I’m attending a Productivity Commission roundtable in Canberra today on the topic ‘Strengthening Evidence-Based Policy in the Australian Federation’. In an attempt to provoke, my paper is titled Evidence-Based Policy: Summon the Randomistas?. Full text here. I’ll have a month to revise it, so all comments are welcome.
One thought on “More Random Musings”
Comments are closed.
Core Economics 
Most social/economic policy evaluations are what are commonly termed ‘process evaluations’. Investigators collect descriptive data about programs, gather the opinions of various participants, and synthesise this with existing knowledge to come up with recommendations.
The problem with ‘outcome evaluations’ (of which randomised trials are the gold standard) is that they can only test a small number of characteristics. Randomised trials can tell you if the program works, but not how to fix or improve it. Process evaluations, even if flawed, are often the most efficient way to address the latter question.
We need to know the answer to both questions. Those people implementing the program need to know how to improve it. Those deciding whether the program should continue to exist, or be expanded, need to know if it works.
Right now, the main institutions commissioning evaluations are at the implementation level. For them, process evaluations make more sense. Randomised control trials are expensive, too narrow and too late in providing results. Governments (and researchers) need to recognise that outcome evaluations are going to be of most value to those higher up in the decision-making process. If central agencies want to know if programs work, they need to take the lead in demanding randomised outcome evaluations.
LikeLike