An Evidence Hierarchy for Social Policymakers

The Economic Roundup, Treasury’s in-house journal, has just released its first issue for 2009. Evidence is a bit of a theme for the issue, and among the articles, I have one that discusses the idea of a medical-style ‘evidence hierarchy’ for social policymakers.* As one possible evidence hierarchy, I’ve suggested that the following might be used:

A possible evidence hierarchy for Australian policymakers

1. Systematic reviews (meta-analyses) of multiple randomised trials

2. High quality randomised trials

3. Systematic reviews (meta-analyses) of natural experiments and before-after studies

4. Natural experiments (quasi-experiments) using techniques such as differences-in-differences, regression discontinuity, matching, or multiple regression

5. Before-after (pre-post) studies

6. Expert opinion and theoretical conjecture

All else equal, studies should also be preferred if they are published in high-quality journals, if they use Australian data, if they are published more recently, and if they are more similar to the policy under consideration.

Naturally, an evidence hierarchy will always be just a rule-of-thumb. If time is short or the issue is new, theory or before-after studies might be the only thing that’s available. But for decision makers who are choosing between a large number of studies, an evidence hierarchy might help discern the wheat from the chaff.

* By ‘social policymakers’, I mean people who formulate social policy, as distinct from those who make policy with their friends over a few beers.

One thought on “An Evidence Hierarchy for Social Policymakers”

  1. Hi Andrew,

    Though randomised trials/natural experiments are one approach to analysing social policies, one wouldn’t want to rule out analysis based on structural models (such as the work done by Michael Keane) which have different advantages (like considering conterfactuals and dealing with dynamic issues).  Stephen King’s paper at the Industry Economics Conference at La Trobe a few years ago could provide some useful guidelines for this sort of evidence.



Comments are closed.

%d bloggers like this: