The Economic Roundup, Treasury’s in-house journal, has just released its first issue for 2009. Evidence is a bit of a theme for the issue, and among the articles, I have one that discusses the idea of a medical-style ‘evidence hierarchy’ for social policymakers.* As one possible evidence hierarchy, I’ve suggested that the following might be used:

A possible evidence hierarchy for Australian policymakers

1. Systematic reviews (meta-analyses) of multiple randomised trials

2. High quality randomised trials

3. Systematic reviews (meta-analyses) of natural experiments and before-after studies

4. Natural experiments (quasi-experiments) using techniques such as differences-in-differences, regression discontinuity, matching, or multiple regression

5. Before-after (pre-post) studies

6. Expert opinion and theoretical conjecture

All else equal, studies should also be preferred if they are published in high-quality journals, if they use Australian data, if they are published more recently, and if they are more similar to the policy under consideration.

Naturally, an evidence hierarchy will always be just a rule-of-thumb. If time is short or the issue is new, theory or before-after studies might be the only thing that’s available. But for decision makers who are choosing between a large number of studies, an evidence hierarchy might help discern the wheat from the chaff.

* By ‘social policymakers’, I mean people who formulate social policy, as distinct from those who make policy with their friends over a few beers.