The limits to evidence based policy.

Evidence-based policy is a buzzword that conjures up images of responsible government: difficult decisions taken after a careful examination of the evidence, tailored local experiments, and then implemented using the best advice available. Sounds good, no? As a buzzword, it is a clear winner and something we all want more of.

But how much can we really expect of this buzzword and what will it actually lead to? Let’s have a look at the dangers and benefits.

The dangers and limitations

The first thing to note about the buzzword is that its main association is with rational control. In order to have evidence based policy, you must gather evidence and have a policy process. That means hiring more bureaucrats to do the gathering and the processing. Hence the buzzword is first and foremost a means for a bureaucracy to get their hands on more resources.

The buzzword also works to legitimise more power to a centralised bureaucracy: if you are going to have evidence-based policy, you must be able to implement the centrally-decided policy. Hence it is an excuse to take independent decision-making power away from local actors to the benefit of a supposedly rational centre. One might note the inherent contradiction here: with less power to local actors comes less local experimentation, meaning less policies are tried out and less possibilities for learning. Evidence-based policy is hence an engineer’s view of how an organisation learns (top-down). An alternative view could be that an organisation learns by following successful examples within the own organisations, which is a more organic but far less ‘evidence-based’ view. One might not know why something works and have no evidence that it will work in other places, but by simply following previous successful examples one can nevertheless reasonable hope to improve over time. That is how evolution works (what accidentally happens to fit its environment gets to procreate) and how many markets work: not by a rational gathering and analysing of information but by an almost blind mimicry of accidental success.

These initial thoughts mean that adopting the word ‘evidence-based’ policy implies increased bureaucracy, increased centralised control, and decreased local experimentation.

Then, let us consider the limits of ‘evidence’. New events that have no clear precedent but to which one must react are clearly outside the scope of evidence. What is one supposed to do then, wait until evidence is gathered? For instance, the recent Global Financial Crisis had no clear known precedence. It only vaguely resembled previous recessions. Should this then mean that all kinds of policies that were not implemented before and for which there was no evidence should not have been implemented? Should there have been no fee for the bank guarantee? Should we have stopped migration? Should we have started government work programmes (which were deemed so successful during the Great depression)? Or should we have done nothing for 50 years during which we gather more evidence?

It is clear that in the case of new major shocks hence, evidence-based policy has no meaning. We react to these based on vague rules-of-thumb and a vague understanding of how the whole economic system behaves. Such vague knowledge, based on vague historical evidence, can hardly be called evidence-based, but it is all we have to go on in such situations.

Then, onto the plus side:

The usefulness of evidence

‘Evidence’ to aid better policy is of most use in areas where government involvement is long-term, where the number of alternative policies is large, and where the issues concern complicated systems for which it is basically impossible to tell which policy will work in advance.

What areas are we then talking about? Primarily we are then talking about the delivery of major services like education, health, law enforcement, taxation, and social security. How to organise universities, schools, hospitals, medical insurance, tax payments, police forces, etc. are things for which there is a lot of time available. These services will still be here in 50 years time and it will be up to the government and its bureaucracy to determine how to organise its delivery. It is there that best-practise, discovered anywhere in the world, can in principle be identified, experimented with, and enforced.

The questions on which evidence can be generated include the question of whether we should have a voucher system for education. Should hospitals be paid on the basis of the diseases the patients comes into the hospital with or the number of actual procedures? Should we introduce an automated ‘default’ for tax returns? Should the internal affairs of a police force be connected to outside scrutiny (such as independent legal institutions) or should they be self-contained? Should teachers be paid according to the progress their students make or should they be paid a fixed rate in order to keep the peace within the teachers’ guild? Does harassment of people in welfare make them more likely to find jobs or make them more entrenched?

Questions like these are almost unknowable beforehand, but once they have been tried in a variety of settings, we can hope to learn which one works.

Do’s and dont’s in evidence-based policy.

A couple of obvious points come out of the above for how, in the best of worlds, ‘evidence-based’ policy can be institutionalised. One is that in order to learn about the evidence, major spending departments must have an analytical unit that retains the memory of previous ‘evidence’ and of ‘best-practise’. Such an analytical department must be somewhat separated from the political process, because governments react to the madness of the day and hence by design cannot really react to evidence most of the time. The knowledge of best-practise must thus be insulated against the political system and be allowed to come forward as independent advice every time a politician of a different colour asks for it. Partially, this knowledge is kept outside government entirely, such as when it is housed inside universities. For major spending areas though, the level of specialisation is so immense that it would seems a good idea to have independent research institutes inside particular departments.

A second point is that in order to learn from experiments, there have to be incentives to have open experiments. At the moment, nearly all the incentives of politicians and departments are towards secrecy, particularly for governments who have been in power for a while. This is because of the lop-sided nature of rewards in the competitive world of politics and senior civil service: any verifiable mistake you are associated with is used to crucify you by your opponents, whereas any ‘success’ has many fathers and usually belongs to the previous administration. Hence in order to create incentives, governments partially have to bind their hands via policies on openness (so that outsiders can criticise bad policy on the basis of internal data), would benefit from having scientific units inside departments who would have a vested interest in experiments, and should perhaps also bind their hands in terms of having regular open advise from outside scientific bodies, such as, in Australia, the Productivity Commission.

A third point is that evidence based policy would have to be kept out of those areas which are not long-term, clearly recurring, and complicated. In the areas of macro-economics and trade, for instance, it makes little sense to demand evidence-based policy and it instead is better to have individuals well-schooled in economics and in the history of macro-economics running around the policy makers.

Evidence based policy also needs to be kept from affecting the overall balance of power until there is a clear case for something, i.e. no jumping the gun. If you do jump the gun in those cases where that alters the balance of power, it will be almost impossible to reverse course because you will have created a political ally for the new situation. In the case of teaching for instance, one should first have experiments on teacher-pay and league-tables before setting up a huge measurement bureaucracy to enforce such policies nation-wide. Once in place, that bureaucracy will be very hard to get rid of.

On balance

Evidence-based policy is useful in a political culture that values reasoned input and where the balance of power between various actors (commonwealth, states, councils, business, etc.) is already clearly defined. In a political culture where the balance of power is not clearly defined however, it would be naive to expect the buzzword of ‘evidence-based policy’ to lead to better policy. More likely, it will be used to justify centralisation and the expansion of the civil service.

In the case of Australia, where the commonwealth is slowly gobbling up power from the states, evidence-based policy will probably mean centralisation. Real evidence will have little to do with it.

Author: paulfrijters

Professor of Wellbeing and Economics at the London School of Economics, Centre for Economic Performance

6 thoughts on “The limits to evidence based policy.”

  1. “A third point is that evidence based policy would have to be kept out of those areas which are not long-term, clearly recurring, and complicated. In the areas of macro-economics and trade, for instance, it makes little sense to demand evidence-based policy and it instead is better to have individuals well-schooled in economics and in the history of macro-economics running around the policy makers.”

    That statement is just plain weird.

    Monetary policy is extremely evidence based. Central banks spend enormous time and resources gathering evidence about the impact of monetary policy decisions on the economy, how the economy responds to particular shocks, and what type of monetary policy framework is most conducive to fulfilling its mandate of price stability (or whatever other objectives governments set for them).

    The same for fiscal policy. What evidence do we have about multipliers? Is it welfare enhancing to let automatic stabilisers work. How distortionary are particular taxes? Etc, etc.

    And trade policy should also be evidence based. What is the likely net effect of a particular FTA. Do FTAs divert or create trade? Under what circumstances? If we lower such and such a tariff by X, what impact will this have on welfare. How will that impact be distributed?

    I really don’t know what you are getting at here.

    Evidence based doesn’t mean not being also guided by theory. Or that experience is irrelevent.

    And it doesn’t have to mean more centralisation. For example, the best evidence might be that some sort of competition amongst fiscal authorities within a federation is useful. Or that some policies tend to be better constructed and implemented by lower level governments because they are closer to the needs of the citizenry.

    The problem is not with the concept of evidence based policy. It is that in many circumstances the current government has used it as a buzzword without following through. So, for example, we had the Productivity Commission being shut out of reviewing manufacturing assistance.


  2. LO,
    you and I normally agree more than this. The piece above is mainly about the political economy of the word evidence-based, and hence how it is used to change the current set of arrangements. As such a discussion of many of the existing arrangements that will almost certainly be unaffected, like the RBA, is a little off-topic.
    However, on the topic of monetary policy I think my description of policy being based on somewhat vague historical information and a vague understanding of the whole system is far more apt that true ‘evidence based’. Where are the monetary experiments? Where is the evidence that inflation targeting works better than, say, aggregate wage targeting or monetary base targeting? It is simply not the case that our monetary policy is based on great evidence. It is more the case that we go along with what seems to work, but continue to scan around so that we can change tack as soon as the situation changes a lot. The last GFC was, for my taste, a textbook case of thinking-on-your feet based on vague historical knowledge. The notion that it was all evidence based is simply not true.


  3. Paul
    I guess I take evidence based to mean making use of the best available evidence. Policy analysis always has to deal with the ‘identification’ problem that derives from the absence of clean counterfactuals. But the fact that it isn’t perfect doesn’t mean that it doesn’t have an evidence base. Am I completely happy with the evidence base supporting monetary policy frameworks, fiscal responses, trade policy, minimum wage setting, climate change mitigation, indigenous interventions, etc? No. But I’d much rather a government that was prepared to look dispassionately at the available evidence as an input into decision making, or was prepared to put in place proper programme evaluation frameworks to test policies ex-post, than a government eschewing these things.
    But perhaps we are at cross-purposes here!


  4. LO,
    we clearly want the same thing. After all, who on earth would advocate we should decide policies on blind ignorance? Yet, the ‘reform’ aspect of the evidence-based movement carries the supposition that whatever it replaces it is not so great and is less smart and effective than what will be done in the future. That implicit supposition is where the rabbit goes into the hat and where it makes sense to have a good look at the rabbit before it is pulled out of the hat and we all admire the conjurer.


  5. “evidence-based policy will probably mean centralisation”  I don’t understand either. Do you mean that someone will decide ‘best practice’ and this will be used as a lever to control other governments? If so, maybe it is not so bad (if indeed it is ‘best practice’).
    I would like to see a committment to _gathering_ evidence as a part of good evidence-based policy (disclosure, I work in a research centre, so this is my job). To often I see policies being implemented where there are good reasons to think that they won’t work as intended, but there no committment to gathering good evidence to check this. Of course, part of the reason for this is that the evaluation may well come up with ‘wrong’ answer – and the better the evaluation the harder it is to dismiss.


  6. “Vertical fiscal imbalance” and the Commonwealths insistance that it should decide how tax revenue is spent will lead to centralisation. “Evidence based policy”, or the lack thereof, is simply one of the means they will use for withholding money from the States and Territories.


Comments are closed.

%d bloggers like this: