Sinclair Davidson wrote a long post responding to Rohan Pitchford’s ERA piece (by the way, welcome to the blogosphere Rohan. I hope we will see many more posts from you). The argument presented was that Australians were being held to international standards and elite standards at that. I must admit that I chuckled at that notion because it is hardly the case that the ERA was at American elite standards. If that were the case, the list of acceptable outlets for top-tier journal publications in economics would be 10 to 15 rather than 100. The A* list might be as high as 8. Real American or International elite standards are not being remotely encouraged by whatever the ERA was doing.
The alternative standard proposed was that of “activity.”
There is the old joke that the associate dean (research) can’t read but can count. During my stint in that position, I came to appreciate that maxim. People who were publishing were working – even if I didn’t understand it, or even think it was useful, I knew they were busy doing something that somebody somewhere thought was good enough to publish. That may well be a low standard, but in an environment of high asymmetric information that standard is a good place to start.
If you are publishing, then that is good enough. The surprising thing about that notion is where it is coming from. It isn’t (I thought) coming from someone who does not believe that performance should be immune to market pressures (especially labour performance). It isn’t coming from someone who (I thought) believed that government money should be allocated to mere activity. It is coming from someone very different.
The best way to evaluate this issue would surely be to compare the allocation of money (and remember that is what we are talking about here) that would arise if it were privately allocated to Universities rather than publicly allocated. And we know the answer to that. The US provides that example. In the private sphere the market bifurcates the industry. We have top Universities in the US where the standards — as I have already pointed out — vastly exceed anything in the ERA. And we have others that do not emphasise research at all where academics teach more and don’t have research expectations. That is a perfectly legitimate outcome. Finally, we have some Universities that fall in another category. My favourite example is the group of economists at George Mason. They have explicitly turned away from ERA like standards to focus entirely on public engagement; and to great effect. I don’t agree with their views but I am betting Deans and University leadership just love them because they bring in the students.
Surely what your right-wing economist in this country would call for is an allocation process that mimicked this system. And you won’t get it from the ERA as the ERA is an internal and not a market-based system. The ERA is designed to tell the boss (who is paying the workers) what the workers are doing with the ton of time that isn’t teaching or administration. To be sure, the answer could be public engagement but the ERA is coming out of the ARC and so you aren’t going to get that answer there. You need another measure to complement it.
Which brings me to the other thing floating around in the ERA debate and implied in the international standards argument — a lack of encouragement to local research. First, high quality, Australian data and policy driven research does get published in A* journals. My own personal experience points to that with work with Andrew Leigh and also other work on electricity markets — wholly about Australia — being internationally successful according to elite standards. I’m not alone. It is surely legitimate for the Government to ask itself why Australian focussed research isn’t of international interest or standards.
Second, it is not clear that international research and notice and Australian policy interest are substitutes. Examples abound — especially amongst Australian bloggers — to the contrary. Personally, I would have long welcomed academic performance reports emphasising public engagement. Those of us who do it have long been unrewarded for it. In my case, there were times I felt actively punished for it. That is not a good state of affairs. But that isn’t a reason to give up on research performance measurement, it is a reason to get better at public engagement performance measure. In many respects that would be pretty darn easy starting with media mentions, blog hits, Government report mentions etc.
Third, if we aren’t going to have the ERA maybe we need to be more upfront with the numbers. Last time I looked the writers over at Catallaxy were big on the numbers. They want cost-benefit analyses for all manner of government expenditures. So maybe we should do it for public money spent on academics. For each academic, we can list the amount of their pay coming directly from the Government and their teaching load, along-side a link to their research and public engagement reports. Then the public can look at those and work out whether they are getting value for money. Now I actually don’t think that would be a good idea (I don’t think the information at an individual level would be of good use although in some aggregated form — like the ERA — it may be). Also, I’m not a CBA at all costs type of person. Finally, even if I did think it was the way to go, as I don’t receive public money for my salary it wouldn’t be fair for me to be sniping from the hills on this. But I do have a question as to why supposed hard-nosed economists aren’t calling for it?
Core Economics 
Good point about the A/A* list being a *very* mild benchmark compared with the US.
I don’t think Australian politicians or the public are aware of the vast differences in standards between Australian academia and the US. To be sure, there are pockets of excellence in Australia, but the average standard is low.
Whenever I such things out to friends, they really don’t want to hear it. They prefer to think Australia is as good in academic research as it is in playing cricket and swimming.
I’m sure if they handed out gold medals for publications in econometrica, AER and the like, our standards would rise rapidly.
But with a strong sentiment of academic protectionism-even among economists who argue for reduced protection in industries outside their own-I think we face an uphill battle.
LikeLike
“The average standard is low”. Who is making these objective judgements about academic standards in Australia Rohan? Or is it just self-hate, cultural cringe bias from someone with narrow views. Or are your views validated by the failed ERA experiment. Given our population size I’ll make the alternative objective judgement that I think we do OK. In all fields Australians seem pretty well represented in academe – in core economics, econometrics and in finance. We just exported Joshua for example. But if your first class of objective judgment is stronger than the second I guess you are entitled to launch a crusade to save us all from bad practices. Good luck – particularly as I won’t be around long enough to be one of your victims.
LikeLike
I’m going to agree with Harry here. Where is the evidence that academic standards are low in Australia? As far as I can see the taxpayer pays for students to be educated and they are getting value for money. Our students get jobs, and good salaries and the number of expat Australians with good jobs overseas shows that too. The large numbers of foreign students coming to study here is also evidence of a good university system – responding to market forces.
Sure the government could ask what its getting for its research dollar. Not much – I have argued elsewhere that public funding to research should be cut to zero. But at the same the amount of money to university research from the government is very low. In my school we get much more money from the private sector than we do the public sector – that is the market at work.
The biggest research challenge facing Australian universities is getting the large number of people who never write anything off their backsides. ERA did nothing for that issue.
LikeLike
If you’re looking for objective measures then how about the competitiveness of job market candidates trained in Australia compared with the US/Canada/Europe? Of those expat Australians with good jobs overseas how many of them had their graduate training in Australia? And what’s the proportion of locally trained v international grads hired by Australian universities? Just anecdotally it seems the best Australian students get the best training overseas, predominantly in the US. The authors on this blog are a case in point.
LikeLike
For what it’s worth, here are some citation figures from the ISI Web of Knowledge’s Essential Science Indicators (the figures refer to the period between 2001 and 2011).
Citations within the field of economics and business:
Australia: 30,889…or 1.5 per 1,000 people
United States: 573,662…or 1.9 per 1,000 people
Citations within the field of “social science, general” (this includes medical and health related social science, as well as fields like sociology and political science):
Australia: 86,173…or 4.2 per 1,000 people
United States: 1,092,757…or 3.7 per 1,000 people
(The per capita figures are based on the fact that in 2005 the population of Australia was 20.395 million while the population of the United States was 295.583 million.)
So on this metric at least Australia seems to be doing well in “social science, general”, while doing less well in economics and business. I’m not sure if I would say there are “vast differences in standards between Australian academia and the US” – you have to remember that not everywhere in the United States is Berkeley or Harvard.
On the other hand, there is always room for improvement.
LikeLike
James,
First citations are no good for assessing immediate impact of research, because of the long lag time in collecting them in many fields. We have to be able to tell our junior colleagues where they should publish to get the best chance of good citations, and we should be honest and advise them to try to publish in the top journals.
The reason why we have to advise them to publish in quality outlets, is that citations have to be quality adjusted to have any meaning. And this argues against your point about Australia doing well in citation terms . If an academic has a large volume of publications on an issue of current policy interest in a B journal, for example, then s/he will get lots of citations from people in B/C journals but not many in higher quality journals. When you do this exercise in economics, the Economic Record (our best journal) has something like 3% the impact of the AER.
Even if you go down to the 50th ranked US department, it will dominate the best Australian departments in most fields. The best economics department (on an old ranking) was the ANU at 62nd in the world.
Economists have thoroughly investigated journal rankings, including impact adjusted rankings, that account for the quality of the journal that cites the work. Unfortunately, we do not come out so well in this ranking. You can see an example here
http://onlinelibrary.wiley.com/doi/10.1162/154247603322752566/pdf
What the journal quality debate is about, is getting incentives right for junior academics, so we can best utilize Australian intellectual capital.
LikeLike
Harry, Sinclair:
Joshua makes a powerful point about conservative academics who nevertheless object to opening up their own profession to more competition. How do you respond to this?
LikeLike
Harry, on the evidence for standards I see two professors arguing that we should eliminate an objective performance measure based on global non-manipulable standards, in favor of highly manipulable local standards and/or panel rankings based on a vague ‘frequency measure’. I can’t recall ever having heard this in any other country. The removal of protection always gets some squeals from industry, but we are better off afterwards.
LikeLike
Rohan – Josh is making a whole bunch of assertions, as you do, but provides no evidence. The Australian university system is very competitive already. We compete for staff and students in international markets. We compete to publish papers in journals. We compete for money from government – not something that I approve of, but people who pay my salary do – and we compete for money from the private sector. When you say that there is academic protectionism I struggle to even understand what you mean. How is this protectionism working? Where does it manifest itself?
To be sure, the incentives are perverse in the system, and I would like to see people doing more and better work. But I have no reason to believe the ERA would result in that occuring. The view that you put forward to Harry ‘two professors arguing that we should eliminate an objective performance measure based on global non-manipulable standards, in favor of highly manipulable local standards and/or panel rankings based on a vague ‘frequency measure’. ‘ (sorry is formatting looks funny there) is factually false. Universities can already work out who does what and where they publish. In my previous life as an administrator I had a list of all staff and their research outputs, research student completions, and revenue they had raised. I also had index scores of their performance as supervisors (how long it took to complete, paperwork done on time, pass rates, etc). My counterpart for teaching had teaching evaluation data and so on. I wasn’t actually allowed to so anything with that information but that was a separate issue. The problem that was going on when I was research dean was that active researchers where being penalised because their work didn’t fit into school clusters (so they were being told to stop doing research and do something else or find another research area). The other thing that happened was that young staff were being told not to publish work in B or C journals, or write book chapters. Why would anyone support a mechanism that lead to incentives to stop academics who were doing research to stop that research while doing nothing to get more academics doing research? That is what you and Josh are supporting.
LikeLike
Sinclair,
Academic Protectionism works via the mechanism that Joshua has described: Asymmetric information between university academic staff on one hand, and the taxpayer, via the government, that funds us, on the other.
We should be accountable to our funding body by doing what we have always claimed we do, which is to publish high quality work.
Yet here we have people arguing that we are should not even be allowed to provide a measure of performance. An argument against letting some sun shine on our activities. The industry in Australia is and has been protected due to lack of scrutiny, and entrenched standards set by entrenched senior academics.
Why would an academic think this was bad unless it threatened that academic’s rents?
LikeLike
We have provided plenty of evidence, and have outlined it in our comments:
1) Standards of output in Australia are low, see my last comment:
http://onlinelibrary.wiley.com/doi/10.1162/154247603322752566/pdf
2) The bifurcation overseas into research universities, and teaching colleges. Unlike here where we have 35 universities all claiming to be research universities. See Joshua’s post.
3) As mentioned, people are squealing about the ERA journal ranking is consistent of rents, and rents being threatened. Along with the other evidence, this relationship looks to be causal.
Additional evidence of low standards
4) Browse the journal output of the 40 to 50th ranked in the US and Europe. Compare journal quality.
5) Do the same with tenure standards, i.e. what did they do to get tenure.
6) Ask yourself why the best Australian academics decide to leave and work in the US and Europe.
I could go on….
LikeLike
The government does not ‘fund’ us* – the government is our single largest customer. There is a huge difference.
When I was working on the RQF (before the ERA) I calculated how much our college would lose if the government 100% defunded our research. The PVC laughed when I told him and said ‘Is that all, we’ll just take another x international students’. I then had to make all the arguments that we’re not doing this for money but reputation etc.
* In contrast to the argument that Harry and I are the protectionists as best I can work out it is the G8 who stands to lose the most by the abandonment of the ranking process. To the extent that money would following rankings and ratings they (including you or, at least, your institution) would, no doubt, get more government money. Now it looks like that limited pot could be shared some other way and the spoils of the regulatory capture that the G8 invested so much in won’t materialise. Rent seekers coming undone – so sad.
I would be more than happy, ecstatic in fact, if Joshua’s proposal above was implemented in Australia – money to be privately allocated to universities and let the chips fall where they may.
After all that, I still don’t understand what this ‘academic protectionism’ is – where are these rents? How are they manifest? All you can point to is an asymmetric information problem – something that is ubiquitous in all relationships.
As I said ” The ERA ultimately failed due to the Hayekian information problem – the government tried to codify implicit knowledge. The top journals in any area are reasonably well-known and the top academics are reasonably well-known too, but beyond that explicit measurement becomes very difficult.”
LikeLike
Instead of arguing at each other – provide a concrete example. How would senior academics – say Harry and I – be any better or worse off under either version of the ERA? I can’t speak for Harry but my working conditions are as follows:
Level E, top of pay scale, teach one subject each semester (85% total satisfaction score this semester), a few grad students, serve on a few university make-work committees, on-going employment – no tenure, large office, no parking space. I am losing the large office when we move buildings and the pay is determined by an EBA negotiated by the union (I am not a member and vote against the EBA each time on principle).
LikeLike
Sinclair,
Rents come from asymmetric information, which is very high between academics and the public that funds us, higher than most relationships.
Rents manifest as follows: We start off in a state where standards are set by senior academics isolated from the global community, setting local standards in such a way as to maximize rents from their positions (unless they have strong preferences to do otherwise). New academics have to follow these standards to succeed, and we get caught in a bad equilibrium of low quality output.
Academic Protectionism is the fight to maintain local standards, so as to gain the rents described, against the introduction of global standards.
LikeLike
I can’t speak for you and Harry with any specific knowledge, but from the outside I would have thought that an ERA would be no threat your positions, and would actually help you to guide junior academics and increase standards of Australian publication. I must admit, I am baffled by Harry’s view. He has published in the AER and ReStud in the past, I believe, so he knows how much harder this is and how much stronger a signal of quality it is than, say publishing in the Australian Economic Review (as useful a journal as that is).
LikeLike
Take a look at Rabee’s post on Politicization of academia. I have thought you’d be very worried about nebulous assessments determined by politicians and less worried about an objective journal ranking.
LikeLike
hc said: “In all fields Australians seem pretty well represented in academe – in core economics, econometrics and in finance. We just exported Joshua for example.”
Yes but I’m not sure that that does anything other than reinforce the point Joshua is making. The work that made him exportable is precisely the work that also put him high on ERA lists. To the extent that you want to talk about the international relevance and recognition of Australian economists, ERA is (while clearly imperfect) a far better attempt at measurement than anything previous or, I would argue, anything you have proposed.
Note that I’m not saying that other measures of policy impact aren’t a good idea or aren’t worthy of implementation and funding. But in terms of international recognition the ERA lists are surely the best measure we have ever had.
LikeLike
Why isn’t the price mechanism taking care of these ‘rents’. Those academics who are no good on the global market have to settle for lower salaries in the Australian system. For your argument to work you have to show that senior academics earn US style salaries without hitting US style publication standards.
Why do you think that Harry and I don’t encourage young academics to publish in the best journals they can? Why do you imagine that Harry and I didn’t know the top ranked journals before the government told us what they were?
I agree there is an incentive problem is the university system but the big issue isn’t that some academics publish in lowly ranked journals but that many academics never publish anything at all. It is appalling (and pathetic) when people with PhDs can say “I can’t do research’ and ‘I need more mentoring’, ‘I don’t have any good ideas’ and nobody calls ‘bullshit’. I don’t place the balme on senior academics, I place the blame of lazy administrators. The ERA does nothing to address that issue while creating all the other perverse outcomes that Harry and I have pointed to.
LikeLike
“ake a look at Rabee’s post on Politicization of academia. I have thought you’d be very worried about nebulous assessments determined by politicians and less worried about an objective journal ranking.”
I did – but the existing rankings were politically biased already. The scraping of the ranking has stymied a nice piece that I had prepared on left-wing bias in academia. I’m normally very suspicious of those sorts of claims, but I did find some nice anomalies that would have caused a bit of a kerfuffle.
More generally, there is always going to be some bias when any government process is involved – best to avoid them in all aspects.
LikeLike
Sinclair you said
“the existing rankings were politically biased already”
I’m not sure how many times I have to repeat this poin, but here goes again:
Incorrect. All Australian professors of economics were invited to submit a ranking, and the result was an aggregation that became the A*/A/B/C list. The list itself was very similar to the European Economics association ranking by Kalitzidakis et al: See
http://ideas.repec.org/a/tpr/jeurec/v1y2003i6p1346-1366.html
LikeLike
Regarding your idea that competition will drive salaries down, good academics and rents away; perhaps. And if true this supports my argument that the rent seeking is damaging.
But there are rents that can be earned by perks such as prestige power travel influence. These become quasi rents associated with the long-term job. Further, lower standards will deter good academics from coming back here, lessening competition for jobs.
LikeLike
BE Journals had A and A* rankings? By pure coincidence some of the people performing the ranking exercise had published in those journals?
LikeLike
“Further, lower standards will deter good academics from coming back here, lessening competition for jobs.”
You keep making the argument that Australian universities have low stabdards, yet you yourself with an Australian qualification was able to complete a US PhD. You yourself publish in top journals. How do you explain yourself?
LikeLike
standards not stabards.
LikeLike