Best practice governance of top academic departments

Over the last 15-20 years academic school meetings have gone from rambling and unstructured brawls to dull “executive infomercials”. The former led to marathon meetings. The current model has led to a middle-management culture that often does not take advantage of the very valuable specialists skills of talented, highly trained (and experienced) scholars in the department. Nor does it allow for reasonable checks and balances on the powers of the executive–something that is vital for the management of any group of academics.

Our school at the ANU has adopted a system informed  both by ANU’s academic board and how the world’s top economics departments operate. The following is the template that we have adopted and which has replaced the executive committee.

 

School Meetings

1. Members of the School Meeting are all academic staff at level B and above who are engaged equal to or greater than 50% FTE.  Observer status is given to all other staff in the School.
2. School Meetings are chaired by the Speaker of the School Meeting, who is chosen by the Head of School. The Speaker is a Member of the School Meeting who is not a Professor nor a Chair of one of the Committees nor the Head of School.
3. School meetings shall be held at least twice every semester during the academic year. Additional meetings may be called by the Head of School  or at the written request of twenty-five percent of the Members of the School Meeting. One-half the Membership of the School Meeting constitutes a quorum.
4. Meetings shall be conducted in accordance with Robert’s Rules of Order. Minutes of the meeting and reports submitted to the Meeting shall be kept and made available to the Members.
5. The initial Agenda for regular School Meetings shall be prepared by the Speaker and circulated in written form at least five calendar days prior to the meeting. The initial Agenda includes the items 1. Apologies 2. Confirmation of Minutes of Previous Meeting 3. Reports of Head of School and the Chairs of Committees 4. Workplace Health and Safety 5. Other Business 6. Items added by the Head of School.

 

6. Additional items may be suggested by individual Members and, at the discretion of the Head of School, be added to the agenda for the forthcoming meeting. Alternatively, items may be placed on the agenda by written petition of twenty-five percent of the Members. All such additions to the agenda must occur at least three days prior to the meeting.
7. A simply majority of those present and those sending proxy votes and/or absentee ballots shall decide an issue arising from an item in the initial agenda or an item added at the discretion of the Head of School.
8. A majority above 60% of those present and those sending proxy votes and/or absentee ballots shall decide an  issue arising from the agenda item not described in 7.
9. A majority above 70% of those present and those sending proxy votes shall decide a motion not arising from an issue on the agenda.
10. Proxies may be given from one member to another. Proxies are given in written form to the other member a copy of which must be received by the Speaker three calendar days prior to the meeting. Proxies cannot be specific to particular items on the agenda nor to a particular motion/amendment not arising from the agenda.

 

 

 

The Importance of Peer-Review in Journal, Department, and Individual Research Rankings

Preamble

I recall that some time in the mid 2000s, when the Research Quality Framework (which preceded the current ERA) was being discussed, Arden Bement, the director of the National Science Foundation, was asked what he thought. He responded as one would expect of a serious researcher, by saying that the only method he knows for judging academic outcomes is peer review.

In fact, Peer review is the gold-standard in science. We simply don’t trust any finding, method, conclusion, analysis, study that is not reported in a peer reviewed outlet. Yet there has been rapid growth in the use of indices in ways that have not been tested through peer review and which are being used to measure journal ranking, individual academic performance, and even the standing of departments.

Here, I argue that we should return to published methods that have been tested through peer review.  The unchecked use of non-peer reviewed methods runs the risk of misallocating resources, e.g. if university promotion and appointment committees and bodies like the ARC use them. Even more troubling, is that non-peer reviewed methods are susceptible to manipulation; combined with the problem of inappropriate rewards, this has the potential to undermine what the profession, through peer-review, regards as the most valuable academic contributions.

Economics journal rankings

In the last two decades or so, there has been an explosion in the use of online indices to measure research performance in economics in particular (and academia generally). Thomson-Reuters Social Science Citation Index (SSCI), Research Papers in Economics (RePEc) and Google Scholar (GS) are the most commonly used by economists.

These tools display the set of publications in which a scholar’s given article is cited. While SSCI and GS take their set from the web as a whole, RePEc––hosted by the St Louis Fed–– is different, in that it refers only to its own RePEc digital database, which is formed by user subscription. Further, RePEc calculates rankings of scholars, but only of those who subscribe. Referring to its ranking methods, the RePEc web page states:

This page provides links to various rankings of research in Economics and related fields. This analysis is based on data gathered with the RePEc project, in which publishers self-index their publications and authors create online profiles from the works indexed in RePEc.

While it has been embraced by some academic economists in Australia as a tool for research performance measurement it is important to note that the RePEc ranking methodology is not peer-reviewed. This departure from the usual strong commitment to the process of peer-review by academics is puzzling, given that there is a long history of peer review in economics in in the study of, you guessed it “Journal Ranking”.

A (very) quick-and-dirty modern history

Coates (1971) used cites in important survey volumes to provide a ranking; Billings and Viksnins (1972) used cites from an arbitrarily chosen ‘top three’ journals; Skeels and Taylor (1972) counted the number of articles in graduate reading lists, and; Hawkins, Ritter and Walter (1973) surveyed academic economists. (Source: Leibowitz and Palmer JEL 1984, p78.)

The modern literature is based on a paper by Leibowitz and Palmer in the Journal of Economic Literature,1984. In their own words, their contribution had three key features

…(1) we standardize journals to compensate for size and age differentials; (2) we include a much larger number of journals; (3) we use an iterative process to “impact adjust” the number of citations received by individual journals

Roughly speaking, the method in (3) is to: (a) write down a list of journals in which economics is published, (b) count up the total number of citations to articles in each journal; (c) rank the journals by this count; (d) weight the citations by this count and, finally; (d) iterate. The end result gives you a journal ranking based upon impact-adjusted citations.

The current best method, is Kalaitzidakis et al Journal of the European Economics Association, 2003, hereafter KMS. This study was commissioned by the European Economics Association to gauge the impact of academic research output by European economics departments.

KMS is based on data from the 1990s and, as far as I am aware, has not been updated. No ranking can replace the wisdom of an educated committee examining a CV. However, KMS at least comes from a peer-review process. Unlike simple count methods, it presents impact, age, page and self-citation adjusted rankings, among others.

But even KMS-type methods can be misused: One should be ready to use the “laugh test” to evaluate any given ranking. KMS deliberately uses a set economics journals, roughly defined as journals economists publish in and read. It passes the laugh test because, roughly speaking the usual “top five” that economists have in their heads (AER, Econometrica, JPE, QJE and ReStud) do indeed appear near the top of the ranking, and other prestigious journals are not far behind.

The Economics Department at Tilburg University has included statistics journals in its “Tilburg University Economics Ranking”. The result? “Advances in Applied Probability” beats out the American Economic Review as the top journal: Their list can be found at https://econtop.uvt.nl/journals.php, but you need look no further than their top five to see that this does not pass the laugh test:

  1. Advances in Applied Probability
  2. American Economic Review
  3. Annals of Probability
  4. Annals of Statistics
  5. Bernoulli

Would I be remiss in suggesting that a statistics-oriented econometrician might have had an input into this ranking? Yes I would oops!

Finally, let us turn to the new RePEc impact-adjusted ranking. A laugh-test failure here, among others, is the inclusion of regional FED journals: Quarterly Review, Federal Reserve Bank of Minneapolis is ranked 14–just above the AER; the Proceedings Federal Reserve Bank of San Franscisco is ranked 16 ahead of the Journal of Econometrics at 19 and; Proceedings of the Federal Reserve Bank of Cleveland is 24, ahead of the European Economic Review at 29.

The RePEc top 5 is:

  1. Quarterly Journal of Economics
  2. Journal of Economic Literature
  3. Journal of Economic Growth
  4. Econometrica
  5. Economic Policy

It would be interesting to investigate whether  macroeconomists and policy scholars had influence here.

My conclusions

If we are going to use ranking methods be very careful. Use methods that have emerged over decades of rigorous peer review, like the European Association’s 2003 study by KMS. And stick to their method rigorously lest we all have to retrain in statistics.

Mandatory Economics for US Supreme Court Justices

That mandatory health insurance coverage makes sound economic sense, and has had broad support from economists of all persuasions for decades does not seem to be enough to sway Judges John Roberts, Antonin Scalia and Samuel Alito of the US Supreme Court. Alarmingly, Scalia does not seem to understand the difference between consumption of insurance—the payment in advance for coverage from ex-post unaffordable catastrophic events, and the consumption of broccoli—something cheap that is enjoyed immediately. All that seems to matter to him is a vague argument that compulsion is government overreach.

 

What kind of sophistry of argument they have to invent when the Pandora’s box of analogous cases hits their doorstep? The US, like other countries, mandates that its citizens pay for their military. US citizens cannot insist on reducing their tax bill by the share of military spending in government expenditure. As with health care from emergency departments, citizens know they will be protected by their military regardless of whether they pay for it. Directly analogous is the provision of roads, public hospitals and schools, the cost of courts, Supreme Court Justices salaries, police and public servants, etc. Citizens are forced to pay for these things also and are not allowed the freedom to opt out precisely because we know they cannot realistically be excluded from benefiting.

 

Even if it is a stretch for these Justices to understand the logic behind these things, I wonder what will happen to the compulsory auto-insurance that many states in the US impose on their citizens. Can States in the US do unconstitutional things like that?

 

If the US Supreme Court rejects the Affordable Health Care Act on the basis of the mandate to purchase, it will not only be a triumph of legal sophistry over economic logic, but an act of the most extreme judicial overreach in that Court’s history.

Spence’s “Sydney Sackings” Signal Time for Tenure Nigh

The legacy of history is a powerful factor in the development of Australia’s university system. Originally incorporated in 1850 under the University of Sydney Act* “for the better advancement of religion and morality, and the promotion of useful [my emphasis] knowledge” (section I of the Act), the University of Sydney primarily functioned to educate the Colony’s future leaders in the professions Law, Medicine and Arts.  In 1954 an Act of Parliament incorporated Sydney Grammar School as a pre-vocational feeder school for the University of Sydney.

What is so interesting about this history is the sheer persistence of this original culture; the feel of the University of Sydney as an extension of the private school system in New South Wales, aimed at supplying training for the merchant class, is palpable.

Continue reading “Spence’s “Sydney Sackings” Signal Time for Tenure Nigh”

Where to next?

I recently learned that some of our colleagues now refer to themselves as “Global Economists”. I wonder which graduate school one goes attends to major in this field–I dont see it in the JEL other than as a subfield if International called “global outlook”. Am I being unfair at wondering if “Galactic Economist” is next?

Getting into a top economics PhD programme

For many decades, graduates of Australian bachelor’s degrees in economics have been making the trek to the best PhD programs the world has to offer such as Harvard, Princeton, LSE. This is a testament to the quality of our undergraduate education. But to get into a top PhD program more is required than good training. Here are  some factors/advice I think are important:

1. First class honours in one of the top Australian honours programs; quite
often with a joint degree. If you don’t get a first class honours degree
you will probably be out of the running for a place in one of the top
ten PhD programs.

2. Letters from academics who are known at the institutions to which
the students are applying and who know how to write a  reference. It
makes sense to plan your honours degree by choosing a supervisor who
is internationally known, who knows academics in top programs, and who
has had some success  with placing students in good PhD programs.

3. Thesis Topic. You will probably need to do a solid thesis in the
honours year. Something that highlights your analytical abilities and
your potential to do research at the highest level. Don’t take
shortcuts do a thesis related to well known literature.

4. Well Written Thesis. In most instances you will be sending your
thesis with your application. Make sure that it is well written and
that the thesis is typographically appealing.

5. Top GRE scores. You’re going to have to do the GRE tests and do
well in them. You don’t usually need to do the economics GRE. You
should practice and my advice is for you to travel to Sydney or
Melbourne to do the computerized tests in November.

6. Advice from academics who are “in the know” about the US, and
hopefully have gained this knowledge from having spent time at an
international institution with a good program. When writing up an
application show it to someone who knows how admission committees
work.

7. Send your applications to many places and keep an open mind. It’s
very unlikely that all the top places will offer you a place so you
should apply widely.

8. Offers start coming in  in early March. Stay on top of things and
check your email regularly. Some places may put you on reserve for
funding. They will need to know if you are still interested in them
and will probably contact you in April. The admissions process for US
institutions ends in early May.

9. Take the opportunity to visit. If Harvard and MIT offer you a
place, then you should try and visit them in April. This is not so
that you can negotiate a larger fellowship but it will help you decide
which institution you want to join.

10. Getting into a top PhD program is hard. You should start planning
early in your undergraduate degree.

Academic Protectionism

Following up on Joshua’s post. It takes a lot of effort to do good research and even more to get the top researchers in your field to be sufficiently impressed by your work that they recommend its publication in a leading international journal. For scientific research, these leading journals include ‘Nature’  and ‘Science’.  In statistics they include the Annals of Statistics, JASA and a few others. Physics, in addition to Nature and Science, has Physical Review. In the area of economics there is international consensus on which journals are ranked in the top five, and which belong in the top thirty.

Australia is academically a very small country. What that means is that the research activities of Australian-based academics have no perceptible effects on the regard that the international academic community has for the various publication outlets. In that sense, we are price takers.

When an Australian academic manages to publish in a top journal, the signal is clear: The work has been deemed by leading members of the profession to be excellent and as good if not better than their own.  Such publications provide the academic with significant international exposure and scholarly rewards.

For decades Australia’s academic community has been operating in essentially a protected industry not accountable to international norms and not accountable to the relative values that the international profession places on scholarly achievements. In the area of economics, research performance in many wealthy departments has been well under par by international standards. The protection afforded by lack of accountability meant that individual professors in some departments formulated ad-hoc incentives that were at odds with international standards. There were exceptions. From time to time we saw the emergence of excellent departments such as Melbourne, UNSW and the ANU where top researchers got together and set up incentive structures in line with international standards.

ERA in its first round adopted journal rankings in economics that are more or less is in line with international standards. In doing so it brought to bear values that much of the Australian academe had ignored. It opened up the university sector to international competition and made universities accountable to the values in international market for ideas. As with the removal of other forms of protection, this change has brought a barrage of complaints. This is not to say that there aren’t aspects of the ERA that warrant complaint. However, it is consistent with the notion that some of the rents from a comfortable life and a lack of scrutiny are being threatened.