In a recent article published in The Journal of Economic Perspectives (Vol 28. Nr 3, summer 2014), Conley & Oender follow up on an earlier paper published the year before with others in Economic Inquiry.

In their earlier contribution, drawing on a panel dataset of 14,271 individuals that were awarded Ph.D.s between 1986 and 2000 in U.S. and Canadian economics departments, Conley & Oender investigated how increases in publication delays have affected the life cycle of publications of recent Ph.D. graduates in economics. They found evidence that productivity has significantly diminished in recent relative to earlier cohorts, with productivity being measured by the number of AER-equivalent publications. The result, while intuitive and in line with other evidence such as increasing publication lags , was not robust: when the number of AER-equivalent pages was used instead diminished productivity was less evident.

Another finding, and arguably the more interesting one, was that research publications were yet another extreme example of positive skew, i.e., a reverse J distribution: The top 1 percent of publishing research economists across their whole sample produced 13 percent of (quality-adjusted) research output, and the top 20 percent produced 80 percent of it.

In their new paper Conway & Oender, drill deeper in their data. Their key question is whether the skewness is due to stratification: Is it the top departments that produce those productive scholars, as one might intuit? Well, turns out, the story is rather more fascinating:

“Our evidence shows that only the top 10 – 20 percent of a typical graduating class of economics PhD students are likely to accumulate a research record that might lead to tenure at a medium-level research university. Perhaps the most striking finding from our data is that graduating from a top department is neither necessary nor sufficient for becoming a successful research economist. Top researchers come from across the ranks of PhD-granting institutions, and lower-ranked departments produce stars with some regularity, although with lower frequency than the higher-ranked departments. Most of the graduates of even the very highest-ranked departments produce little, if any, published research. Indeed we find that that PhD graduates of equal percentile rank from certain lower-ranked departments have stronger publication records than their counterparts at higher-ranked departments.” (p. 205 – 206) The authors identify those institutions in their article.

In other words, rankings or pedigree is only so important and can be seriously misleading. And, “if the objective of graduate training in top-ranked departments is to produce successful research economists, then these graduate programs are largely failing.” (p. 212) And, “For graduate students in economics (and also potential graduate students, AO), the message is that becoming a successful research economist is difficult. The good news is that one does not have to go to a top department in order to become a successful research economist.” (p. 212)

An interesting by-product of their research is that publishing a paper before graduation was uncorrelated with productivity up to the tenure decision. (p. 214) If true, and it certainly seems rather counter-intuitive, the signal extraction problem would be even harder.

It is important to repeat that Conley & Oender draw on a panel dataset of individuals that were awarded Ph.D.s between 1986 and 2000 in U.S. and Canadian economics departments. The changes in the economics profession have been considerable in the last couple of decades and, while it is likely that the key findings reported in their paper (reverse J distribution of research publications, coming from a top place being no quality guarantee) are likely to survive a re-analysis with seriously updated data, the finding of the correlation between publication before graduation and productivity up to the tenure decision might not. It’s not called “publish or perish” for nothing.

In sum, signal extraction in economics is hard. Very hard indeed.

In other disciplines, too.