Fewer children left behind

My AFR op-ed today (competing with the ascendancy of a certain ex-monk), is on the impact of US school reforms on student achievement. Full text over the fold.

Mixed Results for School Tests, Australian Financial Review, 1 December 2009

Some of George Bush’s best verbal slips came when speaking about education. “Is our children learning?”, the former President asked one audience. “The illiteracy level of our children are appalling”, he warned another group of bemused listeners. According to Bush, the remedy was clear: “childrens do learn when standards are high and results are measured”.

Signed into law in January 2002, No Child Left Behind (NCLB) was the largest reform US education had seen in over forty years. Like the schooling reforms implemented through COAG by Kevin Rudd and Julia Gillard, the centrepiece of NCLB was standardised testing and public reporting. So understanding the impact of the US reforms may provide a window into the future of Australian schools.

At the heart of NCLB are two requirements: all public school children must be tested annually from grades 3 to 8, and states must report on whether each school is making ‘adequate yearly progress’. The reporting requirements apply not only to the student body as a whole, but also to particular sub-groups: including six racial minority groups, pupils with limited English proficiency, and children from low-income families.

According to its proponents (among them the late Senator Edward Kennedy), public reporting creates stronger incentives for teachers and school administrators to fix problems, encourages educational innovation, and pushes schools to devote more attention to the neediest pupils. By contrast, NCLB’s detractors claim that it devalues non-tested subjects such as art and music, and creates an incentive to focus too much energy on underperforming students (possibly to the detriment of high achievers). Some even darkly allege that publicly reporting test scores might pressure teachers and administrators into falsifying students’ exam results.

In a new study, Thomas Dee (Swarthmore College) and Brian Jacob (University of Michigan) have produced the leading assessment to date of NCLB. The key to the Dee-Jacob analysis is that some states (such as Texas and North Carolina) already had stringent accountability regimes in place. Earlier work has shown that those states which adopted accountability measures in the 1990s tended to have faster growth in student achievement. But NCLB had little impact on the early adopters. So in the Dee-Jacob analysis, states like Texas and North Carolina are used as a ‘control group’, and compared with ‘treatment group’ states who only adopted accountability measures when forced to do so by NCLB.

To measure the impact of the reforms on student achievement, Dee and Jacob use results from the National Assessment of Educational Progress; a test that allows comparison across states and over time, but which is not reported at a school level. Since schools’ incentives are tied to state-based tests, improvements on the national exam ought to reflect substantive learning, and not ‘teaching to the test’ (or ‘cheating to the test’).

Looking across grades and subjects, the Dee-Jacob analysis finds large positive impacts on mathematics, particularly at the primary school level. However, the researchers find little evidence of any impact (positive or negative) on reading scores. This may reflect the general finding that schools can have more influence over numeracy than literacy (since parents are more inclined to read books with their children than they are to do maths puzzles).

Evaluating one concern about NCLB, Dee and Jacob also look at test scores at the top and bottom of the distribution. Reassuringly, they find no evidence that the reforms caused a decline in performance by the best students.

Due to a lack of data, they are unable to delve much inside the ‘black box’, so it is still unclear precisely how public reporting of test scores led schools to improve their performance. A critical next step is to find out precisely how schools responded to NCLB: did they change the curriculum, lengthen the school day, vary their hiring and firing policies, revamp administration, or do something else entirely?

Of course, some will argue that because Australia outperforms the US on the international PISA tests, we should ignore everything that is happening in their schools. Hubris aside, this misses the fact that US students surpass their Australian counterparts on TIMSS, the other major international exam. According to the international league tables, we beat them on broad conceptual skills (PISA), but they outshine us on curriculum-based knowledge (TIMSS).

Perhaps neither side of US politics would like to admit that the result of their 2002 schools reforms is neither disastrous nor transformative, but something in between. Following NCLB, the US has ‘fewer children left behind’. If the same can one day be said of Australia’s school accountability reforms, Rudd and Gillard can be quietly pleased.

Andrew Leigh is an economist in the Research School of Social Sciences at the Australian National University.