Do Teachers Matter?

A psychology study hit the headlines last Friday under the banner ‘Teacher quality makes little difference, study shows’.

AN AUSTRALIAN study has cast doubt over the “teacher effect”, by suggesting differences between teachers play only a minor role in how well a child will learn.

The global study, led by the University of New England, monitored 500 pairs of identical twins during their first three years of school.

They were divided into two groups: twins who shared the same teacher, and siblings who were split across different classrooms.

The same genetics and home life ensured the twins had the same ability to learn. The study was designed to reveal any differences linked to their teacher alone.

Professor Brian Byrne said the finding contradicted the views of educationalists who claimed teacher quality could account for a variance of up to 40 per cent in a child’s learning outcome.

“Our study shows … the ‘teacher effect’ on differences in children’s acquisition of literacy skills in the early years of schooling is no more – and sometimes less – than 8 per cent,” Professor Byrne said.

“This result is certainly in the direction you would expect if teachers do ‘make a difference’ but it is not very large.

Curious to learn more, I wrote to the lead author, Professor Brian Byrne at UNE. Within a few hours, Brian had emailed me back a copy of the study. He has asked me not to post the text, since it is still forthcoming at the Journal of Educational Psychology. Given that teacher effects are a pretty important policy issue, I thought it might be worth posting a handful of thoughts on it.

  1. A multi-country study that follows twins is very cool. The logistical work required to pull together something like this (including administering bespoke tests) is extremely impressive.
  2. In the study, though not so much in the ensuing media, the authors acknowledge that they cannot separate teacher effects from classroom effects. This is important, since the headline from these results could just as easily have been “class size doesn’t matter”.
  3. The authors spend quite a bit of time discussing the education literature on teacher quality, but less on the economics literature on teacher quality. Economists have indeed separated class size effects from teacher quality effects, and their conclusion on the share of variance explained by teachers is around 2-5% in this study, and up to 16% in this study.
  4. Numbers like 2%, 8% and 16% may sound small, but I don’t think we should be surprised to learn that family background is very important. The question is: are teacher quality effects large enough to affect children’s life chances? Two examples: (1) these numbers imply that a 90th percentile teacher is twice as effective as a 10th percentile teacher; and (2) according to my own Australian results on the dispersion of teacher quality, assuming that the impact of having a more effective teacher persists over time, and that Indigenous children typically get teachers at the 25th percentile, these results imply that the black-white test score gap in Australia could be closed in five years by giving all Indigenous pupils teachers at the 75th percentile.
  5. While twins represent a novel way of decomposing teacher effects, they do have their drawbacks. In particular, one might think that having a same-aged sibling creates the possibility of spillovers: if you have a lousy teacher and your twin brother has a great one, wouldn’t you use his notes when doing homework? If true, this will bias downwards the estimated share of variance explained by teacher differences.

I emailed the above comments to Brian last night, who speedily responded (on his birthday!) with the following.

Brian Byrne: Comments on Andrew Leigh’s comments

Andrew has graciously invited me to comment on his notes about our research on “teacher effects” and its media presentation. I agree with him that this important issue deserves an airing because questions of public policy such as teacher preparation and assignment and school “league tables” are best considered in the light of relevant available evidence.

What I do here is cite some of Andrew’s comments and in turn comment on them

There are a couple of factual matters:

The global study, led by the University of New England, monitored 500 pairs of identical twins during their first three years of school.

For the record, there are indeed about 500 pairs of identical twins in the study, plus the same number of fraternal twins. Our analyses for the paper in question employed both types, with similar results, but the UNE publicity person felt, correctly in my view, that readers and listeners would find it easier to grasp the case of identical twins, where the genetics can’t be a source of differences within twin pairs. In addition, we didn’t include our 250 sets of Scandinavian twins in these analyses because with just a few exceptions these twin pairs were kept together in school and so as a sample were not informative for the issue at stake. We had about 720 pairs in total in the analyses.

They (the twins) were divided into two groups: twins who shared the same teacher, and siblings who were split across different classrooms.

We, the researchers, didn’t divide the children. Parents and/or schools made those decisions; fortuitously, over the three years we followed the children, about half were in the same class as their co-twin and about half were in different classes.

Now, to other matters:

“Our study shows … the ‘teacher effect’ on differences in children’s acquisition of literacy skills in the early years of schooling is no more – and sometimes less – than 8 per cent,”

In case you’re interested, that figure comes from the difference, on average, between the intraclass correlation coefficients for reading and spelling results from same- and different-class twins. The average differences, across both country samples, all tests and zygosity, is about .08, which can be read directly as % variance accounted for by classroom-level effects, that is by the difference in what behaviour-geneticists refer to as unique, or non-shared, environment. For example, and from memory, the interclass correlation between monozygotic twins who have the same teacher is about .85, for those with different teachers, about .78. The correlations for dizygotic twins are both lower than these at about .54 and .46 (that’s the evidence for a substantial genetic effect of early reading), but the difference is about the same in absolute terms.

I say “and sometimes less” because, as Andrew points out, we cannot separate teacher influence from other processes operating at the classroom level. Those processes might include class size (though as I read the literature, within reasonable limits size doesn’t matter much), but includes others as well. Herb Marsh, now at Oxford, and others, have identified “classroom climate,” which is independent of the teacher, as influential. The issue is important because to the degree that classroom-level effects are not teacher effects the implications for practice are less clear-cut–practices like teacher rewards, the effects of cumulative exposure to supposed “good” and “bad” teachers, assigning 75th percentile teachers to disadvantaged groups, and so on. It is a very challenging research question to separate teacher from other classroom effects, but an important one.

The question is: are teacher quality effects large enough to affect children’s life chances?

In our article we give an example, similar to Andrew’s helpful ones, of the effect of having a 75th percentile teachers versus a 25th percentile one—about one-third of a standard deviation in reading in a year. But remember that the interpretation of this kind of scenario as down to teachers depends on classroom effects being completely teacher effects.

While twins represent a novel way of decomposing teacher effects, they do have their drawbacks. In particular, one might think that having a same-aged sibling creates the possibility of spillovers: if you have a lousy teacher and your twin brother has a great one, wouldn’t you use his notes when doing homework? If true, this will bias downwards the estimated share of variance explained by teacher differences.

The question of generalizing from twin studies to children (or people) in general is an important one, and one that behaviour-geneticists are very aware of. We discuss this in our article. We actually conclude that, in this case, twin cooperation is not likely to be a source of contamination. We do so on the grounds that if some process like this were operating twins would be more alike than otherwise, and as long as the process applied equally to monozygotic and dizygotic pairs, the estimate of what is called “shared environment” would be considerably higher than it actually is (in fact, it’s pretty close to zero in the Australian subsample). Shared environment refers to factors that twins share, like family and school, that influence the trait of interest.

By the way, it is on the grounds of the small, and generally statistically nonsignificant, shared environment effects, where school-level influences would show up, that I’ve been questioning the proposed school league tables. My opinion is that if significant differences do exist between average school scores in early literacy (and our data are silent on this) they are more likely to be about characteristics of the students in the school than about school-level policies and practices.

Like all scientific investigations, there are limitations in any piece of research, and much more needs to be known about all these issues. Meantime, I’m grateful to Andrew for the chance to discuss them.

(xposted @ andrewleigh.com)

3 thoughts on “Do Teachers Matter?”

  1. Thatnks for this, Andrew. I must say I was puzzled at the news reports of this study. My recollection from my own school days (admittedly a very long time ago) is that some teachers stand out as having really excited me about their subject; others I would have been happy to miss.
    Even if teacher quality is determinable on a linear scale its effect on student attainment may not be linear. It may be only the top few of teachers (top decile perhaps) who really fire up students’ ambition to achieve outstanding performance, and the bottom few, who don’t understand their subject and/or cannot keep order in the classroom, who really stunt achievement. If this is the case, the effect may be substantially obscured by the middle range.
    In any case I have doubts about whether teacher quality corresponds to anything like a linear scale. I recall  the effectiveness of our 5th form Latin teacher who earned our respect by having taught himself Russian so that he could read Pushkin in the original, the 6th form English teacher who segued from an extracurricular excursion into Hemingway into the 18th century English metaphysical poets. These may not be the best way to connect with students at a school in Alice Springs, although the science teacher who encouraged us to catch huhu grubs in the patch of bush at the back of the school, roast them on the pot-belly stove in the classroom and eat them may have connected pretty well.

    Like

  2. Brian says that most of the variation in outcomes is due to child characteristics rather than schools. This  warns us against misuse of school league tables, but is not much of a guide for policy. We can’t change the genetic characteristics of children (yet) and changing other aspects of family life is also difficult. So I think 8% of variance being due to class-room effects should be interpreted as a _large_ effect.
    Though – as Andrew and Brian note, other things go on in classrooms. I would hypothesis that peer effects (the characteristics of the other children in the class) would also be important. These are harder to change in aggregate – but are probably want parents are seeking to find out when they look at school league tables.

    Like

  3. Hello,
    Education is necessary for every one and when you to learn some body or teacher importance is that his/her learning way is sensitive,because student catch your ideas and information very easily. Teacher and student should be cooperate with each other.

    Like

Comments are closed.