In February, Education Next released a report that seemed to credit performance pay with the success of countries who ranked high on the 2009 Program for International Student Assessment survey (PISA). Merit pay proponents hailed the study as proof positive that these measures are necessary if the United States is to improve student achievement. Case closed.
Not so fast, according to the National Education Policy Center (NPEC). The Education Next report, called “Cross Country Evidence on Teacher Performance Pay” and written by Ludger Woessmann of the University of Munich, went unchallenged in the media until the NPEC published an analysis of the findings and concluded that the report was based on “unreliable and invalid” data and made too many broad assumptions to be useful as a policymaking guide.
Matthias von Davier of the University of Oxford took issue with Woessmann’s refusal to remain cautious in his conclusions, which was necessary given the laundry list of qualifications Woessmann himself acknowledges.
Woessmann throws out so many caveats, the reader is immediately left wondering how he could ultimately draw a straight line between merit pay and student achievement in other nations.
Woessman concedes that the sample size is “small” (27 countries). Just including or excluding one nation produces significant shifts in the size of the reported findings. He also notes that the “available information on the extent of performance pay is far from perfect.”
“Although the author lists numerous caveats,” von Davier points out. “His broad conclusions do not heed those cautions.”
Von Davier’s major concern is that the report doesn’t properly take into account different education systems and the specifics of various performance pay plans.(You can read Von Davier’s complete report here.)
“Differences in the way various countries provide so-called performance pay are not properly considered,” von Davier explains. “Perhaps one type is beneficial, while another is detrimental. Additionally, variations in the length of time one or another country has employed such a pay system are not addressed. A system that hasn’t been in place long enough to have a strong effect is treated equally to one that has been in place much longer.”
“Teasing out differences in PISA scores and then attributing those differences to the presence or absence of performance pay is problematic,” von Drier cautions.
Correlation doesn’t equal causation, and even Woessmann acknowledges that what he has observed could be “the opposite of what it seems.”
“Both performance pay and higher levels of achievement could be produced by some set of factors other than those taken into account in the analysis, “ he concedes.
No doubt, but that didn’t stop merit pay advocates from ignoring the author’s own qualifications and trumpet misleading headlines around the Web.
After all, a headline that reads “Countries with performance pay for teachers score higher on PISA tests – of course the two may not have anything to do with each other” wouldn’t get a lot of attention.