What International Comparisons Don’t Tell Us About U.S. Student Achievement

Recent international test scores show what appears to be a poor performance on average by U.S. students compared with their peers in first-world industrialized countries. Consequently, education systems in Finland, Korea, Singapore and other nations have become the gold standard to which other nations aspire.

Except, according to a new study by the Economic Policy Institute, comparing student outcomes across countries with vastly different cultures, education systems and political environments may be misleading if not somewhat irrelevant. Instead, it may be more vital in the U.S. to compare Massachusetts with Connecticut, and Texas with California, rather than North Carolina with Japan or Poland.

In “Bringing it Back Home: Why State Comparisons are More Useful than International Comparisons for Improving US Educational Policy,” the authors argue that international comparisons have merit, but measuring states with other states rather than countries is more valid in evaluating the true quality of education in the U.S.

“It’s very challenging to craft comparisons based on international ratings,” says Emma Garcia, an EPI economist who co-authored the report with Stanford University Professor Martin Carnoy and Tatiana Khavenson, a researcher at the National Research University Higher School of Economics in Moscow. “The scores do not mean that U.S. students are not making progress in comparison with other countries.”

In addition to ambiguous conclusions, international test scores are often used as a political football by pundits, legislators and other policymakers to wrongly conclude that low-test scores by U.S. students compared to their counterparts threatens the nation’s economic future.

“We’ve made tremendous increases in math and science in the U.S.,” Carnoy says.

The question is, he adds: “Why do some states do better than others? And what can we (in the U.S.) learn from states that have done well.”

While international test scores provide a measuring stick of sorts, the report delves into data available on student academic performance in the states over the past 20 years and shows that state-generated successes are a more reliable guide for improving U.S. education. Why?

Because U.S. students attend schools in 51 separate education systems. Although education in the U.S. is regulated under the same federal rules, the federal government does not run school districts. Instead, districts are operated by individual states, plus the District of Columbia.

“We can learn more from (comparing state scores) than why Connecticut didn’t do as well as Finland,” says Carnoy, who along with Garcia and James Harvey, director of the National Superintendents Roundtable, presented the EPI report October 30 at EPI headquarters in Washington, D.C.

“People interested in international systems don’t understand the dynamics of these (51) systems,” Harvey says. “The American education system is not one.”

According to researchers: “Variation in performance of students with similar backgrounds is likely related to specific state policies that might be applied elsewhere in the U.S.”

This synergy between states is in stark contrast to the social, political, educational histories and customs inherently unique within individual countries. In Finland, for example, students receive five years of pre-school, while in Korea parents pay thousands of dollars a year for extracurricular classes so their children can study, among other topics, how to score well on tests.

“Learning how to take tests does not reflect the quality of an education,” says Carnoy. “We have made much more progress in U.S. education than is usually focused on.”

In addition to the EPI report, Carnoy wrote “International Test Score Comparisons and Education Policy: A Review of the Critiques,” a recent analysis produced by the National Education Policy Center (NEPC) and funded by the Great Lakes Center for Education Research and Practice.

The report focuses on critiques of the Program for International Student Assessment (PISA), a test administered every three years to 15-year-old students from randomly selected schools worldwide. Students take a test in reading, mathematics and science, with a focus on one subject in each year of assessment. In 2012, approximately 510,000 students in 65 economies took part in the assessment.

PISA is administered by Organization for Economic Cooperation and Development and began implementing the tests in 2000. Students in the U.S. have participated in PISA since its inception, along with another assessment of eighth-graders, the Trends in International Mathematics and Science Study (TIMSS).

In “Review of the Critiques,” Carnoy states that when Shanghai topped the PISA rankings in 2010, U.S. Education Secretary Arne Duncan said: “We have to see this as a wake up call.”

According to the EPI report, policymakers are incorrect in concluding—based on international tests—that U.S. students are failing to make progress in mathematics and reading particularly due to country averages not being adjusted for the differences in family background, resources and other social capital.

The report states: “When these factors are adjusted, U.S. students perform better than raw scores indicate. Focusing on national progress obscures the progress that disadvantaged students have made in mathematics at the state level as measured by TIMSS and PISA.”

Gains of disadvantaged students in the U.S. have been larger than similar students in other countries while other students in some states have experienced gains as high as those in high scoring countries. For example, students in Massachusetts and Connecticut perform roughly the same on PISA reading test as students in Canada, Finland, and Korea, and higher than France, Germany and England.

There are lessons to be learned from states where students made gains from initially low beginnings and from initially high beginnings, or where students gained in one subject but not the other.

“It is reassuring to know that some states are competitive with higher-achieving nations,” says Harvey.

More analyses are needed to explain the variation across states so that lessons can be learned about specific state policies and practices that account for variation in the performance.

“That was the purpose of the paper,” Carnoy says. “To get people to focus on this and unravel this puzzle.”

  • Ysbeth

    The NEA wasn’t saying that international test scores didn’t matter back when the USA was near the top. That was a long time ago but we still had the same fragmented education system we have today – in fact, back then it was more fragmented.

    • Mary

      The US student population was very different a long time ago as were the education systems in other countries. The increase in poor children and special needs children who are tested today has an definite effect on the results. We test everyone and the other countries don’t.

  • LBD

    The problem in comparing test scores to other countries is simple… the students being educated in each are vastly different. In the UK by age 15 you know if you will plan to study for A levels and go on to Uni. In Germany it’s age 13 that you pick what you will major in Uni. They have students taking tests that they are experts in whereas the US does not expect students to be an expert in say 4 subjects, just adequate in 12 subjects. So comparing apples to oranges doesn’t help. Also what constitutes good writing in one country would be terrible in another.

  • Margaret Denn

    Mary, you are so right. As my building’s Title 1 teacher, I was required to test mixed groups of Title 1 and Special Ed. students on the Washington State tests. The Special Ed students were being tested far above their performance levels. I had one Samoan fourth grade Special Ed. boy who refused to go out to recess because he wanted to do well on the test. He also wanted to work through lunch and lunch recess. I felt so sorry for him. The testing coordinator for my building said that the groups of children that I tested had done better than the regular fourth grade classroom students. I think one explanation for their performance on the tests was that they liked me and wanted to do well for me.

  • HMWW

    The largest difference between many of these nations and the U.S. are the expectations they place on students. Other nations make it a priority to produce experts in particular fields, while the U.S. typically views education as producing “well-rounded”, non-specialized students (at least at an early age). American parents often send their children to school expecting to them to graduate as not only scholars, but athletes, artists, musicians and so on.