THE BLOG

Making Sense of Assessments: State-level NAEP Comparisons Offer Better Policy Guide Than International Assessments

This post is co-authored by James Harvey, director of the National Superintendents Roundtable.

The release of 2015 scores on the National Assessment for Educational Progress (NAEP), or the Nation's Report Card, finds either stagnation or dips in both reading and math in both 4th and 8th grades for the first time in 20 years. The results have prompted much ado. But much of that ado is, at the end of the day, about nothing. There has been speculation that this two-year dip suggests the Common Core isn't working, that the Standards are needed more than ever, and even that the results are to be expected given the 7-year starvation of school budgets since the onset of the Great Recession.

All of this is debatable, but more important points about NAEP have been captured beautifully in a serious, well-designed study that assesses NAEP scores as they were intended to be assessed - longitudinally, and with the benefit of relevant controls. The study, Bringing It Back Home, was produced by Stanford's Martin Carnoy, Emma Garcia of the Economic Policy Institute, and Tatiana Khavenson of the National Research University Higher School of Economics in Moscow, and released by the Economic Policy Institute at the end of October.

Bringing it Back Home makes several important points. The first of which is that, if we are serious about improving education policy, by far the best source of data is NAEP, not PISA or TIMSS or other international assessments that treat our 51 education systems as a single, national system. The second is that, contrary to what PISA scores have been used to suggest, US students have made major progress in recent decades. And, as a corollary, that there is a clear (and telling) difference in that progress in the pre- versus post-NCLB eras. And the third is that poverty trumps every other factor by far.

Not only education policies but social and economic policies play major roles in students' outcomes, and, of course, these policies vary vastly across countries. Yet we have focused largely on trying to tease out or isolate the impacts of schools in those vastly different systems, rather than turning to rich sets of data from systems that are much more comparable - US states - many of which have lessons to impart. As the authors of Bringing it Back Home write, "The lessons embedded in how these states increased student achievement in the past two decades are much more relevant to improving student outcomes in other U.S. states than looking to high-scoring countries with social, political, and educational histories that differ markedly from the U.S. experience." Indeed, as one of us pointed out during the release, it is extremely difficult to derive policy implications from a tiny, elite Alpine haven of 5,600 students such as Lichtenstein and then apply them to the 56 million diverse students in the United States. Not to mention the degree to which we would be reluctant to adapt the practices of a dictatorship like Kazakhstan, even if they did offer the potential to increase student test scores.

Which leads to the second key point; PISA-based claims that US students are stagnating or losing ground are simply untrue. Across the board, US students have made substantial progress in the past two decades. Fourth graders gained an average of 21 points in math between 1992 and 2015, and eighth graders gained nearly as much. Fourth graders also improved in reading, albeit much more modestly. However, the vast bulk of those gains happened in the decade prior to No Child Left Behind. Since 2003, gains have shrunk substantially; with respect to reading, to virtually nothing. This troubling development does not lend much credence to "no excuses" arguments that lack of accountability and low standards have been major culprits affecting PISA (or NAEP) scores. If anything, the shift from a supports-based version of the Elementary and Secondary Education Act to one more focused on standards and accountability has slowed progress.

Finally, the study's careful, three-layer use of data on poverty hammers home what has long been a central, but largely sidelined, finding regarding PISA data: poverty is not an excuse, but a serious problem. Indeed, after controlling for student poverty, and then adding a second control for school-level poverty, the study finds that the next strongest predictor for low growth in NAEP scores is the state poverty rate. Yes, you read that right: the first, second, and third biggest factors driving NAEP scores - and impeding their growth - are poverty, poverty and poverty.

So we finally have useful, rigorous, and relevant state-level data to guide education policy decision-making. Let's not blow it this time. Let's not continue to pretend that mediocre schools are our biggest problem, while tolerating a situation in which millions of children, in the world's wealthiest country, are hungry, homeless, and traumatized. Let's not keep blaming bad teachers for bad social policy. Let's not let sharply inequitable funding of schools mask the need for sufficient funding for all of them.

Rather, let's treat social, economic, and education policies as the innately interwoven drivers of children's school and life trajectories that they are. And let's start putting them all on the right track. We can begin by looking at state policies in the United States that work instead of trying to import fairy dust from abroad.