The United States has 50 distinct states, which means there are 50 distinct definitions of "proficient" on standardized tests for students.
For example, an Arkansas fourth-grader could be told he is proficient in reading based on his performance on a state exam. But if he moved across the border to Missouri, he might find that’s no longer true, according to a new report.
"This is a really fundamental, interesting question about accountability reform in education," Jack Buckley, commissioner of the government organization that produced the report, told reporters on a Tuesday conference call.
The report, written by the Education Department's National Center for Education Statistics, found that the definition of proficiency on standardized tests varies widely among states, making it difficult to assess and compare student performance. The report looked at states’ standards on exams and found that some states set much higher bars for students proficiency in particular subjects.
The term "proficiency" is key because the federal No Child Left Behind law mandates that 100 percent of students must be "proficient" under state standards by 2014 -- a goal that has been universally described as impossible to reach.
The report, released Wednesday, relies on standards used by the National Assessment of Education Progress, the only national-level standardized test, considered the gold standard for measuring actual student achievement. Researchers scaled state standards to match NAEP's and then analyzed differences among state scores in 2005, 2007 and 2009.
They found many states deemed students "proficient" by their own standards, but those same students would have been ranked as only "basic" -- defined as "partial mastery of knowledge and skills fundamental for proficient work at each grade" -- under NAEP.
"The implication is that students of similar academic skills but residing in different states are being evaluated against different standards for proficiency in reading and mathematics," the report concludes.
"Standards" is a buzzword in contemporary education policy, as test scores are quickly becoming the decider in more and more management decisions, from teacher hiring and firing to compensation. Already, under No Child Left Behind, states determine funding for schools by student performance on these state tests.
The states themselves set the standards, however. They do so by choosing "cut scores," the cut-off point to determine distinct performance levels, such as proficient, basic or advanced.
The report notes that changes in the proportion of students ranked "proficient" at the state level did not hold up once scaled to higher national standards. More states showed positive changes in the number of students meeting state proficiency standards than those that met NAEP's proficiency scores.
Tennessee had the lowest standards in 2009 for fourth grade reading, while Massachusetts had the highest. In eighth grade reading, Missouri had the highest standards, though its proficiency rating was well below NAEP's, while Texas set the lowest bar for proficiency.
Tennessee also had the lowest standards on both 2009's fourth and eighth grade math exams, while Massachusetts led the pack with standards above NAEP's standard for "proficient."
Buckley noted that higher standards did not trend with increased performance.
"There's just no clear relationship between the rigor of the standards and the outcome," he said.
On Monday, Secretary of Education Arne Duncan announced that he was moving forward with a plan to use his executive authority to work around NCLB's provisions by granting waivers to states that agree to sign on to specific reforms, which have yet to be announced. Duncan said NCLB had led to a "dummying down" of standards for states. The report, however, shows that standards in some states may have been lowered between 2005 and 2009, but that was not the case between 2007 and 2009.
The report found that among states that changed their assessments, rigor did not necessarily increase between 2005 and 2009, and in some cases, states decreased the difficulty of their assessments. But states that changed testing practices between 2007 and 2009, the report found, set the bar higher.
“Our role in Washington is to support states as they raise standards," Duncan said in a statement in response to the report Wednesday. "In our plan to offer flexibility from No Child Left Behind's one-size-fits-all mandates, we will encourage states to set a high bar and raise their standards." He also mentioned possible changes under the Race to the Top competition, saying "states are working to create the next generation of assessments that will track students' academic growth and measure higher-order thinking skills."
Duncan has stressed the need to measure students at "career and college-ready standards." Tennessee has raised its standards since the 2009 tests measured in the report, and faced a subsequent decrease in the number of students deemed proficient -- a move Duncan has called "courageous." Other states, such as Colorado, are replicating such moves as they gradually adopt the Common Core, a national curriculum. The report's results may no longer apply as more states seek to change their exams and their standards.
This article has been updated to include a statement from Education Secretary Arne Duncan.