A four-year pilot program in Chicago's Public Schools that focused on increasing teachers' access to self-improvement programs was recently evaluated -- and the resulting report found that while teacher retention rates increased, there was little impact on student performance.
The Teacher Advancement Program (TAP), funded by federal Teacher Incentive Fund grants and implemented over a four-year period at 40 "high need" Chicago Public Schools, was evaluated by independent agency Mathematica Policy Research.
Its original purpose was to provide teachers with more mentoring, structured feedback and incentives for self-improvement. Teachers were meant to have more access to leadership positions, and offered financial rewards for improved student performance on standardized tests.
During its test run in Chicago, similar programs were implemented at more than 200 schools across the country, mimicking the Chicago model.
Mathematica's findings suggest the program's expansion may have been premature:
"These changes did not...pay off in terms of higher student achievement within the four-year rollout period in Chicago," study author Allison Seifullah writes in her executive summary. "This result provides a caution to funders investing in future programs in terms of what to expect over a four-year period. However, designers of new policies might consider how to change selected program elements to produce more favorable outcomes in the future."
The study compounds five years of data collected at 34 public and charter schools that completed the pilot program, and compares their performance in the key areas of test score improvement, achievement gap closure and dropout rate reduction to non-TAP schools. (The data was slightly impacted by closures and consolidations at several participating TAP schools during the four-year period.)
Teacher responses to survey questions about their satisfaction were largely the same at TAP and non-TAP schools, with non-TAP teachers expressing a slightly greater sense of support from administrators: 79 percent of TAP faculty felt sufficiently supported, versus 83 percent of non-TAP teachers.
The changes that occurred in Chicago TAP schools, according to the report, did not translate into positive impacts on test scores in any consistent way; TAP had both positive and negative test score impacts across the participating schools. The other notable impact of the program was in raising teacher retention, which the study notes can be positive for the culture of the school, but has some limitations.
"Unfortunately, we are unable to estimate the impact of Chicago TAP on the quality of teachers retained because we lack reliable measures of teacher quality for both Chicago TAP and non-TAP schools," Seifullah writes in the report. Retention has a policy interest, however, due to the high cost of staff turnover.
Mathematica notes that merit-based pay, which was initially touted as one key component of the TAP program, wasn't executed as the initial proposal had intended. Schools were unable to reliably link student achievement data to individual teachers' performances, resulting in minimal bonus awards that were, on average, smaller than the initially-stated targets.
In this respect, TAP can be seen as a predecessor for the current merit-based pay model introduced by Mayor Rahm Emanuel, which offers principals bonuses for narrowing achievement gaps, reducing dropout rates and improving student performance on standardized tests in reading and math. Teachers have objected to the merit pay model, citing TAP's minimal observable impact on student success.
This supporting data from Mathematica's study could further arm CPS faculty and the Chicago Teachers Union in their fight against test score improvement models that target teacher performance--one of many recent issues that have compounded current standoffs between school staff and administrators.