The problem in education reform isn't a lack of good ideas. It's a lack of good ideas implemented with enough clarity, consistency and integrity to actually make a difference in rigorous experiments (and therefore in large-scale application). A recent large-scale evaluation of Response to Intervention (RTI) illustrates this problem once again.
Response to Intervention (RTI) is a strategy for helping students who are struggling to keep up with ordinary classroom teaching. The idea is that following initial instruction (Tier 1), teachers provide mild assistance to students who are having difficulties (Tier 2), such as small-group remediation. Those who continue to struggle might receive more intensive assistance (Tier 3), such as one-to-one tutoring.
RTI has been common in U.S. classrooms for 20 years, and was virtually mandated as part of No Child Left Behind. So it is distressing that a recent study by MDRC, a respected independent research organization, found no positive effects of Tier 2 services in grades 1-3 reading. In fact, there were slight negative effects for first graders receiving Tier 2 services.
Philosophically, I am a supporter of RTI, but I'm even more of a supporter of rigorous evidence. Yet here's a very large, well-done (though not randomized) study of RTI that finds no benefits.
I think the findings of the RTI evaluation speak to a broader problem of education policy. Often, national, state or local policies promote or require uses of broadly defined strategies. RTI is a perfect example. Everyone understands the general idea, but there are thousands of ways to implement RTI in practice.
Studies of broad teaching concepts almost always find that they make little if any difference. The reason is that general concepts are implemented differently from class to class and school to school, and end up on average looking a lot like what teachers were doing before, or are doing in schools that do not claim to be implementing the broad concept. That is, the "experimental" classes are not terribly different from the "control" classes.
As a good example of this problem, in the 1970s and '80s, Madeline Hunter was extremely popular, and she spoke everywhere suggesting effective classroom strategies. Yet several studies found that when teachers were given training and coaching in Madeline Hunter strategies, it made no difference in achievement. Why? The studies also found that control teachers were already using strategies much like those in the Hunter model. It may well be that Madeline Hunter's theories were so popular precisely because they were appealing descriptions of what teachers already were doing. Everyone likes to hear that what they've always done turns out to be supported by research. So, exactly what made Hunter's prescriptions popular also made them no more effective than ordinary teaching, because they were ordinary teaching.
In the case of RTI, the MDRC researchers documented some differences between schools using RTI and those that were not, but there was enormous overlap.
So here's a proposal for what I'll call Bob's Law: General teaching strategies subject to substantially varying interpretations by individual teachers are likely to be transformed into practices much like ordinary teaching. For this reason, they are unlikely to produce better outcomes than the control group does.
This does not mean that ordinary teaching methods are bad, or that teaching methods informed by general concepts are bad, but what it does imply is that if you want to see marked improvements in student achievement across many teachers and schools, you have to have programs that are well-conceived, well-specified and well-supported by top-quality professional development and materials. Even then, not all programs work, but success is at least possible if programs bring about systematic and sensible change in teaching methods.
Back to RTI, I remain hopeful that RTI's strategies can improve student outcomes. However, the approaches to this concept that are likely to work are ones that are specific about all key aspects of the design and help teachers implement approaches that are markedly better than whatever they were using before.