What Counts as Research?

Evidence-based reform in education is popular among educational researchers who like quantitative, randomized research, but that's a small slice of the profession. A much larger portion of the educational research community ranges from skeptical to downright hostile. They don't see a place for themselves in the brave new world of evidence-informed policy.

Evidence-based reform does in fact rely primarily on experiments that are always quantitative and usually randomized. Critics point out that this is just one approach to research and that there are many other equally valid approaches. Why should one be valued (and funded) far above the others?

I think the question really comes down to appropriate research methods for particular research questions. For some questions, there is no valid alternative to a quantitative experiment. For example, imagine that you want to know whether a new math curriculum is more effective than common practices in terms of enhancing achievement. The question itself demands a quantitative approach, to measure math achievement. It demands an experiment, because the question is a comparison between one approach and another. The outcome measures might include authentic assessments as well as tests, the experiment might be randomized or matched, but essentially the comparative question demands a comparative design.

However, there are many other questions that do not lend themselves to quantitative or experimental measures. How does the new math approach change teachers' and students' roles and relationships? How do they change the culture of the school? How does the new program flow from and reciprocally influence local or state policies? How do students' success with the new program correlate with their prior success, demographic categories, or social class? Each of these, and many other questions, demand different research designs: ethnographic, descriptive, correlational, policy, and so on.

Even in randomized experiments, there is usually a qualitative element, included to provide information on what is really happening in the various treatments. Further, if evidence-based reform takes hold in a big way, the demand for all sorts of evidence on a broad array of questions is sure to expand, as policy makers come to understand and value the entire research enterprise.

It will take all of us working together to bring knowledge to bear on critical questions of educational policy and practice. We can respectfully disagree about strategies and methodologies, of course, but a broader interest in the findings of educational research within the policy community seems sure to be beneficial to the research community. Besides, our focus needs to be on what is best for children, not what is best for our favorite methodology.