a resource to watch

The What Works Clearinghouse was established in 2002 to evaluate scientific studies of the effectiveness of various educational interventions, presumably in support of the 'research-based methodology' mandate of the No Child Left Behind (NCLB) Act.

It looks as though WWC has the potential to be a great resource. Briefly, here's what they do; they select a topic to report on, and evaluate the scientific studies that have been published on that topic. They determine whether the study is relevant to their topic of interest, whether it's well-designed, whether the outcome is relevant, and whether the student sample is relevant.

Why spend so much energy determining whether the research study is well-designed? Ideally, if you're trying to choose a curriculum to use, you want unbiased evidence as to whether the curriculum is effective. You don't want to use studies with samples that were plagued by attrition (i.e. teachers quitting the study in midstream), since that introduces biases; you obviously don't want to use studies that draw causal inferences from correlations; and you don't want studies with obvious biases (e.g., with one sample drawn from a more affluent school district). WWC has done the work of deciding whether a study is flawed for us.

They report on whether the study meets their standards of evidence solidly, marginally, or not at all. They summarize the results of the study, and add any caveats about interpreting the study. You can get brief summaries, or the complete report for each intervention, or the entire topic report.

The first topic the WWC tackled was middle school mathematics. I've been looking forward for a long time to reading their reports, and I've just finished reading the brief summaries of middle school curricular interventions. And I'm disappointed. Not in the work of the WWC, which I think is totally cool, but in the utter lack of strong scientific research in this branch of education. It's quite shocking.

Out of the research they reviewed on middle-school curricula, which was very limited to begin with, they found a total of only 4 research articles that met their standards of evidence.

There were a handful more that met their standards with caveats, and a small boatload more that didn't meet them at all. Even throwing in the ones that didn't measure up, there weren't very many papers to look at; a total of maybe 50. There's not enough meat there to base a decision on, especially since scientific studies ought to be replicated before their results are on firm ground.

For a broader idea of how meaty the field of math education research in general is not, have a look at Catherine's post on the California Dept. of Education study. That study team did a similar quality review of the research literature in mathematics education overall, and found that only about 3% of the studies they reviewed were scientifically valid.

This is an important thing for parents to take note of. If you're told by your school that the loathsome curriculum they chose was research-based, don't be deterred. There's only a 3% probability that the research they are citiing was valid. If you want to push them, ask for the citations.

The NCLB is pretty new. Perhaps the new demand for research-based methodologies will generate a new interest in (and funding for) running unbiased, quality studies. If so, WWC could be a great resource to watch.

Back to main page.

Comments