New research raises fears that promising new educational interventions are being ‘unnecessarily abandoned’ – ScienceDaily

Promising new educational interventions are potentially ‘unnecessarily abandoned’ because trials to test their effectiveness may be insufficiently faithful to the original research, a study has warned.

The caveat comes after researchers ran a large-scale computer simulation of more than 11,000 research trials to examine how much ‘fidelity’ influenced the results. In science and the social sciences, “fidelity” is the extent to which tests evaluating a new innovation adhere to the design of the original experiment on which that innovation is based.

In the same way that scientists test a new drug before it is approved, new learning enhancement strategies are often thoroughly evaluated in schools or other settings before being rolled out.

Many innovations are rejected at this stage because trials indicate that they result in little or no learning progress. However, academics have been concerned for some time that in some cases fidelity losses could jeopardize the lawsuit. In many cases, fidelity is not systematically measured or reported.

The new study put that theory to the test. Researchers from the University of Cambridge and Carnegie Mellon University have conducted thousands of computer-modeled trials, with millions of simulated participants. They then looked at the extent to which changes in fidelity changed the ‘effect size’ of an intervention.

They found that even relatively subtle deviations in fidelity can have a significant impact. For every 5% of lost reliability in the simulated follow-up tests, the effect size dropped by 5%.

In real-world settings, this could mean that some high-potential innovations are deemed unsuitable for use because low fidelity skews the results. The study notes: “There is growing concern that a significant number of poor outcomes in educational interventions… are due to lack of fidelity, resulting in the unnecessary removal of potentially strong programs.

The results can be particularly useful to organizations such as the Education Endowment Foundation (EEF) in the UK or the What Works Clearinghouse in the US, both of which assess new research in education. The EEF reports the results of project testing on its website. Currently, more than three out of five reports indicate that the tested intervention resulted in no progress, or negative progress, for students.

Michelle Ellefson, professor of cognitive science at the University of Cambridge’s Faculty of Education, said: “A lot of money is being invested in these trials, so we need to look closely at how well they control fidelity. Reproducibility in research is extremely important, but the danger is that we could reject promising interventions due to violations of reliability and create an unnecessary trust gap between teachers and researchers.

Academics have often referred to a “replication crisis” precisely because the results of so many studies are difficult to replicate. In education, trials are often conducted by a mixture of teachers and researchers. Larger studies, in particular, create many opportunities for inadvertent loss of fidelity, either through human factors (such as misread research instructions) or through changes in the research environment (e.g., the time or test conditions).

Ellefson and Professor Daniel Oppenheimer of Carnegie Mellon University developed a computerized randomized controlled trial that initially simulated an imaginary intervention in 40 classrooms, each with 25 students. They repeated it over and over again, each time adjusting a host of variables – including the size of the intervention’s potential effect, the students’ ability levels and the reliability of the trial itself.

In later models, they added additional confounders that might further affect the results – for example, the quality of school resources or the fact that better teachers might have better-performing students. The study combined representative permutations of the input variables, modeling 11,055 trials in total.

Strikingly, across the entire data set, the results indicated that for every 1% of reliability lost in a trial, the effect size of the intervention also decreases by 1%. This 1:1 match means that even a trial with, say, 80% reliability would see a significant drop in effect size, which could cast doubt on the value of the intervention being tested.

A more granular analysis then revealed that the effect of fidelity losses tended to be larger where a larger effect size was anticipated. In other words, the most promising search innovations are also more susceptible to fidelity violations.

Although confounding factors weakened this overall relationship, reliability had by far the greatest impact on effect sizes in all tests conducted by the researchers.

Ellefson and Oppenheimer suggest that organizations conducting research trials may want to establish firmer processes for assuring, measuring, and reporting fidelity so that their recommendations are as robust as possible. Their article references research from 2013 which found that only 29% of after-school intervention studies measured loyalty, and another study, in 2010, which found that only 15% of after-school intervention studies social work collected fidelity data.

“When teachers are asked to try new teaching methods, it is natural – perhaps even admirable – that they want to adapt the method to the specific needs of their students,” Oppenheimer said. reliable scientists, however, it is essential to follow instructions precisely; otherwise, researchers cannot know whether the intervention will be largely effective. It is really important for research teams to monitor and measure the reliability of studies, in order to draw valid conclusions.

Ellefson said: “Many organizations do an excellent job of independently evaluating research, but they need to ensure that fidelity is both measured and scrupulously verified. Sometimes the correct response when results cannot be replicated may not be to dismiss the research altogether, but to step back and ask why it might have worked in one case, but not in another? »

The findings are published in Psychological methods.

Comments are closed.