‘Flawed’ use of non-significant data ‘killing further research’

Academic papers are routinely incorrectly treating “non-significant” results as proof that experiments had no effect, according to new research. 

A paper authored by researchers from the universities of Manchester, Oxford and Arkansas warns that there is a “widespread misinterpretation of non-significant results”, which may be killing off areas of interest in future research.

The study, published in PNAS, outlines that “non-significant results” in academic research are typically interpreted as meaning “no difference” or “no effect”, “despite long-standing recognition that this is a fundamental misinterpretation”, it says.

However, the interpretation remains “widespread”, featuring in about 50 per cent of research papers and conference presentations, the paper says.

This is one of the “most widespread and problematic misinterpretations in the scientific literature”, it explains, adding that a statistically non-significant result “shows only that the data do not provide strong evidence for a difference”. 

It says such a distinction matters because “findings can arise for two very different reasons: either there is no meaningful difference, or a meaningful difference is present but cannot be detected reliably because of limited sample size or high variability”. 

“Recognising the difference between ‘no evidence of a difference’ and ‘evidence of no meaningful difference’ can substantially improve how scientific results are interpreted, reported, and acted upon.”

It recommends that “equivalence testing” can be used instead to help make the distinction whether “the effect is too small to matter” or proves the evidence “remains inconclusive”, and therefore requires further research to determine its significance.

Co-author Jakub Tomek from the University of Oxford said that “absence of evidence is not evidence of absence” and using data in such ways is “really problematic”.

Tomek said: “This is a very basic issue, yet in life sciences, it routinely passes review and editorial judgement, and then it pollutes the literature because people say ‘there is no difference, this drug doesn’t have an effect’, which kills the development of further research.”

He said that “we did not detect a significant difference” often becomes shortened to “there was no difference”, especially when summarised in further research papers, “and that’s a huge shift”. 

David Eisner, professor of cardiac physiology from the University of Manchester, added: “Inevitably, some published results where people have concluded that there’s no effect will be wrong.”

He said he hoped their findings can help lead to a shift in data interpretation, and for academics to make it clear where there is evidence that something is not “biologically or medically important”, or whether “the data is so scattered that further work is required” to make definitive conclusions. 

Eisner said that this was only “one of several statistical issues which people need to pay more attention to” and forms part of a greater need to “be more careful with the use of statistics”.

juliette.rowsell@timeshighereducation.com