I’ve written before about the replication crisis and the shockingly low reproducibility rates in scientific research. I’ve usually focused on psychology but sadly medicine is even worse. I’ve usually viewed these problems through the lens of reproducible research and have focused on ways reproducible research could help solve the problem. It turns out, though, that there’s another more relevant problem that needs addressing.
Over at the Nautilus Website, Aubrey Clayton has an excellent article on what he and a growing number of scientists believe the problem is: a reliance on statistical inference. Using these techniques was something I was taught at school along with, I’d guess, most Irreal readers. The use of them is standard and probably required in almost every scientific journal. At least, they have been. Now, due to the ongoing crisis with studies not reproducing, some journals are rethinking that policy and even going so far as to outlaw them in the papers they publish.
Clayton starts his article with what he calls three versions of the same story:
- A woman convicted of murder based solely on the fact that her two infant children died shortly after birth.
- A woman who discovers a lump in her breast and after having a mammogram is told the lump is almost certainly malignant.
- The famous Norenzayan study that concluded staring at Rodin’s sculpture, The Thinker, led to decreased religious belief.
They’re the same story, Clayton says, because they all depend on the same flawed statistical inference to reach their erroneous conclusions. The rest of the article discusses what went wrong and how incorrect findings were reached in each case.
Clayton makes a cogent argument—including a worked out example in the case of the woman with the breast lump—that the proper way to analyze these cases is with Bayesian Analysis, that is by using Bayes’ rule. I won’t repeat his arguments here because you should read them for yourself: The ongoing reproducibility crisis has serious implications for all of us, as the story about the woman with the breast lump makes clear. The article is not at all Mathematical so don’t be put off by that.