What is the reproducibility crisis, and what can the academic community to do solve it? originally appeared on Quora - the place to gain and share knowledge, empowering people to learn from others and better understand the world..
Answer by Richard Price, Founder of Academia.edu, on Quora:
The "reproducibility crisis" is the name given to the situation that a large percentage, somewhere between 65% and 90%, of the academic literature is not reproducible. What this means is that if you take the methods of a given paper, and perform those methods in your own lab, between 65% and 90% of the time you won't get the same findings.
The history of the reproducibility crisis is that in 2011, Glenn Begley, who ran the oncology division at Amgen, decided to try to reproduce 53 foundational papers in oncology. He was unable to reproduce 47 of them, which is 89%. Bayer, another pharmaceutical company, reported in the same year that it was unable to reproduce 65% of the papers in its sample of the biomedical literature. Reproducibility has also recently been found to be an issue in psychology and computer science.
I wrote up an article in Wired a few weeks ago on how to solve the reproducibility crisis. Here is the link to the article Scientific papers need better feedback systems. Here's why.
The summary of the article is that we need a new crowd-sourced peer review system in science. This peer review system would do two things:
- Elicit opinions on papers from all academics who read the paper, and not just the two peer reviewers who read the paper prior to publication, which is the current model.
- Provide a reward system for people to share data sets and code. Journals don't publish data-sets and code. We need an academic to be able to go to their tenure committee and say "I've posted a lot of data-sets and code online. I didn't publish them in journals because, as we all know, journals don't publish data and code. But they have gotten a lot of recommendations from senior people in the field, and that should count for something."
This question originally appeared on Quora. - the knowledge sharing network where compelling questions are answered by people with unique insights. You can follow Quora on Twitter, Facebook, and Google+.