I used to be one of the biggest skeptics I knew, second only to my father -- the scientist.
You know what makes a good skeptic? Believe nothing. Question everything. And I always do. Or, at least, that's where I start.
As a holistic wellness practitioner, not only have I read a fair amount of information on certain topics, to quell my own doubts, but I have sought a way to bridge the gap between blind faith and researched data, for the others that look for a deeper knowing, beyond belief.
In the online world, when someone presents a concept that is perceived as unconventional, one of the biggest responses is, "Has it been peer reviewed and published in a scientific journal?"
Fair enough question.
Or is it?
Here's the deal: The ol' "peer reviewed scientific journal" line is now the classical retort to attempt to dismantle ANYTHING that is out of the box, or pushes at the edge of the current, mainstream acceptable norm. It is the retort used to try to prove that the information being presented is completely bogus, or at the very least, unreliable. And while the peer review process does help eliminate a lot of false claims, and therefore I still value it, if you are using that retort, alone, to prove your position, it may not actually prove anything.
Now, stay with me here.
See, I had bought into this idea as well, though always with a bit of my traditional suspicion. I looked for things to be provable and mistakenly believed that I could look to the peer review process (which can then lead to publication in a reputable scientific journal or funding) as a reliable factor. And, to some degree, that is true.
I also thought I had a lay-person understanding of what the scientific peer review process involved.
I assumed it meant that a handful of other experts, in whatever field, reviewed your work, and determined if they could get the same results as your research. If they did, they would give it a thumbs-up, and then you could potentially be published or funded, from there.
So, yes, that's close...but that's not actually how it works.
First of all, let's get clear on the dictionary definition of peer review.
Merriam-Webster defines "peer review" as follows: a process by which a scholarly work (such as a paper or a research proposal) is checked by a group of experts in the same field to make sure it meets the necessary standards before it is published or accepted.
So, yes, it does involve being "checked" by a group of experts. But, as it turns out, it can be very difficult during that "checking" process to detect if there are issues with the research, such as fraudulent data. The reviewers rarely have access to the original data used, and the peer review process is not a time used to duplicate results of the research. So, some of what gets the thumbs up, may not actually be sound science. As it turns out, it appears that the peer review process is a lot more faith based than we want to attribute to science.
Apparently, the reliability of the peer-review process has come into question, within the scientific community, as well.
"Consider a study conducted by BMJ (British Medical Journal), one of the most respected peer-reviewed journals in medicine. BMJ Editor Fiona Godlee and two colleagues took a paper about to be published in their journal and introduced eight deliberate errors. Then they sent the paper to 420 reviewers. The median number of errors detected by the 221 respondents was two. Nobody found more than five, and 16 percent didn't find any errors at all. This seems to suggest that peer review doesn't really increase the quality of published research, or does so to only a small degree." (source)
Check out this article, in the Journal of the Royal Society of Medicine, Peer review: a flawed process at the heart of science and journals, which concludes with this:
"So peer review is a flawed process, full of easily identified defects with little evidence that it works. Nevertheless, it is likely to remain central to science and journals because there is no obvious alternative, and scientists and editors have a continuing belief in peer review. How odd that science should be rooted in belief."
(P.S. this is a respectable journal, in case you are wondering.)
Or, if you are really into scientific studies, you can read, A Reliability-Generalization Study of Journal Peer Reviews: A Multilevel Meta-Analysis of Inter-Rater Reliability and Its Determinants.
While reviewing the peer review process, apparently, some of the studies have shown things such as inconsistency and bias. Imagine that!
"They took 12 studies that came from prestigious institutions that had already been published in psychology journals. They retyped the papers, made minor changes to the titles, abstracts, and introductions but changed the authors' names and institutions...The papers were then resubmitted to the journals that had first published them. In only three cases did the journals realize that they had already published the paper, and eight of the remaining nine were rejected--not because of lack of originality but because of poor quality. Peters and Ceci concluded that this was evidence of bias against authors from less prestigious institutions." (from the aforementioned JSRM article)
Peer review aside, don't even get me started on the studies that are discovering that a lot of research can't be replicated. That just makes my head spin! According to Henry Miller, a physician and molecular biologist, and the Robert Wesson Fellow in Scientific Philosophy and Public Policy at Stanford University's Hoover Institution, "A number of empirical studies show that 80-90% of the claims coming from supposedly scientific studies in major journals fail to replicate." (source)
Ok, so this is what I have landed on. The average person running around wants to use science as the be-all and end-all determining factor if something is true or not. We want proof, damn it! After all, there are too many crack pots running around with their crazy concepts. And the only way we can separate the wheat from the chaff is with science. Right? RIGHT?????
Geez, I wish that was true.
I truly wish that somehow science was exempt from things like bias and corruption.
I would like to pretend that there aren't inner politics and other factors that leave scientists feeling pressure to get published and funded, no matter what.
I would like to pretend that science, like most things, isn't funded by independent parties with something to gain from the results of the findings.
I would like to pretend that the scientific papers published tell the full story of ALL the research that was done, and that the papers that found conclusions that "disagreed" with the desired outcome were not quietly set aside.
But, I can't pretend.
So, I keep asking questions. I keep looking for answers. I keep my mind and heart open to the possibilities of the unknown and the not yet provable.
Look, I love science. In fact, the whole reason I was inspired to write this was because my sister in law (a scientist) and I were small talking about the peer review process. I mean, seriously...who, outside of the industry, SMALL TALKS about the peer review process?
I appreciate peer review. I appreciate the potentiality for quality control and the vetting process that can keep the false information at bay. And I also appreciate that science itself admits that the process needs to be improved.
What I don't appreciate is people's attempts to use the peer-review process, and the subsequent publication or funding of specific research, as the definitive measure of if something is credible or not. If you want to disagree with someone or something, that is fine. I don't fault you there. We are each going to perceive what we want--whatever you choose to base it on---belief or evidence.
Yes, I feel that there is a whole lot of B.S. that exists out there. But, next time someone brings up a concept that seems out there to you, and you want to throw the baby out with the woo-woo bathwater, please consider not just pointing to the peer-review process, alone, as some sort of magical proof that what you trust, is true and provable, and what someone else perceives, is a lot of crap.
Because, as it turns out, we might all be full of IT.