It's been many years since I have trusted anything I read in a medical or psychiatric journal. There is an enterprise wide positive bias; findings never seem to replicate; benefits are hyped; harms are hidden.
Drug companies bear most of the blame -- the research they sponsor is shoddy and market driven. Scientists are also to blame when they torture data so much it will confess to anything. Medical journals are to blame when they publish positive findings from lousy studies and reject negative results from well done studies. And journalists are to blame when they uncritically accept phony claims. It won't surprise you that the most widely cited study in the recent literature is also my favorite: 'Why Most Published Research Findings Are False' by John Ioannidis.
David Healy, M.D. has been a leader in exposing the harms of drug treatments and the tricks of drug companies. He has also attacked the accepted wisdom that evidence based medicine requires the foundation of randomized, placebo controlled clinical trials (RCT). I agree with much of his criticism, but not with his conclusions. Here is a brief synopsis of our debate.
Dr. Healy writes:
Most papers reporting results of drug sponsored randomized clinical trials are ghostwritten by people hired by the drug company. In a world where we usually have no access to the raw data, such papers published in premier medical journals can claim as safe and effective a drug that is in fact dangerous and ineffective. This is bad medical research that leads to bad medical practice.
Proponents of Evidence Based Medicine (EBM) agree that drug company research behavior is egregiously bad, but argue this does not necessarily prove that clinical trials are inherently flawed. The problem supposedly lies in letting industry design, conduct, and implement the trials. Presumably everything would be okay if only the trials were designed and run by Angels or the Government -- studies would then deliver trustworthy, gold standard evidence that cannot possibly be obtained in any other way. EBM proponents favor the presumed objective results of controlled RCTs to the subjectivity of uncontrolled clinical observation. Without RCTs, they claim, we cannot practice Evidence Based Medicine.
Should industry be stripped of the right to ghostwrite trials and be forced to allow access to the trial data they don't own? Sure, but even if run by Angels, trials favor industry because they allow weaker and weaker drugs to enter the market, while hiding the harms.
Here's how. Clinical trials began as a way to demonstrate that experimenters knew what they were doing. If on the basis of good laboratory science demonstrating the pathology of a condition or direct observation of cures we had a drug that corrected a problem, then comparing it to placebo randomly allocated would show the drug to be superior to placebo. This test would not lead to new knowledge -- it would confirm what we think we already know.
The only time we learn something from clinical trials is when a huckster who has duped vulnerable patients into taking a drug for some purpose is forced to compare his nostrum to placebo and the trial shows him to be a conman. A demonstration like this can make a major contribution to patient safety.
In 1962, following the thalidomide crisis and just after the RCT method was developed, there was a political imperative to improve the safety of drugs. Based on the possibility that controlled study might control hucksterism, RCTs got built into the regulatory apparatus. It seemed a good idea to force the pharma camel to pass through the eye of a scientific needle in order to protect vulnerable patients. No-one recognized the pitfalls.
First, in trials we compare a drug to placebo. Even snake oil would come out as having a benefit if we put a large enough number into the trial.
Second, to run an RCT you have to hypnotize investigators and patients to ignore the 99 things a drug does, and focus instead only on one thing -- the thing both pharma or some Angels want you to focus on. There is a risk in generating ignorance in this way in order to nail down one effect while having a kind of hysterical blindness to harms. It's worth taking the risk when we show the drug doesn't work and the drug doesn't then get on the market -- which is what people expected in 1962, but is not what has happened since.
Third, RCTs are supposed to deliver objectivity by controlling confounders, but when both the illness and the drug produce the same symptom, RCTs create ineradicable confounding. They make it possible for drug companies or Angels to pass off the harms caused by treatment as a symptom of the illness being treated. And in medicine, it's very common for both an illness and its treatment to produce overlapping effects.
To then claim, as the rhetoric of Evidence Based Medicine does, that RCTs provide gold standard knowledge about what drugs do is to generate ignorance about ignorance.
This is not just an abstract ignorance. In 1962, it took a year or two from the time of a drug's launch to recognize its major hazards. It now takes decades from the time of first description to the point where the field accepts a link between treatment and hazard.
Drug induced death is likely now the leading cause of death -- it's difficult to be certain because we don't collect the data adequately. And little is being done to remedy our lack of awareness of drug risks.
More of the same RCTs are not the remedy. The calculations that come out of an RCT are only as good as the data (observations) that go in. In the belief that RCTs are more objective than doctors and patients, we have undermined the abilities of people to observe and act on the basis of their observations.
It is when people with a range of biases agree on what they are seeing that we get objectivity -- a mechanical process can never deliver this.
RCTs created Evidence Based Medicine. Most listeners hearing EBM hear Data Based Medicine, but evidence is not data. Statistical calculations derived from aggregates are a way to deal with conmen -- not a guide to the practice of medicine. The individual patient is the datum in medicine.
Thanks, Dr. Healy. I agree with you on the diagnosis of what ails the clinical research literature, but disagree with your suggested cure. If we have learned one thing from the history of medicine, it is the dangerous fallibility of uncontrolled and anecdotal clinical observation. Doctors have killed patients for millennia by bleeding them, administering emetics and cathartics, and prescribing poisons. The power of the placebo effect, the vagaries of chance correlational associations, and observer bias all overwhelm the generalizability of any given clinical observation.
Dr. Healy is right about the fallibility of clinical trials, especially when their basic purpose is
not to help patients, but rather to sell drugs or advance research careers.
But his solution is even more fallible. We have done the experiment for thousands of years and the clear conclusion is that anecdotal medicine is arbitrary and harmful medicine.
The only responsible courses of action are to improve designs and measures, standardize implementation, change sponsors, achieve complete transparency, report harms as thoroughly as benefits, and eliminate hype. With all the limitations, there is simply no substitute for randomized placebo controlled studies -- we must improve them because we can't do without them.
But it is clearly allowing the fox to guard the henhouse to give drug companies the franchise in conducting the studies that lead to the regulatory approval of their products.
The huge financial rewards will inevitably lead to badly biased implementation that cannot be adequately corrected even if there is complete after the fact transparency.
Drug companies should be taxed to support an independent testing agency that would be responsible for conducting consumer oriented clinical trials for safety and efficacy prior to regulatory approval and for collecting later surveillance data on adverse effects. No system will ever be perfect, but independent trials would be a great advance over the junk we have now and far more trustworthy than the uncontrolled clinical observation suggested by Dr. Healy.
Dr. Healy replies:
It's a mistake to generalize from the effects of a drug on the patient in front of me to the average effects of a drug, but it is an even more dangerous clinical mistake to generalize from average effects in studies to the patient in front of me.
The drugs that came on the market in the 1950s before we began using RCTs were more effective and safer -- guided by doctors reporting on what they saw in front of them -- than the drugs we have had since.
My last word: The pioneer drugs of the '50s were all such home runs they could be discovered by clinical serendipity and n-of-1 observation. But the low hanging fruit has all long since been picked. Refinements with much lower effect sizes require systematic study.
Dr. Healy's last word: "Do we want treatments with low effect sizes especially if their hazards are hidden by the trial process?"
My last-last word: We can't expect every useful drug to be a home run.