When Minnesota Teen Challenge (MNTC) responded to my recent blog entry about their anti-drug program, they cited a "study" to back their claims of being an effective treatment for addiction.
What this paper actually shows is how easy it is produce good looking numbers. In that way, it's actually quite instructive for anyone who wants to understand addiction research--or wants to avoid being taken in by exaggerated outcome data.
Even more intriguingly, MNTC failed to mention that the success rate they proudly cited is for adults--but buried in their own study is a much more dismal picture for teenagers.
Their uncontrolled research includes the classic statistical ploys used by drug programs for decades to inflate success rates. At first glance, a claim of 74% of graduates abstinent without relapse for six months sounds pretty good. OK, it's only six months--but often, if you can make it six months at a time without relapsing, you are doing pretty well.
Let's look a little closer at that number and how it was generated, however. Start with the fact that MNTC participants are not just "addicts off the street." They are seeking treatment--either because they have to in order to avoid prison or because they have decided they want to stop using.
Most have been through a detox program to help with withdrawal--and many will have dropped out before completing that. MNTC participants have also consented to attend a highly religious rehab--or had their parents consent for them. This suggests that we have already eliminated many of the addicts who aren't motivated to recover before they even set foot in the door.
Because there is no control group, all of those facts already mean that any success in the program we see could be due to pre-existing motivation: not to anything special about the rehab. Only with a control group of similar people who get no treatment or attend a different rehab can we really tell what works and what doesn't.
The second important clue to pick up is that word "graduates." This means that no one who started but dropped out of the program was counted in the success rates. Since the people most likely to relapse drop out of treatment quickly--and since long-term rehabs like Teen Challenge typically report 50-70% drop out rates-- this means that this 74% does not come from a representative sample from the start.
When you do a clinical trial, you have to include drop-outs as failures to make your results as applicable as possible to the "real world." Why? Well, let's say you have a drug that looks like a fabulous antidepressant in rats. You give it to 100 people and one person stays in the trial and is no longer depressed - but 99 drop out because the drug also causes an intense itching sensation all over the body. Not gonna be a blockbuster--not a useful drug, period. Your stock's gonna tank!
But by MNTC's measure, there's a 100% success with that one person who didn't get the side effect!!!
The executive director of national Teen Challenge has admitted to a 35-40% drop out rate to the New York Times.
That is unusually low for a long term program--so let's use the low end of the typical drop out rate and say that 50% quit without graduating. (MNTC has also said that it uses different practices than the the national organization).
That cuts their 74% success rate in half to 37% if we look at this as what is called an "intention to treat," study and count the dropouts as failures. That already puts them around average compared to other programs. Your average drug program can get about 30-40% clean for six months--but most of their studies have the same problem of having selected for motivated patients before the research even begins.
And with MNTC, there's another wrinkle. When they surveyed the graduates, only 55% agreed to participate in the follow up study. This is another red flag. If you think about it, who is most likely to be available to participate in a follow up on rehab success--the guy who is employed and off drugs or the one in jail or in the basement smoking meth?
Who is most likely to want to talk to a representative of a rehab--the woman who got clean or the one who is now depressed about failure, particularly if it's a religious rehab where failure means return to sin? Even if the surveyors state their independence from the program, it's hard to know whether addicts will actually believe this.
We can't assume that all of the 45% of the graduates who don't respond failed--it's just much more likely that the responders will be successes than failures. So, if we again give them the benefit of the doubt and assume that only half of these have relapsed (again, a very generous assumption), we can say that roughly 31% of the total sample has stayed off drugs for six months. If we count all of the nonresponders as failures, that yields a 20% abstinence rate at six months for graduates.
Which is pretty close to the 15% who get clean, on average at any given time, with no treatment whatsoever! (Of course, if you wanted to lie with statistics, you could say that you "improved recovery rates over no treatment by 33%!!). The extra five percent, however, could well be a selection effect due to the motivated patients being the only ones who started treatment in the first place.
In addition, there's an even more frightening statistic hidden in the MNTC report (and not mentioned by the program's defenders when they complained about my reporting). The 74% six month abstinence rate is for adults.
Only 37% of the teen graduates reported having been abstinent for the most recent six months--and only 29% of teens reported having had no drugs at all since they left the program. And all the same self-selection issues apply to the teen sample--so the real rates are probably below the natural recovery rate.
To be fair, however, the natural recovery rate is probably lower for youths because most people who stop using without treatment do it when their peers are stopping or cutting back, which tends to be in the 20's or later. And, to be even more scrupulous, only 15% of their sample was teens - so these statistics could rely on a sample size too small to be reliable.
Nonetheless, there is absolutely no basis for this program to make any claim of superiority to other treatments-- some of which do demonstrate greater success without playing with data and without the risks carried by being outside of mainstream medicine.