In recent weeks myself and a few others have engaged with some CAM practitioners via Twitter, and one of the things that keeps coming up is a version of the following argument:
CAM Dude: "This study supports my therapy!"
Scientists: "No it doesn't!"
CAM Dude: "Yes it does!"
And so on ad infinitum. The problem seems to come down to a lack of understanding of methodology, so in this post I'm going to walk through a study cited by "drmike001" as supporting acupuncture, and describe what they can and can't show, and what would have to be done to really support the conclusions that he wants to arrive at.
Recently, Mike cited a couple of studies, claiming that they demonstrated that acupuncture worked - in particular this paper by Haake et al, 2007, which produced the following conclusion:
"Low back pain improved after acupuncture treatment for at least 6 months. Effectiveness of acupuncture, either verum or sham, was almost twice that of conventional therapy."
Now, the trial supports two hypotheses: firstly, that real acupuncture works no better than a placebo treatment, and secondly that even a placebo treatment can be surprisingly effective.
Dr. Mike doesn't see it this way, and this is where his apparent lack of understanding of clinical trials comes into play. In order to rationalise the result, he simply states that the sham acupuncture is in fact a form of real acupuncture, and that the study shows that both are effective. It's easy to see how this analysis could be attractive to the layperson, but there are two major problems with it.
Firstly, if Dr. Mike accepts these results, then he must also accept that the study shows that much of the theory behind the "genuine" acupuncture is nonsense. The study showed that careful placement of metal needles to align with imaginary energy points in line with acupuncture "theory" had zero effect on the treatment - randomly sticking cocktail sticks in had just the same effect. In short, he cannot have his cake and eat if - if he claims that the sham technique is genuine, then he must also disown the conventional treatment.
Secondly, Dr. Mike cannot claim that this is a placebo-controlled trial, while also claiming that the placebo is a real treatment. In order for Dr. Mike to test his hypothesis that the effect of what he calls "dry needling" is more than just placebo, he would have to cite or perform a new trial comparing the toothpick treatment with a similarly administered placebo.
Dr. Mike's response to this predictably missed the point: "can you quote any RCT (not acup) in which the 'placebo' arm is 2x as good as an active comparator?"
This is a daft question for two reasons. Firstly, placebo arms are nearly always similarly administered, and that's not the case in this trial. A drug and a placebo administered the same way should have the same degree of placebo effect, therefore the drug isn't going to be less effective. In this trial that isn't done, and so all bets are off.
Secondly, it completely ignores the points raised above - you still need to rule out the placebo effect, because the only demonstrable fact here is that Dr Mike has no idea how large it is for the toothpick therapy. I also strongly suspect that he doesn't realise just how awesome the placebo effect can be. It is a very powerful thing, which is precisely why we use placebo-controlled trials to begin with.
The results of this trial are actually quite fascinating. They are consistent with an increasing body of evidence we have on the clinical effectiveness of the placebo-effect, and this may have profound effects on health policy - I've blogged previously about the possibly role of "fake" treatments in dealing with syndromes that have psychosocial risk factors. Even a quack treatment like acupuncture could conceivably have a role to play.
But this requires open and honest assessment of the options, and what they can and cannot do. As Ben Goldacre put it when the study came out back in 2007:
"Back pain is clearly a problem which requires more than simply pharmaceutical pills. The question is whether an elaborate, expensive, gimmicky and theatrical placebo ritual is an effective use of money, or whether other, cheaper, more pragmatic, honest psychosocial interventions might be more appropriate and cost effective."
And that's another important point there - honesty. If acupuncturists are willing to embrace and accept the evidence that the benefit they provide to patients comes from the theatrical, psyscho-social intervention they perform then I believe they could have a place in in health care. If they continue to suggest to patience that they are tinkering with some mystical energy force then - unless by some fantastical chance they can produce convincing evidence for it - I will continue to have little respect for them.
Edit - Dr. Mike Replies:
So since posting this, Mike has replied to me via e-mail and twitter. There are two points in particularly that he was unhappy with, so in the interests of fairness I'll note them here.
1) Mike believes I misrepresented him when I said that he claimed this study supported acupuncture: "I didn't claim that Haake supports the efficacy of acupuncture - it simply suggests sham acupuncture cannot be a true placebo." My response is that if you're claiming that sham acupuncture wasn't a "true placebo", then by definition you're either claiming that it was an effective treatment, or that the trial was invalid. And of course the study suggests nothing of the sort, since if his assertion that this wasn't a true placebo were true, then the trial, er, wouldn't be placebo controlled. As I said above, you can't have it both ways...
2) Mike points out that "Western medical acupuncture is about sensory neuromodulation not about points and meridians.." Well, firstly as far as the evidence goes they may as well be the same thing, but more importantly the study cited does not test this hypothesis, and in fact looks at Chinese acupuncture versus placebo. So why bring it up?
http://layscience.net/trackback/574
Nice article
On your last point, though, it doesn't seem unreasonable to think that if acupuncturists admit their treatment is a placebo, which is essentially what you're saying by being more honest, then the effect will disappear, and any benefit to the patient will be lost.
Interestingly, if you check out Ben's post that I linked to and some of the studies cited there, there have been trials where even telling the patient that they were taking a placebo didn't diminish the effect. But until we have more information I wouldn't like to assume either way.
Martin is the editor of layscience.net.
Follow Me!
RSS | Twitter
Thank you for helping to explain methodology. I would also like to refer to a great summery by Dr. Steven Novella that references a more recent study where similar misconceptions are taking place in the CAM community.:
http://www.sciencebasedmedicine.org/?p=492
It was this summery and study reference that led Dr Mike to tweet to me:
"if toothpicks work and are equivalent to real acupuncture in cLBP, we should use toothpicks since they are safer "
Wouldn't this mean that all acupuncture is called into question at this point... (no pun intended).
If the placebo and treatment receive similar results and it doesn't matter where you stick the needles, or if you stick the needles or even if you use NEEDLES at all, then what treatment are you left with? At that point there is no acupuncture left to test and you are just randomly prodding and touching patients.
The Cochrane report had an interesting article; http://www.cochrane.org/reviews/en/ab001218.html in which it is suggested that acupuncture may be of use for migraine. As with your blog, it agrees that needle placement is irrelevant.
I was surprised when I read the Cochrane article as I had always thought that acupuncture was another load of old tosh. Again, read the 'Trick or Treatment' book for a full expose.
It's good to be brought up short sometimes.
Clearly in this case, the acupuncture was doing something else, the question is, what?
Be interesting to dig into that one.
As for your methodology argument - it will fall on deaf ears. Humans are very good at picking good news and completely ignoring the inconvenient. Hence the need for the scientific method. A method, which in/conveniently, will be filtered out/ignored by those that don't know better.
Bill.
Er... how do you resist the temptation to twitter back:
"Dude, your argument is most totally bogus"
- or similar? Don't think I'd be able to.
@Bill "As for your methodology argument - it will fall on deaf ears."
That, I think, is the most frustrating thing. He doesn't seem to understand how to construct and evaluate a study to test a hypothesis, and as a result he makes claims that are not just unsupported, but often mutually contradictory. The same goes for many pseudo-scientists.
But as you say, there is a genuine set of phenomena to look at here - the placebo effect, and the impact of psychosocial factors on disease. These are fascinating things, and it's a shame that Dr. Mike is so close-minded to them.
Martin is the editor of layscience.net.
Follow Me!
RSS | Twitter
Haake et al was a large comparative RCT, not a placebo controlled RCT. Placebo is only mentioned in the discussion, which is worth reading Martin.
The trial demonstrated that acupuncture and sham were significantly better than ten physician/physiotherapist visits, exercise and NSAIDs. NSAIDs work better than placebo but are associated with morbidity and mortality related to GI bleeds.
If acupuncture was a new (orthodox) treatment, we would all be wanting to use it to treat cLBP, because it has been shown to be better than what we currently use in this very large RCT.
Efficacy of acupuncture over sham has already been demonstrated in the Cochrane review (Furlan et al). A large trial with a trend in favour of true acupuncture is likely to strengthen the result of the Furlan meta-analysis, not reverse it...
The German government have decided to reimburse acupuncture treatment for cLBP under the state insurance system, principally as a result of this trial, and the cost effectiveness data.
I guess you will all have heard the news on the NICE guidelines by now.
Mike,
It is factually incorrect that placebo is only mentioned in the discussion of Hakke et al. It was mentioned in the intro, methods, results, and discussion and was a specific placebo intervention in the study.
The study was clearly placebo controlled, with "sham acupuncture" acting as the placebo. If you believe that this is not a "true placebo" because it is an active intervention rather than an inactive substance, then any and all "laying on of hands" placebo controlled trials are, by your definition, not "true placebo".
I'd like to know what you believe a "true placebo" is, and then indicate how one would effectively use a "true placebo" in an acupuncture trial.
Obviously, it is hard to design a control or placebo treatment when the treatment involves sticking needles in people.
If sham acupuncture is as good as conventional Chinese acupuncture, then what is the benefit of using conventional Chinese acupuncture over someone administering a sham treatment? What is the mechanism of treatment in sham acupuncture (I.E. why does it work?) In other studies, sham acupuncture (using needles that do not enter the skin but retract into a sheath) have been shown to be more effective than actual needling (So et al 2009)
The running theory is that people believe it works, and so when acupuncture (real or sham) is administered, it brings about a subjective response, or has an actual hormonal/chemical effect - demonstrates the awesome power of placebo.
Acupuncture trial thoughts
I've never really understood how it's possible to do a double blind trial with acupuncture. I would have thought that the action of sticking sham or real needles in is much more dramatic than taking pills?
What i mean is that taking part in a double blind drug trial would be a kind of 'shrug' the shoulders thing. An 'oh well, let see what happens' affair.
But to lie down on a gurney and have needles whizzed into the skin, sham or otherwise, while it's passive from the patients point of view, it's nonetheless dramatic, as you can see needles being stuck in, must create some kind of mental stir?
Maybe I'm wrong.
When people are selected for acupuncture trials, what's the method for picking out either advanced converts, or people like me, deep down skeptics. Either would be unfair.
Interested to hear.
kind regards
bill
@Bill
It's a good point re selection. The recent acupuncture trial in Norway for menopausal hot flushes had horrendous selection issues. I blogged about it (http://tinyurl.com/rxanpj) and the previous experience of acupuncture + expectation levels were incredible. Basically, it looks like the trial recruited a shedload of acupuncture groupies who had a 50% chance of being randomised to free treatment.
Dr Mike tweeted me to highlight 'quite a big placebo effect though!'. Yes, you will see big placebo effects by chance alone on some occasions anyway but the effect of selection from this trial on the final results renders it invalid for any normal population. (esp in the UK where the use of acupuncture is less anyway). I am not personally sure whether I am comfortable calling all this effect placebo either - to a large extent it is a psycho-social intervention. One that could possibly be provided a lot cheaper without the pseudo-science babble and the need to stick pins in folk.
The answer is that patients should be selected without bias on the basis of their disease, not on the basis of their expectations about the intervention. If the trial is appropriately powered the number of skeptics vs pinheads will balance out.
It had always been my impression that telling someone they were receiving a placebo destroyed the placebo effect--that if I walked into an acupuncturists and didn't believe it was more than a placebo and the acupuncturist made no effort to disabuse me of this fact, I probably would not benefit from the treatment. Or am I totally off base?
you're absolutely correct.
the point i was trying to make though was that there's [in my opinion] a big difference between taking sugar pills and having sham needles pushed into your body.
Cause they still penetrate, just slightly, a matter of a few millimetres. but they still do so.
Bill
you're absolutely correct.
the point i was trying to make though was that there's [in my opinion] a big difference between taking sugar pills and having sham needles pushed into your body.
Cause they still penetrate, just slightly, a matter of a few millimetres. but they still do so.
Bill
Err.. Why NSAIDS? Of course they have a morbidity downside, which is why first line painkiller therapy for lower back pain is paracetamol (dirt cheap and far less problematic).
Comparing things to NSAIDS rather than less heavy-duty and better tolerated painkillers sounds to me rather like a sneaky "loading" of the dice so that you can later say "Ah, but NSAIDS can cause dangerous GI bleeds...".
Actually, I should have said "paracetamol AND EXERCISE". What CAM enthusiasts repeatedly elide is that long-term dosing with powerful drugs is NOT really the desired endpoint of giving painkillers for most lower back pain. What is needed is to mobilise the patient. The painkillers help do that, but a lot of the pain is often a consequence of inactivity. "Get 'em up and moving" is the really therapeutic bit, surely?
NSAIDs are relevant because, as anyone who regularly treats patients with back pain will know, they are frequently prescribed by GPs or patients self administer OTC prepartions. Whilst long term use of medication may not be desirable it is a clinical reality. I believe that back pain patients are greatly let down by many NHS GPs and therapists who give inconsistent advice and treatment.
I agree that short term use of suitable pain relief is appropriate and for patients whose pain is not settling mobilisation, exercise and patient education should be considered essential.
The addition of acupuncture and spinal manipulation by osteopaths and chiropractors as reviewed and recommended by NICE offers patients choice and is associated with good clinical outcomes and high levels of patient satisfaction. This model does appear to be what patients and (I believe) most GPs want and is essentially what is happening with patients seeking (if they can afford it)private care.
Rather than taking polarised views on reductionist science vs CAM with all the offensive jargon that fills the blogs we should be looking at ways of working together to understand how we can improve clinical outcomes, improve patient choice, reduce health inequalities and fund further research.
Topical NSAIDs do not have the same side effect profile as oral NSAIDs and appear to be just as effective - which is why they are recommended before oral NSAIDs in the NICE osteoarthritis guidelines. Strangely they get no mention in the chronic backpain guidelines.
This is either because the backpain people used over specific search criteria, or due to biases in the panel. I am still unclear as to how the two guidelines are supposed to relate to each other given that the backpain guidelines explicitly don't consider backpain from osteoarthritis but also rule out any investigations which might establish that whether backpain is (or rather, could be) due to osteoarthritis - which means that most people with osteoarthritic backpain will be treated under the backpain guidelines, even though those guidelines do not consider studies treating osteoarthritic backpain (or backpain from any other known pathology).
@ Bill with the double blind trial comment
I think the dramatic effect of having needles stuck in you, sham or otherwise, would be part of the placebo effect such trials are looking for. Also, I don't think you watch the needles going in. I've had it done a few times when I was younger (my mum was into it then), most of the time I was face down, but even face up you dont tend to crane your neck to look.
I have also heard it said that the sham treatment (ie no needles going in - retracting or some other mechanism) is possible because people don't actually feel it when they do go in, but in my experience I definitely felt them going in (as in under my skin, not just touching the surface), and felt them being taken out too. Would this mean the trial used only people unused to acupuncture? Otherwise those getting the sham treatment would know something was up
That wouldn't prevent the other kind of sham treatment, where needles are placed randomly, but the "theory" behind it is so implausible, its easier to believe that sticking needles in anywhere might do something, so its a shame that is harder to test.
Finally, none of these methods would produce a double blind trial, and I cannot think of a way the trial could possibly be made blind for the doctor/practitioner.
Can anyone think of some clever ways around these problems?