[bpsdb] When I blogged about the adventures of homeopath Jeremy Sherr recently, a couple of people leapt to the defense of his dubious trade. In particular, one poster called "Prolix" made the following challenge: "...skeptics fail to report the studies and meta-analyses which report positive outcome The National Center for Homeopathy has compiled a list of them. It's important to look at the evidence BOTH sides of an argument, not just the side you favor."
I'm grateful to Prolix, because the resource he cited neatly demonstrates why scientists have so little time for homeopathy, and indeed find many homeopaths to be fundamentally dishonest people. In this four part case study over the next few weeks, I'll take you through the NCH's use of peer reviewed research, and try to explain where they've gone so badly wrong.
Leaving aside books and cost-studies, the National Center for Homeopathy lists 56 research articles that it claims support homeopathy, spread across four broad categories: 25 peer reviewed journal papers; 15 "other research"; 6 meta-analyses; 10 "basic research". Now, already there is a really big problem with this list. A quick search for "homeopathy" on Google Scholar throws up 32,000 articles, yet they cite just 56. Many dozens of meta-analyses can be found, yet they cite just 6.
This is cherry-picking in the extreme, and this alone pretty much invalidates their exercise. There are millions of research papers published, and you can find something to support pretty much any position if you look hard enough - what matters in any subject is the overall evidence-base, and if there are ten thousand pieces of research on a topic but you can only find a handful supporting your position, then the chances are it's a pretty poor position. Or let me put it another way, if you asked a random yes/no question to 32,000 people, and 56 answered "yes" but 31,944 answered "no", which do you think would be the most likely answer?
But let's be charitable and ignore that for now, and work through the evidence that they provide. I'll begin with the strongest evidence - the meta-studies. Meta-analysis is a powerful tool of evidence-based medicine, taking lots of studies that have been performed on something and combining them to get a much more accurate picture than you tend to get from individual trials, which of course may be flawed or biased.
They cite just six "meta-studies". I use the term in quotation marks because in fact the first three aren't meta-studies at all, so quite why they've been included I don't know. The kindest thing to say would be that the folks at the NCH don't really understand what a meta-analysis is. The remaining three, all produced by good scientists and published in quality journals, have all been completely misrepresented, generally using selective quote-mining. They cite a 1991 study by Kleijnen [1] as supporting homeopathy, when in fact the authors state that "definitive conclusions about homeopathy could not be drawn, because many of the trials were not of good quality and the role of publication bias was unknown," and describe the evidence as "not sufficient."
The other two studies are by Linde from 1994 and 1997 [2,3], and again they claim that these support homeopathy. In fact the 1997 study concluded that there was, "insufficient evidence that homeopathy is clearly efficacious for any single clinical condition", while again complaining about the poor quality of trials. The conclusion of the 1994 study was equally damning, with the authors noting that: "The quality of evidence in these studies was low with only 43% achieving one half of the maximum possible quality score and only 31% reported in a fashion that permitted reevaluation of the data." In other words, nearly 70% of the studies they found were so poor, they couldn't even properly evaluate their results.
In summary then, of all the dozens of meta-studies available, they cited just six, of which half weren't actually metastudies, and the other half not only didn't say what was claimed, but actually strongly criticized the poor research methodology of homeopaths. The most charitable explanation is that reading and comprehension are not skills that the people at the National Center for Homeopathy are particularly well-endowed with. An alternative possibility is that they are simply liars.
Next time, in Part II, I'll be moving on to the 10 papers in the Basic Research section.
[1] J. Kleijnen, P.Knipschild, & G.ter Riet (1991). Clinical trials of homoeopathy British Medical Journal, 302, 316-323
[2] Linde, K., & Jonas, W. (2006). Are the clinical effects of homoeopathy placebo effects? The Lancet, 366 (9503), 2081-2082 DOI: 10.1016/S0140-6736(05)67878-6
[3] Linde, K., Jonas, W., Melchart, D., Worku, F., Wagner, H., & Eitel, F. (1994). Critical Review and Meta-Analysis of Serial Agitated Dilutions in Experimental Toxicology Human & Experimental Toxicology, 13 (7), 481-492 DOI: 10.1177/096032719401300706
http://layscience.net/trackback/492
So what you are saying is that they are just doing the same thing the pharmaceutical industry regularly does to get their drugs marketed - picking and choosing the studies that support their case e.g. Vioxx, Seroxat etc. You say 'An alternative possibility is that they are simply liars.'I say - a plague on all their houses! - they are all at it.
Exactly, Anon. I was about to make the same point, but you beat me to it.
And bloggers like Ben Goldacre and myself are more than happy to criticize pharmaceutical companies when they do it, and indeed have done so on various occasions, in spite of being the Pharma Shills we clearly are.
But this isn't particularly about marketing, it's about the evidence-base for homeopathy full stop.
Martin is the editor of layscience.net.
Follow Me!
RSS | Twitter
This pharmaceutical industry argument is intriguing - Anon & Carl Olsen, are you *really* saying that because big pharma is being dishonest and playing fast and loose with the statistics, then Alternative Medicine research should be allowed to do the same?
What sort of nonsense is that?
Surely it would be a better world if CAM produced some clear, well-implemented squeaky clean research to throw in the face of Big Pharma to get them to clean up their act?
Your argument of "if they can do it why can't we?" is bananas.
But this isn't about marketing, it's about the evidence-base for homeopathy full stop.
Anonymous' comment wasn't about marketing, either, Martin. As he clearly states, the pharmaceutical companies pick and choose the studies that support their case (indeed they are involved in the design of the studies i.e. animals vs. people) in order to get their drugs marketed, by which he means "approved for sale" by the FDA, etc. For example:
www.wired.com/medtech/health/news/2005/07/68260?currentPage=all
They can only begin marketing once they've garnered that approval.
As for the evidence base for homeopathy, my reading suggests that there is some evidence that it works for some conditions, but the jury is still out. You evidently feel differently. I'll be interested to read your views on it.
When considering Linde et al. (1997), another paper by substantially the same team of authors is also relevant: Linde et al. Impact of study quality on outcome in placebo-controlled trials of homeopathy, J Clin Epidemiol. 1999 Jul;52(7):631-6. This re-analysed the same data, with particular attention to study quality, and concluded that the 1997 paper "at least overestimated the effects of homeopathic treatment". And perhaps it is also worth mentioning Linde and Jonas's 2005 letter to the Lancet (Lancet. 2005 Dec 17;366(9503):2081-2), which while criticizing the 2005 Shang paper (and more strongly criticizing the editorial that accompanied it) stated, "our 1997 meta-analysis has unfortunately been misused by homoeopaths as evidence that their therapy is proven".
Dr*T, you start by asking a question as to what we said, then proceed to answer it incorrectly. That is indeed nonsense. Nowhere did either of us make such a claim. Indeed, Anonymous says, "a plague on all their houses!"
I'm not prepared to go that far without seeing the evidence, which Martin assures us is forthcoming. However, I feel confident in stating that unlike Big Parma, the homeopathic industry hasn't repeatedly covered up evidence that any of their drugs harm the patient. Please correct me if I'm wrong.
Martin, you claim that because the NCH lists 56 articles favorable to homeopathy and you get 32,000 hits for “homeopathy” on Google Scholar, that there are therefore 56 favourable articles and 31,944 unfavorable ones (“…if you asked a random yes/no question to 32,000 people, and 56 answered "yes" but 31,944 answered "no", which do you think would be the most likely answer?”).
Surely I don’t have to point out the many gaping holes in such an approach? If you’re going to take people to task for scientific bias, then I suggest you need to get your own house in order. Just saying.
As for the NCH possibly being liars, well perhaps they are, but if you want to see a real horror story Google Scholar “vioxx aftermath” and click on just about any of the 20,100 entries. Two wrongs certainly don’t make a right, but the wrongs of the homeopathic industry pale in comparison to those of Big Pharma. It’s as if you’re complaining about the pug’s drooling while ignoring the Great Dane shitting all over your carpet.
Lol, that's a slightly disingenuous point - since homeopathic products don't actually contain any active ingredients, just water, there's not really any way that homeopathic practitioners could cover up side effects. The issue with homeopathy is that it simply doesn't do anything that can't be achieved by a similarly administered placebo.
Martin is the editor of layscience.net.
Follow Me!
RSS | Twitter
I don't actually intend to claim that at all, my point is that there are many thousands of studies available, and selectively picking 56 of them that look a bit positive is meaningless. Or more broadly, my point is that it's just plain cherry-picking.
I still don't get the comments about Big Pharma. To repeat, this article isn't about Pharmaceutical company marketing practices, it's purely about the relationship between homeopathy and science. I've strongly criticized pharmaceutical firms in the past, including on Seroxat, the issue of direct to customer marketing, and failures of FDA regulation, and no doubt I will again, I just don't see the need to bring it up in an article about homeopathy.
Martin is the editor of layscience.net.
Follow Me!
RSS | Twitter
The jury has been out for a couple of centuries now. Maybe they aren't coming back.
Fair enough; so your point is correct but your data was wrong.
Yes, Martin, the mechanistic viewpoint has a certain validity, just as does Newtonian physics, but in this age of entanglement theory, the yet-to-be explained but incontrovertible results of countless ganzfeld experiments, etc. there may be a bit more to the story. The only way to know for sure is to continue with careful experimentation. Perhaps, like parapsychologists, homeopathic scientists will learn from and even work with their critics. Only time will tell. Till then, the jury is out.
So do have to ignore the countless research showing homeopathy to be no better than placebo and continue looking for the one unambiguous paper of a homeopathic preparation unequivocally curing a non-self limiting disease? In my view no. The jury isn't out - at some point you have got to put your efforts into something more reasonable.
Johann Hari has a good article about "what about-ery" here.
You need to get over the fact that there can be more than one "bad thing" happening at any one time.
What has Big Pharma research got to do with sloppy homeopathic research?
Dr*T, to quote from Hari's article, to which you linked:
Say it slowly: there can be more than one bad thing in the world. You can oppose American atrocities, and Chinese atrocities. You can be critical of Israel, and of Islamism.
In the article's examples, one set of atrocities is used to excuse others. As I previously explained, nobody here has made such an argument. One can be critical of homeopathic research and allopathic research at the same time, just as "anonymous" was.
The point I tried to make above is that homeopathy's detractors claim that the tests showing modestly positive results lack rigor. This is exactly the same situation as that faced by parapsychologists. In the latter case, they developed extremely rigorous test regimes, sometimes in co-operation with "debunkers", yet the modestly positive results refused to disappear, despite the supposed "impossibility" of psi phenomena. Therefore, I maintain that the jury is out regarding the efficacy of homeopathy until repeated similarly rigorous testing resolves the issue one way or the other. That hasn't happened yet. Claiming a priori that homeopathy can't work simply isn't science.
Actually, the efficacy of homeopathy has only become a major issue in the last few years, as it has increased in popularity (and continues to increase), largely due to the side-effects of virtually all allopathic drugs. It's therefore imperative that carefully designed studies by both proponents and doubters continue to be carried out and critiqued, as they are.