Put her out to pasture with Jeni Barnett

There appears to be a bit more to the story. For instance, The Daily Mail article says,

Dr Wakefield said...‘We didn’t make any claims that MMR was the cause of anything. The exam question completely misrepresents what we said. The Lancet study received no funding whatsoever.’

and

Last night AQA apologised for any ‘misunderstanding’ and removed the GCSE January 2008 Science paper from its website, where it had remained active for schools to use as a test paper.

Martin, your excellent review of the paper is a dead link (404 error). I think this is likely the review you were referring to:

http://briandeer.com/mmr/lancet-greenhalgh.htm

Doh, made a bit of a cock up with the link to the review, I've fixed it now.

Fascinating quote from Wakefield - I missed that when I wrote this at 3am. A couple of interesting things though - first that this is a 15-month-old paper and they've only just complained about it, secondly that it's ludicrous for Wakefield to suggest that he didn't  link MMR and autism.

It's a shame that the original question isn't online, because it would be interesting to see if Wakefield's other protest about funding held up - The Daily Mail quote that question as "Might Dr Wakefield’s research have been biased?", not "Might Dr Wakefield’s Lancet study have been biased?", which if accurate means that Wakefield is either confused, or obfuscating the issue.

It appears to me that several of the principals both for and against the MMR/autism hypothesis had conflicts of interest. I think Wakefield is picking his words extremely carefully when he makes some of his denials, whereas on other claims it would appear that the Times reporter got his facts wrong. Indeed, the whole conflict over the Wakefield paper really is a tar pit the AQA would have been wise to steer clear of, as it now apparently realizes.

All that notwithstanding, the "link" between MMR and autism has been pretty well demolished in the Danish meta-study published in the New England Journal of Medicine. It can be found at:

http://content.nejm.org/cgi/content/full/347/19/1477

Too bad the Daily Mail didn't reference that in its article!

The word I've seen used at Scienceblogs for this kind of thing is "manufactroversy". No actual controversy in the science, but the appearance of controversy can be created, and controversy in politics and the understanding of the lay public perpetuated, by endless bad reporting.

Although I can't pick out any one thing from the question that is factually false, I do think it's in poor taste to use an example from a specific case like that. A hypothetical one would have done just as well.

@Mike I do have some sympathy with the view that it's harsh to name-and-shame a guy in an exam question. That said, there are two factors that swing it back the other way for me:

Firstly, I think it's important for kids to be able to connect science to public issues. The main point of this blog is to massage my ego, but also it's advocate science-based policy. This question gives the kids a real life issue that they can look at and go "huh, if I apply a bit of reasoning to this, I can learn something that applies to my real life."

Secondly, science is based on public criticism. As a scientist you present your work in public, and then you have to defend it in public. Wakefield has already been disgraced in front of his peers, and in front of the public - an exam paper makes little difference now, and public criticism is what you sign up for when you become a scientist.

I recall from my own school days (which were not all that long ago, appearances to the contrary) that the big boys who sat at the back of the class would constantly shout out: "But, sir, what are we ever going to use this for?"

Hypotheticals work perfectly well but "you might recognise this story from the real world, it was rather big news a few years ago" is better for students you're barely hanging onto as it is. I think that's why it was chosen. It's a real world example used as an application of the dry principles that were presumably drummed into the student's heads in class.

I think it'd be horribly poor taste to lie about a real life person, but I'm not so sure I'd say the same about accurately reporting the current state of affairs?

Hooray for reason a voice of reason.

Beezy Marsh is a waste of space and there are all too many like her writing about stuff they don't understand.

Your links to the exam paper and mark scheme are broken

You can find the paper and mark scheme on my site: http://justatheory.co.uk/2009/05/10/daily-mail-reports-gcse-pupils-brain...

Oops, just fixed the links.

“...the latest sorry excuse for health journalism in the Daily Mail”.

Conscientious and well-informed reporting and commentary on issues in applied science is libel. Irresponsible and ignorant misrepresentation of them is a stock-in-trade of the profession.

I can't really see why AQA pulled the paper and mark scheme off their site. They should've stuck by their examiners who set the questions and certainly not apologised.

{Besides we have quite a few copies of the paper at school as well as having it up on our internal network (and almost certainly hundreds of other school science departments).}

There are similar questions posed on many Year 8 test papers too - as part of the "Microbes and Diseases" topic. Will the QCA be pulling that from schemes of work too?

Marsh is one of many serial offenders it seems - so many journalists represent themselves as authorities in fields that they don't even begin to understand.

As for the exam question, seems perfectly sensible to me (Mike's fair point about naming and shaming aside, which I think Martin deals with well...), especially as a year or two after the original Lancet paper, I was given the publication as a tutorial exercise @ uni and told to critically evaluate it - and we pretty much came up with the same flaws as Greenhalgh did.

Martin you're spot on mate, this is exactly he sort of question A-level students ought to be asked - shame the AQA appear to have buckled to some hate-mongering tabloid...

Call me old fashioned, but I don't think this sort of question belongs in a GCSE science paper at all. It could be a useful subject for a more in-depth project to do with the operation of science policy, the way media works and as such fits onto some form of social sciences. Understanding the basic flaws in the methodology, the application of statistics, the distortions involved and the way that scientific results are published is, even in the context of this one study, a complex subject. However, as a straight multiple-choice question it isn't going to demonstrate any knowledge of science whatsover - just a bit of rote learning in class so the right box could be ticked. Much the same could be said about questions involving global warming where they are being dealt with at a superficial policy level.

I'd much rather that questions in science GCSEs actually demonstrated some understanding of the basic science itself. However, these appalling multiple-choice options don't allow for that at all with the incredibly simplistic right/wrong binary switch. What I see is an increasing politicisation of the subject and a neglect of the basic subject itself.

Note that this doesn't mean in any way that I agree with the Daily Mail's handling of any part of the MMR story of course.

Oops - see that it isn't mutiple choice now I've read it - however, it's still not a science question. More a test of comprehension. The questions about recreational drug usage aren't really biology either - more social policy/social sciences. Perhaps the subject of and in-depth project, but not an informed answer based on a few lines of test.

Really a dreadful way of evaluating knoweldge of biology. Whatever happened to studying basics.

This seems to be a general approach to GCSE science apapers - neglecting the basic skills in favour of answering some larger, vaguer and complex "big questions".

@Steve Jones: "however, it's still not a science question."

See I disagree with you there. They're presenting information about a clinical trial, and asking the pupil to evaluate the trial against the scientific method - that's bread and butter medical science and scientific method in my book. It would be nice to do coursework on the topic, but I still think it's a good exam question.

On the broader point, it's also worth pointing out that this is just one part of the science exam - the other papers have a different emphasis, so it's not all about policy.

@Martin on Mon

It's not science as I studied it at school or university. The question doesn't lay out the way the trial was done, what the results were, what controls were used. I was very practiced at carrying out experiments, dealing with observer effects, controlling for other variables (albeit in the much simpler area of physics) and there is nothing in that question about what was done.

If, what had been laid out was how the tests were performed, what the results were so that the pupil could have assessed it then that would be fine (in which case, then why does the case have to be named). However, there is nothing in the question of that sort.

What is included is a statements that essentially say that the parents (and lawyers) had already made their mind up, Dr. Wakefield results were biased by that and the way the study was funded. Now that may well be true, but that's a social science question, not a biology one. There is, of course, a general question about scientific method, but all we really know from the wording is the state of mind of the parents and how the study was funded.

Can I then ask another question - what relevance does the paragraph "some newspapers used parts of the report in scare stories" have to the basic science rather than the reporting of science in the media? Again that tells us nothing about he understanding of biology, but it certainly could be appropriate in a social sciences or media studies course.

I should add I have no time at all for the Mail's part in the whole sad MMR story and it needs to be challenged. However, I think it counter productive to introduce what are leading questions into a GCSE paper on biology.

Whilst I am at it, there are some other dreadful features of that paper. The questions over recreational drug use are very debatable. There is one question that conflates cause and effect (question 1(ii) - the "likely to lead to" is a directly implying there is a causation rather than a correlation; a fundamental mistake made by many policy makers). On question 1(i) you have to ring a yes/know answer to a question where the correct answer is probably that it doesn't eliminate the possibility that using cannabis leads (ie causes) going onto class A drugs. I'm not even quite sure what the "right" answer is in this case.

On the pure science questions, far too many of them can just be answered with a bit of common sense, reading between the lines, basic comprehension and some ability to interpret a few charts. I'm very confident I could wing this exam comfortably and I never even took O level biology. This is also meant to be the higher tier paper.

In all, a dreadful paper, and the worst thing about it is putting ammunition in the hands of the anti-MMR campaigners. To my mind science questions in GCSE should be based on scientific principles and not leading questions. Too much of that of all sorts here.

Good science at its best ...

Dr Martin 'Cock' Robbins, the deluded McScientist said,

"It isn't a matter of opinion, but cold scientific fact that Wakefield's study was bad science, even before Brian Deer established that data was misrepresented by Wakefield."

But Brian Deer has reported that Richard Horton thinks Wakefield's study was good science ... and apparently still does.

http://briandeer.com/mmr/horton-wakefield.htm

I gather Richard Horton is still editor of the Lancet and still dishing the dirt on his former friend and colleague, Andrew Wakefield ... but not under oath.

@CelticLeopard:

1) At no point in the piece you cite does Richard Horton suggest that the study was "good science".
2) The reasons why the study was bad science are clearly explained above. Which of them do you disagree with?
3) Angry ranting is amusing, but probably won't persuade many people, especially when you don't deal with the substantive points raised ;)

Classically storm-in-a-teacup tooth-aching stupid from Beezy.

But then what else is new?

The central problem is a familiar one to anyone who has ever marked a question that tries to test reasoning. Do you give a mark for "the right answer", or do you give it for reasoning?

Let's take a bit of the Marsh article:

Answering ‘yes’ to the question ‘Might Dr Wakefield’s research have been biased?’ earned another mark if it was backed up with a comment about him being paid by ‘parents/lawyers’.

The key word in Marsh's paragraph above, though she appears not to understand its significance, is "if". Namely, if the kids could suggest a legitimate and recognised reason why Wakefield's study might have a bias - and funder bias and/or "potential conflict of interest" are well recognised in medical research - they get a mark.

If they write "Yes 'cos Wakefield iz a bad man", then they don't.

Ditto something like "based on a tiny number of patients". Or "what if he only saw kids whose parents were sure they knew the answer" Marks for reasons, and reasoning. Not for "yes / no".

It is a little bit Mickey Mouse, and it is not a substitute for an understanding of scientific method grounded in considering how one constructs experiments, but it is not brain-washing.

By the way, if you want an insight into the kind of Conspiracy-verse mindset of people like Beezy Marsh, go to the post on my blog that Martin linked up above, and scroll down until you get to the quote from a comment made by Beezy's fellow journalist and MMR conspiracy enthusiast Sally Beck. (NB The original comment was at Left Brain/Right Brain and can be found here). It is quite enlightening.

1. Are you saying that Richard Horton, as editor of the Lancet, does not regret publishing bad science?
2. That Dr Richard Horton didn't think the study was bad science at the time he published it, and still doesn't, is good enough for me.
3. Eh? Where was the 'angry rant' that wasn't going to persuade any deluded folk, including Cock Robbins?

And there was me thinking Celtic Leopard, AKA Cybertiger, AKA Dr XXXX XXXXXXXXX, of the XXXXXXXX Surgery, two faced "jabber when he gets paid, anti jabber when he doesn't" had either been hit by a bus, had choked on his own hypocrisy, or had been banned by his employers from spouting shite on teh interwebs.

Dr XXXXXXXXX - I think you'll find the given middle name of "Cock" is more generally applied to your friend and co-liar, John "Cock" Stone.

(Edit: Apologies for censoring, but these days I have to be a bit careful. - Martin)

@CelticLeopard:

The article you cited says absolutely nothing about the issue of whether the study was bad science or not. If you have some quotes from Horton about the issue then we can link to them. If not, then you're basically just making stuff up.

Also, even if Richard Horton did think it was good science - which as far as I'm aware he has never said - it's a bit weird that you're appealing to the authority of one doctor who agrees with you, yet dismissing the many who don't. Horton is the editor of a magazine, and a man who has published no research on the subject - what makes you accept his authority?

Of course you've completely avoiding answering my question: which of the several reasons I gave why the Wakefield paper is bad science do you disagree with, and why? If you can't answer that, then you don't really have an argument...

The test info is too limited. First: the bit of text does not fully explain the actual story. Any clinician anywhere might make a serendipitous observation that gives rise to a new theory. That serendipity is entirely a normal part of the scientific process. But is not the endpoint of the scientific process. The next steps involve increasingly refined testing of the hypothesis, with falsification as an inherent requirement of the hypothesis. Not in this test is the fact that Wakefield jumped from a happenstance observation to full-blown confidence. He skipped several steps that would have been required to lable this effort "scientific." But that flaw is not in the test.

Additionally: the second test item mentions bias. All science involves a bias. That is exactly why scientists add the science rules-of-the-game on top of these biases. There is not scieitist out there simply sitting around, generating random hypotheses and testing them. No. Instead, scietists dream up various ideas, such as "MMR might lead to autism." Then, to test whether this hypothesis has any truth, scientific methods are used to test and refine the hypothesis. All along, the scientist has a bias. This is exactly why we need science when we evaluate any such claims. As they arise, serendipitously, from random observations. All of this is normal science.

I'd agree that the test info is limited, but then this is a GCSE paper, not a comprehensive treatment of the reasons why the study was flawed.

@CelticLeopard:
Any answer yet?