Better Health: Smart Health Commentary

Featured

Bad Science: Nine Ways To Dress Up Flawed Research

December 4th, 2008 by Dr. Val Jones in Quackery Exposed
No Comments »

This post is a follow up to my book review of Bad Science, located here.

***

I couldn’t help but feel unusually depressed by Dr. Ben Goldacre’s exposé of researchers who resort to trickery to get their articles published in peer-reviewed journals. There are a number of ways to manipulate data and many ways that flawed research is presented to enhance its chances of publication.

Before we get started, I need to point out that “negative trials” – or research results that don’t corroborate the investigator’s original hypothesis/es – are much less likely to be published. People (and/or publishers) are far more interested in finding a needle in a haystack, than hearing that no needle could be found.  This is a driving force in all manner of mathematical convolutions aimed at demonstrating something interesting and to warrant publication. After all, who can blame the researchers for wanting to get their research published, and to have it make a meaningful contribution to their field of study? Who wants to toil for months to years on end, only to discover that their hypotheses were not born out by experimentation, and in fact no helpful conclusions may be drawn whatsoever?

And so, with this intense pressure to find something meaningful in one’s research (either for profit or personal satisfaction and professional advancement) – there are some typical stragegies that researchers use to make something out of nothing. Ben Goldacre reviews these strategies in the voice of an unscrupulous senior pharmaceutical investigator, giving advice to his junior colleague. (Parenthetically, it reminded me of The Screwtape Letters – an amusing book written by C.S. Lewis, featuring the imaginary advice of a senior demon to his junior counterpart as they tempt humans to evil.)

(This passage is taken directly from pages 192-193 of Bad Science)

1. Ignore the protocol entirely. Always assume that any correlation proves causation. Throw all your data into a spreadsheet programme and report – as significant – any relationship between anything and everything if it helps your case. If you measure enough, some things are bound to be positive by sheer luck.

2. Play with the baseline. Sometimes, when you start a trial, quite by chance the treatment group is already doing better than the placebo group. If so, then leave it like that. If, on the other hand, the placebo group is already doing better than the treatment group at the start, then adjust for the baseline in your analysis.

3. Ignore dropouts. People who drop out of trials are statistically much more likely to have done badly, and much more likely to have had side effects. They will only make your drug look bad. So ignore them, make no attempt to chase them up, do not include them in your final analysis.

4. Clean up the data. Look at your graphs. There will be some anomalous ‘outliers,’ or points which lie a long way from the others. If they are making your drug look bad, just delete them. But if they are helping your drug look good, even if they seem to be spurious results, leave them in.

5. The best of five… no… seven… no… nine! If the difference between your drug and placebo becomes significant four and a half months into a six-month trial, stop the trial immediately and start writing up the results: things might get less impressive if you carry on. Alternatively, if at six months the results are ‘nearly significant,’ extend the trial by another three months.

6. Torture the data. If your results are bad, ask the computer to go back and see if any particular subgroups behaved differently. You might find that your drug works very well in Chinese women aged fifty-two to sixty-one. ‘Torture the data and it will conhfess to anything’ as they say at Guantanamo Bay.

7. Try every button on the computer. If you’re really desperate, and analysing your data the way you planned doesn’t give you the results you wanted, just run the figures through a wide selection of other statistical tests, even if they are entirely inappropriate, at random.

8. Publish wisely. If you have a good trial, publish it in the biggest journal you can possibly manage. If you have a positive trial, but it was a completely unfair test, then put it in an obscure journal… and hope that readers are not attentive enough to read beyond the abstract to recognize its flaws.

9. Do not publish. If your finding is really embarrassing, hide it away somewhere and cite ‘data on file.’ Nobody will know the methods, and it will only be noticed if someone comes pestering you for the data to do a systematic review. Hopefully that won’t be for ages.

Bad Science: How To Mislead, Misinform, and Make Mistakes in Medicine

December 2nd, 2008 by Dr. Val Jones in Book Reviews
2 Comments »

I just finished reading Dr. Ben Goldacre’s new book, Bad Science. It received a very favorable review by the British Medical Journal, and so I thought I’d take a look for myself. After all, I am passionate about patient empowerment and worry sincerely for their safety as healthcare is becoming more and more of a “do-it-yourself” proposition.

Ben is a talented writer – his style is straightforward, accessible, and witty. The premise of the book is to expose the underbelly of science – how it’s miscommunicated to the public (via media, PR, and representatives from the snake oil community) and how research is often poorly designed (by uneducated scientists and government agencies, for-profit pharmaceutical companies, and biased physicians).

The case studies presented in Bad Science are especially poignant. Ben has selected a few shining examples of self-promoting figures who have risen to the highest rank of “expert” in the eyes of the media – all the while referring to themselves as “doctor” and yet only having a Ph.D. from an online correspondence school. Their legal bully tactics, fabrication of data to support their proprietary health gimmick, and extreme narcicism – are excellent studies in poor character triumphing over common sense. It is painful to see how successful snake oil salesmen can be, even in these modern and “enlightened” times.

Bad Science carefully dismantles the pseudoscience that underlies many of the claims of alternative medicine. He clearly demonstrates how research can be manipulated to demonstrate a positive effect for any therapeutic intervention, and explains why cosmetic and nutrition research are particularly rife with false positive results.

Ben also explores the role of the human psyche in misunderstanding science. Our deep desire to find a 1:1 correlation between every cause and effect is difficult to overcome. We want 1) to bring artificial simplicity out of complexity, 2) a quick fix in a pill form, 3) to believe in “breakthrough therapies,”4) to read sensational or scintillating news headlines. Unfortunately, science is often coopted to pander to these wants, rather than illuminate the truth.

Finally, Bad Science explores the many ways that statistics can be manipulated to support any claim. In fact, human intuition about math in general is quite flawed, which works against us as we try to understand the data collected by researchers.

I finished the book feeling enlightened but somewhat despairing – yearning to read a sequel, “Good Science” if only to restore my hope in the idea that wise people will have the courage to seek truth over sensationalism, and value objectivity over subjectivity for the greater good of all.

What does Ben Goldacre think we can do to combat the tidal wave of bad science on the Internet? He suggests that people of sound mind blog about the subject as frequently as possible, so that those who are searching for a voice of reason may find one. I blog here and at sciencebasedmedicine.org for that very purpose.

In my next post, I’ll summarize some tips from Bad Science that will help you to recognize when a health message is likely to be inaccurate.