December 4th, 2008 by Dr. Val Jones in Quackery Exposed
No Comments »
This post is a follow up to my book review of Bad Science, located here.
***
I couldn’t help but feel unusually depressed by Dr. Ben Goldacre’s exposé of researchers who resort to trickery to get their articles published in peer-reviewed journals. There are a number of ways to manipulate data and many ways that flawed research is presented to enhance its chances of publication.
Before we get started, I need to point out that “negative trials” – or research results that don’t corroborate the investigator’s original hypothesis/es – are much less likely to be published. People (and/or publishers) are far more interested in finding a needle in a haystack, than hearing that no needle could be found. This is a driving force in all manner of mathematical convolutions aimed at demonstrating something interesting and to warrant publication. After all, who can blame the researchers for wanting to get their research published, and to have it make a meaningful contribution to their field of study? Who wants to toil for months to years on end, only to discover that their hypotheses were not born out by experimentation, and in fact no helpful conclusions may be drawn whatsoever?
And so, with this intense pressure to find something meaningful in one’s research (either for profit or personal satisfaction and professional advancement) – there are some typical stragegies that researchers use to make something out of nothing. Ben Goldacre reviews these strategies in the voice of an unscrupulous senior pharmaceutical investigator, giving advice to his junior colleague. (Parenthetically, it reminded me of The Screwtape Letters – an amusing book written by C.S. Lewis, featuring the imaginary advice of a senior demon to his junior counterpart as they tempt humans to evil.)
(This passage is taken directly from pages 192-193 of Bad Science)
1. Ignore the protocol entirely. Always assume that any correlation proves causation. Throw all your data into a spreadsheet programme and report – as significant – any relationship between anything and everything if it helps your case. If you measure enough, some things are bound to be positive by sheer luck.
2. Play with the baseline. Sometimes, when you start a trial, quite by chance the treatment group is already doing better than the placebo group. If so, then leave it like that. If, on the other hand, the placebo group is already doing better than the treatment group at the start, then adjust for the baseline in your analysis.
3. Ignore dropouts. People who drop out of trials are statistically much more likely to have done badly, and much more likely to have had side effects. They will only make your drug look bad. So ignore them, make no attempt to chase them up, do not include them in your final analysis.
4. Clean up the data. Look at your graphs. There will be some anomalous ‘outliers,’ or points which lie a long way from the others. If they are making your drug look bad, just delete them. But if they are helping your drug look good, even if they seem to be spurious results, leave them in.
5. The best of five… no… seven… no… nine! If the difference between your drug and placebo becomes significant four and a half months into a six-month trial, stop the trial immediately and start writing up the results: things might get less impressive if you carry on. Alternatively, if at six months the results are ‘nearly significant,’ extend the trial by another three months.
6. Torture the data. If your results are bad, ask the computer to go back and see if any particular subgroups behaved differently. You might find that your drug works very well in Chinese women aged fifty-two to sixty-one. ‘Torture the data and it will conhfess to anything’ as they say at Guantanamo Bay.
7. Try every button on the computer. If you’re really desperate, and analysing your data the way you planned doesn’t give you the results you wanted, just run the figures through a wide selection of other statistical tests, even if they are entirely inappropriate, at random.
8. Publish wisely. If you have a good trial, publish it in the biggest journal you can possibly manage. If you have a positive trial, but it was a completely unfair test, then put it in an obscure journal… and hope that readers are not attentive enough to read beyond the abstract to recognize its flaws.
9. Do not publish. If your finding is really embarrassing, hide it away somewhere and cite ‘data on file.’ Nobody will know the methods, and it will only be noticed if someone comes pestering you for the data to do a systematic review. Hopefully that won’t be for ages.
December 2nd, 2008 by Dr. Val Jones in Book Reviews
2 Comments »
I just finished reading Dr. Ben Goldacre’s new book, Bad Science. It received a very favorable review by the British Medical Journal, and so I thought I’d take a look for myself. After all, I am passionate about patient empowerment and worry sincerely for their safety as healthcare is becoming more and more of a “do-it-yourself” proposition.
Ben is a talented writer – his style is straightforward, accessible, and witty. The premise of the book is to expose the underbelly of science – how it’s miscommunicated to the public (via media, PR, and representatives from the snake oil community) and how research is often poorly designed (by uneducated scientists and government agencies, for-profit pharmaceutical companies, and biased physicians).
The case studies presented in Bad Science are especially poignant. Ben has selected a few shining examples of self-promoting figures who have risen to the highest rank of “expert” in the eyes of the media – all the while referring to themselves as “doctor” and yet only having a Ph.D. from an online correspondence school. Their legal bully tactics, fabrication of data to support their proprietary health gimmick, and extreme narcicism – are excellent studies in poor character triumphing over common sense. It is painful to see how successful snake oil salesmen can be, even in these modern and “enlightened” times.
Bad Science carefully dismantles the pseudoscience that underlies many of the claims of alternative medicine. He clearly demonstrates how research can be manipulated to demonstrate a positive effect for any therapeutic intervention, and explains why cosmetic and nutrition research are particularly rife with false positive results.
Ben also explores the role of the human psyche in misunderstanding science. Our deep desire to find a 1:1 correlation between every cause and effect is difficult to overcome. We want 1) to bring artificial simplicity out of complexity, 2) a quick fix in a pill form, 3) to believe in “breakthrough therapies,”4) to read sensational or scintillating news headlines. Unfortunately, science is often coopted to pander to these wants, rather than illuminate the truth.
Finally, Bad Science explores the many ways that statistics can be manipulated to support any claim. In fact, human intuition about math in general is quite flawed, which works against us as we try to understand the data collected by researchers.
I finished the book feeling enlightened but somewhat despairing – yearning to read a sequel, “Good Science” if only to restore my hope in the idea that wise people will have the courage to seek truth over sensationalism, and value objectivity over subjectivity for the greater good of all.
What does Ben Goldacre think we can do to combat the tidal wave of bad science on the Internet? He suggests that people of sound mind blog about the subject as frequently as possible, so that those who are searching for a voice of reason may find one. I blog here and at sciencebasedmedicine.org for that very purpose.
In my next post, I’ll summarize some tips from Bad Science that will help you to recognize when a health message is likely to be inaccurate.
December 1st, 2008 by Dr. Val Jones in Opinion
2 Comments »
I’d like to point out an error I made during a more optimistic time in my online career. Last year on my blog I suggested that physician ratings were “here to stay” so physicians should “embrace the inevitable.” What I hadn’t thought through at the time was the fact that virtually no one would use the ratings tools. I had made a fairly narcissistic assumption: that everyone cared so much about their healthcare experience that they were dying to describe it online.
The truth is that any online tool, portal, social network, or health 2.0 application must deliver a compelling “value proposition” to the user, especially if participation requires any degree of effort. It is human nature to take part in activities that reward us for our time. For example, we may slave over a hot stove because we stand to gain a delicious, satisfying meal in the end; we continue to work at jobs that we dislike because the paycheck makes it worthwhile. But why would a patient fill out a lengthy survey about his or her doctor when there’s no obvious value to them in doing so?
A recent article in Slate (h/t to the ACP Internist) makes a compelling case for why physician-rating sites have such low participation rates as to be fairly useless. The return on investment (time spent filling out a long questionnaire) is extremely low, and is worthwhile to only the most irritated patients. And of course, there is no policing of contributions – physicians can rate themselves into the highest quality rankings by logging in as fictional patients.
So does this mean that there are no worthwhile physician rating tools online? The Slate author would have you believe that there are none. However, I would suggest that Castle Connolly’s America’s Top Doctors list is a reliable, if somewhat limited source. Why? Because teams of staff (who are paid by Castle Connolly) do the heavy-lifting, requiring no effort from patients or online raters. Castle Connolly reviewers first request nominations for physician excellence from within a given specialty and region. Peers nominate others for the honor and then the Castle Connolly staff seek corroborative data from surveys sent to physician peers, hospital administrators, and support staff to ensure that nominated physicians are indeed highly esteemed by many of those with whom they work. In the end, about 10% of physicians are fully vetted and included in the list – and I’d say that the selection process is quite sensitive but not specific. In other words, a physician listed in America’s Top Doctors is likely to be excellent, but many excellent physicians are not captured by the methods.
I spoke to John Connolly in a recent interview about how to find a good physician and you may listen to the podcast here. Locating a good doctor is not too difficult – but finding one that will take your insurance (or still has some slots available for new patients) is another story.
November 30th, 2008 by Dr. Val Jones in True Stories
4 Comments »
The successful “Engage with Grace” campaign resulted in ~95 bloggers promoting end-of-life care discussions with family members over Thanksgiving. Paul Levy called it the first “medical blog rally on the Internet.” I wonder how many readers took the challenge?
I spent Thanksgiving with my sister (mom of 3) and brother-in-law in Michigan. After our dinner (with the kids tucked in for the night) we enjoyed a glass of wine and a game of Cranium (if you haven’t tried this game yet, you might want to pick one up in time for the next group of holidays – it’s like Pictionary, Charades, Trivial Pursuit, and American Idol wrapped into one). I looked for an opportunity to “pop the question” on end-of-life issues.
During a brief lull between rock song humming I casually inquired about whether or not my sister and brother-in-law had a living will. They said they hadn’t thought of it but agreed that it would be important to have one. We discussed various scenarios related to organ donation, end-of-life care, and cremation vs. burial preferences. Things degenerated a bit as I asked what their individual preferences might be for resuscitation under special circumstances (it was almost like a scene from Monty Python – “So, if you had no arms and no legs and you had a 10% chance of normal brain function recovery, would you like to be tube fed? How about if you had one leg and half an arm and a 5% chance of mental recovery? What about if you had 1/2 a liver, no spleen, and only one eye worked, but you COULD do math questions?”) We all had a good laugh at the black humor, but recognized that something important underlay the jesting. There is no doubt that we each had a 100% chance of dying at some point during our lives.
And then something unexpected happened – my brother-in-law looked me in the eye and said, “If anything happened to us, we’d really love it if you took our children and raised them as your own.”
I was very touched and happily agreed to do so. I replied with a wagging finger, “Now this doesn’t mean that you should take up helmet-free motorcycling…”
We all had a good chuckle and returned to Cranium, each feeling a little richer for the experience – we knew how we’d like to be cared for in case of critical illness, and I’d become the proud new godmother of 3.
November 29th, 2008 by Dr. Val Jones in True Stories
4 Comments »
After my recent interview with Dr. David McCarron, I began to think of ways to increase my dairy intake. It occurred to me that I hadn’t had a glass of milk since I was a kid – sure I’d put milk on my breakfast cereal or add it to recipes, but I just never thought of it as a beverage for some reason. I bet many of you feel similarly.
So I went to the store to get some whole milk, and I was almost overwhelmed by the options. There were many different brands (from different dairies) as well as organic options. I wondered if there was a taste difference between them. Which might be the most delicious?
I purchased three different types of whole milk: one from a local dairy, one that was organic, and a generic store brand. I brought them home and asked my husband to participate in a blind taste test. I put samples of each milk in a white, paper cup and asked him which one tasted best.
After several rounds of sipping, my husband confessed that he couldn’t tell a difference between them.
Then an unexpected thing happened. My cat wanted in on the action and jumped up on the table to participate. She had a very clear preference – she chose cup #3, the generic, non-organic store brand. Way to go, kitty – no fancy milk for you!
What’s the lesson of this totally subjective, unscientific study of n=3? Milk is good for you, it’s the best nutritional value for the cost (25 cents/cup – compare that to soda), and there’s no real taste difference between brands. Milk is a legitimate beverage – I’m going to have some more regularly… if I can wrestle it away from my cat.
