December 20th, 2007 by Dr. Val Jones in Book Reviews
No Comments »
Which news source do you trust more: The New York Times or The National Enquirer? Which news reporter would you trust more: Charlie Gibson or Jerry Springer? As it turns out, medical journals and science researchers run the gamut from highly credible and respected to dishonest and untrustworthy. So as we continue down this road of learning how to evaluate health news, let’s now turn our attention to pillar number one of trustworthy science: credibility.
In medical research, I like to think of credibility in three categories:
1. The credibility of the researcher: does the researcher have a track record of excellence in research methodology? Is he or she well-trained and/or have access to mentors who can shepherd along the research project and avoid the pitfalls of false positive “artifacts?” Has the researcher published previously in highly respected, peer reviewed journals?
2. The credibility of the research: does the study design reflect a clear understanding of potential result confounders and does it control for false positive influences, especially the placebo effect?
3. The credibility of the journal that publishes the research: top tier journals have demonstrated a track record of careful peer review. They have editorial boards of experts who are trained in research methodology and are screened for potential conflicts of interest that could inhibit an objective analysis of the research that they review. The importance of careful peer review must not be underestimated. Some say that the quality of a product is only as good as its quality control system. Top tier journals have the best quality control systems, and the articles they publish must undergo very careful scrutiny before they are published.
So as a lay person, how do you evaluate the credibility of a health news report? In practical terms, here’s what I’d recommend:
1. Look at the name of the journal reference – where was the research published? Is it from a top tier journal? R. Barker Bausell considers the following journals to be “top tier:” The New England Journal of Medicine (NEJM), The Journal of the American Medical Association (JAMA), Annals of Internal Medicine, Nature, and Science. I might cast a slightly larger net, but no one will argue that these are certainly some of the most respected journals in medicine and science.
2. Look at the study design described in the research article abstract. Was it a randomized, controlled, double-blind, placebo-controlled trial? Were there more than 50 subjects in each group? Did the authors overstate their conclusions? This sort of analysis is challenging to the lay person – so do it if you can, but if it proves too difficult, fall back on credibility check #1.
3. Look at the primary author of the research. Search for his/her name in the National Library of Medicine’s Medline database and see what other research he or she has done, and where it was published.
If the news report is based on credible research, you may feel confident in taking the results more seriously (so long as the media is representing them accurately). But before you hang your hat on a journal’s reputation, let’s take a look at the other 2 pillars of trustworthy science: plausibility and reproducibility. These two will help you navigate your way through the vast gray zone, where the credibility check doesn’t pass with flying colors – or maybe you’re dealing with neither Charlie Gibson nor Jerry Springer.This post originally appeared on Dr. Val’s blog at RevolutionHealth.com.
December 19th, 2007 by Dr. Val Jones in Book Reviews
No Comments »
Have you ever been surprised and confused by what seem to be conflicting results from scientific research? Have you ever secretly wondered if the medical profession is comprised of neurotic individuals who change their mind more frequently than you change your clothes? Well, I can understand why you’d feel that way because the public is constantly barraged with mixed health messages. But why is this happening?
The answer is complex, and I’d like to take a closer look at a few of the reasons in a series of blog posts. First, the human body is so incredibly complicated that we are constantly learning new things about it – how medicines, foods, and the environment impact it from the chemical to cellular to organ system level. There will always be new information, some of which may contradict previous thinking, and some that furthers it or ads a new facet to what we have already learned. Because human behavior is also so intricate, it’s far more difficult to prove a clear cause and effect relationship with certain treatments and interventions, due to the power of the human mind to perceive benefit when there is none (placebo effect).
Second, the media, by its very nature, seeks to present data with less ambiguity than is warranted. R. Barker Bausell, PhD, explains this tendency:
1. Superficiality is easier to present than depth.
2. The media cannot deal with ambiguity, subtlety, and diversity (which always characterizes scientific endeavors involving new areas of investigation or human behavior in general)
3. The bizarre always gets more attention than the usual.
I really don’t blame the media – they’re under intense pressure to find interesting sound bites to keep peoples’ attention. It’s not their job to present a careful and detailed analysis of the health news that they report. So it’s no wonder that a research paper suggesting that a certain herb may influence cancer cell protein expression in a Petri dish becomes: herb is new cure for cancer! Of course, many media outlets are more responsible in their reporting than that, but you get the picture.
And thirdly, the scientific method (if not carefully followed in rigorous, randomized, placebo-controlled trials) is a set up for false positive tests. What does that mean? It means that the default for your average research study (before it even begins) is that there will be a positive association between intervention and outcome. So I could do a trial on, say, the potential therapeutic use of candy bars for the treatment of eczema, and it’s likely (if I’m not a careful scientist) that the outcome will show a positive correlation between the two.
There are many reasons for false positive results (e.g. wrongly ascribing effectiveness to a given therapy) in scientific research. “Experimental artifacts” as they’re called, are very common and must be accounted for in a study’s design. For fun let’s think about how the following factors stack the deck in favor of positive research findings (regardless of the treatment being analyzed):
1. Natural History: most medical conditions have fluctuating symptoms and many improve on their own over time. Therefore, for many conditions, one would expect improvement during the course of study, regardless of treatment.
2. Regression to the Mean: people are more likely to join a research study when their illness/problem is at its worst during its natural history. Therefore, it is more likely that the symptoms will improve during the study than if they joined at times when symptoms were not as troublesome. Therefore, in any given study – there is a tendency for participants in particular to improve after joining.
3. The Hawthorne Effect: people behave differently and experience treatment differently when they’re being studied. So for example, if people know they’re being observed regarding their work productivity, they’re likely to work harder during the research study. The enhanced results therefore, do not reflect typical behavior.
4. Limitations of Memory: studies have shown that people ascribe greater improvement of symptoms in retrospect. Research that relies on patient recall is in danger of increased false positive rates.
5. Experimenter Bias: it is difficult for researchers to treat all study subjects in an identical manner if they know which patient is receiving an experimental treatment versus a placebo. Their gestures and the way that they question the subjects may set up expectations of benefit. Also, scientists are eager to demonstrate positive results for publication purposes.
6. Experimental Attrition: people generally join research studies because they expect that they may benefit from the treatment they receive. If they suspect that they are in the placebo group, they are more likely to drop out of the study. This can influence the study results so that the sicker patients who are not finding benefit with the placebo drop out, leaving the milder cases to try to tease out their response to the intervention.
7. The Placebo Effect: I saved the most important artifact for last. The natural tendency for study subjects is to perceive that a treatment is effective. Previous research has shown that about 33% of study subjects will report that the placebo has a positive therapeutic effect of some sort.
So my dear readers – if the media wants to get your attention with exaggerated representations of research findings, and the research findings themselves are stacked in favor of reporting an effect that isn’t real… then how on earth are we to know what to make of health news? Luckily, R. Barker Bausell has explained all of this really well in his book and I will attempt to summarize the following principles in the next few posts:
1. The importance of credible scientific evidence
2. The importance of plausible scientific evidence
3. The importance of reproducible scientific evidenceThis post originally appeared on Dr. Val’s blog at RevolutionHealth.com.
November 18th, 2007 by Dr. Val Jones in Health Policy, Opinion
5 Comments »
A recent blog post at Terra Sigillata really disturbed me. The author describes how, in the face of increasing healthcare costs, Medicare now declines coverage of life saving medicines for lymphoma patients. This is one example of rationing healthcare that will become ever more common (as it is in other leading industrialized nations) as we move towards further cuts in government programs and funding. In Canada, expensive chemotherapies are not commonly covered by the national health plan, and in Britain, age is a determinant for transplant eligibility.
But what troubles me about the apparent capriciousness of denying coverage to certain types of cancer patients over others, is that government programs are – at the same time – allocating millions of dollars to researching implausible alternative medicine treatments while denying coverage of proven therapies to patients who will likely die without them.
Take homeopathy, for example. The National Center for Complementary and Alternative Medicine lists homeopathy as an eligible area of research, and boasts several ongoing studies in the area of stroke, dementia, fibromyalgia, and prostate cancer. And yet, there is no plausible mechanism of action to support its potential use as anything more than a placebo. Homeopathy operates on the assumption that water has memory, and that once it has been exposed to certain substances, such as arsenic, it obtains curative properties for illnesses that bear resemblance to poisoning from those very substances (though the water itself may no longer contain a single molecule of the substance).
Research into scientifically implausible theories should not be funded by our tax dollars at the expense of offering life saving treatments to cancer patients. It is time for scientists to stand up and point out that the Emperor has no clothes when it comes to homeopathy and other similarly flawed alternative medical treatments.
As we move towards rationing limited healthcare resources, we have a moral obligation to prioritize the money correctly. “Open-mindedness” is no excuse for poor stewardship.
Dr. Wallace Sampson sums this up in a provocative recent editorial. Here is an excerpt:
We now see accumulation of useless information in journals and information data bases — hundreds of clinical trials (RCTs) on implausible methods, such as homeopathy, unrefined plant products, prayer, and acupuncture. Initial plausibility retreats before two 20th-century development ideologies of relativism — a principle that all facts and opinions have equal or similar value, and postmodernism — that regards facts as social constructions.
Once thought to be too esoteric for relevance to medicine, these twin ideologies now mold the thinking of policy makers and granting agency officials. Ancient and traditional cultural practices are not diminished for lack of plausibility, but are investigated by RCTs because they are there.
Plausibility depends on prior reliable observations, physical and chemical laws, pharmacological principles, and advocates’ economic and legal misadventures. The National Center for Complementary and Alternative Medicine spends $100 million/year on implausible research and training grants. In performing RCTs on implausible proposals, clinical research has taken a wrong turn and departed from rationality.
This post originally appeared on Dr. Val’s blog at RevolutionHealth.com.
September 29th, 2007 by Dr. Val Jones in Quackery Exposed, True Stories
5 Comments »
As some of you faithful readers of Dr. Val may know – I’m a huge fan of massage. Consider it my
addiction, I suppose – it could be worse! And so it may come as no surprise that I had a massage every day of my vacation (7 days in Southern California – sorry I haven’t been blogging as much lately). Yes I’m well and truly tenderized from head to toe. But I have to say that some of the therapists’ chatter was quite amusing to me. I was unsuccessful at completely removing my doctor hat during the experience, and tried not to look quizzically at them as they explained what they were up to and what I needed.
Those of you with healthcare backgrounds may especially appreciate this dialog:
Therapist (scrutinizing my back as I’m face down on a
table): have you seen a chiropractor recently?
Dr. Val: Um, no. Why?
Therapist: Well, two of your ribs are out.
Dr. Val: They’re ‘out?’
Where did they go?
Therapist: A chiropractor can put them back for you so your
muscles won’t pull in the wrong direction.
Dr. Val: Will a chiropractor be able to fix this
permanently?
Therapist: No, you’ll have to keep going. (Adds some eucalyptus lotion). This will bring your red blood cells to the
surface, and the cooling brings white blood cells to the area.
Dr. Val: (considering what a collection of white blood cells
actually do – yuck). Hrmph. That’s a nice massage technique. What are you doing?
Therapist: I’m using my elbows to stimulate repair cells.
Dr. Val: Ahum…
Therapist: You have lactic acid build up in your shoulders
so we have to flush the toxins out with special oils. You should also drink a lot of water.
Dr. Val: What sort of toxins?
Therapist: Like, dirt and metals and stuff that you’ve been
exposed to. You might have eaten fruit
with pesticides on it. Do you eat
organic food?
Dr. Val: Sometimes.
Therapist: Oh, you should only eat organic food. Then you won’t have as many toxins built up.
Dr. Val: How do I know how many toxins I have in my body?
Therapist: Well, your shoulders are really tight and your
ribs are out so I think you probably have a lot. You’ll need a lot of massage and you need to
see a chiropractor. The oils I used on
you will have a calming effect, though.
You’ll probably sleep really well tonight.
Dr. Val: I see (inhaling, exhaling). I hope I do.
** 15 minutes post massage – back at the hotel room **
Husband: You smell funny – like an almond.
Dr. Val: That’s “the calming oil that flushed the toxins out
of my body” today. I have to drink
water.
Husband: Well we’re driving 2 hours up to L.A. so don’t drink too much or we’ll have to
stop along the way.
Dr. Val: The therapist said 2 of my ribs were out and that I
need to see a chiropractor.
Husband: There’s nothing wrong with your ribs. Don’t be silly. Why do you keep getting these massages?
Dr. Val: They feel good.
Husband: I could give you a back rub for free.
Dr. Val: It’s not the same, though.
Husband: Why, because I don’t tell you your ribs are out of
whack?
Dr. Val: Well, they have a proper table…
Husband: I don’t understand you.
Dr. Val: But you like almonds (hugs him).This post originally appeared on Dr. Val’s blog at RevolutionHealth.com.
August 27th, 2007 by Dr. Val Jones in Opinion, Quackery Exposed
5 Comments »
On the recommendation of several members of the blogosphere, I’ve been watching a new British TV Series called “The Enemies of Reason” hosted by Richard Dawkins, a noted atheist and evolutionary biologist at Oxford. The series offers a skeptical review of the claims of alternative medicine practitioners, strung together with Michael Moore-like skill, and designed to showcase the fringiest believers as they concoct wild, pseudoscientific explanations for the mechanism of action of their therapies. It is entertaining and whimsical – though Dawkins himself appears dogmatic and cold as he ultimately builds a case for science as a religion.
Although I agree with Dawkins on many points, I think his approach is somewhat superficial and unnecessarily adversarial. Instead of unmasking kookiness and labeling people as “enemies of reason,” I think it would be more interesting to ask: why are people seeking out pseudoscience? What is the deeper need that scientific medicine is not satisfying? Why are billions spent on alternative therapies? (Please note that the “alternative therapies” that Dawkins evaluates include things like quantum homeopathy, magnetic healing, angelic guidance, and other treatments that don’t have evidence-based underpinnings.)
I think that at the very heart of the matter is that we humans want to feel in control. For millennia we’ve been conjuring up bizarre theories in order to believe that we can influence our destinies and our health. Just take for example the elaborate Egyptian religious myths (health was controlled by one’s ka which required regular food and drink offerings – not to mention all the elaborate embalming rituals to influence a good afterlife). All of these rituals provided the Egyptians with a sense of control over their lives, deaths, and reincarnations. I’m not entirely sure that we humans today are much different in our desire to control our lives. We just manifest it in other ways.
Ironically, science feeds pseudoscience – the more we know, the more definitive we can be about a disease or its prognosis – and the greater the desire to buck against that. And so as we advance in medical understanding, it is not surprising that there is renewed interest in magic as a means of influencing our clinical course as the inalterable progression becomes clearer and clearer. Add to that the fact that the physician-patient relationship has been undermined by a series of unfortunate historical circumstances (the rise of health insurance middle men, decreasing reimbursements, administrative red tape, etc.) and you have a group of dissatisfied patients with chronic diseases that have predictable complications – all seeking alternative outcomes at the hands of any compassionate person who promises to give them some control back. Of course, our “quick fix” culture also gives rise to a preference for simple solutions, rather than complex (though effective) ones. Is there any wonder that snake oil has emerged as a major player in this climate?
Dawkins makes the convincing argument that certain alternative medicine practices rely entirely upon the placebo effect. If this is the case, practitioners of these therapies cannot admit that their remedies are placebos – in so doing they would undermine their potential effects. Therefore, one cannot expect a rational response from them when confronted with evidence that their strategies do not work or are implausible. For the remedies to have a perceived effect, they only need to be believed in by the recipient. The millions of dollars spent by the National Health Service and National Institutes of Health attempting to uncover the mechanism of action of implausible therapies (such as homeopathy) will not influence the millions of faithful believers who turn to such practices for their health. I suppose that once the placebo effect has been scientifically proven, only the skeptics will be convinced by the data.
In the end however, Dawkins’ “war” is not between the evidence based medicine camp and the placebo based medicine group, it’s really an internal battle that each of us faces about our own mortality. The process of coming to terms with health and disease is uniquely personal – some want to be (as Dawkins puts it) coddled, others want the cold hard facts. As for me, all I want is for patients to be able to make informed decisions, not to be misled about therapeutic safety or efficacy, and not to be guided away from known effective treatments and towards known ineffective treatments. I suspect that this is what most people want as well.
This post originally appeared on Dr. Val’s blog at RevolutionHealth.com.
