A study published last month* by a group from Norway reports a systematic review of ME/CFS case definitions. It’s important because the Institute of Medicine panel is likely conducting a similar process as part of its work, but the study is also significant for the startling and overreaching conclusions drawn by the authors.
Quick Summary:
- This study collects and examines case definitions for ME/CFS, and extracts prevalence data. This will be a helpful future reference.
- The authors then step far beyond their own evidence and draw a variety of overreaching conclusions.
- Future analysis must be wary of cognitive bias in order to avoid the same mistakes.
What They Did
Brurberg et al began by searching the literature for case definitions of ME/CFS (a term encompassing CFS, ME, and combinations of the two). The group then looked for validation studies, meaning studies that applied one or more case definitions to patient populations. Of the twenty case definitions identified in the first step, seven have never been validated in a patient population. Thirty-eight validation studies were identified that test one or more case definition, but the authors were very critical of the quality of those studies saying, “We did not identify any study which rigorously assessed the reproducibility or feasibility of the different case definitions.”
The authors categorized the validation studies, and the summaries and Tables 1-4 are actually quite helpful. Five studies independently applied several case definitions to a single patient population, and thirteen studies took patients diagnosed under one definition (usually Fukuda) and sequentially applied case definitions assumed to be more specific. The authors caution that this second group of studies “should be interpreted with great caution” because there were so many differences in diagnostic method and definition application, including different thresholds for exclusion of psychiatric comorbidity. Finally, twenty-one studies presented prevalence estimates and the authors extracted data on prevalence rates for Fukuda, Oxford, and the Australia 1990 case definitions.
Personally, I find the collection of these studies and extraction of data to be both interesting and important for future reference. I think it is very likely that the IOM panel is conducting a similar review of the case definition literature, as well. Rigorous examination of case definitions and their applications is essential for the IOM and for the ME/CFS field in general. Brurberg et al found this body of evidence to be “methodologically weak and heterogeneous, making it questionable to compare the case definitions.” What I found both startling and disturbing is that the authors then go on to draw a number of conclusions that are simply not supported by the evidence they examined.
Insupportable Conclusions
First, the authors state, “We found no empirical evidence supporting the hypothesis that some case definitions more specifically identify patients with a neuroimmunological condition.” But they didn’t look! The validation studies they examined applied case definitions, but frequently by questionnaire or retrospective analysis. The studies examined by the authors did not take a group of patients, separate them by case definition, and then look for signs of a neuroimmunological condition. There are studies that do conduct such experiments – testing for immune markers, gene expression response to exercise, etc. – but the authors did not look at those studies because their systematic review was not designed to capture them. It’s overreaching to conclude that something doesn’t exist when you didn’t actually look for the evidence to begin with.
Second, the authors argue that for clinical purposes, a broad definition can be helpful because it identifies more patients who may benefit from treatment. They point to the PACE trial as showing the effectiveness of CBT and GET “irrespective of the case definition which had been used.” But they fail to mention some of the PACE follow-up papers, including this one in which the PACE authors admit that the application of the CDC definition in assessing recovery “may have been inaccurate because we only examined for accompanying symptoms in the previous week, not the previous 6 months” as required by the CDC criteria. This is just one example of the many documented methodological problems of the PACE trial, all of which are ignored by Brurberg et al. A glaring example of this is the statement that “existing evidence indicates that side effects of cognitive behavioural treatments or graded exercise therapy are negligible.” This conclusion ignores analysis by Tom Kindlon of the problems with reporting harms in studies like PACE, and published data from trials that did not find treatment benefit for CBT/GET.
Third, the authors argue that the organic vs. psychic disorder debate should be abandoned because most medical disorders have a complex aetiology, and psychological treatments can have benefit in “clear-cut somatic disorders.” But they then point the finger at patients: “Unfortunately, patient groups and researchers with vested interests in the belief that ME is a distinct somatic disease seem unwilling to leave the position that ME is an organic disease only. This position has damaged the research and practice for patients suffering from CFS/ME.” This statement is patently outrageous simply on the basis of logic. They present no evidence to support their accusation that the organic disease-only position has damaged research and clinical practice. Furthermore, they completely ignore the very real and logical possibility that the reverse is true! In other words, it is equally possible that the the people with vested interests in the belief that ME/CFS has psychosocial causes are unwilling to leave that position, and have damaged the research and practice for patients suffering from the disease. I can present ample evidence that this is the case, but the authors seem to dismiss the hypothesis outright.
Fourth, the authors conclude that, “Development of further case definitions of CFS/ME should be given low priority.” Instead, “Priority should be given to further development and testing of promising treatment options for patients with CFS/ME.” How is it possible for effective treatments to be developed if there is not an accurate case definition to identify patients for the studies? The authors admit that classifying severity and symptom patterns to identify potential predictors of treatment benefit “might be useful.” I would say that such work is essential. Furthermore, given the strong methodological criticisms of validation studies raised by these authors, their analysis seems to support a strong argument for further work on validating and refining case definitions, rather than throwing up one’s hands and working with what we’ve got.
Conclusion
Overall, the authors’ collection and review of the ME/CFS case definitions and validation studies will be very useful as a future reference. But they leap far ahead of what their review data suggests, and draw multiple conclusions with little or no supporting evidence. By failing to correct for their own cognitive biases, the authors allow assumptions to contaminate their conclusions.
The IOM panel, AHRQ systematic review, P2P Panel, and other researchers must be mindful of this in conducting their own analyses and drawing conclusions. Otherwise, we are at great risk for the cognitive bias and overreaching of Bruberg et al to become embedded in the literature and accepted as true.
*Brurberg K, Fonhus MS, Larun L, Flottorp S, Malterud K. Case definitions for chronic fatigue syndrome/myalgic encephalomyelitis (CFS/ME): a systematic review. BMJ Open2014;4:e003973 doi:10.1136/bmjopen-2013-003973
Guest Post: Wind Up Clock
The final post in this stretch of guest authors comes from Claudia Goodell. Claudia is among the most proactive ME/CFS patients I know, trying to make a new life for herself with this disease while also participating in advocacy.
When I was in graduate school my professor of Auditory Neuroscience and Psychoacoustics lectured us about sound pressure. In teaching us the mathematical equation for sound traveling through the acoustic system, he made sure we understood that if one looked at only the first part of the equation it would appear that an acoustic signal actually gained energy as it passed through the middle ear. However, this increase only compensates for the loss of energy that eventually occurs when the sound enters the fluid filled inner ear. The net amount is actually a slight loss in energy, and if you see the entire equation this is clear. In order to make this point he taught us, “There ain’t no such thing as a free lunch” (TANSTAAFL), meaning that even if something seems like it is free, there is always a cost, no matter how indirect or hidden.
While I didn’t retain much of my hearing science knowledge, I remembered TANSTAAFL, and ME/CFS reminds me of this every single day. It’s as though I am an old fashioned wind up clock ticking along and then running down. As I run out of energy my tick tock sound gets slower and slower. I sit on the table for various intervals, until someone randomly walks by, sees me and decides to rewind my mechanism. I may be mid-way between fully wound and fully spent; sometimes they rewind me all the way, and other times just a few rotations. I never know how much energy I really have. I just keep tick-tock-ing at whatever level I am capable given the amount of energy at any one time.
I worry. I worry that if I stop ticking I’ll suffer a slow, progression of this awful disease that forces me to stop moving. It’s not because I want to stop moving, or because I’d rather sit around than be active. Nothing could be farther from true. But every time I feel well enough to move, and I get out there and do the things I love, at a much reduced level than before the disease I am left feeling a relapse of symptoms for days, weeks or months. This is not motivational, but fortunately I was an athlete before becoming sick, and I am a determined person.
I do all the good things I can to stay in control of my symptoms as best I can. I avoid foods and drinks that my body doesn’t tolerate, and I take only the few medicines and supplements I really need. I insure ample good quality sleep, drink plenty of water, get regular massage, meditate, walk, do yoga, advocate, and I paint. Although this practice gets me close to maintaining some sort of balance between staying somewhat active and being too sick to move, unfortunately none of this is enough to create what could even loosely resemble a full life. I am unable to work, unable to travel without relapsing, unable to participate in sports at a level I would like, and socializing is minimized. So, to quote a famous movie, “I’m not dead yet”, but I’m not really living either. I’m occupying no man’s land with the rest of my fellow patients, and none of us wants to be here.