Systematic Overreaching

A study published last month* by a group from Norway reports a systematic review of ME/CFS case definitions. It’s important because the Institute of Medicine panel is likely conducting a similar process as part of its work, but the study is also significant for the startling and overreaching conclusions drawn by the authors.

Quick Summary:

• This study collects and examines case definitions for ME/CFS, and extracts prevalence data. This will be a helpful future reference.
• The authors then step far beyond their own evidence and draw a variety of overreaching conclusions.

• Future analysis must be wary of cognitive bias in order to avoid the same mistakes.

What They Did

Brurberg et al began by searching the literature for case definitions of ME/CFS (a term encompassing CFS, ME, and combinations of the two). The group then looked for validation studies, meaning studies that applied one or more case definitions to patient populations. Of the twenty case definitions identified in the first step, seven have never been validated in a patient population. Thirty-eight validation studies were identified that test one or more case definition, but the authors were very critical of the quality of those studies saying, “We did not identify any study which rigorously assessed the reproducibility or feasibility of the different case definitions.”

The authors categorized the validation studies, and the summaries and Tables 1-4 are actually quite helpful. Five studies independently applied several case definitions to a single patient population, and thirteen studies took patients diagnosed under one definition (usually Fukuda) and sequentially applied case definitions assumed to be more specific. The authors caution that this second group of studies “should be interpreted with great caution” because there were so many differences in diagnostic method and definition application, including different thresholds for exclusion of psychiatric comorbidity. Finally, twenty-one studies presented prevalence estimates and the authors extracted data on prevalence rates for Fukuda, Oxford, and the Australia 1990 case definitions.

Personally, I find the collection of these studies and extraction of data to be both interesting and important for future reference. I think it is very likely that the IOM panel is conducting a similar review of the case definition literature, as well. Rigorous examination of case definitions and their applications is essential for the IOM and for the ME/CFS field in general. Brurberg et al found this body of evidence to be “methodologically weak and heterogeneous, making it questionable to compare the case definitions.” What I found both startling and disturbing is that the authors then go on to draw a number of conclusions that are simply not supported by the evidence they examined.

Insupportable Conclusions

First, the authors state, “We found no empirical evidence supporting the hypothesis that some case definitions more specifically identify patients with a neuroimmunological condition.” But they didn’t look! The validation studies they examined applied case definitions, but frequently by questionnaire or retrospective analysis. The studies examined by the authors did not take a group of patients, separate them by case definition, and then look for signs of a neuroimmunological condition. There are studies that do conduct such experiments – testing for immune markers, gene expression response to exercise, etc. – but the authors did not look at those studies because their systematic review was not designed to capture them. It’s overreaching to conclude that something doesn’t exist when you didn’t actually look for the evidence to begin with.

Second, the authors argue that for clinical purposes, a broad definition can be helpful because it identifies more patients who may benefit from treatment. They point to the PACE trial as showing the effectiveness of CBT and GET “irrespective of the case definition which had been used.” But they fail to mention some of the PACE follow-up papers, including this one in which the PACE authors admit that the application of the CDC definition in assessing recovery “may have been inaccurate because we only examined for accompanying symptoms in the previous week, not the previous 6 months” as required by the CDC criteria. This is just one example of the many documented methodological problems of the PACE trial, all of which are ignored by Brurberg et al. A glaring example of this is the statement that “existing evidence indicates that side effects of cognitive behavioural treatments or graded exercise therapy are negligible.” This conclusion ignores analysis by Tom Kindlon of the problems with reporting harms in studies like PACE, and published data from trials that did not find treatment benefit for CBT/GET.

Third, the authors argue that the organic vs. psychic disorder debate should be abandoned because most medical disorders have a complex aetiology, and psychological treatments can have benefit in “clear-cut somatic disorders.” But they then point the finger at patients: “Unfortunately, patient groups and researchers with vested interests in the belief that ME is a distinct somatic disease seem unwilling to leave the position that ME is an organic disease only. This position has damaged the research and practice for patients suffering from CFS/ME.” This statement is patently outrageous simply on the basis of logic. They present no evidence to support their accusation that the organic disease-only position has damaged research and clinical practice. Furthermore, they completely ignore the very real and logical possibility that the reverse is true! In other words, it is equally possible that the the people with vested interests in the belief that ME/CFS has psychosocial causes are unwilling to leave that position, and have damaged the research and practice for patients suffering from the disease. I can present ample evidence that this is the case, but the authors seem to dismiss the hypothesis outright.

Fourth, the authors conclude that, “Development of further case definitions of CFS/ME should be given low priority.” Instead, “Priority should be given to further development and testing of promising treatment options for patients with CFS/ME.” How is it possible for effective treatments to be developed if there is not an accurate case definition to identify patients for the studies? The authors admit that classifying severity and symptom patterns to identify potential predictors of treatment benefit “might be useful.” I would say that such work is essential. Furthermore, given the strong methodological criticisms of validation studies raised by these authors, their analysis seems to support a strong argument for further work on validating and refining case definitions, rather than throwing up one’s hands and working with what we’ve got.

Conclusion

Overall, the authors’ collection and review of the ME/CFS case definitions and validation studies will be very useful as a future reference. But they leap far ahead of what their review data suggests, and draw multiple conclusions with little or no supporting evidence. By failing to correct for their own cognitive biases, the authors allow assumptions to contaminate their conclusions.

The IOM panel, AHRQ systematic review, P2P Panel, and other researchers must be mindful of this in conducting their own analyses and drawing conclusions. Otherwise, we are at great risk for the cognitive bias and overreaching of Bruberg et al to become embedded in the literature and accepted as true.

*Brurberg K, Fonhus MS, Larun L, Flottorp S, Malterud K. Case definitions for chronic fatigue syndrome/myalgic encephalomyelitis (CFS/ME): a systematic review. BMJ Open2014;4:e003973 doi:10.1136/bmjopen-2013-003973