They Know What They’re Doing (Not)

This post comes via Mary Dimmock, with assistance from Claudia Goodell, Denise Lopez-Majano, and myself. You are welcome to publish it on your site with attribution to Mary Dimmock.

 

Last week, Jennie Spotila and Erica Verillo posted summaries of just some of the issues with AHRQ’s Draft Systematic Evidence Review, conducted for P2P.

Jennie and Erica highlighted serious and sometimes insurmountable flaws with this Review, including:

  • The failure to be clear and specific about what disease was being studied.
  • The acceptance of 8 disparate ME or CFS definitions as equivalent in spite of dramatic differences in inclusion and exclusion criteria.
  • The bad science reflected in citing Oxford’s flaws and then using Oxford studies anyway.
  • The well-known problems with the PACE trial.
  • The flawed process that used non-experts on such a controversial and conflicted area.
  • Flawed search methods that focused on fatigue.
  • Outright errors in some of the basic information in the report and apparent inconsistencies in how inclusion criteria were applied.
  • Poorly designed and imprecise review questions.
  • Misinterpretation of cited literature.

In this post, I will describe several additional key problems with the AHRQ Evidence Review.

Keep in mind that comments must be submitted by October 20, 2014. Directions for doing so are at the end of this post.

We Don’t Need No Stinking Diagnostic Gold Standard

Best practices for diagnostic method reviews state that a diagnostic gold standard is required as the benchmark. But there is no agreed upon diagnostic gold standard for this disease, and the Review acknowledges this. So what did the Evidence Review do? The Review allowed any of 8 disparate CFS or ME definitions to be used as the gold standard and then evaluated diagnostic methods against and across the 8 definitions. But when a definition does not accurately reflect the disease being studied, that definition cannot be used as the standard. And when the 8 disparate definitions do not describe the same disease, you cannot draw conclusions about diagnostic methods across them.

What makes this worse is that the reviewers recognized the importance of PEM but failed to consider the implications of Fukuda’s and Oxford’s failure to require it. The reviewers also excluded, ignored or downplayed substantial evidence demonstrating that some of these definitions could not be applied consistently, as CDC’s Dr. Reeves demonstrated about Fukuda.

Beyond this, some diagnostic studies were excluded because they did not use the “right” statistics or because the reviewer judged the studies to be “etiological” studies, not diagnostic methods studies. Was NK-Cell function eliminated because it was an etiological study? Was Dr. Snell’s study on the discriminative value of CPET excluded because it used the wrong statistics? And all studies before 1988 were excluded. These inclusion/exclusion choices shaped what evidence was considered and what conclusions were drawn.

Erica pointed out that the Review misinterpreted some of the papers expressing harms associated with a diagnosis. The Review failed to acknowledge the relief and value of finally getting a diagnosis, particularly from a supportive doctor. The harm is not from receiving the diagnostic label, but rather from the subsequent reactions of most healthcare providers. At the same time, the Review did not consider other harms like Dr. Newton’s study of patients with other diseases being diagnosed with “CFS” or another study finding some MS patients were first misdiagnosed with CFS. The Review also failed to acknowledge the harm that patients face if they are given harmful treatments out of a belief that CFS is really a psychological or behavioral problem.

The Review is rife with problems: Failing to ask whether all definitions represent the same disease. Using any definition as the diagnostic gold standard against which to assess any diagnostic method. Excluding some of the most important ME studies. It is no surprise, then, that the Review concluded that no definition had proven superior and that there are no accepted diagnostic methods.

But remarkably, reviewers felt that there was sufficient evidence to state that those patients who meet CCC and ME-ICC criteria were not a separate group but rather a subgroup with more severe symptoms and functional limitations. By starting with the assumption that all 8 definitions encompass the same disease, this characterization of CCC and ICC patients was a foregone conclusion.

But Don’t Worry, These Treatment Trials Look Fine

You would think that at this point in the process, someone would stand up and ask about the scientific validity of comparing treatments across these definitions. After all, the Review acknowledged that Oxford can include patients with other causes of the symptom of chronic fatigue. But no, the Evidence Review continued on to compare treatments across definitions regardless of the patient population selected. Would we ever evaluate treatments for cancer patients by first throwing in studies with fatigued patients? The assessment of treatments was flawed from the start.

But the problems were then compounded by how the Review was conducted. The Review focused on subjective measures like general function, quality of life and fatigue, not objective measures like physical performance or activity levels. In addition, the Review explicitly decided to focus on changes in the symptom of fatigue, not PEM, pain or any other symptom. Quality issues with individual studies were either not considered or ignored. Counseling and CBT studies were all lumped into one treatment group, without consideration of the dramatic difference in therapeutic intent of the two. Some important studies like Rituxan were not considered because the treatment duration was considered too short, regardless of whether it was therapeutically appropriate.

And finally, the Review never questioned whether the disease theories underlying these treatments were applicable across all definitions. Is it really reasonable to expect that a disease that responds to Rituxan or Ampligen is going to also respond to therapies that reverse the patient’s “false illness beliefs” and deconditioning? Of course not.

If their own conclusions on the diagnostic methods and the problems with the Oxford definition were not enough to make them stop, the vast differences in disease theories and therapeutic mechanism of action should have made the reviewers step back and raise red flags.

At the Root of It All

This Review brings into sharp relief the widespread confusion on the nature of ME and the inappropriateness of having non-experts attempt to unravel a controversial and conflicting evidence base about which they know nothing.

But just as importantly, this Review speaks volumes about the paltry funding and institutional neglect of ME reflected in the fact that the study could find only 28 diagnostic studies and 9 medication studies to consider from the last 26 years. This Review speaks volumes about the institutional mishandling that fostered the proliferation of disparate and sometimes overly broad definitions, all branded with the same “CFS” label. The Review speaks volumes about the institutional bias that resulted in the biggest, most expensive and greatest number of treatment trials being those that studied behavioral and psychological pathology for a disease long proven to be the result of organic pathology.

This institutional neglect, mishandling and bias have brought us to where we are today. That the Evidence Review failed to recognize and acknowledge those issues is stunning.

Shout Out Your Protest!

This Evidence Review is due to be published in final format before the P2P workshop and it will affect our lives for years to come. Make your concerns known now.

  1. Submit public comments on the Evidence Review to the AHRQ website by October 20.
  2. Contact HHS and Congressional leaders with your concerns about the Evidence Review, the P2P Workshop and HHS’ overall handling of this disease. Erica Verillo’s recent post provides ideas and links for how to do this.

The following information provides additional background to prepare your comments:

However you choose to protest, make your concerns known!

 

This entry was posted in Advocacy, Commentary and tagged , , , , , , , , , , , , , , , , , , , . Bookmark the permalink.

18 Responses to They Know What They’re Doing (Not)

  1. Gabby says:

    Excellent review of the ‘Review’! Thank you, Mary, Jennie, Claudia and Denise.
    You have highlighted everything that is wrong with this review and p2p process. It would be really laughable that, this is considered a ‘scientific’ study, if it wasn’t for the fact that the outcome is so serious and damaging for us.

    I would go one step further and claim that this is malfeasance from the HHS to have attempted and directed this review in the way that they have. I am convinced that this result is not a surprise to the government agencies that contracted this ‘Review’. I am tempted to say that this was their intent all along. As you have stated; this has been specifically directed in a way that a desired outcome has been orchestrated to follow.

    This is outright malfeasance. There is intent to hurt and harm patients. It is not merely a mistake but, an outright damaging aim by design.

    I hope that all stakeholders; patients, advocates, family and medical community take the time to protest this insidious process.

  2. Jerrold Spinhirne says:

    My thanks to the writers of this excellent summary of the many flaws in the the Draft Review. I was amazed by just how amateurish the Draft Review for the P2P Workshop was. If HHS is going to hire medical writers to endorse their existing policies, at least they could hire competent ones.

    This is a link to my comment on the many problems of the Draft Review looking at just two paragraphs of the Executive Summary. I submitted the comment using the AHRQ webpage linked above. https://drive.google.com/file/d/0B4uD-VyWmIw2bUt0LWlnMzl1Um8/view?usp=sharing

    • Jennie Spotila says:

      Thank you for linking to your excellent letter, Jerry.

      The struggle we’re having in writing our comments is to keep it short! Your analysis is excellent, in-depth, and only covers two paragraphs! We could probably write a 400 page critique of this 400 page review, if we had the time.

  3. Gabby says:

    I just wanted to add this quote:

    “There may be times when we are powerless to prevent injustice, but there must never be a time when we fail to protest.” – Elie Wiesel

    I know some people might think that there is no use protesting because it won’y make a difference. I would counter with the quote above. We need to use our voice to protest.

  4. Ess says:

    @Gabby
    “This is outright malfeasance. There is intent to hurt and harm patients. It is not merely a mistake but, an outright damaging aim by design.”

    YES, it is!! TPTB (The Powers That Be) know how sick we are with ME/CFS–and further, they full well know that this is a BIO disease!!!

    They are playing with fire–allowing a DISabling PLAGUE / pandemic to multiply–affecting male and female children, teens and young people–MEN and WoMEN. This is nothing short of criminal!

  5. Gina B says:

    Thank you for this excellent work on our behalf. I will be protesting as I have from the start. Where will our opposition to this rubberstamp HHS malfeasance be heard with the most unity. I wish I could write as eloquent as reviewers and comments posted above. Since I know my limitations regarding letter writing, which letter is best to endorse? I am aware of others that don’t think they have enough energy to help at this time. So I am asking for other patients as well! This is not the time to remain silent.

    You guys have been amazing from the start. First bringing it to our attention and keeping us informed. Is it asking to much to have a letter of advocacy for all to sign? There may be one already but I haven’t read all the above links yet. One of the hardest parts of living with this for 21 years personally, is the cognition decline that does progress. ME didn’t only steal my life, it stole my self esteem and confidence.

    Tremendous thanks to all that helped critique all the flaws listed.

    • Jennie Spotila says:

      I am not aware of any current petition efforts, Gina, but it has been discussed. At this time, I think it would be best to send a letter/email (even if it’s very short) to NIH – either to Dr. Francis Collins (Director) or to Dr. David Murray (Director, Office of Disease Prevention).

  6. Gina B says:

    Bring up flaws or continue opposing (my gut reaction) as entire P2P is ludicrous, just having non experts involved anywhere with such a complex illness? Using research proven (PACE) harmful, or how can they include definitions that don’t include patients (Oxford, etc.)?

  7. Sandra says:

    As others have expressed, I too, am most grateful for your advocacy work. Twice now I have submitted comments to the P2P on their Draft. I am wondering if anyone knows if a PACE critique exposing all of the flaw of the “study too big to fail” has been or will be submitted. I don’t know all the ins and outs of it and am too sick right now to delve into it. This is such a critical piece that needs to be exposed for what it is. I would be happy and relieved to know if that has been tackled. Thank you.

    • Jennie Spotila says:

      Sandra, thank you for taking the time to submit comments on the evidence review! As it happens, I am tackling PACE in my comments and I hope others will as well, because other people know that material far better than I do.

  8. Anne Ö says:

    Jennie, a question regarding comments to P2P: Is there an 8,000-character limit on all comments, or just on comments submitted via the online form? If one comments by uploading a document, is there a character limit for that?

    • Jennie Spotila says:

      I believe the character limit applies to each field on the comment page form, not document submissions.

  9. Anne Ö says:

    Thanks, Jennie.

    And the deadline is just before midnight on Monday, EST?

    And do we know for certain that a person can submit comments from anywhere in the world? I notice the boxes for the contact information are not set up to include other countries.

    • Jennie Spotila says:

      Yes, the deadline is just before midnight on Monday. I am not aware of any restrictions on country of origin for comments.

  10. Anne Ö says:

    Another question (sorry)…

    Is one supposed to get recieve some kind of e-mail confirming that one’s comment has been received after one submits it via the website?

    http://www.effectivehealthcare.ahrq.gov/research-available-for-comment/comment-draft-reports/?pageaction=displayDraftCommentForm&topicid=586&productID=1976

  11. Anne Ö says:

    Thanks, Jennie, we didn’t. Will try again.

  12. Deborah Waroff says:

    Collins never cares. Go for our new sec of dhhs. @Jennie Spotila

Comments are closed.