What is the purpose of the ME/CFS P2P meeting at NIH? You would think that we would know by now, since Assistant Secretary Dr. Howard Koh first announced the effort in October 2012. But to say the rhetoric has evolved over time would be a kind description.
HHS keeps changing the answers to questions about the purpose of the workshop, what kind of research is on the table, and whether the ME/CFS experts have meaningful input.
To me, it looks more like a bait and switch where the meeting sounded better the further back in time you look, and key information (like the panel being 100% non-experts) was withheld until the very last minute. The reality of this meeting is very different from the picture they portrayed early on.
Are we making a research case definition or not?
First they told us the workshop would create a new case definition:
The NIH has made a commitment to conduct an evidence‐based review of the status of ME/CFS research and also convene a dedicated workshop to address the research case definition for ME/CFS. Dr. Howard Koh, October 3, 2012 CFSAC Minutes, p. 5.
To address the highest priority identified, which was “case definition issues,” the Working Group submitted a competitive application for an Evidence-based Methodology Workshop (EbMW) on ME/CFS coordinated by the NIH Office of Disease Prevention. May 1, 2013, Response from Dr. Howard Koh to CFS Advisory Committee, p. 3
Then they told us it wouldn’t:
The purpose of the Pathways to Prevention Program and the ME/CFS workshop is not —and I repeat, not—to create a new case definition for research for ME/CFS. Dr. Susan Maier, December 11, 2013 CFSAC Minutes, p. 16.
But in the middle, they said the meeting might help with a research case definition:
This will not create a research case definition in the end, but will inform anyone who wants to do research in this area about what aspects of the case definition are really strong, which are really lacking, and how those holes might be filled. Dr. Beth Collins-Sharp, May 23, 2013 CFSAC Minutes, P. 16.
But the meeting is about research, right?
The answer might depend on the day, and the person you ask. Here are the research-oriented answers:
The purpose of an evidence-based methodology workshop is to identify methodological and scientific weaknesses in a scientific area and move the field forward through the unbiased and evidence-based assessment of a very complex clinical issue. Dr. Susan Maier, May 23, 2013 CFSAC Minutes, p. 6.
The takeaways from a systematic review are answers to the key questions that identify where there’s strong evidence, where there are gaps, and some ideas about how those gaps may be filled. Those are called research recommendations. Dr. Beth Collins-Sharp, May 23, 2013 CFSAC Minutes, p. 13.
It has the potential to be both [research and clinical], but understanding that we are a research organization and our focus is to improve the, um, the integrity of the science that is used for translation into clinical care means that we have to focus on besting the science that is used for the evidence. Dr. Susan Maier, Institute of Medicine Public Meeting, January 27, 2014, Minute 0:19.
Bob Miller, who served on the P2P Working Group, certainly thinks the meeting is about research:
NIH is hosting a Pathway to Prevention workshop to identify gaps in scientific research, to guide a path forward for NIH research. Bob Miller, March 11, 2014 CFSAC Transcript, p. 114.
But at other points, it appears the focus is back on the case definition:
The purpose of the Pathways to Prevention Program for ME/CFS is to evaluate the research evidence surrounding the outcome from the use of multiple case definitions for ME/CFS and address the validity, reliability, and ability of the current case definitions to identify those individuals with or without the illness or to identify subgroups of individuals with the illness who might be reliably differentiated with the different specific case definitions. Dr. Susan Maier, December 11, 2013 CFSAC Minutes, p. 16.
Doesn’t this assessment of multiple case definitions and what research tells us about subgroups sound like what the IOM panel is doing right now? And if IOM is already doing this, why do we need a separate process at NIH where the decision makers are ALL non-ME/CFS experts?
The expert gets to decide, right?
I went back through CFSAC minutes and other documents, looking for the first time Dr. Maier or another federal employee told an ME/CFS audience that the P2P Panel would be composed entirely of non-ME/CFS experts. It was January 27, 2014 in her presentation to the Institute of Medicine, when Dr. Maier offered her ill-fated “jury model” analogy. Dr. Susan Maier, Institute of Medicine Public Meeting, January 27, 2014, Minute 6:25.
Just to be clear, the earliest public discussion of P2P was October 2012, but it wasn’t until almost 16 months later that Dr. Maier finally told us that the P2P Panel would have no ME/CFS experts on it. Why did it take so long? Maybe the better question is why January 2014. Would Dr. Maier have talked about her jury model of “They don’t know, they don’t know anything” if I had not already exposed this here on January 6, 2014? Maybe, but it strikes me as more than odd that despite at least two opportunities to tell CFSAC about the “jury model,” she waited until the IOM meeting to actually disclose it.
But the government says Don’t Worry! Your experts are participating!
The working group will meet to develop the questions that will form the basis of the evidence-based review, develop workshop themes and structures, suggest speakers, and develop an agenda for the meeting. The deliverable from this meeting will be a list of questions for the evidence review, themes for the workshop, perhaps a draft agenda, and any speaker names for those who will speak at the meeting. Dr. Susan Maier, December 11, 2013 CFSAC Minutes, p. 16-17.
Our experts and I had real input into the agenda and questions. The Working Group drove the agenda, and we will participate in the Workshop. I believe the prep work for the Workshop is being done with strong representation from our illness, laying the foundation for a good outcome. Bob Miller, January 12, 2014.
It sure sounds like that Working Group finalized the questions for the evidence review:
The Key Questions were defined by the Working Group of content experts at a planning meeting organized by the NIH Office of Disease Prevention. May 2, 2014 Email from CFSAC listserv.
There’s a problem, everybody. Multiple sources who are in a position to know what happened at the January 2014 Working Group meeting told me that the questions in the study protocol were not the questions defined at the meeting. Did something happen between that January meeting and the release of the study protocol? I don’t know whether someone continued to tinker with the questions, or why the Working Group was not consulted. But either the questions have been significantly changed, or the information from my sources is deeply flawed.
Why is this a problem? Well, in addition to all the problems I documented with the study protocol, those questions form the structure of the P2P Workshop. Those questions give us a pretty good idea of what will be on the Workshop agenda, and I will supplement that with additional exclusive information in my next blog post.