This is the second of a two part series. Part One described the controversy and my fact checking. Part Two presents my analysis of the potential danger to the Clinical Care study.
As part of the renewed focus on ME/CFS at NIH, the National Institute of Neurological Diseases and Stroke is conducting an intramural study of ME/CFS at the NIH’s Clinical Care Center. Dr. Avi Nath is the Clinical Director of NINDS and the Principal Investigator on the study. Some of Dr. Nath’s remarks on the more controversial aspects of the study gave me pause, and I set out to understand the reality and context of those comments. What I found not only confirmed my initial reaction, but exposes a danger to the quality of the study because ME advocates and federal employees are failing to effectively communicate with each other.
The Reality Check: Does Bias Matter?
In his webinar comments, Dr. Nath said, “If you’ve got to eliminate all kinds of [scientists], you’re never going to be able to study anything. Rather, you do as you’re designing your study whereby you don’t have to worry about people’s biases.” After reviewing the emails I obtained through my FOIA requests and speaking with Marian Emr of the NINDS Communications Office, I was left with the question: were advocates’ criticism of Dr. Walitt’s involvement in the study scientifically sound, or is Dr. Nath correct that the study can be designed to make bias irrelevant?
Everyone has biases, and it is a continuing challenge in science. I agree with Dr. Nath that one cannot eliminate all bias from all researchers, and so studies must be designed to control that bias as much as possible. This is why studies use control groups, blinding, and other design elements to keep human error and influence to a minimum.
Dr. Walitt denied that he has a bias towards the psychogenic view of ME/CFS when he spoke at the March telebriefing, but that is hard to square with his previous multiple public statements to the contrary. Dr. Nath argued in the webinar presentation that a well-designed study can make investigator bias irrelevant. In an email on February 26, 2016, Dr. Nath wrote that he was “absolutely certain that no such bias can or will occur in our study.”
I described the Walitt/bias controversy to two scientist sources, one of whom is not in a biomedical field and another who is in a biomedical field unrelated to ME/CFS. The first scientist just stared at me open mouthed, aghast that Nath would claim Walitt’s bias didn’t matter. The other said it was “insane” for Walitt to be on the study if he thinks ME/CFS is a psychological disorder.
In the webinar, Dr. Nath used an analogy about AIDS researchers being biased against homosexuals to show that bias is unavoidable and immaterial. He said that screening researchers for “adverse views about gay people” would have prevented progress in AIDS research. He said, “we know that people have all kinds of biases and we shouldn’t worry about those kind of things.”
I don’t think Dr. Nath made the right analogy. The issue that advocates have with Dr. Walitt is not his personal view about whether he likes his research subjects or their behavior. The issue is whether someone who holds the negated and pseudoscientific view that ME/CFS is a psychological disorder should work on the study. I assume that Dr. Nath would screen his collaborators for their opinions on HIV as the cause of AIDS. I can’t see Nath allowing Dr. Peter Duesberg to collaborate on an HIV/AIDS study, since Duesberg claims that HIV is a harmless passenger virus. That is a bias that Nath would never countenance on an HIV study, and I agree that he should not.
The same should hold true in our case. The psychogenic theory of ME/CFS is wrong. We know this. ME is not a psychosomatic illness. Anyone who holds that view is clinging to a disproven and unfounded scientific theory. There is no good reason or excuse for selecting a scientist for this ME/CFS study who blatantly rejects the scientific evidence. Why would it be reasonable to keep Duesberg off an HIV study, but unreasonable for ME advocates to reject the participation of Walitt and others who have said that ME/CFS is a somatoform illness? This is an unacceptable double standard.
The potential bias of investigators on the Clinical Care study is scientifically relevant. Anyone who believes that ME/CFS is a psychological disorder (in whole or in part) should be disqualified from participating in the study. Such a bias represents a significant risk to the quality of the study, especially because of the harm the psychogenic theory has caused over many decades. In my view, ME advocates have raised legitimate and scientifically sound criticisms of the potential bias of Dr. Walitt and other investigators, and NIH must address these criticisms directly. Dr. Nath should not simply brush them aside as irrelevant.
The Reality Check: Are We Antagonizing Scientists?
Towards the end of the April webinar, Dr. Nath said, “people have to be a little bit careful as to how critical you become. . . . And we want to really try and help, but we can’t do that if the very people you want to help become antagonistic towards you.” I filed FOIA requests for the emails to determine whether ME advocates were antagonizing the scientists working on the study and provoking some of them into withdrawing or stepping back. Was the broader context of criticism on social media so huge and unreasonable that scientists would be justified in refusing to work on the study?
My investigation showed that Drs. Nath and Walitt (combined) received fewer than 20 critical emails over the first four months of 2016. A few are strongly worded or confrontational, but most are thoughtful and well-reasoned. In my view, only one can fairly be classified as antagonistic (the self-described “scornful and contemptuous” one). There were no abusive or harassing emails.
Looking beyond the emails to social media, as Marian Emr suggested, shows that criticism was quite high on blogs, Twitter, and discussion forums. However, it’s not clear how closely any of the NIH scientists personally tracked this. Dr. Nath declined my request to speak and Dr. Walitt declined Julie Rehmeyer’s interview request. We haven’t been able to have a conversation about the sources of the antagonism Dr. Nath was referring to, and so our ability to analyze how reasonable his comments were is limited.
In the absence of direct evidence of personal attacks and the like, Dr. Nath’s admonishment that our criticism would “end up antagonizing all these people” implies that we shouldn’t criticize the study, or at least we should do it very nicely. Even though no one sent nasty or abusive emails to NIH scientists, even though we raised legitimate criticisms in our social media space, even though Marian Emr could not confirm that anyone had withdrawn from the study, Dr. Nath is saying we risk insulting scientists who will now withdraw from the study and not want to help us.
One scientist I consulted said that Dr. Nath’s comment about antagonism was unacceptable and outrageous. Patients have to speak out and be part of the process. Scientists must put patients first, not their own feelings. This scientist noted that many researchers are not used to public engagement, and that outspoken patients can be shocking to researchers who have not dealt with it before. They may not be prepared for the justifiable anger of people who have been sick and neglected for years.
My scientist source said it’s also true that there are some nasty people among advocates, and their behavior can taint the whole community. But he pointed out that HIV/AIDS activists are much more intense than ME advocates. As just one example, look at all the protests at the 1996 International AIDS Conference. ACT UP brought the opening ceremonies to a halt, and protested in multiple parts of the conference. Dr. Nath, having worked on HIV/AIDS, should be used to that. But Dr. Nath says ME advocates are too critical, which deflects attention from the substance of those criticisms to the way in which the criticisms were made.
ME advocates have an obligation to point out the flaws in science and policy that affect our lives. We have the right to be angry about how we have been treated, and the ongoing failures to fix the situation. I wholeheartedly agree that we should not harass, abuse or threaten anyone, but there is no evidence of that in the context of the Clinical Care study. Angry or strongly worded emails and tweets are unpleasant to read, but do not automatically equal abuse. Sarcasm does not equal harassment.
ME advocates’ tactics pale in comparison to HIV/AIDS activists. We are not as numerous, as loud, as omnipresent, as angry, or as bold. But Dr. Nath seems to think that our substantive criticisms of the study and a few confrontational emails are enough to risk antagonizing scientists. I don’t understand why. In my experience, scientists generally have robust egos. If HIV/AIDS researchers can deal with AIDS activists, and even partner with them in meaningful and substantive ways, then the same kinds of partnerships should be possible in ME.
Failing to Communicate is Dangerous
Advocates and NIH are talking past each other. Advocates are expressing strong, substantive, and scientifically sound criticisms of a highly significant and long awaited study. Dr. Nath says we’re antagonizing scientists and should be nice. These are two different conversations. Our failure to succeed in having these conversations poses a risk to the ultimate quality of the Clinical Care study results.
Advocates and NIH should have a substantive discussion about the design and conduct of the Clinical Care study. Patients are entitled to a seat at the table. We have things to teach NIH, and NIH has things to teach us. That process is overdue. Both advocates and NIH would also benefit from discussing how we express and receive criticism. But this must be a two-way conversation between equals. Scientists can’t claim they want to take their test tubes and go home because advocates are being mean if they haven’t actually listened to and considered what the advocates are saying. Advocates, on the other hand, can’t claim that their suffering automatically makes them right.
I had some hope that these conversations and others would be made possible by NIH’s publicly stated intention to create a patient advisory panel of some kind. Many advocates have made a strong case that we should not only be involved in all stages of the Clinical Care study, but also in devising NIH’s research strategy for ME/CFS. But the creation of this panel seems to have stalled.
In the April 21st webinar, Dr. Nath said that it “turns out to be much more
complicated than I originally imagined.” He said that the details of who and how many should be selected, and what their role should be, were unclear. Dr. Nath said the “extramural folks” were approaching people for the panel. Dr. Vicky Whittemore confirmed to me that there was nothing to share at that time. Dr. Whittemore later suggested that the CFS Advisory Committee form a working group to examine how patients could be engaged in the agencies’ work, but the group has yet to meet.
Science requires criticism. It’s part of the process, at every single stage. In the case of ME, patients and advocates know far more about this disease than scientists who are new to the field. This is exactly what happened in HIV/AIDS. Those patients were extremely critical, extremely vocal, and extremely active in standing up to the way government scientists wanted to do things. They also secured patient participation in AIDS research at all stages and at all levels, working alongside scientists to improve their research and to learn information that they could take back to their community. Those combined efforts are what accelerated progress in HIV/AIDS.
I wonder if Dr. Nath has missed the point that vociferous criticism is part of our job as advocates. We are obligated to speak out about our disease, our experiences, and the science needed to find answers. We are fully qualified to participate in the scientific enterprise, and our perspectives are necessary.
Without meaningful participation by patients with a diversity of perspectives, the Clinical Care study is at risk. The controversies over the Reeves criteria and functional movement disorder control group are two examples, and the potential bias of Dr. Walitt is another. These controversies might have been avoided if advocates were involved at the early stages of protocol design. I think there are a variety of other ways we could assist as well. For example, the series of tests planned in the study will be quite grueling. We could help adjust the design to make it easier on participants. In addition, I suspect that the team has not thought through the full spectrum of effects that the study will have on patients – before, during and after participation. Again, adjustments to design could collect those data and augment the study’s impact.
I don’t think NIH’s refusal to engage with us as equals, or scientists’ dislike of our criticism, is a nefarious conspiracy. An essay by Kameron Hurley (about a different topic) explains why:
When the internet loses its shit over what, to many, looks like a single insignificant incident unrelated to anything else, it’s easy to say they’re fucking nuts. They’re raging over some perceived slight that’s been blown waaaaay out of proportion. That, in truth, is the easier narrative . . . . It’s easier to say people are crazy than to try and figure out why.
. . . .
Change is messy. It’s angry. It’s uncomfortable. It’s full of angry people saying angry things, because they’ve been disrespected and forgotten again and again and again and again, and they’re tired of being fucking nice because it makes you uncomfortable if they act in any way that is not deferential or subservient to you and your worldview.
This is exactly the situation we are in with NIH and other federal agencies. They seem to think we’re crazy, we’re antagonistic, and we’re overreacting. They want us to tone down the anger, and criticize in a more polite and quiet way. In Dr. Nath’s comments, I hear: “Just go away and let us do the science. We’re the experts here.”
There is no question that Dr. Nath is a world-renowned expert who is capable of designing an excellent and elegant study. But some of the scientific decisions made by NIH, including the involvement of Dr. Walitt, are worthy of criticism. Our criticisms, even the angry ones, deserve a fair hearing.
The appropriate response to our criticisms is not, “You don’t know what you are talking about. Sit down and speak softly, or not at all.” It is not to dismiss us all as antagonistic and crazy because someone received a couple of critical emails. The appropriate response is to listen, and to consider our points. Then we can partner and learn from each other. And that is essential to producing good science.
PACE: Grossly Exaggerated
The bottom line? The PACE trial authors’ claims that CBT and GET are effective treatments for ME/CFS were grossly exaggerated.
Improvers
First, take a look at what the PACE authors’ own reanalysis showed. When they calculated improvement rates using their original protocol, the rates of improvement dropped dramatically.
One would think that the PACE authors would be at least slightly embarrassed by this, but instead they continue to insist:
Sure, twice as many people improved with CBT and GET compared to standard medical care. But 80% of the trial participants DID NOT IMPROVE. How can a treatment that fails with 80% of the participants be considered a success?
Not only that, but the changes in the protocol were like a magic wand, creating the impression of huge gains in function: 60% improved! The true results, however, are close to a failure of the treatment trial.
Recovery
Today’s publication on Dr. Racaniello’s blog presents the analysis of the recovery outcome data obtained by Alem Matthees. Once again, the mid-stream changes to the study protocol grossly inflated the PACE results.
Source: Matthees, et al.
As the graph from the Matthees paper shows, the PACE authors claimed more than 20% of subjects recovered with CBT and GET. Using the original protocol, however, those recovery rates drop by more than three-fold. Furthermore, there is no statistically significant difference between those who received CBT or GET and those who received standard care or pacing instruction. In other words, the differences among the groups could have easily been the result of chance rather than the result of the therapy delivered.
Matthees, et al. conclude, “It is clear from these results that the changes made to the protocol were not minor or insignificant, as they have produced major differences that warrant further consideration.” In contrast, long time CBT advocate Dr. Simon Wessley told Julie Rehmeyer that his view of the overall reanalysis was, “OK folks, nothing to see here, move along please.”
Taken together, the reanalysis of data on improvement and recovery show that the changes in the protocol resulted in grossly inflated rates of improvement and recovery. Let me state that again, for clarity: the PACE authors changed their definitions of improvement and recovery and then published the resulting four-fold higher rates of improvement and recovery without ever reporting or acknowledging the results under original protocol, until now. Furthermore, the PACE authors resisted all efforts to obtain the data by outside individuals, spending £250,000 to oppose Matthee’s request alone.
Conclusions
Tuller’s detailed examination of the PACE trial and these new data analyses raise a number of questions about why these changes were made to the protocol:
The grossly exaggerated results of the PACE trial were accepted without question by agencies such as the Centers for Disease Control and institutions such as the Mayo Clinic. The Lancet and other journals persist in justifying their editorial processes that approved publication of these grossly exaggerated results.
The voices of patients have been almost unilaterally ignored and actively dismissed by the PACE authors and by journals. We knew the PACE results were too good to be true. A number of patients worked to uncover the problems and bring them to the attention of scientists. Their efforts went on for years, and finally gained traction with a broader audience after Tuller and Racaniello put PACE under the microscope.
For five years, the claim that CBT and GET are effective therapies for ME/CFS has been trumpeted in the media and in scientific circles. Medical education has been based on that claim. Policy decisions at CDC and other agencies have been based on that claim. Popular views of this disease and those who suffer with it have been shaped by that claim.
But this claim evaporates when the PACE authors’ original protocol is used. Eighty percent of trial participants did not improve. Not only that, but we do not have any data on how many people in that group of 80% were harmed or got worse. CBT and GET may not be neutral therapies worth trying in case you fall in that lucky 20% who improved spontaneously or due to the treatment. We don’t know how many people got worse with these therapies, so we cannot assess the risks.
The end result is this: the PACE authors made changes to their protocol after data collection had begun, and published the inflated results. But when the original protocol is applied to the data, CBT and GET did not help the vast majority of participants. The PACE trial is unreliable and should not be used to justify the prescription of CBT and GET for ME patients.
As Matthees, et al., stated in their paper: