A shorter version of this article was published on STATNews on January 10, 2019.
One message dominates NIH’s talk about ME/CFS research: submit more high quality grant applications. Funding would increase if there were more high quality grant applications. Give us a chance to prove we’re serious, but send more high quality grant applications. Thank you for your petition with 50,000 signatures, but send more high quality grant applications.
In a field desperate for research funding, one might think that ME/CFS researchers would be flooding NIH with grant applications. Yet that does not seem to be happening. One significant reason why is that NIH’s business-as-usual approach actually increases the barriers to success for ME/CFS grant applications.
A researcher who wants NIH funding for ME/CFS research has to navigate an obstacle course that begins long before the grant application gets in front of reviewers, an obstacle course which arises from NIH’s own broken response to ME. There are at least six questions a researcher must consider in deciding whether to apply for funding:
- Does NIH want my grant? A researcher may decide the answer is no, especially if she wants to generate hypotheses or has been discouraged by NIH’s lack of interest.
- Is there NIH funding for my grant? Given that NIH currently has no Funding Opportunity Announcements targeted at ME/CFS, researchers could very well conclude the answer is no.
- Can I write this grant? On top of the time pressure and institutional challenges that all researchers face, ME/CFS researchers may face additional barriers such as lack of support from institutions, lack of mentors, and the general stigma associated with this disease.
- Who will review my grant? Based on the SEP rosters for the last eighteen years, researchers should expect that some reviewers will be ME/CFS experts but they may not make up the majority.
- Will the SEP members review my grant fairly? Given the unique challenges of the field that are not recognized by non-experts, applicants may conclude the answer is no.
- Who is my competition? There is no set answer to this question. Depending on timing, the competition could be fellow ME/CFS researchers or a much larger and harder to define pool.
Let’s follow a hypothetical researcher as she runs this gauntlet to submit her ME/CFS proposal to NIH.
Does NIH want my grant?
Before the first word is typed on a grant application page, a researcher asks herself whether NIH will be interested in her project. There are multiple reasons why she may conclude that the answer is no.
First, NIH does not fund hypothesis-generating research. A proposal that boils down to “I’m going to look and see what I can see” is not going to succeed. Yet ME/CFS research needs these projects at this stage. This field has not just been ignored; it has been suppressed by decades of stigma and the false narrative that it is caused by deconditioning and depression. Ironically, NIH has tacitly admitted that hypothesis-generating research is needed. The Clinical Care Center study run by Dr. Avindra Nath is designed to collect reams of data that will then generate hypotheses for further research.
Second, a researcher may be deterred by NIH’s demonstrable lack of interest as evidenced by low ME/CFS funding. NIH currently has no Funding Opportunity Announcements targeted at ME/CFS (see below). In addition, NIH funding has been appalling over time–including the 17% decrease in funding last year–so a researcher may conclude that NIH simply isn’t interested in ME/CFS projects.
Third, if the researcher wants to conduct a clinical trial of a drug treatment, she will have trouble at NIH. Dr. Nancy Klimas has tried, but she said, “There is no door to walk through at the NIH” for clinical trial funding in ME/CFS. Klimas attributed this to the fact that the ME/CFS Special Emphasis Panel (see below) does not review clinical trial applications.
Finally, a researcher may be individually discouraged from applying. I have heard multiple stories along these lines, although people are understandably reluctant to go on the record. Dr. Ron Davis went public with the rejection of two of his pre-proposals in 2015. One of the reasons given by the National Institute of Neurological Diseases and Stroke was that, “It was not clear if the proposal falls within the mission of NINDS.”
Is there NIH funding for my grant?
A researcher might decide NIH is interested in her project, but she also has to ask if there is funding available for it. To answer that question, she will look at two general types of Funding Opportunity Announcements.
First, she will look for a Program Announcements, or PA. The PAs are like open house invitations. NIH says, We’re interested in seeing grant proposals in such-and-such area of research. The invitation is really important, because it tells a researcher where the open house is and what time it is happening. There is no guarantee that there will be enough food and booze to go around, but the researcher knows that if she shows up at the specific place and time then she can try to fight her way to the buffet. However, NIH’s last open house invitation for ME/CFS research was issued in 2012 and it expired in 2015. There is nothing whatsoever targeted for ME/CFS at this time.
Incidentally, on the last NIH telebriefing, Dr. Vicky Whittemore said that NIH would no longer be issuing Program Announcements. However, when I followed up with her after the call she said that her comments were premature. Apparently NIH is contemplating moving away from PAs but no announcement has been made yet.
The second type of Funding Opportunity Announcement is the Request for Applications, or RFA. Unlike a general invitation, an RFA is a specific type of funding competition. NIH says, We have X dollars set aside and we want to spend that on this specific type of research. This is more like a competitive swim meet than an open house. You have to qualify for the swim meet in order to compete, but someone is definitely going home with a gold medal. If an ME/CFS researcher is doing the kind of research the RFA wants, then she knows her application has a shot at the set aside funding.
This is why ME/CFS advocates and researchers are constantly asking for RFAs instead of PAs: someone is going to get money out of an RFA competition. This is also why NIH is very reluctant to issue RFAs: it requires NIH to decide in advance how much money it will invest and then set that money aside for the competition.
Just to be clear, NIH issues plenty of RFAs across its full research portfolio, including forty-three RFAs in October 2018 alone. NIH can do this. But NIH has been clear that it has no intention of issuing an RFA in ME/CFS research any time soon.
It’s easy to understand why a researcher might give up on applying to NIH, given the picture I’ve painted thus far. However, let’s assume that our hypothetical researcher has concluded that NIH wants her grant. And despite the fact that there is no funding opportunity targeted at ME/CFS, our researcher has concluded that there is a chance that NIH might fund her grant. Next she has to ask:
Can I write this grant?
Generally speaking, scientific researchers at academic institutions are responsible for obtaining their own funding. Universities do not fund much research themselves. Researchers know they have to write successful grant applications to get funding, and it is essential to their careers to do so.
Yet it is not that simple. An NIH grant application can take several months to write, and that is after months of planning time. The typical NIH application might be 30 pages long, but the applications for the Collaborative Research Center RFA were hundreds of pages long. The more complex the project and the more collaborators involved, the more difficult and time consuming it is to write the application. Submitting successful applications is part of the job description, but so is conducting current research, teaching a full course load, supervising graduate students, and successfully publishing study results. Oh, and there are committee meetings and other administrative duties. The average professor works sixty hours per week.
And our hypothetical researcher does not just need time. She needs support from her institution in the form of equipment, space, and staff. She needs her department head to support her ideas (or at least not actively squash them), and her application must include letters from her institution and collaborators to prove she has that support.
Obviously, this can go wrong in multiple ways, and many of these issues are not limited to ME/CFS research. However, the decades of stigma and misinformation have a unique impact in ME/CFS. Support from institutions and colleagues is harder to come by. Mentors are few and far between. All of the well-known challenges of writing successful grant proposals are multiplied in this field, increasing the difficulty of our hypothetical researcher’s obstacle course. NIH has done nothing to alleviate the challenges that have arisen from its own history with ME/CFS.
Who will review my grant?
NIH’s peer review system is at the core of its funding decisions. The Center for Scientific Review appoints reviewers with relevant expertise to Study Sections and Special Emphasis Panels. The reviewers score applications on a variety of criteria, and come up with an overall impact score. This peer review is not the final decision on an application, but it is critically important. A bad score in peer review is fatal for the application.
Given the importance of the peer review scores, it’s obvious that reviewers must have the appropriate knowledge and expertise. Yet this has not always been the case when it comes to ME/CFS research.
I have been tracking the rosters of the various incarnations of the CFS Special Emphasis Panel, or SEP since 2000, and I’ve seen definite trends. In earlier years, the SEP covered the areas of CFS, Fibromyalgia, and Temperomandibular Joint Dysfunction. As a result, the SEP reviewers were predominantly dentists, psychologists, and pain experts. Between 2000 and 2010, the average representation of CFS experts on the SEP was 15%.
In November 2010, the SEP was assigned a more narrow scope of just CFS (changed to ME/CFS in 2012). The new scope had an immediate effect on the representation of experts on the rosters. Between November 2010 and the end of 2017, ME/CFS experts made up 72% of the rosters on average.
There is one exception, and that is the roster of the SEP that evaluated the applications for the Collaborative Research Centers and Data Management Center in 2017. ME/CFS experts made up only 26% of that roster. There are several possible reasons for this. First, NIH’s conflict of interest policy meant that anyone applying for RFA funding would have been excluded from the roster, along with many of their colleagues. (Read more about the COI policy here ). Second, the nature of the applications required peer review by experts in population studies, computational biology, and other areas outside of ME/CFS research.
Then, the SEP rosters took a puzzling turn in 2018. ME/CFS experts made up 25-44% of the rosters at the three review meetings. I have no explanation as to why the rosters have shifted to include fewer ME/CFS experts. Since the SEP panel is reconstituted for every review cycle, there is also no way to predict representation on future meeting rosters.
Will the SEP members review my grant fairly?
Our hypothetical researcher should be prepared for her application to be reviewed by a variable mix of ME/CFS experts and non-experts, and so she has to wonder if she will get a fair and accurate review score. I am not assuming that non-experts will automatically trash ME/CFS grant applications, but I also do not assume that all SEP members will use the right standards.
First, it is possible that ME/CFS experts on the SEP are not able to assess all aspects of all ME/CFS grant applications. For example, a POTS researcher on the panel may not be familiar with design of genome-wide association studies or computational biology. A psychologist may not be able to critique a study with newer technology like QEEG. Expertise in ME/CFS does not automatically convey expertise in every possible study of the disease.
Second, reviewers bring their own biases with them. Sleep researchers have a different understanding of fatigue than ME/CFS experts. Reviewers may be unfamiliar with post-exertional malaise, including how it differs from fatigue and how to assess it. The worst case scenario is a reviewer who believes the lie that ME/CFS is depression and deconditioning. Dr. Ian Lipkin said that this is exactly what happened with one of his applications in 2014:
I have been in competition now twice to get funded, and the people there who reviewed me gave me abysmal scores. And the critiques of my work were unfair, and one of the people who critiqued my work said, in fact, that this is a psychosomatic illness. I was floored. I protested, and for reasons that are obscure to me this same individual wound up back on the study section, and I got a similar unfundable score.
Third, ME/CFS research has unique challenges that are well-known inside the field, but potentially not understood by scientists outside the field. Case definition is an obvious example of this. The field has used multiple case definitions over the years, some of which have fatal flaws. NIH has refused to select a single gold standard case definition, arguing that researchers should justify their chosen definition in the applications. But how is a non-expert supposed to evaluate that choice and justification? Someone outside the field is probably unfamiliar with the differences between the Oxford, Fukuda, Reeves, Canadian Consensus, and National Academy of Medicine criteria. Outside reviewers will have difficulty assessing the impact of chosen criteria on a study, and they are unlikely to appreciate the challenges of recruiting appropriately diagnosed subjects.
Fourth, peer reviewers will bring expectations from their own fields of study. A cancer or heart disease researcher is used to multi-center studies, with sample sizes in the thousands. That kind of study has been and remains impossible in ME/CFS. Reviewers may have unrealistic expectations about data quality and study design. Non-ME/CFS experts will also be unable to assess whether an area of study is a strategic priority in the field.
The peer review process is a cornerstone of funding decisions at NIH, but it is far from the only factor in play. Our hypothetical researcher faces additional barriers, including her competition.
Who is my competition?
Competition for NIH funding is fierce, not matter what the area of study. NIH’s overall application success rate was 18.7% in 2017. However, an ME/CFS grant application has to compete in ways that put our hypothetical researcher at a disadvantage.
First, an ME/CFS grant application is naturally competing against all the other applications reviewed at a specific SEP meeting. This can actually happen more than once. NIH allows researchers to revise and resubmit applications based on reviewer comments. On the second submission, an ME/CFS application will compete against an entirely new group of applications in front of a different group of reviewers. That means that an application for a proteomic study could have been scored in comparison to other -omics studies in one round, but then compared to POTS or infection studies in the next round. Given that each review meeting has a new roster, new reviewers may have different criticisms of the application than the first group. So our hypothetical researcher could revise her application based on comments from Group 1, and then get entirely new and different criticisms from Group 2.
Second, the competition pool is heavily influenced by the Funding Opportunity Announcement. Recall the open house vs. swim meet analogies I discussed earlier. Those are very different sets of competitors. With a Program Announcement, our hypothetical researcher is competing against everyone else headed for the buffet at the open house. With an RFA, our researcher is competing against just the swimmers in the pool—and someone is guaranteed to win. This will influence the peer review scores. Reviewers for an RFA know that a good score will basically guarantee funding, and select from among the applications in front of them. Normal Program Announcement review is a more diffuse competition, in part because no one is guaranteed funding and the full group of applicants might not be reviewed by the same group of reviewers.
Third, applications that score well at the SEP stage are then sent to the relevant Institute’s Council for consideration. At this level, the application is now competing against all the other applications reviewed at the Council meeting, regardless of the field. For example, an ME/CFS infection study will compete against every other grant coming before the Council of the National Institute of Allergy and Infectious Diseases. The infection study might be fabulous compared to other ME/CFS applications, but not as strong compared to hepatitis and influenza studies that have huge sample sizes, etc. In addition, ME/CFS is not a named priority at any Institute. The ME/CFS immune study might be critical in our field, but the Council (which has no ME/CFS experts on it) might see it as a much lower priority given the Institute goals.
As I pointed out before, ME/CFS has neither an active RFA nor PA. New applications are currently being submitted under very general parent announcements like this one. It invites applications (to any of twenty-three participating Institutes) for defined projects “in scientific areas that represent the investigators’ specific interests and competencies”. This is an incredibly broad net, and the competition is basically all grants being considered in a funding cycle by a particular Institute. There is no target our hypothetical researcher can aim for, other than get the best score she can and cross her fingers.
NIH’s proposed fix for the dismal state of ME/CFS research funding is simply “submit more high quality grant applications.” In order to do that, our hypothetical researcher has to climb over a series of barriers created and maintained by NIH’s actions in ME/CFS research.
Should our hypothetical researcher submit another ME/CFS grant application? NIH sees no reason why she shouldn’t. As a person with ME, I desperately want her to submit one. But will she decide to invest the time and effort, roll the dice, and apply? How many times will she try? Can anyone blame her if she decides to move to another field?
NIH is responsible for erecting and maintaining this obstacle course. Yet they wash their hands of the problem and repeat the refrain, “Send more high quality grant applications.” NIH’s normal approach to encouraging more proposals will not work in ME/CFS.
There is no single silver bullet that can fix NIH’s broken response to ME. However, there are many actions NIH could take to lower the barriers and truly encourage more applications. Difficult problems require complicated solutions. It is time for NIH to tackle this problem with more than just words.