Note: After publishing this post, I discovered that I had inadvertently missed one meeting in 2017. This post was updated on February 12, 2019 to reflect all new calculations. The changes are not significant enough to alter any conclusions.
There is no question that NIH’s funding of ME/CFS research has been minuscule relative to the size of the public health crisis. Review of ME/CFS grant applications at NIH has drawn scrutiny from the public as one contributing factor. The public perception is that the grant review panelists have not been ME/CFS experts, and that this has led to the unfair denial of qualified applications.
That first point—that grant reviewers are not ME/CFS experts—has a factual answer. The second allegation—that the lack of experts has negatively impacted funding decisions—is harder to answer with publicly available information. Nevertheless, in 2013 I embarked on a project to gather the evidence and answer these questions.
This article will focus on the first issue: who is reviewing the applications. My analysis of the data points to two main conclusions:
- A small subset of reviewers (experts and non-experts) wield disproportionate influence because they serve so many times.
- NIH changed its approach to ME/CFS application reviews in November 2010. Since that date, NIH has primarily appointed ME/CFS experts to evaluate the applications.
Let’s begin by reviewing the basics of NIH’s grant review process.
How NIH Reviews Grant Applications
When a grant application is submitted to NIH, a multi-level review process begins. In the first stage, a review panel of non-federal scientists with relevant expertise evaluates and scores the application on a variety of criteria.
The Center for Scientific Review (CSR) at NIH is responsible for selecting reviewers for the panels. CSR manages hundreds of these panels, which fall into two general categories: standing study sections and special emphasis panels. Special emphasis panels (or SEPs) are comprised of temporary members, selected specifically for the applications under review at a single meeting. Most SEPs are used once and then dissolved, but there are a dozen or so recurring SEPs for areas with an ongoing need for review. ME/CFS is one of those topic areas, and its recurring SEP has a new roster for each meeting.
Each study section and SEP is managed by a Scientific Review Officer (SRO). This is not a desk jockey job; the SRO has a substantive impact on the peer review process. The SRO is responsible for selecting scientists for the panel, monitoring potential conflicts of interest, and preparing summaries of the peer review scores and critiques.
Review panel members must have substantial relevant scientific expertise and knowledge of the most current science. SROs look for reviewers who have themselves received major peer-reviewed grants, and who understand the peer review process. The quality of grant application reviews is largely dependent on selecting the right scientists to review them.
The Methods of This Project
The obvious first step for my analysis was to gather all the SEP rosters and look at who served. Study sections and SEPs are federal advisory committees, and as such their membership must be made public. You might think that getting the rosters would be easy. You would be wrong.
In 2013, I looked for the rosters online, and found very few. When I asked NIH about it, I was told that the rosters were not posted publicly “due to threats some previous panel reviewers have received.” (this is an interesting story for another time) I was instructed to file a FOIA request for the rosters. NIH then denied that request, and to make a long story short, it took me two years of appeals to finally obtain the rosters. For several more years, NIH absurdly required me to file a FOIA request for each roster. It took intervention by Dr. Joe Breen in 2016 to finally change CSR’s policy on publishing the ME/CFS SEP roster.
Since one of my main objectives was to identify how many ME/CFS experts participated, I had to define who qualified as an expert. I did not assume that I knew all the experts and could simply rely on name recognition. For purposes of this analysis, I set the expertise bar very low. I defined an ME/CFS expert as anyone who—at the time they served on the SEP—had at least one publication on ME/CFS or had an NIH grant for ME/CFS research.
I compiled all the roster names for the SEP meetings from 2000 through 2018. I searched PubMed for each person’s ME/CFS publications at the time he or she served on a SEP. I also did my best to identify the scientific specialization of all the members by reviewing their institutional profile pages and CVs. Then I looked for the trends and patterns.
Representation As A Whole
Between January 2000 and December 2018, the ME/CFS SEP met 62 times.* A total of 327 people served as reviewers. Of those 327 panelists, 58 (or 17.7%) qualified as ME/CFS experts under my liberal definition.
Half of all reviewers served more than once, and each roster varied between 5 and 36 members. To calculate the average number of times individuals served, I counted the combined roster seats across all the meetings: 836 seats. Of the total 327 panelists, each person served an average of 2.6 meetings. However, the 58 ME/CFS experts served a combined 207 seats, or 24.7% of the total seats. Those 58 experts served an average of 3.6 meetings each.
First finding: Between 2000 and 2018, 17.7% of the reviewers were ME/CFS experts, and they served 24.7% of the total roster seats.
The percentage of ME/CFS experts at each meeting varied between 0 and 100%. Eight meetings included no ME/CFS experts whatsoever, while four meetings were 100% experts. Over the entire time period, ME/CFS experts made up 20% or less of the rosters of 32 meetings.
Second finding: Just over half of the meetings included 20% or less ME/CFS experts, and eight of those meetings included no experts at all.
Of the 327 total individuals who served on the SEP, I identified 65 (20%) that have psychology or psychiatry degrees. Note that this includes researchers who are ME/CFS experts, such as Drs. Jarred Younger and Lenny Jason. Twenty-four people (7.3%) specialize in craniofacial diseases such as Temporomandibular Disorders. Fourteen (4.2%) are sleep researchers. There are six people who appear in more than one of these categories (such as a psychologist specializing in insomnia).
To measure the influence of these specialties, I looked at how many times these individuals served on the SEP. The 65 psychologists served a total of 214 times, or 25.6% of the total seats. Adding in the sleep and craniofacial specialists (and taking the overlaps into account), these three categories combined represent 29% of the total individuals, but 36.7% of the meeting seats.
Third finding: One-third of all reviewers specialize in psychology/psychiatry, sleep, and/or craniofacial areas, and occupied 36.7% of the meeting seats between 2000 and 2018.
As mentioned above, each reviewer served an average of 2.6 times. However, this is a bit misleading because 71% (233 people) served only once or twice, and the remaining 29% served three or more times. The reviewers who only served once or twice occupied just 36% of the review seats. That means 29% of the reviewers (experts and non-experts) occupied 64% of the seats. To be clear, this means just 94 people filled 534 seats between 2000 and 2018 because they served so many times.
Fourth finding: A minority of reviewers (29%) had a disproportional influence on the review process because they served so many times (64% of seats overall).
The ME/CFS Experts
As I stated in the Methods description above, I used a very liberal definition of ME/CFS “expert.” I classified an individual as an expert if he or she had at least one ME/CFS publication or at least one NIH grant for ME/CFS research at the time of service on the SEP. It turned out that there are a few reviewers who served on the SEP prior to having a publication or grant in ME/CFS, and then served again afterward. I adjusted my analysis to take this into account. You can read the entire list of ME/CFS expert reviewers here.
A total of 58 out of 327 reviewers (17.7%) met the expert definition for at least one meeting served. Many of the names will be immediately recognizable as experts, but others may be a surprise. For example, Dr. Ila Singh published on XMRV and then left the ME/CFS field. Dr. Jordan Dimitrakoff co-authored a paper with his colleagues from the CFS Advisory Committee, but he is a pelvic pain specialist and has done no ME/CFS research. Yet under my liberal definition, both are counted as ME/CFS experts. I was also surprised to find five people who were CDC employees when they served on the SEP: Dr. Jim Jones, Dr. Elizabeth Unger, Dr. Alison Mawle, Dr. Mangalathu Rajeevan, and Dr. Alicia Smith. I do not know if it is unusual for CDC employees to serve on NIH grant review panels.
Fifth finding: Using the most liberal definition of ME/CFS expert, only 17.7% of the reviewers qualified. Multiple people on the list were never involved in much ME/CFS research and/or left the field. Five individuals were CDC employees at the time they served on the SEP.
ME/CFS experts served an average of 3.6 meetings each, but this is misleading because 40% of the group served only once. When I removed the one-timers from the calculation, the remaining 35 reviewers served 184 times, which is 89% of the total number of expert seats. Concentrating grant review assignments to such a small number of scientists is risky. One person’s bias, expectations, preferences, and professional experience can shape the direction of NIH funded research, for better or worse. This is especially true for the reviewers who serve most frequently. At the very top of that list are:
- Dr. Fred Friedberg, psychologist, 15 times;
- Dr. Jim Baraniuk, MD, pain and fatigue, 14 times;
- Dr. Italo Biaggioni, MD, pharmacology, autonomic dysfunction, 10 times as an expert (17 times overall); and,
- Dr. Maureen Hanson, genetics and cell biology, 10 times
These four reviewers served a combined 49 times, which is 23.6% of the total expert seats. The heavy influence of Dr. Friedberg is an example of the inherent risk of this approach. While he has worked in this field for more than fifteen years, and has received $3.9 million in NIH grants, he is a psychologist. Proposals that rely on computational biology, cutting edge imaging, or immunology could be challenging for a behavioral psychologist to properly evaluate. There are other ME/CFS experts, including other psychologists like Dr. Jarred Younger, who may be better positioned to review these applications.
Sixth finding: Just 35 ME/CFS experts have served a combined 184 times (89% of expert seats). Just four experts (Friedberg, Baraniuk, Biaggioni, Hanson) have occupied 23.6% of those seats. They have likely wielded great influence on application scores and critiques.
Before and After November 2010
So far, I have presented my findings based on all the rosters from January 2000 to December 2018 combined. That is not the whole story, however. NIH changed its approach to reviewing ME/CFS grant applications in November 2010.
Prior to November 2010, the SEP reviewed grant applications related to Chronic Fatigue Syndrome, Fibromyalgia, and sometimes Temporomandibular Disorders (TMD). The rosters had titles like “CFS/FM SEP” and “CFS/FMS/TMD.” Beginning with the SEP meeting on November 2, 2010, NIH narrowed the focus of the panel to CFS only. The meeting titles changed to “Chronic Fatigue Syndrome” and “Myalgic Encephalomyelitis/Chronic Fatigue Syndrome.”
The name of the SEP was not the only difference. The types of reviewers appointed to the panels changed significantly. Pain researchers and dentists were out, and ME/CFS experts were in.
| Before Nov 2010 | After Nov 2010 | |
|---|---|---|
| Number of Meetings | 36 | 26 |
| Number of Seats | 605 | 231 |
| Meetings with No Experts | 8 (22%) | 0 |
| Meetings with 1-20% Experts | 23 (64%) | 1 (4%) |
| Meetings with 21-50% Experts | 5 (14%) | 7 (27%) |
| Meetings with 51-99% Experts | 0 | 14 (54%) |
| Meetings with 100% Experts | 0 | 4 (15%) |
| Non-expert seats | 538 of 605 (89%) | 91 of 231 (39%) |
| Expert seats | 67 of 605 (11%) | 140 of 231 (61%) |
| Psych/sleep/craniofacial | 275 of 605 (45.5%) | 34 of 231 (14.7%) |
As you can see, beginning with the November 2010 meeting the SEP rosters are almost directly opposite to the earlier rosters. The expert representation went from 11% to 61%, while non-expert representation dropped from 89% to 39%. I do not know why the shift was made at that particular time, but there is no doubt that it was. It seems unlikely that this was the sole decision of the SRO at the time, but I have no documentary evidence that points to how the decision was made.
Seventh finding: Beginning in November 2010, the focus and composition of the SEP shifted dramatically and included substantially more ME/CFS experts than any meetings prior to that date.
As good as things look after November 2010, there is one troubling trend. Eight of the 25 meetings had 50% or less ME/CFS experts. Seven of those meetings were held since April 2017, including the panel that reviewed the RFA proposals in July 2017.
The roster for the RFA review went through multiple iterations. The final version included 37% ME/CFS experts. This roster must have been difficult to put together because there were so many experts participating in one or more of the fifteen proposals reviewed at that meeting. The conflict of interest policy would have excluded many of them from service on the panel.
The panels for the meetings since July 2017 may signal a dangerous shift in approach. All four had less than 50% ME/CFS experts, with the April 2018 meeting including only one expert and seven non-ME/CFS experts. All four rosters were overseen by Dr. Jana Drgonova. What her approach will be going forward remains to be seen.
Eighth finding: The SEP that reviewed the RFA proposals included only 37% ME/CFS experts, possibly due to the conflict of interest policy excluding many reviewers. The use of experts on the normal SEP panels declined to less than 50% after July 2017, for reasons unknown.
Summary
Rather than repeat the legend that ME/CFS grant applications are reviewed by dentists and psychologists, I set out to examine the data on who reviews these applications. My analysis points to two main conclusions.
First, there is an inside/outside club of reviewers. For ME/CFS experts and non-experts alike, a small subset wields great influence through service at multiple meetings. Among ME/CFS experts, 60% of the experts occupied 89% of the expert seats. The top four individuals occupied 23.6% of the seats. Among non-ME/CFS experts, 48% of the reviewers occupied 78% of the non-expert seats. Given how these subsets wield out-sized influence through repeated appearances, one hopes that this is favoring high-quality reviews and not unreasonably negative ones.
Second, these data show that NIH adjusted its approach in November 2010. The reliance on ME/CFS experts jumped overnight, and the SEP was refocused on ME/CFS applications alone. However, the negative trend to use fewer experts in 2018 bears careful watching.
The real question is how these rosters impacted grant funding decisions. My next article will present that analysis.
Recap of Findings:
- Between 2000 and 2018, 17.7% of the reviewers were ME/CFS experts, and they served 24.7% of the total roster seats.
- Just over half of the meetings included 20% or less ME/CFS experts, and eight of those meetings included no experts at all.
- One-third of all reviewers specialize in psychology/psychiatry, sleep, and/or craniofacial areas, and occupied 36.7% of the meeting seats between 2000 and 2018.
- A minority of reviewers (29%) had a disproportional influence on the review process because they served so many times (64% of seats overall).
- Using the most liberal definition of ME/CFS expert, only 17.7% of the reviewers qualified. Multiple people on the list were never involved in much ME/CFS research and/or left the field. Five individuals were CDC employees at the time they served on the SEP.
- Just 35 ME/CFS experts have served a combined 184 times (89% of expert seats). Just four experts (Friedberg, Baraniuk, Biaggioni, Hanson) have occupied 23.6% of those seats. They have likely wielded great influence on application scores and critiques.
- Beginning in November 2010, the focus and composition of the SEP shifted dramatically and included substantially more ME/CFS experts than any meetings prior to that date.
- The SEP that reviewed the RFA proposals included only 37% ME/CFS experts, possibly due to the conflict of interest policy excluding many reviewers. The use of experts on the normal SEP panels declined to less than 50% after July 2017, for reasons unknown.
*There was a meeting scheduled for February 22, 2011 but it was canceled. A meeting was eventually held on March 24, 2011 with a different roster. I have excluded the February meeting from this analysis.
Jennie, you are amazing! This represents a very large amount of focused work, and the results are somewhat disturbing, despite the major shift after 2010. There is still room for much improvement, and I hope NIH takes serious note of what you point out. We all owe you thanks–again!
Can a researcher on the review panel have a funding grant under review by the panel at the same time? If there are a limited number of ME researchers, if they are serving on the review panels, can the panels review their grants?
No, a researcher whose application is under review at a meeting cannot be at that meeting. I wrote about the conflict of interest policies here: https://www.meaction.net/2017/07/21/jennie-spotila-nihs-management-of-conflict-of-interests/
I suspect that this is one contributing factor as to why some reviewers serve so much more than others. If they apply for lots of grants, they are eliminated from the review panel.
This then creates a “circular problem” as ME researchers have to choose whether to submit grants to fund their work or serve as a grant reviewer, but not both.
Given that the SEP roster is new for each meeting, it’s not really a circular problem. There are three meetings a year, and very few researchers are submitting for every single cycle. So a researcher can submit for cycle 1 and then be available to review during cycle 2 or 3 (if NIH invites them to do so).
That then raises the question of whether there is any relation between successful/unsuccessful grant applicants and roster membership. Do you have any data on that?
That is EXACTLY the next question. I have some data, and my next post will be that analysis.
Fabulous work Jennie.
Most disturbing:
1. Fred. Between his perennial presidency of the IACFSME, overseer status on the publication which he inappropriately named “Fatigue” and his premier role as a reviewer psychologist Fred Friedberg dominates this field. Yet his work is invariable not relevant to our health and has yet to produce new information or understanding about the disease. It is mainly behaviour modification oriented, like the English CBT & GET. He is very clever about scooping up grants from different, unexpected parts of NIH but apparently feels no compulsion to turn this into relief for patient suffering. Placidly, he watches us die.
2.CDC. This amounts to doubling up on CDC influence and political commitments; it is contrary to the intent of Congress for CDC to govern granting of monies by NIH.
3.Absence of distinguished scientists. This nation is replete with accomplished scientists working in relevant related areas such as neurology, rheumatology, infectious disease, immunology and more who could be bringing their knowledge and experience to bear to benefit ME research. Their commentary could be worthwhile. Instead NIH relegates us to schlock. But then, we are in the Janitor’s Closet of NIH in the unfunded office of women’s research, as Hillary Johnson has described.
4. The demand that Jennie obtain FOIAs to access information that is supposed to be publicly available amounts to harassment and obstruction and should be part of a congressional investigation.
“I have no documentary evidence that points to how the decision was made.”
This should be the headline, in all caps and a very large typeface. It applies to most or all aspects of NIH policy regarding this illness. No one can say exactly who or how the Policy of No Research is made, and no one is allowed to ask. It just happens. Like magic.
I agree with deboruth’s assessment of Dr Friedberg’s work. He comes across as sympathetic to patients, but his work is ultimately useless. It is just what the NIH ordered, which is why he gets so many grants and is appointed to so many review panels. And by the way, who put a psychologist in charge of the IACFSME?
I would like to know a lot more about the SRO. After all, it’s not who votes that counts, it’s who counts the votes.
There have been six different SROs from 2000-2018. It’s not that there are votes to be counted, though. The SRO tabulates and summarizes the scores and review comments made by the panel members. I think the SROs wield the most power in selecting the reviewers to begin with.
Must have taken an awful lot of time and energy!
Thank you Jennie!
Writing this post took me all of January, but data collection and analysis has taken months of on and off work. I hope it will be helpful to advocates and researchers! Thanks!
Jennie, another stunningly well researched and written investigative report. I’d like to see you eligible for a national investigative reporting award! I look forward to the next installment. The upshot, of course, is just one more way the NIH screws the ME/cfs population, in this case by not casting a wider net for more expert reviewers.
And to ditto others: Fred Friedburg really should be replaced as head of the IACFS/ME. His main interest is self-interest.
Pingback: Who Reviews ME/CFS Applications For NIH ? – The York ME Community
Thank you for this informative post, Jennnie. It is a beautiful gold nugget!
Geography appears to play a role in NIH decision making – at least where ME is concerned. Three of the four most frequent reviewers are based on the east coast (at Stonybrook, Cornell and Georgetown) with the fourth at Vanderbilt in Tennessee.
More than this, all three of the Clinical Research Centers are on the east coast, as is the Data Center.
One possible explanation for the reliance on East Coast reviewers could be cost, e.g. the cost of travel to bring people to NIH. However, I am not sure how many of the review meetings have been in person. In general, study sections meet face to face. However, many of the SEP meetings are half-days, and I wonder if some of them are conducted via the web. Something to look in to, perhaps.
Some SEP participants have told me the meetings are via the web.
I do not know when that started though.
The Federal Register notices for the meetings indicate whether they are in person or via the web. Someone would have to dig back through the Register to look at the meeting notices. Interestingly, the rosters themselves do not indicate whether they are in person or not.
Very impressive Jennie.
I’m dismayed, though, by the harsh attacks on Fred Friedberg. He has ME/CFS and he has been the volunteer, unpaid President of the IACFS/ME for quite some time. As the President of that organization he has always strongly pushed for more physiological research.
It’s disheartening to read blanket condemnations of someone just because some of his research is focused on self-help and coping mechanisms. I would point out that not all of this research is. In 2014 Friedberg published a study indicating that recovery was far less common than studies suggested. In 2012 he teamed up with Staci Stevens on an exercise paper. In 2010 he worked with Lenny Jason to characterize five different types of fatigue. In 2009 he worked with Lenny Jason on activity logs.
There is nothing the matter with the work that Friedberg is doing – his latest study will not get anywhere close to the cause of ME/CFS but there’s nothing the matter with heart rate variability measures to improve pacing, reduce pain and fatigue and enhance coping.
He’s not taking money away from anyone. He’s simply applying for grants – something quite rare in this research community.
Friedberg, as I said who has ME/CFS – has stepped forward and contributed his time and energy to the IACFS/ME. Would that more of our researchers and ME/CFS experts – few of whom have ME/CFS – were as generous.
How that is construed as “self-interest” I have no idea….
I think we need to differentiate between assessment of a person’s body of work and assessment of his or her qualifications to sit on a grant review panel. Having ME/CFS does not automatically qualify someone to serve on the SEP. Being president of IACFS/ME does not automatically qualify someone. And being a psychologist does not automatically DISqualify someone. As I pointed out, Dr. Jason and Dr. Younger both have psychology degrees.
My concern with Dr. Friedberg serving on the SEP is based on whether he is BEST qualified to evaluate grant applications with advanced immunology, computational biology, molecular biology, etc. I’m also concerned with how many times he has served, and his overall influence on grant review over the last nine years (he first served in 2010).
If Fred Friedberg dominates this field – which he certainly doesn’t – The IACFS/ME is pretty much a bit player in all of this – but if he did – then one would have to conclude that he’s putting his money on a physiological approach to ME/CFS since most of the NIH funded research does go to physiological research and that certainly is the IACFS/ME’s thrust.
The Journal was named Fatigue because in order to gain traction it needed to have a title that would attract a broad part of the research community. Naming it M.E. or something like that would have been the kiss of death – only people in this field would read it. Calling it Fatigue creates the possibility of others learning about this field.
In the time leading up to the shift in SEP composition, I wonder what sort of public discussion (CFSAC for instance) there had been about who was on SEPs.
I agree that people are being unfair to Fred. I first met him when we were working on the 1994 10+ Study of ME/CFS Patients who had been ill ten years or more, the brainchild of Maggie McKenzie and Robert Fontanetta. Fred and I provided the scientific expertise and my university paid for the duplication of the questionnaires. The only other funding we had was from patients, who supported the study with nickels and dimes. In all the years since, I’ve never seen any indication that he believes ME/CFS is anything but a physical disease. He does try to help patients develop coping skills. Nothing wrong with that. Also Leonard Jason and recently Lily Cho have been quoting from our 10+ papers. So that study had some use.
As a psychologist, Fred knows more about statistics than I do as an abstract algebraist. He also knows when to seek help when out of his expertise. He called me once when he was on a committee evaluating NIH grants to ask a general question. I was happy to provide the answer, which helped in the evaluation of the grant.
As to who put him in charge of IACFS–we members did! We voted for him just as we voted for Nancy Klimas when she was willing to be President. It is a thankless job and we are lucky Fred is willing to do it.
The big problem with reviewers for grants is for the SRO to identify possible reviewers. I don’t know how this works for NIH, but for NSF in mathematics as well as for peer-review for mathematical papers in journals, a lot of the names are proposed by others in the network of past reviewers as well as people the person choosing potential reviewers knows. The key is to get names into this network of potential reviewers and before the SRO or editor of the journal. In mathematics, to increase the number of female reviewers, we started compiling lists of potentials through the Association for Women in Mathematics as well as the American Mathematical Society. So maybe that is the approach we need to take with NIH. We need to compile lists of potential reviewers who are ME/CFS experts with an outside organization (Solve ME/CFS, perhaps?), who can make the list available to the SRO.
All of you need to understand that people are not paid for reviewing NIH grant proposals nor for doing peer-review of articles for journals. All of this is volunteer work that takes a great deal of time and effort. So some of the problem is that some people won’t volunteer for the work–or don’t have time to travel to Washington DC to sit on an NIH panel.
Finally, I think the change in composition for the SEPs has to do with the change in the types of grants awarded in ME/CFS. Originally, the psych people dominated because the proposals submitted were mostly psych oriented. That was self-perpetuating–the more psych grants awarded, the more psych people submitted grant proposals and the more psych people were assigned to the SEPs to evaluate those proposals. Once we broke through this and started to get more physiologically oriented ME-CFS proposals, the SEP composition changed. The SRO does try to get a SEP panel of experts that reflects the type of grants proposed. So, if many of the ME/CFS grant proposals are concerning immunology, the SRO will try to get immunologists on the SEP. It is not just the ME/CFS part.
Thank you very much, Jenny, for doing all of this work! Knowledge is power, I always say. Now we have facts, we have a better chance of doing something about the problem.
Thank you for sharing these comments. One thing you said actually increased my concerns about Dr. Friedberg. You said, “He also knows when to seek help when out of his expertise,” and I agree that is a good quality in general.
But then you said: “He called me once when he was on a committee evaluating NIH grants to ask a general question. I was happy to provide the answer, which helped in the evaluation of the grant.” I interpret this to mean that Dr. Friedberg did not have the expertise to evaluate a grant (or aspect of a grant) and so he called you for help. Hopefully that did not breach the confidentiality of grant review, since you said it was a “general” question. However, this is an example of a time when he did not have the expertise needed for the grant review. I wonder how many times he has needed such assistance.
Regarding reviewer selection by the SRO, you are correct that there is a network or database of reviewers that the SRO can draw from, as well as searching the literature. I believe that SMCI has submitted, or offered to submit, names of potential reviewers at some point in the past.
Your hard work will help to educate those who don’t really understand the PR & politics around funding research. Thank you.
Hi Jenny,
Thanks for getting us the facts on what is going on at the NIH. The work you are doing is very important in shedding light on what the NIH actually does as opposed to what they say they are doing or will do. I was very disappointed that the CFIDS association never saw fit to monitor the NIH in such a systematic way over time. Thankfully you have filled the void.
Did you every get a feeling for how these numbers compared to SEPs for other diseases? If there is a SEP for the flu, or stomach cancer, or whatever, what percentage of reviewers have published in that disease? Would be interesting.
This is an excellent question! Most of the SEPs meet one time only and then dissolve. I would have to look at the dozen or so ongoing SEPs to see if any are disease specific.
Thanks for this excellent analysis Jennie. You clearly did a tremendous amount of work to just get the information and then analyze it.
One question I had is whether ME/CFS research would be better served by a standing study section with a semi-permanent group of reviewers than by recurring special emphasis panels. On first reading, I thought having an SEP devoted to ME/CFS was a good thing (because the “S” in SEP stands for “special”), but am wondering if that’s true.
For example, there are 14 (!) standing study sections devoted to AIDS and/or HIV research at NIH. Members of these standing committees presumably learn from past studies, helping them better decide what areas of research and which applicants are most likely to lead to progress in the field. With an SEP, it’s possible that such collective knowledge fails to accumulate over time because of the “churning” of panel membership. Would the ME/CFS community be better served by one or more standing study sections as long as they exclude any remaining proponents of psychosomatic explanations of ME/CFS?
The trouble with a standing study section is how to compensate for the small size of the field AND the conflict of interest policy. If Hanson, Davis, Jason, and Younger (literally the first names that came to mind) were members of the study section, then they would have to recuse themselves from any meeting where one of their applications was being reviewed. Even if none of them had an application under submission for a meeting, there might be an application that they did not have the expertise for. A study section would be great in principle, but the challenge is how to staff it over time.