The Statement of Work

Posted on September 30, 2013 by Jennie Spotila

I have obtained a copy of the Statement of Work (SOW) for the Institute of Medicine study on clinical diagnostic criteria for ME/CFS. I am making that document available to you in its entirety through this link. I am also offering my interpretation of some of the most significant provisions of that SOW in this post.

But first, I want to state up front how I got this document and what I’ve done with it. There have been a multitude of accusations and allegations swirling in the ME/CFS advocacy community over the last week, and I figure that someone somewhere will question my involvement, so I’ll just get this out of the way right now. I requested the SOW from Dr. Nancy Lee by email on September 24th. Dr. Lee’s office responded on September 25th with a brief variation on the original announcement. On September 27th, I responded to Dr. Lee’s office pointing out that I had requested a copy of the actual SOW document, and I attached a copy of the SOW from the IOM’s study on treatments for Gulf War Illness as an example. I also filed an expedited FOIA request for the document. Finally, at 6:20 pm on September 30th, Dr. Lee’s office sent me the Statement of Work document I link to above and that I discuss in this post. Simultaneously with publishing this blog post, I am also sending an email to a number of advocates giving them the links to the document and this post. Let me be clear: this post contains my opinions and mine alone. I did not share or discuss the SOW with anyone (except my husband) before publishing this post.

We have a number of lawyers, doctors and other professionals in the advocacy community. I hope they will examine the SOW closely and offer their own analysis. I also hope that our expert clinicians and researchers (including the 35 experts who signed the Open Letter) will also scrutinize the document. We need open dialogue among all the factions in the advocacy and scientific community. If there is any hope of salvaging the IOM process and shaping it to meet the needs of ME/CFS patients and their families, then everyone needs to share information and communicate with one another. We won’t all agree, but we can stop slitting each others’ throats.

On to my analysis – again, the full document can be viewed here. I want to focus your attention on five issues: what disease is being studied; the requirements for involving experts; the requirements for stakeholder input; the project timeline; and whether I think this whole thing is good or bad.

This SOW is seeking a single clinical definition for what could be multiple disorders.

The first page of the SOW states, “HHS will request that the IOM develop consensus clinical diagnostic criteria for this disorder.” (emphasis added) The term “ME/CFS” is used throughout the document and is defined on page 2 as follows:

For the purposes of this document ME/CFS shall be used to refer to Myalgic Encephalomyelitis (ME), Chronic Fatigue Syndrome (CFS), Chronic Fatigue and Immune Dysfunction Syndrome (CFIDS), Neuroendocrine Immune Disorder, and other terminologies in use for this illness. (emphasis added)

I submit to you that this statement tells us a great deal. ME and CFS are not two separate diseases in the view of HHS. This is a value judgment that tells IOM that all the entities referred to by those names are really the same thing. Furthermore, all the entities described in the case definitions listed on page two, and the entity(ies) being studied in the CDC multi-site study, are all the same thing. This does not preclude the IOM committee from reaching a different conclusion, but this assumption is a pretty powerful frame for the scope of work.

My experts may not be your experts.

The biggest bone of contention regarding use of the IOM is the open question of how many ME/CFS experts will be involved in the committee. Most advocates, including me, tried to stop this contract because we feared that experts would not comprise the majority of the committee. The CFIDS Association position announced on September 23rd did not oppose the IOM contract as long as the new clinical definition was created by ME/CFS experts. Now that we have the SOW, we can make a better guess about how this will pan out. On page two, the SOW states that the committee shall have expertise in:

  • the pathophysiology, spectrum of disease, and clinical care of ME/CFS;
  • neurology;
  • immunology;
  • pain;
  • rheumatology;
  • infectious disease;
  • cardiology;
  • endocrinology;
  • primary care, nursing, and other healthcare fields;
  • health education;
  • and the patient/family perspective.

I listed the categories on separate lines based on the placement of semi-colons, which is how the sentence would be understood grammatically. I would also like to point out that this list is different from the description of necessary expertise in the HHS announcement about the contract from September 23rd. Missing is the earlier reference to an expert in behavioral health, but the inclusion of “other healthcare fields” could easily be used as justification for including a psychologist or other behavioral expert. Also missing is expertise in epidemiology and developing clinical case definitions. Finally, I have to point out one additional qualification in the SOW: “Individuals with expertise in other multi-symptom, complex disorders or diseases may also be included on the committee.”

The SOW does not define “expertise,” as in how much experience is needed to qualify as an expert in ME/CFS. In their position issued last week, PANDORA said, “we define an ME/CFS clinical expert as a doctor who has primarily provided clinical care to ME/CFS patients for at least 10 years. We define an ME/CFS research expert as someone who primarily researches ME/CFS, particularly ME/CFS researchers who have published on biomarkers and definitions.” (emphasis in original; page 3 of Position Statement on ME/CFS Criteria). This seems like a logical definition to me, but no such definition is included in the SOW.

Another gap is the distribution of expertise among committee members. The SOW specifies that the expert committee should include 12 to 16 people (the HHS announcement said approximately 15). There is no minimum number of experts for any of the categories, including pathophysiology and clinical care of ME/CFS or the patient/family perspective. A literal reading of the SOW does not even preclude one person from claiming both of those areas of expertise. Not only does the SOW not require that a majority of the committee be subject matter experts, it doesn’t even require more than one of them. And I am certain that I am not the only advocate who finds the reference to “expertise in other multi-symptom, complex disorders” to be ominous.

Stakeholder input is mentioned but not defined.

The SOW includes a requirement for stakeholder input, but a careful reading shows that the requirement is not very specific and could be meaningless. The first topic area of the task described on page two states, “Conduct a study to identify the evidence for various diagnostic clinical criteria of ME/CFS using a process with stakeholder input, including practicing clinicians and patients” (emphasis added). On page three, the SOW describes that process with a bit more detail:

During one or more public meetings, the Committee shall consider the testimony of ME/CFS patients, caregivers, and advocacy groups in addition to relevant subject-matter experts such as clinicians, researchers, service providers, and public health officials.

Sound good? Let’s parse this sentence. First, it specifies “one or more public meetings.” That means only one public meeting is required. There are no requirements as to the duration of that meeting or meetings. By comparison, the public meeting for the IOM’s Gulf War Illness definition study lasted five hours. One hour was devoted to discussion from five patients. Six physicians/researchers had a combined 90 minutes to speak. Open public comment was limited to 30 minutes. I have found no evidence that IOM has employed a comment docket for that panel, like FDA did for the April ME/CFS meeting. Under the terms of this ME/CFS SOW, a five hour public meeting with a single hour dedicated to patients would be sufficient. I don’t know whether the IOM will do a better job in our case, but the SOW does not require it.

There are many different groups included in the SOW “stakeholder input.” Testimony will be considered from patients, caregivers, advocacy groups, clinicians, researchers, service providers AND public health officials. Seven categories, each with very different skill sets and goals. The FDA April 25th meeting spent most of the four hours hearing from patients and caregivers, with a few groups and clinicians sprinkled in. Furthermore, the April 25th meeting posed a series of very specific questions to focus discussion on what was most relevant to FDA, and also used a facilitation format to identify high priority common themes.

Finally, this part of the SOW says the committee shall “consider” the stakeholder testimony. It does not tell the committee HOW to consider the input or how much significance and weight it deserves. There is no stated goal for considering that input or answering stakeholder concerns. The SOW is intentionally broad, so the committee will have complete discretion to disregard all the input it receives at one extreme or rely on that input to shape the entire outcome at the other extreme.

There is a timeline but no start date.

Page three of the SOW lists a draft timeline of the project stages, but provides no start date for the clock to begin ticking. If we assume that last week’s HHS announcement is accurate, then the start date was some time in September.  In that case, the 18 month timeline for this project will conclude by March 31, 2015 (that is also the deadline noted on page four of the SOW). You can see the full timeline in the SOW document, but I want to point out a couple important dates that I calculated by assuming work began at the end of September 2013.

  • By December 2013: “Seek committee nominations, assemble committee, begin background research; contact stakeholders” That means nominations to the committee will be due very soon, and we better get ready for that. There is no explanation of how nominations will be collected or what is meant by “contact stakeholders,” including which groups are included or the purpose(s) of the contact.
  • January -March 2014: “Hold first committee meeting; plan workshops – identify topics and potential speakers; identify information needs; develop plans for receiving stakeholder input” The SOW does not explain what the “workshops” will be, and obviously there is no stakeholder input by this point because plans for collecting it are still being formulated.
  • March-July 2014: “Hold second and third committee meetings with workshops; develop report outline and workplan; hold working group conference calls; seek and receive input from stakeholders” Are the workshops for the committee or for the public? Notice that the report is being outlined at the same time as stakeholder input is being received.
  • August-December 2014: “Hold fourth and fifth committee meetings; draft and revise the report; draft recommendations” Presumably stakeholder input has been concluded. Hopefully it will be considered as the recommendations are drafted. Stakeholder input is not mentioned in the timeline after July 2014.

Good Or Bad?

It is important to understand the process that the IOM typically uses in its studies in order to assess whether this SOW is good or bad for us. Detailed information describing that process is available on the IOM website. I highly recommend reviewing that information. A few things stand out to me:

  • The sponsor of a study has no control over the study once the “statement of task” and budget are finalized. I have not been able to determine if the SOW I analyze here is the same as the finalized statement of task referred to by IOM.
  • Panelists serve without pay.
  • Input is gathered in public meetings, but committee deliberations are held in private.
  • Select criteria for committees: “The committee must include experts with the specific expertise and experience needed to address the study’s statement of task. . . . Having the right expertise is not sufficient for success. It is also essential to evaluate the overall composition of the committee in terms of different experiences and perspectives.”

At the risk of repeating myself, I will say again that the caliber and prominence of the Institute of Medicine is well-deserved. The conclusions published by the IOM have an extraordinary influence over whatever topic is at issue. My quarrel is with the study defined in the SOW, not with the IOM itself.

While some people may trust that the IOM will select the “right” panel for the ME/CFS clinical case definition study, it is my opinion that this is based to some degree on hope or optimism. There is nothing in the SOW that requires the committee to have a majority comprised of ME/CFS experts. It is entirely possible that IOM will appoint more experts than non-experts, but there is no proof that this will be the case. It could just as easily swing the other way. For people who insist this committee will have the right make up, please show me the data that supports this certainty.

I am very afraid that the IOM will appoint more non-experts than experts to the committee. I can’t prove that I’m right, but the SOW and the IOM process leaves holes big enough to drive a truck through. I am also deeply concerned that the extent of stakeholder input is left to the discretion of the committee, and it’s possible that there will be little opportunity to for us participate. Again, I hope I’m wrong but the SOW does not mandate much stakeholder participation. I also do not have much hope that we can force the cancellation of this study. The government is more than preoccupied with the shutdown and budget woes, and there is scant political will for Congress to act on our behalf.

So for my part, I am focused on two things regarding the IOM: participation in suggesting nominees for the committee, and closely tracking this study to maximize transparency and accountability. I want to hear more from the 35 experts who signed the open letter to Secretary Sebelius. I want to hear more from the experts who did not sign the letter, for any reason. And I want to hear more from my fellow advocates. Pick apart this SOW. Let’s crowdsource analysis and action.

 

Posted in Advocacy, Commentary | Tagged , , , , , | 52 Comments

Trust

Posted on September 26, 2013 by Jennie Spotila

Can you trust someone if you think they don’t respect you? There is an abundance of mistrust and disrespect among all participants in the ME/CFS landscape. I think it’s important to talk about because these attitudes have a significant impact on how we make policy decisions and engage with one another. The current conflagration over the Institute of Medicine case definition study is a direct result of the distrustful climate, and we have to see it for what it is if we have any hope of fixing it.

The majority of my fellow advocates do not trust the government and do not feel respected by federal employees. We have a long list of reasons and experiences to support this view. I am going to be very specific about the incidents I have observed and can document. I’m not doing this to stir people up or aggravate what is already a highly charged atmosphere. This information is key to understanding the basis for mistrust and suspicion. I know that non-patients read this blog, and they need to hear specifics too.

Setting the Stage

Problems between ME/CFS advocates and HHS have persisted for decades. There is a heavy legacy that poisons every interaction. I have personally witnessed this. Here are a few incidents that have contributed to my own mistrust of HHS, but I have heard of many many more.

  • When I requested documentation on CFSAC nominations, I was told that the Office of Women’s Health did not keep those records. This was not true, and those documents were produced after I filed a FOIA request.
  • I have another pending FOIA request for additional nomination documents. In a partial response, I was told that all the documents had already been released to me. This is not true, and I have two documents to prove it. I am still waiting for a response to my challenge.
  • There is a well-documented example of the CFSAC violating the Federal Advisory Committee Act. I can document other incidents as well. These seem like squabbles over rules and technicalities, but in my opinion it is indicative of a casual or cavalier attitude of disrespect, an attitude that doesn’t see us as being very important.
  • We have unresolved allegations on the record that Dr. Nancy Lee threatened to remove one or more CFSAC members for expressing their views.
  • On more than one occasion, my polite requests for information have not even been acknowledged by the individual or agency I addressed.
  • On more than one occasion, I have witnessed a federal employee complaining to one advocate about other advocates. This is unprofessional and disrespectful.

After Assistant Secretary Koh said that CFSAC was the path for stakeholders to engage with HHS, I joined other advocates in trying to reach across the divide to have respectful dialogue about our concerns. But in the last year, I have seen that divide grow wider and deeper as our concerns were dismissed or rebuffed again and again.

But the mistrust and disrespect flows both ways. Federal employees have indicated that they feel under siege or targeted by ME/CFS advocates. Their negative experiences with advocates must contribute to their views of the community, and I would not be at all surprised if they mistrust us. I can document a few examples, but I’ve heard of many many more.

  • A federal employee complained to me about the volume of emails that patients send HHS, and also complained that some of those emails are nasty. This was indicative of that employee’s view of the advocacy community, both in complaining to me about my fellow advocates but also the employee’s negative experience of interacting with advocates.
  • Dr. Nancy Lee said at the CFSAC meeting on June 14, 2012: ““I just want everyone to know that all of the people up here, both our non-federal committee members and our federal committee members, are committed to this topic [ME/CFS].  . . . We are very committed to this or we wouldn’t be here.” I noted at the time that this comment seemed oddly defensive, coming right after a public comment segment and the lunch break.
  • Dr. Susan Maier (NIH) said at the CFSAC meeting on May 23, 2013: “I have had people laugh at me; I’ve had people accuse me of giving them information that is wrong; I’ve had people tell me that I’m not helping them.  . . .  I’m tired of people telling me that I’m wrong and being misquoted.”

These incidents are just the tip of the iceberg. I know my fellow advocates can cite dozens upon dozens of examples of things HHS, the agencies, and individual employees have done or said that demonstrated disrespect for ME/CFS patients. These incidents have directly contributed to the pervasive mistrust of HHS. On the other hand, we have to acknowledge that those individual employees are human. Their experiences may have contributed to them distrusting us.

And Then There’s IOM

Into this existing atmosphere of mistrust and wariness, the Institute of Medicine project and how it was handled by HHS was the equivalent of dropping a match on a lake of gasoline. This is total conjecture on my part, but I suspect that the way HHS behaved in the last month was motivated in part by their distrust of the advocacy community. But by behaving in this manner, HHS demonstrated incredible disrespect for the community and guaranteed the explosive reaction they got from us. Look at this timeline summary:

  • Spring 2013: HHS develops the idea of tasking the Institute of Medicine to create a new clinical case definition for ME/CFS. We don’t know who was involved, when it started, or how it evolved. But we do know that not a single word was said to the public or the advocacy community about this. Several sources have told me that HHS made a conscious decision not to consult us.
  • August 27, 2013: The sole source solicitation notice is published on FedBizOpps. No announcement is made by the Office of the Assistant Secretary or anyone else. An ME/CFS advocate finds the notice and begins spreading the word.
  • August 27-September 2, 2013: The advocacy community gets organized and begins protesting the project via email and other means.
  • September 2, 2013: The intent to contract with the IOM is announced via the CFSAC listserv.
  • September 4, 2013: The sole source solicitation is canceled due to “all of the concern from the public.” A second update states that “HHS will continue to explore mechanisms to accomplish this work,” but no further details are offered. The email campaign continues.
  • September 12, 2013: CFSAC announces via its listserv that it continues to work on a contract with IOM and that information will be released AFTER it is finalized. The issue will be on the agenda of the November CFSAC meeting.
  • September 4-September 17, 2013: Multiple advocates, including myself, try to engage with the Office of Women’s Health and other people at HHS to get any information about the IOM contract, and attempt to offer constructive input. None of us are successful. I am not aware of anyone at HHS reaching out to the ME/CFS community in any way.
  • September 17, 2013: We learn that the contract will be signed before September 30th.
  • September 23, 2013: We learn that the IOM study has been authorized, and will begin this month. Apart from the listserv announcement, no further information is offered. My inquiry for additional information has been ignored.
  • September 23, 2013: We learn that 35 clinicians and researchers sent an open letter to Secretary Sebelius urging her to adopt the (nicknamed) Canadian Consensus Criteria and abandon the IOM effort. To my knowledge, there has been no response from HHS.

Walking this timeline as I have, and knowing the prevalence of mistrust and disrespect going into this situation, I still find it shocking. The intentional failure to engage the advocacy community before or during this controversy is astonishing. The piss-poor communications via the listserv made everything worse. I feel like we are on an elevator, and HHS has cut the cables. We are plummeting down and I have no trust that there is a soft landing waiting for me.

Every single advocate I know asked themselves: why did they withdraw the contract and then pursue it in complete secrecy? How can anyone trust that this IOM process will be good for us? If their intentions are good, and they truly plan to use experts to create the new case definition, then why in the world didn’t they say so at the beginning? Why didn’t they show us the basic respect of telling us what was going on? At the same time, I imagine that HHS employees were thinking something like “Don’t tell them anything because they’ll freak out and cause trouble.” But if that is the case, then their distrust simply guaranteed that they would elicit the response they feared. And I have to believe that if the relationship between advocates and HHS had not deteriorated to the degree it has in 2012-2013, then perhaps our reaction to this current debacle would not have been the same.

We Are Where We Are

So here we are. The IOM study is authorized, and I presume it will begin shortly. I must note here that we still have no definitive information about it beyond the listserv announcement. Thirty-five experts are now on record saying that no further study is needed to adopt the CCC. Advocates continue to press this point and are requesting that the IOM study be canceled. This document gives a superb summary of why.

The distrust is truly toxic to finding the right solutions. Advocates don’t trust HHS, and can point to a multitude of reasons why. HHS (or at least a number of its employees) don’t trust the ME/CFS advocates, and may even actively dislike or disrespect us, and can point to their own multitude of reasons why.

Maybe HHS will listen to the thirty-five experts – or Congress – and adopt the CCC. Maybe more experts will join the call. Maybe the IOM study will go forward in a way that relies on experts and adopts the CCC or something similar. Maybe the IOM study will go forward and result in a definition that lumps us all under a giant nonspecific umbrella of fatigue.

In my conversations with advocates, it seems like one’s degree of trust in the government dictates which Maybe you believe is true. Since no one at HHS is talking to me – despite multiple attempts on my part – I can’t speculate about how they see the Maybes.

What I do know with absolute certainty is that our entire environment has been poisoned by mistrust, fear and disrespect. The actions of HHS in the last month have shocked even the most moderate of advocates. We need a sincere and meaningful demonstration by HHS that they truly understand our concerns. If we cannot stop the IOM contract, then we will expect and demand that HHS adhere to everything they promised in the listserv announcement, and more.

As I have shown, there is more than enough distrust, suspicion, fear and misgivings to go around. All sides have contributed to spilling that gasoline. But HHS is the one who lit the match and threw it on the spill. HHS is the one who needs to put out the fire. If someone at HHS steps up to do that, advocates have to be willing to let the fire go out as long as there is demonstrable transparency, accountability and commitment to get this right. Maybe that will allow us to step back from the terrible precipice on which we stand.

 

Posted in Commentary | Tagged , , , , , , , | 31 Comments

The Experts

Posted on September 24, 2013 by Jennie Spotila

Developments came fast and furious last night. First, the CFIDS Association announced that it would not oppose the IOM contract, but would actively call for the IOM to choose a panel that matches the CFS Advisory Committee recommendation from October 2012. Within a few hours, CFSAC announced on its listserv that the IOM contract would go forward and provided some additional information. And then late last night, we learned that thirty-five ME/CFS experts sent an open letter to Secretary Sebelius urging her to adopt the Canadian Consensus Criteria.

This letter starts with a bang: “[W]e have reached a consensus on adopting the 2003 Canadian Consensus Criteria (CCC) as the case definition for this disease.” Many of the people who signed the letter have been using the CCC for some time, but this is the first time to my knowledge that a group of experts has come together to support adoption of CCC in writing.

The letter goes on to urge HHS to adopt the CCC as the single case definition for all Department activities, both research and clinical uses. Then the next big statement:

[W]e strongly urge you to abandon efforts to reach out to groups such as the Institute of Medicine (IOM) that lack the needed expertise to develop “clinical diagnostic criteria” for ME/CFS. Since the expert ME/CFS scientific and medical community has developed and adopted a case definition for research and clinical purposes, this effort is unnecessary  . . . [and] threatens to move ME/CFS science backward by engaging non-experts in the development of a case definition for a complex disease about which they are not knowledgeable. (emphasis added)

That is bold. The IOM contract has just been signed, but these experts are saying: No. I do not know what efforts the signatories will make to push this message and perhaps urge that the IOM contract be rescinded. But this is the strongest advocacy statement that we’ve seen from a group of research and clinical experts in quite some time, especially in the form of an open letter to Secretary Sebelius.

There are some big names on this letter. Ten of the signatories are current or former members of CFSAC. Four of the seven clinicians participating in CDC’s multisite study have signed on to this letter. Four are co-authors of the CCC and twelve are co-authors of the ME-International Consensus Criteria. You can see a complete list at the end of this post.

But there are also some big names missing from this letter. Three of the clinicians participating in the CDC multisite study are not listed, and other giants in the field like Dr. Anthony Komaroff have not signed. I do not know what process this group followed in drafting this consensus letter, so we cannot assume that someone not listed actually opposes this position. Perhaps more experts will sign on, now that this is public and gaining momentum.

The big question is what next? How will HHS reconcile investing in the IOM contract in the face of this open consensus letter? Will others join? Will the signatories actively advocate for the rescinding of the IOM contract? Is there sufficient consensus across the field? What position with the CFIDS Association and other organizations adopt in light of this letter? Will the IACFS/ME support this effort? (two of the signatories are on the IACFS/ME board)

The good news is that things are moving. Our experts are coming together in consensus, and speaking out. That’s fantastic! But we’re still in a pickle, aren’t we. On the one hand, HHS signed the IOM contract over the vociferous objections of many in the ME/CFS community. On the other hand, many prominent researchers have now joined together and told HHS that they should abandon the contract and move forward with the CCC. This is in line with the CFSAC October 2012 recommendation as well – achieve consensus of the experts on a case definition starting with the CCC.

The question I’m asking myself today is: IS ANYBODY LISTENING? There are so many moving parts, many at cross purposes. How do we pull it all together to move forward productively, based on the science? We cannot allow anyone to lose sight of the fact that the ultimate goal must be better research and clinical care for patients.

 

 Signatories CFSAC member CCC author ICC author CDC Multisite
Dharam Ablashi No No No No
Lucinda Bateman 2005-2010 No Yes Yes
David Bell 2003-2005 No Yes No
Gordon Broderick No No Yes No
Paul Cheney No No No No
John Chia No No Yes No
Kenny De Meirleir No Yes Yes No
Derek Enlander No No No No
Mary Ann Fletcher 2012-2016 No No No
Ronald Glaser 2007-2011 No No No
Maureen Hanson No No No No
Leonard Jason 2007-2012 No No No
Nancy Klimas 2007-2012 Yes Yes Yes
Gudrun Lange No No No No
Martin Lerner No Yes No No
Susan Levine 2010-2014 No No No
Alan Light No No Yes No
Kathleen Light No No No No
Peter Medveczky No No No No
Judy Mikovits No No Yes No
Jose Montoya No No No No
James Oleske 2005-2010 No No No
Martin Pall No No Yes No
Daniel Peterson No Yes No Yes
Richard Podell No No No Yes
Irma Rey No No No No
Chris Snell 2007-2012 No No No
Connie Sol No No No No
Staci Stevens 2003-2006 No Yes No
Rosemary Underhill No No No No
Marshall Williams No No No No
Birgitta Evengard No No No No
Sonya Marshall-Gradisnik No No Yes No
Charles Shepherd No No No No
Rosamund Vallings No No Yes No

 

Posted in Advocacy, Research | Tagged , , , , , , , | 31 Comments

Contract Signed

Posted on September 23, 2013 by Jennie Spotila

This message was sent out via the CFS Advisory Committee listserv this evening:

We are pleased to announce that the Institute of Medicine (IOM) will begin conducting a study on Diagnostic Criteria for Myalgic Encephalomyelitis/Chronic Fatigue Syndrome this month.  This study grew out of the 2012 CFSAC recommendation to the Secretary to convene a workshop to “reach a consensus for a case definition useful for research, diagnosis and treatment of ME/CFS.”  Because the use of and audience for case definitions for research and clinical care are very different, HHS decided that separate processes were needed to develop them.  The NIH is convening an Evidence-based Methodology Workshop process that will consider case definitions appropriate for ME/CFS research and the IOM will address the clinical diagnostic portion of the recommendation.

The Office on Women’s Health/HHS is co-sponsoring the IOM study with several agencies that participate in CFSAC.  A group of voting and non-voting members of the committee participated in the development of the Statement of Work.

The IOM has agreed to perform the following tasks over the next 18 months:

·         Conduct a study to identify the evidence for various diagnostic clinical criteria of ME/CFS using a process with stakeholder input, including practicing clinicians and patients;

·         Develop evidence-based clinical diagnostic criteria for ME/CFS for use by clinicians, using a consensus-building methodology;

·         Recommend whether new terminology for ME/CFS should be adopted;

·         Develop an outreach strategy to disseminate the definition nationwide to health professionals.

To accomplish these tasks, the IOM has also agreed that:

·         the IOM committee will include approximately 15 members with expertise in the following areas: epidemiology; clinical medicine/primary care and other health care fields, particularly with expertise in ME/CFS, including neurology, rheumatology, immunology, pain, infectious disease, behavioral health, cardiology, endocrinology; and scientists and physicians with experience in developing clinical case definitions.  The IOM will ask interested parties, including expert clinicians, researchers and patient advocates, to make suggestions for nominees to the committee.

·         the IOM, in addition to the committee meetings, will sponsor open meetings to enlist the comments and concerns of patients, family members and other caretakers, health educators, health care professionals, and advocacy groups.  During these meetings, an open phone line and email address will be available to those who cannot attend in person.

·         as the committee reviews the literature, efforts that have already been completed on this topic area will be considered, including the 2003 ME/CFS Canadian Consensus Definition, the 2007 NICE Clinical Guidelines for CFS/ME, the 2010 Revised Canadian ME/CFS definition, the 2011 ME International Consensus Criteria, and data from the ongoing CDC multi-site clinical study of CFS. In an effort to minimize overlap and maximize synergy, the committee will seek input from the NIH Evidence-based Methodology Workshop for ME/CFS.

·         the committee will distinguish between disease subgroups, develop a plan for updating the new criteria, and make recommendations for its implementation. Any recommendations made by the committee will consider unique diagnostic issues facing people with ME/CFS, specifically related to: gender, across the lifespan, and specific subgroups with substantial disability.

HHS is proud to be making this investment in ME/CFS. We have heard the advocates’ concerns and trust that many have been addressed by the information provided in this email.  It is our hope that a widely accepted clinical definition and a clear distinction from case definitions for clinical trials and research will aid in advancing clinical care, drug development, and basic translational research for ME/CFS.

________________________________________________________

The CFSAC Support Team

I hope to provide more detail and analysis tomorrow.

 

Posted in Advocacy | Tagged , , , , , | 7 Comments

Lips Are Sealed

Posted on September 21, 2013 by Jennie Spotila

Whatever happened to that investigation we requested after the last CFS Advisory Committee meeting? Good question.

You may recall that at the May 2013 CFSAC meeting, voting member Eileen Holderman alleged that she and two other members had been intimidated and threatened with expulsion from the Committee for expressing their views. On June 12th, I joined forty-five other organizations and individuals in requesting that the General Counsel of the Department of Health and Human Services undertake an investigation of those allegations.

On June 21st, the General Counsel referred the matter to the Office of the Assistant Secretary, which oversees the CFSAC. Since then? We haven’t heard word one. We’ve received no updates and no responses to inquiries. More than 90 days have passed since our original request, and less than 60 days remain until the November CFSAC meeting.

The importance of this really can’t be overstated. Two members are on record saying that they have been intimidated for expressing their views. The public can have no confidence in any deliberations or recommendations made by a federal advisory committee where members are being intimidated by the Designated Federal Officer.

As I have said from the very beginning, I do not know if the allegations are true. That’s why we requested an investigation: to find out. The best thing the Assistant Secretary can do is conduct a full and impartial investigation, and report his findings to the public. We have no way of knowing if that process is underway because no one is responding to our requests for information. All I need to hear is “we’re looking into it and will issue a report by such and such date.” I don’t think this is an unreasonable expectation.

Yesterday, Mary Dimmock made yet another request for information. This is the full text of her letter to Assistant Secretary Koh:

 

Dear Assistant Secretary Koh,

It has been more than 90 days since I asked General Counsel William Schultz to investigate whether the CFS Advisory Committee’s Designated Federal Officer Dr. Nancy Lee had intimated voting members of the Committee. Forty-six organizations and individuals joined me in that request, and expressed deep concern after the allegations of intimidation were made on the record at the May 23, 2013 CFSAC meeting.

General Counsel Shultz referred the matter to your office on or about June 21, 2013. I wrote to you on August 22, 2013, requesting an update on the status of the investigation. Neither you nor anyone from your office has responded to my inquiry.

The CFS Advisory Committee is scheduled to meet on November 12 and 13, 2013. If the allegations of intimidation are true, then the integrity and function of the Committee is materially impaired. No one can have confidence in an advisory committee that has been compromised by federal intimidation directed at minority or dissenting viewpoints. Failure to objectively investigate or take necessary action will similarly undermine public confidence in the transparency and accountability of the Committee and the Department.

Therefore, I once again request a written update from your office on the status of this investigation. If the investigation is completed, I request a written summary of the findings and any action that has been taken. Given the time sensitive nature of this investigation, I ask that you respond to me in writing by October 15, 2013. Thank you for your prompt attention to this matter.

Now we wait. Again.

 

Posted in Advocacy | Tagged , , , , , , | 12 Comments

Pedal to the Metal

Posted on September 20, 2013 by Jennie Spotila

figur hat akku leerI feel like an engine that has been gunning too long. I’m revving revving revving and I can smell something burning and a gear is about to blow. The rest of my life is being neglected. I haven’t been able to cook in weeks. I’m not sleeping well. The house and paperwork and all those annoying daily life things are completely trashed. That’s pretty much the effect of all the advocacy work going on, and I am definitely not the only patient struggling mightily. And I’m one of the LUCKY ones who has a little energy to spare to pour into it. There are patients who can’t sit up in bed without assistance, and they’re just struggling to survive.

There’s lots of other news to talk about and analyze, including the FDA’s report from the April 25th meeting. But I don’t have the bandwidth to write those posts, and most patients don’t have the spare energy to read them.

We cannot get distracted from our mission! If we want to stop the IOM contract for a new ME/CFS case definition, then we have to make that our only focus through September 30th! We have ten days, friends. Ten days to stop a contract that could hurt us for decades to come. This is no time to waver or stagger or stop.

Keep emailing Secretary Sebelius: use my template or adapt it for yourself.

Ask your healthy friends and families to email the Secretary: again, I have a template but feel free to adapt it.

Target Congress and the President: great templates from Liz Willow.

Keep emailing! I know it’s tedious and hard to do every day. Please do what you can. This is a critical moment in our struggle. Anything you can do to help will make a difference.

Posted in Advocacy, Occupying | Tagged , , , , , , | 5 Comments

To The Healthies

Posted on September 19, 2013 by Jennie Spotila

We need all the help we can get right now! HHS is bound and determined to sign the IOM contract by September 30th. Advocates are working like mad to stop it. So in addition to sending my own emails to HHS today, I am also emailing my healthy friends and family to ask for their help.

Here’s the email template I am asking my healthies to use. Feel free to adapt to your own needs. Make sure that you add the right information to the first paragraph and signature line. BUT PLEASE reach out to your family, friends, colleagues, and neighbors! Ask them to send an email to support you!

 

TO: Kathleen.Sebelius@hhs.gov

CC: howard.koh@hhs.gov, txf2@cdc.gov, Tomfrieden@cdc.gov, Marilyn.Tavenner@cms.hhs.gov, margaret.hamburg@fda.hhs.gov, Mary.Wakefield@hrsa.hhs.gov, collinsf@mail.nih.gov, richard.kronick@hhs.gov, cfsac@hhs.gov, MEACTNOW@yahoo.com

Subject: Help My Friend With ME/CFS

Secretary Sebelius,

ME/CFS is a devastating illness that has left my friend <insert name> disabled for nineteen years. There are no treatments approved for her disease. She is homebound, and has lost her career and plans for a family.

I learned this week that your Department is unilaterally pursuing a contract with the Institute of Medicine to redefine ME/CFS, despite the objections of people like my friend and without informing members of your federal advisory committee for the disease.

I do not understand why you would do this. I do not understand why you would ask the Institute of Medicine to create clinical criteria for ME/CFS when the IOM has never done this for another disease. There are many doctors and researchers who are experts on ME/CFS, and it seems to me that you should be asking them to define this disease.

My friend needs your help. ME/CFS should be defined by experts familiar with this complex disease. I ask that you stop this IOM contract. Mistakes are being made, and it will only get worse if non-experts are given power to pronounce judgment on ME/CFS. Please stop this IOM contract. Please act to protect and help people with ME/CFS, and those of us who care about them.

Sincerely,

<sign your name>

Posted in Advocacy | Tagged , , , , , , | 2 Comments

Insult, Meet Injury

Posted on September 17, 2013 by Jennie Spotila

Did you think that your efforts to tell HHS to freeze the IOM contract to create a clinical definition for ME/CFS were successful? Did you have the impression that your government would listen to your concerns?

Join me in mourning the death of my faith in the process.

We have learned that HHS is not only continuing to pursue this contract, but that they have every intention of signing it by the end of the fiscal year on September 30th. This is not fear mongering or over reaction. We have confirmation that this is true. Furthermore, HHS’s stated intention is to share information with the community only after the contract has been signed.

Yes, I know they just canceled a solicitation for such a contract on September 4th, saying, “Because of all of the concern from the public surrounding this potential sole source requisition, we have decided to discontinue this request.” And yes, I know they said on September 12th that “We continue to work on a contract with the Institute of Medicine (IOM) to develop recommendations for clinical diagnostic criteria.  When the contract is finalized, we will provide additional information via the CFSAC listserv and website.  This topic will be included as an agenda item for the November webinar.” That listserv announcement creates the impression that nothing will be finalized before the November meeting if it’s going to be on the agenda, doesn’t it? Nope. That listserv announcement was smoke and mirrors.

There has been no consultation with our community, despite our public objections to their first attempt to do this. Instead, they pulled the first contract when we started to push back, but simply changed tactics and continued to negotiate the contract another way. And we are not the only people being cut out of this process. I have information from multiple sources confirming that the voting members of the CFSAC have not been consulted or informed about the IOM contract at any stage. Not earlier this year when the process began, and not in the last few weeks when it seemed to be blowing up.

I tend to be a reasonable person. I tend to respond to these controversies professionally, giving people the benefit of the doubt. I am not a conspiracy theorist. Today, I am shaking with rage. I cannot come up with a plausible, charitable way to interpret the facts here. They failed to announce the first contract notice, and withdrew it as soon as we objected. They strung us along with we’ll-discuss-in-November. Meanwhile, they failed to consult the people the Secretary has appointed to advise the Department on, among other things, “impact and implications of current and proposed diagnosis and treatment methods for CFS.” (CFSAC Charter, emphasis added). And at the same time, they plan to execute this contract by September 30th so they can obligate the (substantial) funds required out of FY2013 money. Under these circumstances, it is impossible for me to trust that the IOM contract will be structured in my best interest. I pray that I am wrong. I pray that if this case definition process goes forward that it will produce a definition that accurately reflects my disease. But the actions and statements of HHS, particularly the Office of Women’s Health, have given me no confidence in such an outcome.

I ask that you join me in SPEAKING LOUDER until our government hears us and takes appropriate action. Every day, we will be emailing Secretary Sebelius and her key deputies. There are multiple email templates available, including here and here. I’m sharing my email, and feel free to adapt it to your own needs:

 

TO: Kathleen.Sebelius@hhs.gov

CC: howard.koh@hhs.gov, txf2@cdc.gov, Tomfrieden@cdc.gov, Marilyn.Tavenner@cms.hhs.gov, margaret.hamburg@fda.hhs.gov, Mary.Wakefield@hrsa.hhs.gov, collinsf@mail.nih.gov, richard.kronick@hhs.gov, cfsac@hhs.gov, MEACTNOW@yahoo.com

Subject: Stop the IOM Contract to Redefine ME/CFS

Secretary Sebelius,

Your Department, acting through the Office of the Assistant Secretary, is unilaterally pursuing a contract with the Institute of Medicine to redefine ME/CFS, despite the objections of the affected patient community and without informing members of your federal advisory committee for the disease. I have no alternative but to ask that you stop this contract and listen to the subject matter experts on this disease.

OASH recently published a sole source solicitation for the Institute of Medicine to “develop consensus clinical diagnostic criteria for this disorder.” The request was swiftly cancelled after ME/CFS advocates objected to the backroom secrecy, short response time, and the IOM’s patent lack of experience in creating accurate and meaningful case definitions for diseases. Despite those objections, OASH has stated its intention to sign this contract (without publishing a sole source notice) by September 30th.

Your own CFS Advisory Committee recommended that you convene a “stakeholders’ (ME/CFS experts, patients, advocates) workshop in consultation with CFSAC members to reach consensus for a case definition useful for research, diagnosis and treatment of ME/CFS beginning with the 2003 Canadian Consensus Document case definition and its utility for diagnosis and treatment of ME/CFS.” CFS Advisory Committee recommendation, October 2012. Not only has the recommendation been ignored, but OASH has concealed the plans for this IOM contract from the voting members of the Advisory Committee.

Your Department is progressing a contract to the IOM with no assurance that the actual clinical and research ME/CFS experts would be the ones staffing the consensus panel. Your Department has concealed its plans from the public and your own Advisory Committee. Your Department is pursuing this despite the overwhelming objections of the ME/CFS community.

I have no confidence that your Department and its agencies will act to protect me or to ensure that I can receive accurate diagnosis and adequate treatment of my disease. Your agencies and personnel have repeatedly ignored the hallmark symptoms of this disease, disregarded the case definitions authored by ME/CFS experts, and resisted meaningful engagement to address the scientific questions that have plagued this disease for decades. This latest episode has only increased, rather than allayed, my concerns. Your Department has failed to act in good faith at every opportunity in the last month, and I have no choice but to oppose this.

I urge you to consult with the researchers and clinicians who have actually worked on ME/CFS, and not rely on those with no direct experience or whose expertise is limited to overlapping or related conditions. We need transparency, accountability, and direct consultation with the experts, patients and advocates in order to have any hope of moving forward with an accurate case definition for ME/CFS. At present, your Department is only creating barriers to meaningful progress.

I ask that you stop this IOM contract. I ask that you bring all the stakeholders to the table to address this situation as equals. The climate of mistrust and bad dealing that these actions have created will haunt us for years to come, and a case definition that does not reflect the disease will set back research and treatment efforts for decades. None of us can afford for your Department to get this wrong.

Sincerely,

Jennifer Spotila

I will keep you informed as things develop. I hope you will join me in this effort!

 

Posted in Advocacy | Tagged , , , , , , , | 21 Comments

Cut Back or Cut Out?

Posted on September 16, 2013 by Jennie Spotila

Last week, the CFS Advisory Committee announced via its email listserv that the fall meeting will be held on November 12th and 13th. Advocates were shocked to learn that the meeting will be held from 12pm to 5pm each of those days (as opposed to the two full days we usually have), and that the meeting will occur via webinar and not in-person. So is this a cut back from budget necessity, or is it another attempt to cut patients and advocates out of the process?

Sequester and Legalese

According to the email announcement, “Because of budget constraints and the government-wide Sequestration, federal advisory committees have been told to conduct at least some of their meetings as webinars or use other similar formats.” As of this writing, I have been unable to independently confirm such an instruction to advisory committees.

I have not received a response to my email inquiry to the DHHS Committee Management Officer. Furthermore, several HHS advisory committees have not made this move to webinar meetings. Both the National Vaccine Advisory Committee and the Advisory Committee on Human Research Protections are holding in-person meetings this fall. This move to a shorter webinar meeting could very well be an effect of sequestration. All the agencies have had to cut back. But are all committees required to change their format? Or was CFSAC singled out for such a cutback by someone at HHS? We don’t have enough information to say for sure.

It is legal for a federal advisory committee to meet via webinar or other remote communication. However, the meeting must still comply with the requirements of in-person meetings: it must be accessible to the public; the forum must accommodate a reasonable number of members of the public; anyone can file a written statement; and anyone can speak (within agency guidelines). The meeting must be accessible in real time, and the Designated Federal Officer must be continuously present.

Neither the CFSAC Charter nor Bylaws mandate the number of days for meetings. Traditionally, there have been four days of meetings per year. In May 2010, there was a single day meeting, but that was to free up the budget for a full day of science presentations in October (plus a two day meeting). As much as we are used to those four days a year, there is no requirement for it. But what will they cut? Will public comment time be reduced? There never seems to be enough time to cover the two day agenda. What steps will be taken to make discussion more efficient? Even if the budget cut is unavoidable, how will CFSAC ensure a minimum negative impact?

Silencing

The IOM fiasco has created grave concerns in the advocacy community. And CFSAC has fanned the flames with this statement in the listserv announcement:

We continue to work on a contract with the Institute of Medicine (IOM) to develop recommendations for clinical diagnostic criteria.  When the contract is finalized, we will provide additional information via the CFSAC listserv and website.  This topic will be included as an agenda item for the November webinar. (emphasis added)

Now honestly, what the hell does that mean? So far, I have been unsuccessful in my attempts to get more information. Regardless of what I am able/unable to learn, I think it is abundantly clear that the fall CFSAC meeting is going to be controversial and contentious.

First, we have the unresolved allegations of attempts to intimidate voting CFSAC members for expressing their views. To date, there has been no substantive response to advocates’ request for a formal investigation. As we said in June, “Threats or intimidation of voting members for expressing their views, particularly by the DFO, would materially impair the CFSAC’s ability to formulate recommendations to the Secretary.” There is a huge cloud hanging over this meeting as a result. Will we hear the full and honest opinions of the voting members? Or will the webinar format make it easier to silence certain opinions or select individuals? I’ve moderated phone meetings myself, and it is extremely challenging to give everyone an equal chance to speak, especially when there is disagreement. Without a way for members to virtually raise their hands in some way, I fear that it will be difficult for some members to get a fair shot – even if only because some are more outspoken than others.

Second, the prospect of an IOM contract to create a new clinical case definition has galvanized the advocacy community. We were able to force the withdrawal of the original contract, but the issue is clearly not dead. Advocates aren’t sitting around waiting for the fall meeting. Instead, action continues in the effort to ensure that actual experts are the majority in any case definition effort – and more action is planned. But the fall CFSAC meeting is an important forum for the community to express its views, and I have no doubt that we would have done so. HHS must know this too. The move to a webinar meeting format deprives us of the ability to speak through actions such as signs, standing together during public comment, and other silent physical actions. We have other options, and they are being explored. But intentionally or not, HHS has partially silenced us by canceling the in-person meeting. In the current advocacy climate, it is difficult for many people to trust that HHS did not intend this effect.

Not Backing Out

So is the reduction in meeting time and change to webinar format purely an effect of sequester? Or is funding a convenient reason to make changes that attempt to silence voting members or advocates?

From my perspective, it doesn’t matter. We’ll never be able to prove a malicious intent to silence advocates. And even if the change is purely a byproduct of sequester that will affect every advisory committee, the bottom line is the same. It will be more difficult for the committee to accomplish its job in the time allotted, and in a way that ensures equal input from the voting, non-voting and ex officio members. There will be less time and less opportunity for us to provide input and make our views known.

So what should we do? We adjust. We find ways to express our views to HHS and the CFSAC members. If we are once again victims of budget priorities, speak louder. If this is an attempt to silence us, speak louder. Does everyone hear me? SPEAK LOUDER. I am not backing out. You shouldn’t either.

 

Posted in Advocacy, Commentary | Tagged , , , , , , | 9 Comments

Opportunity Lost

Posted on September 10, 2013 by Jennie Spotila

The CDC hosted a conference call for ME/CFS patients and advocates today. The highlight of the call was a presentation from Dr. Ian Lipkin about his pathogen and immunology work in ME/CFS. But we received an important update on the CDC multisite study, and it remains to be seen whether advocates will accept what we were given.

You may recall that at the May 2013 CFSAC meeting, advocates were aghast at Dr. Unger’s statement that the CDC multisite study would not use two-day cardiopulmonary exercise testing (CPET) despite the research showing that this protocol produces evidence of post-exertional malaise, metabolic dysfunction, and is a potential diagnostic marker for ME/CFS. When questioned, Dr. Unger said she had not discussed the protocol with Dr. Chris Snell or Staci Stevens (who created it). This seemed like yet another example of CDC having an opportunity to do good science and intentionally choosing not to do so.

On July 22, 2013, eleven groups and thirty-one individuals sent a letter to CDC requesting, among other things, that the multisite study use the two-day protocol. Here’s what they said:

The two-day CPET regimen known as the Stevens Protocol provides gas exchange and other objective and measurable results “which can’t be faked.” With properly trained personnel in place, this test can be done using technology which has been used in hospitals and other facilities for decades. Having CPET testing performed by trained personnel on subjects involved in the multi-site clinical assessment should be considered a TOP PRIORITY in order to maximize standardized data and take advantage of the opportunity provided by this important CDC-initiated study.

We cannot over-emphasize the importance of measuring and understanding post-exertional malaise (PEM) in this study. PEM is most often the largest obstacle to activities of daily living, gainful employment, exercise, and more. A combination of data from the two-day CPET test and the on-line cognitive test that is already planned will provide the data needed for effective analysis of this debilitating symptom.

Dr. Unger responded in writing on August 30th, but for unknown reasons the advocates did not receive her response until today. Both the letter and Dr. Unger’s comments on the call today explain why CDC has chosen to do one day of maximal effort testing, followed by 48 hours of cognitive testing and symptom measurements. Especially important (and highlighted in the excerpt below) is Dr. Unger’s representation of Dr. Snell’s opinion on the protocol:

To address concerns regarding the cardio-pulmonary exercise testing (CPET) in the second stage of the study, I would like to share additional details, and the rationale that we used to select the one-day maximal exercise test. Our primary objective is to measure the exercise capacity in as many of the enrolled patients as possible using a standardized protocol, and to monitor the post-exertional response for 48 hours with online cognitive testing and visual analogue scales of fatigue, pain, and symptoms. Maximal CPET with one day of testing and 48-hour follow-up of cognition was developed in consultation with Dr. Gudrun Lang (cognition) and Dr. Dane Cook and Connie Sol (exercise). The exercise protocol was discussed also with Dr. Chris Snell. Dr. Snell favors the two-day test because it gives more information, however he believes the one-day maximal CPET will provide useful information. We chose the one-day test so that more patients could be tested. The two-day test would require an additional overnight stay for those patients who travel long distances to attend clinic and excludes those who are most severely affected because of the heavy physical toll. In developing the protocol, we strived to find a balance between testing that would yield meaningful data in the broadest representation without placing an unnecessary burden on the patients.

I immediately asked Dr. Snell if this was an accurate representation of his comments, and he said it was. He commented:

As you know, we believe that the 2 day test provides important metabolic data as well as potential to objectively document fatigue following physical exertion. I do believe, for most patients, a single max test will elicit PEM which should affect the post-test cognitive scores and fatigue scale scores. The CPET data however may not be a true reflection of physiological function post exertion for all patients.

On balance, I am happy that the CDC chose to use a validated protocol for functional assessment that does incorporate objective measures of effort. This is infinitely preferable to dubious “sub-maximal” tests. I did indicate that the study was still worthwhile even absent the second test. On what may be a selfish note, I am disappointed that the study does nothing to validate the diagnostic value of repeated CPET testing for ME/CFS. It was briefly mentioned that this might be part of subsequent studies.

So is this CDC protocol a reasonable compromise? I’m sure it was influenced by budget, to some degree. CPET testing is expensive (as I can personally attest), and creates a serious burden of recovery. CDC is choosing to compromise by using a single maximal test and then measuring the effect on patients. Will a one day test be sufficient to demonstrate PEM, including the cognitive and physical symptoms of a crash? Will advocates be satisfied, especially in light of Dr. Snell’s support of CDC’s protocol for this study?

I think CDC will capture good data this way, but it won’t be a complete demonstration of PEM and the metabolic dysfunction that characterizes ME/CFS. The second day of testing captures the significant drop in VO2max, oxygen consumption at the anaerobic threshold, peak workload, and workload at the anaerobic threshold. The second day results differentiate ME/CFS patients from other illness groups. It is possible that the CDC multisite results will not do so without that second day of testing. In my opinion, that is a huge missed opportunity.

 

Posted in Advocacy, Research | Tagged , , , , , , , | 21 Comments