What If the Institute of Medicine Wrote a Report and Nobody Followed it? - the Case of the Standards for Developing Trustworthy Guidelines

For over 20 years, clinical practice guidelines (CPGs) have been touted to improve health care quality and control costs.  Enormous numbers of guidelines have been developed, but with seemingly little impact on health outcomes.  While some of those leading health care organizations have predictably blamed individual practitioners for obstinately ignoring or challenging guidelines, there is increasing evidence that maybe the guidelines themselves are part of the problem.

 An Example of a Guideline that Apparently was Not Trusted

One example Dr Wally Smith and I have taught in our recurring mini-course on why physicians fail to follow guidelines (and otherwise appear not to practice in accord with others' wishes) is that of the guidelines on management of depression in primary care.  Most existing guidelines urge physicians to screen patients for (presumably mild-to-moderate) depression and treat them aggressively, with emphasis on the use of the newer anti-depressants.  These guidelines, in turn, were based on numerous published randomized clinical trials that showed that these drugs were safe and efficacious.  Yet multiple studies showed that physicians failed to follow these guidelines, and various attempts to improve their adherence did little.  So for years the assumption was that physicians at best experienced practical and system barriers to follow these guidelines, and at worst were ill-informed or irrational. 


However, information that came out gradually during the early part of the 21st century suggested that perhaps the problem was within the guidelines, not the health care professionals.  First, documents produced during New York Attorney General Eliot Spitzer's lawsuit against GlaxoSmithKline about the marketing of one of these drugs (Paxil, paroxetine) suggested that the company had suppressed clinical trial data that reflected poorly on the drug (See Kondro W.  Drug company experts advised staff to withold data about SSRI use in children.  Can Med Assoc J 2004; 170: 783.  Link here.)    These suspicions were later fleshed out  by consideration of documents further disclosed in litigation (e.g., see this post and its links).  Then several studies, most particularly that by Erick Turner and colleagues, showed that numerous trials of new anti-depressants had been suppressed, that is, never published (Turner et al. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med 2008; 358:252-260.  Link here).  When the results of these trials were added to those that were published, the efficacy of anti-depressants was no longer so clear.  So maybe the guidelines that physicians did not follow were not trustworthy, and should not have been followed in the first place.

Would IOM Standards to Improve Guideline Trustworthiness Help?

So in 2011, the prestigious Institute of Medicine released a report on the development of  better standards to produce more trustworthy guidelines (Clinical Practice Guidelines We Can Trust.  Link here.)  We posted about that report here, but noted that it was receiving little other attention, an example of the anechoic effect. 

A few weeks ago, an article appeared documenting a study meant to assess the the trustworthiness of clinical practice guidelines published soon after the IOM report.  Its title telegraphs the results. ( Kung J, Miller RR, Mackowiak PA. Failure of clinical practice guidelines to meet Institute of Medicine standards: two more decades of little, if any progress.  Arch Intern Med 2012.  Link here.)

Methods and Results

The investigators selected a random sample of 114 individual guidelines available during June, 2011 stratified by 26 clinical topics. The versions of the guidelines used were those archived in the National Guideline Clearinghouse (NGC) maintained by the Agency for Healthcare Research and Quality (AHRQ).


The goal of the study was to "examine adherence to the IOM standards" by guidelines published after the standards were published.  Actually, the study only assessed adherence to 18 of the 25 standards espoused by the IOM (because the remaining seven were "too vague and subjective to be analyzed.")

Furthermore, the criteria used to determine if a specific guideline met each of the three items above were rather lax:

In evaluating each guideline summary, care was taken to be as liberal as possible in considering that a standard was met when the individual guideline summary provided any information pertaining to that particular standard.

Nevertheless, using these lax standards to only evaluate adherence to 18/25 guidelines, the authors found that "the overall median number of IOM standards satisfied (out of 18) was 8 (44.4%) ....  Fewer than half of the guidelines surveyed met more than 50% of the IOM standards."

An examination of the details of the study's methods reveals things are even worse than that.

Analyzing the Study's Methods to Find that Things Are Worse Than They Seem

Review of the study's methods show that they provided a very optimistic view of adherence to the IOM standards.  As noted above, the study did not look for adherence to all of the IOM standards.  Moreover, those they did consider were simplified and made less rigorous.  For example, Standard 2 from the IOM on management of conflict of interest (COI) was:

  STANDARD 2
Management of conflict of interest (COI)

2.1
Prior to selection of the Guideline Development Group (GDG), individuals being considered for membership should declare all interests and activities potentially resulting in COI with development group activity, by written disclosure to those convening the GDG.
- Disclosure should reflect all current and planned commercial (including services from which a clinician derives a substantial proportion of income), non-commercial, intellectual, institutional, and patient/public activities pertinent to the potential scope of the CPG.

2.2
Disclosure of COIs within GDG
- All COI of each GDG member should be reported and discussed by the prospective development group prior to the onset of their work.
- Each panel member should explain how their • COI could influence the CPG development process or specific recommendations.

2.3
Divestment
- Members of the GDG should divest themselves of financial investments they or their family members have in, and not participate in marketing activities or advisory boards of, entities whose interests could be affected by CPG recommendations.

2.4
Exclusions
- Whenever possible GDG members should not have COI.
- In some circumstances, a GDG may not be able to perform its work without members who have COIs, such as relevant clinical specialists who receive a substantial portion of their incomes from services pertinent to the CPG.
- Members with COIs should represent not more than a minority of the GDG.
- The chair or co-chairs should not be a person(s) with COI.
- Funders should have no role in CPG development.

However, the study boiled all this down to three items:
-  COIs stated
-  Chair has COI
-  Co-chairperson has COI

Thus the study did not address standards requiring full and complete disclosure (not just some disclosure) of all COIs; consideration of how the COIs might influence the particular guideline; divestment of specific types of conflicts of interest, that is, financial investments, and cessation of participation in marketing activities or advisor boards; minimization of conflicts of all members of the committee; and barring of participation of funders in guideline development.

Even so, the guidelines assessed did a very poor job upholding even these few liberalized standards regarding conflicts of interest.  Of the guidelines assessed, less than half, 46.8% provided ANY disclosure of conflicts of interest.  Those written by sub-specialty societies were particularly opaque in this regard.  Less than one-third, 29.3%, provided any disclosure.  Thus the majority of guidelines assessed were not at all transparent about conflicts of interest affecting the guideline development process.

Furthermore, of the 46.8% of all the guidelines which made any disclosures of conflicts of interest, 71.4% admitted their chair people HAD a conflict of interest.  Thus, only (0.468 * [1 - .714]) = 13.3% provided assurance that they fulfilled the single requirement (from standard 2.4 above) that the chair person did not have a COI.  For the guidelines written by sub-specialty societies, by a similar calculation, only 12.2% provided an assurance that the chair had no COI.  (The proportions providing assurance that the co-chair people had no COI were even lower.)  Thus the vast majority of guidelines did not clearly follow two straight-forward standards for minimizing the effects of conflict of interest, that the guideline committee chair and co-chair should not have any relevant conflicts. 

Given the miserable results concerning even minimal adherence to some of the IOM report's conflict of interest standards, it is likely that almost no published guidelines from 2011 came close to fulfilling the full set of IOM standards.  Despite the best efforts of the IOM, it appears that guideline developers have not progressed at all towards providing trustworthy guidelines. 

Summary

An editorial (Shaneyfelt T. In guidelines we cannot trust.  Arch Intern Med 2012) accompanying the article by Kung and colleagues summarized its results thus:

The same problems that have plagued guideline development continue to plague guideline development; namely, their variable and opaque development methods, their often conflicted and limited panel composition, and their lack of significant external review by stakeholders throughout the development process.   As a result, the trustworthiness of guidelines is limited.

While guidelines may have seemed to be a promising method to improve health care in the early 1990s, they have failed to live up to that promise.  Shaneyfelt was not optimistic they would improve in the future:

I am not optimistic that much will improve.  No one seems interested in curtailing the out-of-control guideline industry.

On the other hand, in my humble opinion, it is not that on one is interested in better guidelines.  It would clearly be in the best interests of patients and the public, and of health care professionals who care about the quality of their practice and the outcomes of their patients to curtail that industry.  The issue is why patients', the public's, and professional's interests were ignored.

Neither Shaneyfelt nor Kung et al discussed why there has been so little attention to patients' and the public's health, and to health care professionalism in all this.  For a quick answer, we do not have to look far on Health Care Renewal. 


In fact, the IOM report on guideline development from 2011 was a serious challenge to the powers that be in health care.  In particular, it challenged the cozy relationships that had grown up among the organizations that undertook guideline development and the health care professionals on guideline panels on one hand and organizations that stand to gain were specific guidelines to favor their products, services, and agendas on the other.  The standards mandated transparency and honesty about conflicts of interest affecting guideline committees and the organizations which assembled them, and if upheld would have greatly reduced these relationships.

Now it turns out that the guideline standards have been honored mainly in the breach.  Of course, these standards, while increasing trustworthiness, would have cost a lot of medical societies considerable commercial funding, and would have cost a lot of health care professionals on guideline panels considerable personal wealth.  These standards would probably also have cost a lot of companies whose products and services were addressed by guidelines to lose revenue.  So it is not surprising that the IOM standards were ignored.  Their implementation would have cost too many people who are financially benefiting from the status quo too much money.  And these people, that is, leaders of professional societies dependent on commercial outside funding, health care professionals and academic used to financial support from commercial interests, and health care corporations are good at making sure their interests are not ignored, even if their interests conflict with those of patients, the public, and well-intentioned health care professionals.  

So, the flouting of the well reasoned IOM guideline standards adds one more reason for patients and the general public to distrust modern health care and all those who "deliver" it, even to distrust well-intentioned health care professionals who have not been able to distinguish themselves from their colleagues who are too happy to help commercial interests while taking commercial money.  If health care professionals want to regain the public's trust, they could do worse than publicly declaring their intention to show that their practice in the future will be guided by trustworthy guidelines based on clinical research evidence and knowledge of biomedical science, drawn up by health care professionals independent of commercial interests.

The Institute of Medicine Releases Reports on Practice Guidelines and Systematic Reviews Which Generate Few Echoes

Two days ago, the prestigious US Institute of Medicine released two reports on important health care issues, clinical practice guidelines and systematic reviews.  Systematic reviews of the relevant clinical research have been advocated by evidence-based medicine proponents as the appropriate basis for clinical and policy decisions.  Clinical practice guidelines have been advocated by many health researchers, policy makers, and clinicians as the best way to encapsulate the evidence to inform clinical and policy decision making.  Both reports suggested series of standards for how systematic reviews and clinical practice guidelines should be developed. 

These topics are of general importance to clinicians, health services researchers, and health policy makers.  The Institute of Medicine, part of the US National Academy of Sciences, is one of the most authoritative sources of opinion on medicine and health care.  Therefore, one would think that these reports would have gotten wide notice, and would hardly required Health Care Renewal to create some echoes.

However, a Google News search today produced only six "hits" relevant to these reports, including the original press release.  All are in specialized medical/ health care news outlets.  None are in the national media, and none are from major medical/ professional journals or societies. 

Let me suggest a theory about why these two major reports have generated so few echoes so far.  Let me quote from the summary of the report on clinical practice guidelines:

Most guidelines used today suffer from shortcomings in development. Dubious trust in guidelines is the result of many factors, including failure to represent a variety of disciplines in guideline development groups, lack of transparency in how recommendations are derived and rated, and omission of a thorough external review process. To be trustworthy, clinical practice guidelines should:
• Be based on a systematic review of the existing evidence;
• Be developed by a knowledgeable, multidisciplinary panel of experts and representatives from key affected groups;
• Consider important patient subgroups and patient preferences, as appropriate;
• Be based on an explicit and transparent process that minimizes distortions, biases, and conflicts of interest;
• Provide a clear explanation of the logical relationships between alternative care options and health outcomes, and provide ratings of both the quality of evidence and the strength of recommendations; and
• Be reconsidered and revised as appropriate when important new evidence warrants modifications of recommendations.
Additionally, as reflected in the committee’s standards for developing trustworthy clinical practice guidelines, guideline development groups optimally comprise members without conflict of interest. The committee recognizes that in some circumstances, a guideline development group may not be able to perform its work without members who have conflicts of interest—for example, relevant clinical specialists who receive a substantial portion of their incomes from services pertinent to the guideline. Therefore, the committee specifies that members of the guideline development group who have a conflict of interest should not represent more than a minority of the group.

So it seems that the report on clinical practice guidelines emphasized two issues highly relevant to Health Care Renewal, the need for transparency in guideline development, and the need to avoid conflicts of interest affecting the development process. The two first standards for guidelines are about transparency and minimization of conflicts of interest.  Similarly, the report on systematic reviews also included fairly tough standards to minimize conflicts of interest.

We on Health Care Renewal go on and on about the need to maximize transparency in health care, and particularly in health care leadership and governance, and about the need to disassemble the now pervasive web of conflicts of interest that has entangled health care.  However, as we know, these are not popular topics among the leadership of health care, which includes many individuals who have greatly benefited from lack of transparency and pervasive conflicts of interest.  We know these topics make these leaders, and many of those who report to, or work or associate with them very uncomfortable.

So unfortunately, I am not surprised that the two new and likely authoritative reports from the Institute of Medicine, despite that organization's prestige, have started off relatively anechoic.  It also unfortunately likely that they will remain relatively anechoic. 

In 2009, the IOM issued an authoritative report on conflicts of interest in medicine and health care which suggested fairly tough standards to decrease such conflicts and their influence (also see post here).  Since 2009, I just found 53 citations to that report in the medical literature using the ISI Web of Science, for a rate of 27/year.  In 1999, the IOM issued a report on medical errors, "To Err is Human."  Since then, it has received 1374 citations, a rate of 115/year. 

As we noted above, the topic of conflicts of interest seems to make the powers that be in health care very uncomfortable.  In contrast, "To Err is Human" was widely interpreted to mean that physicians make a lot of dangerous errors, and the best way to decrease them is to impose more controls by bureaucrats, managers, and executives (even if that was not its intent).  Thus, that report could be twisted to fit the talking points of the powers that be, and hence has been anything but anechoic.

So while Health Care Renewal is hardly a powerful tool for creating publicity, I thought we should try to get the word out about the new IOM reports on clinical practice guidelines and systematic reviews.  Every little bit helps.

Meanwhile, the deathly quiet reception these reports have gotten so far emphasizes the need to combat the anechoic effect.  As long as the powers that be can command billions of dollars to influence the health care conversation through their marketing, public relations, and lobbying departments, expect the discussion not to question what they do, and how they benefit from the status quo to the financial and health detriment of patients and the population.

We will not be able to truly reform health care until we can speak openly about what threatens health care values and what needs to be done about these threats.

New CMS Chief Donald Berwick: a Trojan Horse for Quackery?

On July 7, President Obama appointed Dr. Donald Berwick as Administrator of the Centers for Medicare and Medicaid Services (CMS). Dr. Berwick, a pediatrician, is well known as the CEO of the non-profit Institute for Healthcare Improvement (IHI), which "exists to close the enormous gap between the health care we have and the health care we should have — a gap so large in the US that the Institute of Medicine (IOM) in 2001 called it a 'quality chasm'.” Dr. Berwick was one of the authors of that IOM report. His IHI has been a major player in the patient safety movement, most notably with its "100,000 Lives Campaign" and, more recently, its "5 Million Lives Campaign."

Berwick's CMS gig is a "recess appointment": it was made during the Senate's July 4th recess period, without a formal confirmation hearing---although such a hearing must take place before the end of this Senate term, if he is to remain in the position. A recent story suggested that Obama made the recess appointment in order to avoid a reprise of "last year's divisive health care debate." The president had originally nominated Berwick for the position in April, and Republicans have opposed "Berwick's views on rationing of care," claiming that he "would deny needed care based on cost."

A "Patient-Centered Extremist"

If there is a problem with the appointment, it is likely to be roughly the opposite of what Republicans might suppose: Dr. Berwick is a self-described "Patient-Centered Extremist." He favors letting patients have the last word in decisions about their care even if that means, for example, choosing to have unnecessary and expensive hi-tech studies. In an article for Health Affairs published about a year ago, he explicitly argued against the "professionally dominant view of quality of health care":

I think it wrong for the profession of medicine—or any other health care profession, for that matter—to “reserve to itself the authority to judge the quality of its work.” I eschew compromise words like “partnership.” For better or worse, I have come to believe that we—patients, families, clinicians, and the health care system as a whole—would all be far better off if we professionals recalibrated our work such that we behaved with patients and families not as hosts in the care system, but as guests in their lives. I suggest that we should without equivocation make patient-centeredness a primary quality dimension all its own, even when it does not contribute to the technical safety and effectiveness of care.

A new definition. My proposed definition of “patient-centered care” is this: The experience (to the extent the informed, individual patient desires it) of transparency, individualization, recognition, respect, dignity, and choice in all matters, without exception, related to one’s person, circumstances, and relationships in health care.

Does this mean that Dr. Berwick would also eschew professional, i.e., expert, judgment in favor of patients' wishes? In a word, yes:

Evidence-based medicine sometimes must take a back seat. First, leaving choice ultimately up to the patient and family means that evidence-based medicine may sometimes take a back seat. One e-mail correspondent asked me, “Should patient ‘wants’ override professional judgment about whether an MRI is needed?” My answer is, basically, “Yes.” On the whole, I prefer that we take the risk of overuse along with the burden of giving real meaning to the phrase “a fully informed patient.”

Dr. Berwick is not so naive as this opinion might suggest. He envisions a "mature dialogue" in such a case, and argues that "if, over time, a pattern emerges of scientifically unwise or unsubstantiated choices...then we should seek to improve our messages..." He also admits that there might be an occasional patient whose demands are so unreasonable that "it is time to say, 'No'." That exception, he argues, should not dictate the rule.

There are situations in which most civilized people would agree with Dr. Berwick's view of 'patient-centeredness'. In both the Health Affairs article and in his recent address to the 2010 graduating class of the Yale School of Medicine, he offered real examples of petty, arbitrary hospital rules causing unnecessary sorrow for patients and their loved ones. It is in such contexts that he makes a convincing case that health professionals ought to behave "as guests in their lives." In an interview for the New York Times, he argued:

We don’t have a standard of services or processes that are comfortable for patients. We have built a technocratic castle, and when people come into it, they are intimidated.

Nothing to disagree with there. To create that standard, moreover, would not undermine settled medical practice ethics---it would celebrate them, even as it rightly embarrasses the profession for having taken so long to do so.


Enter the Woo

Eschewing the scientific basis for modern medical practice, however, is another matter. In February of 2009, Dr. Berwick gave a 'keynote' address at the IOM and Bravewell Collaborative-sponsored Summit on Integrative Medicine and the Health of the Public. He shared the podium with Mehmet Oz, Dean Ornish, Senator Tom Harkin, and other advocates of pseudoscientific health claims. I wrote about the conference at the time, mainly to call attention to its misleading use of the term "integrative medicine": literature emanating from the Summit characterized it as "preventive" and "patient-centered," whereas the only characteristic that distinguishes it from modern medicine is an inclusion of various forms of pseudomedicine.


I noticed that Dr. Berwick was on the speaker roster, which I found disappointing: I imagined that he had either gone over to the Dark Side or, perhaps, was sufficiently naive about the topic to have been duped; or, more likely, that he had cynically accepted the offer to further his ambitions. I didn't bother to listen to his speech until the CMS appointment was announced a few days ago.

It is troubling, to say the least. Dr. Berwick did not argue, as he had in the NYT piece, that "If we doctors feel a person is going to make unwise choices, we have to take on the responsibility of being teachers, educators and informers." Rather, he praised his fellow speakers, most of whom were spouting nonsense, for their "reach" and "eloquence." He praised the IOM for its "glorious record...in pursuit of better designs in health care...traditional, allopathic curative care and now migrating into this distinguished and important new arena." He mentioned homeopathy and acupuncture, not to wonder why they should be promoted as effective, but merely to warn that they will fail---presumably in some economic sense---if they try to compete with each other for reimbursement.

Such language, and Dr. Berwick's very presence at the Summit, were a far cry from advocating "patient-centeredness." What they amounted to was a generous endorsement of pseudoscientific practices and of the socio-political movement that promotes them. Even granting some naivete on his part (he called himself "an amateur at this topic"), he must have known this. Such an endorsement, unlike tearing down the "technocratic castle," has ethical implications at least as profound as those that Dr. Berwick tacitly or explicitly relies upon to support his arguments for patient-centeredness.

"Physicians have no Immunity to Moral or Ethical Constraints"

The relevant medical ethics treatises (reviewed here) are in substantial agreement that it is unethical for physicians to prescribe scientifically implausible methods or to refer patients to other practitioners for the same purpose. They are also in agreement that it is unethical to prescribe a placebo to a patient while claiming that the treatment has specific biologic activity---a point that has been vigorously argued in the UK this year, with regard to homeopathy. These ethical tenets are not mere odes to nerdy, sciency thinking; they are matters of honesty and integrity---fundamental bases for ethical interactions between physicians and patients.

In 1983, philosophers Clark Glymour and Douglas Stalker published an article in the New England Journal of Medicine titled “Engineers, cranks, physicians, magicians.” They framed modern medicine as follows, comparing it to what was then called "holistic medicine" (the article is quoted extensively here):

Medicine in industrialized nations is scientific medicine. The claim tacitly made by American or European physicians, and tacitly relied on by their patients, is that their palliatives and procedures have been shown by science to be effective. Although the physician’s medical practice is not itself science, it is based on science and on training that is supposed to teach physicians to apply scientific knowledge to people in a rational way.

The practice of medicine in the United States and in other industrialized nations is a form of consultant engineering...

That statement is just as accurate now---even more so, in this era of Evidence-Based Medicine---as it was nearly 30 years ago, even if some might find the likening of medicine to engineering displeasing. Nor is it at odds with almost any definition of "patient-centeredness," other than one that presumes that the patient's desires trump the physician's ethics:

A physician engineer can act as consoler; nothing in either logic or social psychology forbids it. But certain combinations are impossible or extraordinarily unlikely. A physician engineer cannot honestly claim powers of magic or occult knowledge. The principles governing scientific reasoning and belief are negative as well as positive, and they imply that occult doctrines are not worthy of belief. Moreover, physician engineers have no immunity to moral or ethical constraints. On the contrary, they are by training and by culture enmeshed in a tradition of rational thought about
the obligations and responsibilities of their profession.

Dr. Berwick---if he really believes what his presence and words at the "Integrative Medicine" Summit imply---is playing with ethical fire. (If, as I hope, he doesn't really believe those things, he's playing with ethics of another kind). Will we begin to see pseudomedicine "integrated" into Medicare and Medicaid? That is certainly the expectation of those who observed Dr. Berwick's performance at the Summit, and who appear intent to hold him to his word.

KA

Two other blogs that have addressed this issue are:

Dr. RW: Not only evidence based medicine but science based medicine may take a back seat in Donald Berwick's vision for patient centered care

Dr. David Gorski: Dr. Donald Berwick and “patient-centered” medicine: Letting the woo into the new health care law?