Comparative Effectiveness Research

The Institute of Medicine defines CER as “the generation and synthesis of evidence that compares the benefits and harms of (various) methods to prevent, diagnose, treat, and monitor a clinical condition or to improve the delivery of care.”

CER compares two or more care options (e.g. treatments, prevention measures, diagnostic tools) to determine which work best, for whom, and in what circumstances.

For example, a CER study might compare joint replacement surgery with a new drug in treating severe osteoarthritis of the knee. Researchers would examine the benefits as well as the disadvantages of both options. The study would generate evidence with relevance for different subgroups, such as women under age 65 or overweight men.

Better Evidence for More Informed Health Choices

When faced with making a health-related decision, like how to treat an illness or injury, you need sound scientific evidence to help you weigh the alternatives. But most research studies don’t do side-by-side comparisons. They focus on one option at a time, measuring whether it works in an ideal clinical setting rather than in typical conditions.

Research for the Real World

We’re working to make study findings more relevant by advancing an approach known as Comparative Effectiveness Research (CER). Examining large, diverse populations, CER studies compare two or more options to identify which work, for whom, and in what circumstances. They produce scientific evidence that applies to the real world, enabling patients and health professionals to make more informed decisions.

Professional Training

Integrative health professionals and researchers around the world are learning how to conduct CER studies through the Institute’s international research methods training program. Led by Nova Institute Scholar and CER expert Claudia Witt, a module of this intensive course is dedicated to CER. Participants come away able to conduct their own pragmatic trials.

Guidelines for CER Studies

Institute Scholar Claudia Witt led the development of two effectiveness guidance documents (EGD). These guidelines outline best practices and ensure uniformity in the way that integrative medicine researchers conduct comparative effectiveness studies of acupuncture and traditional Chinese medicine. See the EGD for acupuncture research.

CER Forums

The Institute organized a series of forums, inviting a wide range of stakeholders—including patients, health professionals, policy makers, and payers—to explore CER’s potential and lay the foundation for applying it to integrative medicine.

research icon

Stakeholder Symposium on the Evidentiary Framework for Complementary and Integrative Medicine  

Partnering with the Center for Medical Technology Policy, the Nova Institute brought together 45 experts and stakeholders in November 2009 to examine opportunities and obstacles in conducting rigorous, clinically useful studies in integrative medicine. “I think this debate is happening throughout all the Western industrialized nations,” said symposium participant George Lewith, professor of health research at University of Southampton. “How do we get effective healthcare decision-making without having to investigate every little small element of treatment with a placebo controlled randomized trial?”

Read the whitepaper, “Physician Perspectives on Comparative Effectiveness Research: Implications for Practice-based Evidence,”  Global Advances in Health and Medicine, 2012

Comparative Effectiveness Research & Integrative Medicine Online Symposium

This virtual forum in April 2010 addressed three major areas:

  • Stakeholder needs: When making healthcare decisions, what information do we need? What outcomes matter to us and therefore should be measured? How can high-quality CER support the needs of all stakeholders
  • Research Methods: If commonly used research methods don’t yield the information we need, what research designs can? If new methods are considered less ‘rigorous’ than existing ones, is it possible to achieve both rigor and relevance in clinical research? How can we balance the internal validity and the practical application of clinical research?
  • Cost-effectiveness: What do we know about the cost-effectiveness of the complementary and integrative approach to healthcare? How can CER help us understand the potential of integrative medicine to lower the national healthcare bill? Can CER give complementary and integrative medicine a place in the coverage and reimbursement system?
  • Interdisciplinary Consensus Workshop on the Development of an Effectiveness Guidance Document for Acupuncture Research
    This July 2011workshop jump-started the process of creating an effectiveness guidance document, a set of guidelines to steer the design of comparative effectiveness research studies on acupuncture. The outcome, published in BMC Complementary and Alternative Medicine, took existing research design procedures into account and identified opportunities to integrate health economic analysis.

Frequently Asked Questions

The most commonly used research methods don’t always produce the kind of evidence people need to make sound decisions about their health care or the care of their patients. While conventional studies are scientifically rigorous, they often fall short in offering insights that can be applied in the real world.

CER is of great benefit to those who are not well-served by typical studies, such as children, the elderly, seriously and terminally ill patients, and minorities. CER studies are capable of generating evidence that can contribute substantially to personalized care.

With increasing cost pressures on health care, knowing which care options are most effective for which people is more important than ever.

Most studies are conducted in large teaching institutions, and care options (e.g., treatments, preventative measures, diagnostics) are delivered in a very standardized way. By contrast, CER studies examine options as they are most likely to be used in the real world.

Conventional studies usually compare treatments to placebos and seldom look at complex strategies, such as the combination of medication, exercise, and diet to treat cardiovascular disease. CER studies are different. They compare treatment options to one another, examining both the benefits and harms. They are very well suited for evaluating complex strategies.

Patients in most conventional research studies resemble one another. Those who have more than one medical condition or are taking multiple medications are typically eliminated. CER, on the other hand, uses a broad methodology and deviates from the strictly controlled trial framework by adjusting design elements, such as eligibility criteria.

CER places a strong emphasis on involving stakeholders (e.g. patients, health care providers). Unlike conventional research practices, CER identifies the kind of evidence stakeholders want and establishes research priorities and protocols accordingly.

The old paradigm of clinical research can be likened to passengers on a city bus. The passengers are being driven from point to point, sometimes bypassing or falling short of their destinations. These anonymous participants have no say about the route, the speed, or their fellow passengers.

CER offers a new paradigm with a different metaphor: The car pool. Each participant brings unique skills, perspectives, resources and a personal sense of urgency about getting to a specific destination.

On the bus, it doesn’t matter how urgent things are for you. You arrive when the bus arrives. The stop may or may not be close to your destination. In the car pool, each member drives, offers direction, and contributes personal knowledge and experience.

Comparative effectiveness research has the potential to put patients in the driver’s seat, or at least share the driver’s seat with other stakeholders. When properly designed, CER offers new evidence, insight into subgroups, and timely, relevant results.

When we measure efficacy, we’re trying to answer the question: Did the treatment, intervention, or therapy perform as we expected it to perform in a perfect or ideal environment? In other words, did the intervention produce a detectable specific effect when all circumstances were optimized?

This type of research excludes as many complicating influence factors as possible. A good example is the typical drug study, such as a comparison of a new drug to reduce high blood pressure with a placebo or another drug in patients with no other medical problems and who are reminded frequently to take all of their study medications.

When we measure effectiveness, we are trying to answer the question: Did this treatment work in a real-world setting, similar to how health care would usually be delivered and used?

The key question in this approach is to determine which treatment works best for which patients and under what circumstances. A good example of an effectiveness study is a study that compares a drug with an exercise program to reduce high blood pressure in patients treated by community physicians.

To have a clear picture about the clinical evidence for an intervention, both efficacy and effectiveness are relevant. However, as its name clearly implies, comparative effectiveness research focuses on developing evidence of effectiveness. And it often involves comparing at least two widely used treatment options, since those are the type of choices that patients and doctors commonly face.

Understanding the distinction between efficacy and effectiveness in clinical studies is critical to our understanding of CER and relates directly to the purpose of CER as described by the Institute of Medicine, to assist decision makers in making informed decisions.

Three elements of study design influence the placement of each study within the efficacy effectiveness continuum:

  • Eligibility criteria of study participants
  • Details of the patients’ treatment during the study (treatment protocol)
  • Parameters used to measure the treatment effect (outcome measures)

Let’s examine each of these in detail:

The eligibility criteria of the study participants. For example, a study that included only highly selected patients of younger age without co-morbidities and co-medication who are expected to have a better reaction to the intervention would fall on the efficacy side of the continuum. Maximum effectiveness would mean to include all types of patients who are affected by the disease without excluding subgroups of patients with risk factors e.g. high blood pressure, which might predict a less favorable response to the treatment.

The treatment protocol. A study in which a very standardized treatment protocol as study intervention (e.g., a fixed exercise program without any flexibility) would fall on the efficacy side of the continuum. Maximum effectiveness would suggest a flexible treatment protocol as implemented in routine or usual care (e.g., exercise program individually adapted to the age and co-morbidities and preferences of the patient).

The outcome measures. Objective outcome measures such as death and laboratory data would fall on the efficacy side of the continuum. Maximum effectiveness would rely more heavily on patient-reported outcome measures (e.g., symptom scales or quality of life measures).

When study participants are highly selected, the treatment is standardized, and all major outcomes are not directly experienced by patients, such a study would provide information on efficacy more than effectiveness.

In this case, the study results would be very valid for similar patients in similar settings, but less so when applied under other conditions. Thus the study is said to have a higher internal validity. This means that the results of the study are less influenced by other factors, for example, beliefs and health behaviors. We may reasonably conclude that the results actually represent the effect of the treatment we have researched.

However, when study participants represent a mixture of typical patients, the treatment protocol is more flexible, and the outcome measures are more patient-centered or subjective, the study measures effectiveness more than efficacy. When the study measures effectiveness, the results are regarded as more relevant to real-world health care decisions. In other words, this study is considered to have a higher external validity.

It’s important to keep in mind that CER does not only mean doing studies that represent the extreme range of effectiveness studies. The “art” of CER requires that researchers deliberately sacrifice some degree of internal validity in order to increase the generalizability, relevance, feasibility and timeliness of research results. The right balance of the desirable features of research cannot be determined rigorously through a scientific process. The optimal balance emerges from the process of engaging stakeholders in a discussion about each element of study design, in light of current evidence, and what sorts of decisions must be made.

Comparative Effectiveness Research Resources

Effectiveness Guidance Documents

Articles

Organizations

  • Patient-Centered Outcomes Research Institute (PCORI) is authorized by Congress to conduct research to provide information about the best available evidence to help patients and their health care providers make more informed decisions. PCORI’s research is intended to give patients a better understanding of the prevention, treatment, and care options available and the science that supports those options.
  • Center for Medical Technology Policy (CMTP) defines and publishes methodological standards and guidance for CER, develops processes and products for clinical research, and facilitates dialogue around relevant policy issues.

Healing is facilitated through safety, persistence, and trust.

  • Persistence: “People did not simply progress through this sequence and experience healing. The healing journey was a recursive, back and forth process. They found helpers, used the skills/resources that those helpers provided, found other helpers that provided more resources and used those skills and resources. As this process continued, people experienced a gradual amelioration of their suffering. Although many despaired at times, all demonstrated the quality of persistence—they refused to give up.”
  • Safety & Trust: “To connect to helpers, it was essential for people to feel safe in those relationships and able to trust that the person would be a helper and not a barrier to healing. Persons whose wounds included a violation of trust were especially careful about testing the safety of new relationships.”

Resources support us as we heal. They include reframing, responsibility, and positivity. “Making connections enabled participants to acquire and refine resources and skills that were essential in their healing journey. People also brought their own personal strengths to the journey.”

  • Reframing: “A particularly important skill was the ability to reframe—that is to look at suffering through a different lens.” This does NOT mean minimizing trauma or pain, but rather it often means the opposite: understanding what happened was wrong, unfair, or uncontrollable and that we are not to blame for it.
  • Responsibility: While we don’t have control over what happened to us, we are the only ones who can help ourselves heal. “A third essential resource that people acquired or refined was the ability to take an appropriate amount of responsibility for their healing journeys. They participated actively in the process of healing. Once again, some participants already had developed this skill, and some acquired or refined it from their helpers.”
  • Positivity: “Another resource that people acquired or refined during their healing journey was choose to be positive—that is to have some optimism about their situation.” People have varying predispositions to positivity. In the study, positivity was important in helping people heal. This doesn’t mean a toxic positivity, but rather simply finding some good in life and feeling hopeful about our situations.

“Connection to others was an essential part of all the healing journeys.” Humans are social creatures, and even the most introverted of us need close relationships. Friends and family add meaning and value to life and help support us, in good times and bad. 

When we experience relational trauma, relationships can feel scary, but reestablishing safety and trust in relationships is where the healing happens. (To be clear, we do not mean reestablishing safety and trust with abusers, but rather finding other healing relationships.) 

“When safety and trust had been established, people were able to connect with helpers. The nature of the behaviours of helpers that fostered healing ranged from small acts of kindness to unconditional love.”

  • “Moving from being wounded, through suffering to healing, is possible. It is facilitated by developing safe, trusting relationships and by positive reframing that moves through the weight of responsibility to the ability to respond.”
  • “Relationships with health professionals were among these but were not necessarily any more important to the healing journey than other kinds of helpers, which included family members, friends, spirituality and their God, pets, support groups, administrators, case workers and supervisors.”

Healing probably means different things to different people, but one definition that emerged from the study is: “The re-establishment of a sense of integrity and wholeness.” 

Healing was an emergent property that resulted from each individuals’ complex healing journey, a result of bridged connections between resources and relationships. “…they gradually found relief from suffering and began to exhibit emergent characteristics: a sense of hope, self-acceptance, and a desire to help others—the immediate precursors to healing.”

 In varying degrees, “they were able to transcend their suffering and in some sense to flourish.” 

  • Helping Others: We find meaning in helping others. “Understanding that suffering gives the strength and experience to help others in similar situations.”
  • Hope: We begin to have hope that we will not always feel this bad. A Crohn’s patient said, “I think gradually I realized that I was going to feel better. I did have days when I actually didn’t vomit, when I did feel better. And I think gradually I came to believe that maybe I could have a normal life again.”
  • Self-Acceptance: We see our inherent value and understand that we are not to blame for our suffering. A participant living with HIV said, “I’m really proud of myself. I think that now I still want to live. I don’t want to die, and I really love myself a lot. I have a lot of comfort in myself.”

Suffering is the ongoing pain from wounding. 

There is debate about whether or not one actually needs to experience suffering on the path to healing.

Wounding happens when we experience physical or emotional harm. It can stem from chronic illness or by physical or psychological trauma for which we do not have the tools to cope, or a combination of those factors. 

“The degree and quality of suffering experienced by each individual is framed by contextual factors that include personal characteristics, timing of their initial or ongoing wounding in the developmental life cycle and prior and current relationships.”

Characteristics: How predisposed someone may be to wounding/how many tools and resources someone may have to deal with trauma/illness.

Lifestages: Developmental timing plays an important role in the impact of trauma — young children often do not have the same resources as older adults.

Relationships: Relationships can provide solace and support for those suffering, while lack of healthy relationships can prolong suffering.