Comparative Effectiveness Research

The Institute of Medicine defines CER as “the generation and synthesis of evidence that compares the benefits and harms of (various) methods to prevent, diagnose, treat, and monitor a clinical condition or to improve the delivery of care.”

CER compares two or more care options (e.g. treatments, prevention measures, diagnostic tools) to determine which work best, for whom, and in what circumstances.

For example, a CER study might compare joint replacement surgery with a new drug in treating severe osteoarthritis of the knee. Researchers would examine the benefits as well as the disadvantages of both options. The study would generate evidence with relevance for different subgroups, such as women under age 65 or overweight men.

Better Evidence for More Informed Health Choices

When faced with making a health-related decision, like how to treat an illness or injury, you need sound scientific evidence to help you weigh the alternatives. But most research studies don’t do side-by-side comparisons. They focus on one option at a time, measuring whether it works in an ideal clinical setting rather than in typical conditions.

Research for the Real World

We’re working to make study findings more relevant by advancing an approach known as Comparative Effectiveness Research (CER). Examining large, diverse populations, CER studies compare two or more options to identify which work, for whom, and in what circumstances. They produce scientific evidence that applies to the real world, enabling patients and health professionals to make more informed decisions.

Professional Training

Integrative health professionals and researchers around the world are learning how to conduct CER studies through the Institute’s international research methods training program. Led by Nova Institute Scholar and CER expert Claudia Witt, a module of this intensive course is dedicated to CER. Participants come away able to conduct their own pragmatic trials.

Guidelines for CER Studies

Institute Scholar Claudia Witt led the development of two effectiveness guidance documents (EGD). These guidelines outline best practices and ensure uniformity in the way that integrative medicine researchers conduct comparative effectiveness studies of acupuncture and traditional Chinese medicine. See the EGD for acupuncture research.

CER Forums

The Institute organized a series of forums, inviting a wide range of stakeholders—including patients, health professionals, policy makers, and payers—to explore CER’s potential and lay the foundation for applying it to integrative medicine.

research icon

Stakeholder Symposium on the Evidentiary Framework for Complementary and Integrative Medicine  

Partnering with the Center for Medical Technology Policy, the Nova Institute brought together 45 experts and stakeholders in November 2009 to examine opportunities and obstacles in conducting rigorous, clinically useful studies in integrative medicine. “I think this debate is happening throughout all the Western industrialized nations,” said symposium participant George Lewith, professor of health research at University of Southampton. “How do we get effective healthcare decision-making without having to investigate every little small element of treatment with a placebo controlled randomized trial?”

Read the whitepaper, “Physician Perspectives on Comparative Effectiveness Research: Implications for Practice-based Evidence,”  Global Advances in Health and Medicine, 2012

Comparative Effectiveness Research & Integrative Medicine Online Symposium

This virtual forum in April 2010 addressed three major areas:

  • Stakeholder needs: When making healthcare decisions, what information do we need? What outcomes matter to us and therefore should be measured? How can high-quality CER support the needs of all stakeholders
  • Research Methods: If commonly used research methods don’t yield the information we need, what research designs can? If new methods are considered less ‘rigorous’ than existing ones, is it possible to achieve both rigor and relevance in clinical research? How can we balance the internal validity and the practical application of clinical research?
  • Cost-effectiveness: What do we know about the cost-effectiveness of the complementary and integrative approach to healthcare? How can CER help us understand the potential of integrative medicine to lower the national healthcare bill? Can CER give complementary and integrative medicine a place in the coverage and reimbursement system?
  • Interdisciplinary Consensus Workshop on the Development of an Effectiveness Guidance Document for Acupuncture Research
    This July 2011workshop jump-started the process of creating an effectiveness guidance document, a set of guidelines to steer the design of comparative effectiveness research studies on acupuncture. The outcome, published in BMC Complementary and Alternative Medicine, took existing research design procedures into account and identified opportunities to integrate health economic analysis.

Frequently Asked Questions

The most commonly used research methods don’t always produce the kind of evidence people need to make sound decisions about their health care or the care of their patients. While conventional studies are scientifically rigorous, they often fall short in offering insights that can be applied in the real world.

CER is of great benefit to those who are not well-served by typical studies, such as children, the elderly, seriously and terminally ill patients, and minorities. CER studies are capable of generating evidence that can contribute substantially to personalized care.

With increasing cost pressures on health care, knowing which care options are most effective for which people is more important than ever.

Most studies are conducted in large teaching institutions, and care options (e.g., treatments, preventative measures, diagnostics) are delivered in a very standardized way. By contrast, CER studies examine options as they are most likely to be used in the real world.

Conventional studies usually compare treatments to placebos and seldom look at complex strategies, such as the combination of medication, exercise, and diet to treat cardiovascular disease. CER studies are different. They compare treatment options to one another, examining both the benefits and harms. They are very well suited for evaluating complex strategies.

Patients in most conventional research studies resemble one another. Those who have more than one medical condition or are taking multiple medications are typically eliminated. CER, on the other hand, uses a broad methodology and deviates from the strictly controlled trial framework by adjusting design elements, such as eligibility criteria.

CER places a strong emphasis on involving stakeholders (e.g. patients, health care providers). Unlike conventional research practices, CER identifies the kind of evidence stakeholders want and establishes research priorities and protocols accordingly.

The old paradigm of clinical research can be likened to passengers on a city bus. The passengers are being driven from point to point, sometimes bypassing or falling short of their destinations. These anonymous participants have no say about the route, the speed, or their fellow passengers.

CER offers a new paradigm with a different metaphor: The car pool. Each participant brings unique skills, perspectives, resources and a personal sense of urgency about getting to a specific destination.

On the bus, it doesn’t matter how urgent things are for you. You arrive when the bus arrives. The stop may or may not be close to your destination. In the car pool, each member drives, offers direction, and contributes personal knowledge and experience.

Comparative effectiveness research has the potential to put patients in the driver’s seat, or at least share the driver’s seat with other stakeholders. When properly designed, CER offers new evidence, insight into subgroups, and timely, relevant results.

When we measure efficacy, we’re trying to answer the question: Did the treatment, intervention, or therapy perform as we expected it to perform in a perfect or ideal environment? In other words, did the intervention produce a detectable specific effect when all circumstances were optimized?

This type of research excludes as many complicating influence factors as possible. A good example is the typical drug study, such as a comparison of a new drug to reduce high blood pressure with a placebo or another drug in patients with no other medical problems and who are reminded frequently to take all of their study medications.

When we measure effectiveness, we are trying to answer the question: Did this treatment work in a real-world setting, similar to how health care would usually be delivered and used?

The key question in this approach is to determine which treatment works best for which patients and under what circumstances. A good example of an effectiveness study is a study that compares a drug with an exercise program to reduce high blood pressure in patients treated by community physicians.

To have a clear picture about the clinical evidence for an intervention, both efficacy and effectiveness are relevant. However, as its name clearly implies, comparative effectiveness research focuses on developing evidence of effectiveness. And it often involves comparing at least two widely used treatment options, since those are the type of choices that patients and doctors commonly face.

Understanding the distinction between efficacy and effectiveness in clinical studies is critical to our understanding of CER and relates directly to the purpose of CER as described by the Institute of Medicine, to assist decision makers in making informed decisions.

Three elements of study design influence the placement of each study within the efficacy effectiveness continuum:

  • Eligibility criteria of study participants
  • Details of the patients’ treatment during the study (treatment protocol)
  • Parameters used to measure the treatment effect (outcome measures)

Let’s examine each of these in detail:

The eligibility criteria of the study participants. For example, a study that included only highly selected patients of younger age without co-morbidities and co-medication who are expected to have a better reaction to the intervention would fall on the efficacy side of the continuum. Maximum effectiveness would mean to include all types of patients who are affected by the disease without excluding subgroups of patients with risk factors e.g. high blood pressure, which might predict a less favorable response to the treatment.

The treatment protocol. A study in which a very standardized treatment protocol as study intervention (e.g., a fixed exercise program without any flexibility) would fall on the efficacy side of the continuum. Maximum effectiveness would suggest a flexible treatment protocol as implemented in routine or usual care (e.g., exercise program individually adapted to the age and co-morbidities and preferences of the patient).

The outcome measures. Objective outcome measures such as death and laboratory data would fall on the efficacy side of the continuum. Maximum effectiveness would rely more heavily on patient-reported outcome measures (e.g., symptom scales or quality of life measures).

When study participants are highly selected, the treatment is standardized, and all major outcomes are not directly experienced by patients, such a study would provide information on efficacy more than effectiveness.

In this case, the study results would be very valid for similar patients in similar settings, but less so when applied under other conditions. Thus the study is said to have a higher internal validity. This means that the results of the study are less influenced by other factors, for example, beliefs and health behaviors. We may reasonably conclude that the results actually represent the effect of the treatment we have researched.

However, when study participants represent a mixture of typical patients, the treatment protocol is more flexible, and the outcome measures are more patient-centered or subjective, the study measures effectiveness more than efficacy. When the study measures effectiveness, the results are regarded as more relevant to real-world health care decisions. In other words, this study is considered to have a higher external validity.

It’s important to keep in mind that CER does not only mean doing studies that represent the extreme range of effectiveness studies. The “art” of CER requires that researchers deliberately sacrifice some degree of internal validity in order to increase the generalizability, relevance, feasibility and timeliness of research results. The right balance of the desirable features of research cannot be determined rigorously through a scientific process. The optimal balance emerges from the process of engaging stakeholders in a discussion about each element of study design, in light of current evidence, and what sorts of decisions must be made.

Comparative Effectiveness Research Resources

Effectiveness Guidance Documents

Articles

Organizations

  • Patient-Centered Outcomes Research Institute (PCORI) is authorized by Congress to conduct research to provide information about the best available evidence to help patients and their health care providers make more informed decisions. PCORI’s research is intended to give patients a better understanding of the prevention, treatment, and care options available and the science that supports those options.
  • Center for Medical Technology Policy (CMTP) defines and publishes methodological standards and guidance for CER, develops processes and products for clinical research, and facilitates dialogue around relevant policy issues.