Measuring Effectiveness Of Health Service Interventions: Challenges And Potential Solutions
Editor’s Note
This post is part of a Health Affairs Blog short series, “Enhancing Value By Evaluating Health Care Services.” The series discusses ways to extend the use of tools for clinical and economic evaluation beyond medical technologies to the services and procedures that account for the bulk of health care spending; the goal is to create a more robust evidence base for the effectiveness and value of health care services. The posts in the series were completed with support for the authors from the Research Consortium for Health Care Value Assessment, a partnership between Altarum and VBID Health, through a grant from the Pharmaceutical Research and Manufacturers of America (PhRMA). PhRMA extended complete independence to Altarum to select researchers and specific topics. Health Affairs retained review and editing rights.
Billions of dollars are wasted each year in the United States health care system on ineffective health service interventions such as inpatient procedures and screenings. Experts often point to the use of data on clinical- or cost-effectiveness to help inform which interventions are high or low value. However, evidence-based methods to measure value are not commonly applied to health service interventions. Meanwhile biopharmaceuticals have been overrepresented in comparative- and cost-effectiveness studies as a proportion of health care spending.
The disproportionate focus on biopharmaceuticals should be surprising. The cost of inpatient and outpatient services accounts for more than 60 percent of per capita heath spending in the U.S. and over three-quarters of the difference in total health spending between the U.S. and other industrialized countries. And price of services vary substantially, mostly due to hospital market power. For example, a cesarean section is four times the price in San Francisco, CA as it is in Knoxville, TN.
However, as others have noted, the evidentiary standard for drugs is different than for other types of health care for a number of reasons. Regulatory mandates require companies to collect evidence on a biopharmaceutical treatment’s efficacy and safety to determine whether the drug’s benefits outweigh its known risks. There are established endpoints that the Food and Drug Administration and manufacturers agree to in advance of clinical trials that may lead to eventual market approval. These endpoints are often the basis or starting point for comparative effectiveness research.
By contrast, in most instances health service interventions can come to market either without regulatory approval on safety and effectiveness or without running randomized controlled trials (RCTs) to generate that evidence. Therefore, the regulatory requirements for biopharmaceuticals offer a much easier starting point to generating evidence on value.
Given the scale of spending devoted to health services, evidence generation on the comparative effectiveness of health service interventions should be a national priority. To make progress, we must understand the challenges, as well as the potential solutions, that the large field of health economics and outcomes research can employ. Challenges include defining and distinguishing effectiveness from broader concepts of quality of care; measuring health outcomes that matter most to patients; designing effectiveness studies when faced with non-random allocation of groups in clinical practice; measuring effectiveness based on existing administrative claims and electronic medical record (EMR) data; and overcoming a lack of understanding the full cycle of care from a patient perspective.
This post provides an overview of these challenges. It offers potential solutions for researchers, providers, and funders, and a path forward to better measurement of effectiveness for health service interventions.
Defining Effectiveness Of Health Services
Evaluating the broader quality of health service interventions is multi-dimensional and shaped by complex interactions between setting, personnel, accessibility, financial sustainability, and of course efficacy and effectiveness. A full discussion of all factors related to the quality of health service interventions is beyond the scope of this post. However, the initial framework for evaluating the quality of medical care from Avedis Donabedian reminds us that effectiveness, defined by production of health and satisfaction, is central to quality. Building from Donabedian’s original framework, I rely on more recent definitions that define effectiveness as related to the impact of a health service intervention (e.g., screenings, procedures) on health outcomes important to patients in real-world clinical practice settings, as opposed to ideal environments.
What Health Outcomes Matter Most To Patients?
Defining outcomes for measuring effectiveness should be an important starting point for evaluation of all interventions. Providers often optimize quality of care or process-related outcomes such as compliance with current clinical guidelines or surgical techniques. However, these provider-related outcomes are attempting to answering regulatory and sustainability questions rather than answering whether an intervention improves health outcomes for a given patient.
In recognition of the gaps in outcomes measurement that reflect what matters to patients, there are emerging efforts to redefine outcomes for measuring quality, effectiveness, and value of health care. The International Consortium for Health Outcomes Measurement (ICHOM) defines health outcomes as “the results people care about most when seeking treatment, including functional improvement and the ability to live normal, productive lives.” The National Health Council (NHC) has undertaken an initiative to ensure measurement better reflects the “broad range of impacts a disease and its treatment have on a patient’s daily life.”
Imbedded in the definitions from ICHOM and NHC are the concepts of living longer and better. But the challenges of measuring both for health service interventions becomes even more complex than for biopharmaceuticals. For example, how can we link one up-front screening procedure to long-run survival? Or assess quality-of-life changes for routine screenings or surgical procedures where functional and general health status may change over shorter or longer durations? To address these questions, we need to use advanced design methods, continue to invest in data infrastructure, and understand a patient’s journey through the health care system.
Designing Effectiveness Studies
Ideally, all evidence on effectiveness of an intervention would be based on studies in which an intervention is randomized to one group and a placebo or standard of care to another group. However, RCTs are also subject to challenges, such as limitations on generalizability and gaps in information on health outcomes or other relevant areas such as equity or patient preferences. Regardless, many health service interventions are not required to undergo rigorous regulatory approval steps, or are simply not practical in a real-world setting, so observational studies are necessary in measuring effectiveness.
Observational studies can be not only necessary but advantageous. While these studies can be plagued by design issues, potentially generating spurious results on effectiveness, the field of health services research has produced methodological solutions around most confounding or measurement problems using real world data.
Challenges In Measuring Exposures And Outcomes
While design is a crucial step in measuring effectiveness, data gaps on exposures and outcomes have received little attention in comparison. The effectiveness of a health service intervention should be based on an examination between an exposure to that intervention and the resulting health outcomes relevant to patients. However, isolating both exposures and outcomes from the broader concept of quality of care can be challenging. Specifically, in the absence of RCTs, the measurement of both exposures and outcomes often relies on data sources that reflect billed services as opposed to real-time encounters.
On the exposure side, structure and process in health care, such as staff competency or established protocols, may or may not influence the performance a particular intervention. On the outcomes side, recording encounters in hospital or outpatient settings may simply reflect regulatory requirements or what is reimbursable rather than what resources were used and outcomes achieved. For example, in a study on hospital report cards for hospital-acquired pressure ulcers, authors found hospital performance scores varied widely depending on whether administrative or surveillance data were used. Ultimately, the authors concluded administrative data were inappropriate for comparing pressure ulcer outcomes across hospitals.
Omission or underreporting of outcomes may not be the only challenge. Given that billed services are recorded as diagnostic or procedural, there is little information on severity of diagnoses and patient-reported health status, among other key sources of information that input to effectiveness estimation.
Recent advances in linking administrative claims data with EMR data and/or survey questionnaires have provided helpful measurement solutions. For example, PCORnet® has collected and linked EMR data with enrollment and claims files on over 60 million individuals. Other efforts are more in isolation, such as specific to academic medical centers. These solutions offer new opportunities for evidence generation on effectiveness of health service interventions. But even with some of these recent data infrastructure solutions that combine EMR with claims, we are still left with gaps. For example, how generalizable are exposures and outcomes from a limited group of academic medical centers that may not represent the bulk of care in the United States? Do current data infrastructure solutions have enough supplemental information to overcome the gaps left by EMR and claims? Are the data publicly available so researchers around the country can estimate effectiveness of various health service interventions? And most importantly, are we currently collecting health outcomes that matter most to patients?
The Importance Of Understanding The Full Cycle Of Care
Many of the challenges already discussed are a result of our lack of understanding around the patient journey through the health care system. We persist in asking questions such as, “How can this be made better?” referencing a quality of care question, when we should be asking, “What goes on here?” In other words, we should be measuring the full cycle of care to understand all the resources used when treating a patient up to the health outcomes ultimately achieved. More recently, Professors Kaplan and Porter have advanced this concept for costing in health care through time-driven activity-based costing (TDABC), which attempts to comprehensively understand the time, type, and quantity of resources used when treating patients. This involves mapping out the entire process of care, personnel and time involved, and equipment and space needed—all at the patient level.
A review of the literature found approximately 25 TDABC studies have been conducted on health service interventions from inpatient and outpatient surgery to psychiatry services. Results suggest TDABC helped overcome challenges with current costing methods and can inform bundled payment scenarios on an efficient budget. The field of health economics and outcomes research should borrow from these advanced costing methods to help define the scope of effectiveness measurement, including the link between exposures and outcomes. Understanding the full cycle of care should not be limited to cost, and these methods represent a great opportunity to advance the measurement of effectiveness as a supplement to TDABC.
Summary And Considerations For Future Research
Until there is more research devoted to generating data on effectiveness of health service interventions, we will continue to look under the proverbial lamppost to address rising health care costs. Addressing the gap in evidence will require greater investment in a broader scope of effectiveness research, and a shift in thinking about health service interventions as contributing to health benefits. Instead of focusing on broader quality-of-care initiatives, providers and researchers should focus on linking exposures to outcomes that truly matter to patients. This shift in thinking needs to include not only researchers, but providers and funders of research.
For researchers, we should ensure our study on effectiveness of health service interventions represents effectiveness on outcomes that matter to patients and not broader concepts of quality of care; we should acknowledge limitations about how available data sources may underreport or omit exposures and outcomes important to patients; prioritize the most important information using existing research tools such as value of information; and more broadly develop publicly available data tools and dashboards that can track the full cycle of care for patients.
For providers, defining exposures and outcomes beyond sustainability will only benefit practice and improve the health of patients. Relying on established and standardized instruments to measure health outcomes is an ongoing effort requiring collaboration with groups such as ICHOM and NHC. Further, understanding the full cycle of care requires engagement with financial teams and researchers to link past exposures with current and future outcomes.
For funders of research, continued funding for data infrastructure solutions is needed, including continued incentivization of common data models. While current efforts are promising, supplements are still needed to understand the multiple touchpoints for a patient through the health system. Linking data sources on exposures, outcomes using standardized patient health status instruments, costs, and biomarkers, among other variables, will provide fruitful research opportunities on effectiveness for decades. And publicly funded data models should be made available to qualified researchers with few barriers to access after necessary data privacy concerns are addressed.
Given the scale of spending on hospital and provider services, greater investment in proper measurement of effectiveness is urgently needed. Truly understanding effectiveness will require considerable investment and collaboration, but the benefits could help change the core of our health care system from simply producing quality metrics to producing value for patients.