key: cord-0944387-ucdg18of authors: Byron, Sepheen C; Roth, Lindsey; Acton, Ryan M; Shen, Angela title: Harnessing electronic clinical data to report adult and prenatal immunization quality measures date: 2021-07-19 journal: J Am Med Inform Assoc DOI: 10.1093/jamia/ocab125 sha: 18d46a5bc94ceb3f34c0c752d3315ed1d058626d doc_id: 944387 cord_uid: ucdg18of OBJECTIVE: To explore the use of health plan quality measures specified for electronic clinical data to monitor immunizations. MATERIALS AND METHODS: We analyzed 2018 data submitted by health plans reporting 2 new Healthcare Effectiveness Data and Information Set measures assessing receipt of clinically recommended vaccines among pregnant women and adults. We analyzed the number of plans reporting a valid performance rate and electronic data source used. We consulted expert panels and reviewed coverage rates from other sources to understand the results. RESULTS: We received 136 data submissions across commercial, Medicaid and Medicare plans and 87 submissions across commercial and Medicaid plans for the adult and prenatal immunization measures, respectively. These submissions represent approximately 15% of possible submissions. Plans used claims, registries and electronic health records. Mean performance rates for adult immunizations were 21.2 (commercial), 14.0 (Medicaid) and 19.5% (Medicare). Mean rates for prenatal immunizations were 33.1 (commercial) and 16.7% (Medicaid). DISCUSSION: Results from the first year of reporting 2 electronic clinical data measures suggest health plans can feasibly report these measures and are seeking electronic data to supplement claims. Comparison of rates to other national results showed lower than expected rates for the adult immunization measure. However, prenatal immunization rates were on par with those from a national survey, suggesting this measure is closer to use for quality improvement. CONCLUSION: Quality measure reporting that encourages connection to electronic data sources is a step forward in performance monitoring and improvement. The use of electronic sources may advance health information exchange for patient care. Quality measures are tools for understanding the performance of the health care system on important dimensions of care and can promote adherence to evidence-based practices. [1] [2] [3] [4] Measures can be particularly effective when used within mechanisms that provide explicit incentives for high performance, such as value-based purchasing and pay-for-performance initiatives. [5] [6] [7] The Healthcare Effectiveness Data and Information Set V R (HEDIS) is a national measurement set that assesses how well Medicare, Medicaid, and commercial health insurance plans manage the care of their enrolled populations. 8 HEDIS assesses performance using a set of metrics that range from preventive services to behavioral health care and chronic disease management. In 2018, HEDIS added 2 new measures that assess vaccinations in adults and pregnant women using a new method that better incorporates electronic clinical data. The Adult Immunization Status (AIS) and Prenatal Immunization Status (PRS) measures assess whether adults and pregnant women, respectively, received routine vaccines according to clinical guidelines. Health plans are ideally situated to monitor immunizations given the numerous venues through which adults receive vaccines (eg, physician offices, retail pharmacies, the workplace). Health plans can aggregate this information across these diverse provider locations and use this information to assess coverage and target quality improvement efforts. The electronic data method encourages more structured data capture and sharing across systems that have been built for immunizations monitoring, such as immunization information systems (IIS). These considerations make adult immunization measures an ideal test case for understanding how quality measures may facilitate and encourage the flow of electronic clinical data across providers and plans and vice-versa-and how multiple sources of information can increase robustness of reporting of vaccination uptake. This article describes the first year of reporting for these 2 HEDIS measures specified for structured electronic clinical data reporting. Immunization rates among adults are lagging Immunizations are among the most effective primary preventive services and are responsible for substantial declines in morbidity and mortality across vaccine-preventable diseases. 9 The Advisory Committee on Immunization Practices (ACIP) recommends 4 routine vaccines for the general adult population. 10 Adults aged 19 years and older are recommended to receive influenza vaccine annually and tetanus, diphtheria, and pertussis (Tdap) or tetanus and diphtheria (Td) vaccine every 10 years. Adults age 50 years and older are recommended to receive herpes zoster vaccine, and adults age 65 years and older are recommended to receive pneumococcal vaccine. Pregnant women are a priority population to receive influenza and Tdap vaccines. Immunization of pregnant women delivers protective antibodies transplacentally to the fetus and confers protection in infants too young to receive certain childhood immunizations. 11 ACIP recommends influenza vaccination at any time during pregnancy. Tdap vaccine is recommended for each pregnancy, ideally between 27-and 36-weeks' gestation in order to maximize passive antibody transfer to the infant. 12, 13 Despite these recommendations, immunization rates remain low, and recent coverage estimates fall well below national goals. 14 For example, in 2017, less than half of adults reported receiving the Tdap vaccine; the number was as low as 25% in some states. 15 For the 2017-2018 influenza season, only 37% of adults reported receiving an influenza vaccine. Among adults age 18-49 years, uptake was as low as 27%. 16 In 2017, among pregnant women, estimated rates were 50% for administration of both influenza and Tdap vaccines. 17 These gaps in care can result in significant morbidity and mortality from vaccine-preventable illnesses. Since 2010, influenza has resulted in 140 000 to 960 000 hospitalizations annually. 18 During the 2017-2018 season, 10% of deaths across all ages were attributable to pneumonia and influenza. 19 Declines in herd immunity have more far-reaching ramifications. For example, pertussis has recently increased among children despite high rates of pediatric vaccine uptake-likely due to increased transmission through adults. 20 This is particularly concerning, as pertussis-related mortality is highest among newborns. 21 Given persistently low rates of vaccinations among adults, national efforts were enhanced in recent years to improve uptake of adult vaccines. 22 In 2016, the National Vaccine Program Office within the US Department of Health and Human Services released a National Adult Immunization Plan, which aimed to promote coordinated action across all stakeholders in order to guide implementation of adult immunization standards across the US. 23 Specific priorities include evaluating and advancing targeted quality improvement initiatives and supporting the infrastructure for improved monitoring, such as use of electronic health records (EHRs) and IIS. IIS are population-based repositories that record all immunization doses administered by participating providers to persons residing within a specified geographic area. 24 Quality measures in particular were named as a means to assess, monitor, and drive up vaccination coverage rates. Given the ardent use of measures to demonstrate quality and value, much work has been undertaken to improve the data from which measures draw. Traditionally, measures are calculated using administrative data alone or supplemented with manual medical record review of a sample of the target population when needed. In the context of health care quality measures, administrative data include health insurance claims, enrollment, and other transactional data used for business purposes. The merits and pitfalls of these methods have been studied extensively. [25] [26] [27] [28] [29] In general, administrative data are feasible yet limited in the types of clinical quality concepts they can measure; medical record review enables measurement of concepts of higher interest yet is resource intensive given the time needed to obtain charts, manually review and retrieve information, and interpret nonstandard information. 25, 30 Despite limitations, these 2 data collection methods serve as the basis for measures in many of the major performance monitoring programs, including HEDIS. 7, 8, 31 Recognizing the constraints of the current methods, and coupled with increased access and availability to electronic sources, HEDIS added a new reporting method that more formally incorporates electronic clinical data into quality measures. Measures that use the HEDIS Electronic Clinical Data Systems method draw from data found in health plan claims and enrollment files, EHRs, case management systems, and clinical registries (including IIS). 32 In order to qualify for use as a source of data for reporting, data must reside in structured fields, which allow the data to be queried automatically. The goal of this reporting method is to move away from manual and time-consuming processes, such as medical record reviews, and to encourage better recording, sharing, and use of electronic clinical data. A requirement of the HEDIS electronic data reporting method is that plans must ensure care teams can access data at the point of care-a key principle of sound clinical decision support. The 2 immunization-focused measures recently added to HEDIS are specified for the Electronic Clinical Data Systems reporting method. In order to understand the feasibility of incorporating such data for immunizations reporting, this study assesses the results of health plans' use of the Electronic Clinical Data Systems method to assess vaccination coverage among adults and pregnant women. We conducted an analysis of data submitted for AIS and PRS after the measures' first year of reporting. Data represent a snapshot of care using information gathered in the 2018 measurement year. To calculate the measures, health plans assess the number of enrolled members who meet eligibility criteria. For AIS, adults 19 years of age and older who are enrolled for the duration of the 1year measurement period in commercial, Medicaid, or Medicare plans are eligible for the measure. The PRS measure uses deliveries as the unit of measurement. For each delivery, the woman must be enrolled in a commercial or Medicaid plan for a minimum of 28 days prior to delivery through the delivery date to be eligible for the measure. Deliveries in which the woman delivered prior to 36 weeks gestation are removed. This exclusion ensures providers have optimal opportunity to provide vaccinations according to guidelines, which recommend vaccinating for Tdap at 27-36 weeks gestation. 13 These criteria define the measures' denominators. For AIS, adults are numerator compliant if documentation indicates the required (age-determined) vaccines were received according to the time interval specified in the measure, which aligns to ACIP guidelines. For example, the Tdap rate assesses whether adults age 19 and older received at least 1 Td/Tdap vaccine within the last 10 years. The measure calculates a rate for each of the 4 recommended vaccines. For PRS, deliveries are assessed for whether the woman received influenza and Tdap vaccinations during the pregnancy period. Annually, HEDIS measures are calculated and aggregated by health plans, validated by certified auditors, and submitted to the National Committee for Quality Assurance via an online data submission portal. Data for measures using the Electronic Clinical Data Systems method are reported by data source, which allows for an analysis of reporting by type of data. Health plans follow a mutually exhaustive hierarchy when categorizing numerator events by data source, which restricts each numerator event to be associated with only 1 data source. The hierarchy starts with point-of-service data: health plans are directed to first search for numerator events in EHRs, move to health information exchanges/clinical registries/IIS, then to case management registries and, finally claims. Rates are considered reportable if they meet measure specifications as well as HEDIS programmatic guidelines. For each measure, we assessed feasibility through an analysis of the submitted data. Feasibility in this case refers to whether a sufficient number of health plans are able to report a measure as specified. It is informed by whether plans have sufficient numbers of members eligible for the service or intervention (ie, denominator) and whether a plan can identify that the service or intervention was provided (ie, numerator). For this analysis, we determined feasibility by assessing the number of submitting health plans with a reportable rate. Reportable rates were defined as those that were deemed free of audit concerns and met a minimum denominator criterion of 30 members (for AIS) or 30 deliveries (for PRS). We assessed face validity by reviewing our findings with a multistakeholder expert panel. The panel consisted of representatives from academia, state and federal users, health plans, clinician groups, immunization data experts, and patient advocacy groups. The multistakeholder nature of the panel ensures a variety of per-spectives are represented, which supports measurement that balances the desirable attributes of relevance, scientific soundness, and feasibility. 30 In addition, we describe the performance rate by type of health plan (commercial, Medicaid, Medicare, as applicable) and the type of data source (claims alone compared to claims supplemented with additional electronic data sources). To determine whether differences in performance were statistically significant, we calculated independent sample t-tests of rates across plan type and data-source type. To gain a sense of whether performance rates reflect expected vaccine coverage, we examined rates from other measures and national coverage studies. For AIS, we reviewed rates calculated using the HEDIS Influenza and Pneumococcal Vaccination CAHPS V R measures , 33 the Centers for Disease Control and Prevention (CDC) National Health Interview Survey (NHIS), 34 and the Indian Health Service Adult Immunization Composite measure. 35 For PRS, we reviewed rates reported from a CDC Internet panel survey of pregnant women. 17 Rates of other measures or studies are not a direct comparison, as they use different specifications and data collection methods. However, these rates provide a rough approximation to help us understand whether the AIS and PRS rates are performing as one might expect. The AIS measure received a total of 136 submissions across commercial, Medicaid, and Medicare lines of business. The PRS measure received a total of 87 submissions across commercial and Medicaid lines of business. These submissions represent approximately 15% of expected submissions for these measures. We estimated expected submissions by ascertaining the number of submissions received in the same year for HEDIS measures applicable to similar populations. The number of members meeting measure denominator criteria among submitting plans are described in Table 1 Overall, performance rates were wide-ranging depending on the vaccine and product line. Among the general population of adults, average performance was lowest for receipt of zoster vaccine. As few as 1.6% of Medicaid adults were documented as having received the zoster vaccine. Average performance of health plans was highest for receipt of Tdap/Td vaccine. Across commercial, Medicaid, and Medicare plans, approximately 25% of adults were documented as having received the Tdap/Td vaccine. Among deliveries, fewer pregnant women were documented as having received influenza compared to Tdap, with results much lower across Medicaid plans compared to commercial plans. Distribution of performance rates is reported in Table 2 . Most reporting health plans used multiple electronic clinical data sources to calculate the 2 measures. We used violin plots to illustrate the reporting patterns by data source profile for AIS ( Figure 1 ) and PRS (Figure 2) . A violin plot is a mirrored, smoothed probability density plot and can be a helpful way to visualize a distribution of values. Convex areas of the plot indicate a higher frequency of values in that area; concaved areas indicate a relatively lower frequency of values in that area. 36 In general, plans that sought data from sources beyond claims had higher performance rates compared to plans that reported using solely claims. For example, Medicaid plans that reported PRS using claims data alone had a mean performance rate of 19%; those using claims data plus additional electronic clinical data sources had a rate of 30%. Further, plans reporting using claims data alone exhibited a narrower range of performance rates compared with plans reporting using additional data sources. T-test results showed that, by and large, the observed differences in performance were generally significant at the P < .05 level. We assessed the hypothesis that Medicaid plan performance was lower than commercial plan performance. We also assessed the hypothesis that claims-only performance was lower than performance when claims were supplemented with additional electronic clinical data. Both hypotheses held true with a few exceptions where rates were nearly identical (see Supplementary Tables S1-S2) . When comparing AIS performance rates to those calculated for other national measures that use different methods and data sources, AIS measure rates were lower. For example, the mean influenza rate for commercial plans was 18.7% compared with a coverage rate of 45.4% reported from the 2017 NHIS. The Tdap rates were closer, with an average performance of 29.4% among commercial plans versus 31.7% as measured by the NHIS. For zoster, the rate was 6.1% on average among commercial plans versus 34.9%. For pneumococcal vaccine, the rates were 20.3% from AIS compared to 69% from the NHIS. Average performance rates reported by commercial health plans for the PRS measure were more similar to prenatal immunization rates reported in national studies. For influenza vaccination, the mean PRS rate was 41%; for Tdap vaccination, the rate was 63%. In a CDC Internet Panel Survey in 2018, the rates were 55% and 59% for commercially insured women. Our multistakeholder panels concluded the performance results for AIS were lower than expected based on published studies and expert knowledge. They noted that the low rates of performance in light of what we would generally expect suggest health plans are challenged by the difficulty of retrieving data. Our expert panels were more encouraged by the PRS measure results, which approximated expected rates based on the literature and expert knowledge. This analysis demonstrated that quality measures that incorporate standardized electronic clinical data hold promise. Based on the first year of reporting for 2 new immunization measures that use struc- tured data from electronic clinical data systems, health plans are beginning to seek information beyond claims to understand immunization coverage across their adult and prenatal populations. Given the available method and platform to do so, plans can leverage available data to better understand gaps in care among their members and use these data to increase utilization of meaningful services. While the majority of data used to calculate the AIS and PRS measures came from claims, many plans sought information from additional elec-tronic clinical data sources, which improved performance rates. About 40% of Medicaid and Medicare plans used additional data sources beyond claims to report the measure; nearly 80% of commercial plans did so. There are several limitations to these findings. First, as noted, the total number of submissions for these measures represents approximately 15% of possible submissions. Thus, the results are based on a limited number of reporting plans. For example, plans submitting measures using this reporting method may be more advanced in their electronic data capture capabilities. If so, our results may overestimate health plans' ability to access electronic data. Nevertheless, the results represent a first look into health plans' use of structured electronic data beyond claims and are encouraging-particularly in light of the voluntary nature of these measures, which are not yet publicly reported or required in evaluation programs. Related, the number of Medicaid plans submitting the measures was far fewer than commercial and Medicare plans, which may account for the lower Medicaid performance results. Early indicators of HEDIS reporting show that the number of Medicaid plans reporting both AIS and PRS is increasing 3-fold. It is possible that Medicaid plans may be on a slightly slower pace due to more limited resources, but that these plans will begin reporting on par with commercial and Medicare plans. Last, the results are based on an assumption that health plans with a higher rate are performing closer to a "true" vaccination rate, and that this higher performance is due to use of data sources beyond claims. However, it is possible that health plans using claims alone are performing at their highest level, and that supplementation with additional electronic clinical data sources would not improve the rate. This is an observation that is relevant across quality measures, which may reflect dimensions of quality other than care receipt, such as data sources, workflow, documentation practices, or infrastructure. Related, when comparing rates of AIS and PRS to other national measures, it is assumed that these rates should be approximately similar. However, there may be legitimate reasons for differences that go beyond data sources, such as sampling and other methodological attributes. While a true vaccination rate may not be knowable, the measures may still serve as an important tool for monitoring improvement and encouraging the use and sharing of electronic clinical data to the benefit of patients. David Blumenthal, when serving as the director of the federal Office of the National Coordinator for Health Information Technology, noted, "Information is the lifeblood of modern medicine. Health information technology is destined to be its circulatory system." 37 The electronic data connections demonstrated by the HEDIS immunization measures, while nascent, are critical to the improvement of patient care and long overdue. For example, the use of gestational age as a quality measure data element encourages recording of this information in a structured data field-a standard that could vastly enhance the ability to measure other aspects of health care for women in the perinatal stage. Measures assessing the receipt of immunizations among adults, including pregnant women, provide a case for the use of electronic clinical data for quality measurement, which in turn can increase compliance to evidence-based clinical preventive services. This analysis demonstrated that measures that use such data are feasible and that health plans are seeking information beyond claims to understand immunization coverage. This is important, as the adult schedule for routinely recommended vaccines extends across a number of years. Tdap/Td, for example, is recommended every 10 years --a time span that extends far beyond the information most health plans would have in traditional claims-based data sources. Additionally, many adults receive their vaccines at different locations and from a wide range of providers. Influenza vaccine is given across settings that may fall outside the typical range of data sources sought out by health plans. The ability to share data for quality measurement and improvement presents a business case for continuing to build the infrastructure for data sharing and interoperability among data sources (eg, between a provider's EHR and the IIS). Findings from this analysis suggest that health plans are beginning to build the connections needed to retrieve and share information critical to supporting immunizations uptake. Results were lower than expected for adult immunizations compared to other national data sources and judged by expert panels, suggesting the measure may be reflecting continued data access issues. However, the results for prenatal immunizations approached national coverage rates in the literature. Based on the robustness of the findings, the Prenatal Immunization Status measure was incorporated into HEDIS public reporting in measurement year 2020. 38 Public reporting, in addition to the use of these measures in reporting programs, should bolster the case for continued investment in the infrastructure to support immunizations monitoring-infrastructure that will be critical, particularly in the face of the COVID-19 pandemic and for monitoring the mass vaccination efforts underway. This work was partially funded by the National Vaccine Program Office, Department of Health and Human Services (contract numbers HHSP233201700024C and HHSP233201600063C). SB, LR, RA, and AS contributed to the conception of the work. RA provided the data analyses. All authors contributed to drafting the article, critical revision of the article, and final approval of the version to be published. The data underlying this article cannot be shared publicly, as they are based on testing data that has not been approved for public reporting by the National Committee for Quality Assurance. A framework for assessing the performance of health systems Developing a clinical performance measure Using performance measurement to drive improvement: a road map for change Introduction and overview of the conceptual framework for a national quality measurement and reporting system What role does performance information play in securing improvement in healthcare? A conceptual framework for levers of change Centers for Medicare & Medicaid Services. About the Marketplace Quality Initiatives. Centers for Medicare & Medicaid Services Quality Initiatives HEDIS Measures and Technical Resources. National Committee for Quality Assurance Impact of vaccines universally recommended for children-United States Advisory committee on immunization practices recommended immunization schedule for adults aged 19 years or older-United States Vaccines in pregnancy: the dual benefit for pregnant women and infants Prevention and control of seasonal influenza with vaccines: recommendations of the advisory committee on immunization practices-United States, 2019-20 influenza season Prevention of pertussis, tetanus, and diphtheria with vaccines in the United States: recommendations of the Advisory Committee on Immunization Practices (ACIP) US Department of Health and Human Services. Immunization and Infectious Diseases. Healthy People Adult Vaccination Coverage General Population Report. Centers for Disease Control and Prevention AdultVaxView General Population Vaccination Coverage: Influenza. Centers for Disease Control and Prevention Flu-VaxView Influenza and Tdap vaccination coverage among pregnant women-United States ?CDC_AA_refVal¼https%3A%2F% 2Fwww.cdc.gov%2Fflu%2Fabout%2Fdisease%2Fburden.htm. Accessed Update: influenza activity in the United States during the 2017-18 season and composition of the 2018-19 influenza vaccine Prevention of pertussis through adult vaccination Trends in pertussis among infants in the United States A pathway to developing and testing quality measures aimed at improving adult vaccination rates in the United States National Vaccine Program Office. National Adult Immunization Plan. US Department of Health and Human Services About Immunization Information Systems. Immunization Information Systems Comparison of administrative-only versus administrative plus chart review data for reporting HEDIS hybrid measures Treatment patterns for prostate cancer: comparison of medicare claims data to medical record review Administrative data for quality improvement Assessing quality using administrative data Identification of metastatic cancer in claims data Developing measures for pediatric quality: methods and experiences of the CHIPRA pediatric quality measures program grantees Quality of Care Performance Measurement HEDIS Electronic Clinical Data System (ECDS) Reporting. HEDIS and Performance Measurement Specifications for Survey Measures Vaccination Coverage among Adults in the United States, National Health Interview Survey Evaluation of an adult immunization composite measure in the 101: Visualizing Distribution and Probability Density. Mode National Committee for Quality Assurance. ECDS News: Public Reporting for Measurement Year 2020. HEDIS Electronic Clinical Data System (ECDS) Reporting Supplementary material is available at Journal of the American Medical Informatics Association online. Sepheen Byron, Lindsey Roth and Ryan Acton are employees of the National Committee for Quality Assurance (NCQA), which owns and stewards HEDIS. However, analyses of measures and their use in NCQA programs are overseen by an independent, multistakeholder voting panel. Angela Shen has no competing interests.