key: cord-1035371-0nzltsqp authors: Messino, Paul J.; Kharrazi, Hadi; Kim, Julia M.; Lehmann, Harold title: A Method for Measuring the Effect of Certified Electronic Health Record Technology on Childhood Immunization Status Scores Among Medicaid Managed Care Network Providers date: 2020-09-12 journal: J Biomed Inform DOI: 10.1016/j.jbi.2020.103567 sha: c7e0ae150a6060bb9a599b1de430a07ee9827cda doc_id: 1035371 cord_uid: 0nzltsqp OBJECTIVE: To provide a methodology for estimating the effect of U.S.-based Certified Electronic Health Records Technology (CEHRT) implemented by primary care physicians (PCPs) on a Healthcare Effectiveness Data and Information Set (HEDIS) measure for childhood immunization delivery. MATERIALS AND METHODS: This study integrates multiple health care administrative data sources from 2010 through 2014, analyzed through an interrupted time series design and a hierarchical Bayesian model. We compared managed care physicians using CEHRT to a propensity-score matched comparisons from network physicians who did not adopt CEHRT. Inclusion criteria for physicians using CEHRT included attesting to the Childhood Immunization Status clinical quality measure in addition to meeting “Meaningful Use” (MU) during calendar year 2013. We used a first-presence patient attribution approach to develop provider-specific immunization scores. RESULTS: We evaluated 147 providers using CEHRT, with 147 propensity-score matched providers selected from a pool of 1253 PCPs practicing in Maryland. The estimate for change in odds of increasing immunization rates due to CEHRT was 1.2 (95% credible set, 0.88 to 1.73). DISCUSSION: We created a method for estimating immunization quality scores using Bayesian modeling. Our approach required linking separate administrative data sets, constructing a propensity-score matched cohort, and using first-presence, claims-based childhood visit information for patient attribution. In the absence of integrated data sets and precise and accurate patient attribution, this is a reusable method for researchers and health system administrators to estimate the impact of health information technology on individual, provider-level, process-based, though outcomes-focused, quality measures. CONCLUSION: This research has provided evidence for using Bayesian analysis of propensity-score matched provider populations to estimate the impact of CEHRT on outcomes-based quality measures such as childhood immunization delivery. In 2009, the U.S. embarked on incentivizing providers to adopt government-certified 53 electronic health record technologies (CEHRTs), aiming to reduce costs and inefficiencies in the 54 health care system, while improving quality [1] (abbreviations are listed in Supplementary 55 Materials, Table 1 ). Electronic clinical quality measures (eCQMs) were introduced along with 56 CEHRTs as an approach to quantify the improvement in clinical performance and outcomes. One administrative data is lacking [10] . HEDIS managed care health care quality measures are 70 calculated at the health plan level and not the individual provider level. [11] . 71 The literature assessing the correlation of quality measures to the use of CEHRT suggest 72 that quality improvement is more likely if the CEHRT functionality tracks closely with the 73 quality metric and if providers use the CEHRT functionality effectively. The functionalities of 74 best-practice alerts, order sets, and panel-level reporting, have led providers to receive 75 statistically significantly higher "meaningful use" quality scores related to the EHR functionality 76 [12]. These "best practice alerts," or clinical decision support (CDS), pertained to the 77 preventative services of tobacco cessation, breast cancer screening, colorectal cancer screening, 78 pneumonia vaccination, and body mass index screening [12, 13] . 79 CEHRT-based CDS focuses on prompting providers to take some type of action on 80 particular patients, usually during or immediately surrounding the patient encounter. CDS may 81 be effective as a childhood-vaccine intervention because it takes advantage of the presence of the 82 child during well-and sick-child visits to reduce the likelihood of missed opportunities to 83 administer or catch up on vaccines. How the CDS is implemented within the clinical workflow 84 can differentially impact the effect of CEHRT on vaccination rates, with some studies showing 85 improvements in vaccine rates [14, 15] while others not showing the expected improvements [16 86 With standardized Electronic Health Record (EHR) systems, Medicaid agencies will soon 87 be able to collect patient-level data by provider to compare like providers to like providers 88 longitudinally, allowing for pay-for-outcomes models. However, until the adoption of certified 89 EHRs is widespread, the integration of health-care related data sources is easier, patient 90 attribution is precise and accurate, and, electronic clinical measures become more commonplace, 91 any longitudinal analysis of quality must rely on HEDIS-like quality methodologies. Maryland Medicaid uses the HEDIS Childhood Immunization Status score, Combination 7 (HEDIS 93 immunization measure), as a standard measure to quantify immunization rates for its child 94 population. The primary objective of this study is to develop a methodology for estimating the effect This research used a retrospective, interrupted time series to assess the association 107 between CEHRT use and a provider-specific HEDIS immunization measure (Figure 1 ) [17] . The providers who adopted EHR at any point in this study. The providers who met MU in 2013 and attested to meeting NQF#0038 (the 136 "intervention group") were compared to PCPs who gave immunizations but had not adopted 137 CEHRT (the "comparison group" Because researchers and health system administrators will likely be using this method A full-variable balance plot provides a better visualization technique to assess the impact 209 of PSM on creating a suitable comparison group for analysis. Figure 3 shows the balance plot of 210 our propensity score matching and comparing each variable pre-and post-matching. Table 6 . We restricted each provider in each calendar year to a primary group as providers 305 practice with many groups, and because this analysis recognizes that the intervention group is The relationship between the data used for this analysis is complex due to its hierarchical 331 structure ( Figure 4) . The data includes three nested levels: groups, providers, and time. Providers We specified hierarchical models, expressed as Bayesian analysis Using Gibbs Sampling 345 (BUGS) models and non-informative conjugate priors [28] . A non-informative conjugate prior is 346 a higher-level prior distribution that is within the same family as the underlying distribution it is 347 feeding within the model, but its distribution is specified to be vague [29] . Using BUGS, the 348 model focused on provider vaccination rates as affected by base vaccination rates, behavior of 349 the rest of the provider's group, and other relevant covariates. The second "level" was the 350 difference in vaccination rates across the years, which in turn is a function of CEHRT status. Likelihood was derived by Equation (2), which states that the likelihood of theta, the 352 parameter you care about, given y, the data you observed, is the product of each of the 353 probabilities of y from y 1 to y n , given theta, a probability whose expression is known (e.g, To estimate the effect of CEHRT on HEDIS immunization measure, we monitored the Figure 1 for the trace and kernel density plots. As shown in Table 2 , 413 comparing the intervention group to its comparison group showed a 21% estimated improvement 414 in the odds of meeting HEDIS immunization measure (Combination 7) due to EHR use. This study has some limitations: (1) As with all studies that utilize PSM to develop 510 comparison groups, this study assumes non-ignorability or selection-on-observables to define 511 covariates for propensity score matching. The authors attempted to mitigate this weakness by The authors declare no conflicting interests. Childhood Immunization Schedule; Board on Population Health and Public Health Practice The Childhood Immunization Schedule and Safety Priorities among effective clinical 562 preventive services Timeliness of childhood vaccinations in the 565 United States: days undervaccinated and number of vaccines delayed Timeliness of 569 childhood immunizations National, state, and selected local area 572 vaccination coverage among children aged 19-35 months -United States Annual Report on the Quality of Care for Children in Medicaid and CHIP 576 Health and Human Services Secretary National Committee for Quality Assurance. States Using NCQA Programs Using the Hybrid Method to Calculate 585 Measures for the Child and Adult Core Sets. Technical Assistance Brief The Search for Value in Health Care: A Review of the National 590 Committee for Quality Assurance Efforts Associations between healthcare quality and use of electronic health record functions in 594 ambulatory care Electronic medical record reminders and 597 smoking cessation activities in primary care Utilizing health information technology to improve vaccine 601 communication and coverage Impact 605 of Clinical Alerts Within an Electronic Health Record on Routine Childhood Effect of Provider Prompts on Adolescent Immunization Rates: A Randomized 611 Use of interrupted time series analysis in evaluating health care 614 quality improvements R: A language and environment for statistical computing Provider characteristics and behaviors as 622 predictors of immunization coverage Medicare's Physician Quality Reporting System 625 (PQRS): quality measurement and beneficiary attribution Primary care physicians' links to other 629 physicians through Medicare patients: the scope of care coordination Variable Selection for Propensity Score Models The Central Role of the Propensity Score in 637 Observational Studies for Causal Effects A Two-Step Bayesian Approach for Propensity Score Analysis: 640 Simulations and Case Study Causal inference in statistics : a primer Bayesian analyses: Where to start and what to report, European 647 Basic Bayesian Methods Bayesian Graphical Models using MCMC The BUGS project: Evolution, critique 661 and future directions CODA: Convergence Diagnosis and Output 664 Analysis for MCMC Matching as nonparametric preprocessing for reducing 667 model dependence in parametric causal inference Bayesian Data Analysis A Bayesian χ 2 test for goodness-of-fit Maturation of Electronic Health Record Functions Among US Hospitals: Retrospective Analysis 680 and Predictive Model Task Force on the status and future direction of EHRs Assesing electronic health record 687 implementation challenges using item response theory A Population Health Measurement 691 Framework: Evidence-Based Metrics for Assessing Community-Level Population Health in the 692 Global Budget Context, Popul Health Manag A proposed national research and development 695 agenda for population health informatics: summary recommendations from a national expert 696 workshop IT-enabled Community Health Interventions: Challenges Comparing Population-based Risk-stratification 702 Model Performance Using Demographic, Diagnosis and Medication Data Extracted From 703 Outpatient Electronic Health Records Versus Administrative Claims Evaluating the Impact of Prescription Fill 707 Rates on Risk Stratification Model Performance Assessing markers from ambulatory 711 laboratory tests for predicting high-risk patients Assessing the Impact of Body 714 Mass Index Information on the Performance of Risk Adjustment Models in Predicting Health 715 Care Costs and Utilization The Impact of Social Determinants of Health on 719 Hospitalization in the Veterans Health Administration The Association Between Neighborhood 723 Socioeconomic and Housing Characteristics with Hospitalization: Results of a National Study of 724 A public health perspective on using electronic health 727 records to address social determinants of health: The potential for a national system of local 728 community health records in the United States Prospective EHR-based clinical trials: the challenge of 732 missing data Challenges in data quality assurance for electronic health 735 records, Stud Health Technol Inform A State-wide Health IT Infrastructure for 738 Population Health: Building a Community-wide Electronic Platform for Maryland's All-Payer 739 Global Budget Assessing electronic health record 742 implementation challenges using item response theory Public Health and Epidemiology Informatics: Recent 746 Research and Trends in the United States What's Past is Prologue: A 750 Scoping Review of Recent Public Health and Global Health Informatics Literature Public and Population Health Informatics: The 754 Bridging of Big Data to Benefit Communities Effects of the COVID-19 Pandemic on 758 Routine Pediatric Vaccine Ordering and Administration -United States Conceptualization, Methodology, Software, Formal analysis, Investigation, Data Curation Writing-Original Draft ☒ The authors declare that they have no known competing financial interests or personal relationships 770 that could have appeared to influence the work ☐The authors declare the following financial interests/personal relationships which may be considered 773 as potential competing interests  Reusable method for estimating impact of CEHRT on health care quality Method applies Bayesian analysis, estimates impact of CEHRT on immunization scores 784  No statistically significant difference in the odds of meeting immunization score