key: cord-0700692-06y8w7ht authors: Baxter, Sally L; Apathy, Nate C; Cross, Dori A; Sinsky, Christine; Hribar, Michelle R title: Measures of electronic health record use in outpatient settings across vendors date: 2020-11-19 journal: J Am Med Inform Assoc DOI: 10.1093/jamia/ocaa266 sha: 090cbec57b76f0b21e293dc70964b63c6536e128 doc_id: 700692 cord_uid: 06y8w7ht Electronic health record (EHR) log data capture clinical workflows and are a rich source of information to understand variation in practice patterns. Variation in how EHRs are used to document and support care delivery is associated with clinical and operational outcomes, including measures of provider well-being and burnout. Standardized measures that describe EHR use would facilitate generalizability and cross-institution, cross-vendor research. Here, we describe the current state of outpatient EHR use measures offered by various EHR vendors, guided by our prior conceptual work that proposed seven core measures to describe EHR use. We evaluate these measures and other reporting options provided by vendors for maturity and similarity to previously proposed standardized measures. Working toward improved standardization of EHR use measures can enable and accelerate high-impact research on physician burnout and job satisfaction as well as organizational efficiency and patient health. Although electronic health records (EHRs) have been widely adopted, 1,2 their use has been associated with decreased physician satisfaction, decreased productivity, and burnout. [3] [4] [5] [6] [7] Single-center studies concerning physicians' EHR use and attendant burden are difficult to generalize. 8, 9 To address this challenge, some studies have used audit logs from a single vendor to study EHR use across institutions. 4 ,10 Yet the burgeoning literature concerning EHR audit log analysis remains somewhat limited in generalizability due to variations in measurement. 9 Absent standardized measures of EHR use that can be utilized across vendor platforms and institutions, broad conclusions regarding the impact of EHRs on burden and burnout will remain elusive. In an effort to improve standardization, core EHR use measures using log data were recently proposed for outpatient physicians (Table 1) . 11 These measures capture major domains of EHR use that impact clinical outcomes and physician satisfaction. 3, [5] [6] [7] Some EHR vendors provide "off-the-shelf" EHR use measures for their clients. 12 These are widely scaled solutions that offer opportunity to learn across healthcare settings, but concerns have been raised regarding vendor-provided measure validation, across-vendor inconsistencies in task and measure definitions, and within-vendor changes to measurement methodologies over time. 13 This study's purpose was to describe the current state of vendor-provided outpatient EHR use measures. Understanding vendors' development of these measures can help inform the ongoing process of developing standardized measures. Based on internal data from the Office of National Coordinator for Health Information Technology, we included the following vendors with leading outpatient market share in the USA: Epic Systems V R (Verona, WI), Cerner Corporation V R (Kansas City, MO), AllScripts Healthcare Solutions V R (Chicago, IL), eClinicalWorks V R (Westborough, MA), athenahealth V R (Watertown, MA), and NextGen Healthcare V R (Irvine, CA). We asked vendor representatives about the availability of EHR use measures from December 2019 to May 2020. For vendors with available measures, we collected data regarding measure definitions via semistructured interviews with representatives, including technical personnel directly involved in developing and implementing off-the-shelf measures. Interviews involved exchanges between two or three members of the vendor team and two or three members of our research team. For Epic and Cerner, where we had direct user access (eg, via clinical practice at our affiliated health systems), we also explored the vendor-provided EHR usage platforms directly. We describe vendors' individual approaches to measuring development and operationalization and assessing alignment to the previously proposed measures 11 to inform ongoing standardization efforts. Vendors vary in the maturity of development of outpatient EHR use measures ( Figure 1 ). NextGen has not developed any programs to accumulate usage data. Although eClinicalWorks makes EHR use metadata accessible for development of custom measures by thirdparty vendors, it does not provide any native measures. athenahealth is in the process of developing measures, but these are not yet available. The remaining vendors (Epic, Cerner, and AllScripts) have developed EHR use measures in platforms available to clients. AllScripts offers an analytics platform based on patient cohorts ("population sets") to generate EHR utilization reports. The population sets include broad cohorts (eg, all visits, inpatients, outpatients) and complex cohorts with specific definitions, often based on client requests (eg, sepsis, type 1 diabetics admitted with ketoacidosis). Utilization reports are generated for a given population set and include the following domains: clinical documentation, computerized physician order entry, order set utilization, alerts, knowledge-based medication administration, and "EHR audit reports." Within these domains, user reports can count both number of activities and time Although Cerner does not directly provide this measure to users, a dashboard dedicated to ordering and personnel does include the numbers of orders placed by providers and care team members, via computerized physician order entry (CPOE) and other means. This dashboard contains the values necessary to compute the percentage of orders with team contribution. expenditure per activity. Visualization options include an "event timeline by user" that displays event actions and times for individual physicians, and-at the site or system level-an event density map that depicts the distribution of EHR tasks across users over time. Cerner provides EHR use measures via the Lights On platform, 12 which summarizes commonly reviewed measures in a dashboard and benchmarks user metrics to national specialty metrics. Measures are calculated at the user, specialty, facility, and health system levels, and aggregated to monthly averages by default. Lights On can be configured to record measures across outpatient, inpatient, and emergency department venues, with measures spanning 42 domains (eg, alerts, documentation, orders). Lights On provides broad measures of EHR use within each domain, such as types and counts of activities (eg, number of documents signed), proportions of clinical activity meeting certain criteria (eg, percentage of prescriptions sent electronically), time associated with EHR use activities (eg, afterhours work, defined as time spent in the EHR before 6 AM, after 6 PM, and on weekends), and how frequently clinicians use recommended workflows. These measures are normalized to a per-patientseen denominator, measured as the number of unique patients for whom a provider signed a clinical note during the time period. Measure data in Lights On are available at daily, weekly, monthly, quarterly, and yearly levels, and users can download historical monthly data for all measures. Users can visualize measures over time at different levels of aggregation, and Lights On includes functionality to perform ad hoc pre/post analyses of the impact of system changes (eg, upgrades or workflow configuration changes) on use measures at more granular time intervals. Users can access definitions of measures via help links within Lights On, and Cerner maintains wiki-style resource pages with additional details of measure methodology. Epic provides EHR use measures via Signal, a platform providing measures in the following domains: in-basket, orders, notes, workload, clinical review, and "other" (assorted measures such as total time in system per day and time in schedule per day). Measures are generated at the provider level and can be aggregated to the specialty, department, or health system levels. Some measures are normalized with different denominators (eg, time in notes per day and per appointment). Several measures describe after-hours work, such as time outside scheduled hours (with 30-min buffers before the first appointment and after the last appointment), time on unscheduled days with no appointments, time outside 7 AM to 7 PM, and "pajama time," which is time outside 7 AM to 5:30 PM on weekdays and time on weekends. Measures are calculated based on monthly reporting periods. Measures can be displayed longitudinally over the course of a workday or month-to-month and are compared with averages of all Epic users of the same specialty. Besides time-based measures, Epic also offers measures of use of personalization/efficiency tools. Detailed measure definitions are documented in a library. Currently, this platform is available for only outpatient attending physicians and not other physicians (eg, inpatient providers, residents/fellows) or clinician types (eg, nurses). There are several differences between vendor-provided measures and proposed measures. Total EHR time and time on documentation, prescriptions, inbox, and outside work are all variably defined. For example, vendors' measures normalize to per-patient or perday, whereas the proposed measures suggest normalizing to 8 hours of scheduled patient time. There are also varying definitions of the numerator for "work outside of work." For documentation, vendors also are more likely to split subtasks (ie, separating clinical review and note-writing) rather than aggregating them like the proposed measure. For inbox-related work, vendors measure time strictly in the inbox, whereas the proposed measure would also include time on immediate actions related to inbox messages. Finally, undivided attention is unavailable across all vendors. We assessed vendor-provided EHR use measures and compared them with measures recently proposed by a multidisciplinary informatics workgroup. 11 Our key findings were: (1) several vendors were in early stages of measure development; (2) differences between vendor-provided measures and proposed measures carry important implications for interpretation and cross-vendor comparison; and (3) ongoing work is needed to improve standardization. First, although three vendors offered measures and welldeveloped platforms, the remaining vendors have not developed measures or are still developing them. This represents an opportunity for early engagement regarding measure definitions and standardization. Even for vendors with existing platforms, there are some domains (eg, undivided attention) that are still undeveloped, again representing an opportunity for early alignment with vendors as measure libraries expand. Second, differences in measure definitions have several important implications. Vendors often normalize measured time as per patient or per day. However, per-day calculations do not account for variations in physician schedules, ie, half-day vs full-day clinics. In contrast, the proposed measures normalize to 8 hours of scheduled patient time. This enables a more generalizable comparison across variable scheduling templates, but does not account for varying clinical volumes. Both time and clinical volume are important domains for measure standardization. Future work and engagement with physicians and healthcare organizations can inform how best to incorporate clinical volume into an expanded set of standardized measures. Difficulty of standardization is readily apparent in work outside of work measures. Variations across vendors illustrate the difficulty of measuring this construct, which is particularly important given that after-hours EHR use has been associated with physician burnout. 3, 5, 14 Vendors' calculations may vary depending on whether physicians had multiple shorter clinic sessions on different calendar days vs a longer clinic session on a single day. For example, vendorprovided measures subtract buffer periods before and after clinic hours from work outside of work time, which is arbitrary and may not reflect the scheduling practices or preferences of specific clinicians. In contrast, the proposed measure is normalized to 8 hours of scheduled patient time without any subtraction of time. However, the proposed measure could overestimate work outside of work if the physician is still seeing patients in the clinic past the time of the last scheduled appointment. This may happen when clinics run long or when patients have multistep appointments that include intake activities and ancillary testing; thus, this clinic work may not be accurately labeled as "outside" work time. Overall, work outside of work remains challenging to define for vendors, researchers, and clinicians alike. Another example of difficulties in accounting for provider time are demonstrated through time spent on inbox management. Inbox and messaging time may be underestimated by vendor-provided measures that strictly count time spent viewing or writing messages. In addition, using the inbox for specific tasks (eg, embedded order entry) may vary among different EHR systems. Resolving inbox messages often requires work in areas of the EHR other than the inbox, 7 and may even involve tasks that are not reflected in EHR log data at all (eg, phone calls to consulting providers or to patients). The proposed measure would include these actions, although best practices regarding how to define actions related to inbox messages are still evolving. 14, 15 This is critically important because inboxrelated activities pose a well-documented burden on physicians, 7 which will only increase with widespread adoption of telehealth and asynchronous patient engagement in response to the COVID-19 pandemic. [16] [17] [18] [19] Of note, vendors reported that measure data for reporting periods at the onset of the pandemic were skewed due to rapid fluctuations in the volume of both inbox messages and patient appointments. These examples illustrate that both "time on the clock" and clinical volume are important domains for measure standardization. Ongoing work is needed for standardizing EHR use measures. One strategy would be to evaluate aligning vendor-provided measures into the standardized proposed measures. This may depend on developing standardized terminology around EHR use and audit log elements, especially because vendors offer different levels of granularity. Furthermore, to calculate these measures, vendors often use other logs, which are more granular than the regulated audit logs needed for compliance with federal regulations. Developing standardized terminology for EHR logs is ongoing. In the near term, matched measures on the same vendor platform can enable crossinstitution research, 12,20 a valuable first step towards more generalizable evidence. Over the long term, standardization will not only require consensus on appropriate measure definitions, but also interorganizational collaboration and ongoing maintenance. Although prior studies have criticized vendors, 13 the initiatives of vendors to develop these measures and engage in conversations about standardization, even without top-down regulatory requirements, is encouraging. Our study had limitations. We did not examine measures for inpatient and other settings. We did not review all vendor-provided measures and focused on those pertaining to the previously proposed core measures, which have not been extensively validated. Next, we focused on large EHR vendors; future studies are needed to evaluate smaller EHR vendors. Finally, we focused on measure definitions but did not analyze various forms of data visualization on vendors' platforms. Developing best practices around data visualization would be valuable for future investigation. In summary, our findings help identify high-priority alignment of EHR vendor measures against a set of proposed standardized outpatient EHR use measures. We identify measures readily available from vendors that can be used in studies of EHR use and attendant burden. Understanding variations and working toward standardization will facilitate future work to compare studies across diverse organizations in the effort to measure and improve physician burnout. This study was supported in part by NIH grants T15LM011271 and T15LM012502. Each author made substantial contributions to the conception or design of the work; was involved in drafting the work or revising it critically for important intellectual content; gave final approval of the version to be published; and has agreed to be accountable for all aspects of the work. Supplementary material is available at Journal of the American Medical Informatics Association online Office of the National Coordinator for Health IT. Office-Based Physician Health IT Adoption and Use. Health IT Dashboard j Data Percentage of office-based physicians using any electronic health record (EHR)/electronic medical record (EMR) system and physicians that have a certified EHR/EMR system, by U.S. state. National Center for Health Statistics Electronic health records and burnout: time spent on the electronic health record after hours and message volume associated with exhaustion but not with cynicism among primary care clinicians The impact of electronic health record use on physician productivity Relationship between clerical burden and characteristics of the electronic environment with physician burnout and professional satisfaction Physician burnout: contributors, consequences and solutions Physicians' well-being linked to inbasket messages generated by algorithms in electronic health records Effects of health information technology on patient outcomes: a systematic review Using electronic health record audit logs to study clinical activity: a systematic review of aims, measures, and methods Variation in physicians' electronic health record documentation and potential patient harm from that variation Metrics for assessing physician activity using electronic health record log data Physician time spent using the electronic health record during outpatient encounters: a descriptive study Have you got the time? Challenges using vendor electronic health record metrics of provider efficiency Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations Use of electronic health record access and audit logs to identify physician actions following noninterruptive alert opening: descriptive study COVID-19 transforms health care through telemedicine: evidence from the field Virtually perfect? Telemedicine for Covid-19 Telehealth for global emergencies: implications for coronavirus disease 2019 (COVID-19) Telehealth transformation: COVID-19 and the rise of virtual care Multi-center analysis of electronic health record use among ophthalmologists The authors wish to thank Vaishali Patel, PhD, from the Office of the National Coordinator for Health Information Technology for providing data regarding vendors' market share, and Genna Cohen, PhD, and Jessica Boi from Mathematica for critical input and review of the project. The authors also wish to thank the EHR vendor representatives who assisted with providing information for this analysis and reviewed and approved the manuscript to verify accuracy of EHR use measure descriptions. We are also grateful to the National Research Network (NRN) Measures Workgroup for comments and insights throughout the study. SLB acknowledges grant funding from National Institutes of Health/National Library of Medicine (award number T15LM011271) and the Heed Ophthalmic Foundation. NCA acknowledges funding from the National Institutes of Health/National Library of Medicine (award number T15LM012502). NCA also holds stock in Cerner Corporation.