key: cord-1000869-ex7ebksd
authors: Damaske, Janet; Walsh, Wendy; McKay, Jennifer
title: CME in the Time of COVID-19: Educating Healthcare Professionals at the Point-of-care and Improving Performance Outcomes
date: 2020-10-16
journal: Journal of European CME
DOI: 10.1080/21614083.2020.1832798
sha: 3b70930a55d3508c256bb7c7caafcc3b5054661e
doc_id: 1000869
cord_uid: ex7ebksd

UpToDate® is a point-of-care clinical resource used by nearly 2 million clinicians worldwide. Users in the USA and other regions can obtain continuing education credits after researching clinical questions at the point-of-care. As part of the CME credit redemption process, participants provide feedback on the information researched during the UpToDate® learning activity. We examined the impact of UpToDate® searches on clinical decision-making related to COVID-19. Between January 1 and 31 August 2020, UpToDate® added more than 40 topic reviews on various aspects of COVID-19 diagnosis and management, and developed new methods for delivering COVID-19 content to our learners. During the observation period, participants accessed COVID-19-related topic reviews over 7.5 million times. Data collected from CME activity evaluations and user feedback suggested that the learning activity had a significant impact on clinical decision-making. Over 94% reported that they modified their management strategies as a result of using UpToDate®, and 97% reported that use of UpToDate® led to improvement in care. These findings support the benefit of self-directed, point-of-care learning activities on the clinical management of patients during a global pandemic.

When the World Health Organisation (WHO) declared COVID-19 a pandemic on 11 March 2020, clinicians could not rely on decades of medical research or their own clinical experience to treat the novel virus. Tasked with caring for infected patients and preventing viral spread, health-care workers were left with an urgent need for current, practical information. Continuing education providers saw this as a call to action to develop high-quality resources for clinicians fighting the disease.

As a provider of CME/CE/CPD, UpToDate® (a clinical resource produced by Wolters Kluwer) was first accredited by the USA' Accreditation Council for Continuing Medical Education (ACCME) in 2000. UpToDate® has maintained ACCME accreditation and gained recognition as a continuing education resource by regulatory bodies from regions around the world including Australia, Asia, Europe, and South America. UpToDate® is used by nearly 2 million clinicians worldwide.

UpToDate® offers internet point-of-care activities, wherein participants can search over 12,000 medical topics using clinical terms, phrases, or questions. The platform provides clinicians with immediate access to original, peer-reviewed content designed to support decision-making in clinical practice. This resource may be accessed at the point-of-care or used for indepth individualised study. Clinicians translate this knowledge into improved strategies for patient care while earning credit to fulfill local licencing and certification requirements.

Independent research underscores the validity and use of UpToDate® and demonstrates the system is working: health-care workers are getting the right answers quickly, and patients are receiving better care. Over 80 research studies suggest that use of UpToDate® results in improved hospital performance and patient care. Researchers at Harvard published a study in 2011 showing a correlation between UpToDate® use and improved lengths of stay, mortality rates, and quality of every condition on the Hospital Quality Alliance (HQA) metrics [1] . A 2012 study conducted at Singapore's National University Hospital showed that 37% of the time, UpToDate® use led to changes in investigation, diagnosis, or management [2] .

Together with external contributors and peer reviewers, the editorial staff at UpToDate® continuously reviews the literature to develop actionable, evidence-based content and disseminate information as efficiently as possible. In early 2020, when COVID-19 cases were rising and clinicians needed point-of-care access to current best practices, many learners turned to UpToDate®. During the week of March 15 alone, users searched UpToDate® for COVID-19 information 319,880 times, growing to 921,207 times by the end of March.

UpToDate's® unique editorial model provided a framework for an immediate and dynamic response to the COVID-19 crisis. A number of assessment measures, including content output, user input, usage reports, and search metrics, indicated UpToDate® made substantial positive impact on practice in the first eight months of its response to COVID-19.

A well-refined editorial workflow and a team of expert and clinically active contributors enabled the editorial staff to quickly release new UpToDate® content. COVID-19-related content development began in January 2020, when a contributor notified Infectious Disease editors at UpToDate® of a new coronavirus: "I just read this morning that we likely have a new coronavirus that we should keep an eye on. The Wuhan pneumonia outbreak has now several cases that have had a new coronavirus. Almost no other information". Shortly afterwards, the UpToDate® in-house editors added a section on "2019-nCoV" to the "Coronaviruses" topic, which rapidly expanded and became its own dedicated topic in February.

Since the first section was added, editors have performed daily scans of published information from WHO, Centers for Disease Control and Prevention (CDC), and other literature; managed discussions through content drafts with external authors and editors; and fielded questions from users that helped identify gaps. As of August 31, the editorial team has created and published 42 COVID-19 topics, updated the material 581 times, and added 47 COVID-19 graphics, spanning 20 of the 25 specialities covered in UpToDate®. External faculty have been highly engaged, reviewing and responding to changes within the same day, often within an hour or two of receipt.

When learners seek immediate answers to pressing questions, UpToDate's® design provides an easy to use, practice-based learning resource. Search data show that from January 1 through 31 August 2020, terms related to coronavirus were searched in UpToDate® more than 4.1 million times, underscoring the critical need for COVID-19 information. The following existing design features allow for quick and easy access to relevant content:

• Scrollable, clickable, and carefully curated outlines for individual topics appear on the search results screen and within each topic. • A "Summary and recommendations" section appears directly underneath the topic title on the search page and at the top of the topic outline. • Graded recommendations, based on the international GRADE approach, help the clinician determine the level of evidence behind the treatment recommendations. 1 • Within the topic text, references and other supporting evidence are included. References link to the abstracts in PubMed, allowing learners to access the evidence cited in topics. • "See links" within the topic and its outline enable learners to access other sections and navigate to related topics with one click. • UpToDate® staff continuously review and optimise search results for "time to answer" and accuracy.

The UpToDate® design includes a linked "feedback button" on all topics. The feedback feature empowers participants to identify content gaps and inform editorial decisions. When feedback is received, it is triaged by speciality editors, who either improve the search experience so that existing content can be found easier or work with external contributors to update topics and address previously missing information. Feedback from resource-limited participants within and outside the USA helps editors gauge how well they are meeting global health needs.

In 2009, UpToDate® partnered with Better Evidence, a program that has donated over 20,000 complimentary UpToDate® subscriptions in more than 150 low-and medium-income countries. This donations program serves doctors and patients in resource-limited countries, with the added benefit that feedback from a larger user base helps expand the faculty's understanding of clinical needs in diverse settings. This global approach became even more significant as COVID-19 spread from country to country, affecting communities at different times and to different degrees. As an example, one clinician in India credited UpToDate® with providing crucial information in preparing the workforce to face the virus. A health-care worker in Nepal wrote, "Now the whole world is suffering from COVID-19 pandemic and we in Nepal are also not spared too. Since UpToDate is providing regular updates and it has helped us in getting knowledge, resources and providing continuous updates to government. In today's world I can't imagine my life without UpToDate in managing patients".

Direct communications through the feedback button and the comments received from recipients of UpToDate® via the donations program underscore the importance of developing educational content that is useful for clinicians practising internationally. The Better Evidence program strongly contributes to improvements in the standard of care for patients across the globe.

In response to the pandemic, UpToDate® developed additional instructional design features that enable participants to easily navigate to COVID-19-related content.

Drawing on prior experience with alerting readers to critical content during natural disasters and disease outbreaks, a box was added below the main UpToDate® search field to highlight COVID-19 content. Links were grouped so that participants could go directly to clinical topics, society guidelines, patient education, questions and answers, and UpToDate® Pathways.

Building on the existing feedback feature, participants' questions specific to COVID-19 were collected and triaged daily by a dedicated team of editors. The most common were organised into "questions and answers" topics, which include links to more in-depth information. For instance, in "Coronavirus disease 2019 (COVID-19): Questions and answers", users could click on the following question from the outline: "What are the major coagulation abnormalities in patients with COVID-19?" They would be brought to a brief explanation, as well as a "see link" to the relevant section of the core topic "Coronavirus disease 2019 (COVID-19): Hypercoagulability".

Search analytics strongly suggested that UpToDate's instructional design was working. Between January 1 and 31 August 06, 15,200 unique users in 193 countries accessed topics related to COVID-19 over 7.5 million times. 2 The current most-viewed topic in the program, "Coronavirus disease 2019 (COVID-19): Epidemiology, virology, and prevention", has been viewed over 3.3 million times since it was published on February 6.

Users consistently find answers in less than 1.5 minutes (median time, 81 seconds on web browser and 69 seconds on mobile app). Successful COVID-19 searches in February averaged 91.4% but dropped to 87.8% by May. During this time, the medical community's understanding of COVID-19 was evolving rapidly. UpToDate® contributors continued to focus on updating topics with new evidence for diagnosis and treatment options. Successful searches started trending upward again after May; by the end of August, approximately 88.9% of learners found an answer, indicating participants were successfully locating answers and applying information at the point-ofcare.

As a point-of-care learning activity, UpToDate® participants are awarded credit for each completed learning cycle, which includes the following steps: documenting the clinical question, identifying the topics consulted in UpToDate®, and reflecting on the application of findings to practice. Data collected from the reflection demonstrate that use of UpToDate® reinforced learners' clinical management strategies (64.1%) as well as modified their strategies (26.8%) in many situations. Based on the number of COVID-19-related searches completed in the first eight months of 2020, we estimate that 1.1 million decisions regarding approach to patient care were revised as a result of participation in UpToDate®. These data suggest that learners were successfully translating the UpToDate® content into practice and making improvements to their performance and patient management strategies.

Participants also answered questions about UpToDate's impact on performance and patient care as part of the post-activity CME evaluation. From January 1 through August 31, over 98% of participants reported UpToDate® reinforced their clinical management strategies. Ninety-four per cent of participants found that patient management strategies were modified as a result of using UpToDate®, and 97% responded that use of UpToDate® led to improvements in care. Ninety-seven per cent of participants found use of UpToDate® was an effective way to find information at the point-of-care.

In addition to requesting feedback using the CME evaluation, we measure effectiveness over time by following up with participants via email. Each month, a select number of learners are sent a four-question survey within a few days of redeeming CME credits. The survey asks the participant to provide an example of a question they researched in UpToDate® and to describe whether they made any practice changes as a result, whether any barriers prevented them from making those changes, and whether UpToDate® had a measurable impact on patient outcomes or provider performance.

This routine survey was adapted to determine how participants have been using UpToDate® to research COVID-19. Learners who indicated on their CME activity evaluation that they had performed a COVID-19-related search between January 1 and 30 June 2020 were sent an email asking them to consider the survey questions in the context of that search. The survey was also sent to participants who had provided feedback regarding COVID-19 through our feedback system. Of the participants who answered the survey, 82% provided overall positive feedback, indicating that UpToDate's content regarding COVID-19 helped them care for patients, and more than 50% reported that their use of UpToDate® had a measurable impact on patient outcomes or provider performance.

Participants provided the following examples of practice changes that they made as a result of their research in UpToDate®:

• … from not giving even nebs to patients with secondary bronchospasm to understanding how to approach an elevated D-dimer, everything in between was evolving at the speed of light. It was very difficult to feel that we were doing the best possible for the sickest patients, but UpToDate was always available with the latest opinion from the best practices and with the perspective from studies all around the world. This last issue is very important because many countries had had experience with different diagnostic and therapeutic approaches already. • … advances in understanding the epidemiology and clinical progression of COVID-19 was useful. For example, I began supplying patients with COVID-19 who stayed home during their illnesses with inexpensive pulse oximeters. Some COVID patients will begin to desaturate the blood oxygen levels before they become clinically distressed. This allowed me to give them better advice on when to go to the hospital. It also allowed me to reassure them, when this was required.

The participants' responses, which were reviewed by appropriate in-house editors, emphasise that UpToDate's® method of continuous publishing is ideal for sharing reliable information during a fastmoving health crisis.

Several mechanisms are used to evaluate the impact of UpToDate® on performance and patient care. UpToDate® uses a review system that collects information regarding changes in learners' performance from participant data and research studies. User data show that our content answers specific clinical questions at the point-of-care and improves patient management. The COVID-19 pandemic required us to create content quickly using our established process and interface, and also to adapt and build on our model. We piloted new ideas for interface design, presenting a growing collection of topics in an easily managed search box. We also developed a new process to quickly triage a deluge of questions from readers, presenting the answers in new and continually updated questions and answers topics.

UpToDate® continues its efforts to expand its user base so that more people have access to our COVID-19 information. Since mid-March, UpToDate® has granted free access to anyone seeking COVID-19 information. In a time when clinicians are stretched thin, some even practising in complete isolation, UpToDate® provides practice-changing information, while also allowing clinicians to earn CME credit, save time, and improve performance in their essential, lifesaving work.

Use of UpToDate and outcomes in US hospitals

Utility of the electronic information resource UpToDate for clinical decision-making at bedside rounds

We wish to thank our Wolters Kluwer colleagues Hongli Li, Product Software Engineering Manager; Denise Gilpin, Vice President of Content Editing; Brogan Calkins, Senior Content Editing Associate; Peter Bonis, Chief Medical Officer; and Ted Post, UpToDate® Editor in Chief for their contributions and support of our work.

Ms. Damaske has nothing to disclose. Ms. Walsh has nothing to disclose. Ms. McKay has nothing to disclose.