key: cord-0850756-rbw4n62c
authors: Kawczak, Steven; Fernandez, Anthony; Frampton, Bethany; Mooney, Molly; Nowacki, Amy; Yako, Matthew; Stoller, James K.
title: Observations from Transforming a Continuing Education programme in the COVID-19 Era and Preparing for the Future
date: 2021-08-17
journal: Journal of European CME
DOI: 10.1080/21614083.2021.1964315
sha: 264e1b861c424dcc2880021781c790c879acc1f5
doc_id: 850756
cord_uid: rbw4n62c

The COVID-19 pandemic has significantly disrupted and transformed continuing education in the health professions to be reliant on digital learning modalities. This retrospective observational study of a large, international health system’s continuing education programme compares educational activities offered, participation, and learning outcomes pre- and intra-pandemic to assess the impact of digitisation advanced because of the pandemic. There was a significant increase in internet-based activities that filled the gap of cancelled or postponed live, in-person activities to keep healthcare professionals up to date in their specialities and prepared to handle the clinical and hospital demands of the pandemic. Compared to live, in-person education, virtual activities were offered in shorter increments, reached a much larger amount of participants, and were equally effective in achieving learning outcomes. Questions remain regarding business model implications to generate adequate revenues to cover costs of virtual education. Additionally, there is a general inadequacy of digital learning environments to coalesce groups and meet social needs. Regardless, the efficiencies and effectiveness of digital modalities will be a primary method of teaching healthcare professionals going forward.

The COVID-19 pandemic has significantly disrupted all aspects of life globally. One disruption within healthcare relates to Continuing Education (CE) activities, which were transformed in both educational focus (pandemic preparedness) and delivery (digital modalities). Many CE activities, as with a majority of other educational efforts, have traditionally been designed with formats optimised for live, in-person settings. These include long-established activities like grand rounds and regional, national, and international conferences, in addition to more recently instituted activities like hospital and team improvement activities. However, with the pandemic suddenly restricting group gatherings to limit viral spread, these in-person educational activities were cancelled, and replaced by alternative, synchronous and asynchronous virtual format strategies to ensure that critical educational activities continued.

The educational focus also quickly changed as healthcare organisations and caregivers were confronted with a wide-range of needs in order to adequately, safely, and effectively respond to this pandemic. Thus, prompt and regular CE became paramount for success, and an abundance of CE initiatives were rapidly designed as knowledge emerged about COVID-19 with the related needs of organisations and caregivers in mind [1, 2] . At the same time, smalland large-scale societies felt a need to continue educating their members to stay up-to-date with advances in their respective diseases and specialities.

As a result, the COVID-19 era has transformed CE. Virtual education has become widespread, and we have been forced to learn how to more effectively design, reach, and teach learners with digital tools. The necessary CE modifications during this time have no doubt resulted in long-lasting innovations. However, this era has also shown that virtual activities are not entirely ideal, and that a measured return to in-person settings is desired in order to meet certain learning and social goals.

Our organisation's CE programme endured the full impact of the COVID-19 pandemic. Since March 2020, essentially all live internal and external CE converted to virtual formats, with the small exception of internal team-based skills training critical for COVID-19related hospital care, thus forcing our CE team to innovate quickly. The shift to virtual formats in our programme spotlighted issues surrounding technical competencies, unproven business models for virtual education, and ensuring that an effective learning environment was implemented. To help activity directors and planners prepare for successful transition, instruction and guidance concerning how to design and implement live, synchronous virtual education had to be offered. Shifts in funding, marketing, communication, educational format, delivery format, and audience engagement all had to be considered to achieve success.

After more than a year of implementing digital education activities and learning to improve them during this time, we currently assess the changes that COVID-19 forced upon our CE programme, activities, and learners. At the forefront of this analysis is the value of digital education and whether or not online environments can achieve educational results comparable to live, in-person settings. This before-after study presents an analysis of data collected pre-and during the pandemic in order to identify both successes and opportunities for improvement. Finally, we discuss how our CE programme's experiences during COVID-19 are likely to affect the overall design of CE in the future, post-COVID-19 era.

We performed a retrospective before-after observational study comparing our institution's CE programme over a two-year period (January to December 2019 pre-pandemic; January to December 2020 intra-pandemic) to identify the impact of the COVID-19 pandemic on the types of educational activities offered, participation, and self-reported learning outcomes, as well as the business models and implementation processes for digital activities. Institutional IRB approval for this study was obtained (IRB # 21-512).

Annually, data are compiled across our institution's CE programme based on activity type, number of learners reached, and credits offered. Activity types are established by the Accreditation Council for Continuing Medical Education (ACCME) and American Medical Association (AMA), and the types offered in our programme include: Course, Regularly Scheduled Series, Internet Live Course, Performance Improvement, Internet Searching and Learning, Internet Activity Enduring Material, Enduring Material, Journal-Based CME, and Manuscript Review [3, 4] .

Evaluation surveys are used to assess outcomes for each educational activity. Evaluation questions are designed based on activity goals, and our programme offers optional templated questions. These templated, self-reported learner evaluation questions are based on Moore's framework on achieving learning objectives and performance changes [5] . We compared results of outcome questions for a subset of our institutional CE programme's live courses that converted to an internet live format in 2020 due to the COVID-19 pandemic to observe how the modality of instruction change may have affected learning outcomes. A revised evaluation template was implemented in early 2020, prior to knowledge of the impending pandemic, which resulted in a different question asked in 2020 about achieving each learning objective compared to the prior iteration of the course.

• Original: Were objectives (each objective is listed) met? Response options: "met" or "not met" • Revised: Please rate the degree to which the activity has impacted your ability to (objective listed): Response options: "high impact," "moderate impact," "no impact"

To allow comparison across modalities, we bundled the responses of "high impact" and "moderate impact" as equivalent to "met" for the achievement of learning objective outcomes.

The question pertaining to performance change was "Based on the information presented in this activity, will you change your clinical practice behaviours? Response options: very likely, likely, somewhat likely, not at all, not applicable

Associations among modality of instruction and learning outcomes (achievement of learning objectives and performance change) were assessed using chisquare tests. Values of p < 0.05 were considered statistically significant. All analyses were conducted with JMP Pro 15 statistical software (SAS Institute, Cary, NC).

Comparison of CE programme activities in 2019 and 2020 showed a number of striking differences ( Table  1) . As expected, based on changes implemented during COVID-19, there was a significant increase in the number of internet enduring activities (2.2-fold increase), regularly scheduled series (1.4-fold), and internet live courses (192.5-fold), and a concomitant significant decrease in live in-person courses (3.1-fold decrease) and performance improvement activities (3.4-fold decrease) in 2020 compared to 2019. These findings indicate that activity directors did not abandon educational initiatives during the pandemic, but rather mainly transitioned them to live virtual formats. Still, overall there was a 1.5-fold decrease in live (live in-person + internet live) activities in 2020 compared to 2019. On the other hand, no significant change was seen in the frequency of journal or manuscript reviews.

In terms of learner engagement, there was a significant increase in participation, credits offered, and credits claimed for regularly scheduled series (RSSs), but decreased participation in enduring activities, likely reflecting a focus of new 2020 RSSs on COVID-19 (e.g. developments in disease management, caregiver safety, and updates on organisational policies regarding COVID-19 care). Thus, the pandemic prompted critical and urgent education, unlike antecedent enduring materials. Another contributing factor was likely the relaxation of continuing education requirements for licensure and maintaining board certification requirements for physicians due to COVID-19's disruption of education plans [6, 7] .

Our CE programme offers many conferences that typically recur on a regular basis and target learners outside of our organisation at regional, national, and international levels. Sixteen such activities that were scheduled in 2020 elected to postpone until after the pandemic abated and did not take place. Directors of 32 of these activities that had previously occurred inperson made the decision to convert to a live virtual format in 2020. These were selected to investigate because they focus on audiences external to the organisation, depend on outside revenue sources (registration fees, educational grants, exhibits), and are recurring in nature (e.g. annual). Table 2 summarises the comparison of these previously in-person with currently live virtual courses. The analysis shows that activity planners modified their activity agendas to be briefer for live virtual formats compared to live and inperson formats, thus resulting in fewer credit hours in most cases (overall 57.2% decrease). In many such cases, awareness that attendees from all over the world may participate contributed to a briefer programme in order to accommodate a wide range of time zones. Courses that offered similar credits in both formats were mainly board review activities, which needed to provide comprehensive content to satisfy course objectives. Nine courses provided increased content and credits in the live virtual format compared to the prior year's live, in-person format. In these cases, planners took advantage of less "networking" or "break" time built into their live, in-person agendas and added additional content in their 2020 virtual activity agendas to avoid unusually long breaks. Next, learning outcomes were compared between live, in-person and virtual modalities (Table 3 ). There were a total of 23 courses held virtually in 2020 that pivoted from prior year in-person activities and that used comparable evaluation questions on achieving learning objectives. The results show a 0.6% decrement with virtual courses for these paired activities. When considering all programmes offered live and/or inperson, even if they did not repeat and pair to another course, the results show similar achievement levels (97.5% met learning objectives in live vs. 97.9% virtual). While both of these comparisons achieved statistical significance, significance is being driven by the very large sample size for virtual activities and the observed small absolute differences are unlikely to carry practical importance. Both comparisons show that learning objectives are largely met regardless of modality.

There were a total of 10 courses held virtually in 2020 that repeated from a prior year in-person format and which used comparable evaluation questions on performance change ( Table 3 ). The results of participant-reported performance change shows a higher proportion of "Very Likely" ratings for live programmes (6% higher) than virtual and a higher proportion of "Likely" ratings for virtual programmes (11% higher) than live. In aggregating "Very Likely" with "Likely", the proportion reporting performance change was 80.8% live vs. 85.6% virtual. When considering all programmes offered live and/or in-person, even if they did not repeat and pair to another course, the proportion reporting performance change "Very Likely/Likely" was 78.6% live vs. 83.5% virtual. Again, both results are statistically significant, but as with learning objectives, the observed effect is unlikely to carry practical importance.

Given the challenges in attaining informative statistical analyses, we analysed open-ended feedback regarding the virtual format of courses from comments obtained from these questions: "Please share any other comments or recommendations" and "Other comments on this educational activity". There were 769 verbatim comments from these courses. Most feedback (n = 508) pertained to the educational content and effectiveness of the speakers, although there were some comments about the virtual format (n = 261).

Most feedback pertaining to a virtual format was positive (71%). There were a variety of reasons that participants indicated satisfaction, with some of the more prominent themes being: (1) Convenience-normally could not participate, (2) Efficient/minimal work disruption, (3) Less expensive registration or overall cost savings, (4) No travel required, and (5) More opportunities for diverse speakers. Also, some feedback regarding the virtual format was negative (29%), including: (1) Technical difficulties, (2), Did not meet needs for interaction, (3) Education design not as effective virtually, and (4) Activity not held at a convenient time (e.g. held during workday or hours not optimal for participant's particular time zone) ( Table 4 ).

An obvious and important aspect of implementing educational activities is obtaining sufficient financial support to meet budgetary needs. Therefore, we examined and compared revenue sources for CE activities based on their modality. Revenue sources for these courses typically include educational grants from commercial supporters, exhibit fees from various organisations, and registration fees from participants, all of which are used to offset the costs of production. Table 5 shows variations in revenue sources for inperson and virtual format courses. Total revenue decreased by 43% across our entire programme, but there was a partially offsetting cost savings of approximately 30% in expense reductions on average for live events.

Commercial support remained strong in 2020 during the pandemic, even slightly increasing (by 6%) for the virtual events (Table 5 ). This suggests an overall appetite for grantors to support education regardless of the format. Although not obvious from our data, our experience was that grantors did not initially extend support early during COVID-19 for programmes wishing to transition to virtual formats, and delays were incurred in obtaining approvals. However, this improved by the Summer and Fall of 2020, indicating that grantors eventually had increased confidence in supporting virtual education

As with nearly all academic centres, the COVID-19 pandemic caused prompt and profound changes in many aspects of routine healthcare practices, including CE. In a hospital environment, the production and use of CE changed to new education interventions for improving knowledge, competency, and behaviour related to diagnosing and treating patients with COVID-19 based on rapidly evolving evidence. Disseminating COVID-19-education was critical to both controlling community spread, improving outcomes of affected patients, and keeping providers safe [1, 2] . Importantly, our results demonstrate that learning outcomes were comparable among the live and virtual modalities and that overall, both formats were successful, without obvious setbacks related to the pivot to virtual offerings. On this basis, it suggests that no harm was done and we conclude that a virtual setting is just as effective as live modalities, if not more so. As the tremendous difference in the number of learners precluded informative statistical analyses, there remains a need for more granular investigation of paired educational content to further test for differences.

For the broader healthcare community, CE remained essential to continue learning within disciplines in order to ensure optimal outcomes, as well as to satisfy licensure and continued certification expectations for some. This propelled a global shift of nearly all live conferences, courses, and annual society/association meetings to virtual formats to ensure that they continued to reach learners. Many education providers have reported successful transitions to offering CE in virtual formats, and have been exploring effective models of virtual engagement scaled to the scope of their audience and learning objectives. The result has been the development of a wide range of digital learning activities, from small group case conferences to international level congresses, leveraging technologyenhanced learning, including blended strategies, and increased use of tools such as social media and podcasting to engage learners [8] [9] [10] [11] [12] [13] [14] . These innovative solutions have been largely developed due to the need for change and finding ways to reach learners and maintain community. These trends invite the obvious question as to what changes will persist as the world normalises post-pandemic.

As shown by the comparison, virtual solutions have been well-received and have demonstrated efficiencies for the learner and provider; at the same time, the transition to virtual learning has left some longing for a return to in-person gatherings for social reasons, networking, and to restore informal information exchanges [15] [16] [17] [18] [19] . All these factors will need to be weighed for future design in the post-COVID era. Intra-hospital education, such as regularly scheduled series like morbidity and mortality activities, case conferences, grand rounds, and other team-based education forums, such as communications activities, are poised to benefit long term from the efficiencies of virtual settings, where education objectives and an effective environment can be continued in a closed, psychologically safe setting in which learners know one another [20] . This CE programme analysis underscores several noteworthy themes. First, the demand for CE remained strong during the pandemic, and even increased on a programme-wide scale. Not surprisingly, the most significant growth was related to COVID-19 education offered in live virtual, ondemand online and in RSS activities. These programmes were offered through partnership with clinical areas, organisation leadership and technology experts to create time-sensitive and critical content on COVID-19 preparedness and safety. Also, training on a wide range of topics to refresh dormant clinical skills and to prepare clinicians for possible redeployment to areas outside their usual scope of practice was also necessary to care for hospitalised patients. Also, education was needed to teach caregivers how to offer virtual patient encounters, especially regarding technology and communication competencies and to provide routine virtual care for all patients, including those unaffected by COVID-19. The ACCME's streamlined requirements for COVID-19 education significantly enhanced our ability to offer credit for these activities. The substantial number of participants and credits related to these activities, coupled with an overall sense of pride and satisfaction among our caregivers concerning our institutional response to the pandemic alone supports the value of virtuallyoriented education.

The decline in performance improvement activities is concerning, but perhaps not unexpected given the momentous changes in routine healthcare endeavours and ability to focus on team-based gaps in care performance brought on by COVID-19 disruptions. Additionally, a switch to virtual longitudinal engagement for teams and coaching required lengthier planning. We expect and look forward to seeing these restart in virtual and/or in-person formats.

Our data indicate generally increased participation rates in virtual vs. prior live CE activities. We believe that the three main drivers of this trend were that: (1) Most courses charged lower registration fees or were offered free, (2) The need to travel to a location for attendance and associated expenses was eliminated, and (3) At least early in the pandemic, patient care volumes decreased in many areas (e.g. related to early cancellation of elective care) during the COVID-19 pandemic, allowing healthcare professionals unexpected time to focus on education.

Similarly, to explain the observed decreased revenue associated with 2020 courses, likely factors include: (1) Commercial entities' decreased interest in virtual exhibitions due to lack of perceived value and their internal regulatory guidance not being designed around virtual interactions, (2) Reduction in the number of exhibit opportunities that were offered to companies, (3) Reduction in fees charged for virtual exhibit opportunities compared to in-person events, and (4) Hesitation by commercial entities to exhibit virtually due to complicated approval processes and risk of unknowingly breaking regulatory guidelines in relation to virtual formats.

Decreases in registration fees were likely related to several factors. First, there was an intentional effort to minimise or eliminate fees due to the disruption of learners' and learners' institutions' budgets by the pandemic. Also, many programmes decreased in length and had correspondingly lower registration fees. Lastly, the costs of producing virtual meetings are much lower with no expenses due food and beverage, venue, travel and signage as are typically needed for an in-person event.

On the other hand, some courses kept registration fees at similar levels to prior years, such as for board reviews, which retained a similar volume of content and experienced increases in attendance due to the ease of access. Importantly, the success with reaching participants suggests that learners are willing to pay for virtual format content, presumably because they perceive the value will be similar to an in-person format. Although this goes against the trend for the majority of courses, it may have important implications for future virtual activities that need to consider registration fees to meet budgetary demands.

In terms of directly comparing live, in-person courses that converted to virtual live formats, our major findings included a shortening of curricula for virtual formats, a significant increase in participation in virtual format courses, and high satisfaction with the overall ability of virtual courses to meet objective goals and to elicit behaviour-changing practice changes. Of particular note are comments surrounding the ease of participating as a learner in a virtual meeting and the elimination of travel, as well as access to international experts who may otherwise not have been able to present (i.e. international thought leaders). Again, these findings overwhelmingly support the conclusion that a virtual format can be successful and underscores the likelihood of their persistence going forward.

Adding to the surveyed responses, our experience underscores several other observations about the pivot to virtual CE. First, participants and faculty were very flexible and grateful to have some sort of "normalcy" when virtual CE education was offered during the pandemic. Second, we have observed that the frequency and impact of technical glitches when presenting or attending virtual sessions have diminished over time, as learners and faculty became more facile with the various technical platforms (e.g. Microsoft Teams, Zoom, etc.). Over time, we modified how education was produced to ensure easy access for both participants and faculty. There has been a trend to utilise pre-recorded lectures and having live panels or workshops to reduce amount of possible technical errors by the faculty while still maintaining audience interaction in a live virtual setting.

Third, we perceived a shortening of registration lead time for virtual events than for in-person events. We ascribe this shorter lead time to participants' having an easier time in their schedules to commit to participating, paired with a lack of need to travel and taking time away from the office [21] . Also, for courses that lacked registration fees, we observed a high frequency of "no shows". We speculate that learners had less "skin in the game" for free activities, prompting lower commitment, which was also evident for free events prior to the pandemic.

Fourth, from an accreditation point of view, the shift to virtual education was based on shorter, compressed timelines, which caused our CE team to change processes to accommodate implementation needs. The compression of traditional timelines and processes likely reflected two main factors: (1) the ACCME's streamlined requirements regarding COVID-19 education, and (2) the enhanced ease of planning virtual courses, e.g. due to relief from not needing to plan for a venue, food and beverage, and travel. Before the pandemic, the traditional planning timeline started 10-14 months prior to a course date for in-person courses. Planning a live virtual meeting during (and after) the pandemic now takes our CE team half the amount of time for activities for a public audience. The required time is even less for activities within our hospital setting.

Finally, a consequence of the pandemic was improved methods for obtaining feedback and credit claiming. In the live virtual setting, we now utilise QR codes that link the learner to the evaluation and a credit claiming process directly from the screen. Specifically, a QR code is shown at the end of every CE session, allowing the learner to scan the code which takes them to a site which allocates credit for participating in the activity. This process was developed for virtual RSS sessions, and was immediately embraced by busy clinical colleagues. Also, the evaluation approach for RSS was changed to be intermittent instead of for every session. This decrease of frequency improved efficiency for learners and has not diminished our ability to obtain feedback. These changes were innovations for administering credits and evaluation hastened by the needs of the pandemic, which we expect will endure beyond the pandemic.

While this analysis examines the broad impact of the pivot to virtual CE following the pandemic, several limitations of the analysis warrant comment. First, the analysis is based on a before-after comparison, which introduces potential bias related to natural temporal changes and interim innovation. While the acuity of change suggests that the changes were attributed to the pandemic, only a concurrent, randomised head-to-head comparison of live vs. virtual CE can establish causality. Second, the analysis is limited to the first 12 months of the pandemic. It remains unclear whether and how extensively virtual CE activities will revert to prior pre-pandemic practices once the pandemic abates or if hybrid models are designed. Similarly, it remains unclear whether rates of satisfaction and participation in CE will continue as the pandemic continues and whether they will persist in a post-pandemic world. Third, the learner outcomes data in this analysis are based on self-reported achievement of meeting learning objectives and stated (rather than observed) commitments to change practice behaviour. Documenting objective changes in actual practice behaviours is needed to establish that virtual CE is as impactful as live CE, as has been demonstrated in prior studies [22] . Finally, the analysis is based on experience at a single institution, so that conclusions about the generalisability of these findings will require validation in future, multicenter studies.

While the durability of the observed changes is uncertain currently, valuable lessons have been learned in the transformation of our CE programme to digital formats, with overall themes of effectiveness in achieving learning outcomes, participation increases, and efficiencies for learners and educators. These lessons will surely inform future decisionmaking and we should remain confident in utilising virtual modalities. Notwithstanding the demonstrated benefits of virtual CE, the absence of social networking that accompanies virtual participation and the loss of "human collisions" that can spawn innovation and coalesce groups are losses that will require further study, but are real factors justifying revival of in-person education, at least in part [19] . We expect to now think of digital options first, and then weigh learner interactivity and social needs in deciding when to return to in-person settings. How CE modulates in a post-pandemic world to preserve the benefits of virtual education while restoring some live, face-to-face meetings surely invites future analysis.

Rapid retooling, acquiring new skills, and competencies in the pandemic era: implications and expectations for physician continuing professional development

Rapid continuing professional development interventions at a large tertiary care center in response to the COVID-19 pandemic

Accreditation Council for Continuing Medical Education. What kind of activity types can be reported in PARS? Accreditation Council for Continuing Medical Education Web site

American Medical Association AMA PRA credit system frequently asked questions activity types

Achieving desired results and improved outcomes: integrating planning and assessment throughout learning activities

Federation of State Medical Boards. States modifying continuing medical education requirements in response to COVID-19. Federation of State Medical Boards Web site

American Board of Medical Specialties. Statement on Credentialing During COVID-19. American Board of Medical Specialties Web site

Continuing medical education in the time of social distancing: the case for expanding podcast usage for continuing education

Harnessing the power of social media to support a professional learning network during the COVID-19 pandemic

Medical conferences in the post-COVID world: a challenge, and an opportunity

Virtual morning report during COVID-19: a novel model for case-based teaching conferences

Impact of COVID-19 on medical education: introducing homo digitalis

Continuing medical and student education in dermatology during the coronavirus pandemic -a major challenge

Virtual education in neurosurgery during the COVID-19 pandemic

Physically distant, educationally connected: interactive conferencing in the era of COVID-19

Implementation and impact evaluation of a virtual orthopaedic continuing medical education conference in a low-resource country

Epub ahead of print

Ophthalmology conferences in the coronavirus disease 2019 era

Traditional and virtual congress meetings during the COVID-19 pandemic and the post-COVID-19 era: is it time to change the paradigm?

On the power of spontaneous collisions: preserving culture and effectiveness in the pandemic

Building your "educational peloton:" cycling together for success during uncertain times

Lessons from 9 months of virtual learning -what it means for planning hybrid continuing education. Alliance for Continuing Education in the Health Professions Web site

The impact of a quality improvement continuing medical education intervention on physicians' vaccination practice: a controlled study

The authors declare no relevant financial or non-financial competing interests.

The authors confirm that the data supporting the findings of this study are available within the article and its tables.

No grants or funding were obtained from any organization for this study.

http://orcid.org/0000-0002-3487-2291