key: cord-0879977-5itujanl authors: Martin, Alister; Raber, Joshua P.; Shayer, Desiree; Lai, Deborah; Goodcoff, Adam; Kannikal, Jasmine; Raja, Ali S.; He, Shuhan title: Get waivered remote: Nationwide, remote DEA-x waiver course in response to COVID-19 date: 2021-10-19 journal: Digit Health DOI: 10.1177/20552076211048985 sha: 24f9857a479dbe286ccc4f7480cecf3a5dde3432 doc_id: 879977 cord_uid: 5itujanl PROBLEM: As of 2020, less than 5% of physicians in the United States have a drug enforcement administration-X waiver to prescribe buprenorphine. The coronavirus-2019 pandemic restricted in-person gatherings, including traditional drug enforcement administration-X waiver courses. As a result, in-person conferences have needed to adopt remote formats. Many programs identified a gap between educational delivery and the faculty skills required to deliver content remotely. APPROACH: To address the need for high-quality remote learning, Get Waivered designed and implemented a novel experience for clinicians, called Get Waivered Remote. An educational session was live-streamed via Zoom™. To foster interactivity, like in-person didactic conferences, participants were polled to facilitate discussion among presenters, learners, and facilitators during the broadcast OUTCOMES: The RE-AIM framework was used for evaluation. Our program had a Reach encompassing 814 users that participated during the live-streamed event; Effectiveness with 73.79% reporting being somewhat familiar or very familiar with the practice of opioid dependency treatment with approved buprenorphine medications; Adoption with 95.15% reporting a favorable experience and 92.23% reporting it was similar or more enjoyable than their usual teaching; Implementation with 450 messages sent by 281 users to engage with presenters and other learners via Zoom chat in real time. NEXT STEPS: Get Waivered Remote provides a proof-of-concept that a broadcast with a concurrent, interactive remote learning platform is feasible, low cost, and simple to execute. Further study is required to assess the ability of our group to maintain this innovation and also to measure its impact on the treatment of opioid use disorder. Buprenorphine was approved by the food and drug administration for the treatment of opioid dependance on October 8, 2002 , but as recently as 2016 only 10% of patients who needed this treatment had access to medication. One of the challenges patients face in gaining access is the limited number of providers who are able to prescribe buprenorphine. The Drug Addiction Treatment Act of 2000 (DATA 2000) requires physicians to complete an 8 h training course and receive a waiver, often known as a Drug Enforcement Administration (DEA) X waiver. The Comprehensive Addiction and Recovery Act of 2016 expanded the waiver process to nurse practitioners and physician assistants after 24 h of training. However, recent estimates indicate that less than 5% of physicians in the United States had obtained an X waiver to prescribe buprenorphine in early 2020, creating a significant barrier to care for patients with Opioid Use Disorder (OUD). On January 14th, 2021, in an effort to ease federal requirements to grant more physicians the authority to prescribe buprenorphine, the U.S. Department of Health & Human Services expanded access to medication-assisted treatment (MAT) by exempting physicians that possess DEA registrations from the 8 h training requirement needed to obtain an X waiver. However, deemed published prematurely, these notable guidelines were nullified by the newly inaugurated Biden administration several days later, essentially reverting to the original DATA 2000 requirements. Furthermore, these unsuccessful legislative changes, compounded by a coronavirus (COVID)-2019, have exacerbated the barriers to care faced by patients with OUD, introducing new challenges to providing patient care, clinician training, and the accessibility and delivery of evidence-based care. Importantly, measures to suppress the spread of the virus, through social distancing, broad limits on in-person gatherings, and self-isolation, 1 create distinct barriers to traditional forms of medical education, including X waiver training courses. As a result, COVID-19 has required, and produced, dramatic innovations in online educational resources. 2 COVID-19 and the opioid epidemic COVID-19 has had a disproportionate impact on the already vulnerable patients who struggle with opioid addiction. Patients with OUD are at a higher risk of the negative sequelae associated with COVID-19. 3, 4 Protective measures, like social distancing and self-isolation, both exacerbate the drivers of addiction, namely anxiety and isolation, increasing the risk of relapse, while simultaneously decreasing access to treatment. Outpatient clinics and other treatment programs have struggled to provide care in accordance with social distancing guidelines. Ensuring uninterrupted access to evidence-based MAT, with agents such as methadone and buprenorphine, is integral to counteract these additional challenges. 5 Training related to both the prevention and treatment of opioid overdose with the use of evidence-based prescribing practices leads to improvements in clinician preparedness to address OUD. 6, 7 To treat OUD with buprenorphine, clinicians must undergo additional training to receive an "X" waiver from the DEA, 5 yet despite the documented efficacy of MAT, less than five percent of physicians in the United States had obtained an "X" waiver certifying them to prescribe buprenorphine prior to the emergence of COVID-19. 8 Physicians obtain this waiver by completing an 8-h course while nurse practitioners and physician assistants are required to complete a 24-h course. 9 These sessions are generally presented to live clinician audiences in group settings of ten to twenty providers. However, the recently implemented social distancing measures prompted the cessation of in-person training sessions, 2 leading to fewer opportunities to obtain the DEA-X certification. While successful virtual conferences have been previously described as an effective solution for other educational venues, there has been no large-scale approach to replicate the in-person "X" waiver course using virtual platforms. This paper describes the pilot of Get Waivered (GW) Remote, an interactive educational conference designed and implemented by GW. We aim to describe the implementation of the GW Remote Course and use the reach effectiveness adaptation implementation maintenance (RE-AIM) framework to evaluate our program outcomes to suggest potential opportunities for future improvement. The GW team delivered and hosted a nationwide, remote DEA-X waiver training course that would serve to get clinicians their DEA-X waiver using a live, synchronized, and interactive digital platform. We aimed to mitigate the challenges created during the COVID-19 pandemic which hindered clinician education including social distancing requirements that limited in-person gatherings. Our previous work focused on the three key behavioral patterns that were found to limit the number of clinicians who completed the "X" waiver process. 10 These barriers were identified as: (1) absence of a social norm; (2) hassle bias in obtaining the waiver; and (3) a lack of salience in treating OUD. We tailored the course around these three elements to maximize the user experience. First, our course sought to bolster social norms around treating OUD using evidence-based practices. On our website, we displayed the session's enrollment numbers and provided real time enrollment data to individuals seeking to enroll. We hoped that interested parties would see that hundreds of their colleagues had already enrolled, increasing their motivation to enroll as well. Previous research has found that faculty and residents were often unclear about their institution's stance on the treatment of OUD 10 which adds to an absence of social norms; therefore, our course was jointly sponsored by the American College of Emergency Physicians, the largest society for emergency medicine (EM) physicians, effectively demonstrating new, pro-MAT social norms at the specialty level. Second, the GW Remote course sought to decrease the level of hassle bias experienced in obtaining the DEA-X waiver. Hassle bias describes the obstacles that can make a task more difficult, thereby decreasing the likelihood of completing the task. 10 This occurs when individuals delay completing a task or by reducing follow-through. By introducing a remote, digital course focused on maximizing ease of access, the session aimed to decrease this hassle bias. This was accomplished in two ways. First, the traditional course requires that clinicians drive to a central location where the course is held. GW Remote offered clinicians an opportunity to access the training in a more convenient fashion, via an online platform, reducing the hidden financial and time costs associated with traveling to and from the course training location. Second, by laying out an easy-to-follow process to waiver training completion on a user-friendly, navigable website, and presenting information about how to sign up for the waiver process in a streamlined fashion, we aimed to address the administrative complexity associated with completing the waiver process. After completing the free online course, participants were educated on how to submit their certificate of completion to substance abuse and mental health services administration (SAMHSA) to receive the waiver. Third, our program attempted to mitigate the lack of salience in treating OUD, which also acts as a barrier to obtaining the waiver. 10 Salience refers to the idea that individuals focus on pieces of information that are more prominent to them and ignore those that are less so. In doing so, a bias is created in engaging in behaviors that are more striking and perceptible. This may not be so apparent in treating OUD because the long-term benefits and outcomes in treating these types of patients are often hidden and at times not readily apparent to the clinician who experienced the initial encounter. To counteract these biases, the recruitment materials used to promote the GW Remote course highlighted the urgency of treating OUD in the context of COVID-19, linking waiver training to a highly salient issue among Emergency Medicine physicians in May 2020. Further, the training was delivered by DEA waivered speakers that introduced clinical vignettes containing reallife patient situations highlighting stories of recovery. Course content and its method of delivery were developed by Providers Clinical Support System. Presenters had experience with DEA-X waiver courses previously delivered in-person. Clinicians (faculty, residents, nurse practitioners, and physician assistants) and medical students and nurse practitioner (NP)/physician assistant (PA)s in training were eligible to register for the course. The didactics were delivered via the Zoom platform on May 20th, 2020 from 10 AM to 6 PM Eastern Standard Time, totaling the required eight hours. During this time, participants studied a series of modules covering all aspects of treating OUD patients with buprenorphine. The course allowed for participants to interact in a variety of ways, both within the zoom platform and on external social media platforms ( Figure 1 ). In addition to the core presentation, the digital, remote platform allowed individuals to ask questions which were then answered in real time by additional staff with clinical and administrative experience with both completing the waiver process and prescribing buprenorphine. To measure the engagement of the participants, poll questions were sent to all participants during the course to measure learning and retention. Additionally, the event was made free for participants through sponsorships from the foundation for opioid response efforts (FORE) foundation and by a grant from SAMHSA. We utilized the RE-AIM model to evaluate our training session's outcomes. The five RE-AIM dimensions include the following, Reach, Efficacy, Adoption, Implementation, and Maintenance. These categories are examined at both the level of the individual and the organization to assess the delivery, reception, and overall efficacy of public health interventions. According to Glasgow et al., "reach" is a measure of individual participation; more specifically, it refers to the percentage of individuals in a defined population who are affected by a new program or policy, broken down by demographic categories. "Effectiveness" measures outcomes of the program in terms of end results, in this case, completing the GW Remote course. "Adoption" is the proportion of individuals or organizational settings that affect change to adopt a given policy or program and corresponds to predicting future behavior. "Implementation" refers to the extent to which the program is delivered as it was originally intended. Finally, "maintenance" is assessed at an organizational and community level to evaluate the extent to which the policy becomes routine and part of regular practice. 11 Below, we report results for the first four of the elements of the RE-AIM model. Appendix 1 contains a matrix of our data sources and analyses and Appendix 3 addresses the survey development and deployment process. All physicians, nurse practitioners, physician assistants, and medical students were invited to participate. In total, we received 1179 enrollments during the enrollment period. A total of 814 participants joined the Zoom meeting delivered on May 20th, 2020, completed a precourse survey, and participated in the first GW Remote pilot. Of those who participated, 40.79% (n = 332/814) were physicians, 18.06% (n = 147/814) were residents, 3.44% (n = 28/814) were fellows, 5.90% (n = 48/814) were nurse practitioners, 14.13% (n = 115/814) were physician assistants, and 17.69% (144/814) were medical students. 59.46% (n = 484/814) participants learned about Get Waviered Remote via email, 15.97% (n = 130/814) from a text from a friend, 7.13% (n = 58/814) from the GW Website, 6.51% (n = 53/814) from Facebook, 6.14% (n = 50/814) from Instagram, and 4.79% (n = 39/814) from Twitter. Postcourse survey revealed that 55.34% of participants were in or entering Emergency Medicine as a specialty (n = 57/103). Between August 6, 2019-June 6, 2020, the GW Remote website's Google Analytics revealed: (1) Users: There were a total of 8919 users (Tables 1 and 2 We surveyed participants about their experience with GW Remote using a postcourse survey, with a 12.65% response rate (n = 103/814 participants). 95.15% rated their overall experience about what they had expected or better (n = 98/103). When presented with a likert scale and asked how likely one would be to recommend the remote course to a friend or colleague, 88.35% (n = 91/103) rated a score of 7 or greater. When compared to standard in-person courses, 92.23% of participants (n = 95/103) felt that course delivery via the digital remote platform was equivalent to or better than their usual academic proceedings. When assessing Q&A comment features, we found that 14.67% of total comments (n = 66/450) were related to a technological issue. 34.52% (n = 281/814) of participants interacted with others enrolled through the chat function or Q&A, sending a total of 450 messages over the course of the 8 h of the course (Tables 3 and 4 ). The GW Remote course was an effective solution for delivering live, real time educational content while limited by in-person gathering restrictions due to COVID-19. The delivery platform also allowed for collaborative interaction among speakers, moderators, and attendees, something rated positively by participants. Our digital conference implementation and delivery style may provide a framework for future medical education conferences set to impact public health policy across the nation. GW is an initiative that was founded with the goal to expand the waiver network by encouraging and assisting clinicians to get their DEA-X waiver. The GW team includes highly motivated individuals ranging from boardcertified physicians with tremendous experience in emergency medicine to aspiring physicians of various backgrounds looking to better understand and act on the opioid crisis. The GW Remote X waiver training course is a novel implementation of the classroom-based DEA-X waiver training course and the largest known course at the time of this manuscript. It has the potential to scale the DEA-X waiver course to a national level among healthcare providers in the setting of COVID-19 and beyond. It was seen as a convenient option for providers interested in completing their X waiver training, successfully reducing the hassle bias associated with low completion rates. Further, the postsurvey results showed that remote courses can be effective educational opportunities for medical professionals, comparable to or even better than, in-person courses. This indicates that training opportunities can be made more convenient to providers, thereby increasing the number of providers who complete additional training, without reducing the quality of the course. Results of the comment analysis suggest that most participants that submitted comments sought to obtain information with the intention of receiving a real time response from course moderators or facilitators rather than from other participants. Among this predominantly unidirectional comment thread, the most frequent poster intentions were contributions either seeking information or nonquestion comments. It may be worth further investigating why there was not much bidirectional conversation among the course's participants. Furthermore, the implications of more or less bidirectional interaction in real time comments are not known. Therefore, in future studies, surveying the level of participation with others via questions, comments, and answers may be important in gauging the overall user experience and success of course delivery. Most participants posted questions about course content, receiving course credit, and others submitted concerns primarily related to the remote learning technology. Potential reasons for the last two topics include heterogeneity in user technological ability, connectivity issues, uncertainty on the impact of switching devices, and lack of periodic attendance checks. Clearer communication on recordkeeping, particularly that the webinar platform automatically records user participation would be helpful in the future. Finally, our framework can be extrapolated to other campaigns and other new educational experiences that may allow for audiences to have the ability to ask questions in real time to concept experts. This application will allow those to fulfill learning collaboratives organized by authorities from other institutions. Lastly, the final component of the initiative takes a proactive step towards finding new organizations and members to adopt a protocol for more expansive coverage. Overall, of the 8919 users who visited the site, representing a 13% conversion rate. It could be worth investigating why there were nearly double the number of male users compared to female users. It may be worth focusing development efforts on the Enrollment and Home pages, which together receive nearly half of all traffic. More than onefifth of all users who visit the Enrollment page continue to other pages on the site, which may indicate that users want more information before registering. Though the top sources of traffic were Direct and Social, which would encompass GW's email and Instagram/Twitter campaigns, higher engagement came from Organic and Referral traffic, which indicates a stronger ROI from content developments, SEO, and public relations. The final dimension of the RE-AIM model is to examine maintenance. According to Glasgow et al. maintenance is evaluated on both an individual and organizational- 11 and in our case, this includes the number of clinicians utilizing the evidence-based approach of prescribing buprenorphine in the treatment of OUD. Maintenance also measures the extent to which the implementation of the policy becomes stable overtime. 11 Given the nature of this dimension's evaluation, this assessment will require more time for adequate evaluation. We plan to follow up with participants in the future to survey the impact the waiver course had on their prescribing practices, which will allow us to assess the extent to which a remote waiver course helped overcome the obstacles clinicians face in providing addiction treatment during the COVID-19 pandemic. Given widespread OUD, real time, digitally delivered, interactive medical education courses such as GW Remote have the potential to positively impact the treatment of patients nationwide. Contributorship: All authors contributed equally, reviewed, edited, and approved the final version of the manuscript. Declaration of conflicting interests: The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. Ethical approval: This project was undertaken as a Quality Improvement Initiative at Massachusetts General Hospital, and as such was not formally supervised by the Institutional Review Board per their policies. Not applicable, because this article does not contain any studies with human or animal subjects. Funding: The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was funded by the Foundation for Opioid Response Efforts (FORE). The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies or stance, either expressed or implied, of FORE. FORE is authorized to reproduce and distribute reprints for Foundation purposes notwithstanding any copyright notation hereon. Guarantor: SH. Informed consent: Not applicable, because this article does not contain any studies with human or animal subjects. ORCID iD: Joshua P Raber https://orcid.org/0000-0003-2330-5387 Trial registration: Not applicable, because this article does not contain any clinical trials. Google Analytics (Google, LLC, Mountain View, CA) was installed on GetWaivered and used to track user data during the initial stages of the operation (August 6, 2019-June 6, 2020). This allowed collection of data related to user behavior emanating from a user's interaction with the website. The data could come from avenues such as the uniform resource locators of the pages the user viewed, the location of the user, and the type of device being used to access the application. Google Analytics also collected information about the nature of the visit such as the content viewed and length of the session. Such information was summarized in a real time, interactive dashboard format on the Google Analytics website, which could only be accessed by logging in with a user account that had been granted permission by the development team to access the data. Google Analytics did not collect any personally identifiable information and presented all collected data in aggregate form, thus mitigating the ethical concerns that could arise from user behavior research. 12, 13 Comments This study included data obtained from a precourse and postcourse surveys completed by participants and the question-and-answer feature of the course platform, which allowed for the real time exchange between presenters, moderators, and participants. The contents of the Q&A were exported to a spreadsheet for further analysis by the authors of the study. We retrospectively reviewed and analyzed the question-and-answer contents of the exported data. The data were manually coded by one author of this paper. Those involved in coding collaborated to include a consensus on codebook definitions, confirm categorical data assignments, and ensure reliability. The examination of question-and-answer participation, poster intention (PI), and comment features (CF) were qualitatively assessed modeling a previously used coding framework which examined the effectiveness of academic social networking sites and how scholars exchanged information and resources. [14] [15] [16] Poster intention determined the purpose and expectation of the participant commenting. Content features examined the details of the post. If a single comment contained more than one question or comment, it was coded for each individual question or comment submitted. The following category designations were used to denote poster's intentions and assigned a value 1-5: "Seeking information," "Seeking discussion," "Non-question," "Answering a question," and "Furthering discussion." A single post could be assigned multiple PI values. The following category designations were used to denote comment features and assigned a value 1-9: "Providing factual info," "Providing personal experiences," "Making an inquiry-Administrative questions about the waiver process," "Making an inquiry-Course content," "Making an inquiry-Technological," "Making an inquiry-Attendance/Course Credit," "Providing opinions," "Requesting resources," and "Off topic." A single post could be assigned multiple CF values. My department/hospital strongly encouraged me to do this. My friend/colleague strongly encouraged me to do this. The new barriers to treatment facing patients because of COVID19 motivated me. What was the most important barrier preventing you from getting waivered previously? * (select one) The associated monetary costs. The associated time and hassle. It was not mandated or encouraged by my hospital/ department. There was no clear benefit. I did not know how to initiate the process. 1-I strongly disliked it, would never do this instead of regular conference/sessions 2-I disliked it compared to regular conference/sessions 3-I didn't like it as much as regular conference/ sessions 4-Neutral 5-It was marginally better than regular conference/ sessions 6-I liked it compared to regular conference/sessions 7-I highly preferred it, would prefer this to regular conference/sessions Other How likely is it that you would recommend remote course to a friend or colleague? * (select one) 1 2 3 4 5 6 7 8 9 10 Prior to this experience, how familiar were you with the practice of opioid dependency treatment with approved buprenorphine medications? * (select one) Not at all familiar Not so familiar Somewhat familiar Very familiar Extremely familiar NOW AFTER this experience, how familiar were you with the practice of opioid dependency treatment with approved buprenorphine medications? * (select one) Not at all familiar Impact of non-pharmaceutical interventions (NPIs) to reduce COVID19 mortality and healthcare demand Rethinking residency conferences in the Era of COVID-19 COVID-19 and substance use disorders: recommendations to a comprehensive healthcare response An epidemic in the midst of a pandemic: opioid use disorder and COVID-19 Medications for opioid use disorder save lives. The national academies collection: reports funded by national institutes of health Opioid overdose prevention education for medical students: adopting harm reduction into mandatory clerkship curricula Opioid overdose prevention training with naloxone, an adjunct to basic life support training for first-year medical students Maternity care and buprenorphine prescribing in new family physicians Get Waivered: a resident-driven campaign to address the opioid overdose crisis Evaluating the public health impact of health promotion interventions: the RE-AIM framework Evaluating information seeking and use in the changing virtual world: the emerging role of Google analytics An objective approach to evaluating an internet-delivered genetics education resource developed for nurses: using Google analytics™ to monitor global visitor engagement Information exchange on an academic social networking site: a multidiscipline comparison on researchgate Q&A An emerging form of public engagement with science: ask me anything (AMA) sessions on Reddit r/science Addressing immediate public coronavirus (COVID-19) concerns through social media: utilizing Reddit's AMA as a framework for public engagement with science Not so familiar Somewhat familiar Very familiar Extremely familiar What did you like about the event? (open-ended response) What did you dislike about the event? (open-ended response) Final Thoughts See Table 5 and Figures 2 and 3. Table 5 . Contains a matrix of all the data sources and our analysis approach for each. Types of data Analysis Categorical and numerical data Geographical demographic, gender, and traffic source estimation dataQualitative and quantitative analysis of descriptive statistics within Google analytics results, including the following metrics: We retrospectively reviewed and analyzed precourse and postcourse surveys content's exported data. The data were exported to Microsoft Excel and organized by question. The data were manually coded and analyzed by two authors of this paper. Appendix 3 explains our survey development process and contains our survey tool. In order to construct the surveys used in this study, one evaluation team member (SH) modeled previous surveys that had been designed to evaluate ALiEM Connect. Bearing in mind the five RE-AIM dimensions, the current surveys were constructed to evaluate the efficacy domain, which seeks to appraise the usefulness of a program to its participants. As such, many of the survey items sought to assess participant experience during the Get Waivered Remote course.After producing precourse and postcourse survey questions, the survey tool was created using SurveyMonkey (San Mateo, CA, USA). The surveys were first piloted to other members of the Get Waivered team. Following the pilot, members provided feedback about question readability, quality, applicability, and overall survey experience. Results from the feedback obtained during the pilot were incorporated into the final forms. Please see Parts A and B below for the full surveys that were distributed to participants. Email-An email was sent to all participants enrolled in the course. What is the most important reason you chose to get your DEA-X waiver at this time? * (select one) A recent patient encounter motivated me. This was a convenient time/format to do this.