key: cord-1055268-luzqwe7p
authors: Saraghi, Mana; Fonner, Andrea M.; Riley, Cara J.; Brady, Jason W.
title: Administering oral and written board examinations remotely during the COVID‐19 pandemic
date: 2021-06-10
journal: J Dent Educ
DOI: 10.1002/jdd.12699
sha: 002bf1c3ae7bc1bb6e7b12a07efd0165d1e71228
doc_id: 1055268
cord_uid: luzqwe7p

nan

The ubiquitous shift to remote learning and testing due to the COVID-19 pandemic affected the candidates for board certification through American Dental Board of Anesthesiology (ADBA). Upon consultation with various other dental specialty boards, it became apparent that some organizations did administer remote examinations, but none had done so without the use of established testing centers. The ADBA determined the use of testing centers to be costprohibitive due to the comparatively smaller scale; there are approximately 30 candidates scheduled for each ADBA exam. After much consideration and an initial postponement to October 2020, the ADBA opted to adapt the inperson exam to a remote format without the use of a testing center. In addition to offering the exam safely during the COVID-19 pandemic, the ADBA established three main objectives for remote exam administration: (1) avoidance of technical problems, (2) preservation of exam integrity, and (3) maintenance of exam consistency.

Technical issues arising from the new remote format of the exams were avoided with careful planning and communi-cation. The ADBA conducted one video conference (Zoom Video Communications Inc.) to explain the examination process to the candidates and answer their questions in advance of the exams. Technical problems for the ADBA written exam were mitigated because most candidates were familiar with the testing software program (ExamSoft Worldwide Inc.). Since 2018, both the ADBA written board exam and the ADBA in-training examination (ITE) have been administered using the same testing software program. Most candidates had prior experience using the testing software as the ITE is offered annually to all dental anesthesiology residents, in which they download the testing software to their personal laptops or tablets and are proctored by their respective residency programs. Prior to the written board exam, functionality of the testing software was verified by requiring each candidate to complete an unscored practice test. The practice test consisted of basic knowledge questions, unrelated to anesthesiology. It verified the candidates ability to install the software, launch the exam, flag questions for follow-up, open figures associated with the questions, use the calculator, answer questions, complete the exam, and finally upload the exam for submission. One written exam candidate had a documented learning disability and a request for extra time was accommodated with 6 h for the 4-h exam.

Administration of the oral exam required more careful technical coordination as five individuals (one candidate and four examiners) met remotely through video conferencing software (Zoom Video Communications Inc.). Examiners received a schedule with links to each virtual room. The duration of the exam remained the same as previous years, with an additional 15 min added between exam slots to allow for technical difficulties or delays. The proctors provided each oral exam candidate with a computer and noise-cancelling headphones and connected them to their assigned virtual exam rooms. Candidates were connected to two examiners for the adult exam, and then to two new examiners for the pediatric exam. An information technology (IT) expert was hired to install software and ship laptops to testing sites (16 h) and for technical assistance throughout the 2 days of exams (24 h). In anticipation of personnel and technical difficulties, the ADBA maintained a roster of back-up examiners and proctors as well as a comprehensive series of exam room links through an alternate video conferencing software (Google Meets and Google LLC.)

Although honesty and integrity are expected of every candidate, several measures were utilized to protect the integrity of the exam. 1 The proctor verified the candidates' identity and obtained their signature on a confidentiality agreement. The proctors prevented exam breaches: they ensured that all of the candidate's personal items remained outside of the room and collected any scrap paper and writing implements after the exam. Each written exam candidate was allowed to have his or her own exam device, power cable, and noise-cancelling headphones. The written exam could only be unlocked upon receipt of the exam password from the proctor. Once unlocked, the testing software blocked the user's access to other applications, such as documents or web browsers. The oral exam candidates were provided with an ADBA-owned computer with internet access. The proctors served as the meeting host and connected and disconnected the virtual meetings. Both the oral and written board exams were traditionally administered in one centralized location. Due to travel restrictions, both exams were administered at various regional testing sites secured by the ADBA (including dental schools, residency programs, private practices, and private residences), which minimized the amount of travel required for each candidate, and each proctored location had COVID-19 precautions in accordance with Centers for Disease Control and Prevention and regional guidelines.

The final objective was to maintain consistency between the previous exams and the exams given virtually. The candidates were first given an opportunity to postpone their exam for 1 year with no penalty to their board eligibility window. However, once the candidates chose to take the exam, the results and impact on their board eligibility would stand. The content of the exams was developed and implemented into the exam in the same manner as past years. Each year, the oral board examiners participate in an all-day calibration session to discuss and confirm the scoring rubric. For the virtual exam, the ADBA added an additional calibration session 1 week prior to the exam to ensure consistency in scoring methods. The scoring rubrics and passing score cutoffs for both exams remained consistent with previous years.

The execution of the examinations exceeded the ADBA's expectations. No major technical issues presented during either exam. It is assumed that exam integrity was maintained as no unusual or suspicious behavior was reported by the proctors, examiners, or candidates. The majority of candidates who had registered for spring 2020 exams opted to take the exam in October rather than defer: 16 of 27 candidates for the written exam, and 25 of 37 for the oral exam.

Although an indirect measure of exam consistency, passing rates for both exams were consistent with previous years. Historically, approximately 80.0% of candidates pass the written exam. The 2020 passing rate of 87.5% could be attributed to the relatively small number of candidates (n = 16), as one more failing score would reduce the passing rate to 81.3%. A potential selection bias is possible: individuals who were willing to take the test rather than defer may have been stronger candidates, and may have benefited from an additional 6 months to study.

The oral exam pass rate was 80.0%, consistent with previous years. Following completion of the oral exam, all scoring sheets were reviewed by the President and Vice President of Oral Exams. Scoring sheets for each candidate were screened for consistency between the two examiners; large discrepancies or any questions were flagged for review. Since the virtual format of the exam allowed for easy recording, a third examiner was able to independently review any exams with large scoring variations. The ADBA reviewed the scoring metrics for each examiner and found that there was no significant scoring variation for any examiner in comparison to past exams.

A post-exam survey (SurveyMonkey, SVMK Inc.) was sent electronically to both oral and written exam candidates upon the completion of the exam. The questions assessed the ease with which candidates felt they were able to access information and complete exam registration through the ADBA website as well as their satisfaction with the administration of the exam. The response rate was low: eight responses (50.0%) for the written and three (12.0%) for the oral exam. Most of the respondents gave consistently strongly positive feedback on the entire survey, and a minority of respondents were consistently negative. This low response rate and response pattern was consistent with past years. One possible explanation for a low response rate is that candidates were stressed, exhausted, and perhaps dejected if they felt they performed poorly. The majority of positive responses may be attributable to a response bias, where candidates felt pressure to provide positive feedback. In addition, there may have been a self-selection bias as positive respondents may be conscientious and diligent individuals who attributed their perceived ease of exam access with the efforts of the Board. It is possible to achieve a higher response rate if exam results were withheld pending survey completion. However, the Board feels strongly that a voluntary survey may help avoid perception of coercion.

The remote format offered several advantages to the oral exam. The ability of the ADBA President, lead examiners, and IT expert to remotely oversee each exam room at once, invisible to the examiners and candidates, allowed for helpful oversight and consistent pacing. The ability to remotely view the exam also provided a valuable learning experience for two newly appointed examiners who were able to discreetly audit the exam in real time. Remote exam administration was cost-effective. Expenses for live exams are approximately $50,000 and include conference rooms and examiner airfare, hotels, and food and beverage. The remote exam cost approximately $15,000 and included software (ExamSoft for written and Zoom for oral exams), hardware (eight laptops and headphones for oral exams), technical support (Robell technologies, 40 h of IT time), and thank you gifts for proctors and examiners who volunteered their time and expertise.

The ADBA's objectives for the exam were met while achieving the overall goal of administering the examinations safely and effectively during the COVID-19 pandemic. The ADBA will offer remote written and oral examinations in the spring of 2021.

American Dental Association Principles of Ethics and Code of Professional Conduct, with Official Advisory Opinions Revised to

The authors would like to thank our ADBA Executive Director, Ms. Amy Sarno, for her dedication, Mr. Matt Robell for his IT expertise, all our examiners and proctors, and the Board of Directors: Dr. Martin Boorin, Dr. Lily Hu, Dr. Christine Quinn, Dr. Phillip Yen, and Dr. Andrew Young.