key: cord-0887945-qq43jf6n
authors: Vishwanathan, Karthik; Joshi, Ankit Nitinkumar; Patel, Mit Jitendra
title: Perception of Medical Undergraduate Students about an Electronic Practical Examination In Orthopaedics During the COVID-19 Pandemic
date: 2021-11-29
journal: Indian J Orthop
DOI: 10.1007/s43465-021-00578-0
sha: 3c79a11508f0f32d0c286dd9086888cedc8cfefb
doc_id: 887945
cord_uid: qq43jf6n

BACKGROUND/PURPOSE OF STUDY: No study has evaluated the perception of medical undergraduate students to the electronic objective structured practical examination (e-OSPE) in orthopaedics. The aim of the present study is to evaluate the feasibility and perception of the medical undergraduate students to e-OSPE conducted by the department of Orthopaedics to assess problem-solving and clinical decision-making skills of medical undergraduate students. METHODS: Medical undergraduate students of second and third year at our medical college who completed the orthopaedic clinical posting and appeared for the orthopaedic practical examination during the second wave of the COVID-19 pandemic were included in this prospective observational study. Students appearing for the exams from 20th March 2021 to 26th June 2021 were invited to complete the questionnaire immediately after the e-OSPE. Internal consistency of the survey questions was assessed using Cronbach’s alpha. RESULTS: 272 out of 312 eligible students completed the survey and the survey response rate was 87.2%. Nine groups of medical undergraduate students gave the orthopaedic practical exams from 20th March 2021 to 26th June 2021. 91.2% students felt that the e-OSPE represented a valid modality of evaluation of essential orthopaedic practical knowledge during the COVID-19 pandemic. The overall reliability of the 19 questions included in our survey was very high (Internal consistency: Cronbach’s alpha = 0.88). CONCLUSION: The e-OSPE was well received by the medical undergraduate students at our institute and the students had a positive perception about the new examination technique used in orthopaedics during the COVID-19 pandemic.

The onset of the COVID-19 pandemic had prompted medical colleges in India to shift to online delivery of medical teaching to avoid compromising the teaching-learning process [1, 2] . The COVID-19 pandemic has harmed postgraduate training in orthopaedics across various countries [3] [4] [5] [6] . Malhotra et al. from AIIMS, Delhi have described their experience of conducting postgraduate orthopaedic examination during the COVID-19 pandemic and the students taking the examination and the examiners gave positive feedback about the examination process [7] . Suggestions have been given for conducting postgraduate orthopaedic examination during the pandemic [8, 9] . Experience from India has shown that electronic objectively structure practical examination (e-OSPE) is a feasible, valid, and reliable alternative in comparison to the traditional OSPE in the field of biochemistry during the COVID-19 pandemic [10, 11] .

There was a huge surge in COVID-19 cases in major cities of our state during the second wave, and hence, another lockdown was imposed in our city from 23rd March 2021. The medical undergraduate students attended offline and online lectures and the orthopaedic department devised a novel and innovative method (image-based short question-answer practical examination) for the assessment of students.

Medical undergraduate students attended orthopaedic posting, but to maintain social distancing and safety of students, we conducted an image and video-based examination. No study has evaluated the perception of medical undergraduate students to the e-OSPE in the speciality of orthopaedics. The present study aims to evaluate the feasibility and perception of the medical undergraduate students to the novel electronic objective image-based short questions-answers conducted by the Department of Orthopaedics to assess problem-solving and clinical decision-making skills of medical undergraduate students.

Approval from the Institutional Ethics Committee for human research was obtained for the conduct of the present study and the participating medical students gave online written consent. Participation of the medical undergraduate students in the post-examination feedback was voluntary. There were no rewards or incentives for participating in the study.

Medical undergraduate students of 2017 (third clinical posting), 2018 (second clinical posting), and 2019 (first clinical posting) batches at our medical college who completed the orthopaedic clinical posting and appeared for the orthopaedic practical examination during the second wave of the COVID-19 pandemic and those giving consent for voluntary participation were included in the study. Students appearing for the exams from 20th March 2021 to 26th June 2021 were invited to participate in this prospective observational study. Students that did not appear for the clinical exam and those that appeared for the exam but did not give consent for participation were excluded from the study.

The practical examination consisted of 5-10 image-based questions. Students and faculty members were given verbal and written instructions regarding the pattern of the examination. The students were instructed to write the answers in pen and paper format, and were instructed not to answer any phone calls during the duration of the examination that lasted from 20 to 30 min.

The examination was conducted using Google meet or Zoom meet application. The entire batch of students was equally divided amongst various faculty members. Each faculty member was allotted 4-5 students for invigilation. Each faculty member posted the meeting link on the WhatsApp group of the batch of students posted in orthopaedics 1 day before the examination. The Microsoft PowerPoint slides were posted to the allotted invigilators the evening before the examination. The examination commenced at 9:30 am. All students were instructed to keep the mobile cameras on and to unmute the mikes for the entire duration of the exam. The cameras had to remain focused on the student's face. Each slide had a clinical picture, clinical video, the image of radiograph, the image of bone, the macroscopic image of pathology specimen, or the image of an orthopaedic implant or appliance. All questions about the photograph/video were posted on the slide adjacently. The time and marks allotted for each question were posted in the upper part of each slide. After a designated time of 3-5 min, the invigilators automatically changed the slides.

After the exam was over, the students were instructed to scan their answer sheets using a mobile scanning application and were instructed to convert it into a single PDF file and post it to the invigilator on WhatsApp. The invigilators were instructed to ensure clarity of the answer sheets in PDF format before allowing the students to leave the Zoom meet/ Google meet. All the answer sheets were collected electronically by the invigilators and then posted answer sheets to two senior faculty members for assessment to maintain uniformity in evaluation. Resubmission of answer sheets was not allowed once the student left the meeting room.

Once the examination was over and all answer sheets were submitted to the respective invigilators, a formative feedback session was organized online to discuss the correct answers and tips were given on how to approach problembased questions.

One day before the examination instruction was given to all students to wear masks, maintain social distancing etiquettes and get their writing pads and papers. All students assembled in the orthopaedics ward teaching room that was located on the second floor of the hospital. All students were asked to keep their mobile phones inside their bags.

During the orthopaedics term, approximately 36-40 students were posted out of the total batch of 150 students. The majority of the groups had a cohort of 37 students posted in orthopaedics. The entire batch of students of 37 students was divided into two groups of 19 and 18 students. The offline examination was conducted in the seminar room of the orthopaedics department that was located on the third floor of the hospital. The first batch of 19 students went to the orthopaedics seminar room, while the second batch waited in the teaching room under the supervision of intern doctors. The questions were in Microsoft PowerPoint format and were projected on a large screen with the help of a portable LED projector. Faculty members of the orthopaedic department were present as invigilators during the exam. After the ending of the exam for the first batch, all students of the first batch were requested to assemble in another teaching room on the third floor under the supervision of another intern doctor. Then, the second batch of 18 students was instructed to come to the orthopaedics seminar room under the supervision of the intern doctors. Once the second batch of students took their seats for the examination, the first batch of students was then free to use their mobile phones. The second batch of students then completed the offline examination, and after the examination, feedback was given to both batches of students. This method ensured that all students had the same set of questions no communication was allowed between the two batches of students, as the use of mobile phones was not allowed during the conduct of examination of both the batches.

The questionnaire used in the present study was adapted from previously published studies [12] [13] [14] [15] . The participants were invited to give responses to the questions based on the five-point Likert scale (Table 1 ). The options were: strongly agree, agree; neither agree nor disagree, disagree, and strongly disagree.

The Principal Investigator announced the details of the survey soon after the clinical exam and advertised the invitation on the WhatsApp group created for students posted in Orthopaedics Department. The link to the survey questionnaire (26 items) on Google Forms was posted on the What-sApp group of the batch of students soon after the exam within a few minutes (20th March 2021, 26th March 2021, 10th April 2021, 23rd April 2021, 1st May 2021, 21st May 2021, 22nd May 2021, 18th June 2021, and 26th June 2021). The administrator closed the survey response after 48 h. The students were able to review their answers, but editing of responses was not possible after submission.

Categorical data were presented as proportions and percentages. IBM SPSS version 20 was used for the statistical analysis. For the Likert response, data were presented as mean, median, standard deviation, and interquartile range (25th percentile to 75th percentile).

Internal consistency of the survey questions was assessed using Cronbach's alpha. A Cronbach's alpha value ≥ 0.75 is an indicator of good correlation amongst all the questions and suggests that the questions are measuring similar domains [16] . Figure 1 shows the justification for including and excluding participants. 272 out of 312 eligible students completed the survey (response rate was 87.2%). Nine groups of medical undergraduate students gave the orthopaedic practical exams from 20th March 2021 to 26th June 2021. Table 2 

The response of the participants to various survey questions is summarized in Table 1 , Fig. 2 . 96.7% of participants agreed or strongly agreed that the questions were based on the application of knowledge. 93.8% of participants agreed or strongly agreed that the exam was well structured. 90.4% of participants agreed or strongly agreed that the exam was well administered. 92.6% of medical undergraduates agreed or strongly agreed that the number of questions was adequate. 85.3% of participants agreed or strongly agreed that the questions were clear. 85.3% of students agreed or strongly agreed that the case details provided in the slides were sufficient. 84.6% of students agreed or strongly agreed that the quality of clinical image, radiographic image, and video were sufficient to reach a diagnosis. 84.6% of participants agreed or strongly agreed that adequate time was given for each question. 93% of students agreed or strongly agreed that the new exam format led to the evaluation and promotion of their problem-solving and decision-making skills in orthopaedics. 89.7% of students agreed or strongly agreed that their progress was evaluated. 90.4% of medical undergraduate students agreed or strongly agreed that the present exam format was acceptable to them. 91.2% of students agreed or strongly agreed that the new exam format represented a valid modality of evaluation of essential orthopaedic practical knowledge during the COVID-19 pandemic. 89% of students agreed or strongly agreed that the present exam format was the most practical method for assessment of the application of knowledge during the COVID-19 pandemic. 91.5% of students agreed or strongly agreed that the exam format was standardized and uniform for all students.

For the open-ended question on strengths of the new exam format, 161 participants (59.2%) gave a positive response, six participants (2.2%) gave a neutral response, and three participants (1.1%) gave a negative response. Some of the positive responses from the participants are summarized in Table 3 .

For the open-ended question about the limitations of the new exam format, 71 participants (26.1%) were satisfied and perceived the absence of limitations, 31 participants (11.4%) felt that the exam should have had a viva-voce component and exposure to real patients, 19 participants (7%) felt that time for each question was inadequate, 14 participants (5.1%) had issues with Internet and mobile phone connectivity, six participants (2.2%) felt that the exam was an "all or none" format wherein either you know the answer or you do not know the answer, five participants (1.8%) felt that more exposure to such questions should have been given during small group teaching, three participants (1.1%) felt that the offline exam pattern was more time-consuming, two participants (0.7%) felt that less clinical information was given in the slides, and one participant (0.4%) perceived the exam format to be stressful.

For the open-ended survey question on suggestions for improvement in the exam format, 97 participants (35.7%) were satisfied and felt that no change was needed in the new exam format, 13 participants (4.8%) suggested increasing the time for answering each question, 12 participants (4.4%) have recommended sensitization and training of tackling problem-based questions during small group teaching, 12 participants (4.4%) have suggested the addition of multiplechoice questions to the exam format, 3 participants (1.1%) suggested incorporation of viva-voce component, one participant (0.4%) has suggested increasing the number of questions, and one participant (0.4%) felt that the exam should be more patient-oriented.

The overall reliability of 19 questions included in our survey with a five-point Likert response was very high (Internal consistency: Cronbach's alpha = 0.88). This suggests that all the questions were measuring the same domain of participant perception to the e-OSPE. 

x Did not undertake the survey (n = 36) x Undertook survey but did not give consent (n = 4)

We need to adapt to the challenges posed in orthopaedic education of postgraduate students [17] , interns [18] , and undergraduate students during the present pandemic.

There is a paucity of evidence related to the use of innovative assessment methods. Though many articles [19, 20] have discussed methods to improve orthopaedic education, no study has addressed the question of the practical application of knowledge of undergraduate medical students in orthopaedics.

Medical undergraduate students of our medical college had a positive perception of practical orthopaedic assessment using the e-OSPE during the COVID-19 pandemic. Our sample of 272 participants was comparable to previously published studies [12, 15, 21] . On the five-point Likert scale, the median values of questions on examination being well structured, the examination being well administered, and clarity of questions asked were comparable to a previously reported study [15] . However, the median values of questions about the reduction in bias due to influence of social and personality-related factors, evaluation of progress, acceptability of new exam pattern, exam being the most practical method to evaluate the application of knowledge, and exam being standardized for all students were higher in the present study compared to the study by Majumder et al. [15] . The study by Majumder et al. [15] was conducted in the West Indies, was a summative university examination of final-year MBBS using the traditional OSCE method, and included all topics except orthopaedics. Our study pertained to formative assessment, and hence, probably students were under less stress compared to those appearing for summative assessment and hence the difference in perception of stress levels. The above differences could be plausible explanations for the difference in observed values. On the five-point Likert scale, the mean values of questions about helping in the evaluation of progress and examination pattern being acceptable were higher in our study compared to that reported by Furnmedge et al. [13] . The study by Furmedge et al. had a considerably larger sample size and evaluated the role of traditional OSCE for formative, low stake assessment in various colleges in the UK. The proportion of students agreeing to questions on clarity of questions, provision of adequate time to answer the questions, and the ability of the exam to evaluate problem-solving abilities was higher compared to previously reported values [12, 21] . Both the studies evaluated the traditional OSCE, whereas our study evaluated e-OSPE and this could be one reason for the difference in values. Assessment of medical students is challenging during the present pandemic, and in some countries, there has been a cancellation of clinical exams, and in some centres, online assessments relying on identification of clinical signs from videos were used [22] . A survey amongst medical undergraduate students in the UK has revealed that about 37% of students perceived that their training in orthopaedics was poor [23] . Though the study included only final-year students and the mean duration of clinical posting in orthopaedics was 2.5 weeks, many students perceived that they lacked competence in basic orthopaedic skills. In India, the total duration of clinical posting in orthopaedics ranges from 8 to 10 weeks and the new guidelines from the National Medical Council of India aim to provide competency-based training to medical undergraduates.

Objective question patterns such as Single Best Answers (SBA) and Extended Matching Questions (EMQ) could be assessed in theoretical exams, however; practical exams should assess the reasoning and decision-making abilities of the students. Our examination pattern included the use Table 3 Responses from participants regarding strengths of the new examination format "Problem solving based questions requires knowledge and based on the results of the exam self evaluation can be done" "Clinical case base question and how to reach a diagnosis" "It was less stressful and almost all topics were covered unlike viva where we read everything n just 5 min viva is there so this exam is more scoring. problem based questions were there so it'll b helpful to us in PG entrance exam..because pattern is quite same" "Clinical base question solving skills get strong and discussion after exam gets us know our mistakes and how to approach a question" "Based on only practical knowledge" "Less crowded area, images and videos" "No patient is required, application of knowledge assessed" "In current circumstances, this was the only exam that was conducted with all COVID-19 precautions and measures so far. Appropriate clinicalbased questions were incorporated" "Great format and less chance of copying " "Clinical orientation & our ability of quick thinking in limited time. + same questions to everyone makes it unbiased" "Student friendly" "Online zoom meet in presence of invigilator" "The exam was well balanced about how to reach diagnosis by our knowledge with image and tests" "It is based on checking problem solving attitude of students. Based on clinical cases. And writing paper in front on examiner makes it trustworthy processes of evaluation and reduces chances of malpractice" "Bias will be reduced and got sufficient time to explain what we know which is not possible everytime in viva. And also the last part post-exam for discussion of questions was great" "Less stressful and conceptual thinking" "Clinical questions were well structured, though the questions were a bit difficult, looking at the positive side one question cleared more than one concept" "The exam did impart proper knowledge in the form of assessing an Orthopaedic patient and helped me know how the investigations were carried out when after exams discussion was there and answers were discussed. Broad spectrum information in form of diagnosing, clinical finding and then investigations. I really appreciate the way exam was conducted and would like to give such exams in future also" "Image based learning which helps us in clinical situations" "Frankly speaking some Gujarati medium students not have that kind of English fluency which is must require in Viva, so due to that reason they could not express themselves very well. So this is the strength of this format that they can write and don't lose their confidence" "The photos presented during the exam which makes us think practically are the strength of exam" "It will inculcate fast decision making quality, because there is a time constraint for a particular question, it will prevent overthinking while in exam" "More of practical knowledge and skill assessment rather than routine fact checking. Students are encouraged to think in application based manner so is quite valuable in our clinical practice to become a competent clinician" "Sharpening of the clinical knowledge on the basis of the clinical findings given In the questions. 2. Vertical integration of the medical subjects" of open-ended questions to evaluate the problem-solving abilities of the medical students. Boscardin et al. [24] have reported the beneficial effects of open-ended questions and their ability to predict performance in future high stake examinations.

End of clinical posting assessment is a continuously evolving process and the feedback obtained from the medical undergraduate students reaffirms our beliefs about maintenance of standard and quality of the evaluation process. Constructive feedback from the students helps us to improve the assessment process, so that it helps the students to prepare for high stakes' summative assessment examinations for obtaining their medical licence and to gain entry to postgraduate training posts. A formative evaluation method should be able to encourage and motivate the students to perform better in future assessments, and should be able to give an idea to the students about their progress in the subject.

All students get the case of similar complexity and there is less chance of luck factor playing a role. It is possible to examine students' knowledge and problem-solving capabilities based on various orthopaedic conditions. The advantage of the novel examination method as compared to traditional OSCE is summarized in Table 4 . The advantages of the present examination system include its ability to test problemsolving skills, clinical reasoning abilities, interpretation, and decision-making abilities of students. The objective was to evaluate "Knows how" domain of Miller's pyramid instead of testing recall knowledge. This pattern of examination is likely to benefit the medical undergraduate students for their future high stake examinations.

There are no valid scales to assess the perception of medical undergraduate students to new methods of examination and the scale used in our study was based on questionnaires used in previously published studies [12, 13, 15] . In the present prospective study, the students were allowed to submit the responses within 48 h from the time of examination to minimise recall bias. Offline teaching was suspended for the medical students at our medical college, and hence, we had shifted to complete online teaching and assessment from 22nd March 2021 to 15th June 2021. We had offline teaching for the students from 11th January 2021 to 20th March 2021 and from 16 to 26th June 2021. In our study, six batches (66.7%) had online teaching and online assessment, two batches (22.2%) had hybrid methods with online teaching and offline assessment, and one batch (11.1%) had offline teaching and an offline assessment.

The present e-OSPE has some disadvantages that need acknowledging. The new examination format would evaluate the interpretation of history given by the patient but would not evaluate history taking abilities of the student. The examination would only evaluate the inspection skills of the student, but would not be able to evaluate palpation skills or ability to test the passive range of motion or ability to perform special tests. In addition, our examination pattern would not be able to evaluate communication skills and other professional skills that are evaluated during the viva-voce examination. However, there are better ways to determine professional skills such as multisource feedback. All the exams conducted by our department were e-OSPE patterns and we did not compare our results to the traditional OSPE.

Ours is the first study to describe the perception of medical undergraduate students to the novel assessment method used for formative assessment in the subject of orthopaedics during the COVID-19 pandemic. Our study deals with the perception of medical students from a single medical college, and hence, we would need to ascertain whether reproducible results could be obtained from other medical colleges. Our sample size was large and the response rate was high.

The e-OSPE was well received by the medical undergraduate students at our institute and the students had a positive perception of the new examination technique used in orthopaedics during the COVID-19 pandemic. We need to have contingency plans in place should we need to revert to online educations methods at short notice. The transition can be smooth only if we are well prepared in advance and the orthopaedic teaching fraternity shares its innovative teaching and assessment methods for undergraduate and postgraduate students. We have described our experience with the new format of examination for formative, low stake assessment and the encouraging feedback from our study which makes this format a viable option even for summative assessment should the pandemic force such a situation in the future. Table 4 Comparison of advantages between traditional OSPE/OSCE and e-OSPE *Time duration for ensuring rotation of students, avoiding interaction among students' sub-groups, collection of answer sheets, and allowing time for students to collect their belongings and disperse to the waiting room Traditional OSCE Image-and Video-based SQA (e-OSPE)

Number of stations 5-10

Total duration required for a batch of 38 students (considering 3 min duration for each station) Exam will need to conducted for students in four batches ( Additional arrangements for students To ensure students who gave the exam do not mix with students yet to give the exam For offline exam, waiting time would be 15 min only, because once the second batch starts giving the exam, the first batch of students can leave from the waiting hall One-way traffic system needs to be implemented For online exam, there is no need for waiting of students, because the exam is administered simultaneously to all students at the same time Students who have given the exam will have to wait for longer time till the entire examination process gets over

For offline exam, just 2 observers are sufficient (one to administer the exam and second to supervise the students) For online exam, 8 observers needed

Medical faculty perception toward digital teaching methods during COVID-19 pandemic: experience from India

Rapid transition to online teaching program during COVID-19 lockdown: experience from a medical college of India

The impact of COVID-19 pandemic on orthopaedic resident education: a nationwide survey study in South Korea

The role of orthopaedic trainees during the COVID-19 pandemic and impact on post-graduate orthopaedic education: a four-nation survey of over 100 orthopaedic trainees

Impact of the COVID-19 pandemic on orthopaedic and trauma surgery training in Europe

Insight into the changing patterns in clinical and academic activities of the orthopedic residents during COVID-19 pandemic: a cross-sectional survey

Conducting orthopaedic practical examination during the Covid-19 pandemic

Virtual postgraduate orthopaedic practical examination: a pilot model. Postgraduate

Coordinating a postgraduate orthopaedic exam during the COVID-19 pandemic

The transition from objectively structured practical examination (OSPE) to electronic OSPE in the era of COVID-19

Evaluation of e-OSPE as compared to traditional OSPE: a pilot study

Students' concerns about the pre-internship objective structured clinical examination in medical education

Developing doctors: What are the attitudes and perceptions of year 1 and 2 medical students towards a new integrated formative objective structured clinical examination?

A pilot survey of student perceptions on the benefit of the OSCE and MCQ modalities

An evaluative study of objective structure clinical examination (OSCE): Students and examiners perspectives

Tracing the steps of survey design: a graduate medical education research example

The changing face of orthopaedic education: searching for the new reality after COVID-19

Virtual" boot camp: orthopaedic intern education in the time of COVID-19 and beyond

The past, present, and future of orthopedic education: lessons learned from the COVID-19 pandemic

Evaluating the early impact of the COVID-19 pandemic on sports surgery fellowship education

Medical students' perception of objective structured clinical examination: a feedback for process improvement

The coronavirus (COVID-19) pandemic: adaptations in medical education

Current state of undergraduate trauma and orthopaedics training in United Kingdom: a survey-based study of undergraduate teaching experience and subjective clinical competence in final-year medical students

Predicting performance on clerkship examinations and USMLE Step 1: what is the value of open-ended question examination?

Acknowledgements The authors would like to express their sincere thanks to all the participants of the study and the teaching faculty of the Department of Orthopaedics for their cooperation in the smooth conduct of the exams.

All authors contributed to the study conception and design. Material preparation, data collection, and analysis were performed by KV, AJ, and MP. The first draft of the manuscript was written by KV and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.Funding This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

The authors have no conflict of interest to declare related to the present manuscript.

The study proposal was approved by the Parul University Institutional Ethics Committee for Human Research (PUIECHR/ PIMSR/00/081734/3601). All procedures followed were in accordance with the ethical standards of the responsible committee on human experimentation (institutional and national) and with the Helsinki Declaration of 1975, as revised in 2008.Informed consent Informed consent was obtained from all patients for being included in the study.