key: cord-0995521-0yz4bic0
authors: Yousefi, Mehdi; Ebrahimi, Zahra; Fazaeli, Somayeh; Mashhadi, Leila
title: Continuous multidimensional assessment system of medical residents in teaching hospitals
date: 2022-04-05
journal: Health Sci Rep
DOI: 10.1002/hsr2.573
sha: 2356c61408091c68d6076fa841e0de449d83d4ef
doc_id: 995521
cord_uid: 0yz4bic0

BACKGROUND AND AIMS: Evaluation of learners is considered as one of the most important principles in education, especially in the clinical fields. Continuous assessment can be used to provide appropriate feedback to students about their strengths and weaknesses. In this regard, this study is aimed to design a system of continuous assessment of medical residents (SCMARs). METHODS: This study was performed using a combination of qualitative methods, including focus group discussion, expert panel, and Delphi technique. The study population consisted of all the stakeholders involved in the evaluation process of medical residents in Imam Reza Hospital Complex (IRHC) in Iran. This study was conducted in three phases, including identification of subthemes and indicators, production of a primary framework for the SCMAR, and agreement on the subthemes of the SCMAR. The nominal group technique was used for generating priority information. Data analysis was performed during the agreement review stage with the Excel software version 2016. RESULTS: The finalized SCMAR consisted of 10 main themes and 38 subthemes. The themes included objectives, evaluators, areas, and indicators of evaluation, evaluation periods, evaluation requirements, data collection, data sources, point assignment and data analysis methods, reporting, and feedback dissemination methods. Five areas of evaluation and 11 indicators of evaluation were proposed. CONCLUSION: A comprehensive assessment system that continuously evaluates the performance of Medical Residents can be used as a stimulus to improve the quality of educational processes. The present study was aimed to address this need by designing a framework for such a system.

In the last 20 years, there have been obvious changes in the development and research for assessment methods in medical education. 1 Students play a crucial role in the improvement of the performance of the educational and treatment system. Moreover, they are responsible for the treatment of patients, especially in unconventional shifts, in medical education centers. 2 One of the most important and challenging issues in clinical education is student evaluation. 3 Therefore, comprehensive assessment helps identify the strengths and weaknesses of education and take steps to reform the educational system by strengthening the positive aspects and eliminating the shortcomings. 4 Furthermore, clinical evaluation of students through direct observation in practical and real situations will guarantee their ability to cope and predict clinical events in specific patient conditions. 5 Clinical assessment methods that are accompanied by feedback promote learning and are able to evaluate the student in areas that are difficult to evaluate by traditional methods. 6, 7 Studies performed by Ogunyemi et al., 8 Goldstein and Zuckerman, 9 and Stark et al. 10 about the assessment of the professional skills and behavioral competence of residents suggest that assessment requires more than one group of evaluators. In other words, multiple groups of evaluators need to be involved in the assessment. 11, 12 Moreover, to improve and enhance the quality of clinical education, its status should be constantly evaluated. 13 According to the above-mentioned studies, a desirable method should continuously assess the learners and guarantee the participation of all stakeholders in the process, while giving them feedback on the evaluation results. Therefore, due to the importance of assessment based on the integrated indicators and the need to facilitate and accelerate access to the collected information, Imam Reza Hospital Complex (IRHC) decided to design and implement system of continuous assessment of medical residents (SCMARs).

The education and treatment managers of the hospital started to design and implement SCMAR to reduce the problems.

This qualitative study was conducted in three phases. The research environment was IRHC in Mashhad, Iran. IRHC is a large public teaching hospital with 1000 beds. It provides medical care for more than 6000 hospitalized patients and about 19,000 patients in the Emergency Department per month. There are~300 faculty members and 500 students in various fields of medicine in IRHC.

The main research method in this phase was focus group discussions with stakeholders involved in the medical resident assessment process. Members of the focus group discussion included six medical residents, education and health deputy of the hospital, head of the educational services department of the hospital, and two head nurses in the wards with the highest number of residents. The group had three meetings, 2 h long. They discussed the most important indicators of educational accreditation standards. Moreover, after reviewing the related literature in the focus group discussions, the requirements of clinical ethics and the performance of medical learners were discussed. The literature required at this step was prepared by the narrative review approach, using the following words and phrases in search engines Google Scholar, PubMed, Science Direct, EMBASE, and Web of Science databases, Assessment, Teaching hospital, Medical education, and medical resident.

In the meetings, each member was allowed to comment on the areas that needed improvement. Moreover, it was emphasized that the members should listen to the statements and views of the group members with respect and present their complementary opinions after they finish talking. To prevent deviation from the research objectives, the answers were guided if necessary. 14 At this phase, the knowledge, views, and attitudes of members about the SCMAR were extracted and after reaching a consensus, the aspects of this system were identified.

This phase is mainly designed to produce a primary framework for the SCMAR. Its main research method was the expert group meeting. In this phase, the nominal group technique proposed by Potter et al. 15 was used in five steps. In the first step, the study was introduced, which was followed by an explanation of the session. The topic in the session, a summary of the findings of the previous stage of the study, was presented in a session for 10 min. Then, the questions and ambiguities of the participants regarding familiarity with the study were answered.

The second step was based on the protocol of the silence to generate the idea. This step was conducted in three parts: in the first part, during a session, people were asked to think about the themes of SCMAR. Their opinions in response to the following questions were gathered: "Is this theme appropriate for SCMAR?" and "Are the themes defined providing the necessary comprehensiveness to evaluate the performance of medical residents?" Participants were given 20 min to think about these questions and generate their own ideas.

In the second part, which was conducted in two sessions, participants were asked to think about the following questions, which were about the axes with at least 75% of participants agreed: "Are these subthemes appropriate for the intended theme? And are the subthemes comprehensive enough? And what other subtheme do you suggest?"

In the third part, people were asked to think about the indicators related to the field of axis and residents' evaluation indicators and to generate their ideas on the following questions: "Are these indicators appropriate?" and "Are the indicators comprehensive enough?" and "What other indicators do you suggest?"

In the third step, for each part of the previous step, ideas were created based on Ron Robin's model. 16 Accordingly, all individuals were asked to introduce their ideas and no ideas were rejected or discussed at this step by the team leader and others. All items were displayed on the video projector screen without any discussion about the titles. This step was continued until everyone introduced all their ideas on the next topic and all ideas were recorded. The next step was a group discussion to clarify the ideas. At this step, the first volunteer was asked to talk about the opinions of her/his colleagues and discuss the issues. Afterward, everyone was given the opportunity to express their opinions.

The final step was to vote on the ideas. At this step, voting was done for each part. Items with more than 75% of the votes remained in the model and others were excluded. The result of the steps was the creation of a basic framework for continuous evaluation of medical residents, which was considered as input for the next step, which is the Delphi method.

In this phase, we conducted an expert panel and all the possible options for components in the primary framework were considered.

Components of the primary framework of SCMAR, which was obtained from the prior phase, were confirmed by the Delphi method. Purposive sampling was used for data gathering. To ensure the representativeness of the panel members, expert members included the Medical Education Committee, the research management team, and members of the focus group discussion (Table 1 ). In addition, the snowball sampling method was used to identify other participants. In the first round, themes of the Primary Framework for SCMAR were sent to the panel members. Panel members were asked to express their viewpoints about each theme with three choices: agree (1), disagree (2) , and without viewpoint (0). Themes with more than 75% Consensus were used for the secondary framework. Themes with Consensus between 50% and 75% were entered into the second round of Delphi and others were excluded.

After gathering the panel member's statements, we summarized the viewpoints in an appropriate format for feedback, so that each member received an abstract of the panel viewpoints as well as a reminder of the scores that the member had assigned to each theme. In addition, the panel members were invited to a second round. Based on the findings of data analysis in the second round of Delphi, there remained no themes with an agreement between 50% and 75%. In the final step of consensus development, the panel members were invited to see the feedback and discuss their viewpoints about the appropriateness of each theme and subtheme based on their own professional decision. The interviews focused on the applicability, conformity, and relationship between themes and approaches to amend data gathering and future steps in the expansion of SCMAR. All participants were consented to participate in the study.

In this study, six participants did not participate in the second round of Delphi due to their busy work schedules. As shown in themes and 43 subthemes were identified (Table 2) .

According to the designed assessment system, the required data for evaluation is collected from various sources, including the head of the ward, the deputy director of education of the hospital, the faculty members working in the ward, the head nurse, and the patients in the ward. The evaluation period is 6 months and is based on the rotation of the learners. Moreover, the data collection method is a combination of interview, observation, and document review methods (Figure 1 ).

One of the most important themes of the designed SCMAR is the theme of areas and indicators of evaluation. Based on the steps taken in this study, the evaluation indicators were defined in five areas, including individual performance, patients, equipment, hospital policies, and education ( Table 3 ). The checklist items had two options (i.e., yes and no), which could be scored zero and one. Accordingly, the lack of the desired performance in each indicator in <60% of the evaluated items was scored 0, F I G U R E 1 Continuous multidimensional assessment system for medical residents.

Column Indicators Areas According to previous reviews of comprehensive performance evaluation programs, they have been most influential in the axis of observation and evaluation. 7, 17 The influence of the observation axis indicates the importance of a comprehensive and constructive evaluation system. 17 In the present study, the aspects and components of the continuous evaluation system were reviewed using the ideas of experts in four basic steps to guarantee a complete evaluation. Based on the results of a study conducted in Iran using the Objective Structured Clinical Examination (OSCE) method, the feedback was positive and the students were content with this evaluation method. 18 In previous studies, students had stated that the OSCE test covered a wide range of clinical knowledge and competencies, identified learning weaknesses, and was equitable. One of the obtained categories was preparation as a process of adaptation; accordingly, by preparing and studying to succeed in various assessments, the students' learning improved regardless of the stress. 19 However, the above-mentioned evaluation system had some weaknesses; for instance, regarding the simulation of the clinical environment, it was stressful and was also held in inappropriate locations and at inappropriate times. 18, 20 These results are consistent with the evaluation system designed in the present study in terms of direct observation; nevertheless, the evaluations in this study will be performed in a clinical environment, which is one of the highlights of the SCMAR. In the comprehensive OSCE test, the learner might have a good basic knowledge and also know how to use it in different situations, but not be able to perform well when faced with patients. 19 Among the main themes identified in this study were feedback dissemination methods. Continuous assessment in all shifts is conducted by the provision of feedback on the performance of medical residents to improve their performance. In a study conducted on medical students in the Neurology Department, Zhao et al. 21 This study aimed to create a framework and consider the characteristics of effective feedback to benefit from the mechanisms of dissemination of feedback, including emails, letters, and group meetings, morning rounds through a summary of the evaluation results based on the individual, group, and educational level. It seems that the continuous assessment of residents promotes professionalism, teamwork, and communication. 25 Moreover, the provision of feedback to students can be effective in teaching medical ethics. 26 Research has shown that medical students frequently encounter moral conflicts during the course of their study, and that the greatest weakness in their communication with patients is introducing themselves. [27] [28] [29] [30] In this study, evaluation of respectful behavior towards patients and respect for their privacy and the principle of confidentiality, appropriate interaction with the staff, and introduction of themselves to the patient were among the indicators evaluated in the continuous assessment system of medical residents. This assessment system promotes professional ethics which plays an important role in communication with patients.

Due to the outbreak of the coronavirus disease-19 pandemic during the final sessions of the assessment system design, it was not possible to have face-to-face interaction and in-depth interviews with residents to obtain their statements and comments. Therefore, discussions with residents as stakeholders involved in the evaluation process were conducted virtually via e-mail and video call.

Implementation of a SCMAR in the form of a system or application that can be used on smartphones and tablets will greatly increase its effectiveness and establishment. This system should be web-based, provide the ability to define access levels, define users and create a user panel, be user-friendly, provide advice and guidance for users, perform evaluations and provide online performance feedback instantly, provide cross-sectional and process reports, provide reports for individuals, groups, and educational categories, have an archive of evaluation records, be easily updated, send reports in the form of e-mail and text message to managers, allow the managers to view activity reports based on the period, section, and subaxis in the relevant folder, allow incentives or disciplinary control measurements by choosing the name of the person, their position, and the type of action in question, provide warnings and reminders, and allow the graphic presentation of the information. 31 It is suggested that an intervention study be carried out by implementing the SCMAR to generalize the obtained results with more confidence.

Evaluation in the form of a SCMAR can help the managers, faculty members, and residents of medical departments to make sure that the tasks are performed more accurately and quickly with fewer costs. It also plays an important role in the promotion and YOUSEFI ET AL. | 7 of 9 institutionalization of professional ethics and the establishment of effective communication with the patients. It should also be noted that the above-mentioned evaluation system should be performed continuously and provide the necessary feedback to the evaluated groups and individuals. This will increase the possibility of correcting professional behaviors and activities in the evaluated individuals and also increase the effectiveness of education. This has been one of the most important goals of designing a SCMAR in this study.

All of the authors participated in designing the study. Mehdi Yousefi:

Conceptualization and methodology, project administration, writing - 

The authors declare no conflicts of interest.

The data sets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

This study was approved by the Research Ethics Committee of Tehran University of medical sciences, Tehran, Islamic Republic of IRAN (Ethical code: IR.TUMS.NIHR.REC.1400.012). Written informed consent was obtained from all the participants. All methods were carried out in accordance with relevant guidelines and regulations.

A comparison of two methods of teaching reflective ability in Year 3 medical students

Survey of clinical competency in fourth-year medical students of Guilan University of Medical Sciences in 2011: a short report

Designing a tool to assess medical students' clinical competency in pediatrics

Implementation of a Mini-Clinical Evaluation Exercise (Mini-CEX) program to assess the clinical competence of postgraduate trainees in pediatrics

Creating an ultrasound scholarly concentration program for medical students

Learning to prescribe and instruct exercise in physiotherapy education through authentic continuous assessment and rubrics

Formative feedback with in-class question bank utilization improves resident satisfaction with general surgery didactics

From the eye of the nurses: 360-degree evaluation of residents

A perspective on 360-degree evaluations

Impact of a 360-degree professionalism assessment on faculty comfort and skills in feedback delivery

Patient, faculty, and selfassessment of radiology resident performance: a 360-degree method of measuring professionalism and interpersonal/communication skills

Reliability of the Interprofessional Collaborator Assessment Rubric (ICAR) in Multi Source Feedback (MSF) with post-graduate medical residents

Evaluation of clinical education in pediatric wards of hospitals affiliated to Shahid Beheshti University of Medical Sciences according to the Ministry of Health Standards in 2015

Resident perceptions of the impact of duty hour restrictions on resident-attending interactions: an exploratory study

The nominal group technique: a useful consensus methodology in physiotherapy research

The diagnosis of IBS in primary care: con-sensus development using nominal group technique

Nominal group technique: an effective method for obtaining group consensus

UK Council for Clinical Communication in Undergraduate Medical Education. National survey of clinical communication assessment in medical education in the United Kingdom (UK)

Designing, implementation and evaluation of OSCE to assess nursing students clinical competence in Jiroft Faculty of Nursing and Midwifery

Effects of OSCE on learning, satisfaction and test anxiety of nursing students: a review study

The objective structured clinical exam (OSCE): a qualitative study exploring the healthcare student's experience

The effect and analysis of multiple assessment methods in the assessment of clinical ability of medical students

Evaluation of competencies related to personal attributes of resident doctors by 360 degree

Active learning of respiratory physiology improves performance on respiratory physiology examinations

Stakeholder management in international projects

Advanced assessment of medical students clinical performance: challenges methods and approaches

Alignment of ethics curricula in medical education: a student perspective

Interns' communication with patients during interview: the perspectives of patient and observer

Initial assessment of communication skills of intern doctors in history-taking

A novel peer-directed curriculum to enhance medical ethics training for medical students: a single-institution experience

Ethics teaching in a medical education environment: preferences for diversity of learning and assessment methods

Continuous multidimensional assessment system of medical residents in teaching hospitals