key: cord-0687900-ofb7at5d
authors: Olivares Olivares, Silvia Lizett; Hernández, Ramón Ignacio Esperón; Corolla, Miriam Lizzeth Turrubiates; Alvarez, Juan Pablo Nigenda; Sánchez-Mendiola, Melchor
title: MOOC Learning Assessment in Clinical Settings: Analysis from Quality Dimensions
date: 2021-01-09
journal: Med Sci Educ
DOI: 10.1007/s40670-020-01178-7
sha: b408e477d9cc4f03f3b78aab0724e06b8b477c28
doc_id: 687900
cord_uid: ofb7at5d

Online learning is becoming a fundamental modality of learning in medical education, and can be of great help during global crisis like the current COVID-19 pandemic. The MOOC (massive open online course) mode of e-learning is increasing its penetration worldwide, as a valid teaching approach to reach large populations. A major challenge in clinical education is the assessment of medical students and residents in clinical settings, and there is substantial evidence that the current situation requires improvement. The goal of this study was to evaluate the quality dimensions of a MOOC titled “Learning assessment in clinical settings,” developed by three Mexican universities in the Coursera platform. A mixed-method study design was used to assess the quality dimensions of the MOOC in two phases: pilot and implementation. The best-rated aspects were learning resources and pedagogical perspective, and those with opportunity for improvement were collaboration and time management. Assessment might be learned through a MOOC format, especially for participants willing to engage with educational technology and self-direction.

The COVID-19 pandemic declared by the World Health Organization (WHO) has slowed down the entire world, changing multiple economic and social sectors, including higher education. Schools and universities have been forced to transition to an online format to protect the academic community. Education is being abruptly transformed into distance and self-directed learning. In consequence, medical educators around the globe are changing their courses, practices, and other activities to provide their teaching from home. The most innovative educational institutions have already implemented distance education as part of academic programs, faculty development, and continuing medical education. They are moving forward to guarantee the programs' continuity, and those reluctant to change are now struggling to maintain educational services. Because of the crisis, the short-term future of every aspect of life is uncertain, but distance learning is the safest modality that will prevail.

Well-planned online experiences are meaningfully different from courses offered to respond to a crisis [1] ; MOOCs (massive open online courses) offer education with highquality standards to a large number of participants around the world who have internet access at little or no cost [2] . According to Daniel, Vázquez, and Gisbert [3] , these courses have the potential to offer better educational experiences within learning communities where students and teachers from multiple institutions and geographic locations participate remotely. According to these authors, MOOC can be understood as an essential approach for learning in higher education, not only in countries where they are already offered but also in less economically developed countries.

MOOC programs are becoming an alternative to reduce the cost compared to traditional training. Therefore, they may increase the perceived value for participants. MOOCs intend to expose students to new disciplines, stimulate new interests, learn ideas, and create a learning community [4] . In medical education, several MOOCs are now delivered as independent education or as a complement to classroom courses [5] . During COVID-19, MOOCs become an opportunity for distance learning about topics of interest to users.

There is some skepticism that MOOCs can provide a satisfactory learning experience, no matter what technologies are used [6] . Online learning carries a stigma of being of lower quality than face-to-face learning, despite research showing otherwise [1] . Another criticism of MOOCs is the difficulty of teaching psychomotor skills, although the possibility of integrating virtual patients in MOOCs for clinical reasoning competencies is being explored [7] . The emergent move to online teaching during COVID-19 could accelerate the decolonization of curricula, and increase the perception that this type of learning has several advantages [8] . Then, medical educators may understand through their own mandatory experience, how distance learning may be an option for teaching and learning.

Regarding faculty development, Coursera MOOCs provide completion certificates useful as evidence of microcredentials for participants. However, there is currently a shortage of educational interventions available for medical educators, targeting the global community of Spanishspeaking health professionals. The National Board of Medical Examiners (NBME) created the Latin American Grants Program for projects related to assessment in medical education (https://www.nbme.org/about-nbme/contributionsopportunities/latin-american-grants). Faculty in health sciences require practical training in educational evaluation. Teacher training should encourage a comprehensive, patientcentered education, emphasizing the assessment of clinical competence [9] . Additionally, a MOOC is an alternative for teacher development since it is delivered on a flexible platform with a personal learning environment designed for autonomous learning [10] . The current pandemic crisis is an opportunity for educators to prepare themselves on important pedagogical competences as student assessment.

The present document describes the quality dimensions results of the MOOC "Learning assessment in Clinical settings" which three universities in Mexico designed as part of a project funded by the NBME. The online format also requires a quality model for the design, production, maintenance, and evaluation of a MOOC as part of the educational strategy.

Several authors have proposed dimensions to assess the quality of a MOOC. Daniel et al. [3] consider these aspects as follows: teaching model, monetization, certification, and adaptive learning. The teaching model focuses on the student acquiring useful skills and knowledge that can be applied in real-world contexts. Monetization refers to making MOOC courses financially viable using different approaches that can reduce the costs of other online learning models while offering comparable or even higher learning quality. Certification has two aspects of MOOCs. The first is that it opens the door to revenue from course fees; secondly are the issues of how learning is assessed, authenticated, and valued by employers. Adaptive learning refers to making MOOC courses more personalized by collecting all the data for analysis and interpretation to improve the educational activities, the courses delivered, the learning experience, and the investments of exclusive educational offers. Zhu, Bonk, and Sari [11] list three categories: resources, pedagogy, and logistics. The first category refers to the affordance of MOOC platforms, support from the host institution and the platform, and the available intellectual and hardware resources. The pedagogical category focuses on the learning objectives, assessment methods, course length, course content, flexibility, and collaborative learning support. The logistics category refers to the amount of time that instructors spent designing the MOOC (planning, assessment, and teaching). DeBoer, Ho, Stump, and Breslow [12] consider the following as important variables: enrollment, participation, curriculum, and achievement. Enrollment refers to know the user commitment to particular pathways for certification, auditing, etc., in order to allow a better description and analysis of the users. Participation refers to the differentiated pathways created by the individual goals or objectives of the students, such as attendance, number of clicks, total number of hours spent in the course, homework time, and partial and final exams. Curriculum refers to the number of resources in the course, which users can choose according to individual preferences, for which there is no correct, prescribed way to proceed. Finally, achievement is defined by the individual goals of the user, rather than criteria set by instructors. Other authors suggest tips for MOOCs associated with preparation, design, implementation, and improvement (Table 1) .

Alemán, Sancho, and Gómez [14] propose a model of four dimensions to assess a MOOC quality: Pedagogical perspective, functional features, technological features, and time management. Pedagogical perspective relates to content, learning resources, pedagogical approach, and evaluation. Functional features evaluate autonomy and user control, user friendliness, and functionality. Technological features evaluate visual environment, design, versatility, navigation, interaction, and dialogues. Time management addresses the agenda and timing for each activity.

The pedagogical perspective should be the most important quality standard to be considered by the design team. Many faculty members believe that MOOC cannot replace a teacher because of the lack of interaction with the instructor. However, some studies have shown that collaboration and peer assessment promote critical thinking and an effective way to create a free, open, and highly interactive global community [15, 16] . To create an engaging environment, it is vital to understand learners' motivations to start a MOOC, their expectations, and their preferred learning process [11] . Quality Matters [17] offers a list of requirements for online learning.

The MOOC "Learning evaluation in Clinical settings" has the objective of disseminating how teaching develops in various clinical settings, from the point of view of the student and the teacher. The program starts with an overview of traditional medical education and its evolution over time with an emphasis on assessment. The participant learns to analyze what elements are important in the assessment of learning, to improve decisions about what to assess and why. Each assessment tool is described considering Patient-Centered Learning principles, validity and quality measures, and feedback practices to improve student performance.

The MOOC "Learning evaluation in Clinical settings" was organized on four modules: Participants understand the main changes through the history of medicine and medical education considering several teaching environments. They recognize that the common denominator in clinical teaching throughout history continues to be patient-centered care, regardless of changes in medical schools. The content includes the relationship between clinical cases and learning objectives.

Direct observation (360°, portfolios, MiniCEX, work-based assessment). On the first blueprint of the program, there were five modules to be distributed to each university participating in the project. Still, after the first design session, the topics were integrated and mixed in four modules. The course has the purpose of developing teachers, students, and health professionals on concepts and applications for evaluation in clinical settings based on the principles of Patient-Centered Learning [18] . The collaboration process to define the structure of the program was assessed by experts from the NBME and consensed after several face-to-face and online sessions.

The content has a variety of resources as videos, infographics, power point presentations, and reading materials. There are included interactive forums, self-assessments, and projects aimed to implement the learning into the practice. Similar to other MOOCs, because of the large number of participants, activities are tested by automated quizzes with multiple-choice answers, and assignments are reviewed by peers also active on the course [19] .

The functional features are commonly called user friendly. Navigation inside the platform should be clear and easy to understand for every user. MOOC participants may have diverse backgrounds and levels of knowledge, so the links should take participants into activities of each module in a very intuitive direction [11] . References should be presented appropriately to invite users to learn deeper the topics of their interest.

The MOOC "Learning evaluation in Clinical settings" is presented with sequential topics and activities which might be accessed on both computer and mobile devices. The videos were limited to 6 with a length range from 5 to 8 min in order to enhance the participant experience. Contents may be downloaded to be accessed without a Wi-Fi connection.

Technological features relate to the beauty of materials considering colors, fonts, graphical design, and multimedia design. According to Graham [19] , in a MOOC, the course content is delivered via video lectures. Darby [20] does not recommend posting a video providing a standard speech for a classroom; instead, the recording should capture the personality and passion of the instructor in ways different from what he might do in person, yet authentic. Quality Matters [17] recommends recording short multimedia videos with easy access to additional software and plugins. Besides the speaker, video editing provides opportunities to complement materials with other pictures, movement, data, and sound that expands the learning experience.

An important feature from the course is the interactive forums and peer-review activities where participants interact with each other to promote learning. An earlier study showed that one important success factor from students that completed the MOOCs was the multiple times posts on discussion forums [4] . According to Hoy [21] , students that participate in online discussion forums get clarification, share thoughts, and build a sense of community.

The MOOC "Learning evaluation in Clinical settings" includes some videos that were recorded with professional equipment provided and edited by an audiovisual production company. A team of digital designers enriched the materials developed by faculty members and provided programming into the platform to enhance the user experience.

Time management is crucial for medical educators. MOOCs provide learners with the ability to work at their own pace [19] . It is considered a primary advantage of a MOOC the flexibility of time and place for learners, allowing them to take the course when and where they prefer as most convenient. Zhu et al. [11] describe that MOOCs benefit participants to develop time management skills. Medical educators usually have tight schedules, and sometimes they even lack the interest to teach [22] .

The purpose of the study was to evaluate the quality dimensions of the MOOC "Learning assessment in Clinical settings" from the participants' perspectives.

A mixed, descriptive, and transactional method was applied considering two phases: Pilot (qualitative) and massive implementation (quantitative). During the pilot, a total of 57 faculty members from three universities were invited to register as participants on the course. As a qualitative phase, three focus groups were performed with 21 faculty members out of the 33 that completed the pilot course: 10 from Tecnologico de Monterrey, 8 from UNAM, and 3 from Westhill University. Faculty feedback in a health sciences' MOOC was also applied by Sitzman, Jensen, and Chan [16] . Those faculty members provided feedback related to how conclusions resonated with their understandings and experiences in a MOOC. The focus groups proposed by Savin-Baden and Howell Major [23] aim to provide information to the researcher about the opinions of a group on a topic, to document ideas, and to highlight inconsistencies of opinion among members of a community. The process to guide each focus group session was with questions related to the quality dimensions proposed by Aleman et al. [14] . Each topic was reviewed, discussed, and analyzed by participants. Researchers stratified the excerpts by emergent coding [24] . Participants signed consent forms to participate in the study and approved the recording of the sessions. Results were applied to improve design before go-live implementation.

For the massive implementation, a quantitative approach was selected. There was designed a survey based on Alemán et al.'s [14] survey to assess quality dimensions considering the emergent coding from the pilot phase. The questionnaire has 12 items and Cronbach' alpha of 0.97, to assess quality during implementation phase ( Table 2 ).

In October, the course accumulated 4712 total learners who started the program, 2701 active learners with completed checkpoints, and 325 with certificate. Regarding participants' profile, most of them have 25-34 years old (43.4%) followed by 18-24 years old range (26.2%). They are employed full time (39.8%) followed by unemployed (16.5%). For the present study, the survey only considered participants who have already completed the course to assess the MOOC's quality effectively. A total of 190/322 participants completed the study, which represents a 59% response rate.

Qualitative results from pilot are presented in Table 3 . Some quote examples are included by code. The codes are The technical and aesthetic quality of the multimedia materials is adequate The graphic design of the pages is attractive, and the resources are dynamic Functional features Navigation on the pages is structured, simple, and ergonomic The instructions for the exercises are clear and easy to understand organized, starting with the ones with best perceived value from participants, and lastly with codes related to topics with more opportunities for improvement. The resultant quality dimensions from Table 3 were classified into two groups, Academic content and Digital platform ( Fig. 1) . Academic content involves the process of learning competencies, both generic and disciplinary. Generic competences promoted and enhanced with MOOCs are creativity, critical thinking, self-direction, collaboration, and inquiry through learning resources. Disciplinary competencies from The program has enough support for practice with your students. I did see it as useful to have tips for managing my group better.

At some point, Patient Centered Learning seems repetitive for me.

I enjoyed Patient Centered Learning content to introduce it in the cases.

It seems to me that promotes self-management, which is also the value of the online course.

There is feedback from the self-assessments, but sometimes it did not give you the feedback as you had answered Activities develop critical thinking. There may be other activities to improve it.

There are challenging activities that allowed us to review what we previously knew and identify what was needed.

Some sections of the activities do not invite reflection. A part of the course does. Collaboration

There are activities in which a colleague will feed you back, which allows analyzing the colleague's point of view.

I like the platform, but face-to-face is also essential due to the interaction with my colleagues. I do not feel comfortable with people I do not know.

Discussion in forums depends on others' commitment. We contribute to the debate, but we do not go back to see what they said.

The forums did not promote discussion. I Sometimes the participation is so extensive that it is difficult to read what they were writing. Learning resources The additional links are okay. They are not overloaded and arouse curiosity.

There are additional links to deepen the topics.

There are summaries, tables, maps, and other supports. It was well thought out what resources to use.

It motivated me to investigate. The readings are interesting Time management

The schedule helps to manage the time. It includes the hours for each topic.

The time of the units is enough, and you just have to manage the times well.

We did not have time to go deeper, and we went more to the multiple-choice. There was no time. I wanted to land in my quick work.

I consider online courses faster than face-to-face.

The time indicated limited the work that must be sent for feedback. The times were not enough for me. Digital aesthetics Very graphic, handy charts Visual quality is adequate. It does not hurt your eyes. The colors are beautiful and warm.

The page design is attractive, and the resources are very dynamic.

I found the presentation of the images very very good. I like the visual, and I thought that how it was developed was good. Functional features It was easy to access the content, it is clear. It is easy for me to use the phone, you can download the reading without the internet.

Activity instructions are clear.

The infographic summarizes everything that was seen on the subject. Activity instructions are clear.

Browsing in the course is structured and fast.

There was difficulty getting to some places where you wanted to work. the MOOC "Learning assessment in clinical settings" are medical teaching and assessment. Participants valued content, exercises, and resources to assess Patient-Centered Learning with cases. However, there were contrasts in the answers. While some participants considered some material repetitive, others appreciated the inclusion of this module on the course. They also referred to how they have been introducing new practices in their clinical settings not considered before the training.

Digital platform refers to media production and preparation to enhance the learning experience, with an online welldesigned and friendly environment, customized to the participant's schedule and context. Quality Matters [17] recommends clear instructions and expectations, a sense of community and interaction, and a variety of resources for emergency remote learning. Similarly, Darby [20] offers tips for faculty to become a better online teacher, including clear expectations, content organization, and visual appeal.

Certainly, Academic content is the priority in online courses, but a friendly Digital platform prevents users' frustration. The current situation with COVID-19 has forced emergency remote learning, which lacks adequate planning and multimedia production with high-quality standards. However, Academic content should be addressed to guarantee the continuity of education. This crisis also represents an opportunity for faculty development. Figure 2 shows the trends by month of the number of participants since the MOOC go-live in January 2020. It is evident that the lockdown in March caused an increasing interest in the program.

Quality dimensions results for the MOOC "Learning assessment in clinical settings" are presented in Fig. 3 . Scores are highly consistent with pilot perceptions of participants. The best-rated aspects were learning resources and 

The use of digital platforms and assessment methods for education is exponentially growing since the COVID-19 pandemic. The future of medical education is uncertain and volatile, but MOOCs are becoming an excellent opportunity to educate faculty members and students with no instructor dependence. Right now, plenty of medical educators are already attending COVID-19 patients or they are waiting to be called to the clinical trenches. It is highly probable that some medical teachers will have little availability for their students.

Learning assessment in clinical settings in a MOOC format could be challenging for some medical educators, especially older ones. Participants had the opportunity to critically reflect on their prior knowledge and regarding evaluation to selfimprove their practices considering Patient-Centered Learning principles. This program involves educational technology both in the content and the delivery. Several tools described in the course refer to the use of technology to assess students during clinical practices, which are also taught in the MOOC through the digital platform.

It is evident that students' interactions with technology can change their motivation to advance in the course. Since MOOCs are open to diverse profiles, a friendly environment is required for every age and level of digital skills. This profile diversity in participants should be promoted as an opportunity to create a global interactive community of learners, developing self-guided individuals. Today more than ever it is imperative to collaborate to improve health care and education, to become better professionals, and to survive the crisis.

Funding Financial support was provided by the National Board of Medical Examiners Latin American Grants Program.

Data Availability The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

Conflict of Interest The authors declare that they have no conflict of interest.

Code Availability Not applicable. 

The difference between emergency remote teaching and online learning

Perception and use of massive open online courses among medical students in a developing country: multicentre crosssectional study

The future of MOOCs: adaptive learning or business model

Coursera's introductory human physiology course: factors that characterize successful completion of a MOOC

Instructional design quality in medical massive open online courses for integration into campus education

Online learning: how to make a MOOC

Beyond xMOOCs in healthcare education: study of the feasibility in integrating virtual patient systems and MOOC platforms

Covid-19 is our best chance to change universities for good. Support The Guardian

Aprendizaje centrado en el paciente: Cuatro perspectivas para un abordaje integral. México: Editorial Medica Panamericana

Figura de los facilitadores en los Cursos Online Masivos y Abiertos (COMA/MOOC): nuevo rol profesional para los entornos educativos en abierto. Rev Iberoam de Educ a Distancia

Instructor experiences designing MOOCs in higher education: pedagogical, resource, and logistical considerations and challenges

Course": reconceptualizing educational variables for massive open online courses

Twelve tips for developing and delivering a massive open online course in medical education

Hacia un modelo de calidad para cursos en-línea, masivos y abiertos (MOOC): Caso de un MOOC para el desarrollo profesional docente (Disertación Doctoral)

MOOC pedagogy: gleaning good practice from existing MOOCs

Creating a global community of learners in nursing and beyond: caring science, mindful practice MOOC

Lista de requerimientos para la Docencia Remota de Emergencia

Education in the digital age: MOOCs, TED Talks, and other nontraditional educational offerings

How to be a better online teacher. The Chronicle of Higher Education

MOOCs 101: an introduction to massive open online courses

Clinical competence assessment: development of a mobile app to enhance patient centerderness

The essential guide to theory and practice. Madrid: Routledge Inte-ractive

The coding manual for qualitative researchers

Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations