key: cord-0802893-vw576fhf
authors: Grech, Joseph; Grech, Jessica
title: Nursing students’ evaluation of a gamified public health educational webinar: A comparative pilot study
date: 2021-03-06
journal: Nurs Open
DOI: 10.1002/nop2.826
sha: 1f1d8a8a7b19d0ec7bd53560f2e6013a86aee646
doc_id: 802893
cord_uid: vw576fhf

AIM: To compare undergraduate nursing students’ evaluations of a gamified educational webinar to a non‐gamified version. Both webinars covered the topic “Determinants of health,” part of public health education. DESIGN: A post‐test evaluation design with a comparison group, and an additional qualitative component. METHODS: First‐year students reading for a Bachelor of Science in Nursing at the Malta College of Arts, Science and Technology were randomly allocated into two classes of 26 students each, and to the gamified and non‐gamified webinars. The revised “Students Evaluations of Educational Quality” questionnaire was sent to all participants. Participants were also asked to provide comments on their learning experience. RESULTS: A total of 40 participants who attended the gamified and non‐gamified webinars completed the questionnaire. Educational quality was perceived as “good” to “very good,” in both groups. Most participants in the gamified webinar group remarked that gamification helped to increase their engagement and interaction.

Following the declaration of the COVID-19 pandemic, most nurse educators had to resort to online teaching and learning. Online learning or e-learning provides online access to learning, thus eliminating distance or timescale limitations that are inherent with traditional class-based teaching (Regmi & Jones, 2020) . Nonetheless, poor levels of student interaction and engagement have been identified within the nursing literature (Patterson et al., 2012) .

Gamification is a recent interactive and innovative teaching modality that can engage and attract nursing students in learning (Brull & Finlayson, 2016) . Gamification, which is the use of gamebased techniques, has proven to be effective in the engagement of students in education, particularly in subjects typically perceived as boring (Kapp, 2012) . While Gentry et al. (2019) found that the use of serious gaming/gamification in healthcare students' education could result in increased student satisfaction, knowledge and skills when compared to traditional education, the authors did not differentiate between serious gaming and gamification. Although these tend to be used simultaneously, these are quite different in concept.

According to Fatta et al. (2019) in serious games, fully fledged games are developed to deliver curriculum or skills to enhance the learning experience and increase understanding. In developing serious games, the notion is to offer a richer concept of playing, such as the use of simulation in learning clinical skills. Conversely, in gamification, the educator does not carry out a fully fledged game and only utilizes gaming concepts/elements (e.g. awarding points for questions correctly answered) within non-gaming elements, that is his/ her teaching practices, to increase student involvement and motivation to learn (Fatta et al., 2019) .

Public health education, covered in undergraduate nursing education, often proves to be a challenge for educators, especially for supporting and retaining engagement for learning (Marin-Kelso, 2013) . The use of online teaching modalities that enable realtime interaction, such as webinars, with the addition of gamification, may be particularly useful in the delivery of such a module.

Underpinning gamification is the concept of motivation (Kapp, 2012) .

Motivation can be intrinsic, because the student may undertake the gamified learning process, for example participate in a quiz out of the enjoyment or the learning it provides, or the sense of achievement it brings; or it can be extrinsic, where the quiz is taken to obtain some sort of reward, praise or admiration from others (Kapp, 2012) .

According to the self-determination theory, which has been widely used to differentiate between successful and unsuccessful gamified processes (Rutledge et al., 2018) , the core elements that need to be considered in gamification are autonomy, competence and relatedness (Kapp, 2012) . For a game to be motivating enough for its users, players must feel that they are in control of their actions, competent to carry out the tasks, and they need to feel related to the other players who are also playing (Kapp, 2012) . As opposed to the selfdetermination theory which is focused on intrinsic motivation, operant conditioning looks at extrinsic motivation and suggests that students' motivation is more likely to increase every time they are rewarded, such as through the provision of points (Kapp, 2012) . In fact, in their systematic review, Subhash and Cudney (2018) noted that the assignment of points and use of leaderboards were important game elements for successful gamification. To motivate player activity and action, the use of time-constrained activities is also suggested (Kapp, 2012) . The use of timed questions helps to activate player action and encourages students to work under pressure (Kapp, 2012) .

Research on undergraduate healthcare students' evaluations of gamified education is limited (Gentry et al., 2019) . Nonetheless, students who have experienced gamification highlighted a positive educational experience (Aktekin et al., 2018; Kinder & Kurz, 2018; Pettit et al., 2015) , remarking increased learning (Kinder & Kurz, 2018; Roche et al., 2018) and other good qualities such as increased engagement and interactions (Aktekin et al., 2018; Pettit et al., 2015) .

Students attributed the gamified component to the creation of a positive competitive environment, an increase in motivation and encouragement to follow the course, and a supporting tool for retaining knowledge (Aktekin et al., 2018) . No studies on students' evaluations which utilized gamification in challenging subjects such as public health were identified. Moreover, no studies which reported the delivery of nursing education using webinars with the addition of gamification were found. The recent closure of most educational institutions because of the COVID-19 pandemic, where academics have been constrained to deliver exclusive online teaching and learning methods, emphasizes the need to address the above-mentioned research lacunae.

The purpose of this pilot study was to identify undergraduate nursing students' evaluations of a gamified educational webinar as compared to the evaluations of a non-gamified version. Both webinars addressed the topic "Determinants of health," an important concept within public health education. Information from this pilot study, collected from users' perspectives, will inform the researchers on the use of gamification in online public health education for fullscale evaluation. 

The study's participants were first-year Bachelor of Science in Nursing Studies students at the Malta College of Arts, Science and Technology (MCAST), who had just started their public health module at the time of the study (March 2020). Any other nursing students were thus ineligible to participate. The first-year students were already randomly allocated into two classes of 26 students each, on enrolment of their studies. To participate in this pilot study, the first-year students had to attend the "Determinants of Health"

webinar.

The psychometrically sound questionnaire, the "Students Evaluations of Educational Quality" (SEEQ), which looks at teaching from a multi-dimensional aspect (Marsh & Hocevar, 1991; Marsh & Roche, 1992) , was utilized. Although developed more than 35 years ago by Marsh (1982) , this is one of the most widely used and universally acceptable tools for students' evaluation of teaching methods (Grammatikopoulos et al., 2015) . Constant high levels of validation and reliability scores have been highlighted in the literature in various countries (Balam & Shannon, 2010; Coffey & Gibbs, 2001; Marsh, 1982; Marsh & Hocevar, 1991; Marsh & Roche, 1992) .

The SEEQ was also used amongst nursing students (Balam & Shannon, 2010) .

This tool has nine distinct components (consisting of 31 items) which assess: "learning"; "enthusiasm"; "group interaction"; "individual rapport"; "breadth"; "examinations"; "assignments"; and "overall", by using a 5-point continuous Likert scale, "very poor" (1), "poor,"

"moderate," "good" and "very good" (5), for each item (Marsh, 1982) .

Given that students were asked to evaluate their learning experience on a sole topic of the public health module, the latter three compo- In the gamified webinar, the same presentation was uploaded on Mentimeter ® , a real-time application for creating presentations that enables interactivity through various means such as quizzes (Mentimeter, 2020) , and delivered on Microsoft Teams ® .

A six-question quiz, having a multiple-choice format (two or three responses) covering basic and simple concepts about the determinants of health, was spread out within lecture content and delivery.

To ensure student autonomy (as suggested in the self-determination theory), students were invited to join the game as a fun way of learning, and not as a means of assessment, by going on menti.com and entering a unique code. Questions, which were allocated points as a means of extrinsic motivation (operant conditioning), had a 20-s time limit. The students who gave the right answer fastest got higher points. After each question, after discussing the students' responses, a leaderboard slide was displayed showing the students' ranking. At the end of the lecture, another leaderboard slide displaying the top scores was shown. The same questions forming the quiz were also presented in the non-gamified webinar to generate discussion.

Participants were unaware (blind) that webinars differed between groups. The online questionnaire was sent to the participants at the end of each webinar, accepting responses until that same evening.

All quantitative analyses were done using IBM SPSS Statistics Version 26. Descriptive statistics (frequencies "n," percentages, mean and standard deviation "SD") were used to characterize the sample and the revised SEEQ variables. Given the small sample, nonparametric tests were used to identify any statistically significant differences between the gamified and non-gamified webinars samples' characteristics and their mean revised SEEQ scores. Fisher's exact test was used to identify any differences for nationality, gender and highest qualification, while the Mann-Whitney U test was carried out to identify differences in the mean participants' age. The

Mann-Whitney U test was also utilized to identify any differences between the groups' mean scores of each of the revised SEEQ's items and their respective components. 

Before carrying out the study, a research proposal listing all relevant ethical considerations was sent to the Institution Review Board at MCAST for approval. No ethical issues were envisaged, and the study was approved.

An information letter was attached to the questionnaire. To support participants' self-determination, students were informed that participation and their answers, or non-participation, did not affect their grades whatsoever. Participation in the questionnaire was voluntary and actual participation implied informed consent. No names, email addresses or personal identifiers were collected thus ensuring anonymity.

Twenty-four students participated in the gamified webinar, while 25 students attended the non-gamified session. The report generated by Mentimeter ® showed that all students who attended the gamified webinar (24) Table 1 . The statistical tests carried out showed that participants did not differ significantly by their characteristics. All participants in the gamified webinar group confirmed that they had participated in the quiz.

Tables 2 and 3 display the participants' ratings for each indicator of the six distinct components in the revised SEEQ (Coffey & Gibbs, 2001) . When considering quality in educational methods as "good" or "very good," the majority in both groups perceived their teaching and learning experience as so. This was noted particularly for the items falling under "Individual rapport" and "Group interaction" (mean values listed in Table 4 ), where almost all or all students in both groups gave high ratings. The lowest "good/very good" scores were identified amongst participants in the non-gamified webinar group for "Instructor enhanced presentations with the use of humour" (66.7%) and for "Instructor's style of presentation held your interest during class" (71.4%), and amongst students in the gamified webinar group for the item, "Instructor gave the lecture that facilitated taking notes" (73.7%). As shown in Table 4 , the mean scores of the six distinct components were not significantly different between groups. Furthermore, no statistically significant differences were identified for each item.

Fifteen students in the gamified webinar group and 12 students in the non-gamified webinar group provided comments on their learning experience. Following initial analysis, comments were found to be related to positive feedback or suggestions for improvement.

Most participants (n = 10) in the gamified webinar group highlighted the benefits in having participated in the quiz, in terms of increased engagement and interaction; "Quiz game … made us more involved" (participant in the gamified webinar group [GG] no. 2), "The lecture was very engaging and fun … enabled students to discuss among ourselves." (GG 9), "The quiz was nice as it kept us alert to the lecture." (GG 10), and "quiz … I feel it encouraged us more to stay focused and concentrate" (GG 14). Four participants in the non-gamified webinar group and one participant in the gamified webinar group (who did not attribute his/her comment to gamification;

"I think it was a good lecture, the message of the lesson was sent and I understood everything that the lecturer said." GG 6) remarked that it was a good lecture; "he is quite good in delivering the lecture"

(participant in the non-gamified webinar group [NGG] no. 5), and "everything was fine and up to date" (NGG 7).

Various suggestions were provided for improving educational quality (identified categories and participants' excerpts listed in Table 5 ). Five students, four of whom participated in the nongamified webinar, suggested that the lecturer should interact more on an individual basis. While two students who participated in the gamified webinar suggested that more questions could have been included in the quiz, four participants suggested that time to respond to quiz questions should have been longer. The use of humour was encouraged by two participants in the non-gamified webinar group and one in the gamified webinar group.

Overall, students in both groups considered the quality of their educational experience as "good" to "very good." This was also remarked by some students when expressing their views on attending the gamified and non-gamified webinars. Such findings relate to the literature, where the majority of nursing students (88%, 95% CI:

79%-97%) were satisfied or very satisfied with the use of webinars for learning (Williamson et al., 2009 ).

In our study, almost all students in both groups reported "good/very good" "Individual rapport" and "Group interaction."

Nonetheless, five students, four of whom were in the non-gamified webinar group, remarked in their comments that the lecturer should interact more on an individual basis. This highlights the importance of real-time instructor interaction in online nursing education (Seckman, 2018; Wolf, 2018) , which students might miss within online education. Given that gamification was found to increase student interaction, this was perceived less by students in the gamified webinar group.

More students in the gamified webinar group than in the nongamified webinar group rated the item, "Instructor's style of presentation held your interest during class" as "good/very good,"

suggesting that the use of gamification was a useful educational tool to gain students' interest. As was identified in the litera- While three students from both groups pointed out the need of "more humour" when providing comments on their learning experience, less students in the non-gamified webinar group found that the presentation was enhanced with the use of humour. It is likely that since students in the gamified webinar group found that the style of the presentation held their attention, they might have perceived less the need (or lack) of humour.

Participants in the gamified webinar group were less likely to rate the item, "Instructor gave the lecture that facilitated taking notes" as "good/very good." In the gamified webinar, which also lasted two hours, the lecturer had to deliver topic content and allocate time for the quiz and a discussion of the students' responses. This might have resulted in a faster-paced lecture which created some difficulties for students who were taking notes. 

To our knowledge, this is the first study that compared students' evaluation of a gamified webinar to a non-gamified one amongst undergraduate nursing students following a public health module.

Student evaluation was carried out after the delivery of one nongamified/gamified webinar thus ensuring that results and suggestions for improvement were noted before embarking on a full-scale evaluation. Moreover, it is likely that a longer intervention, that is comparing gamified and non-gamified webinars across the whole module, would have then been subject to bias. This is because over time, students in both groups, being from the same institution, were more likely to disclose their learning experience, becoming aware of the noticeable difference in educational delivery. Consequentially, participants in the non-gamified group would have been more likely to give a lower rating of the educational quality of the sessions when compared to those in the gamified group, since they would be aware that they are missing the additional gaming experience. A full-scale evaluation, by recruiting classes from different institutions, and evaluating the use of gamification in webinars throughout the public health module, is thus recommended. This would also make up for the small sample limitation of this study.

It is also noteworthy to mention that this study did not carry out pre-tests as a means to ensure parity between groups. 

The authors declare no conflict of interest.

The first author was the leader in this research. He participated in all phases of the study. He also delivered the webinars and collected the data on his own and wrote the initial draft. The second author participated in the other phases of the study, including the conception and design of the study, the review of the literature, the interpretation of results and the drafting of the article. Both authors have approved the final paper submitted.

The raw data that support the results of this study are available from the corresponding author upon a reasonable request.

Joseph Grech https://orcid.org/0000-0002-2976-0201

Joseph Grech @JosephGrech19 Teaching the social determinants of health in undergraduate medical TA B L E 5 Main categories identified from the students' suggestions for improving educational quality

Interact with individual students "Be more interactive with the students as individuals" GG13 (n = 5) GG13, NGG6, NGG10, NGG11, NGG12 "Include the students more" NGG6 "I think by asking more individually the students, like we do in class" NGG10 Gamification-longer time limit "more time to read the questions during quiz" GG3 (n = 4) GG1, GG3, GG4, GG11 "quiz … I think that there was not enough time allocated to read and answer the questions" GG11

More humour "Be more humorous" GG13 (n = 3) GG13, NGG3, NGG8 "more humor" NGG3 Slower teaching pace "More time could be given while explaining" GG7 (n = 2) GG7, NGG3

Gamification-more questions "The lecturer could have added more questions in the quiz" GG7 (n = 2) GG7, GG12

Less busy slides "more visual interpretations … especially when there was a lot of information on a slide" NG3 (n = 2) GG8, NGG3

Use of videos "Maybe involving videos" NGG7 (n = 2) NGG4, NGG7 a Participants from the gamified webinar group are denoted as "GG," while those from the non-gamified webinar group are denoted as "NGG."

A scoping review

Game-based learning and gamification: Searching for definitions

Serious gaming and gamification education in health professions: Systematic review

Assessing the Students' Evaluations of Educational Quality (SEEQ) questionnaire in Greek higher education. Higher Education

The gamification of learning and instruction gamebased methods and strategies for training and education

Gaming strategies in nursing education

Communicating, motivating and teaching the significance of public health

SEEQ: A reliable, valid, and useful instrument for collecting students' evaluations of university teaching

The multidimensionality of students' evaluations of teaching effectiveness: The generality of factor structures across academic discipline, instructor level, and course level

The use of student evaluations of university teaching in different settings: The applicability paradigm

Student outcomes of distance learning in nursing education: An integrative review. CIN -Computers Informatics Nursing

Student perceptions of gamified audience response system interactions in large group lectures and via lecture capture technology

A systematic review of the factors -enablers and barriers -affecting e-learning in health sciences education

Educational analytics: A new frontier for gamification? CIN: Computers, Informatics, Nursing

Gamification in action: Theoretical and practical considerations for medical educators

Impact of interactive video communication versus text-based feedback on teaching, social, and cognitive presence in online learning communities

Computers in human behavior gamified learning in higher education: A systematic review of the literature

How reliable are Students' Evaluations of Teaching (SETs)? A study to test student's reproducibility and repeatability

Undergraduate nurses' and midwives' participation and satisfaction with live interactive webcasts

Nursing education: An integrative review