key: cord-1030482-dkxdd330
authors: Kaur, Anjuli; Singh, Sid; Chandan, Joht S; Robbins, Tim; Patel, Vinod
title: Qualitative exploration of digital chatbot use in medical education: A pilot study
date: 2021-08-31
journal: Digit Health
DOI: 10.1177/20552076211038151
sha: 8b9ace44c126f6a52528d9589b6b30cfadfb2a25
doc_id: 1030482
cord_uid: dkxdd330

PURPOSE: During the coronavirus disease 2019 pandemic, face-to-face teaching has been severely disrupted and limited for medical students internationally. This study explores the views of medical students and academic medical staff regarding the suitability and limitations, of a bespoke chatbot tool to support medical education. METHODS: Five focus groups, with a total of 16 participants, were recruited using a convenience sample. The participants included medical students across all year groups and academic staff. The pre-determined focus group topic guide explored how chatbots can augment existing teaching practices. A thematic analysis was conducted using the transcripts to determine key themes. RESULTS: Thematic analysis identified five main themes: (1) chatbot use as a clinical simulation tool; (2) chatbot use as a revision tool; (3) differential usefulness by medical school year group; (4) standardisation of education and assessment; (5) challenges of use and implementation. CONCLUSIONS: Both staff and students have clear benefits from using chatbots in medical education. However, they documented possible limitations to their use. The creation of chatbots to support the medical curriculum should be further explored and urgently evaluated to assess their impact on medical students training both during and after the global pandemic.

Chatbots are an application of the emerging field of artificial intelligence. It underpins the knowledge of whether machines can use evidenced-based algorithms to enhance efficiency, communication and dialogue. Chatbots have the potential to simulate a natural dialogue either by audio or text and have the capacity, to perform complex tasks, by interacting with human users, as demonstrated by notable examples such as the Amazon Alexa. 1 A simple flowchart, as illustrated in Figure 1 , demonstrates a flow of conversation between the user (the medical student) and the chatbot. The user input are keywords from a command by a user which would be recognised by the chatbot, based on which the chatbot will decide to answer the question, allowing a flow of conversation to occur.

The focus of the study was to qualitatively explore the views of medical students and medical educators in integrating the use of chatbots into the medical curriculum, using focus groups. The context of the study was based on making current simulations which were being conducted across one National Health Service (NHS) trust, more efficient, with the use of chatbots. This would mean that the number of health care professionals available would not limit the number of medical students being allowed to complete the simulation as a chatbot could serve numerous medical students at one time. This would mean that the simulations would no longer be limited to just one NHS site.

The chatbot market in healthcare is expected to grow over £410 million from 2019 to 2025 2 particularly in primary care. Currently, chatbots in healthcare, developed by Babylon Health, 3 have predominantly been used to facilitate patient consultations. Despite the advancements in the use of chatbots for patient-facing services, such as those support by Babylon Health, 3 there are no chatbots that are actively used to support medical students in education. As face-to-face teaching is being limited for clinical medical students by coronavirus disease 2019 (COVID-19), a chatbot could augment teaching that is being delivered remotely by medical schools.

There is an increasing body of research showing how technology-mediated machine learning can increase quality of learning processes, to achieve better outcomes for medical students. Studies have shown a positive effect of using mobile learning devices, such as smartphones and tablets, in clinical environments, to support the learning of medical students. 4 There is substantial evidence that a range of skills can be improved using virtual patient simulations. 5 A further study illustrated that a simple chatbot can be created for a tutoring system, 6 and that interviewing patients can be practiced using a chatbot. 7 To conclude, it was shown that students who used these chatbots were attaining higher grades. 8 However, a recent paper suggested that teachers need to be willing to promote technology amongst students to influence their use. 9 Despite research suggesting how chatbots can improve education the implementation of these chatbots are rarely researched. There are currently limitations in our understanding of how to formally incorporate chatbots into medical education curricula due to a lack of understanding of their application and lack of evidence for consideration by regulators. We therefore need a study to examine the views of students and staff as to the use of chatbots in medical education. To our knowledge, this is the first paper that directly assesses the use of chatbots in British medical education and the first international paper with a formal focus on group methodology.

We performed a pilot implementation and assessment of Chatbot technology at Warwick Medical School. Warwick Medical School is a United Kingdom Medical School located in Coventry, with 1900 medical students. At Warwick Medical School, students participate in casebased learning (CBL) exercises and 'Clinically Observed Medical Education Training (COMET)' 10 simulation sessions at one NHS trust base. Feedback for these sessions are very positive, however significant human resources, such as experienced postgraduate tutors are required, and the cost of mannequins to help aid simulation. Each station requires one tutor and on average there are four stations per COMET tutorial. Topics include emergency care, clinical consultations and practical procedures. We propose that chatbot technology could potentially be integrated to facilitate such sessions both at Warwick Medical School and internationally. The aim of this study is to assess the suitability and limitations of creating a chatbot, which can be implemented into a medical curriculum. The use of chatbot technology is particularly useful in resourcelimited environments and in response to the COVID-19 pandemic where there have been, and remain, significant disruptions to face-face medical education training.

Assessing the perspectives of medical students and medical educators is vital to forming a long-term plan for the future optimisation of chatbot design and widespread implementation into medical education curricula. Their input is a strong predictor of how well students might engage and benefit. We adopted a pedagogical theory approach to understand teaching practices across medical education. This required understanding the students' perspective along with the medical educators' perspective. Collaboratively this was expected to provide a focus on where chatbots could be integrated into the medical education curriculum

This was a study conducted over a 2-month period at Warwick Medical School. Medical students from years 1 to 4 and academic medical staff were recruited by AK. Medical students and medical educators were invited to take part in the study via course-wide emails.

Face-to-face focus groups, averaging three participants per group, took place at Warwick Medical School. The focus groups were semi-structured in nature supported by a predetermined topic guide, as shown in Figure 2 . The topic guide explored topics such as the uses of chatbots in teaching practices common to the medical school, such as CBL, and their perception of using chatbots to learn from, instead of traditional methods such as lectures and textbooks. The topic guide questions were designed following exploration of the literature and remained iterative in nature. This allowed for the questions to change as the focus groups went on, in case new important topics emerged. The focus groups lasted between 24 and 36 min and were recorded on an encrypted recording device by AK.

Each transcript was transcribed verbatim. The analysis of the data was carried out alongside conducting focus groups by AK. This ensured that themes that were raised in prior focus groups, were addressed and open for discussion in the forthcoming focus groups. The transcripts from the focus groups formed the raw data for the study. Data was analysed using NVivo 11.0 software; a thematic framework approach 11 was used for the thematic analysis in five steps:

1. Familiarisation of the data collected by transcribing and re-reading the transcripts. 2. Identifying common themes amongst the focus groups and coding appropriately. 3. Indexing particular sections of the discussions which corresponded to a theme or an idea. 4. Charting the data in illustrative forms respective of the themes identified. 5. Mapping and analysis of key characteristics laid out in the charts created to obtain a conclusion.

Following recruitment, there were 13 participating medical students and three staff members. In total 16 participants took part in the focus groups as outlined in Table 1 . There appeared to be an overall student consensus during each focus group, and no reports were made of feeling uncomfortable whilst expressing views. Five main themes were identified which were further categorised into subthemes as shown in Table 2 .

Participants expressed that potential use for a chatbot would be to use it as a patient simulator that a medical student could interact with.

It's helping you practice before going into a real patient setting.

(Year 1, participant b) However, some participants raised concerns about how a chatbot could limit the capacity at which the chatbot could be used to learn or practice from.

Depends how descriptive you have to be with your own responses, to get the right trigger words to get the responses.

(Year 3, participant c)

Participants expressed that one area where chatbots could be used in are where students have to practise history taking.

On the clinical skills side of things, it lets you hone history taking skills.

(Year 1, participant c) Some participants suggested that audio chatbots could be used to practice for telephone consultations.

Would be useful for doing phone consultations, because you lose that human contact straight away and having something to use to practice on before trialling on a patient would be useful.

(Year 3, participant c) However, some participants expressed the limitations of taking a history using a chatbot which would therefore not be beneficial for their learning.

You will lose the non-verbal aspect.

(Year 3, participant a) I would prefer a real patient than interacting with a chatbot. They will never have the subtilties and the nuances that a patient can present with and that experience.

(Academic faculty, participant 3)

Another limitation which participants outlined were that if facilitators involved with their learning were replaced by chatbots they would be losing out on personal experience that health care professional brings during teaching sessions.

With real life facilitators the nurses are actually bringing their experience […] The tips and advice that they would be able to provide would be really useful.

Participants were also asked about COMET sessions whereby a chatbot could potentially replace facilitators and felt that the chatbot would not be taken seriously and would not be a beneficial exercise.

If a student asks a question which isn't phrased in a way that the chatbot couldn't understand […] You would then technically be speaking to an inanimate object […] wouldn't exactly be taken seriously or even be helpful.

( Participants were worried about how they would later interact with patients if they were being taught by chatbots for a significant part of their medical education. However, participants mentioned how using a chatbot would create a safe space whereby students can practice using chatbots prior to seeing patients in real life, early in their training.

It does mean we can practice in a safe space without it being stressful […] I don't think that it matters as much, because you're not taking someone's time and it would be less stressful.

( Good for pharmacology mechanism of action and things like that could easily be delivered by a chatbot. It could be a way to make that livelier and more interactive.

(Academic faculty, participant 3)

Many participants felt that learning would be more efficient if they could locate resources using a chatbot.

It would be good to use as an index to find out more about a presentation which could direct you to internal lectures and textbooks.

( 

Participants expressed that first-year students would be more suited to using a chatbot in their early training instead of final year students.

If you want to practice taking a history, especially in the earlier years, that would be very useful. 

Participants suggested that using a chatbot part of their training could ensure some level of standardisation amongst teaching opportunities.

With the chatbot you get a relative standardised test setting.

There is a whole subjectivity in marking, it might actually standardise marking and make it more fairer.

(Academic faculty, participant 1)

The majority of participants expressed numerous challenges of the use and implementation of chatbots. There was a negative perception of using chatbots in medical education in contrast to one which was positive.

One participant had a positive perception of using chatbots in medical education. 

The use of chatbots is emerging in a wide variety of industrial sectors, including more recently in the healthcare setting, however their use in medical education is largely untested. This study advances knowledge of areas in which a chatbot can be integrated into a medical curriculum. In this pilot study, we identified the following five themes: patient simulation, revision tools, suitable users, standardisation of testing and the perception of chatbots. These themes elicited strengths and limitations of the potential adaptations and uses of chatbots.

A major theme identified in the study was the use of chatbots for patient simulation. Many participants felt that a chatbot could be suitably used to practice history taking where the chatbot acts as the patient. This is consistent with findings of a metanalysis 5 which provided evidence that in contrast to traditional education, clinical reasoning and procedural skills are improved whilst using virtual patient simulations. Medical students who are currently based at an NHS trust, often take part in hospital patient simulations. The participants felt that replacing the facilitators with a chatbot would not be an ideal situation as they learn from the facilitators and from the experiences that they share. Further from this, many participants expressed concern about their professional and empathic development when dealing with patients. This was consistent with the findings of another research group 12 whereby physicians felt that chatbots would not be able to comprehend the emotional aspect when dealing with patients and thus would not provide a realistic experience for the student. Participants further felt that practicing patient consultation and simulations would hinder their ability to interact with patients in real life because consultations are complicated, which is a thought echoed by previous studies. 13 One research group 14 has suggested that there are challenges when interacting with chatbots due to cultural and language differences. However, these can be tackled by exploring the user experience and mimicking a range of changes to user responses based on trigger words such as 'pain' to allow the chatbot to respond in an empathetic manner.

Given the results of the thematic analysis in regard to using chatbots for patient simulations, we can propose a chatbot for virtual diabetes clinics, during the COVID-19 pandemic. As face-to-face teaching has been limited this would be an effective way of preparing medical students to manage diabetes mellitus as it too, is part of an on-going global epidemic. There are approximately 415 million 15 people living globally with diabetes. It has become a challenge to provide efficient and effective management of diabetes; this unfortunately leads to overwhelming complications. Diabetes is particularly suited to the use of digital technologies as teaching aids due to its quantitative nature importance across primary, secondary and tertiary care.

The Alphabet strategy 16 is a framework for the management of diabetes. It incorporates the core components of diabetes care, in a simple mnemonic-based checklist It consists of: (1) advice on lifestyle changes such as smoking, exercise, diet and vaccine recommendations; (2) blood pressure targets; (3) cholesterol measurement with targets; (4) glucose control with targets; (5) annual eye exams; (6) annual foot exams; (7) use of guardian drugs: aspirin, angiotensin converting enzyme inhibitors/angiotensin receptor blockers and statins. Collectively this checklist ensures the reduction of complications caused by diabetes such as cardiovascular disease, retinopathy and nephropathy.

The novel contribution of this study is that it proposes an approach to educate medical students in the preparation of managing patients with diabetes, using a chatbot. The objective of the chatbot would be to run a virtual diabetes clinic consultation, in order to assess medical students or clinicians, using the Alphabet Strategy approach. The Alphabet strategy has already proved to be a useful instrument for patient education. By incorporating the Alphabet strategy into a chatbot, a technical solution is provided in training medical students and clinicians on the task of achieving effective and efficient diabetes care for all patients.

The second major theme identified was using a chatbot as a revision tool. Many participants felt that using a chatbot to learn simple concepts of pharmacology would make it more interactive. Participants also expressed that an information retrieval chatbot would be useful in making their revision less time consuming when it comes to locating resources. This is consistent with the findings presented by previous studies 6 whereby a simple chatbot was created for a tutoring system which provided answers for administrative questions. The 'Johari Window Model' 17 is applicable when discussing revision. It outlines what is and is not known to self, as shown in Figure 3 . This can therefore be applied to the development of a chatbot for medical education. The chatbot could allow the student to 'bypass' areas of learning that they feel competent in, thus spending more time on areas that need further competency development (Figure 4) .

The third theme identified was the differential usefulness by medical school year group. The majority of participants felt that first-year students would benefit from using a chatbot to practice history taking as it was a safe space and can help build their confidence before meeting real patients. This concept of interviewing patients using a chatbot has been tested before in the form of a 'Virtual Patient Bot'. 7 Students were able to type their message and the chatbot replied back either in text or audio. A study subsequently completed 8 showed that students who used the chatbot gained higher grades in comparison to those that did not.

The fourth theme identified was the standardisation of education and assessment. Participants commented on how experiences differ in regard to their examiners. A chatbot could therefore provide a standardised experience for each student and limit personal bias from a human examiner. However, this would still result in limitations of providing feedback.

The final theme identified was the challenges of use and implementation of using a chatbot. Many participants felt that in a course where patient care is integral to their learning. Substituting a chatbot with real patients would hinder their development and feel that this would put them at a disadvantage, in comparison to those who got more patient exposure. These perceptions were based on participants who have not used a chatbot in their education. Previous research 9 has shown that the willingness of teachers in promoting technology, could influence the student's positive response towards said chatbot, which today is an emerging technological concept.

One strength of the study was the use of focus groups as a methodology. This provided a direct interaction with the participants, enabling key questions along with exploring a wide range of opinions. The rigorous approach to thematically analyse the dataset ensured flexibility to build and explore on further discussions with the latter focus groups. An additional strength of the study included getting a student perspective among all year groups of the medical school. Despite shared ideas and opinions, participants from each year shared personal experiences which contributed to identifying particular student groups which are more likely to benefit from a chatbot and those that would not greatly benefit from a chatbot, in their opinion.

There were limitations in the sampling framework used for this study. A convenience sample is a non-probability sampling method. This meant that participants were selected based on their availability. This therefore meant that there was a risk of not equally targeting participants which represented the population being investigated. 18 However, as this was a pilot study and thematic saturation was achieved, as the emerging themes were similar in the latter focus groups, no further participants were required.

Despite emailing the entire medical school cohort there was a low uptake on the offers for participation. Many of the participants that did get involved, stated they were unclear about what a chatbot was and felt intimidated about taking part in a discussion with limited knowledge. The participants were briefed prior to the focus groups to ensure them that substantial knowledge about chatbots was not required to take part, which in hindsight posed as a limitation. This meant that when certain situations were discussed it was difficult for participants to visualise possible uses for a chatbot. If participants had current knowledge of chatbot advancements, perhaps an immediate suitable use of a chatbot could be identified for future developments. However, as this study was a detailed qualitative analysis, the limited number of participants allowed an in-depth assessment of the participant's perceptions.

Another limitation was that a member check was not done for the study. This would have allowed validation of the data collected by the participants which could have been done by appraising the transcribed discussions from the focus groups. 19 However, participants were provided with opportunities to contact the principal researcher after the focus group.

A final limitation of the study was that the analysis was conducted by the team so there will be an element of bias, as there is a possibility that during analysis points were found to validate a specific theme, however saturation was met whilst gathering data and therefore this was reduced.

The conclusions which were drawn from this pilot qualitative study, highlighted areas which require further research to confirm its findings. A further qualitative study using: a larger sample size, participants who develop chatbots for education and members of the medical education community and who utilise technology to enhance learning. Such participants would be able to provide clearer insight into the production of a minimum viable product which could be further evaluated. This study has suggested implications for such a product.

The study highlighted that medical students and their educators believe that there is scope for integrating a chatbot into the medical curriculum and more importantly, its potential contribution to changing or supplementing the way medicine is taught today. This study identified that there is a need for medical institutions to promote the use of technology in order to encourage its students to explore the various programmes which are being developed for students' learning. Medical schools have a commitment in improving education for their students and this is one potential way of improving how medicine can be taught outside of the lecture hall.

Whilst it is important to understand the limitations and appropriateness of replacing patient contact with a chatbot to improve medical education, it is also important to understand that a lot of time is spent away from clinical placements. There will always be an accepted belief that medical students learn best by meeting patients and shadowing doctors on the wards. However, there clearly is a niche in the timeline of medical education at the beginning of the degree, where these clinical placements have not yet been established. A chatbot could help create a safe environment for students to learn and develop their skills, before moving into a clinical environment. This study has shed light on areas where a chatbot could be integrated during these transition periods in order to enhance their medical school experience.

Chatbots are currently underused in British medical education. (1) They are not being developed for the correct needs of medical students. (2) They may not be encouraged by institutions, as traditional trial and tested methods are often utilised. (3) Students are unaware they exist and are therefore unsure how a chatbot can benefit their learning. The key findings of this study were that there is an opportunity to integrate a chatbot into the British medical curriculum either as a patient simulator, a revision tool and a tool to standardise examinations. This study also highlighted that a chatbot would be most useful during the earlier years of medical training rather than the later years. However, due to the limitations of the study, we may be underestimating the value of chatbot use in the later years of medical school, as its value could potentially come from being a virtual teaching assistant. The perceptions of chatbot use by medical students need to be urgently addressed with education on its benefits and limitations.

To conclude, the aim of building a chatbot for medical students would be to produce a chatbot which could be used to reduce tutor input and promote self-learning, something which many of the participants expressed in the study. Frequent use of the chatbot would then in turn logically improve competence as outlined by the General Medical Council framework on the 'Generic Professional Capabilities' especially when 'creating effective learning opportunities for students and doctors' 20 and increase performance feedback with regular use. This would therefore improve patient safety. The chatbot could be used as a protocol for standard assessment either by (a) the medical school or (b) the medical students during independent learning. Chatbots could prove to be great tools for medical students. We have demonstrated a clear example of how a chatbot can be implemented into the medical education system to run a virtual diabetes clinic, using the Alphabet Strategy, for managing patients with diabetes during the COVID-19 pandemic. Students can engage and utilise chatbots to ensure that they are regularly practicing key skills and knowledge, which is required of them to be competent doctors for the NHS especially during the COVID-19 global pandemic which has disrupted medical education globally.

Contributorship: All listed authors have (1) made substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data; (2) drafted the article or revised it critically for important intellectual content; and (3) approved the final version to be published.

Declaration of Conflicting Interests: The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Disclaimers: This article formed part of the medical studies of AK at the University of Warwick and was submitted as part of an ongoing assessment.

Ethical approval: Ethical approval for the methodology was received from the Biomedical & Scientific Research Ethics Committee at the University of Warwick (BSREC) (ref:

BSREC-CDA-SSC2-2019-46 (02/10/2019)). Prior to initiating the focus groups, written and verbal consent was obtained from each participant. All focus groups took place in private settings and were recorded on an encrypted recording device issued from Warwick Medical School. All focus groups were transcribed keeping participant identity anonymous.

ORCID iDs: Anjuli Kaur https://orcid.org/0000-0002-7803-306X Tim Robbins https://orcid.org/0000-0002-5230-8205

Echo and Alexa Devices by Amazon.co.uk. Amazon.co.uk

Healthcare Chatbots Market -Global Opportunity Analysis and Industry Forecast

Mobile learning in medicine: an evaluation of attitudes and behaviours of medical students

Virtual patient simulations in health professions education: systematic review and meta-analysis by the digital health education collaboration

Medchatbot: an UMLS based chatbot for medical students

Data representation and algorithms for biomedical informatics applications

A multi-institutional randomized controlled trial of adjuvant web-based teaching to medical students

Teacher attitude towards use of chatbots in routine teaching

COMET: clinically observed medical education tutorial -a novel educational method in clinical skills

Qualitative Data Analysis For Applied Policy Research. 1994. The Qualitative Researcher's Companion

Physicians' perceptions of chatbots in health care: cross-sectional webbased survey

Complex consultations and the 'edge of chaos'

Designing for Health Chatbots

Diabetes Prevalence

Alphabet strategy for diabetes care: a multi-professional, evidence-based, outcome-directed approach to management

The Johari window, a graphic model of interpersonal awareness

Sampling methods in clinical research; an educational review

Member checking