key: cord-0904696-pdcjptu3
authors: Upadhyay, N.; Wadkin, J.C.R.
title: Can training in diagnostic radiology be moved online during the COVID-19 pandemic? UK trainee perceptions of the Radiology-Integrated Training Initiative (R-ITI) e-learning platform
date: 2021-06-18
journal: Clin Radiol
DOI: 10.1016/j.crad.2021.06.003
sha: aa40cc2e2e8e44b86be14582e8dd9428f81fc2ec
doc_id: 904696
cord_uid: pdcjptu3

AIM: To assess trainee perceptions of the Radiology-Integrated Training Initiative (R-ITI)) e-learning modules. MATERIALS AND METHODS: A mixed methodology approach was used, with triangulation between a thematic analysis of eight semi-structured interviews from radiology trainees and trainers, and a contextual analysis of 60 free-text feedback comments and module ratings from trainees after completion of R-ITI modules. RESULTS: Three broad themes emerged: “learning the subject matter”, “learning the role,” and “e-learning preferences”. Superficial learning techniques were prevalent when “learning the subject matter” during early training, with e-learning resources providing a good pedagogical fit for this learning. Much of what was considered “learning the role” of the radiologist was learned at the workplace. This included topics with inherent subjectivity, which were difficult to convey with e-learning. Trainees' “e-learning preferences” favoured modules that incorporated many imaging cases with layer annotation, clinical relevance, and self-assessment. CONCLUSIONS: The ease of reproducing imaging studies using the R-ITI platform represents a huge potential for e-learning. Content tailored to the learning needs of the trainee, the appropriateness of the subject matter for an online platform, and the design of the e-learning modules are important considerations. Radiology training also involves important tacit learning and discussions around subjective topics, which are difficult to capture on this platform.

In contrast to other medical specialities, diagnostic radiology has the potential to benefit from a privileged relationship with e-learning. Imaging studies, which represent the crux of the work performed in diagnostic radiology, can be effectively reproduced using electronic media.

In diagnostic radiology, trainees develop image interpretation skills by obtaining an appreciation of normal findings and signs of pathology on imaging studies. These studes can easily be replicated effectively on online platforms and interrogated using web-based tools, in much the same way that radiologists interrogate studies using workstations, in their routine clinical practice.

The potential advantages of conducting radiology training online are numerous. Well-designed online modules are able to demonstrate complex anatomy much more effectively than a textbook, while allowing information to be disseminated widely and in a cost-effective manner. 1, 2 Radiology trainees traditionally travel to specialist centres to gain experience of rarer pathologies, which would not be routinely encountered at their base hospital. Trainees in schemes that do not have links with such specialist centres may be at a disadvantage during postgraduate examinations, and in their future practice.

The advantages associated with the ability to reproduce imaging studies in an e-learning format led to the Royal College of Radiologists (RCR) in the UK, in conjunction with the Department of Health, developing the Radiology-Integrated Training Initiative (R-ITI), which has been one of the largest e-learning projects in the world. 3 The R-ITI platform, designed to support the learning of clinical radiology trainees, includes online modules covering the full range of radiology subspecialties, medical physics and professional skills (Electronic Supplementary Material Table S1 ).

The R-ITI provides the advantage of increased accessibility to learning. As many traditional training schemes are oversubscribed, and with a need to increase the number of qualified radiologists, the requirement for a more flexible way of meeting training requirements in the UK has been highlighted. 2 Although there has been documentation of the use of elearning strategies for undergraduate radiology teaching, 4e6 there is a general paucity of literature on how radiology trainees engage with e-learning materials. Furthermore, although these existing studies document positive outcomes for e-learning resources, with a focus on participant satisfaction, there is limited evidence on what motivates student engagement.

During the COVID-19 pandemic, the importance of elearning and online delivery of educational activities has become more pertinent than ever. Restrictions on travel and the ability to gather multiple people in a single venue have highlighted the importance of online distance learning.

There is a need to better understand what motivates radiology trainees to interact with the R-ITI e-learning platform, and how they perceive these resources. A greater understanding of this will have important implications for how e-learning might be used to deliver radiology training during the ongoing COVID-19 pandemic and beyond.

A mixed methodology approach 7 was used for this study, with analysis of semi-structured interviews and online feedback of RITI e-learning modules. Participants were invited to participate from conventional radiology training schemes in London and the East of England. Purposive sampling was performed, so that both junior and senior trainees were represented, with participants from each group selected on a first-come, first-served basis, prior to the beginning of the COVID-19 pandemic. A series of semistructured interviews with two junior trainees (both in their second year of training), three senior trainees (two in their fourth year and one in their fifth year of training), and three consultants with formal training roles were conducted, each lasting 45 minutes (Fig 1) . The participants selected all came from training schemes, which involved rotating between teaching hospitals and district general hospitals placements.

The feedback left by trainees for online R-ITI e-learning modules, which cover the core radiology curriculum was also analysed. Trainees voluntarily leave modules a score between 1 and 5, with 5 being the highest, alongside written free text feedback. There were a total of 60 completed reviews available on the platform for analysis. These reviews covered 56 modules, with three reviews for one module and two reviews for two modules. The reviewed modules covered a range of subspecialites and related topics: cardiac, thoracic/respiratory, musculoskeletal, gastrointestinal, obstetrics and gynaecology, paediatrics, head and neck, neuroradiology, radionuclide radiology, physics, basic science, professional skills and introductory modules. Permission was granted by the Royal College of Radiologists to perform this analysis.

Thematic analysis of the interview data was performed using a previously described framework. 8 The data were transcribed and inductive coding was performed. The codes were categorised into sub-themes. These were re-reviewed and then overarching themes were generated.

For the 60 R-ITI module reviews, a content analysis was performed. The numerical feedback scores were also compared with the content of the written feedback to assess for any trends. As a validated scoring system is not available, critical analysis of the R-ITI modules was performed on the feedback left by trainees, for whom the modules had been designed.

Two different types of triangulation were used: "triangulation of sources" and "triangulation of methods". 9 Triangulation of sources was used by interviewing both trainees and trainers. It was hoped that these complimentary perspectives would provide a broader picture of radiology training. Triangulation of methods was used by comparing the thematic analysis from the interviews with the analysis of the feedback for R-ITI modules.

This study considered the milestone of passing the final radiology examinations as the divide between junior trainees and senior trainees. This is in keeping with the division used by trainers and the UK radiology community in general.

Three broad themes emerged from the interviews. These were; "learning the subject matter", "learning the role," and "e-learning preferences". Salient quotes relating to the three themes are demonstrated in Table 1 . Despite the R-ITI elearning resources being largely optional at the schemes represented by the interviewees, all the interviewees reported using e-learning resources regularly. All trainees interviewed reported voluntarily choosing modules that were recommended by peers, or had good reviews posted about them.

In "learning the subject matter", most trainees commented on the large volume of material junior trainees need to learn to pass the exams during the first 3 years of their training (Table 1, 1A) . Superficial learning methods were found to be favoured during this period, due to the intensive nature of the revision required. For the written exams taken during this period, the trainees highlighted the importance of memorising lists of differentials. These would often be learned out of context, separate from the imaging ( Table 1 , 1B,C). The consultant trainers interviewed also looked for trainee knowledge of differential lists as a sign of competence and associated this with a perceived likelihood of exam success (Table 1 , 1E,F).

There was a clear difference between trainees' approach to written exams and learning at the workplace. For the written exams, there was a focus on learning relevant lists of factual knowledge, not on the practical applications of this information. E-learning resources lend themselves well to this superficial learning technique and therefore online modules were commonly used to reinforce this learning. Although there was the volition to engage with deeper learning strategies, it was felt this was not possible at this stage of training due to the volume of material that needed to be covered for the written examinations. Some trainees felt however, that learning in depth was good preparation both for image interpretation examinations and for their future role as a consultant radiologist (Table 1, 1D) .

Consultant trainers that were interviewed also felt that a strong factual knowledge base allowed radiologists to add value to the multidisciplinary team meetings. In particular, knowledge of the statistical accuracy of radiology examinations was highlighted. Recalling the parameters of a test, such as sensitivity and specificity, alongside advising colleagues using this specialist knowledge, was therefore considered part of the role of the radiologist (Table 1, 2FeH) . This factual knowledge could be easily reproduced through e-learning resources for radiology trainees. The practical application of this knowledge, however, required first-hand experience of the role of a radiologist in the workplace.

For the trainees interviewed, much of what was considered of "learning the role" of the radiologist was learned in the workplace. This included developing attitudes towards uncertainty, learning how to communicate this uncertainty, and the duty to protect patients from risks associated with unnecessary radiation exposure (Table 1 , 2B,C,E). This learning was often impressed upon the junior trainees when they were supervised by consultants or were working alongside senior trainees on-call in radiology departments. Junior trainees demonstrated malleability in their perception of risk, which often depended upon the beliefs of the supervising consultant (Table 1, 2B) .

Both trainees and trainers alike generally felt that learning how to convey uncertainty in radiology reports using e-learning was not effective. The subjectivity of this concept was found to be difficult to capture in an e-learning module and therefore practical experience was required in order to learn this aspect of the speciality.

In gauging "e-learning preferences", there were numerous examples of advantageous features of online learning modules provided by both the trainees and trainers. Consultant trainers reported that they often directed trainees to e-learning resources due to time pressures at work. A sound knowledge of the R-ITI e-learning content and modules available allowed trainers to mitigate the effects of service-provision pressures on training (Table 1, 3D) . R-ITI modules and other e-learning resources, which had a lot of imaging cases, colour coding, and layer annotation, were valued by trainees over text-based modules (Table 1 , 3A,B). Modules that focused on the differentials for imaging findings, rather than the details of the specific diagnoses, were found to be useful because they better reflected the pragmatic challenges faced by trainees at work (Table 1, 3C) .

Trainees completing R-ITI e-learning left scores for the individual modules, with 65% of modules receiving a score of 4 or 5 out of a total of 5 (Fig 2) . Almost 95% of the modules that were given the highest rating (4 or 5) were associated with positive comments about content (18/19 comments).

Of the modules that were given the lowest rating (1 or 2), associated negative comments were related to technical issues or module content.

Triangulation between the written R-ITI module feedback and the interviews demonstrated that a number of points mentioned by interviewees were also expressed in the feedback comments (Electronic Supplementary Material Table S1 ). The importance of annotation on the e-learning modules was mentioned by a number of interviewees, and in feedback comments. The importance of well-designed opportunities for self-assessment was also highlighted in both the interviews and the feedback comments.

The biggest discrepancy between the interviews and the feedback comments related to technical difficulties when using the system. None of the interviewees had suggested problems with this, although almost 30% of the R-ITI feedback (18 comments) mentioned technical issues. In over 60% of these cases (11 comments), technical difficulties led to problematic navigation through the module, and a resulting poor comment on the feedback.

Both the greatest amount of criticism and the greatest amount of praise from the feedback comments related to the content of the modules. Negative comments related to Table 1 Example interviewee quotes from semi-structured interviews, arranged in the emerging themes from the thematic analysis coding.

Interviewee quotes 1. Learning the subject matter Trainees A) "There's so much to get through. It kinda takes over your life. Every 6 months ... in the library like a madman" B) "It's weird because you learn how to describe these bone lesions, but you don't know what they look like e a bit silly really." C) "It's nice seeing other people in the group teaching ... It's more sociable. But to learn the lists you have to sit down with Chapman [book name] and just keep going through them." D) "I enjoyed revising for the 2Bs much more than the 2As. You feel like you're learning stuff that's actually useful for the job." Trainers E) "You can tell they've improved when they come back from the DGH [District General Hospital] then tell you the differential for something while you're in the middle of looking it up!" F) "I keep telling them you've got to know your lists. The ones that listen pass." 2. Learning the role Trainees A) "When there's lots of urgent work to do it's important to prioritise. Every clinician thinks their scan's most important." B) "I had one boss who had a go at me for "sitting on the fence" too much ... So I changed my reports and became more definitive." C) "Modules on statistics are useful ... But I think your attitude towards [risk] is a really personal thing." D) "The list of phrases were useful. But the way they suggest you use them is just one person's view." E) "Some teams would fry a patient with CT scans every day they could. When you tell the FY1 the risk of cancer associated with their CT request you often hear a stunned silence over the phone. I tell them to go away and discuss it with their registrar." Trainers F) "... only the radiologist can tell you which is going to be the most useful test ... [certain specialities] just don't get that some tests just aren't good enough to pick up the pathology ... and that some tests will just give you non-specific results." G) "We're clinical (emphasised) radiologists ... We need to remind the other specialties that were not just technicians ..." H) "Can you call back and put 100,000 people through invasive tests to pick up one cancer, which may not even affect that patient's outcome?" In some situations there are no black-and-white answers." 3. E-learning preferences Trainees A) "The ones rated five stars tend to have lots of cases with good annotations and explanations ... Some of the modules are just full of text ... You can get that from a textbook." B) "The best ones are the ones where you can test yourself -and then the answer has the main points annotated on the imaging. Then it's clear where you've gone wrong." C) "The modules which work through differentials are more useful than modules that go through a disease in detail." Trainers D) "The R-ITI's been a good way to get the trainees to do some background reading before our sessions. I find there's a lot less time to teach now ... If the trainees know a bit to start with, we can make better use of our time together." modules taking too much time to complete, or containing too much material. The positive comments generally related to the design of a module, the clinical relevance of the content, or good explanations of the subject matter. Negative comments were noted relating to topics that do not lend themselves well to the transmission of facts, and are more commonly associated with the discussion of a plurality of views. Examples of this included a module on medical ethics and a module on safety in radiology. This highlighted the difficulty of translating topics associated with a discussion of opinions into an e-learning format and resonated with the importance of radiology trainees learning the "role of the radiologist" at the workplace.

Examples of social learning online were scarce; however, there were examples of discussions on forums covering topics that would not be found in textbooks, such as "hot topics" in emerging areas of radiology. For many of these topics, there was no consensus approach or evidence-based guidelines; therefore, discussion with fellow experts was valued.

E-learning offers a potentially effective platform to train radiologists and is particularly relevant during the COVID-19 pandemic, when opportunities for face-to-face teaching may be limited. The R-ITI e-learning platform is one of the most comprehensive online radiology resources available for trainees; it is freely accessible to UK trainees but requires a paid licence for trainees and radiologists based outside the UK. This study aimed to provide a better understanding of how trainees perceive and interact with such online resources.

The trainees' use of e-learning in this study was predominantly underpinned by an emphasis on fact acquisition. Superficial, fact-based learning was described by trainees as a requirement for the earlier written examinations. Furthermore, rote learning was encouraged by trainers so that the relevant factual "lists" required for the workplace were acquired. A trainees' aptitude for being able to remember these lists appeared to be a surrogate measure of competence for the trainers interviewed.

Trainees described using the R-ITI e-learning resources for deeper learning strategies, when there were highquality images with layer-annotation and opportunities for self-assessment. When preparing for their image interpretation examinations, some trainees described deeper learning approaches to understand the relationship between different anatomical structures. This may allow an appreciation of the three-dimensional configuration of anatomy with learners building on their existing knowledge to create more comprehensive three-dimensional models, although enhancing logical and critical thinking ability. 10, 11 Although trainees in this study reported enjoying deeper learning, external factors such as stress and the need to pass exams, may influence the trainees' choice of learning strategy. 12 This may explain why junior trainees, in the midst of written examinations, which required a large volume of material to be covered, emphasised the use of rote learning. Senior trainees, who no longer had exam pressures and were training in a chosen subspecialty area of interest, were more likely to describe deeper learning strategies.

The use of different learning strategies for different types of examinations, as demonstrated in this study, has also been previously recognised. Students have been shown to be more likely to use surface learning approaches for multiple choice question (MCQ) written examinations in contrast to deeper learning approaches for assignment essays. 13 The extensive use of MCQ examinations in radiology is therefore likely to perpetuate the emphasis on rote learning within the radiology community, especially during the initial years of training, which may influence the way elearning is utilised.

Trainees described learning during normal working hours, which related to the development of concepts of uncertainty. Such learning might be difficult to capture with e-learning. Trainees' attitudes towards uncertainty were influenced by the differing views of supervising consultants. The interviews from the senior trainees suggest that exposure to this variability may be part of a formative process for the trainee, as it allows them to develop their own opinion on risk and uncertainty when they practice independently. This formative process may also explain the strong metacognitive skills demonstrated by trainees when they described the variability in the communication of risk amongst consultants. This important formative learning would be difficult to recreate on an e-learning platform. Indeed, by reifying this sociocultural learning with electronic resources, appreciation of this variability, and the associated learning, may be lost.

There was also evidence of tacit learning from peers, most notably when trainees were shadowing colleagues oncall, which would also be difficult to recreate using elearning. This learning related to identification of their professional roles and developing the associated behaviours that their peers felt were required when starting on-call. This was also demonstrated when trainees described learning the role of the "gatekeepers" to imaging (Table 1 , 2A) and the role of the radiologist in multidisciplinary team meetings. It is difficult to foresee how the complexities associated with this type of sociocultural learning might be captured by e-learning platforms. With improvements in technology, face-to-face activities have seen a recent move over to remote teleconferencing, such as with remote multidisciplinary team meetings. This may offer scope for recreating the learning opportunities associated with social interaction on an online platform.

Some examples of collaborative learning using online forums, which has been considered an effective learning tool 14 were provided by a senior trainee and consultant. Both used forums to discuss cutting-edge areas of imaging which related to their chosen subspecialty. As these emerging specialist areas are not covered in conventional textbooks, the interviewees found it helpful to discuss ideas with experts that had similar interests. This is reflected in the move to virtual conferences for many radiology societies brought about by the COVID-19 pandemic. For these participants, the peer support from the online forum served as an effective means to engage with expertise knowledge and reduced the requirement to travel long distances. It was interesting that the junior trainees did not engage with forums. This is likely to be because the more basic, established knowledge that they required was readily accessible through other media such as textbooks and e-learning modules.

Trainees identified the pedagogical advantages of the ability to switch on and off layers of annotation over imaging, which helped highlight salient points and also facilitated self-assessment, although colour coding helped to clarify discussion points. E-learning modules designed to incorporate these elements although allowing the flexibility to choose engagement in a superficial or deep manner, are more likely to be well-received by trainees. R-ITI module feedback suggests that attention must also be given to the user experience, which includes minimising technical issues, ensuring the length of learning modules are appropriate and incorporating enough opportunities for interaction and selfassessment.

There were limitations to this study. Although the strength of the qualitative interview component of this study was to allow an insight into the tacit areas of learning that might not be appreciated with quantitative methods, and as per convention recruitment was stopped when it was it was felt that thematic saturation was reached, 15 the sample size means caution must be taken with the generalisability of the interview findings. Furthermore, the interviewees in this study were all taken from conventional radiology training programmes. In the newer radiology academies, there is an emphasis on a blended learning approach, with trainees receiving didactic teaching, and dedicated time for elearning. Given the more established integration of elearning at the academies, it could be that these resources are received more favourably by trainees at these centres. It is also possible that those that responded to the invitation to interview may have had strong views or interests in elearning, which may not be representative of the wider trainee cohort. Consideration of e-learning for procedural skills was also beyond the scope of this study, but is an important consideration in radiology training. These areas warrant further consideration in future studies.

This study suggests that trainee engagement with elearning is strongly influenced by context. The radiology community places a significant emphasis on fact acquisition, through both workplace-based training and the written examinations. E-learning platforms, such as the R-ITI, are well placed to facilitate this knowledge transmission. When external pressures promoting rote-learning are less at the forefront, trainees are more likely to engage with deeper learning strategies, and well-designed modules that are rich in imaging content can facilitate this; however, tacit learning is an important feature of radiology training, which is difficult to reify with current e-learning technology. The important learning from discussions around subjective topics is also currently difficult to capture with this platform.

Radiology education: a glimpse into the future

The radiology integrated training initiative (R-ITI)

Radiology: R-ITI. E-Learning for Healthcare

Evaluation of use of e-Learning in undergraduate radiology education: a review

E-learning for chest x-ray interpretation improves medical student skills and confidence levels

Evaluation of a new elearning framework for teaching nuclear medicine and radiology to undergraduate medical students

Qualitative research: an essential guide to theory and practice

Using thematic analysis in psychology

Enhancing the quality and credibility of qualitative analysis

The human cadaver in the age of biomedical informatics

Catching up

Motivation of radiology residents: an interaction between personal and environmental factors

The influence of assessment method on students' learning approaches: multiple choice question examination versus assignment essay

PBL online: a framework for collaborative e-learning. In: Problem-based learning online

Semistructured interviewing in primary care research: a balance of relationship and rigour