Promoting Evidence-based Veterinary Medicine through the online resource ‘EBVM Learning’: User feedback

¹Bristol Veterinary School, University of Bristol, Langford, Bristol, BS40 5DU, UK
²Vet Partners, Spitfire House, Aviator Ct, York, YO30 4UZ, UK
³St. George’s University, School of Veterinary Medicine, True Blue, Grenada, West Indies
⁴Clinic for Animal Reproduction, Faculty of Veterinary Medicine, Freie Universität Berlin, Germany
⁵RCVS Knowledge, Belgravia House, 62–64 Horseferry Road, London, SW1P 2AF, UK
⁶School of Veterinary Medicine and Science, University of Nottingham, Sutton Bonington Campus, Leicestershire, LE12 5RD, UK
⁷Veterinary Epidemiology, Economics and Public Health Group, The Royal Veterinary College, Hawkshead Lane, North Mymms, Hatfield, Hertfordshire, AL9 7TA, UK
^8aInformation Services Group, University of Edinburgh, Easter Bush, Midlothian, EH25 9RG, UK
^8bDeanery of Clinical Sciences, College of Medicine & Veterinary Medicine, University of Edinburgh, Edinburgh, EH16 4SB, UK
⁹University of Agricultural Sciences and Veterinary Medicine of Bucharest, 59 Marasti Bd., 011464, 1st district, Bucharest, Romania
¹⁰Centre of Evidence Based Dermatology, University of Nottingham, King's Meadow Campus, Nottingham, NG7 2NR, UK
¹¹School of Veterinary Science, University of Liverpool, Leahurst Campus, Neston, CH64 7TE, UK
¹²Medical Sciences Library, University Libraries, Texas A&M University, College Station, TX 77843–4462, USA
¹³Dept. of Medicine and Surgery, Chattogram Veterinary and Animal Sciences University, Zakir Hossain Road, Chattogram–4225, Chattogram, Bangladesh
¹⁴Atlantic Veterinary College, University of Prince Edward Island, 550 University Ave, Charlottetown, PEI, C1A 4P3, Canada
¹⁵Faculty of Veterinary Science, University of Pretoria, Onderstepoort, 0110, South Africa
^*Corresponding Author (sarah.baillie@bristol.ac.uk)

‘EBVM Learning’ is a freely available resource created in 2015 by an international team with the support of RCVS Knowledge. The resource comprises a series of online modules teaching the fundamental concepts of evidence-based veterinary medicine (EBVM) (Ask, Acquire, Appraise, Apply & Assess) supported by case studies, exercises, worked examples and quizzes. The aim of the current study (undertaken in 2019) was to review ‘EBVM Learning’ to ensure its ongoing relevance and usefulness to the range of learners engaged in EBVM. Feedback was gathered from stakeholder groups using website statistics and feedback forms, a survey and semi-structured interviews to provide a combination of quantitative and qualitative data.

Website statistics revealed an international audience and a steady increase in visitors exceeding 1,000 per month in August 2020. Feedback via the online form (n=35) and survey (n=71) indicated that the resource was well structured, with an appropriate level and amount of content, useful examples and quizzes and the majority of respondents would use it again. Semi-structured interviews of educators (n=5) and veterinarians (n=8) identified three themes: features of the ‘EBVM Learning’ resource (strengths, suggestions for improvement), embedding the resource in education (undergraduate, postgraduate) and promoting EBVM (challenges, motivation for engagement). At a project team workshop the results were used to plan updates to the existing content and to identify new ways to promote learning and engagement. An updated version of ‘EBVM Learning’ was developed.

‘EBVM Learning’ is helping to produce the next generation of evidence-based practitioners and enabling to engage in the concepts of EBVM as part of their clinical practice.

There is a clear need to include research evidence in veterinary practice to ensure clinical guidelines and decision-making are informed by the latest findings and continue to evolve and adapt (Doig, 2008; Holmes & Cockcroft, 2004; Feetham & Raffan, 2014; Dean et al., 2017; and Dean & Heneghan, 2019). However, in medicine it has been shown that there can be a significant time lag between the publication of clinically important research and a change in treatment recommendations made by experts (Antman et al., 1992; Glaziou 2005; and Walker et al., 2019). Evidence has become more easily accessible to all health practitioners including veterinarians, for example, through publication of evidence syntheses and open access publishing. Patients and owners also have access to some of this information. As a result, there has been growing pressure from the public, professional bodies, practitioners and students to ensure all professions, including veterinary medicine, continue to be science-driven and evidence-based (Dean et al., 2017; Huntley et al., 2017; and RCVS Code of Professional Conduct for Veterinary Surgeons, 2020).

The practice of evidence-based veterinary medicine (EBVM) can be defined as the ability to choose the best clinical approach for patients through rational decision-making supported by sound clinical reasoning, the current best scientific evidence, and an understanding of risk management (Arlt et al., 2012). The circumstances of the patient and owner also need to be taken into consideration (Armitage-Chan, 2020). However, there are recognised challenges to performing evidence-based (veterinary) medicine (EB(V)M) when in clinical practice including limited access to databases, lack of time to search for key literature, lack of skills to effectively appraise the quality of identified research, and difficulties in applying the evidence and assessing the impact (McColl et al., 1998; Kastelic, 2006; and Vandeweerd et al., 2012). The situation has been improved by the production of evidence syntheses, a systemic structured search of the evidence which is then summarised for clinicians in an easily accessible format (a Knowledge Summary or Critically Appraised Topic (CAT)). A number of such initiatives have been introduced in recent years (BestBETs for Vets; RCVS Knowledge; Kasch et al., 2017; and Brennan et al., 2020). The use of EBVM will further improve over time if the necessary skills are taught in the veterinary curriculum to enable future practitioners to perform EBVM in a busy practice environment (Janicke et al., 2020).

It is recognised that teaching everything (“just in case knowledge”) in curricula is no longer possible and there is a need for a change in direction towards teaching students the skills to find, appraise and utilise evidence in practice (“just in time knowledge”) (Dean et al., 2017). However, the challenges in teaching these skills e.g. recognising uncertainty, understanding how evidence does or does not fill the gaps in our knowledge, can be considerable. In EBVM, the decision-making process is complicated by a variety of types of evidence e.g. primary research versus narrative text or expert opinion; in turn, there is a greater need for students to understand how to make decisions to accommodate greater levels of uncertainty (Holmes & Cockcroft, 2004). Another barrier is finding the time to incorporate teaching these skills into already overcrowded curricula (Shurtz et al., 2017) and contextualising the process, so it is presented as an integral part of clinical decision-making and not just a theoretical concept (Dean et al., 2017). Learning is enhanced by embedding the task into realistic clinical scenarios, similar to those that would occur in veterinary practice (Arlt et al., 2012). A number of initiatives have been introduced in veterinary curricula often culminating in students creating a critically appraised topic; for example, in first year (Hardin & Robertson, 2006), as a clinical pharmacology assignment (Fajt et al., 2009) and as part of clinical reproduction (Arlt et al., 2012) and farm animal rotations (Steele et al., 2013). A toolbox has been developed for veterinary educators to consult when reviewing or introducing EBVM in a curriculum (Janicke et al., 2019).

In 2013, a workshop was held at the Centre for Evidence-based Veterinary Medicine (CEVM), University of Nottingham to bring together those involved in teaching EBVM at each of the UK veterinary schools (Dean et al., 2017). Following the workshop, the team was extended to include international partners in a project (‘EBVM Learning I’) to produce an open-access coherent online resource with a series of modules for teaching the fundamental concepts of EBVM. Over 20 people were involved in developing the resource, including veterinarians who teach EBVM, librarians and a website designer. The team represented 10 veterinary schools and four countries (UK, Germany, Romania and Canada). The resource was divided into distinct sections based on the steps of EB(V)M: The Five As (Heneghan & Badenoch, 2006). First, constructing a relevant patient question (Ask) and then searching for the evidence to answer the question (Acquire). Next, critically evaluating the resulting evidence for quality and applicability to the question (Appraise) and then using the evidence, as appropriate, in the clinical context (Apply). Finally, monitoring and evaluating the outcomes and impact on practice (Assess). Each section included interactive content and provided context through case-studies, exercises, clinical examples, videos and quizzes. The resource ‘EBVM Learning’ was launched by RCVS Knowledge in November 2015.

The aim of the current project ‘EBVM Learning II’ (undertaken in 2019) was to review the online resource ‘EBVM Learning’ to ensure its ongoing relevance and usefulness to the range of learners engaged in EBVM. The project team was extended to include more international partners from universities in Grenada, USA, South Africa and Bangladesh, representation from veterinary nursing and several UK-based practitioners. Feedback was gathered from stakeholder groups using website statistics, a survey and semi-structured interviews to provide a combination of quantitative and qualitative data. The information was used to update the existing content and identify areas for development and innovative approaches to promote learning and engagement.

Online resource usage and user feedback

Basic information about visits to the ‘EBVM Learning’ website was collected by RCVS Knowledge using Google Analytics. The data from the time the website was launched in mid-November 2015 to the end of January 2019 were analysed and included visitor numbers, country of origin and the percentage of return visits.

On completion of the resource, users were invited to provide feedback via an online form and were asked to provide free-text responses to questions about what they liked, did not like, what could be improved and their reason for using the tutorial. The data were analysed to identify recurring comments (by author SB). Users were also asked if they would use the resource again and recommend it to others (yes, no, or unsure).

Survey of user experience

A more in-depth survey was developed, informed by the questions and feedback data from the online form, to investigate further the user experiences of the resource to date. Following a successful pilot by academic colleagues to check clarity of questions and online functionality, the survey was delivered through Online Surveys (www.onlinesurveys.ac.uk) to veterinary students and veterinarians who had used ‘EBVM Learning’. The survey included basic demographic data (gender, country) and veterinarians were asked for their year of graduation, area of work and context for using the resource. A Likert scale (strongly disagree, disagree, undecided, agree, strongly agree) was used for questions about the resource’s content, usefulness, aspects of functionality, time commitment, if participants would use it again and recommend it to others. Free-text questions asked what they liked the most, the least and for any suggested improvements, including ways to make the ‘EBVM Learning’ resource more engaging and relevant.

Two groups of veterinary students were identified, both of which had recently completed a course incorporating the resource: the year 4 cohort of the 5 year BVSc programme at the University of Bristol, UK and the semester 2 cohort of the 4 year DVM program at St George’s University, Grenada. Administrative staff contacted the students via email including a link to the survey and a project information sheet. Student responses were anonymous.

The veterinarian survey was promoted to the veterinary profession through RCVS Knowledge media outlets and an email sent to those registered on a postgraduate EBVM module at the University of Liverpool, UK. At the end of the survey, participants were asked if they would be willing to be contacted (by email) for a follow-up interview to explore their experiences and suggestions in more detail. In order to maintain anonymity, the email details were excluded from the analysis of the survey response data.

The survey was open for a period of 6 weeks. Data were exported into Excel spreadsheets and the Likert items were collated and summarised graphically. The free-text responses were reviewed to identify recurring comments (by author SB).

Interviews

Two groups were selected for interview to gain more insight into ‘EBVM Learning’ from the perspective of practitioners and educators using the resource in their teaching. Practitioners were approached if they volunteered to be contacted for follow-up interview via the online survey; eight volunteered and all were interviewed, six having previously completed an EBVM module (for the Certificate in Advanced Veterinary Practice), another moved into education recently. Five educators were interviewed as a convenience sample, identified through project team contacts across four continents. The backgrounds of these educators included veterinary, human health care and epidemiology. They were all using ‘EBVM Learning’ in some format within their undergraduate and/or postgraduate EBVM teaching. Interviewees were provided with an information sheet describing the purpose of the research study and implications of participation, and returned a signed consent form.

A template was developed for the semi-structured interviews with the same questions for the two groups covering: current use and application of the resource, experience of the resource and suggestions for improvements. A pilot interview informed a minor amendment to the order of questions. Interviews were conducted via Skype or mobile phone, recorded and transcribed, whilst maintaining the interviewee’s anonymity.

Transcripts were analysed using inductive thematic analysis (Braun & Clark, 2012). One interview from each group was analysed independently by two researchers (by authors ES and SW) before meeting to reach a consensus on the sub-themes. The same sub-themes were identified in both interview groups. The remaining transcripts were coded by ES to produce cohesive themes for the two cohorts.

Project team workshop

A 1 day workshop was hosted by RCVS Knowledge in London where the research findings were presented and the updates to ‘EBVM Learning’ were discussed. RCVS Knowledge demonstrated features of new courseware that will be used to host the updated version of the resource. Project team members then worked in author groups for each ‘EBVM Learning’ section to plan the specific changes.

Ethical approval

The study received approval from the Faculty of Health Sciences Research Ethics Committee, University of Bristol, UK (# 81624), the Veterinary Research Ethics Committee, University of Liverpool (# VREC800) and the Institutional Review Board at St George’s University, Grenada (# 19020).

Online resource usage data and user feedback

Over the period monitored (mid-November 2015 – end of January 2019) Google Analytics recorded 20,925 visitors, of which approximately 15% were returning visitors. There was a steady increase over the first 2 years with an average of just under 500 visitors per month. This upward trend continued and exceeded 1,000 visitors per month for the first time in August 2018. The visitors represented an international audience with most participants in the UK (33%) and North America (30%) followed by Australia (5%). Visitors from other countries included Ireland, France, Russia, Grenada and Brazil (each between 1 and 2%).

The online form had been completed by 35 users. Most (66%) had used the resource as part of a course (undergraduate or postgraduate) with other reasons being to find out more about using EBVM in practice (9%), to teach students (9%) and to learn how to appraise a paper (6%) while a few did not provide a reason (10%). When asked about using it again: 80% said ‘Yes’, 17% ‘Unsure’ and one did not answer. When asked about recommending it to others: 89% said ‘Yes’, 8% ‘Unsure’, 0% ‘No’ and one did not answer. Recurring comments in response to the question 'What did you like the most?' were that it was well structured (31%), and had good examples (31%), quizzes (20%) and references/links (17%). The aspects liked the least were that some parts were too wordy (46%) and/or too long (29%). Suggestions for improvement mostly related to more interactivity (26%) e.g. case examples, quizzes, videos, and an estimate of time required per section.

Survey of user experience

The basic demographic information for all participants is displayed in Table 1. The student response rates were 26 (18%) and 26 (23%) for the UK and Grenada cohorts respectively. Eighteen veterinarians responded, 10 (58%) of whom were working primarily in clinical practice; the main reason for accessing the online resource was as part of a university postgraduate EBVM module.

Table 1: Demographic data from the online surveys of students and veterinarians

The Likert scale responses are shown in Figure 1. There were a few missing data points for some questions, but all participants were included. The majority replied positively to questions relating to the usefulness, ease of navigation, clarity of information, time for completion, and use of quizzes and references. Most participants (particularly veterinarians) agreed that they would use the resource again and/or recommend it to others. When asked if ‘the content was too much’ and also asked ‘if the content was too little’ most participants disagreed with both statements, suggesting that the volume of content may be about right.

Figure 1: Likert scale data from the online surveys of students and veterinarians

The analysis of the free-text comments identified recurring comments, similar for both students and veterinarians. The ease of navigation and clarity of the layout were appreciated: “Well presented. Logical. Stepwise”, as were the quizzes and examples. A recurring criticism was that some sections were a bit long, wordy and dry: “blocks of text need breaking up”. Suggestions for improvement included having more of the existing interactive content (e.g. quizzes), more case examples, and using visual content (e.g. tables, images) to replace text.

Interviews

Three major themes, each with sub-themes, were identified through the analysis of interview transcripts. The themes encompassed views about the resource and EBVM more broadly. Illustrative quotes have been included from both groups; practitioners (P) and educators (E).

1. Features of the ‘EBVM learning’ resource

This theme explored the strengths and weaknesses of the resource, with findings aligning closely with those from the user feedback and online survey.

Strengths

Comments in this sub-theme illustrated that the website was easy to use, and the content was at an appropriate level:

“Everything’s there that needs to be. All the guidance is really, really good” (P4)

The examples used throughout the resource added clinical relevance while the quizzes improved engagement and provided learners with feedback on their progress:

“It has good examples to really help explain the material” (E2)

“I thought having a quiz at the end of each section was quite useful, because it did tell you whether you’d picked up the right information or not” (P1)

Comments from both groups described that the resource had assisted learners’ skill development and had been helpful for practitioners:

[Ask, Acquire] “I found it really sped me up in terms of, now if I’ve got a question I don’t know the answer to, I’m quite good at finding out that information” (P2)

[Ask, Appraise] “most people with a background in first opinion practice… may not have had much exposure to evaluating evidence, or even things like creation of PICO questions… and the resource provided really good explanations” (P4)

 

Suggestions for improvement

Suggestions for improving the usability included reducing the blocks of text in some sections and increasing interactivity throughout, having a printable version and podcasts.

Additionally, interviewees mentioned the need to raise awareness as many did not know about the resource before being directed to it through a postgraduate course or as an educator:

“I don’t think it’s something I would necessarily have stumbled upon” (P5)

Educators noted that veterinary librarians appeared to be well acquainted with the resource, as it was disseminated proactively through their networks.

2. Embedding the resource in education

This theme drew together the experiences of interviewees as they embedded the resource in undergraduate teaching, and as learners or educators at the postgraduate level.

Teaching undergraduates

Educators used the resource in different ways. Some provided students with time in the curriculum to work through each section; others provided the resource as a reference tool; and some used the resource to prepare for their own teaching:

“we’re actually integrating it into a professional development course” (E2)

“it was useful. I certainly went back to look at things, to check things before I had meetings with the students” (P8)       

Educators were using a mixture of didactic teaching, small group work and self-directed learning. Integration throughout the curriculum provided scaffolding for the teaching of EBVM: introducing library skills in early years, progressing to PICO formation, evidence gathering and appraisal, and leading to students producing a Knowledge Summary:

“the EBVM course material is arranged vertically through the curriculum. So they build up knowledge and experience through the programme” (E4)

Those who required students to work through the resource suggested including a way to track student engagement “completion certificate” (E3) and incorporating tips for ways to use it in teaching: 

“examples of projects that could be done and ways of incorporating that into a programme” (E3)

Several challenges were mentioned including time pressure in the curriculum being a reason for not fully incorporating the resource. Another challenge was a lack of experience amongst staff, for example, to supervise EBVM related projects:

“there was a lot of variability based on whether they had the experience previously or not” (E2)

Additionally, when using the resource as a reference tool, comments indicated that it was not easy to search and find a relevant subsection. Another challenge related to how to engage all students and ensure the teaching is clinically relevant:

“some students do it well. Some students really struggle... I really thought they would get behind this idea…[i.e.] Knowledge Summaries. I think the more you make it clinically relevant the better” (E2)   

Both practitioners and educators commented on the need for repetition of the skills:

“feedback from the students is that they do forget things between the years” (E4)

“Instead of being something that you just get taught… it’s almost something that needs to be incorporated in everything…” (P5)

 

Postgraduate teaching

Many comments highlighted the usefulness of the resource although finding the time to study was sometimes an issue. A recurring theme across practitioners was the sense of “wishing I’d done it sooner” both as a veterinary student and as part of postgraduate study:

“The last module [of the certificate] was the free choice module… I thought it [EBVM] looked really interesting and I really regretted it that I didn’t do it as my first module. I think it would have helped so much, being taught that at the start of my certificate… because I found it so useful” (P2)

Additionally, access to evidence was much easier from a university environment whereas in practice this could be a significant barrier:  

“if they’re free or they’re something that you’ve got access to through doing the certificate module then that’s great. But actually, if you’re a general person in practice who doesn’t have that many professional memberships that would give them access to journals, it’s all very well reading the title of a paper, but then you can’t get access to it” (P1)

 

3. Promoting EBVM

Both challenges and motivators of EBVM within the context of the wider profession were identified.

Challenges 

One challenge related to EBVM being in its infancy – an emerging field. There was a sense of a lack of awareness of EBVM within the profession, partly because it is a relatively recent addition to curricula:

“I had no idea about all this, you know, [the] evidence-based veterinary medicine thing” (P3)

Consequently, for educators, a major challenge in promoting EBVM was linked to the lack of experience, their own and that of colleagues:

“there was a lot of variability [in staff] based on whether they had the experience previously or not. And that was one of the things we got back from the student feedback was there was a huge variability in the amount of mentoring they were getting, or the quality of mentoring they were getting” (E2)

Practitioners experienced several challenges for engaging with EBVM, centring around a perceived lack of support for EBVM compounded by a lack of time, lack of access to evidence, variable skills in scientific reading and writing, and difficulties in being able to implement and apply EBVM in clinical practice:

“I think time is the biggest one. And then access is the second one. And the third is knowing what to do with it” (P4)

“I think the thing that’s hard with ‘Apply in a clinical setting’ is that it’s never, [or] it’s rarely relevant. … you go, “Oh, no-one’s actually researched it in this population, so I’ll just have to use this population instead”” (P8)

 

Motivators for engagement

The reasons that practitioners did engage with the resource related to a desire to develop skills in themselves and others, to understand the connection with clinical practice, and career progression:

“it’s not something that necessarily I had all the tools to do effectively. So I did it…to give me a bit of confidence...a way to make it more clinically relevant to me” (P5)

For educators, including epidemiologists and clinicians, some of their motivation was to raise awareness in the profession by developing an interest in EBVM amongst undergraduate students.

Motivated leadership was considered a major influence on the uptake of EBVM in practice, which could be provided by a senior veterinarian or by practices promoting specific EBVM initiatives. In these situations, there was greater prioritisation of EBVM in the practice with, for example, protected time for further education courses and/or facilitating the learning of others:

“we had some of the vet nurse clinical coaches in and I went through how we teach EBVM and just tried to give them some ideas of how to incorporate it into practice” (E5)

“Since doing the EBVM module …. I’ve sat down with individual vets and taught them how to search” (P2)

There was a crucial role identified for leaders in the veterinary profession in taking responsibility for the generation of more evidence, which linked to an important point raised by practitioners about the challenges surrounding the lack of evidence (comparing EBVM with EBM):

“I could think of lots of questions I wanted to ask and then realised there was zero evidence” (P6)

“I remember coming across the Cochrane database… wouldn’t that be amazing if we had this for vets, so hopefully in time to come, we’ll get there, yes” (P2)

Other suggestions for promoting engagement in EBVM related to facilitating learning by creating a “community of learning” (P4), integrating regular clinical club meetings and journal clubs to practise skills, the use of an EBVM network to provide mentorship, online discussion forums, and greater publicity via e-news (e.g. InFocus) and social media platforms (e.g. Facebook).

Updating ‘EBVM Learning’

During the workshop, the project team developed a plan to update ‘EBVM Learning’ based on the research findings. Feedback had been positive about the structure of the resource and therefore the existing sections were maintained. Authors representing each section reviewed the content identifying areas that needed updating and checked the validity of hyperlinks. Improvements were proposed for each section by, for example, identifying areas that could be shortened and/or consolidated, breaking up or replacing long sections of text with other formats e.g. tables and, images, incorporating more clinically relevant examples and new interactive functionality available via the new courseware. As well as enhancing the overall learner experience such changes would be likely to benefit those for whom English is not a first language. Other features included the option to download the complete course and a completion certificate. Following the workshop, the groups continued to edit their sections coordinated by the project manager (author ES). The updated version of ‘EBVM Learning’ is available at https://learn.rcvsknowledge.org/ebvm-learning

One point of discussion during the workshop had focused on how to address the challenges practitioners encountered in engaging with EBVM while maintaining a resource that was already used in undergraduate and postgraduate teaching. A decision was made that a ‘one size fits all’ approach was not appropriate and therefore there was a need to create an additional version for practitioners in the future that was more slimline and practitioner-focused.

This article reports on experiences of using the online resource ‘EBVM Learning’ in veterinary curricula and practice. There has been increasing awareness of the importance of evidence-based practice and the challenges encountered in translating primary research into better quality healthcare (Heneghan et al., 2017; and Dean & Heneghan, 2019). Since the launch of ‘EBVM Learning’ in 2015, the resource has been successfully integrated into undergraduate and postgraduate courses and has the potential to provide a sound basis for using EBVM in practice. The further development of ‘EBVM Learning’ through this project will help to keep the resource relevant.

Interestingly, themes emerged through the interviews about EBVM generally not just about experiences using the resource, although the latter was the focus. Our findings indicated that the level of engagement in EBVM across the profession has been variable with motivated practitioners keen to upskill. Historically, the inclusion of EBVM teaching in undergraduate curricula had been limited and whilst recent research shows that awareness is improving, there is a need for more insight into the wider profession’s understanding of EBVM and how barriers to its practice can be overcome (Huntley et al., 2017). A future project being undertaken by the team, ‘EBVM III’, aims to further investigate the profession’s challenges to inform the development of an additional practitioner-focused resource.

‘EBVM Learning’ – how is it helpful?

‘EBVM Learning’ provides educators with a freely available online resource, authored by an international team with experience teaching EBVM. Feedback gathered via an online form, surveys and interviews consistently indicated that the resource was easy to use, had a structure that supported a logical and progressive development of skills, with relevant examples and quizzes helping to consolidate learning. Feedback also indicated areas for improvement which informed the update of ‘EBVM Learning’, including a reduction of large blocks of text and improving interactivity.

Recent literature suggests that successful teaching of EB(V)M involves embedding the concepts throughout a curriculum and ensuring learning is contextualised within clinical training to reinforce skill development (Maggio et al., 2015; and Janicke et al., 2020). A blended learning approach, where students were set relevant tasks alongside didactic teaching, has been shown to be effective at improving student attitudes to EBM (Ilic et al., 2015). The design of ‘EBVM Learning’ supports its integration within a veterinary curriculum with activities designed to support scaffolded active learning, when core knowledge is extended through problem-solving tasks (May & Silva-Fletcher, 2015), enabling students to build their knowledge and skillset alongside their clinical training. Further work is needed to determine the most effective means of providing contextualised EBVM training. Comments were raised by educators as to the variable quality of mentoring for such student activities, pointing to a need for more training.

In the wider profession there is an increasing appetite for EBVM (Huntley et al., 2017) and further understanding is required as to how EBVM training can be brought to a wider audience. In our research, some practitioners remarked on how useful ‘EBVM Learning’ had been for their everyday work, wishing they had done it sooner. Additionally, some of the perceived and real barriers felt by practitioners in their quest for integrating EBVM into clinical practice were highlighted when discussing EBVM more generally: time in a busy clinical environment, access to databases and journal articles, and difficulties applying EBVM in practice. This is potentially where evidence syntheses such as Knowledge Summaries, BestBETs and CATs can assist decision-making for busy clinicians. However, to be able to use them they need to be aware they exist and then understand the purpose and limitations, syntheses of evidence also need to be critiqued.

Time and busy workloads pose a barrier to practitioners using EB(V)M (Vandeweerd et al., 2012; and Zwolsman et al., 2013) and our interviewees echoed this point. In coming years, greater emphasis on EBVM is likely to result from the increasing proportion of graduates who will already be familiar with the concepts of EBVM through their training and as EBVM is embedded in postgraduate courses and practice cultures, such as clinical audit. Suggestions to increase EBVM uptake included the creation of EBVM podcasts, increasing practice-based journal clubs and support for scientific reading and writing skills.

Librarians (information specialists) are cited as an underused resource (Shurtz et al., 2017) and our data suggest that librarians have a vital role in dissemination of the resource and are often involved in teaching, particularly the EBVM steps Ask and Acquire. Recent initiatives, such as greater access to databases, journal articles and training from RCVS Knowledge Library and staff, have the potential to provide invaluable assistance to practitioners wanting to engage more with EBVM.

A recent manifesto for promoting evidence-based medicine (Dean & Heneghan, 2019) suggested several groups are required to work together to improve the evidence base, making evidence relevant and producing better guidelines. Interviewees commented that they did not know how to apply their newly found evidence in a clinical context and suggested facilitation from “communities of learning” such as online mentorship and discussion forums.

Although most respondents indicated that they would use the resource again and recommend it to others, Google Analytics revealed that only approximately 15% were returning visitors suggesting re-use was relatively low. Practitioners acknowledged time as an issue and students may not have returned because of other obligations, workload and exams. Furthermore, our research primarily gathered feedback on user experiences and additional work is required to determine any long-term benefits and the potential impact on the practice of EBVM.

Limitations of the study

The response rate through the online feedback form was low. The online surveys were conducted to gather additional feedback although the practitioner numbers remained low. This may reflect practitioners’ reported lack of awareness and time to complete the survey and/or to engage with the resource. Most of the practitioners who did respond were involved in an online postgraduate EBVM module, hence our research is relying on feedback from those who were already more actively engaged in EBVM and therefore may not represent the views or knowledge of the wider profession.

The small sample size for the interviews limited breadth of inclusion yet added depth to our dataset; the researchers recognised that data saturation was not reached and additional relevant information would be likely to emerge from a larger sample size. Educators selected for interview comprised a convenience sample, based on project team contacts, ensuring familiarity and use of the resource within a curriculum. Practitioners self-selected for inclusion in the interviews, selecting for those who are actively engaging in and promoting EBVM.

Additionally, our research primarily gathered feedback on user experiences and further work is required to determine any long-term benefits, learning outcomes and the potential impact on the practice of EBVM.

Conclusions

EBVM is gaining momentum and ‘EBVM Learning’ has the potential to make a valuable contribution to the training of the next generation of EBVM practitioners across the globe.

Our research indicated that ‘EBVM Learning’ in the current format was unsuitable for most busy practitioners, yet there is still a need to support those who do not have easy access to formal EBVM training and courses. Therefore, further work is being conducted in a follow-on project ‘EBVM III’ which aims to identify the needs of practitioners more specifically to enable EBVM to be better integrated into everyday practice. The research will inform the development of a slimline version of ‘EBVM Learning’ specifically for practitioners.

The authors have no conflicts of interest to declare.

The authors would like to thank those who participated in the surveys and the interviews. The projects to develop ‘EBVM Learning’ and evaluate and update the resource were both funded by grants from RCVS Knowledge.

1. Antman, E.M., Lau, J., Kupelnick, B., Mosteller, F. & Chalmers, T.C. (1992). A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts. Treatments for myocardial infarction. Journal of the American Medical Association. 268(2), 240–248. DOI: http://dx.doi.org/10.1001/jama.1992.03490020088036
2. Arlt, S.P., Haimerl, P. & Heuwieser, W. (2012). Training Evidence-Based Veterinary Medicine by Collaborative Development of Critically Appraised Topics. Journal of Veterinary of Medical Education. 39(2), 111–118. DOI: http://dx.doi.org/10.3138/jvme.1111.112R
3. Armitage-Chan, E. (2020). Best practice in supporting professional identity formation: Use of a professional reasoning framework. Journal of Veterinary Medical Education. 47(2), 125–136. DOI: http://dx.doi.org/10.3138/jvme.0218-019r
4. BestBETs for Vets. Available from: http://www.bestbetsforvets.org/
5. Braun, V. & Clarke, V. (2012). Thematic analysis. In: Cooper, H., Camic, P.M., Long, D.L., Panter, A.T., Rindskopf, D. & Sher, K.J. (Eds.), APA handbook of research methods in psychology. Vol. 2. Research designs: Quantitative, qualitative, neuropsychological, and biological (p. 57-71). American Psychological Association. DOI: http://dx.doi.org/10.1037/13620-004
6. Brennan, M.L., Arlt, S.P., Belshaw, Z., Buckley, L., Corah, L., Doit, H., Fajt, V.R., Grindlay, D.J.C., Moberly, H.K., Morrow, L.D., Stavisky, J. & White, C. (2020). Critically Appraised Topics (CATs) in Veterinary Medicine: Applying Evidence in Clinical Practice. Frontiers in Veterinary Science. 7. DOI: https://doi.org/10.3389/fvets.2020.00314
7. Dean, R., Brennan, M., Ewers, R., Hudson, C., Daly, J.M., Baillie, S., Eisler, M.C., Place, E.J., Brearley, J., Holmes, M., Handel, I., Shaw, D., McLauchlan, G., McBrearty, A., Cripps, P., Jones, P., Smith, R. & Verheyen, K. (2017). The challenge of teaching undergraduates evidence-based veterinary medicine. Veterinary Record. 181(11), 298–299. DOI: https://doi.org/10.1136/vr.j3441
8. Dean, R. & Heneghan, C. (2019). Do we need an evidence manifesto? Veterinary Record. 185(2), 58–59. DOI: https://doi.org/10.1136/vr.l4653
9. Doig, G.S. (2008). Evidence-based veterinary medicine: what it is, what it isn’t and how to do it. Australian Veterinary Journal. 81(7), 412–415. DOI: http://dx.doi.org/10.1111/j.1751-0813.2003.tb11551.x
10. Fajt, V.R., Brown, D. & Scott, M.M. (2009). Practicing the Skills of Evidence-Based Veterinary Medicine through Case-Based Pharmacology Rounds. Journal of Veterinary Medical Education. 36(2), 186–195. DOI: http://dx.doi.org/10.3138/jvme.36.2.186
11. Feetham, L. & Raffan, E. (2014). Better research reporting for better patient care. Veterinary Record. 175(21), 535–536. DOI: http://dx.doi.org/10.1136/vr.g7167
12. Glasziou, P. & Haynes, B. (2005). The paths from research to improved health outcomes. Evidence-Based Nursing. 8, 36–38. DOI: http://dx.doi.org/10.1136/ebn.8.2.36
13. Hardin, L.E. & Robertson, S. (2006). Learning evidence-based veterinary medicine through development of a critically appraised topic. Journal of Veterinary Medical Education. 33(3), 474–478. DOI: http://dx.doi.org/10.3138/jvme.33.3.474
14. Heneghan, C. & Badenoch, D. (2006) Evidence-based Medicine Toolkit, Second Edition, Blackwell Publishing, BMJ Journals. 71. DOI: https://doi.org/10.1002/9780470750605.ch12
15. Heneghan, C., Mahtani, K.R., Goldacre, B., Godlee, F., Macdonald, H. & Jarvies, D. (2017). Evidence based medicine manifesto for better healthcare. The BMJ. 357. DOI: http://dx.doi.org/10.1136/bmj.j2973
16. Holmes, M. & Cockcroft, P. (2004). Evidence-based veterinary medicine 1. Why is it important and what skills are needed? In Practice. 26(1), 28–33. DOI: http://dx.doi.org/10.1136/inpract.26.1.28
17. Huntley, S.J., Dean, R.S. & Brennan, M.L. (2017). The Awareness of the International Veterinary Profession of Evidence-Based Veterinary Medicine and Preferred Methods of Training. Veterinary Sciences. 4(1), 15. DOI: http://dx.doi.org/10.3390/vetsci4010015
18. Ilic, D., Nordin, R.B., Glasziou, P., Tilson, J.K. & Villanueva, E. (2015). A randomised controlled trial of a blended learning education intervention for teaching evidence-based medicine. BMC Medical Education. 15(39), 39–49. DOI: http://dx.doi.org/10.1186/s12909-015-0321-6
19. Janicke, H., Johnson, M.A., Baillie, S., Warman, S., Stone, D., Paparo, S. & Debnath, N.C. (2019). Toolbox: Creating the Next Generation of Evidence-Based Veterinary Practitioners and Researchers: What are the Options for Globally Diverse Veterinary Curricula? AAVMC. Available from: https://www.aavmc.org/programs/civme/toolkit/
20. Janicke, H., Johnson, M.A., Baillie, S., Warman, S., Stone, D., Paparo, S. & Debnath, N.C. (2020). Creating the Next Generation of Evidence-Based Veterinary Practitioners and Researchers: What are the Options for Globally Diverse Veterinary Curricula? Journal of Veterinary Medical Education.47(5), 647–658. DOI: https://doi.org/10.3138/jvme.2019-0098
21. Kasch, C., Haimerl, P., Heuwieser, W. & Arlt, S. (2017). Evaluation of a CAT Database and Expert Appraisal of CATs Developed by Students. Journal of Veterinary Medical Education. 44(4), 676–685. DOI: http://dx.doi.org/10.3138/jvme.0416-083R1
22. Kastelic, J.P. (2006). Critical evaluation of scientific articles and other sources of information: An introduction to evidence-based veterinary medicine. Theriogenology. 66(3), 534–542. DOI: http://dx.doi.org/10.1016/j.theriogenology.2006.04.017
23. Maggio, L.A., Cate, O.T., Irby, D.M. & O'Brien, B.C. (2015). Designing evidence-based medicine training to optimize the transfer of skills from the classroom to clinical practice: applying the four component instructional design model. Academic Medicine. 90(11), 1457–1461. DOI: http://dx.doi.org/10.1097/ACM.0000000000000769
24. May, S.A. & Silva-Fletcher, A. (2015). Scaffolded Active Learning: Nine Pedagogical Principles for Building a Modern Veterinary Curriculum. Journal of Veterinary Medical Education. 42(4), 332–339. DOI: http://dx.doi.org/10.3138/jvme.0415-063R
25. McColl, A., Smith, H., White, P. & Field, J. (1998). General practitioners’ perceptions of the route to evidence based medicine: A questionnaire survey. The BMJ. 316, 361–365. DOI: http://dx.doi.org/10.1136/bmj.316.7128.361
26. RCVS Code of Professional Conduct for Veterinary Surgeons. (2020). Available from: https://www.rcvs.org.uk/setting-standards/advice-and-guidance/code-of-professional-conduct-for-veterinary-surgeons/
27. RCVS Knowledge's Knowledge Summaries. Available from: https://knowledge.rcvs.org.uk/evidence-based-veterinary-medicine/knowledge-summaries
28. Shurtz, S., Fajt, V., Heyns, E.P., Norton, H.F. & Weingart, S. (2017). Teaching Evidence-Based Veterinary Medicine in the US and Canada. Journal of Veterinary Medical Education. 44(4), 660–668. DOI: http://dx.doi.org/10.3138/jvme.1215-199R
29. Steele, M., Crabb, N.P., Moore, L.J., Reyher, K.K., Baillie, S. & Eisler, M.C. (2013). Online tools for teaching evidence-based veterinary medicine. Journal of Veterinary Medical Education. 40(3), 272–277. DOI: http://dx.doi.org/10.3138/jvme.0113-010R1
30. Vandeweerd, J-M., Kirschvink, N., Clegg, P., Vandenput, S., Gustin, P. & Saegerman, C. (2012). Is evidence-based medicine so evident in veterinary research and practice? History, obstacles and perspectives. The Veterinary Journal. 191(1), 28–34. DOI: http://dx.doi.org/10.1016/j.tvjl.2011.04.013
31. Walker, A.J., Pretis, F., Powell-Smith, A. & Goldacre, B. (2019). Variation in responsiveness to warranted behaviour change among NHS clinicians: novel implementation of change detection methods in longitudinal prescribing data. The BMJ. 367. DOI: http://dx.doi.org/10.1136/bmj.l5205
32. Zwolsman, S.E., van Dijk, N., Te Pas, E. & Wieringa-de Waard, M. (2013). Barriers to the use of evidence-based medicine: knowledge and skills, attitude, and external factors. Perspectives on Medical Education.2(1), 4–13. DOI: http://dx.doi.org/10.1007/s40037-013-0039-2