key: cord-0747859-3snej6lu
authors: Maggio, Lauren A.; Costello, Joseph A.; Norton, Candace; Driessen, Erik W.; Artino Jr, Anthony R.
title: Knowledge syntheses in medical education: A bibliometric analysis
date: 2020-10-22
journal: Perspect Med Educ
DOI: 10.1007/s40037-020-00626-9
sha: 66c4fe4d480f753c9fa818e421eb9f8091d41bbd
doc_id: 747859
cord_uid: 3snej6lu

PURPOSE: This bibliometric analysis maps the landscape of knowledge syntheses in medical education. It provides scholars with a roadmap for understanding where the field has been and where it might go in the future, thereby informing research and educational practice. In particular, this analysis details the venues in which knowledge syntheses are published, the types of syntheses conducted, citation rates they produce, and altmetric attention they garner. METHOD: In 2020, the authors conducted a bibliometric analysis of knowledge syntheses published in 14 core medical education journals from 1999 to 2019. To characterize the studies, metadata were extracted from PubMed, Web of Science, Altmetrics Explorer, and Unpaywall. RESULTS: The authors analyzed 963 knowledge syntheses representing 3.1% of the total articles published (n = 30,597). On average, 45.9 knowledge syntheses were published annually (SD = 35.85, median = 33), and there was an overall 2620% increase in the number of knowledge syntheses published from 1999 to 2019. The journals each published, on average, a total of 68.8 knowledge syntheses (SD = 67.2, median = 41) with Medical Education publishing the most (n = 189; 19%). Twenty-one types of knowledge synthesis were identified, the most prevalent being systematic reviews (n = 341; 35.4%) and scoping reviews (n = 88; 9.1%). Knowledge syntheses were cited an average of 53.80 times (SD = 107.12, median = 19) and received a mean Altmetric Attention Score of 14.12 (SD = 37.59, median = 6). CONCLUSIONS: There has been considerable growth in knowledge syntheses in medical education over the past 20 years, contributing to medical education’s evidence base. Beyond this increase in volume, researchers have introduced methodological diversity in these publications, and the community has taken to social media to share knowledge syntheses. Implications for the field, including the impact of synthesis types and their relationship to knowledge translation, are discussed. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (10.1007/s40037-020-00626-9) contains supplementary material, which is available to authorized users.

its evidence base in an effort to empower medical educators to act as evidence-informed teachers [1] . In response to this call and inspired by the evidence-based medicine movement, the Association for Medical Education in Europe established the Best Evidence in Medical Education (BEME) Consortium, which called upon the medical education research community to create knowledge syntheses [2] . More than two decades later, medical education has seen a steady rise in knowledge syntheses; yet, there has been limited effort to understand this growth and map the landscape of these knowledge syntheses in medical education. This lack of understanding is problematic because without knowing where the field has been it is difficult to chart our future to ensure a robust evidence base moving forward.

Knowledge syntheses are defined as "the contextualization and integration of research findings of individual research studies within the larger body of knowledge on the topic" [3] . Gordon asserts that in medical education the creation of knowledge syntheses is as "vital as the production of novel primary research" [2] . Knowledge syntheses can provide medical educators with the means to move from reliance on eminence-and experience-based opinion or individual studies-to the utilization of resources that holistically consider and integrate the best available evidence [4] . Thus, it is unsurprising that interest in knowledge syntheses has resulted in the publication of several instructional guides on how to conduct knowledge syntheses in medical education [5] [6] [7] and the establishment of grant funding opportunities to support their creation [8] . What is more, systematic reviews are often a gateway for medical students starting their own lines of research [5] and are also essential for graduate students entering the field. To this end, BEME reports a 50% increase in the number of reviews undertaken by authors over a recent five-year period [9] .

Over the last decade, broad bibliometric reviews of the literature have been undertaken to identify trends in medical education publishing, [10] [11] [12] recognize the most cited articles in the field, [13] characterize author networks across international borders, [14] and describe associated social media attention [15, 16] . While these studies provide important insights into the overall landscape of the medical education literature, they fail to fully illuminate knowledge syntheses and their characteristics, which can have implications for researchers and educators who strive to act as evidence-informed practitioners. For example, medical educators are encouraged to publish and utilize knowledge syntheses to guide their practices (e.g., curricular design, student remediation), but without an awareness of the current state of knowledge syntheses both actions may be quite challenging for many and suboptimally undertaken for most.

More recently, members of our author team conducted a meta-synthesis of BEME knowledge synthe-ses, attempting to identify the degree to which the reviews were ready for translation into practice and to summarize their features [17] . Although valuable, this meta-synthesis examined a limited sample of knowledge syntheses (BEME reviews only) that underwent several intense review steps prior to publication; as such, the generalizability of this work is limited.

The bibliometric analysis reported here aims to map the broader landscape of knowledge syntheses in medical education to provide scholars with a roadmap for understanding both where the field has been and where it might go in the future. In particular, we examine the venues in which knowledge syntheses are published, the types of syntheses conducted, the citation rates they produce, and the altmetric attention they garner. In doing so, we hope to provide a detailed summary of the characteristics of medical education knowledge syntheses over the past 20 years to inform those who wish to conduct knowledge syntheses, those who are considering publishing such work, and those who provide funding for these efforts.

We conducted a bibliometric analysis of knowledge syntheses. A bibliometric analysis aims to use quantitative methods to describe characteristics of publications (e.g., journal articles) and their publication patterns, in order assess the current state of a field and provide insight into its overall structure [18] . Although we did not conduct a systematic review or meta-analysis, where appropriate, we report our methods in alignment with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses [19] to provide transparency and facilitate replication of our approach.

On March 26, 2020 we queried PubMed using a search strategy designed by two authors trained in information science. The purpose of this systematic search was to broadly capture knowledge syntheses using a combination of keywords, such as knowledge synthesis, literature review, meta-synthesis, and controlled vocabulary terms (Appendix of the Electronic Supplementary Material). We limited our search to the last 20 years (1999 to 2019) to correspond with the establishment of the BEME initiative.

We restricted our search to 14 journals that have been previously identified as core medical education journals [10, 17] On March 31, 2020 we queried Altmetric Explorer to obtain information on each article's Altmetric Attention Score, which is a web-based measure of an article's impact, with an emphasis on social media outlets as sources of data (e.g., whether the article was tweeted, saved to Facebook, covered by the news media, etc.). Also on this date, we queried the Unpaywall database, which tracks the accessibility of journal articles, to determine each article's open access status. Next, on April 10, 2020 we searched WoS to obtain additional metadata (e.g., number of times cited, funding information). To provide context in the broader field of medical education, we also downloaded citation data for all articles in the sample for the selected time period. All data were downloaded and managed in GoogleSheets [20] .

Knowledge syntheses were included if they met the above definition as written by Canadian Institute of Health Research [3] . We also included knowledge syntheses in which authors reviewed and synthesized the content and structure of research studies to contribute understanding on a topic (e.g., knowledge syntheses that defined a concept based on the literature or identified how studies were conducted). We excluded articles focused solely on the bibliometric properties of knowledge and articles focused on the mechanics of conducting a knowledge synthesis. We also excluded articles that solely analyzed documents that were not findings of research studies, such as syntheses of promotion and tenure guidelines or curricular documents. We also excluded articles that included a review of the literature, but were primarily focused on powering an additional research approach such as a Delphi study or survey.

To determine inclusion, two authors independently reviewed the titles and abstracts of all citations and then met via conference call to discuss coding discrepancies. A third author was available to facilitate any coding disagreements.

All data were extracted from each included article's citation metadata. For example, to determine review type (e.g., systematic review, scoping review, etc.), we relied solely on the text available in the title or abstract.

Descriptive statistics were calculated using Google-Sheets [20] and IBM SPSS Statistics [21] . We also calculated a Spearman's rank-order correlation to examine the relationship between the age of the knowledge syntheses and two of the collected variables.

We identified 2210 citations of which 963 met the inclusion criteria (Fig. 1 ). Included citations represented 3.1% of total articles published in the overall journal sample between 1999 and 2019 (n = 30,597).

Knowledge syntheses were published in all years studied (1999 to 2019) with 49% (n = 470) published between 2015 and 2019, and 84% (n = 805) published between 2009 and 2019 ( Fig. 2 ). On average, there were 45.9 knowledge syntheses published annually (SD = 35.85, median = 33). The fewest knowledge syntheses were published in 1999 (n = 5) and the most in 2019 (n = 136), representing an overall percent increase of 2620% across the 20-year period. In contrast, over the same time period, non-knowledge synthesis articles in these journals experienced an overall percent increase of 204%. 

We identified 21 types of knowledge syntheses, which accounted for 631 publications. Knowledge synthesis type was unspecified for 34% (n = 332) of the publications. Systematic reviews were the most prevalent knowledge synthesis type (n = 341; 35.4%) followed by scoping reviews (n = 88; 9.1%), systematic reviews with meta-analysis (n = 62, 6.4%), and narrative reviews (n = 50; 5.2%) (Tab. 2). Additionally, systematic reviews were the top knowledge synthesis type for each journal except for Clinical Teacher, which predominantly published narrative reviews. The following review types each represented less than 10 articles (<10% of the sample): qualitative, thematic, historical, guideline, meta-narrative, synthetic, rapid, meta-review, methodological, systematic scoping, umbrella, mapping, meta-synthesis, and meta-ethnography.

Funding was reported for 285 (30%) knowledge syntheses, with systematic reviews (n = 122; 43%) being the most commonly funded knowledge synthesis type. Forty (15%) of these knowledge syntheses reported more than one funder. The US National Institutes of Health was the most commonly reported funder (n = 41; 14%). However, a variety of funders were credited for their support including, but not limited to, national and regional agencies (e.g., the Canadian Institute of Health Research, [22] the European Commission; [23] ) professional societies (e.g., such as the Society of Directors of Research in Medical Education [8, 24] and Society of Hospitalist Pharmacists of Australia; [25] ) and foundations (e.g., the Arnold P. Gold Foundation and the Bill and Melinda Gates Foundation [26] ). Multiple authors listed funding assistance from their home institutions.

Knowledge syntheses, for which we were able to extract data from WoS (n = 847), generated a total of 45,566 citations. Citation counts ranged from 0 to 1629 and, on average, knowledge syntheses were cited 53.80 times (SD = 107.12, median = 19). For comparison, over the same time period, all other articles published in these journals (e.g., research reports, editorials, perspectives, commentaries, etc.) were cited an average of 15.30 times (SD = 39.22, median = 5).

Fifty-one knowledge syntheses had not been cited at the time of data extraction. Of these uncited knowledge syntheses, 41 (80%) were published in 2019. The most cited knowledge synthesis, a 2005 BEME systematic review on simulation, has received 1602 citations (Tab. 3). The top three cited knowledge synthesis types, of those that were specified, were critical reviews (mean = 80.35, SD = 149.37, median = 46), systematic reviews (mean = 57.64, SD = 134.06, median = 21), and systematic reviews with meta-analysis (mean = 52.76, SD = 90.06, median = 17) (Tab. 2).

There was a strong positive correlation between the age of the synthesis and its citation rate (Spearman's Knowledge syntheses in medical education: A bibliometric analysis ρ = 0.79, p < 0.01), indicating that older reviews were cited more often than more recent reviews.

The majority of knowledge syntheses received social media attention (i.e., altmetric attention; n = 771; 80%) generating 15,149 total mentions across 10 altmetric outlets (e.g., Twitter, Facebook There was a weak negative correlation between the age of the synthesis and its Altmetric Attention Score (Spearman's ρ = -0.24, p < 0.01), indicating that newer reviews attained a slightly higher altmetric score than older reviews.

The majority of social media attention was received on Twitter (n = 14,172 tweets), with activity generated by 6308 unique tweeters from 105 countries. The next most prevalent sources of altmetric attention were Facebook (n = 314 posts), the news media (n = 251 mentions), blogs (n = 200 posts), and policy documents (n = 99) (Tab. 4). Articles were not mentioned in seven additional outlets tracked by Altmetric, including patent documents, Pinterest, Weibo, LinkedIn, Q&A resources, Reddit, and syllabi.

The article with the highest altmetric score (806) was, "A systematic review of the effectiveness of flipped classrooms in medical education", published in 2017 in Medical Education [36] . This article was mentioned in 87 news media stories. This media coverage focused on the transition from in-person to online learning in the wake of the COVID-19 pandemic. For context, this knowledge synthesis has the second highest altmetric score of all articles published in the selected journals. The next highest knowledge synthesis had an altmetric score of 281 [37] . Additionally, journals in our sample had varied access policies. For example, Academic Medicine makes articles freely available on its website one year after the article appears in print.

Over the past two decades, there has been considerable growth in knowledge syntheses in medical education, including an increased variety of synthesis types and the emergence of scoping reviews as an important approach for synthesizing the medical education literature. Based on our analysis, we have provided a detailed summary of the evolution and characteristics of medical education knowledge syntheses, which we hope will encourage those individuals conducting, publishing, using, and funding knowledge syntheses to take an evidence-informed approach to those actions.

Our findings confirm a robust growth in the number of knowledge syntheses in medical education. While the overall growth of non-review journal articles has also been quite large (204% growth in our sample), the growth of knowledge synthesis has been more than 10 times that number (2620%). We predict this trend will continue, paralleling the growth of funding opportunities (e.g., SDRME's annual funding opportunity), [8] medical education graduate programs, [39] and related supporting initiatives such as BEME and the community's broader desire for synthesized evidence to keep current with research in the field and improve educational practice. 17 Additionally, when compared with other article types, knowledge syntheses are more highly cited, which makes them attractive to both authors and editors. For example, 80% of the top cited articles in Medical Education were review articles [40] .

In the current academic climate, the count of an individual's citations is usually factored into faculty promotion and tenure decisions [41, 42] . Also, from an editor's perspective, journals are typically judged by their impact factor, a citation-based metric. In both cases, this emphasis on citations may make it difficult for authors and editors to pass up the opportunity to write and publish a review article that stands to accrue up to three times as many citations as another article type. If our prediction of continued growth proves to be correct, then it is critical that knowledge syntheses are optimized for use. A recent meta-synthesis of BEME reviews indicated that while such reviews are quite valuable, there was much room for improvement in relation to accessibility and relevance [43] . Additionally, a survey of medical educators reported that 20% of participants never or rarely used knowledge syntheses to inform their educational decisions [44] . From our perspective, this is a lost opportunity for individuals and institutions to advance their evidenceinformed practice. In many ways, it is also a waste of resources if one considers the substantial time and energy required to complete a high-quality knowledge synthesis (one that might never get translated into practice) [7] . Future research should consider identifying potential barriers to uptake of knowledge syntheses specific to medical education and attempt to identify best practices for their creation, dissemination, and ultimate translation into practice. Moreover, our findings suggest that the medical education community is producing knowledge syntheses at an everincreasing rate. As such, we believe the community should consider ways to make use of these syntheses to better inform their educational practices.

We identified 21 types of knowledge syntheses. This variety aligns with research in other fields that has identified the presence of between 14 and 25 different types of knowledge syntheses [45, 46] . The large variety of knowledge synthesis types in medical education appears to be a somewhat recent trend, with only systematic and critical reviews dating back to 1999, and with scoping and realist reviews appearing in just the last decade. Gordon suggests that this proliferation of synthesis types relates to the shifting scope of investigators' research questions and aims [2] . He notes that investigators are not only focused on synthesizing literature to answer whether an educational approach works (e.g., by undertaking a systematic review), but also to discern what works, for whom, and in what contexts (e.g., by publishing a realist review) and to understand why it works and how (e.g., by conducting a meta-ethnography).

Our documentation of the increasing variety of knowledge syntheses highlights a need to ensure that consumers of these reviews are aware of the various types. In particulate, improved awareness is needed so that consumers can better understand the situations in which knowledge syntheses are best applied, as well as to better judge the quality of those reviews. For example, educators who are most familiar with the positivist tradition of systematic reviews may find themselves in strange territory when reading or attempting to execute a scoping review which, by design, is not designed to provide a "single answer" like a systematic review. Instead, scoping reviews provide readers with a map of the information landscape on a topic in an effort to help readers subjectively interpret the available evidence [47] . As Thomas et al. recently noted, "the philosophical stance scholars adopt during the execution of a scoping review, including the meaning they attribute to fundamental concepts such as knowledge and evidence, influences how they gather, analyze, and interpret information obtained from a heterogeneous body of literature" [44] . Therefore, anyone considering a knowledge synthesis must first understand the various types so that they can then identify the most appropriate synthesis type to answer their research question or satisfy their study aim [44] . To meet this need, several published articles compare and contrast knowledge syntheses types and provide practical advice for their conduct and use [45] [46] [47] [48] .

Knowledge translation is defined as a dynamic and iterative process that includes the synthesis, dissemination, exchange, and ethically sound application of knowledge to improve health, provide more effective health services and products, and strengthen the health care system [3] . In this review, we focus on the knowledge synthesis component of the knowledge translation process, which is critical. We propose, however, that increased accessibility for these articles, as indicated by higher altmetric scores for newer reviews, provides a glimpse into efforts to disseminate this knowledge and build a bridge to span the knowledge to practice gap. In this study, we observed that 80% of knowledge syntheses received altmetric attention with mentions predominantly on Twitter. In medical education, tweets have been observed to have a small positive effect on article page views of the tweeted articles [49] . In addition, we found that 69 knowledge syntheses were integrated into 99 policy documents, which suggests a translation of knowledge into practice. While these findings are promising, we encourage medical education researchers to consider other ways in which social media and policy integration can be leveraged as a knowledge translation tool for sharing knowledge syntheses.

This study should be considered in light of its limitations. In particular, although we conducted a comprehensive search for reviews, it is possible we inadvertently excluded one or more reviews. In addition, our search was limited to 14 core medical education journals, which did not include specialty-specific medical education journals (e.g., Academic Emergency Medicine) or medical education journals from specific world regions (e.g., African Journal of Health Professions Education). In addition, for citation data, we relied on WoS, which uses selective indexing, and so we were unable to obtain citation data for 125 (13%) knowledge syntheses. Future work might consider alternate citation sources such as Google Scholar, Dimensions, or an amalgamation of such services, with the caveat that each service has a slightly different approach for data collection, each of which having its strengths and weaknesses. Lastly, we did not assess the quality of the included knowledge syntheses and instead chose to focus on the growth and bibliometric characteristics of those publications. Therefore, we believe quality assessment may be an area ripe for future research.

Over the past 20 years, medical education researchers have begun to answer Harden's call to develop the field's evidence base [1] by publishing nearly a thousand knowledge syntheses across 14 core journals. In addition to a 2620% increase in knowledge synthesis volume, medical educators have responded by introducing the field to new review types and by utilizing methods, such as posting to social media, for communicating about and sharing knowledge syntheses. As we look to the future of medical education theory, research, and practice, we envision continued production of knowledge syntheses, which-if disseminated, interpreted, and applied effectively-could empower our community to act as evidence-informed educators.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Best evidence medical education

Are we talking the same paradigm? Considering methodological choices in health education systematic review

Knowledgetranslation

The best evidence medical education (BEME) collaboration: into thenextdecade

Twelve tips for medical students to conduct a systematic review

Twelve tips for undertaking a focused systematic review in medical education

Conducting systematic reviews in medical education: a stepwise approach

The Society of Directors of Research in Medical Education. SDRME sponsored scholarship

Best evidence medical education (BEME) review process redesigned for methodological rigor, shorter timeline and author support

50 years of publication in the field of medical education

Research topics and trends in medical education by social network analysis

A bibliometric analysis of evaluative medical education studies: characteristics and indexing accuracy

The top-cited articles in medical education: a bibliometric analysis

Productivity in medical education research: an examination of countries of origins

Beyond citation rates: a real-time impact analysis of health professions education research using altmetrics

Comparing alternative and traditional dissemination metrics in medical education

Examining the readiness of best evidence in medical education guides for Knowledge syntheses in medical education: A bibliometric analysis integration into educational practice: a meta-synthesis

Bibliometric indicators and analysis of research systems: methods and examples

Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement

IBM SPSS statistics for Windows, version 26.0. Armonk, NY: IBM Corp

Core competencies or a competent core? A scoping review and realist synthesis of invasive bedside procedural skills training in internal medicine

Training of adult psychiatrists and child and adolescent psychiatrists in Europe: a systematic review of training characteristics and transition from child/adolescent to adult mental health services

Standards for reporting qualitative research: a synthesis of recommendations

Enhancing education activities for health care trainees and professionals using audience response systems: a systematic review

Medical education in Sub-Saharan Africa: a literature review

Features and uses of high-fidelity medical simulations that lead to effective learning: a BEME systematic review

Systematic review of depression, anxiety, and other indicators of psychological distress among US and Canadian medical students

Reflection and reflective practice in health professions education: a systematic review

A critical reviewof simulation

A systematic review of faculty development initiatives designed to improve teaching effectiveness in medical education: BEME guide no

Toward a definition of competency-based education in medicine: a systematic review of published definitions

Does simulation-based medical education with deliberate practice yield better results than traditional clinical education? A meta-analytic comparative review of the evidence

Effectiveness of problem-based learning curricula: research and theory

A best evidence systematic review of interprofessional education: BEME guide no

A systematic review of the effectiveness of flipped classrooms in medical education

Flipped classroom improves student learning in health professions education: a meta-analysis

Evidence regarding the utility of multiple mini-interview (MMI) for selection to undergraduate health programs: a BEME systematic review: BEME guide no. 37

Health professions education graduate programs are a pathway to strengthening continuing professional development

In search of educational efficiency: 30 years of medical education's top-cited articles

Use of the journal impact factor in academic review, promotion, and tenure evaluations

Impact factor of medicaleducationjournalsandrecentlydevelopedindices: can any of them support academic promotion criteria?

exploring the relationship between altmetrics and traditional measuresofdisseminationinhealthprofessionseducation

Scoping reviews in health professions education: challenges, considerations and lessons learned about epistemology and methodology. Adv Health Sci Educ Theory Pract

A scoping review identifies multiple emerging knowledge synthesis methods, but few studies operationalise the method

A typology of reviews: an analysis of 14 review types and associated methodologies

Knowledge syntheses in medical education: demystifying scoping reviews

Varieties of integrative scholarship: why rules of evidence, criteria, and standards matter

To tweet or not to tweet, that is the question: a randomized trial of twitter effects in medical education