Received 12/11/2017 Review began 12/20/2017 Review ended 12/29/2017 Published 01/03/2018 © Copyright 2018 Quinn et al. This is an open access article distributed under the terms of the Creative Commons Attribution License CC-BY 3.0., which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Curated Collections for Educators: Five Key Papers on Evaluating Digital Scholarship Antonia Quinn , Teresa Chan , Christopher Sampson , Catherine Grossman , Christine Butts , John Casey , Holly Caretta-Weyer , Michael Gottlieb 1. Department of Emergency Medicine, SUNY Downstate College of Medicine 2. Faculty of Health Sciences, Department of Medicine, Division of Emergency Medicine, McMaster University 3. Emergency Medicine, University of Missouri Columbia 4. Pulmonary and Critical Care Medicine, Virginia Commonwealth University Health Systems 5. Section of Emergency Medicine, Louisiana State University Health Sciences 6. Department of Emergency Medicine, Ohiohealth Doctors Hospital 7. Department of Emergency Medicine, Oregon Health & Science University 8. Department of Emergency Medicine, Rush University Medical Center  Corresponding author: Antonia Quinn, antonia424@gmail.com Disclosures can be found in Additional Information at the end of the article Abstract Traditionally, scholarship that was recognized for promotion and tenure consisted of clinical research, bench research, and grant funding. Recent trends have allowed for differing approaches to scholarship, including digital publication. As increasing numbers of trainees and faculty turn to online educational resources, it is imperative to critically evaluate these resources. This article summarizes five key papers that address the appraisal of digital scholarship and describes their relevance to junior clinician educators and faculty developers. In May 2017, the Academic Life in Emergency Medicine Faculty Incubator program focused on the topic of digital scholarship, providing and discussing papers relevant to the topic. We augmented this list of papers with further suggestions by guest experts and by an open call via Twitter for other important papers. Through this process, we created a list of 38 papers in total on the topic of evaluating digital scholarship. In order to determine which of these papers best describe how to evaluate digital scholarship, the authorship group assessed the papers using a modified Delphi approach to build consensus. In this paper we present the five most highly rated papers from our process about evaluating digital scholarship. We summarize each paper and discuss its specific relevance to junior faculty members and to faculty developers. These papers provide a framework for assessing the quality of digital scholarship, so that junior faculty can recommend high-quality educational resources to their trainees. These papers help guide educators on how to produce high quality digital scholarship and maximize recognition and credit in respect to receiving promotion and tenure. Categories: Medical Education, Quality Improvement, Other Keywords: curated collection, digital scholarship evaluation, academic promotion, medical education Introduction And Background The manifestation of scholarship is changing. Whereas traditionally, academia only recognized clinical research, benchwork, peer reviewed publications, and grant funding as markers of success, other forms of scholarship were defined in the late 20th century by Ernest L. Boyer. Dr. Boyer, through his seminal work, Scholarship Reconsidered, created a major paradigm shift to 1 2 3 4 5 6 7 8 Open Access Review Article DOI: 10.7759/cureus.2021 How to cite this article Quinn A, Chan T, Sampson C, et al. (January 03, 2018) Curated Collections for Educators: Five Key Papers on Evaluating Digital Scholarship. Cureus 10(1): e2021. DOI 10.7759/cureus.2021 https://www.cureus.com/users/38270-antonia-quinn https://www.cureus.com/users/23655-teresa-chan https://www.cureus.com/users/37821-christopher-sampson https://www.cureus.com/users/54765-catherine-grossman https://www.cureus.com/users/54768-christine-butts https://www.cureus.com/users/54766-john-casey https://www.cureus.com/users/54767-holly-caretta-weyer https://www.cureus.com/users/35471-michael-gottlieb include other forms of scholarship, namely scholarship of integration, scholarship of application/engagement, and scholarship of teaching and learning [1]. With the advent of the digital age, however, disruptive technologies like social media are now pushing us even closer towards yet another paradigm shift. In the age of the printing press, publishing and distributing ground-breaking new ideas were controlled by publishing houses. In today’s age, the Web 2.0 applications now allow for ease of publication at an unprecedented level. This has led to a veritable explosion of certain types of digital products, including blogs, podcasts, and tweet chats [2-4]. However, recent trends have allowed for differing approaches to scholarship, including digital publication [5-6]. While there is a compelling case that these new forms of scholarship are actually no different from prior technologies (e.g. Is a blog post not just the modern interpretation of a text book? Isn’t a podcast just a more easily distributed recording of a lecture?), their ease of digital publication makes quality surveillance imperative [7]. As increasing numbers of trainees and faculty turn to online educational resources, it is imperative to evaluate digital resources for quality. Trainees and practicing physicians alike are inconsistent in gestalt recommendations of online educational content [8-9]. In order for academia to accept these disruptive forms of scholarship, it is imperative these publications be scrutinized and rigorously assessed for content and quality in the same way as reserved for traditional scholarship. In 2017, the Faculty Incubator was created by the Academic Life in Emergency Medicine (ALiEM) team to create a community of practice (CoP) for early career educators. In this online forum, members of this CoP discuss and debate topics relevant to modern clinician educators. To that end, we created a one month module focused on learning technologies, with digital scholarship as a prominent point of discussion. Since there is an emerging literature base on this important topic, our team sought to identify key literature about how to evaluate the quality of digital scholarship. This paper utilized a consensus-based review process to determine which papers may assist junior educators who wish to learn more about how they can evaluate their digital scholarship. These papers provide a framework for assessing digital scholarship for quality so that junior faculty can recommend quality educational resources to their trainees. Furthermore, by highlighting the parts of digital scholarship that produce high quality, future writers of online content may be able to better shape their work in order to maximize credit when applying for promotion and tenure. Review Methods In the first month of the ALiEM 2017-2018 Faculty Incubator, we discussed the topic of digital scholarship. We monitored the proceedings of this CoP from May 1st-31st, 2017. The monitored online discussions involved both junior faculty members and senior faculty mentors. While discussions occurred, we gathered the titles of papers that were cited, shared, and recommended within our online discussion forum and compiled these into a list. This list was then expanded via two other methods. First, we consulted two content experts, Brent Thoma and Jonathan Sherbino, for suggestions. Next, we posted a call for important papers regarding the evaluation of digital scholarship on Twitter. We ‘tweeted’ requests to have participants of the #FOAMed and #MedEd online communities provide suggestions for important papers on the topic of peer review. 2018 Quinn et al. Cureus 10(1): e2021. DOI 10.7759/cureus.2021 2 of 12 Once the augmented list was completed, we conducted a three-round voting process inspired by the Delphi methodology. Similar methods were used on our previous papers to build consensus on the most important papers to feature [10-13] [14-16]. Readers will note that this was not traditional Delphi methodology because our raters included novices (i.e. junior faculty members, participants in the Faculty Incubator), as well as experienced/expert medical educators (i.e. clinician educators, all of whom have published >10 peer reviewed publications, who serve as mentors and facilitators of the ALiEM Faculty Incubator). Rather than only including experts, we intentionally involved junior educators to ensure we selected papers that would be of use to a spectrum of educators throughout their careers. Results Our Faculty Incubator discussions yielded 37 papers, and the expert recommendations and social media calls yielded a total of one additional article. Our process provides a rank-order listing of all these papers in the order of perceived relevance, from the most to the least relevant. Our top five papers are expanded upon below. After our third round of voting, we had a tie for the final article leading to a fourth round of voting. We included the article not chosen as the fifth paper as an 'honorable mention'. Our ratings of all 38 papers are listed in Table 1, along with their full citations. Citation ROUND 1. Initial mean scores (SD). Max score 7 ROUND 2. % of raters that endorsed this paper ROUND 3. % of raters that endorsed this paper ROUND 4. Tie break round Top 5 papers Cabrera, et al. [6] 6.6(0.52) 100% 87.5% 1 Thoma, et al. [17] 5.4(1.30) 87.5% 87.5% 1 Chan TM, et al. [18] 6.3(0.89) 87.5% 75.0% 2 Colmers, et al. [19] 5.8(1.39) 87.5% 75.0% 2 Sherbino, et al. [5] 6.0(1.41) 75% 50.0% 62.5% 5 Gottlieb, et al. [20] 5.8 (0.71) 87.5% 50.0% 37.5% Honorable Mention Thoma B, et al. [21] 5.8 (1.28) 50% 0% Chan T, et al. [22] 5.5(1.20) 50% 0% Frank JR, et al. [23] 5.5(1.69) 50% 12.5% Cameron CB, et al. [24] 5.3(1.49) 75% 12.5% 2018 Quinn et al. Cureus 10(1): e2021. DOI 10.7759/cureus.2021 3 of 12 Lin M, et al. [25] 5.3 (1.49) 75% 12.5% Krishnan K, et al. [8] 5.1(1.49) 62.5% 37.5% Chan TM, et al. [26] 5.0(1.69) 50% 0% Thoma B, et al. [27] 5.0(2.07) 25% 0% Lin M, et al. [28] 4.9(1.13) 62.5% 0% Flynn L, et al. [29] 4.9(1.25) 37.5% 0% Sherbino J, et al. [30] 4.9(1.64) 37.5% 0% Boyer EL. [1] 4.6 (2.33) 50% 0% Sterling M, et al. [31] 4.6 (1.19) 25% 0% Purdy E, et al. [32] 4.5 (0.76) 12.5% 0% Paterson QS, et al. [33] 4.4 (1.69) 12.5% 0% Eysenbach G [34] 4.4 (1.51) 50% 0% Roland D, et al. [35] 4.1 (1.55) 0 0% Lumba- Brown A, et al. [36] 4.1 (0.83) 12.5% 0% Jordan J, et al. [37] 4.0 (1.93) 0 0% Boulos MN, et al. [38] 4.0 (1.31) 12.5% 0% Sutherland S, et al. [39] 3.9 (1.64) 12.5% 0% Diug B, et al. [40] 3.9 (0.83) 0 0% Lin M, et al. 3.8 (0.89) 12.5 0% 2018 Quinn et al. Cureus 10(1): e2021. DOI 10.7759/cureus.2021 4 of 12 [41] Raine T, et al. [42] 3.8 (1.83) 0 0% Riddell J, et al. [43] 3.8 (0.89) 0 0% Thoma B, et al. [44] 3.6 (1.60) 0 0% Ke Q, et al. [45] 3.4 (2.13) 0 0% Hillman T, et al. [46] 3.4 (1.60) 0 0% Pronk NP, et al. [47] 3.0 (2.39) 0 0% Flynn S, et al. [44] 3.0 (1.31) 0 0% Solomon DJ, et al. [48] 2.8 (1.58) 0 0% Langdorf MI, et al. [49] 2.6 (1.77) 0 0% TABLE 1: The complete list of 38 literature items collected by the authorship team Discussion The following is a list of papers that our group has determined to be of interest and relevance to junior faculty members and more senior colleagues who may be in charge of faculty development. The accompanying commentaries are meant to explain the relevance of these papers to junior faculty members and also highlight considerations for senior faculty members when using these works for faculty development workshops or sessions. 1. Thoma B, Chan T, Benitez J, Lin M. Educational Scholarship in the Digital Age: A Scoping Review and Analysis of Scholarly Products, The Winnower 1: e141827. 77297, 2014, DOI: 10.15200/winn [17]. Summary: This article applies the more widely accepted Boyer model of scholarship to classify digital works [1]. A literature review and coding schema allow the different forms of digital media presented to be reclassified into Boyer’s four subtypes of scholarship: teaching, integration, application, and discovery [1]. Various types of digital scholarship were also reviewed and summarized. Over 85% of digital scholarship was mapped to the teaching subtype, although some mapped to more than one subtype. The weakness most associated with digital scholarship was ensuring rigorous scrutiny for quality, which is presumed to occur with traditional scholarship. A framework, such as the one proposed, is valuable as the development 2018 Quinn et al. Cureus 10(1): e2021. DOI 10.7759/cureus.2021 5 of 12 and distribution of digital scholarship continues to increase and these works are integrated into academic portfolios. Relevance to junior faculty members: Associating various types of digital scholarship with a well-established and accepted education framework allows junior faculty to strengthen their academic portfolio. Demonstrating how their digital works fit into the established subtypes of Boyer’s scholarship framework can assist junior faculty when explaining their work to a promotion and tenure committee. This may be particularly valuable by normalizing digital scholarship into terms that promotion and tenure committees may be more familiar with. Relevance to faculty developers: Thoma and colleagues go beyond demonstrating how digital scholarship can fit into the framework of Boyer’s four subtypes of scholarship to make a case for developing tools that can examine the impact and quality of these digital products [17]. The growing number of digital products identified in the literature emphasizes the increased use of digital technology for educational scholarship with most being categorized into Boyer’s scholarship of teaching and learning [1]. The authors concluded that there is no compelling evidence that digital products are more effective but, given their widespread use, further research is needed in order to assess their quality. 2. Cabrera D, Vartabedian BS, Spinner RJ, Jordan BL, Aase LA, Timimi FK. More Than Likes and Tweets: Creating Social Media Portfolios for Academic Promotion and Tenure. Journal of Graduate Medical Education. 2017 Aug;9(4):421-5 [6]. Summary: More education faculty are focusing their scholarly efforts on the creation, curation, and dissemination of free, open-access medical education. In this paper, Cabrera and colleagues propose a framework to incorporate social media scholarship into current academic promotion and tenure systems. They provide recommendations of best practices for institutions in implementing a system for recognizing social media and digital scholarship for academic promotion. This includes the development of focused guidelines and strategic priorities for social media use, the development of a clear impact grid to identify types of social media activities considered for academic tenure, and methods by which the quality and impact of scholarship can be measured. Suggested measures of impact include altmetrics such as page views, peer review of the work, and objective measures of quality such as those laid out by Sherbino and colleagues [5]. An example impact grid is also included in the paper. Faculty scholars are encouraged to create and maintain a social media scholarship portfolio, much the same as they would an educator’s portfolio for their work in clinical and didactic teaching, curriculum design, and formal mentorship. The description of one’s scholarly work in social media should remain consistent with Glassick’s framework and include clear goals, adequate preparation, appropriate methods, significant results, effective presentation, and reflective technique [50]. Relevance to junior faculty members: This paper lays out a clear framework for presenting social media and digital scholarship for academic promotion and tenure. Junior faculty should develop and curate a social media scholarship portfolio, which includes a statement of social media scholarship philosophy, academic niche, audience, objectives, and platforms. All aspects of social medial scholarship should be presented and include original content creation, curation of content, community management, platform administration, data analysis, and a durable record of scholarship (eg, permalinks, cached content). A clear description of how social media scholarship aligns with the junior faculty member’s overall career development plan and program of scholarship will provide a cohesive picture for the promotion and tenure committee at the time of review. Relevance to faculty developers: This is an essential paper for educators who engage in social 2018 Quinn et al. Cureus 10(1): e2021. DOI 10.7759/cureus.2021 6 of 12 media to receive proper recognition when applying for promotion and tenure. The authors outline a list of potential social media scholarship avenues, as well as their relative impact. It also provides a guideline for developing local institutional criteria for social media-based scholarship in addition to a formal definition based upon the paper by Sherbino and colleagues [5]. This resource may be provided to one’s academic appointment committee when applying for promotion, but would be better utilized in advance to help develop local criteria. 3. Colmers IN, Paterson QS, Lin M, Thoma B, Chan TM. The Quality Checklists for Health Professions Blogs and Podcasts, The Winnower 2: e144720. 08769, 2015, DOI: 10.15200/winn. 144720.08769 Colmers, et al. This article is distributed under the terms of the Creative Commons Attribution.;4 [19]. Summary: Blogs and podcasts are highly popular in medical education and have grown exponentially without significant oversight or quality control. This paper seeks to create an easy-to-use checklist for educators to evaluate and identify the educational value of a blog or podcast. To do so, the authors used a three-step process. Initially, they performed a literature review with the goal of extracting quality indicators of secondary educational resources. These findings then underwent qualitative analysis to isolate those that would be applicable to blogs or podcasts, yielding a total of 151 indicators. To further refine these quality indicators, the authors then surveyed a large and varied group of expert producers of online medical education content to determine which of this initial group of indicators was most important. This analysis yielded a more manageable 14 indicators for blogs and 26 for podcasts. To avoid bias in surveying only experts in content development, the authors also surveyed general medical education experts. From the initial group of 151 quality indicators, the general medical experts identified three indicators for blogs, one for podcasts, and nine for both blogs and podcasts. Once these surveys were completed, the authors compiled the findings from both the expert producers of online content and the general medical education experts to form simplified checklists for both blogs and podcasts. The checklists are similarly divided into three sections (ie, credibility, content, and design) and contain a series of questions that can be answered yes, no, or unclear. Relevance to junior faculty members: Many educators and students enjoy blogs and podcasts due to their entertainment value, brevity, and style. They offer a welcome break from more traditional methods such as textbooks or lectures. However, there is a tendency to overlook the content and quality of a blog or podcast when blinded by other factors, such as the aforementioned entertainment value. This checklist offers junior faculty, who may have limited experience in evaluating online educational materials, a simple and rapid method to assess the worth of a blog or podcast as a teaching tool. By focusing on key aspects of the blog or podcast, this checklist ensures that the junior faculty will be able to accurately assess its quality. Relevance to faculty developers: This paper is invaluable for faculty development for both consumers and producers of digital scholarship. Assessing the quality of podcasts and blogs based on personal gestalt has been shown to be unreliable [8]. This paper provides an easily usable checklist for developers, editors, and end-users of blogs and podcasts. This checklist helps guide the producer to focus their efforts to produce a quality product. Editors can utilize rubric to provide quality feedback to writers to optimize the peer review process, advancing the overall quality of digital scholarship. Educators can guide learners to critically appraise blogs and podcasts prior to their use. 4. Chan TM, Grock A, Paddock M, Kulasegaram K, Yarris LM, Lin M. Examining Reliability and Validity of an Online Score (ALiEM AIR) for Rating Free Open Access Medical Education Resources. Annals of Emergency Medicine. 2016 Dec 31;68(6):729-35 [18]. 2018 Quinn et al. Cureus 10(1): e2021. DOI 10.7759/cureus.2021 7 of 12 Summary: This article compared a global rating gestalt-based Likert scale versus a five-domain rating scale (ALiEM approved instructional resources score (AIR Score)) to evaluate free open access medical education (FOAM) blog posts. The AIR Score included ratings of: Best Evidence in Emergency Medicine rater scale, content accuracy, educational utility, having an evidence- based medicine construct, and provision of author and literature references. The AIR score had moderate congruence with the gestalt evaluations and required nine raters to achieve adequate interrater reliability per article. Relevance to junior faculty members: This article is important to junior faculty as it provides an initial scaffold for evaluating existing resources, as well as a framework for creating quality online FOAM resources. As evaluation tools for FOAM resources evolve, it will be important to keep up with literature to adapt publications as needed to stay within best practices. Relevance to faculty developers: This paper is of two-fold importance for those interested in faculty development. First, it is a good example of applying measurement science to an educationally-relevant scoring system. This can open up a discussion around the methodology used by the authors to generate ‘validity evidence’ for a particular scoring system, which can be a nice way to compare and contrast with known diagnostic test validation methods. Moreover, it is an example of how you can generate multiple wins from a single educational innovation [20]. Notably, this group has authored a paper describing their innovation [28] and subsequently analyzed the data they acquired whilst running their innovation processes to generate validity evidence around the scoring system they created. 5. Sherbino J, Arora VM, Van Melle E, Rogers R, Frank JR, Holmboe ES. Criteria for Social Media- based Scholarship in Health Professions Education. Postgraduate Medical Journal. 2015 Oct 1;91(1080):551-5 [5]. Summary: Sherbino and colleagues recognized that substantial questions were present on how social media-based scholarship was viewed through traditional scholarship assessment methods [5]. Using a consensus approach, they were able to define the criteria for social media- based scholarship. Following a facilitated session of both in-person and virtual health professions educators, a consensus statement was produced. This statement required that social media-based scholarship in health professions must: be original, advance the field of health professions education, be archived and disseminated, and provide the community the ability to comment on and provide feedback in a transparent fashion. A process was also defined that listed the standards of criteria for authorship. With respect to the impact on education, the authors agree that evidence of a transparent critical appraisal is required. Also, the innovations must have the potential to impact health professions in a rapid or broad fashion. Ease of accessibility is also a concern. Alternative metrics were suggested as a way to demonstrate the dissemination and impact of social media-based scholarship. They concluded that the health professions education community should champion social media-based scholarship as a legitimate educational pursuit. Relevance to junior faculty members: Given the use of social media-based educational scholarship among junior faculty, it is important to know criteria that can be applied in order to make one’s scholarship more robust and of high quality. This enables a junior faculty member’s productivity to fit criteria that can help them when viewed by evaluators for academic advancement. Relevance to faculty developers: With increasing use of social media within academic medicine, it is essential that junior faculty understand what constitutes scholarship. This paper builds upon the work of Glassick to define scholarship with social media and is complementary to the above paper by Cabrera [6, 50]. This may be a valuable resource to ensure that one’s efforts meet 2018 Quinn et al. Cureus 10(1): e2021. DOI 10.7759/cureus.2021 8 of 12 the criteria for scholarship in order to properly categorize and support projects when applying for promotion and tenure. It is equally important to ensure that faculty understand what does not constitute scholarship, so as to realign one’s efforts with a project that meets the criteria and may maximize their efforts [20]. Honorable Mention: Gottlieb M, Chan TM, Sherbino J, Yarris L. Multiple Wins: Embracing Technology to Increase Eff iciency and Maximize Efforts. AEM Education and Training [20]. This paper was highly-ranked by our team, but ultimately not selected because it does not bear weight on how to actually evaluate digital scholarship. Instead, we summarize it here because we feel that it is a valuable “how-to” article that may guide how junior faculty members align projects and harness the power of technology to efficiently increase their scholarly efforts [20]. While it does not help those seeking evaluative tools for judging the quality of digital scholarship, it describes well how one might integrate digital scholarship into their practice. Limitations Many of the final articles share one or more of the authorship group of this manuscript. These same authors served as senior mentors and content experts for the month covering digital scholarship for the Aliem Faculty Incubator and therefore were essential in the writing of this manuscript. As with more systematic reviews, individuals with a good grasp of key literature and content experts, such as active researchers in a particular field, are best served to identify papers. We attempted to minimize selection bias by “tweeting” requests for additional articles covering the evaluation of digital scholarship as well as multiple rounds of the modified Delphi. The chosen articles not only had to cover the topic of evaluating digital scholarship, they had to be beneficial for junior faculty. Articles not chosen via the modified Delphi methodology may not have been appropriate for junior faculty. Conclusions This paper describes five key articles on the evaluation of digital scholarship. We believe this resource will be valuable for clinician educators to evaluate digital scholarship, while also providing guidance for producers of digital content to create a robust product that meets the definition of scholarship. Additional Information Disclosures Conf licts of interest: In compliance with the ICMJE uniform disclosure form, all authors declare the following: Payment/services info: All authors have declared that no financial support was received from any organization for the submitted work. Financial relationships: All authors have declared that they have no financial relationships at present or within the previous three years with any organizations that might have an interest in the submitted work. Other relationships: All authors have declared that there are no other relationships or activities that could appear to have influenced the submitted work. Acknowledgements The authors would like to thank Dr. Brent Thoma and Dr. Jonathan Sherbino for their input as content experts on the topic of digital scholarship and for providing references on the topic. The authors would like to acknowledge Dr. Michelle Lin and the 2017-18 Academic Life in 2018 Quinn et al. Cureus 10(1): e2021. DOI 10.7759/cureus.2021 9 of 12 Emergency Medicine (ALiEM) Faculty Incubator participants and mentors for facilitating the drafting and submission of this manuscript. References 1. Boyer E: Scholarship Reconsidered: Priorities of the Professoriate. Princeton University Press, Lawrenceville, NJ; 1990. 2. Cadogan M, Thoma B, Chan TM, Lin M: Free Open Access Meducation (FOAM): the rise of emergency medicine and critical care blogs and podcasts (2002–2013). Emerg Med J. 2014, 31:76–7. 10.1136/emermed-2013-203502 3. Chan TM, Thoma B, Radecki R, Topf J, Woo HH, Kao LS, et al.: Ten steps for setting up an online journal club. J Contin Educ Health Prof. 2015, 35:148–54. 4. Lin M, Joshi N, Hayes BD, Chan TM: Accelerating knowledge translation: reflections from the online ALiEM-Annals Global Emergency Medicine Journal Club experience. Ann Emerg Med. 2017, 65:469–74. 10.1016/j.annemergmed.2016.11.010 5. Sherbino J, Arora VM, Van Melle E, et al.: Criteria for social media-based scholarship in health professions education. Postgrad Med J. 2015, 91:551–5. 10.1136/postgradmedj-2015-133300 6. Cabrera D, Vartabedian BS, Spinner RJ, et al.: More than likes and tweets: creating social media portfolios for academic promotion and tenure. J Grad Med Educ. 2017, 9:421–5. 10.4300/JGME-D-17-00171.1 7. Roland D, Trueger S, Thoma B, et al.: FOAM helps to bridge the knowledge translation gap . Emerg Physician Int. 2016, 18:34. Accessed: November 8, 2017: http://www.epijournal.com/articles/246/foam-helps-to-bridge-the-knowledge-translation- gap. 8. Krishnan K, Thoma B, Trueger NS, et al.: Gestalt assessment of online educational resources may not be sufficiently reliable and consistent. Perspect Med Educ. 2017, 6:91–8. 10.1007/s40037-017-0343-3 9. Thoma B, Sebok-Syer SS, Krishnan K, et al.: Individual gestalt is unreliable for the evaluation of quality in medical education blogs: a METRIQ study. Ann Emerg Med. 2017, 70:394–401. 10.1016/j.annemergmed.2016.12.025 10. King A, Boysen-Osborn M, Cooney R, et al.: Curated collection for educators: five key papers about the flipped classroom methodology. Cureus. 2017, 9:e1801. 10.7759/cureus.1801 11. Thoma B, Gottlieb M, Boysen-Osborn M, et al.: Curated collections for educators: key papers about program evaluation. Cureus. 2017, 9:1–10. 10.7759/cureus.1224 12. Gottlieb M, Chan T, Fredette J, et al.: Academic primer series: five key papers about study designs in medical education. West J Emerg Med. 2017, 18:705–12. 10.5811/westjem.2017.4.33906 13. Cooney R, Chan TM, Gottlieb M, et al.: Academic primer series: key papers about competency- based medical education. West J Emerg Med. 2017, 18:713–20. 10.5811/westjem.2017.3.33409 14. Yarris L, Gottlieb M, Scott K, et al.: Academic primer series: key papers about peer review . West J Emerg Med. 2017, 18:721–8. 10.5811/westjem.2017.2.33430 15. Boysen-Osborn M, Cooney R, Gottlieb M, et al.: Academic primer series: key papers about teaching with technology. West J Emerg Med. 2017, 18:729–36. 10.5811/westjem.2017.2.33076 16. Chan T, Gottlieb M, Quinn A, et al.: Academic primer series: five key papers for consulting clinician educators. West J Emerg Med. 2017, 18:311–7. 10.5811/westjem.2016.11.32613 17. Thoma B, Chan T, Benitez J, Lin M: Educational scholarship in the digital age: a scoping review and analysis of scholarly products. The Winnower. 2014, 1:1-12. 10.15200/winn.141827.77297 18. Chan TMY, Grock A, Paddock M, et al.: Examining reliability and validity of an online score (ALiEM AIR) for rating Free Open Access Medical Education resources. Ann Emerg Med. 2016, 68:729–35. 10.1016/j.annemergmed.2016.02.018 19. Colmers IN, Paterson QS, Lin M, Thoma B, Chan TM: The quality checklists for health professions blogs and podcasts. The Winnower. 2015, 2:1–7. 10.15200/winn.144720.08769 20. Gottlieb M, Chan TM, Sherbino J, Yarris L: Multiple wins: embracing technology to increase efficiency and maximize efforts. AEM Educ Train. 2017, 1:185–90. 10.1002/aet2.10029 21. Thoma B, Sanders J, Lin M, et al.: The social media index: measuring the impact of emergency medicine and critical care websites. West J Emerg Med. 2015, 16:242–9. 2018 Quinn et al. Cureus 10(1): e2021. DOI 10.7759/cureus.2021 10 of 12 https://eric.ed.gov/?id=ED326149 https://dx.doi.org/10.1136/emermed-2013-203502 https://dx.doi.org/10.1136/emermed-2013-203502 https://www.ncbi.nlm.nih.gov/pubmed/?term=26115115 https://dx.doi.org/10.1016/j.annemergmed.2016.11.010 https://dx.doi.org/10.1016/j.annemergmed.2016.11.010 https://dx.doi.org/10.1136/postgradmedj-2015-133300 https://dx.doi.org/10.1136/postgradmedj-2015-133300 https://dx.doi.org/10.4300/JGME-D-17-00171.1 https://dx.doi.org/10.4300/JGME-D-17-00171.1 http://www.epijournal.com/articles/246/foam-helps-to-bridge-the-knowledge-translation-gap http://www.epijournal.com/articles/246/foam-helps-to-bridge-the-knowledge-translation-gap https://dx.doi.org/10.1007/s40037-017-0343-3 https://dx.doi.org/10.1007/s40037-017-0343-3 https://dx.doi.org/10.1016/j.annemergmed.2016.12.025 https://dx.doi.org/10.1016/j.annemergmed.2016.12.025 https://dx.doi.org/10.7759/cureus.1801 https://dx.doi.org/10.7759/cureus.1801 https://dx.doi.org/10.7759/cureus.1224 https://dx.doi.org/10.7759/cureus.1224 https://dx.doi.org/10.5811/westjem.2017.4.33906 https://dx.doi.org/10.5811/westjem.2017.4.33906 https://dx.doi.org/ 10.5811/westjem.2017.3.33409 https://dx.doi.org/ 10.5811/westjem.2017.3.33409 https://dx.doi.org/10.5811/westjem.2017.2.33430 https://dx.doi.org/10.5811/westjem.2017.2.33430 https://dx.doi.org/10.5811/westjem.2017.2.33076 https://dx.doi.org/10.5811/westjem.2017.2.33076 https://dx.doi.org/10.5811/westjem.2016.11.32613 https://dx.doi.org/10.5811/westjem.2016.11.32613 https://dx.doi.org/10.15200/winn.141827.77297 https://dx.doi.org/10.15200/winn.141827.77297 https://dx.doi.org/10.1016/j.annemergmed.2016.02.018 https://dx.doi.org/10.1016/j.annemergmed.2016.02.018 https://dx.doi.org/10.15200/winn.144720.08769 https://dx.doi.org/10.15200/winn.144720.08769 https://dx.doi.org/10.1002/aet2.10029 https://dx.doi.org/10.1002/aet2.10029 https://dx.doi.org/10.5811/westjem.2015.1.24860 10.5811/westjem.2015.1.24860 22. Chan T, Trueger NS, Roland D, Thoma B: Evidence-based medicine in the era of social media: scholarly engagement through participation and online interaction. CJEM. 2017, 1–6. 10.1017/cem.2016.407 23. Frank JR, Cheung WJ, Sherbino J, et al.: CAEP 2016 academic symposium: how to have an impact as an emergency medicine educator and scholar. CJEM. 2017, 19:S16–S21. 10.1017/cem.2017.339 24. Cameron CB, Nair V, Varma M, et al.: Does academic blogging enhance promotion and tenure? a survey of US and Canadian medicine and pediatric department chairs. JMIR Med Educ. 2016, 2:e10. 10.2196/mededu.4867 25. Lin M, Thoma B, Trueger NS, et al.: Quality indicators for blogs and podcasts used in medical education: modified Delphi consensus recommendations by an international cohort of health professions educators. Postgrad Med J. 2015, 91:546–50. 10.1136/postgradmedj-2014-133230 26. Chan T, Thoma B, Krishnan K, et al.: Derivation of two critical appraisal scores for trainees to evaluate online educational resources: a METRIQ study. West J Emerg Med. 2016, 17:574–84. 10.5811/westjem.2016.6.30825 27. Thoma B, Chan TM, Paterson QS, et al.: Emergency medicine and critical care blogs and podcasts: establishing an international consensus on quality. Ann Emerg Med. 2015, 66:396– 402. 10.1016/j.annemergmed.2015.03.002 28. Lin M, Joshi N, Grock A, et al.: Approved Instructional Resources Series: a national initiative to identify quality emergency medicine blog and podcast content for resident education. J Grad Med Educ. 2016, 8:219–25. 10.4300/JGME-D-15-00388.1 29. Flynn L, Jalali A, Moreau KA: Learning theory and its application to the use of social media in medical education. Postgrad Med J. 2015, 91:556–60. 10.1136/postgradmedj-2015-133358 30. Sherbino J: Education scholarship and its impact on emergency medicine education . West J Emerg Med. 2015, 16:804–9. 10.5811/westjem.2015.9.27355 31. Sterling M, Leung P, Wright D, Bishop TF: The use of social media in graduate medical education. Acad Med. 2017, 92:1043–56. 10.1097/ACM.0000000000001617 32. Purdy E, Thoma B, Bednarczyk J, Migneault D, Sherbino J: The use of free online educational resources by Canadian emergency medicine residents and program directors. Cjem. 2015, 1717:101–6. 10.1017/cem.2014.73 33. Paterson QS, Thoma B, Milne WK, Lin M, Chan TM: A systematic review and qualitative analysis to determine quality indicators for health professions education blogs and podcasts. J Grad Med Educ. 2015, 7:549–54. 10.4300/JGME-D-14-00728.1 34. Eysenbach G: Can tweets predict citations? Metrics of social impact based on Twitter and correlation with traditional metrics of scientific impact. J Med Internet Res. 2011, 13:e123. 10.2196/jmir.2041 35. Roland D, Spurr J, Cabrera D: Preliminary evidence for the emergence of a health care online community of practice: using a netnographic framework for twitter hashtag analytics. J Med Internet Res. 2017, 19:e252. 10.2196/jmir.7072 36. Lumba-Brown A, Tat S, Auerbach M, et al.: PEMNetwork barriers and enablers to collaboration and multimedia education in the Digital Age. Pediatr Emerg Care. 2016, 32:565–9. 10.1097/PEC.0000000000000885 37. Jordan J, Jones D, Williams D, Druck J, Kuehl DR: Publishing venues for education scholarship: a needs assessment. Acad Emerg Med. 2016, 23:731–5. 10.1111/acem.13003 38. Boulos MNK, Maramba I, Wheeler S: Wikis blogs and podcasts: a new generation of web- based tools for virtual collaborative clinical practice and education. BMC Med Educ. 2006, 6:41. 10.1186/1472-6920-6-41 39. Sutherland S, Jalali A: Social media as an open-learning resource in medical education: current perspectives. Adv Med Educ Pract . 2017, 8:369–75. 10.2147/AMEP.S112594 40. Diug B, Kendal E, Ilic D: Evaluating the use of twitter as a tool to increase engagement in medical education. Educ Health (Abingdon). 2016, 29:223–30. 41. Lin M, Sherbino J: Creating a virtual journal club: a Community of Practice using multiple social media strategies. JGME. 2015, 7:481–2. 10.4300/JGME-D-15-00070.1 42. Raine T, Thoma B, Chan TM, Lin M: FOAMSearch.net: a custom search engine for emergency medicine and critical care. Emergency Medicine Australas. 2015, 27:363–5. 10.1111/1742- 6723.12404 43. Riddell J, Patocka C, Lin M, Sherbino J: JGME-ALiEM hot topics in medical education: analysis 2018 Quinn et al. Cureus 10(1): e2021. DOI 10.7759/cureus.2021 11 of 12 https://dx.doi.org/10.5811/westjem.2015.1.24860 https://dx.doi.org/10.1017/cem.2016.407 https://dx.doi.org/10.1017/cem.2016.407 https://dx.doi.org/10.1017/cem.2017.339 https://dx.doi.org/10.1017/cem.2017.339 https://dx.doi.org/10.2196/mededu.4867 https://dx.doi.org/10.2196/mededu.4867 https://dx.doi.org/10.1136/postgradmedj-2014-133230 https://dx.doi.org/10.1136/postgradmedj-2014-133230 https://dx.doi.org/10.5811/westjem.2016.6.30825 https://dx.doi.org/10.5811/westjem.2016.6.30825 https://dx.doi.org/10.1016/j.annemergmed.2015.03.002 https://dx.doi.org/10.1016/j.annemergmed.2015.03.002 https://dx.doi.org/10.4300/JGME-D-15-00388.1 https://dx.doi.org/10.4300/JGME-D-15-00388.1 https://dx.doi.org/10.1136/postgradmedj-2015-133358 https://dx.doi.org/10.1136/postgradmedj-2015-133358 https://dx.doi.org/10.5811/westjem.2015.9.27355 https://dx.doi.org/10.5811/westjem.2015.9.27355 https://dx.doi.org/10.1097/ACM.0000000000001617 https://dx.doi.org/10.1097/ACM.0000000000001617 https://dx.doi.org/10.1017/cem.2014.73 https://dx.doi.org/10.1017/cem.2014.73 https://dx.doi.org/10.4300/JGME-D-14-00728.1 https://dx.doi.org/10.4300/JGME-D-14-00728.1 https://dx.doi.org/10.2196/jmir.2041 https://dx.doi.org/10.2196/jmir.2041 https://dx.doi.org/10.2196/jmir.7072 https://dx.doi.org/10.2196/jmir.7072 https://dx.doi.org/10.1097/PEC.0000000000000885 https://dx.doi.org/10.1097/PEC.0000000000000885 https://dx.doi.org/10.1111/acem.13003 https://dx.doi.org/10.1111/acem.13003 https://dx.doi.org/10.1186/1472-6920-6-41 https://dx.doi.org/10.1186/1472-6920-6-41 https://dx.doi.org/10.2147/AMEP.S112594 https://dx.doi.org/10.2147/AMEP.S112594 https://www.ncbi.nlm.nih.gov/pubmed/?term=28406107 https://dx.doi.org/10.4300/JGME-D-15-00070.1 https://dx.doi.org/10.4300/JGME-D-15-00070.1 https://dx.doi.org/10.1111/1742-6723.12404 https://dx.doi.org/10.1111/1742-6723.12404 https://dx.doi.org/10.4300/JGME-D-16-00067.1 of a multimodal online discussion about team-based learning. J Grad Med Educ. 2017, 9:102– 8. 10.4300/JGME-D-16-00067.1 44. Thoma B, Paddock M, Purdy E, et al.: Leveraging a virtual Community of Practice to participate in a survey-based study: a description of the METRIQ study methodology. AEM Educ Train. 2017, 1:110–3. 10.1002/aet2.10013 45. Ke Q, Ahn YY, Sugimoto CR: A systematic identification and analysis of scientists on Twitter . PLoS One. 2017, 12:1-17. 10.1371/journal.pone.0175368 46. Hillman T, Sherbino J: Social media in medical education: a new pedagogical paradigm . Postgrad Med J. 2015, 91:544–5. 10.1136/postgradmedj-2015-133686 47. Pronk NP, Ankel FK: Building resilience into the workplace: bending the system to adapt . ACSM's Health & Fitness Journal. 2017, 21:44–7. 48. Solomon D: The impact of digital dissemination for research and scholarship . ecancer. 2014, 73:83–7. 10.3332/ecancer.2014.ed44 49. Langdorf M, Lin M: Emergency medicine scholarship in the digital age . West J Emerg Med. 2016, 17:511–2. 10.5811/westjem.2016.8.32283 50. Glassick CE: Boyer's expanded definitions of scholarship, the standards for assessing scholarship, and the elusiveness of the scholarship of teaching. Acad Med. 2000, 75:877–80. 10.1097/00001888-200009000-00007 2018 Quinn et al. Cureus 10(1): e2021. DOI 10.7759/cureus.2021 12 of 12 https://dx.doi.org/10.4300/JGME-D-16-00067.1 https://dx.doi.org/10.1002/aet2.10013 https://dx.doi.org/10.1002/aet2.10013 https://dx.doi.org/10.1371/journal.pone.0175368 https://dx.doi.org/10.1371/journal.pone.0175368 https://dx.doi.org/10.1136/postgradmedj-2015-133686 https://dx.doi.org/10.1136/postgradmedj-2015-133686 https://experts.umn.edu/en/publications/building-resilience-into-the-workplace-bending-the-system-to-adap https://dx.doi.org/10.3332/ecancer.2014.ed44 https://dx.doi.org/10.3332/ecancer.2014.ed44 https://dx.doi.org/10.5811/westjem.2016.8.32283 https://dx.doi.org/10.5811/westjem.2016.8.32283 https://dx.doi.org/10.1097/00001888-200009000-00007 https://dx.doi.org/10.1097/00001888-200009000-00007 Curated Collections for Educators: Five Key Papers on Evaluating Digital Scholarship Abstract Introduction And Background Review Methods Results TABLE 1: The complete list of 38 literature items collected by the authorship team Discussion Limitations Conclusions Additional Information Disclosures Acknowledgements References