References that anyone can edit: review of Wikipedia citations in peer reviewed health science literature References that anyone can edit: review of Wikipedia citations in peer reviewed health science literature OPEN ACCESS M Dylan Bould staff anesthesiologist 1, Emily S Hladkowicz research assistant 2, Ashlee-Ann E Pigford research assistant2, Lee-Anne Ufholz director3, Tatyana Postonogova research associate4, Eunkyung Shin research assistant 5, Sylvain Boet staff anesthesiologist 6 1Department of Anesthesiology, Children’s Hospital of Eastern Ontario, University of Ottawa, 401 Smyth Road, Ottawa, ON, Canada, K1H 8L1; 2Department of Anesthesiology, Ottawa Hospital Research Institute, Ottawa; 3Health Sciences Library, University of Ottawa, Ottawa; 4Allan Waters Family Simulation Centre, Li Ka Shing Knowledge Institute, St Michael’s Hospital, Toronto, ON, Canada; 5Department of Surgery, St Michael’s Hospital, University of Toronto, Toronto; 6Department of Anesthesiology, The Ottawa Hospital, University of Ottawa, Ottawa Abstract Objectives To examine indexed health science journals to evaluate the prevalence of Wikipedia citations, identify the journals that publish articles with Wikipedia citations, and determine how Wikipedia is being cited. Design Bibliometric analysis. Study selection Publications in the English language that included citations to Wikipedia were retrieved using the online databases Scopus and Web of Science. Data sources To identify health science journals, results were refined using Ulrich’s database, selecting for citations from journals indexed in Medline, PubMed, or Embase. Using Thomson Reuters Journal Citation Reports, 2011 impact factors were collected for all journals included in the search. Data extraction Resulting citations were thematically coded, and descriptive statistics were calculated. Results 1433 full text articles from 1008 journals indexed in Medline, PubMed, or Embase with 2049 Wikipedia citations were accessed. The frequency of Wikipedia citations has increased over time; most citations occurred after December 2010. More than half of the citations were coded as definitions (n=648; 31.6%) or descriptions (n=482; 23.5%). Citations were not limited to journals with a low or no impact factor; the search found Wikipedia citations in many journals with high impact factors. Conclusions Many publications are citing information from a tertiary source that can be edited by anyone, although permanent, evidence based sources are available. We encourage journal editors and reviewers to use caution when publishing articles that cite Wikipedia. Introduction Launched on 15 January 2001, Wikipedia is self described as “a free, collaboratively edited, and multilingual internet encyclopedia supported by the non-profit Wikimedia Foundation.”1 As of 2012 Wikipedia is the largest online reference site,2 3 and it is reported to be the most used online healthcare resource globally.4 However, the assessment of Wikipedia as a credible source for information has been debated since its origin.5 The impermanent nature of Wikipedia entries and concerns about quality have been raised as important matters.6 The literature on Wikipedia has concentrated on evaluating content and ensuring that users have access to appropriate information.5 Only 13% of Wikipedia articles had identifiable errors when assessed by academics.7 Giles and colleagues found that the number of factual errors, omissions, or misleading statements in Wikipedia articles was comparable to the Encyclopaedia Britannica.8 In general, Wikipedia articles reference academic literature,9 10 and medical articles are overseen by WikiProject Medicine, which is driven by editorial oversight.2 4 Several studies have confirmed the accuracy of health specific Wikipedia articles and discuss the potential value in education of patients.2 3 8 11-15 Wikipedia has offered an innovative way to provide free access to information for people around the world.2 4 5 To further improve the quality of these articles, medical professionals are being encouraged to contribute to Wikipedia.2 16-18 Studies have also examined groups that use Wikipedia, noting that, in addition to patients and nursing students,2 3 10 19 medical students and residents commonly access Wikipedia to acquire health information.20-23 Although physicians have been discouraged from relying on Wikipedia,16 one study showed that in practice use of Wikipedia is as high as 70% among junior physicians.22 In general, consultation of Wikipedia is a growing trend among academics. For example, among more than 1000 authors in Nature, 17% reported that they consulted Wikipedia on a weekly basis.8 As the public and Correspondence to: M D Bould dbould@cheo.on.ca No commercial reuse: See rights and reprints http://www.bmj.com/permissions Subscribe: http://www.bmj.com/subscribe BMJ 2014;348:g1585 doi: 10.1136/bmj.g1585 (Published 5 March 2014) Page 1 of 10 Research RESEARCH o n 5 A p ril 2 0 2 1 b y g u e st. P ro te cte d b y co p yrig h t. h ttp ://w w w .b m j.co m / B M J: first p u b lish e d a s 1 0 .1 1 3 6 /b m j.g 1 5 8 5 o n 6 M a rch 2 0 1 4 . D o w n lo a d e d fro m http://creativecommons.org/licenses/by-nc/3.0/ http://www.bmj.com/permissions http://www.bmj.com/subscribe http://crossmark.crossref.org/dialog/?doi=10.1136/bmj.g1585&domain=pdf&date_stamp=2014-03-05 http://www.bmj.com/ healthcare professionals increasingly accept Wikipedia as a source of information, new questions emerge such as the appropriateness of citing Wikipedia in academic publications. The controversial nature of citing Wikipedia as a reference source for academic information is threefold. Firstly, a theoretical concern exists that anyone with access to the internet can alter Wikipedia. This raises questions about the spread of unintentional misinformation, which is fuelled by acts of intentional vandalism that have gained media attention and have challenged the public’s perception of Wikipedia.24-26 However, the use of a wiki model addresses the fear that that the information on Wikipedia is not guaranteed to be correct. The concept of a wiki is based on the assumption that the majority is correct: if someone writes an erroneous statement on Wikipedia, the probability is considered to be high that someone else from the majority that is presumed to be correct will identify the error. This model has many advantages and allows Wikipedia to be a free source of a vast amount of collaborative information.2 4 5 However, the scientific community still has concerns about the academic integrity of a model that in theory could be edited by anyone. Secondly, the changing nature of Wikipedia makes permanent versions difficult to access. Although Wikipedia maintains a detailed history of previous Wikipedia pages, current academic citation systems rarely require a citation to include time stamped access information detailed to the second. As Wikipedia pages are in constant change, unlike paper encyclopaedias in which readers can confidently find the permanent cited source, readers might find confirming that the Wikipedia reference is the exact version cited challenging, particularly if detailed time stamps are not included.27 Thirdly, citing tertiary sources such as Wikipedia, which are resources that compile or provide digests of secondary sources, has literary problems.28 Secondary sources are books, articles, or unpublished literature that provide an interpretation of primary sources, which include original data, manuscripts, records, or documents. International guidelines state that authors should provide direct references to original research sources,29 so citing Wikipedia or any other tertiary source in the academic literature opposes literary practice. To date, only one study has examined the frequency of citation of Wikipedia in the academic literature.30 To our knowledge, no one has examined the frequency of Wikipedia citations in the health science literature and the ways in which Wikipedia citations are used. We aimed to evaluate the prevalence of Wikipedia citations in indexed health science journals, identify the health science journals that publish articles with Wikipedia citations, and determine how Wikipedia is being cited. Methods Data collection In February 2012 one investigator (LU) searched the ISI Web of Science and Scopus online databases to identify articles in the peer reviewed health science literature that had directly cited Wikipedia since its inception in 2001. She searched references of all articles in both databases on the same day for the word “Wikipedia” or any possible derivation to account for spelling errors. The word could appear anywhere in the reference, so not all articles necessarily referenced Wikipedia but could be articles about Wikipedia. To focus on indexed health science literature, we refined the results by using Ulrich’s Database to identify articles from journals indexed in at least one of Medline, PubMed, or Embase. We retrieved full text for the articles (including reviews, original research, editorials, letters to the editor, and case reports) identified in the search through the University of Ottawa library and associated Canadian inter-library loan services. We excluded citations when articles were not written in English or when full text was unavailable. Although citations originated in the health science literature, the topic of the citation could extend beyond health, so we included all Wikipedia pages that were cited in the academic literature. In the event that an article cited Wikipedia more than once, we considered all citations and created duplicate entries for that article. We used Thomson Reuters Journal Citation Reports (JCR) to collected journals’ 2011 impact factors (a measure that reflects how frequently the average article in a journal has been cited in a particular period).31 32 JCR impact factors provide quantitative evidence about the position of one journal in relation to the competition and offer an approximation of the prestige of a journal. When interpreting the impact factor, one should consider only journals within the same subject category as the scores are relative.32 JCR coverage includes the world’s leading journals and offers a systematic, quantifiable means to critically evaluate them by measuring the influence and impact of research at the journal and category levels, as well as showing the relation between citing and cited journals. We did not recover other journal metrics because of the unique nature of JCR and its pre-eminence as a proxy to measure a journal’s importance within a field. Data extraction/coding system Using an iterative process, we developed a descriptive coding strategy. Three investigators (MDB, SB, ESH) reached consensus on the distinction between categories. After independently reviewing 15% of citations, the three investigators met as a team to discuss their coding strategies. They developed an initial coding strategy and recoded citations with the new guide. The team continued this iterative process of coding, meeting, and refining the guide before the three authors reached agreement and the coding strategy was finalised. Emergent sub-codes were integrated into the coding system as they arose. Using the coding system, one investigator (ESH) systematically coded all Wikipedia citations. To ensure coding reliability, another investigator (AEP) coded 100 random citations with the same coding system. Analysis We used SPSS 17.0 for statistical analysis. We calculated descriptive statistics for the frequency of Wikipedia citations by thematic code, the impact factor of journals citing Wikipedia (median, range), and the frequency of these statistics by year. We used the intra-class correlation coefficient to assess the inter-rater reliability. Results We recovered 2359 publications (2307 from Scopus and an additional 52 from Web of Science) in our search. After excluding duplicates, non-English journals, and articles that did not directly cite Wikipedia, we retrieved full text for 1433 articles from 1008 indexed journals, with 2049 Wikipedia citations. In total, 2011 impact factors were available for 1420 citations in 980 articles from 650 indexed journals and were found in the JCR database. Table 1⇓ shows the frequency of Wikipedia citations since 2001, the nature of citations (thematic code), and related impact factor information (median and range). The intra-class correlation No commercial reuse: See rights and reprints http://www.bmj.com/permissions Subscribe: http://www.bmj.com/subscribe BMJ 2014;348:g1585 doi: 10.1136/bmj.g1585 (Published 5 March 2014) Page 2 of 10 RESEARCH o n 5 A p ril 2 0 2 1 b y g u e st. P ro te cte d b y co p yrig h t. h ttp ://w w w .b m j.co m / B M J: first p u b lish e d a s 1 0 .1 1 3 6 /b m j.g 1 5 8 5 o n 6 M a rch 2 0 1 4 . D o w n lo a d e d fro m http://www.bmj.com/permissions http://www.bmj.com/subscribe http://www.bmj.com/ coefficient for the two coders was 0.91 (P<0.001), indicating a high degree of inter-rater reliability. More than half of the citations were coded as definitions (n=648; 31.6%) or descriptions (n=482; 23.5%). For the purpose of this study, of the 13 categories, we considered citations (n=82; 4.0%) from only two categories (Citations about Wikipedia, and Wikipedia used in methods) to be the most appropriate uses of Wikipedia, as in these cases Wikipedia was the original source of information. Furthermore, we recovered 97 (4.8%) citations in which Wikipedia was cited in place of an original research study. The median impact factor of journals citing Wikipedia was 2.0 and has remained fairly consistent over time. However, the total number of Wikipedia citations has increased each year since 2004 except for between 2009 and 2010. The figure⇓ shows the number of articles that cite Wikipedia at least once by year up to 21 November 2013. Consistently since the inception of Wikipedia, journals with high impact factors have continued to publish references to Wikipedia. Table 2⇓ illustrates the journals with the top 25 highest impact factors that emerged in our study. These journals accounted for 2.2% (n=3) of cases in which Wikipedia was the most appropriate citations (as previously defined). Table 3⇓ specifies the titles and types of articles published in these journals, including the number of Wikipedia citations in each article. Journals with available impact factors that cited Wikipedia more than 10 times included Explore: The Journal of Science and Healing (n=53), Child’s Nervous System (n=26), BMC Bioinformatics (n=21), Theoretical Biology and Medical Modelling (n=20), Journal of Medical Internet Research (n=17), Health Information and Libraries Journal (n=13), BMJ (n=13), JALA—Journal of the Association for Laboratory Automation (n=12), and American Journal of Forensic Medicine and Pathology (n=11). Discussion Our findings illustrate a relatively small but increasing frequency of citations of Wikipedia in indexed health sciences literature. We retrieved 2049 Wikipedia citations, in 1433 articles from 1008 journals indexed in Medline, PubMed, or Embase. Most of these citations occurred after December 2010 and were consistently found in journals with low impact factors and journals without impact factors, as well as in journals with high impact factors, including Nature, Science, and the BMJ. Using the descriptive coding strategy, we found a wide variety of uses for Wikipedia citations in the literature, the most common being definitions and descriptive statements. Strengths and limitations As the first study to describe the citation practices of Wikipedia (how, where, when, and so on) in indexed health science literature, our study adds insight into the role that Wikipedia may play in academic literature. We also provide information about the quality of the journals that cite Wikipedia, measured through impact factors from the Journal Citation Reports. The study searched citations from two well recognised databases (Scopus and Web of Science) and included all English language journals available in full text at Canadian university libraries, so some citations may have been excluded from the search because of the search criteria. However, the study used two comprehensive databases that probably captured most citations from journals with reaching influence as well as high impact factors. Although we report an increasing number of Wikipedia citations, the health science literature is expansive; therefore, papers that have cited Wikipedia remain a small minority of all published papers, albeit with increasing frequency. Comparison with other studies Active research on Wikipedia has examined the role of academic citations in Wikipedia, content validity, and interactions between Wikipedia and user groups.5 The only other study to examine Wikipedia in the scholarly references was completed by Park in 2011.30 In this bibliometric analysis, Park also used the ISI Web of Science and Scopus databases to identify the total number of studies (n=1746), as well as leading authors, their institutional affiliations, most frequent publication sources, main academic fields, and other statistics on the frequency with which scholarly articles cite Wikipedia. Our study took Park’s analysis further. Firstly, we used Ulrich’s database to refine search findings to be specific to health sciences literature. Secondly, we examined how Wikipedia is being cited. Both studies show a general increase in frequency of citations over time, suggesting that if Park’s data were to extend beyond 2010 they would continue to show increasing Wikipedia citations. While Park identified the publications that cited Wikipedia most often, our study also provides impact factors, adding information about journals’ impact. Contextualisation and policy implications Just because more researchers are citing Wikipedia does not necessarily justify it as a valid source of information for citation. Wikipedia itself has cautioned against using Wikipedia as a source,33 and some universities have gone so far as to ban students from citing Wikipedia.34 Recognising that learning modalities are changing and evolving increasingly towards online and e-resources,21 23 35 36 we believe that ensuring that peer reviewed academic literature aligns with the International Committee of Medical Journal Editors (ICMJE) guidelines is still important. Relevant to this study, the ICMJE guidelines state that “Readers should therefore be provided with direct references to original research sources whenever possible.”29 Although people have been argued that Wikipedia is comparable to an encyclopedia in terms of accuracy,8 37 this argument overlooks the literary problems associated with citing a tertiary source intended to direct researchers to an appropriate reference.34 We echo the findings of other researchers who believe that Wikipedia should not be cited when a more authoritative—that is, primary, permanent, peer reviewed, evidence based—source exists6; however, Wikipedia may be the most appropriate source to cite for a definition of Wikipedia and in situations in which Wikipedia is used as part of the scientific methods (for example, a search strategy). Outside of those rare instances, arguing that citation of Wikipedia in the academic literature is appropriate is difficult. Our “original research” category further illustrates how Wikipedia was often cited when the authors could have cited an original research study. For example, one article stated that “Some researchers propose that vitamin D supplementation may be beneficial in the treatment and prevention of some types of cancer,”38 and rather than citing the original sources provided on the Wikipedia page, the author instead cited Wikipedia. Evidence based medicine requires that clinical decisions are made using professional expertise in conjunction with replicable evidence from systematic research.39 As Wikipedia entries are constantly changing, they may be difficult to access in the form that was originally cited by the time the work is published, reducing the reader’s ability to reproduce the author’s process.27 In a study by Peoples,6 most citations to Wikipedia did not reference the date and time at which the entry was visited. Although Wikipedia does have a history function that allows users to access previous versions of the page, this function may not be obvious to most users, and determining which version No commercial reuse: See rights and reprints http://www.bmj.com/permissions Subscribe: http://www.bmj.com/subscribe BMJ 2014;348:g1585 doi: 10.1136/bmj.g1585 (Published 5 March 2014) Page 3 of 10 RESEARCH o n 5 A p ril 2 0 2 1 b y g u e st. P ro te cte d b y co p yrig h t. h ttp ://w w w .b m j.co m / B M J: first p u b lish e d a s 1 0 .1 1 3 6 /b m j.g 1 5 8 5 o n 6 M a rch 2 0 1 4 . D o w n lo a d e d fro m http://www.bmj.com/permissions http://www.bmj.com/subscribe http://www.bmj.com/ was accessed for citations that do not denote the date and time of the visit can be difficult. If Wikipedia is to be cited in the future, not only the date accessed but also the time should be included so that future researchers can access the entry as it was originally viewed. Thus, the process of peer review should include close attention to the references that appear in articles before recommending them for publication and potentially approving information that contradicts evidence based research. Future directions Although the proportion of retrieved Wikipedia citations in the indexed health science literature was relatively small, the increasing trend of citation is still important to note. The study of how Wikipedia is used in the academic literature is relatively young, and beginning an open discourse around the appropriateness of using non-permanent Wikipedia citations in the academic literature seems timely. The relationship between academic publication and Wikipedia remains largely understudied, and international guidelines such as those of the ICJME, World Association of Medical Editors, and Council of Science Editors lack editorial guidance on the subject. Although our study does not provide evidence of harm by these citations, even those that substitute for a permanent primary research source, we emphasise the need for a consistent voice on how Wikipedia should be used in the academic literature. This study begs the broader question of “what is an appropriate reference source?” Is it something that is generated by the scientific method, is replicable, and undergoes rigorous peer review or is it something that is collaboratively generated, readily open to editing, and broadly accepted? We call attention to the need to work with the Wikipedia community to establish guidelines not only for reviewers and editors but also for academics drafting publications. Conclusion An increasing number of peer reviewed academic papers in the health sciences are citing Wikipedia. The apparent increase in the frequency of citations of Wikipedia may suggest a lack of understanding by authors, reviewers, or editors of the mechanisms by which Wikipedia evolves. Although only a very small proportion of citations are of Wikipedia pages, the possibility for the spread of misinformation from an unverified source is at odds with the principles of robust scientific methodology and could potentially affect care of patients. We caution against this trend and suggest that editors and reviewers insist on citing primary sources of information where possible. Contributors: MDB and SB were involved in the conception and design of the study, the analysis of the data, and critical review of the manuscript. ESH was involved in the conception and design of the study, the analysis and interpretation of data, and critical review of the manuscript. AEP was involved in the analysis and interpretation of data and drafted the manuscript. LU was involved in the conception and design of the study, interpretation of the data, and critical review of the manuscript. TP and ES were involved in collection and analysis of data and critically revised the manuscript. All authors had full access to all data and reviewed the final version to be published. MDB and SB are the guarantors. Funding: The study was funded by the Children’s Hospital of Eastern Ontario Department of Anesthesiology, University of Ottawa, and the Ottawa Hospital Department of Anesthesiology, University of Ottawa. The funding sources had no role in the study design; the collection, analysis, and interpretation of data; or the writing of the article and the decision to submit it for publication. Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work. Ethical approval: Not needed. All the data presented are in the public domain. Citation information and full text articles were acquired through the ISI Web of Science and Scopus databases. As the accepted practice is not to seek consent to report on information in the public domain, we did not do so for this study, and nor did we seek approval from our research ethics board. Transparency: The lead author (MDB) affirms that this manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained. Data sharing: No additional data available. 1 Wikipedia. Wikipedia. 2012 [updated 19:11, 28 October 2012; cited 30 October 2012]. http://en.wikipedia.org/wiki/Wikipedia. 2 Heilman JM, Kemmann E, Bonert M, Chatterjee A, Ragar B, Beards GM, et al. Wikipedia: a key tool for global public health promotion. J Med Internet Res 2011;13:e14. 3 Thomas GR, Eng L, de Wolff JF, Grover SC. An evaluation of wikipedia as a resource for patient education in nephrology. Semin Dial 2013;26:159-63. 4 Heilman J. Online encyclopedia provides free health info for all. Interview by Fiona Fleck. Bull World Health Organ 2013;91:8-9. 5 Okoli C, Mehdi M, Mesgari M, Nielsen FÅ, Lanamäki A. The people’s encyclopedia under the gaze of the sages: a systematic review of scholarly research on Wikipedia. 2012. http: //dx.doi.org/10.2139/ssrn.2021326. 6 Peoples LF. The citation of Wikipedia in judicial opinions. Yale Journal of Law and Technology 2009;12. 7 Chesney T. An empirical examination of wikipedia’s credibility. First Monday 2006;11(11). 8 Giles J. Internet encycolpedias go head to head. Nature 2005;438:900-1. 9 Nielsen FÅ. Scientific citations in Wikipedia. First Monday 2007;12(8). 10 Haigh CA. Wikipedia as an evidence source for nursing and healthcare students. Nurse Educ Today 2011;31:135-9. 11 Czarnecka-Kujawa K, Abdalian R, Grover SC. The quality of open access and open source internet material in gastroenterology: is Wikipedia appropriate for knowledge transfer to patients? [abstract M1042]. Gastroenterology 2008;134:A325-6. 12 Giles J. Wikipedia rival calls in the experts. Nature 2006;443:493. 13 Rajagopalan MS, Khanna VK, Leiter Y, Stott M, Showalter TN, Dicker AP, et al. Patient-oriented cancer information on the internet: a comparison of wikipedia and a professionally maintained database. J Oncol Pract 2011;7:319-23. 14 Van der Marel S, Duijvestein M, Hardwick JC, van den Brink GR, Veenendaal R, Hommes DW, et al. Quality of web-based information on inflammatory bowel diseases. Inflamm Bowel Dis 2009;15:1891-6. 15 Richardson CG, Hamadani LG, Gotay C. Quantifying Canadians’ use of the internet as a source of information on behavioural risk factor modifications related to cancer prevention. Chronic Dis Inj Can 2013;33:123-8. 16 Metcalfe D, Powell J. Should doctors spurn Wikipedia? J R Soc Med 2011;104:488-9. 17 Callis KL, Christ LR, Resasco J, Armitage DW, Ash JD, Caughlin TT, et al. Improving Wikipedia: educational opportunity and professional responsibility. Trends Ecol Evol 2009;24:177-9. 18 Wikipedia. Wikipedia:Plain and simple guide for medical editors. 2013 [updated 13:43, 17 May 2013 ; cited 4 December 2103]. http://en.wikipedia.org/wiki/Wikipedia:Plain_and_ simple_guide_for_medical_editors. 19 Leithner A, Maurer-Ertl W, Glehr M, Friesenbichler J, Leithner K, Windhager R. Wikipedia and osteosarcoma: a trustworthy patients’ information? J Am Med Inform Assoc 2010;17:373-4. 20 Allahwala UK, Nadkarni A, Sebaratnam DF. Wikipedia use amongst medical students—new insights into the digital revolution. Med Teach 2013;35:337. 21 Duran-Nelson A, Gladding S, Beattie J, Nixon LJ. Should we Google it? Resource use by internal medicine residents for point-of-care clinical decision making. Acad Med 2013;88:788-94. 22 Hughes B, Joshi I, Lemonde H, Wareham J. Junior physician’s use of Web 2.0 for information seeking and medical education: a qualitative study. Int J Med Inf 2009;78:645-55. 23 Judd T, Kennedy G. Expediency-based practice? Medical students’ reliance on Google and Wikipedia for biomedical inquiries. Br J Educ Technol 2011;42:351-60. 24 Vandals prompt Wikipedia to ponder editing changes. ABC News 2009 Jan 28. 25 Martin L. Wikipedia fights off cyber vandals. The Guardian 2006 Jun 18. 26 Kleeman J. Wikipedia fights vandalism. New Zealand Herald 2007 Apr 2. 27 Knapp MM. eBay, Wikipedia, and the future of the footnote. Theatre Hist Stud 2008;28:36-41. 28 Prytherch RJ. Harrod’s librarians’ glossary and reference book: a directory of over 10,200 terms, organizations, projects and acronyms in the areas of information management, library science, publishing and archive management: Ashgate Publishing, 2005. 29 International Committee of Medical Journal Editors. Recommendations for the conduct, reporting, editing, and publication of scholarly work in medical journals. 2013 [cited 15 Feb 2014]. www.icmje.org/icmje-recommendations.pdf. 30 Park TK. The visibility of Wikipedia in scholarly publications. First Monday 2011;16(1). 31 Garfield E. The Thomson Reuters impact factor. Curr Contents 1994;25:3-7. 32 Reuters T. Journal Citation Reports. 2013. http://thomsonreuters.com/products_services/ science/science_products/a-z/journal_citation_reports/. No commercial reuse: See rights and reprints http://www.bmj.com/permissions Subscribe: http://www.bmj.com/subscribe BMJ 2014;348:g1585 doi: 10.1136/bmj.g1585 (Published 5 March 2014) Page 4 of 10 RESEARCH o n 5 A p ril 2 0 2 1 b y g u e st. P ro te cte d b y co p yrig h t. h ttp ://w w w .b m j.co m / B M J: first p u b lish e d a s 1 0 .1 1 3 6 /b m j.g 1 5 8 5 o n 6 M a rch 2 0 1 4 . D o w n lo a d e d fro m http://www.icmje.org/coi_disclosure.pdf http://en.wikipedia.org/wiki/Wikipedia http://dx.doi.org/10.2139/ssrn.2021326 http://dx.doi.org/10.2139/ssrn.2021326 http://en.wikipedia.org/wiki/Wikipedia:Plain_and_simple_guide_for_medical_editors http://en.wikipedia.org/wiki/Wikipedia:Plain_and_simple_guide_for_medical_editors http://www.icmje.org/icmje-recommendations.pdf http://thomsonreuters.com/products_services/science/science_products/a-z/journal_citation_reports/ http://thomsonreuters.com/products_services/science/science_products/a-z/journal_citation_reports/ http://www.bmj.com/permissions http://www.bmj.com/subscribe http://www.bmj.com/ What is already known on this topic The use of Wikipedia as a source of academic information has been debated since its origin, but it is increasingly cited in peer reviewed health science literature Although studies have examined the content and vailidity of Wikipedia, no evidence has been published about how this resource is being used in health science literature What this paper adds Although a few instances exist that may warrant using Wikipedia as a reference, Wikipedia is often cited when permanent, evidence based sources are available Authors, reviewers, and editors should use caution when publishing articles that include Wikipedia citations 33 Wikipedia. Wikipedia:Citing Wikipedia. 2013 [updated 19:49, 5 June 2013; cited 6 June 2013]. http://en.wikipedia.org/wiki/Wikipedia:Citing_Wikipedia. 34 Waters NL. Why you can’t cite Wikipedia in my class. Commun ACM 2007;50:15-7. 35 Matava CT, Rosen D, Siu E, Bould DM. eLearning among Canadian anesthesia residents: a survey of podcast use and content needs. BMC Med Educ 2013;13:59. 36 Mostyn A, Jenkinson CM, McCormick D, Meade O, Lymn JS. An exploration of student experiences of using biology podcasts in nursing training. BMC Med Educ 2013;13:12. 37 Reavley NJ, Mackinnon AJ, Morgan AJ, Alvarez-Jimenez M, Hetrick SE, Killackey E, et al. Quality of information sources about mental disorders: a comparison of Wikipedia with centrally controlled web and printed sources. Psychol Med 2012;42:1753-62. 38 Chakraborti C, K. Vitamin D as a promising anticancer agent. Indian J Pharmacol 2010;43:113-20. 39 Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn’t. BMJ 1996;312:71-2. Accepted: 10 February 2014 Cite this as: BMJ 2014;348:g1585 This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/3.0/. No commercial reuse: See rights and reprints http://www.bmj.com/permissions Subscribe: http://www.bmj.com/subscribe BMJ 2014;348:g1585 doi: 10.1136/bmj.g1585 (Published 5 March 2014) Page 5 of 10 RESEARCH o n 5 A p ril 2 0 2 1 b y g u e st. P ro te cte d b y co p yrig h t. h ttp ://w w w .b m j.co m / B M J: first p u b lish e d a s 1 0 .1 1 3 6 /b m j.g 1 5 8 5 o n 6 M a rch 2 0 1 4 . D o w n lo a d e d fro m http://en.wikipedia.org/wiki/Wikipedia:Citing_Wikipedia http://creativecommons.org/licenses/by-nc/3.0/ http://www.bmj.com/permissions http://www.bmj.com/subscribe http://www.bmj.com/ Tables Table 1| Frequency of Wikipedia citations since 2001, categorised by code and impact factor Median (range) impact factor Total citations—No (%)Coding strategy 1.9 (0.1-31.2)648 (31.6)A statement defining a word, phrase, item, or other symbol. This includes definitions of chemical formulas and equations that are not otherwise supported by original research references. Example: Caregiver is an internationally accepted term for unpaid people who are caring for someone requiring support due to a disability, frailty, mental health problem, learning disability or old age (Wikipedia) Definition 1.9 (none*-31.2)482 (23.5)Description of a process, system, or event. This differs from a definition, in that a process is explained without necessarily defining a term. Example: The key component of clinical audit is that performance is reviewed (or audited) to ensure that what should be done is being done, and if not, it provides a framework to enable improvements to be made (Wikipedia) Descriptive statement 1.7 (0.1-36.3)277 (13.5)Description of a historical event. This can include the date, location, and a description of the events that took place. Example: In 1984 a beautiful 18-year-old coed, Libby Zion, died in a New York City hospital after being improperly treated by sleep-deprived residents (Wikipedia) Historical 2.0 (none*-16.1)161 (7.9)Includes demographic and gross domestic product information (population size, including census information), geographical information, ratios, percentages, and averages. Example: West Bengal is an agriculture-dependent state in eastern India. It occupies only 2.7% of India’s land area, although it supports over 7.8% of the Indian population, and is the most densely populated state in India (Wikipedia) Statistics 2.9 (0.7-8.8)59 (2.9)Reporting results from a research study when on the Wikipedia page the original research is referenced but in the article the authors referenced Wikipedia, or the authors imply something like “research suggests that. . .” with no support other than Wikipedia. Example: Epidemiologic studies have shown that individuals who consume high levels of b-carotene in their diet and those with high levels of b-carotene in their blood have significantly reduced risk of lung cancer (Wikipedia) Original research 2.8 (0.6-14.1)38 (1.9)Tools used, such as a laws, formulas, or scales that could have been referenced using an original source. Example: The paediatric GCS (Wikipedia) and PNG TB score 5,6 (Table 1) were recorded for all children 1.2 (0.4-13.7)91 (4.4)Description of a person’s life or accomplishments. This may include dates. Example: In 1895, Wilhelm Conrad Roentgen reported the discovery of the x-ray (Wikipedia) Biographical 2.8 (0.6-13.8)78 (3.8)Citation about the Wikipedia, describing what it is, how it is used, and so on. Example: A case in point is Wikipedia, the largest encyclopedia in the world. With over 3.5 million registered users (Wikipedia) Citation about Wikipedia 1.8 (0.5-5.7)55 (2.7)Describing what Web 2.0 is and examples; Web 2.0 are resources in which multiple users continually update and remix data.21 Example: During the first decade of the 21st century, we have witnessed the birth of Web 2.0, heralded by a set of powerful tools designed to enhance creativity, information sharing, collaboration, and functionality of the Web, and to transfer power to the end user (Wikipedia) Web 2.0 citation 1.8 (0.5-6.8)50 (2.4)Any images, including maps, people, structures, figures, and so onImages 2.0 (0.5-7.8)42 (2.0)Author of article says to search Wikipedia for further details and references Wikipedia. Example: For a very interesting explanation of the impact factor and the debate around its use, go to Wikipedia’s “Impact Factor” entry (Wikipedia) Recommended reading 2.1 (0.1-9.9)37 (1.8)Unclear language, unable to understand what part of the citation is a reference to Wikipedia, reference number does not match citation, missing reference number, and so on. Example: http://en.wikipedia.org was listed in the reference list, but the matching reference number was absent from the text Unclear citation/ referencing 1.8 (1.0-2.8)27 (1.3)Replicating a direct quote from a person, movie, or book quoted in Wikipedia. Example: Ralph Waldo Emerson, an American philosopher, said: “Nothing great was ever achieved without enthusiasm” (Wikipedia) Direct quotation 4.1 (4.1-31.2)4 (0.2)Wikipedia is directly used in the research methods of the paper. Example: We compared the rise to fame of the most famous people of different eras. We took all 740,000 people with entries in Wikipedia, removed cases where several famous individuals share a name, and sorted the rest by birth date and frequency (Wikipedia) Wikipedia in methods 2.0 (none*-36.3)2049 (100)Total *Journal indexed in Journal Citation Reports does not have assigned impact factor. No commercial reuse: See rights and reprints http://www.bmj.com/permissions Subscribe: http://www.bmj.com/subscribe BMJ 2014;348:g1585 doi: 10.1136/bmj.g1585 (Published 5 March 2014) Page 6 of 10 RESEARCH o n 5 A p ril 2 0 2 1 b y g u e st. P ro te cte d b y co p yrig h t. h ttp ://w w w .b m j.co m / B M J: first p u b lish e d a s 1 0 .1 1 3 6 /b m j.g 1 5 8 5 o n 6 M a rch 2 0 1 4 . D o w n lo a d e d fro m http://en.wikipedia.org/ http://www.bmj.com/permissions http://www.bmj.com/subscribe http://www.bmj.com/ Table 2| 25 journals with highest impact factors that have cited Wikipedia since 2001 No (%) of citations for which Wikipedia is most appropriate source*Total No (%) of Wikipedia citationsImpact factorPeriodical name 82 (100)2049 (100)—All recovered citations 0 (0)1 (0.05)36.3Nature 1 (1.2)3 (0.1)31.2Science 0 (0)1 (0.05)30.4Nature Reviews Neuroscience 0 (0)1 (0.05)22.5Nature Medicine 0 (0)3 (0.1)18.4Journal of Clinical Oncology 0 (0)5 (0.2)17.4The Lancet Infectious Diseases 0 (0)4 (0.2)16.7Annals of Internal Medicine 0 (0)5 (0.2)16.3PLoS Medicine 0 (0)2 (0.1)16.1Clinical Microbiology Reviews 0 (0)3 (0.1)14.7Circulation 0 (0)3 (0.1)14.2Journal of the American College of Cardiology 0 (0)13 (0.6)14.1BMJ 1 (1.2)2 (0.1)13.8Journal of the National Cancer Institute 0 (0)1 (0.05)13.7Molecular Psychiatry 0 (0)2 (0.1)12.5Nature Reviews Neurology 0 (0)1 (0.05)11.7Gastroenterology 0 (0)5 (0.2)9.9Journal of the American Chemical Society 0 (0)1 (0.05)9.2Clinical Infectious Diseases 1 (1.2)2 (0.1)9.0Genome Biology 0 (0)1 (0.05)8.8Nature Reviews Cardiology 0 (0)5 (0.2)8.8Schizophrenia Bulletin 0 (0)1 (0.05)8.1Diabetes care 0 (0)1 (0.05)8.0Nucleic Acids Research 0 (0)4 (0.2)7.8Science Translational Medicine 0 (0)3 (0.1)7.5Annals of Surgery 3 (2.2)73 (3.6)—Total *Citations for which Wikipedia is original source of information (citations about Wikipedia, and Wikipedia used in methods) were considered most appropriate citations of Wikipedia. No commercial reuse: See rights and reprints http://www.bmj.com/permissions Subscribe: http://www.bmj.com/subscribe BMJ 2014;348:g1585 doi: 10.1136/bmj.g1585 (Published 5 March 2014) Page 7 of 10 RESEARCH o n 5 A p ril 2 0 2 1 b y g u e st. P ro te cte d b y co p yrig h t. h ttp ://w w w .b m j.co m / B M J: first p u b lish e d a s 1 0 .1 1 3 6 /b m j.g 1 5 8 5 o n 6 M a rch 2 0 1 4 . D o w n lo a d e d fro m http://www.bmj.com/permissions http://www.bmj.com/subscribe http://www.bmj.com/ Table 3| 25 journals with highest impact factors: article titles, types, and number of Wikipedia citations No of Wikipedia citationsType of articleArticleJournal 1LetterTracking the rupture of the Mw = 9.3 Sumatra earthquake over 1,150 km at teleseismic distance Nature 1PerspectiveChallenges and opportunities in mining neuroscience dataScience 1PerspectiveOn the future of genomic data 1Research articleQuantitative analysis of culture using millions of digitized books 1PerspectiveGene therapy: can neural stem cells deliver?Nature Reviews Neuroscience 1CommentaryPerspectives on the properties of stem cellsNature Medicine 1CorrespondenceReply to R.I. Haddad et alJournal of Clinical Oncology 4SeriesEmergence of medicine for mass gatherings: lessons from the HajjThe Lancet Infectious Diseases 1SeriesNon-communicable health risks during mass gatherings 2History of medicineAkhenaten and the strange physiques of Egypt’s 18th dynastyAnnals of Internal Medicine 1History of medicineThe nomogram epidemic: resurgence of a medical relic 1Medicine and public policyIndividualized guidelines: the potential for increasing quality and reducing costs 1Policy forumThe global health system: linking knowledge with action—learning from malariaPLoS Medicine 1Research articleMotor vehicle crashes in diabetic patients with tight glycemic control: a population-based case control analysis 1Research articleCardiovascular risk with non-steroidal anti-inflammatory drugs: systematic review of population-based controlled observational studies 1PerspectiveMeasuring the true costs of war: consensus and controversy 1Policy forumGlobal health philanthropy and institutional relationships: how should conflicts of interest be addressed? 2ReviewBed bugs: clinical relevance and control optionsClinical Microbiology Reviews 2ControversiesThere is a role for industry-sponsored education in cardiologyCirculation 1ReportsACC/AHA/ACR/ASE/ASNC/HRS/NASCI/RSNA/SAIP/SCAI/SCCT/SC MR/SIR 2008 key data elements and definitions for cardiac imaging: a report of the American College of Cardiology/American Heart Association task force on clinical data standards (writing committee to develop clinical data standards for cardiac imaging) 1ReportsACC/AHA/ACR/ASE/ASNC/HRS/NASCI/RSNA/SAIP/SCAI/SCCT/SC MR/SIR 2008 key data elements and definitions for cardiac imaging: a report of the American College of Cardiology/American Heart Association task force on clinical data standards (writing committee to develop clinical data standards for cardiac imaging) Journal of the American College of Cardiology 9Research articleHow citation distortions create unfounded authority: analysis of a citation networkBMJ 2Research methods and reporting; Statistics notes Statistics notes: brackets (parentheses) in formulas 1Feature; Christmas 2011: Oral traditions The royal road to healing: a bit of a saga 1Research articleCD4 cell count and viral load monitoring in patients undergoing antiretroviral therapy in Uganda: cost effectiveness study 1CommentaryRole of science in the treatment of breast cancer when tumor multicentricity is present Journal of the National Cancer Institute 1Brief communicationEffect of misclassified underlying cause of death on survival estimates of colon and rectal cancer 1Original articleDo reasons for major depression act as causes?Molecular Psychiatry 2ReviewNature Reviews Neurology 1Reviews and perspectivesRegenerative medicine and the gutGastroenterology 2CommunicationsLuminescent graphene quantum dots for organic photovoltaic devicesJournal of the American Chemical Society 1ArticleWhite light emission and second harmonic generation from secondary group participation (SGP) in a coordination network 1ArticleENDOR/HYSCORE studies of the common intermediate trapped during nitrogenase reduction of N 2H 2, CH 3N 2H, and N 2H 4 support an alternating reaction pathway for N 2 reduction No commercial reuse: See rights and reprints http://www.bmj.com/permissions Subscribe: http://www.bmj.com/subscribe BMJ 2014;348:g1585 doi: 10.1136/bmj.g1585 (Published 5 March 2014) Page 8 of 10 RESEARCH o n 5 A p ril 2 0 2 1 b y g u e st. P ro te cte d b y co p yrig h t. h ttp ://w w w .b m j.co m / B M J: first p u b lish e d a s 1 0 .1 1 3 6 /b m j.g 1 5 8 5 o n 6 M a rch 2 0 1 4 . D o w n lo a d e d fro m http://www.bmj.com/permissions http://www.bmj.com/subscribe http://www.bmj.com/ Table 3 (continued) No of Wikipedia citationsType of articleArticleJournal 1ArticleDielectric and thermal effects on the optical properties of natural dyes: a case study on solvated cyanin 1Photo quiz (answers)A woman with knee pain and soft-tissue calcificationClinical Infectious Diseases 1SoftwareProteopedia—a scientific ‘wiki’ bridging the rift between three-dimensional structure and function of biomacromolecules Genome Biology 1PerspectiveCan we dramatically reduce the incidence of coronary heart disease?Nature Reviews Cardiology 5Special featureBeating the odds—nothing is impossible, its just a road less traveledSchizophrenia Bulletin 1National standards and reviewNational standards for diabetes self-management educationDiabetes Care 1ArticleRfam: Wikipedia, clans and the “decimal” releaseNucleic Acids Research 4PerspectiveThe NCGC pharmaceutical collection: a comprehensive resource of clinically approved drugs enabling repurposing and chemical genomics Science Translational Medicine 3ReviewA review of natural orifice translumenal endoscopic surgery (NOTES) for intra-abdominal surgery: experimental models, techniques, and applicability to the clinical setting Annals of Surgery No commercial reuse: See rights and reprints http://www.bmj.com/permissions Subscribe: http://www.bmj.com/subscribe BMJ 2014;348:g1585 doi: 10.1136/bmj.g1585 (Published 5 March 2014) Page 9 of 10 RESEARCH o n 5 A p ril 2 0 2 1 b y g u e st. P ro te cte d b y co p yrig h t. h ttp ://w w w .b m j.co m / B M J: first p u b lish e d a s 1 0 .1 1 3 6 /b m j.g 1 5 8 5 o n 6 M a rch 2 0 1 4 . D o w n lo a d e d fro m http://www.bmj.com/permissions http://www.bmj.com/subscribe http://www.bmj.com/ Figure Number of articles that cite Wikipedia at least once by year since 2001. *Data from February 2012 to November 2013 were not included in detailed analysis No commercial reuse: See rights and reprints http://www.bmj.com/permissions Subscribe: http://www.bmj.com/subscribe BMJ 2014;348:g1585 doi: 10.1136/bmj.g1585 (Published 5 March 2014) Page 10 of 10 RESEARCH o n 5 A p ril 2 0 2 1 b y g u e st. P ro te cte d b y co p yrig h t. h ttp ://w w w .b m j.co m / B M J: first p u b lish e d a s 1 0 .1 1 3 6 /b m j.g 1 5 8 5 o n 6 M a rch 2 0 1 4 . D o w n lo a d e d fro m http://www.bmj.com/permissions http://www.bmj.com/subscribe http://www.bmj.com/