The Increasing Importance of Systematic Reviews in Clinical Dermatology Research and Publication EDITORIAL © 2006 The Society for Investigative Dermatology www.jidonline.org 2357 The Increasing Importance of Systematic Reviews in Clinical Dermatology Research and Publication The number of published systematic reviews is growing rapidly, and such reviews are receiving increased attention from sci- entists, editors, policy makers, and consumers. Unlike most traditional review articles, quality systematic reviews use an explicit and systematic predefined methodology to minimize bias and to increase the precision of measurements of treatment effects. Yet methodological flaws can exist in systematic reviews that limit their utility. To reduce unnecessary duplication of clinical trials and ensure that scarce resources address the most pressing research needs, institutional review board panels and dermatology journals should consider requiring systematic reviews or reference to existing systematic reviews as a pre- requisite for protocol approval and publication of clinical trials. What are systematic reviews? Systematic reviews differ from traditional narrative expert- opinion reviews in their comprehensive sys- tematic approach to summarizing health-care evidence (Levin, 2001) and are now required by some biomedical journals (for example, The Lancet) for clinical trial publication (Young and Horton, 2005). The systematic approach starts with formulation of a clear question, followed by a thorough search for all relevant evidence, a critical appraisal of that evidence using predeter- mined criteria, quantitative pooling (metaanaly- sis) of similar study results when appropriate, and finally, interpretation of that evidence (Bigby et al., 2003). Authors of systematic reviews con- ducted within the Cochrane Collaboration make a commitment to update the review periodically with the goal of fine-tuning the research in light of any new studies that come along. If performed correctly, these reviews can provide complete, up-to-date, and unbiased measurements of treat- ment effectiveness by using metaanalytical sta- tistics. High-quality systematic reviews may also indicate whether existing evidence is consistent and can be generalized across patient popula- tions or variations in treatment (Collier et al., 2005; Parker et al., 2004). Why might systematic reviews be important? Results of isolated randomized controlled trials are frequently contradicted by subsequent stud- ies (Ioannidis, 2005). Under the most rigorous study design conditions, a well-planned single study, even if prospective and randomized, rarely provides definitive results, and primary studies tend to overestimate treatment effects. A recent study reported that 32% of a set of studies with at least 1,000 citations were either contradicted by or reported stronger effects than subsequent studies (Ioannidis, 2005). Relying on single high-profile clinical trials can therefore be harmful to patients’ health. Well-designed randomized controlled trials are excellent when looking at effectiveness, though many fall short in quality reporting of safety and adverse events associated with an inter- vention. Quality systematic reviews often have increased power and decreased bias as com- pared with the individual studies they include, and the careful pooling of treatment effects can provide the most accurate overall assessment of an intervention. Benefits of systematic reviews in biomedical literature. Systematic reviews may save both lives and resources. In addition to adopt- ing a systematic approach that minimizes bias, systematic reviews of all published and unpublished randomized controlled trials also have the potential through appropriate use of metaanalysis to produce a more precise estimate of treatment effect so that small but clinically important effects become apparent among a group of apparently conflicting sin- gle trials. The human cost of failing to produce such systematic reviews is illustrated by the Journal of Investigative Dermatology (2006) 126, 2357– 2360. doi:10.1038/sj.jid.5700560 2358 Journal of Investigative Dermatology (2006), Volume 126 © 2006 The Society for Investigative Dermatology life-saving potential of a systematic review conducted to examine infant sleeping position and sudden infant death syndrome. Although advice to place infants on their backs to sleep was widely available in the early 1990s, the authors of the sudden infant death syndrome study showed that the mortality benefit of this sleep position would have been apparent if a systematic review had been performed any time after 1970 (Gilbert et al., 2005). Such a review potentially could have saved 60,000 infant lives in the United Kingdom, Europe, the United States, and Australia. Quality systematic reviews also possess the potential to save biomedical resources. Cumulative metaanalytical tech- niques on 64 trials investigating the effectiveness of aprotinin showed that the effectiveness of the drug was apparent after only 12 trials (Fergusson et al., 2005). Thus the systematic review of aprotinin and perioperative bleeding identified 52 unnecessary trials. Had a systematic review been performed after the twelfth study, the treatment effect would have been apparent, duplicate trials would have been avoided, and patients would have experienced the benefit of a useful drug ten years earlier. Antenatal corticosteroid therapy for fetal lung matura- tion reduces mortality, respiratory distress syndrome, and intraventricular hemorrhage in preterm infants (Crowley, 2000). Although corticosteroids are routinely used today to accelerate fetal lung maturity in infants at risk for prema- ture delivery, the medical community did not embrace this treatment unanimously during the 1970s and 1980s despite repeated randomized trials providing evidence supporting their use. A systematic review provided incontrovertible evidence in favor of antenatal corticosteroid therapy and revealed that tens of thousands of premature babies have needed more extensive therapy, suffered, and died unneces- sarily (Crowley, 2000). These are just three examples of the costs of failure to perform systematic, up-to-date reviews of randomized con- trolled trials of health care in areas of medicine outside of dermatology. The question remains, therefore, to what extent such messages apply to dermatology. Systematic reviews in dermatology Risk of melanoma from indoor UV tanning. Systematic reviews may reveal clear outcomes when individual stud- ies report widely varying, and even contrasting, results. This point is illustrated by a recent systematic review performed to examine the risk of malignant melanoma in relation to artificial UV radiation (Gallagher et al., 2005). Because exposure to artificial UV radiation through sunlamp and sunbed use may be intense and intermittent, concern arose that these devices might increase the risk of developing melanoma (Elwood and Jopson, 1997). Studies performed to identify any potential risk of developing melanoma from exposure to sunless tanning devices have, however, includ- ed low numbers of exposed individuals and have reported inconclusive results (Elwood and Jopson, 1997). By com- bining individual study results, Gallagher and colleagues (2005) were able to identify significant increased risk of melanoma in people exposed to artificial tanning devices. Trials of topical immunomodulating medications and the “me-too” phenomenon. Systematic reviews may suppress the “me-too” phenomenon common in today’s industry- driven research environment. New medications are marketed heavily to physicians and consumers. Trials of new pharma- ceutical agents often fail to compare crucial active compara- tors with the study drug. Licensing systems for medicines in Europe and the United States only require new drugs to show efficacy above placebo and vehicle, leading to a large influx of new medications to the market with considerable increas- ing costs to government drug budgets (Morgan et al., 2005). Results from studies lacking comparison with traditional, often cheaper, therapies lead to confusion within the medi- cal community regarding the efficacy of new agents. Simply because a newer medication is superior to a placebo does not mean that it is better than traditional, standard-of-care therapy. To examine the trials used to support the use of a new class of medication for the treatment of atopic derma- titis, a systematic review combined data from randomized controlled trials to determine whether topical pimecrolimus was more effective than other treatments (Ashcroft et al., 2005). Of 11 trials using pimecrolimus, eight compared the drug with vehicle only. Strikingly, none of the trials includ- ed a comparison with what is perhaps the most appropriate active comparator for mild atopic dermatitis, twice-daily 1% hydrocortisone. Mirroring the aprotinin example, the efficacy of pimecrolimus as compared with vehicle at 3 weeks was evident after completion of three trials. Although additional trials are sometimes required to demonstrate efficacy in differ- ent groups, such as adults versus children or different ethnic groups, and government drug administrations may require the repetition of clinical trials, some of the remaining trials could be viewed as unnecessary and a waste of research resources. Further studies are not needed to establish the effectiveness of pimecrolimus as compared with placebo. Limitations of systematic reviews. Like all scientific research methodologies, systematic reviews have limitations. Extensive labor is involved in the creation of a quality sys- tematic review, and the finished product can be cumbersome to read for those not familiar with review methodology. One study reported that the median length of a Cochrane system- atic review was 15 printed pages, and a substantial number exceeded 30 pages (Johansen et al., 2001). Funding for these time-consuming systematic reviews is currently limited, and most are done by volunteers interested in performing them. Access to quality reviews is limited too, as databases of sys- tematic reviews like the Cochrane Database are available online only by subscription in some countries that have not arranged national procurement. The methodology associated with conducting a system- atic review is complex, and expertise is required at each step, from question creation to data collection, analysis and interpretation. Errors can occur at any step in this complex process and potentially can lead to meaningless and misin- terpreted data. Disease- and intervention-definition method- ology used prior to the collection and analysis of data can greatly influence which studies are ultimately included. EDITORIAL © 2006 The Society for Investigative Dermatology www.jidonline.org 2359 EDITORIAL Systematic review questions must be designed carefully so that the greatest number of appropriate trials is included in the final analysis. Therefore, quality control is essential. A common concern expressed by those familiar with systematic reviews is the effect heterogeneity may have on review outcomes. Heterogeneity in a systematic review can be described as a measure of the intervention variability and study variation that exist among included trials. It can be dif- ficult to know when it is appropriate to combine results from included trials. The decision to do so should rest on a solid understanding of trial heterogeneity. A critical eye is needed for the interpretation of the results of a systematic review, one that is better able to focus on the results by looking through sometimes complicated methodology. To be representative of all available evidence, systematic reviews must be periodically updated. Updating has been defined as a discrete event with the aim of searching for and identifying new evidence to incorporate into a previously completed systematic review (Moher and Tsertsvadze, 2006). It is also often difficult to determine when it is appropriate to update a systematic review, as this decision must rest on an analysis of information evolution and the quantity of avail- able studies. The changing biomedical environment and systematic reviews. The number of systematic reviews has grown exponentially since 1975 (Egger et al., 2001), and a grow- ing number of agencies are performing and using systematic reviews (Atkins et al., 2005). Systematic reviews now have an increased presence in policy at the government level, with reviews influencing coverage decisions made by the Centers for Medicare and Medicaid Services, consensus conferenc- es, and other policy initiatives (Atkins et al., 2005; Tugwell et al., 2006b; Moynihan, 2004). Through the creation of an Equity Group, the Campbell Collaboration Equity Methods Group and the Cochrane Collaboration Equity Field aim to use systematic reviews to improve health disparities world- wide (Tugwell et al., 2006a). Systematic reviews epitomize evidence-based medicine; they are regarded by some as the pinnacle of the evidence hierarchy (Guyatt et al., 1995) and are more cited than any other study design (Patsopoulos et al., 2005). Weighing the pros and cons: should dermatology jour- nals insist on systematic reviews? Using systematic reviews to link past to present research improves study quality and effectiveness and may save lives and resources, but extensive labor is involved in performing, publishing, and updating a systematic review. This extra work would especially be noticeable in the field of dermatology, where the numbers of systematic reviews published to date are few — only 3% of all Cochrane Reviews from a search of the Cochrane Database of Systematic Reviews and the Database of Abstracts of Reviews of Effectiveness were deemed relevant to dermatol- ogy (Parker et al., 2004). The requirement that all new studies be preceded by a systematic review also creates extra work for the journals that must ensure this before publication of new information. Given the paucity of high-quality systematic reviews in dermatology, insisting on systematic reviews could impart an extensive labor requirement on investigators and potentially serve as an impediment to the publication of current ongo- ing studies. Because journal submission and peer and edi- torial review are late steps in the publication process, more efficient policy might involve requiring systematic review for approval at the institutional review board level. Some govern- ment bodies, such as the United Kingdom Medical Research Council, already insist on a systematic review or reference to a published one before considering funding new clinical trials. This requirement puts the proposed research into the context of the existing body of medical evidence and helps to avoid duplicative study. Although systematic reviews are powerful tools, caution must be exercised in the consideration of policy change that could potentially hinder clinically useful research. Insisting that all new studies should be preceded by a systematic review may be a bit drastic for dermatology at this time. Insisting on the need to mention whether or not a system- atic review has been done, and how the existing body of evi- dence dictated the need for a new study, does seem entirely reasonable and achievable, however — a move that could discourage the nth placebo-controlled trial on another “me- too” product. Current publication methodology may lead to selective publication of results that are more favorable to study spon- sors and to publication of trials that deviate from original study protocols (Al-Marzouki et al., 2005; Smith, 2005). A newly proposed system for reporting clinical trials would require the posting of a systematic review on the Internet as well as any subsequent study protocols related to the review (Smith and Roberts, 2006). This system would be freely accessible to patients, researchers, and editors and would (1) force investigators to follow the original trial design, thereby preventing selective publication of results and misleading post hoc analyses; (2) allow for feedback at any stage of the trial; (3) allow research teams contemplat- ing undertaking new trials to see whether their proposed work has already been done; and (4) permit those report- ing completed trials to refer to key ongoing trials. Similarly, more efforts need to be directed at registering all clinical tri- als prospectively in a publicly searchable database such as Current Controlled Trials (http://www.controlled-trials.com) or the Cochrane Skin Group’s trial register (http://www. nottingham.ac.uk/ongoingskintrials/), a free resource that has recently been updated to incorporate the World Health Organization’s latest recommendations for trials registra- tion. The Journal of Investigative Dermatology requires reg- istration of all clinical trials that started enrollment after July 1, 2005 prior to publication consideration (Williams and Stern, 2005). The Journal of Investigative Dermatology fol- lows International Committee of Medical Journal Editors standards and information regarding this policy can be found at the journal's website (http://www.nature.com/jid/ author_instructions.html#clinical-trials-registration). Redundant studies waste valuable resources, including those of the funding body, researchers, and, most importantly, 2360 Journal of Investigative Dermatology (2006), Volume 126 © 2006 The Society for Investigative Dermatology patients, who usually participate in clinical studies for altru- istic reasons. A policy at the journal or institutional review board level requiring any new study to relate its findings to the body of existing evidence should help to define the need for the study and sharpen its methodology. This requirement would help dissuade investigators from performing duplica- tive trials to better ensure that scarce resources available for clinical dermatology research are targeted toward the most urgent research gaps. CONFLICT OF INTEREST The authors state no conflict of interest but are all enthusiastic participants in the Cochrane Collaboration. ACKNOWLEDGMENTS This work was supported by grant T32 ARO7411 from the US National Institutes of Health, Bethesda, Maryland, USA (to S.R.F.); grant K07 CA92550 from the National Cancer Institute, Bethesda (to R.P.D.); the University of Colorado Cancer Center; and the United Kingdom National Health Service Research and Development Programme (to H.C.W.). Scott R. Freeman1, Hywel C. Williams2 and Robert P. Dellavalle1,3 1University of Colorado at Denver and Health Sciences Center, Department of Dermatology Research Laboratories, Aurora, Colorado, USA; 2Centre of Evidence-Based Dermatology, Nottingham University Hospital National Health Service Trust, University of Nottingham, Nottingham, United Kingdom; and 3Department of Veterans Affairs Medical Center, Dermatology Service, Denver, Colorado, USA Correspondence: Dr. Robert P. Dellavalle, Department of Veterans Affairs Medical Center, Dermatology Service, 1055 Clermont Street, #165, Denver, Colorado 80220, USA. E-mail: robert.dellavalle@uchsc.edu REFERENCES Al-Marzouki S, Roberts I, Marshall T, Evans S (2005) The effect of scientific misconduct on the results of clinical trials: a Delphi survey. Contemp Clin Trials 26:331–7 Ashcroft DM, Dimmock P, Garside R, Stein K, Williams HC (2005) Efficacy and tolerability of topical pimecrolimus and tacrolimus in the treatment of atopic dermatitis: meta-analysis of randomised controlled trials. BMJ 330:516 Atkins D, Fink K, Slutsky J (2005) Better information for better health care: the Evidence-based Practice Center program and the Agency for Healthcare Research and Quality. Ann Intern Med 142:1035–41 Bigby M, Williams H (2003) How to critically appraise systematic reviews and meta-analyses. In: Evidence-Based Dermatology. (Bigby M, Williams H, Diepgen T, Herxheimer A, Naldi L, Rzany B eds) BMJ Publishing Group: London, pp 49–55 Collier A, Johnson K, Heilig L, Leonard T, Williams H, Dellavalle RP (2005) A win-win proposition: fostering consumer involvement in the Cochrane Collaboration Skin Group. J Am Acad Dermatol 53:920–1 Crowley P (2000) Prophylactic corticosteroids for preterm birth. Cochrane Database Syst Rev 2:CD000065, doi:10.1002/14651858 Egger M, Smith GD, Altman D (eds) (2001) Systematic Reviews in Health Care: Meta-analysis on Context. BMJ Books: London, p 4 Elwood JM, Jopson J (1997) Melanoma and sun exposure: an overview of published studies. Int J Cancer 73:198–203 Fergusson D, Glass K, Hutton B, Shapiro S (2005) Randomized controlled trials of aprotinin in cardiac surgery: could clinical equipoise have stopped the bleeding? Clin Trials 2:218–32 Gallagher RP, Spinelli JJ, Lee TK (2005) Tanning beds, sunlamps, and risk of cutaneous malignant melanoma. Cancer Epidemiol Biomarkers Prev 14:562–6 Gilbert R, Salanti G, Harden M, See S (2005) Infant sleeping position and the sudden infant death syndrome: systematic review of observational studies and historical review of recommendations from 1940 to 2002. Int J Epidemiol 34:874–87 Guyatt GH, Sackett DL, Sinclair JC, Hayward R, Cook DJ, Cook RJ (1995) User’s guides to the medical literature. IX. A method for grading health care recommendations. JAMA 274:1800–4 Ioannidis JP (2005) Contradicted and initially stronger effects in highly cited clinical research. JAMA 294:218–28 Johansen HK, Moustgaard R, Gotzsche PC (2001) Anatomy of the Cochrane Review. 9th International Cochrane Colloquium; October 2001; Lyon, France. Oral presentation. Cochrane 1:op008 Levin A (2001) The Cochrane Collaboration. Ann Intern Med 135:309–12 Moher D, Tsertsvadze A (2006) Systematic reviews: when is an update an update? Lancet 367:881–3 Morgan SG, Bassett KL, Wright JM, Evans RG, Barer ML, Caetano PA et al. (2005) “Breakthrough” drugs and growth in expenditure on prescription drugs in Canada. BMJ 331:815–6 Moynihan R (2004) Evaluating Health Services: a Reporter Covers the Science of Research Synthesis. Milbank Memorial Fund: New York. Available at http://www.milbank.org/reports/2004Moynihan/040330Moynihan.html Parker ER, Schilling LM, Diba V, Williams HC, Dellavalle RP (2004) What is the point of databases of reviews for dermatology if all they compile is “insufficient evidence”? J Am Acad Dermatol 50:635–9 Patsopoulos NA, Analatos AA, Ioannidis JP (2005) Relative citation of various study designs in the health sciences. JAMA 293:2362–6 Smith R (2005) Medical journals are an extension of the marketing arm of pharmaceutical companies. PLoS Med 2:e138, doi:10.1371/journal. pmed.0020138 Smith R, Roberts I (2006) Patient safety requires a new way to publish clinical trials. PLoS Clinical Trials 1:e6, doi:10.1371/journal.pctr.0010006 Tugwell P, Petticrew M, Robinson V, Kristjansson E, Maxwell L, Cochrane Equity Field Editorial Team (2006a) Cochrane and Campbell Collaborations, and health equity. Lancet 367:1128–30 Tugwell P, Robinson V, Grimshaw J, Santesso N (2006b) Systematic reviews and knowledge translation. Bull World Health Organ 84:643–51 Williams HC, Stern RS (2005) Prospective Clinical Trial Registration. Journal of Investigative Dermatology (2005) 124, viii–x; doi:10.1111/j.0022– 202X.2005.23665.x Young C, Horton R (2005) Putting clinical trials into context. Lancet 366:107–8 EDITORIAL