key: cord-0891463-opfgegin authors: Kahale, Lara A.; Elkhoury, Rayane; El Mikati, Ibrahim; Pardo-Hernandez, Hector; Khamis, Assem M.; Schünemann, Holger J.; Haddaway, Neal R.; Akl, Elie A. title: Tailored PRISMA 2020 flow diagrams for living systematic reviews: a methodological survey and a proposal date: 2022-01-28 journal: F1000Res DOI: 10.12688/f1000research.51723.3 sha: 6b3167b44a2515d6518fdb0095425e21832ad110 doc_id: 891463 cord_uid: opfgegin Background: While the PRISMA flow diagram is widely used for reporting standard systematic reviews (SRs), it was not designed for capturing the results of continual searches for studies in living systematic reviews (LSRs). The objectives of this study are (1) to assess how published LSRs report on the flow of studies through the different phases of the review for the different updates; (2) to propose an approach to reporting on that flow. Methods: For objective 1, we identified all LSRs published up to April 2021. We abstracted information regarding their general characteristics and how they reported on search results. For objective 2, we based our proposal for tailored PRISMA approaches on the findings from objective 1, as well as on our experience with conducting Cochrane LSRs. Results: We identified 279 living publications relating to 76 LSRs. Of the 279 publications, 11% were protocols, 23% were base versions (i.e., the first version), 50% were partial updates (i.e., does not include all typical sections of an SR), and 16% were full updates (i.e., includes all typical sections of an SR). We identified six ways to reporting the study flow: base separately, each update separately (38%); numbers not reported (32%); latest update separately, all previous versions combined (20%); base separately, all updates combined (7%); latest update version only (3%); all versions combined (0%). We propose recording in detail the results of the searches to keep track of all identified records. For structuring the flow diagram, we propose using one of four approaches. Conclusion: We identified six ways for reporting the study flow through the different phases of the review for the different update versions. We propose to document in detail the study flow for the different search updates and select one of our four tailored PRISMA diagram approaches to present that study flow. 2, we based our proposal for tailored PRISMA approaches on the findings from objective 1, as well as on our experience with conducting Cochrane LSRs. Results: We identified 279 living publications relating to 76 LSRs. Of the 279 publications, 11% were protocols, 23% were base versions (i.e., the first version), 50% were partial updates (i.e., does not include all typical sections of an SR), and 16% were full updates (i.e., includes all typical sections of an SR). We identified six ways to reporting the study flow: base separately, each update separately (38%); numbers not reported (32%); latest update separately, all previous versions combined (20%); base separately, all updates combined (7%); latest update version only (3%); all versions combined (0%). We propose recording in detail the results of the searches to keep track of all identified records. For structuring the flow diagram, we propose using one of four approaches. Conclusion: We identified six ways for reporting the study flow through the different phases of the review for the different update versions. We propose to document in detail the study flow for the different search updates and select one of our four tailored PRISMA diagram approaches to present that study flow. Any reports and responses or comments on the article can be found at the end of the article. During the coronavirus disease 2019 (COVID-19) pandemic, health research has proliferated exponentially 1 . Systematic reviews are essential to synthesize the evidence and inform policy and practice. Given the pace of research publication, those reviews need to be kept up to date. Living systematic reviews (LSRs) are an emerging type of systematic review that involves the continual search of the literature and incorporation of relevant new evidence, soon after it becomes available 2 . While many evidence synthesis groups are engaged in conducting LSRs or living network meta-analyses, others have developed living databases or living maps, including resources specific for COVID-19 literature [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] . An essential component of systematic reviews is to keep track of and report the number of records captured while searching the scientific literature and details of the selection process 18 . The PRISMA statement recommends the use of the PRISMA flow diagram to depict the flow of studies through the different phases of the systematic review 19 . While the PRISMA flow diagram is a widely used tool for reporting original systematic reviews, it was not designed to capture the results of continual searches typically used in LSRs. Hence, it's unclear how authors of LSRs address the issue of presenting results of these continual searches. The objectives of this study were (1) to assess how published LSRs report on the flow of studies through the different phases of the review for the different updates; and (2) to propose an approach to documenting and reporting on the flow of studies through the different phases of a LSR, for the different updates. For objective 1, we collected relevant data as part of a larger methodological survey aiming to assess the methods of conduct and reporting of LSRs. We have described the details of that study in a previously published protocol 20 . Briefly, we identified all living reviews published up to April 2021 available from the following electronic databases: Medline, EMBASE and the Cochrane library (see extended data 21 of Khamis et al. 20 for the search strategy). An eligible living review was either (1) a protocol for an LSR, (2) a base version of an LSR, (3) a full update version of an LSR, (4) a partial update version of an LSR, or (5) a combination of any of these (e.g., one living review may constitute of a protocol, a base version, and a full update version; another living review may constitute of only a Box 1 the definition of each type of living reviews. We have made the following edits from version 2 to version 3: • We updated the affiliations of co-authors LAK, REK, IM, and NH. • We updated the search from July 2020 to April 2021, our sample now includes 76 LSRs (270 living publications) instead of 32 LSRs (108 living publications). Hence, we updated the methods sections of the abstract and maintext. • We updated the survey findings in Table 1 (characteristics of the living publications), Table 2 (reporting on study flow), and Figure 1 (Summary of the four tailored PRISMA flow diagram approaches) with the analysis of the 76 LSRs (270 living publications). Hence, we updated our results sections of the abstract and main-text. • Our conclusions did not change. • Our suggestions of approaches to documenting and reporting LSR study flow did not change. • We referenced under implications for practice a web-based App that can facilitate producing flow diagrams for LSRs using the four approaches suggested. • We discussed under implications for research our plans and in developing extension to the PRISMA 2020 statement for LSRs. • We did some copy-editing. For objective 2, we base our proposal for tailored PRISMA 2020 flow diagram approaches on the findings from objective 1, on our experience conducting Cochrane LSRs, and our methodological work on designing and reporting living evidence. Since 2017, our group has been responsible for the first series of three Cochrane LSRs, all of which address anticoagulation in patients with cancer 22-24 . We conducted the base search in February 2016. Since then, we have been updating the search on a monthly basis. Through this experience, we have been able to apply and refine the guidance for conducting LSRs endorsed by the living evidence network group 25 . Specifically, we explored solutions for the reporting of the study flow that would address different scenarios. Our goal was not to be prescriptive and narrow, but rather to cover all possible resulting flows by reviewing the LSRs we identified based on objective 1. Two authors developed a draft of the tailored approaches to presenting the study flow, and then circulated to the author team for review and suggestions for improvement. We used REDCap to collect and manage the data abstraction process. All data were exported from REDCap and analyzed using Stata v. 13 26,27 . Our search identified a total of 279 living publications relating to 76 LSRs. Table 1 shows their general characteristics. Of the 279 living publications, 11% were protocols, 23% were base versions, 50% were partial updates, and 16% were full updates. The median number of living publications per LSR was 2 (Interquartile range 1-4). Of the 76 living reviews, 22% were published in the Cochrane library, 63% were related to COVID-19, and 25% had a base version published as a rapid review. The majority were related to clinical topics (70%). Table 2 shows the results for the reporting on the study flow. Most base versions and full updates used a flow diagram to report on the search results (96% and 93% respectively), whereas only one partial update presented a flow diagram. In addition, none of the 279 living publications reported in their methods section how they plan to report on the study flow. Among the 184 update versions ( Figure 1 ): • 21% reported the search results for the base version and for each update version separately. • 11% reported on the search results for the base version separately and for all update versions combined. • 13% reported the search results for the latest update version separately and for all previous versions combined (including the base). • 6% reported the search results for all the different versions combined. • 26% reported the search results for the latest update version only. • 23% did not report the search results at all (e.g., 'new studies identified and integrated' without specifying the number). We show in Figure 2 a snapshot of the 'cumulative' tab of the spreadsheet that keeps track of all records. It shows the study flow for a hypothetical example for an LSR published first in January 2020 (i.e., base version) and updated on a monthly basis up to August 2020. Each row corresponds to a different update version. The columns present the following information for each update (columns B to E): the number of records received, deduplicated, included at title and abstract screening, and included at full-text screening (i.e., newly included reports). Additional columns (F to I) present the distribution of the newly included reports as relating to either: (1) new studies, (2) previously included studies, (3) ongoing (unpublished) studies, or (4) preprints. After manually entering the information in the first five tabs (corresponding to the different search sources) the total is automatically computed in the 'cumulative' tab. The proposed spreadsheet can act as a basis for a tailored PRISMA flow diagram for LSRs. For structuring the flow diagram for LSR, one can select one out of four tailored PRISMA 2020 flow diagram approaches: • Approach 1: presenting the search results of the different versions separately (i.e., base and each update separately) ( Figure 3 ). • Approach 2: presenting the search results for the different versions combined (i.e., including base and all update versions) (Figure 4 ). • Approach 3: presenting the search results for the base version separately, and the results of all update version combined ( Figure 5 ). • Approach 4: presenting the results of the latest update version separately, and the results of all previous versions (including the base) combined ( Figure 6 ). In our Cochrane reviews, we applied the second proposal where we present the results for the different searches combined. This study found that authors of LSRs are not consistent in reporting on the flow of studies through the different phases of the review for the different update versions. Thus, we propose to document in detail the study flow for the different search updates and select one of four tailored PRISMA 2020 flow diagram approaches to present that study flow. To our knowledge, this is the first methodological survey that assesses how LSR authors report on the flow of studies through the different phases of the review for the different update versions of LSRs. In addition, the research expertise on our team covers both living approach and regular updating of traditional SR. We believe that our assessment forms a vital baseline and allows us to propose best practices for visualization options to improve consistency whilst the production of LSRs is still at a relatively early stage. Indeed, this survey is part of a larger methodological survey aiming to assess the methods of conduct and reporting of LSRs 20 , that would allow us to update our findings in the future. Authors tend to produce more partial updates of LSRs rather than continually updating the full systematic review. This might seem like a pragmatic approach particularly for a rapidly growing research field and when methods do not seem to change from one update to another. The heterogeneity observed in the ways LSR authors report on the study flow is likely to be explained by the lack of clear guidance on how to do so. We built our proposal on the PRISMA 2020 flow diagram and provide four approaches to tailor the needs for continual searchers used in LSR. The fourth approach is the closest to the current PRISMA 2020 flow diagram as it presents the results of the latest update version separately and the results of all previous versions (including the base) combined. In addition, we proposed three other different approaches to provide options to LSR authors and publishing journals. Authors should choose one or the other approach based on the number of new citations, presentation preferences, and the impression of what provides the greatest transparency in reporting. Whatever approach one decides to follow, for transparency purposes, the systematic reviewers should ideally archive previous versions of the flow diagram (e.g., in an appendix). One major challenge will be to accommodate a large number of updates in the same diagram; some approaches would work better than others in that case. Also, advanced information technology solutions may allow fitting a large number of updates. A web-based prototype is available that allows readers to explore different reporting options across these four approaches: an R package (https://github.com/nealhaddaway/ livingPRISMAflow) and web-based ShinyApp (https://estech. shinyapps.io/livingprismaflow/) were developed that allow users to enter their own data (e.g., from the spreadsheet suggested above) to produce a bespoke flow diagram according to their desired approach or to create their own interactive diagram that allows readers to toggle between different versions of the same data 28 . Advanced information technology can also be utilized to simplify updating and tracking the change in all LSR sections including the PRISMA diagram. It would be optimal to develop the base version in a certain platform where all SR and LSR sections are reported as units (i.e., title, authors, background, objectives, inclusion criteria, effect estimate for outcome x). With each update and for every unit, the author has the luxury to keep the same text (if no change has occurred) or edit (if change has occurred). Each unit can be updated in a differential speed based on certain criteria. The edits could be highlighted to visualize the change. For a certain section, one would easily have access to the entries in the previous versions and possibly visualize a trend across the different versions (i.e., cross-sectional view for that specific item). For example, dynamic documents can be developed using 'R markdown', a document preparation system, where static text can be combined with in-line code and 'code chunks' that produce instantly updatable documents given a modified input 29 . This study is part of a bigger project aiming to develop extension to the PRISMA 2020 statement for LSRs (please see registration form on EQUATOR network website: Equator Network. PRISMA for LSR -Extension of PRISMA 2020 for living systematic reviews. 2021; Accessed from https://www. equator-network.org/library/reporting-guidelines-under-development/reporting-guidelines-under-development-for-systematic-reviews/#LSR]. This project will pilot the proposed approaches for documenting the study flow and for structuring the living flow diagram. In addition, qualitative studies would be helpful to explore: (1) the feasibility and acceptability by LSR authors, publishers, and users towards the proposal; and (2) what the end-users would like to see in an LSR update. LSR authors are not consistent in reporting the flow of studies through the different phases of the review for the different update versions. We propose to document in detail the study flow for the different search updates. Authors can select one of our four tailored PRISMA 2020 flow diagram approaches to present that study flow until detailed guidance will become available. Improving the reporting of study flow in LSR methodology is essential for incorporating living evidence when developing living guidance, particularly in the context of an urgent response 30,31 . Underlying data All data underlying the results are available as part of the article and no additional source data are required. 1. This is an interesting proposal to solve the problem of study flow reporting in living systematic reviews (LSRs). As LSRs increase in number, the methodology and reporting requirements need to be well described and usable. I am not sure these particular suggested approaches are the most practical. Spreadsheets are not easily inserted into publications, but it is a worthwhile question to be asking, and working from the existing publications is a good starting point. Thank you for your positive feedback and practical advice. We agree that spreadsheets are not easily publishable, that is why we recommend using it as a tool for the LSR authors to document in detail the study flow as opposed to using it as a reporting tool. We now elaborate in the discussion how this spreadsheet would feed in an app (shinyapp) that creates nicely designed flows for these four approaches well suited for insertion into a publication. We agree with this comment. This study is a methodological survey aiming to summarize what LSR authors are currently reporting. As this is not a guidance document, we are not making any recommendations. However, this study is part of a bigger project aiming to develop extension to the PRISMA 2020 statement for LSRs. For that purpose, we will be following the EQUATOR Network's guidance for developing health research reporting guidelines which include, but is not limited to, engaging stakeholders, including methodologists, LSR end-users, Cochrane groups, journal editors, etc. in a Delphi exercise. In addition, we will be doing a scoping review of current guidance documents and methods papers, and a qualitative study with stakeholders. Living Systematic Reviews (LSRs) are updated as new evidence becomes available and gained popularity during the Covid-19 pandemic. This manuscript describes the way PRISMA 2020 flow diagrams are handled in 32 LSRs with a view to recommend how this can best be done. The topic is relevant albeit a bit narrow and the manuscript is written clearly. However, there're some issues that should to be solved in the next version of the manuscript. It's disappointing that no clear recommendation but four alternative recommendations are given without any guidance which one to select when. That sounds a bit like 'anything goes'. The reduction from the six approaches found in the LSRs to date to the four recommended is not very impressive. I was also surprised that the recommendations were solely based on the experience of the authors. Why is no attempt made to consult survey methodologists and end-users of LSRs, e.g. by performing a Delphi study? Also Cochrane Methods Groups and the editors of the Cochrane Handbook seem not to have been approached with a request to state their view on the issue. The findings presented are part of a larger project on the methods of LSRs about which near to nothing is said in the manuscript. That makes one wonder whether this is not too small a part of the harvest to be optimally useful. Please explain why this element on flow charts is separated from the rest. The data set is quite small: 32 LSRs of which 8 are only available as study protocol, 12 have only one (base) version, and 12 have one or more updates. Why is no indication provided of the corresponding imprecision, e.g. by presenting 95% confidence intervals? The bottom half of table 2 presents how the study flow is reported among the 12 LSRs that got at least one update. I recommend to do this for all 32 LSRs included, assuming that when no update is yet reported the envisioned handling of flow charts should be specified in either the review protocol or the base version of the review. Is the study design appropriate and is the work technically sound? Partly Are all the source data underlying the results available to ensure full reproducibility? Yes Network's guidance for developing health research reporting guidelines which include, but is not limited to, engaging stakeholders, including methodologists, LSR end-users, Cochrane groups, journal editors, etc. in Delphi exercise. In addition, we will be doing a scoping review of current guidance documents and methods papers, and a qualitative study with stakeholders. We have elaborated about this research under implications for future research. 2. The findings presented are part of a larger project on the methods of LSRs about which near to nothing is said in the manuscript. That makes one wonder whether this is not too small a part of the harvest to be optimally useful. Please explain why this element on flow charts is separated from the rest. Thank you for your query. This paper is the first published study within the larger project which aims to explore how LSRs authors are currently reporting, conducting, and publishing LSRs. We cite the protocol of that project (Khamis 2019). This study focuses on reporting flow diagrams, another study will focus on general characteristics and delay in updating LSRs, and a third study will focus on methodological features of LSRs. We have now updated our results to include data from up to April 2021, and our dataset now includes 279 publications of 76 LSRs. Thank you for your comment. Unfortunately, that was the state of the science. However, as noted in the previous comment, we have now updated our search to include results up to April 2021 Including data from 279 publications of 76 LSRs. We avoided including confidence intervals for the simplicity of presentation and given the small sample size. We are happy to include these if the editor prefers so. A large number of COVID-19 interventional clinical trials were registered soon after the pandemic onset: a descriptive analysis Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap The impact of the COVID-19 pandemic on selfharm and suicidal behaviour: update of living systematic review Protocol for a "living" evidence summary of primary research related to Covid-19 Repurposing Existing Medications for Coronavirus Disease 2019: Protocol for a Rapid and Living Systematic Review. medRxiv. 2020. Publisher Full Text Interventions for treatment of COVID-19: a protocol for a living systematic review with network meta-analysis including individual patient data (The LIVING Project) A living systematic review protocol for COVID-19 clinical trial registrations Baseline results of a living systematic review for COVID-19 clinical trial registrations Ventilation techniques and risk for transmission of coronavirus disease, including COVID-19: a living systematic review of multiple streams of evidence PubMed Abstract | Publisher Full Text | Free Full Text We would like to acknowledge Dr. Mathew Page for his revision for the manuscript. College of Medicine and Public Health, Flinders University, Alice Springs, Northern Territory, Australia This is an interesting proposal to solve the problem of study flow reporting in living systematic reviews (LSRs). As LSRs increase in number, the methodology and reporting requirements need to be well described and usable.I am not sure these particular suggested approaches are the most practical. Spreadsheets are not easily inserted into publications, but it is a worthwhile question to be asking, and working from the existing publications is a good starting point.The authors have done what they set out to do, but I suggest further work is needed before a recommended method of study flow reporting is settled upon. Are all the source data underlying the results available to ensure full reproducibility? Yes Competing Interests: No competing interests were disclosed.Reviewer Expertise: Systematic reviews, evidence-based practice. Author Response 07 Jan 2022 Lara Kahale, American University of Beirut, Beirut, UKThe Reviewers' comments are in bold font and our replies in regular font. Extracts from the text are in italic fonts with changes underlined. We have indicated the sections where revisions have been made in our manuscript. The Reviewers' comments are in bold font and our replies in regular font. Extracts from the text are in italic fonts with changes underlined. We have indicated the sections where revisions have been made in our manuscript. Living Systematic Reviews (LSRs) are updated as new evidence becomes available and gained popularity during the Covid-19 pandemic. This manuscript describes the way PRISMA 2020 flow diagrams are handled in 32 LSRs with a view to recommend how this can best be done. The topic is relevant albeit a bit narrow and the manuscript is written clearly. However, there're some issues that should to be solved in the next version of the manuscript. 1. It's disappointing that no clear recommendation but four alternative recommendations are given without any guidance which one to select when. That sounds a bit like 'anything goes'. The reduction from the six approaches found in the LSRs to date to the four recommended is not very impressive. I was also surprised that the recommendations were solely based on the experience of the authors. Why is no attempt made to consult survey methodologists and end-users of LSRs, e.g. by performing a Delphi study? Also Cochrane Methods Groups and the editors of the Cochrane Handbook seem not to have been approached with a request to state their view on the issue.Thank you for your comment. This study is a methodological survey aiming to summarize what current LSR authors are reporting. Given that the full methodology of living systematic reviews is still emerging, we prefer not to make firm recommendations but lay out options.