key: cord-0815560-wo4ngl0a
authors: Iannizzi, Claire; Dorando, Elena; Burns, Jacob; Weibel, Stephanie; Dooley, Clare; Wakeford, Helen; Estcourt, Lise J; Skoetz, Nicole; Piechotta, Vanessa
title: Methodological challenges for living systematic reviews conducted during the COVID-19 pandemic: a concept paper
date: 2021-09-12
journal: J Clin Epidemiol
DOI: 10.1016/j.jclinepi.2021.09.013
sha: af04ba883b98712cf4950dd10ed38b60f4549473
doc_id: 815560
cord_uid: wo4ngl0a

Background: A living systematic review (LSR) is an emerging review type that makes use of continual updating. In the COVID-19 pandemic, we were confronted with a shifting epidemiological landscape, clinical uncertainties and evolving evidence. These unexpected challenges compelled us to amend standard LSR methodology. Objective and outline: Our primary objective is to discuss some challenges faced when conducting LSRs in the context of the COVID-19 pandemic, and to provide methodological guidance for others doing similar work. Based on our experience and lessons learned from two Cochrane LSRs and challenges identified in several non-Cochrane LSRs, we highlight methodological considerations, particularly with regards to the study design, interventions and comparators, changes in outcome measure, and the search strategy. We discuss when to update, or rather when not to update the review, and the importance of transparency when reporting changes. Lessons learned and conclusions: We learned that a LSR is a very suitable review type for the pandemic context, even in the face of new methodological and clinical challenges. Our experience showed that the decision for updating a LSR depends not only on the evolving disease or emerging evidence, but also on the individual review question and the review teams’ resources.

A living systematic review (LSR) is an emerging systematic review type, which makes use of continual updating and ongoing surveillance of emerging research evidence [1] . Regular searches ensure that the systematic review includes the latest available findings and remains up to date [1] . Therefore, LSRs are most suitable for high-priority topics with substantial uncertainty, and where new evidence is published regularly [1] . In a series of four papers, the various aspects of LSRs have been discussed and elaborated on in detail [1] [2] [3] [4] . Cochrane published the first version of a Cochrane LSR series in 2017 [5] , and in 2019 released guidance on the conduct and publication of Cochrane LSRs [6] .

In the context of the Coronavirus disease 2019 (COVID-19) pandemic, we are confronted with a shifting epidemiological landscape, clinical uncertainties, a lack of evidence and a rapidly evolving evidence base. As methodologists conducting LSRs during the pandemic, we have recognised the need and opportunity to respond to new and unexpected challenges by amending our standard systematic review methodology.

Our primary objective is to discuss some of the challenges faced when conducting LSRs in the context of the COVID-19 pandemic, and to provide methodological guidance for others doing similar work.

To accomplish these objectives we draw on the experience and lessons learned from the author teams of two Cochrane LSRs [7, 8] and the methodological approaches used in other selected LSRs [9] [10] [11] [12] [13] [14] . When referring to 'our' experiences, we refer either to 'review one', investigating convalescent plasma for COVID-19 treatment [7] or 'review two', investigating international travel-related control measures for containing the COVID-19 pandemic [8] . The additional six LSRs were selected based on several characteristics (journal, complexity of methodology and topic, number of included studies and update strategy) to cover a broad variety in terms of LSR characteristics, see supplementary table 1. We highlight methodological considerations related to when a living review question is reasonable, particularly with regards to study designs, types of interventions and comparators, changes in outcome measures and the search strategy. We discuss when to update, or rather when not to update the review and the importance of transparency when reporting methodological changes.

To address the uncertainties related to COVID-19 research and adapt to the evolving evidence landscape, certain methodological elements needed special consideration for ensuring that LSRs are a reliable, up-to-date source of evidence that respond to the urgent health situation.

Our experiences and further methodological approaches identified through other LSRs are elaborated in the following sections and summarized in Traditionally, evidence-based medicine has applied a hierarchy of evidence according to study design in order to achieve an adequate quality of evidence in systematic reviews and draw meaningful and valid conclusions. For standard intervention reviews, for example, randomised controlled trials (RCTs) are at the top of this hierarchy, followed by cohort studies and case-control studies in the middle of the evidence pyramid, and case series or reports at the bottom [15] .

As response to clinical uncertainties in the COVID-19 pandemic and to ensure that no relevant evidence was excluded, in conducting review one we initially started with broad study design inclusion criteria. We planned to include RCTs preferentially, and to include other study designs, e.g. observational studies, only if insufficient RCT-evidence was available. We had to eventually adapt this initial plan, as we soon realised that refining inclusion criteria is an interactive process [7] . Identified studies did not report data for all our review outcomes, and in particular, some of the RCTs did not report safety data for the control group. Thus, for a better understanding of the frequency of unintended effects, we made the post-hoc decision to also include safety data from prospectively registered controlled and uncontrolled studies. Similarly, other selectively identified LSRs included observational studies at an early stage of the COVID-19 pandemic due to a paucity of RCTs [7, [9] [10] [11] 13] . The lesson learned here is that authors should always rely on the best available evidence, which may be dependent on the outcome and will likely evolve and change rapidly over time. As LSR authors, we aim to synthesise and critically appraise all available evidence at a given time, but to update the review as more and potentially more trustworthy evidence becomes. For example, we have seen that observational studies reported on positive outcomes for several interventions, e.g. convalescent plasma [7] or hydroxychloroquine [11] , but were later shown to have little or no therapeutic effect against COVID-19 in higher quality studies and systematic review updates.

Generally ignored by the classical hierarchy of evidence, modelling studies have rarely been used in answering questions of intervention effectiveness, with systematic reviews focusing instead on experimental and sometimes observational evidence. This lack of consideration of modelling studies is at least partially because such studies simulate data on interventions and/or outcomes, which often require multiple sometimes questionable assumptions, rather than observing and measuring them directly. For some COVID-19 questions, modelling studies represented the sole evidence source, and it became clear early on, that decisionmakers, despite the limitations of such studies, were using such studies to inform decisions. In the first version of review two [8] , we included any type of modelling study due to the lack of experimental and even observational evidence.

Separate from study design, we also discussed which types of publication to consider for our reviews, e.g. peer-reviewed articles, preprints, abstracts, letters, etc. The experiences during this pandemic have shown the risks and benefits of using preprints, i.e. prompt availability versus validity and reliability (or lack thereof). Due to the prompt availability, several LSRs included preprints [7] [8] [9] [10] [11] [12] [13] [14] . However, preprints must be handled with caution as they are not peer reviewed and results might still change [7-9, 12, 13] . Using preliminary preprint findings instead of data from the most updated preprint version, or the peer-reviewed journal publication could lead to different results or implications for review updates. It is challenging to identify updates of preprints, especially when the DOI remains unchanged [9] [10] [11] [12] [13] [14] . As soon as the full-text journal publications became available, some review authors prioritised these and reassessed the preprints [7, 8, 12, 13] . To address remaining uncertainties, sensitivity analysis excluding preprints can be helpful to investigate the robustness of results [7] .

We learned that the interventions and comparators assessed by LSRs in the pandemic context evolved over time and needed adaptations. According to the Cochrane Handbook for Systemic Reviews of Interventions, it is important to consider and minimise the clinical and methodological heterogeneity between studies, to allow a valid comparison and a reliable pooled effect [16] . For review one [7] , our overall main comparison remained unchanged; however we did adapt how we defined the specific intervention and comparator. Specifically, we noticed that, because there was and is no real standard care available, the best supportive care options differed widely across contexts. For instance, we observed that Chinese studies often used Traditional Chinese Medicine as part of patient care [17] . Studies that were initiated early in the pandemic often used hydroxychloroquine [18] , and studies that recruited patients after July 2020 often used corticosteroids [19] . Another challenge was that for most co-interventions the evidence on safety and effectiveness remained uncertain. Hence, we tried to account for bias due to unequal distribution of co-interventions across study groups. We analysed individuals with mild and moderate to severe symptoms separately, based on existing hypotheses regarding the intervention modes of activity and our evolving understanding of COVID-19 progression to assure comparability of study participants [7] .

At the beginning and throughout the course of the pandemic, robust and relevant outcome measures were not clear. We based our outcome selection on the COMET (Core Outcome Measures in Effectiveness Trials) initiative for COVID-19 patients [20] . As more evidence became available, we found that outcome measures needed to be refined. Thus, our outcome set was never "final" but constantly evolving. We noted for instance in review one [7] , that there was broad diversity in the assessment and reporting of the clinical status or disease progression, with standard reporting measures changing over time as well. This increased the potential for heterogeneity in outcome measurement and reporting across studies. We could not find a solution to reasonably combine data and provided narrative syntheses without metaanalysis for respective outcomes. Changing outcomes were also identified in the Juul review, which added an additional post hoc outcome for their update [9] .

We used the Cochrane guidance for LSR search methods to develop our initial search strategy. According to this guidance, there is a particular interest for LSRs to keep ongoing and emerging evidence up to date through regular searches of electronic databases, clinical trial registries and other potential sources [6] . The search strategies also need to be updated, as relevant terms, keywords or database filters may change [6] .

One challenge for review two, related to maintaining searches over time [8] . The changing database landscape required constant amendments to the search strategy and literature

Research Articles Downloadable Database, an early and comprehensive source of pre-print articles, was discontinued in mid-2020, but is now completely covered by the WHO COVID-19 Global literature on coronavirus disease database. A challenge for review one involved the dynamic nature of electronic databases [7] , with existing databases changing and new ones becoming available. Therefore, it was not sufficient to rely only on the traditionally utilised databases such as PubMed, Embase or CENTRAL. Some reviews also explored new COVID-updated public database for study references, particularly efficient for update searches of LSRs [21] . Also, the L*VE platform, was used in some LSRs [7, 10, 14] . The information specialists involved in review one used the website of "COVID-END" [22] and the EPPI Centre [23] , providing guidance and listing the various COVID-19 registries, to get an overview of the numerous newly available and often overlapping registries. Another challenge identified in review one was that no suitable screening software exists to respond to the evolving inclusion criteria [7] . Because of the rapid emergence of new evidence, most LSRs decided to run a complete search each week [7, 9, 10, 14] or month [11] .

A further challenge with review one was tracking the ongoing studies [7] , as the estimated study completion dates indicated in the study registries were sometimes unreliable. Therefore, it was of utmost importance to track ongoing studies through regular contact with the main investigators. For previously identified ongoing platform trials [7] or preprints [9] [10] [11] [12] [13] [14] some authors decided to perform regular manual checks for new updates.

When to update is an important issue to discuss when planning and conducting a LSR, and is highly context dependent. There is no clear standard for how frequent or at which time point updates of LSRs should be performed and published [1] . According to Cochrane, updates can be planned either when it is likely that newly identified evidence has an impact on the review conclusions or at a fixed-interval schedule when more emerging evidence is expected [6] . The panel for updating guidance for systematic reviews recommends an individualised updating approach, where the responsibility for the update decision depends on the personal resources of the authors and the editorial team [24] . The Annals of Internal Medicine provides detailed author guidance on updating and publishing paths for LSRs, which suggests committing to publish either surveillance comments, alerts of new evidence or a new article with major updates [25] .

In the context of the COVID-19 pandemic, we defined additional components to consider when deciding on updating a LSR. Based on a figure from the Cochrane LSR guidance illustrating the LSR workflow [6] , we reproduced a similar figure and adapted it according to our experience (blue components), to visualise the decision process for updating or publishing a LSR (see Figure 1 ). The part of the flowchart that we altered most relates to running the search. Here we added three additional considerations that can affect the decision of whether to update, publish, neither or both: the policy relevance, the importance of the study and funding. Each of these three considerations is described in more detail below. Authorization and statement by the US Food and Drug Administration (FDA) that convalescent plasma may be effective in treating COVID-19 was an important consideration [26] in deciding to publish an update of review one [7] , even though the conclusion of our review did not change after considering additional study data. Regarding review two [8] , the development of WHO guidance on COVID-19 mitigation in the aviation sector heavily influenced the decision of when to update [27] .

We also added "waiting for important study" as an additional aspect to consider in the updating decision. As ongoing studies identified in previous versions of review one came to completion [7] , we faced the decision of when to update. In order to make optimal use of resources, we ultimately decided to tie our updates to the completion of larger, well-planned studies, e.g. platform trials. These were most likely to produce higher-certainty evidence, and we felt that an update including such studies would be of optimal value to the end user.

Through regular communication with study investigators, we were able to identify when an 'important' study for our PICO was going to be published. For review two [8] , and potentially for other less clinical, more public health-type PICOs, diverse modelling studies and smaller observational studies can be important. The author guidance of the annals of internal medicine for instance suggests a "major update" when new evidence is substantive. Here an "important study" could be a large, well designed study in case of previous inconsistent or lower quality studies, or could be several new studies of differing size and quality [25] .

Funding could influence or delay the conduct of updates and should therefore be considered already before starting a review update and running the search. If there are no resources to conduct the review update it is possible that the steps following the search cannot be conducted. This concerns not only financial resources but also to time and personnel. In one LSR the review authors indicated that they excluded grey literature for instance due to resource limitations, which could have hindered identification of important new evidence [13] .

Transparent and traceable reporting of changes to the methodology of LSRs is a challenge, and there is currently no PRISMA reporting guideline for LSRs. According to Cochrane guidance, a LSR requires the explicit reporting of certain factors, such as the screening, whether the review incorporated new evidence, and the methods changes [6] . Cochrane has established a transparent structure for reporting the differences between protocol and review in standard reviews [6] . This standard structure can be used for any review, but it may not be adequate for LSRs. Thus, we feel that there is a need for a similar structure specific for LSRs.

In the context of a living PICO and methodology, we found it highly relevant to report in a transparent way the differences between the protocol and first version of the review, as well as between the review updates. When looking at how changes from other LSRs were reported, one LSR included a section at the end of the review on "changes between protocol and review" [9] . Others indicated mainly the changes between review versions or updates, either briefly in the discussion, data supplements [9, 12, 14] , or through update alerts published between review versions [10, 11] . One review included a section "Updates from version 1" before the introduction of the review [13] . For review one [7] , we decided to include an overview table of changes titled 'Summary of PICO development from protocol stage to current review version', which can be incorporated by other LSR authors. This table summarises the main PICO elements (e.g. participants, interventions, comparators, outcomes, study design and methodological changes) of the protocol and review version changes. For the latter, it is also important to report changes of the review´s results (e.g. the number of included studies, the certainty of the evidence) and the authors conclusion (see Table 2 ). We found it important to emphasise that the choice of study design eligibility was not a selective post-hoc approach. Ideally, the methodology for each update should be adapted and (re)finalised at the beginning of each version. Studies that are excluded based on different criteria for the updated PICO could be listed chronologically in a 'supplemental evidence set' or incorporated in a modified PRISMA flowchart, and thereby increase transparency of the screening and study selection process [28] .

Based on our experiences in the planning and conducting of LSRs in a pandemic environment and challenges identified from approaches used in other LSRs, we can conclude that a LSR is a highly suitable review type for the pandemic context, even in the face of new methodological and clinical challenges. Our experience also demonstrated that updating the methods of a LSR, or the LSR itself, is dependent not only on the evolving disease or the emerging evidence, but also on the individual PICO and the capacity as well as resources of the review team. For a living PICO, we described the importance of transparently reporting the differences between the protocol and the review, as well as between each review update.

These lessons learned could be valuable for future pandemic preparedness. An implication for further research and discussion is when to 'retire' and discontinue the updating of a review.

Herewith, I confirm that the following contributions were accurate and agreed by all authors:

This work was funded by the CEOsys project (Grant Nr 01KX2021), a scheme issued by the Network of University Medicine (Nationales Forschungsnetzwerk der Universitätsmedizin (NUM)) by the Federal Ministry of Education and Research of Germany (Bundesministerium für Bildung und Forschung (BMBF)).

There are no declarations of interest.

Six authors, including Claire Iannizzi; Elena Dorando; Jacob Burns; Stephanie Weibel; Nicole Skoetz and Vanessa Piechotta, are funded by the CEOsys project (Grant Nr 01KX2021), a scheme issued by the Network of University Medicine (Nationales Forschungsnetzwerk der Universitätsmedizin (NUM)) by the Federal Ministry of Education and Research of Germany (Bundesministerium für Bildung und Forschung (BMBF)), but this is not leading to a conflict of interest for this concept paper.

Living systematic review: 1. Introduction-the why, what, when, and how

Living systematic reviews: 2. Combining human and machine effort

Living Systematic Review N. Living systematic reviews: 3. Statistical methods for updating meta-analyses

Living Systematic Review N. Living systematic reviews: 4. Living guideline recommendations

Parenteral anticoagulation in ambulatory patients with cancer. The Cochrane database of systematic reviews

Guidance for the production and publication of Cochrane living systematic reviews: Cochrane Reviews in living mode

Convalescent plasma or hyperimmune immunoglobulin for people with COVID-19: a living systematic review. Cochrane Database of Systematic Reviews

International travel-related control measures to contain the COVID-19 pandemic: a rapid review

Interventions for treatment of COVID-19: Second edition of a living systematic review with meta-analyses and trial sequential analyses (The LIVING Project)

Ventilation Techniques and Risk for Transmission of Coronavirus Disease, Including COVID-19

Hydroxychloroquine or Chloroquine for Treatment or Prophylaxis of COVID-19

Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal

The impact of the COVID-19 pandemic on self-harm and suicidal behaviour: update of living systematic review

Clinical manifestations, risk factors, and maternal and perinatal outcomes of coronavirus disease 2019 in pregnancy: living systematic review and meta-analysis

New evidence pyramid

Cochrane Handbook for Systematic Reviews of Interventions version 6

Travel-related control measures to contain the COVID-19 pandemic: a rapid review

Use of Convalescent Plasma Therapy in Two COVID-19 Patients with Acute Respiratory Distress Syndrome in Korea

Treatment With Convalescent Plasma for Critically Ill Patients With Severe Acute Respiratory Syndrome Coronavirus 2 Infection

Core Outcome Measures in Effectiveness Trials

Evaluation of the comprehensiveness, accuracy and currency of the Cochrane COVID-19 Study Register for supporting rapid evidence synthesis production

Guide to key COVID-19 evidence sources

Resources relating to COVID-19

When and how to update systematic reviews: consensus and checklist

Issues Emergency Use Authorization for Convalescent Plasma as Potential Promising COVID-19 Treatment, Another Achievement in Administration's Fight Against Pandemic

Evidence to recommendations: COVID-19 mitigation in the aviation sector. World Health Organization

Tailored PRISMA 2020 flow diagrams for living systematic reviews: a methodological survey and a proposal

The experience report was partly developed from the framework of the CEOsys project. We would like to thank those involved at CEOsys for their support (covid-evidenz.de/).The research was part of a project supported by the Federal Ministry of Education and Research (NaFoUniMedCovid19, funding number: 01KX2021; part of the project "CEOsys").The contents of this document reflect only the authors' views and the German Ministry is not responsible for any use that may be made of the information it contains.We thank the entire author team of the review 'Convalescent plasma or hyperimmune We thank Toby Lasserson for his contribution to the input of this guidance paper and experience report.We thank Maria-Inti Metzendorf, information specialist, for her contribution on the search strategy section.

 Figure 1 . LSR decision flowchart for updating and publishing the review, adjusted for the context of rapidly emerging diseases (Amendments: components in blue are the added steps to the original Cochrane flowchart in grey and purple [1] )