key: cord-0902056-5a3ewky1 authors: Beecher, Claire; Toomey, Elaine; Maeso, Beccy; Whiting, Caroline; Stewart, Derek C.; Worrall, Andrew; Elliott, Jim; Smith, Maureen; Tierney, Theresa; Blackwood, Bronagh; Maguire, Teresa; Kampman, Melissa; Ling, Benny; Gravel, Christopher; Gill, Catherine; Healy, Patricia; Houghton, Catherine; Booth, Andrew; Garritty, Chantelle; Thomas, James; Tricco, Andrea C.; Burke, Nikita N.; Keenan, Ciara; Westmore, Matthew; Devane, Declan title: What are the most important unanswered research questions on rapid review methodology? A James Lind Alliance research methodology Priority Setting Partnership: the Priority III study protocol date: 2021-07-23 journal: HRB Open Res DOI: 10.12688/hrbopenres.13321.1 sha: ef734910f281fcc9e0d79489d88177129ac93dd7 doc_id: 902056 cord_uid: 5a3ewky1 Background: The value of rapid reviews in informing health care decisions is more evident since the onset of the coronavirus disease 2019 (COVID-19) pandemic. While systematic reviews can be completed rapidly, rapid reviews are usually a type of evidence synthesis in which components of the systematic review process may be simplified or omitted to produce information more efficiently within constraints of time, expertise, funding or any combination thereof. There is an absence of high-quality evidence underpinning some decisions about how we plan, do and share rapid reviews. We will conduct a modified James Lind Alliance Priority Setting Partnership to determine the top 10 unanswered research questions about how we plan, do and share rapid reviews in collaboration with patients, public, reviewers, researchers, clinicians, policymakers and funders. Methods: An international steering group consisting of key stakeholder perspectives (patients, the public, reviewers, researchers, clinicians, policymakers and funders) will facilitate broad reach, recruitment and participation across stakeholder groups. An initial online survey will identify stakeholders’ perceptions of research uncertainties about how we plan, do and share rapid reviews. Responses will be categorised to generate a long list of questions. The list will be checked against systematic reviews published within the past three years to identify if the question is unanswered. A second online stakeholder survey will rank the long list in order of priority. Finally, a virtual consensus workshop of key stakeholders will agree on the top 10 unanswered questions. Discussion: Research prioritisation is an important means for minimising research waste and ensuring that research resources are targeted towards answering the most important questions. Identifying the top 10 rapid review methodology research priorities will help target research to improve how we plan, do and share rapid reviews and ultimately enhance the use of high-quality synthesised evidence to inform health care policy and practice. Systematic reviews summarise existing research and use structured and 'explicit methods to identify, select and critically appraise relevant studies, and collect and analyse data from these studies' 1 . Systematic reviews, therefore, provide a robust synthesis of existing research for a given topic and are typically seen as the cornerstone of research evidence for informing health policy and practice decision-making. However, systematic reviews require significant resources (time, expertise, funding) to conduct and can take up to two years to complete 2 . While full systematic reviews can, and have been, completed rapidly, rapid reviews have emerged more commonly as a form of evidence synthesis in which certain steps of the systematic review process are omitted or simplified to produce information more efficiently within limited resources to inform healthcare decisions 2,3 . Evidence suggests that rapid reviews are increasingly being commissioned, and the results used, to inform decision making by policymakers and funders as a more resource efficient alternative to conventional systematic reviews 4, 5 . The coronavirus disease 2019 (COVID-19) pandemic has further increased the demand for rapid reviews due to the need to provide time-sensitive information to decision-makers 6 . Although growing demand has led to a steady rise in rapid review publications generally since 2010 3, 7 , albeit often poorly reported 8 , there are challenges in identifying rapid reviews commissioned specifically by organisations as they are often not published in peer review journals 3 . This may, in part, be due to the suggestion that commissioned rapid reviews are considered more relevant in the context of narrow questions rather than broader questions of wider relevance that may require the focus of a conventional systematic review 5, 9 . Despite an evident rise in the commissioning and conduct of rapid reviews, there is an absence of high-quality evidence underpinning some decisions about how they are planned, done and the findings shared. There is also limited evidence on the relative impact of different simplifications or omissions, for example, restricting searches to published material or by date/language or conducting single reviewer screening or data extraction 10,11 , on the validity and application of findings 3 commonly used in rapid reviews. Furthermore, there is no universally accepted definition of a rapid review 3,6,12,13 and debate exists regarding the use of the word 'rapid', with several different synonyms previously identified within the literature 7,10 . These debates have been further complicated in the era of the COVID-19 pandemic where it has been highlighted that rapid reviews cannot be categorised simply based on the time taken to complete a review given that well-resourced conventional systematic reviews have been conducted to a high standard, rapidly. Overall, rapid reviews are 'poorly understood, ill-defined diverse methodologies supported by limited published evidence' 3 , with a need for further research to establish a robust methodological evidence base 14 . Without such an evidence base, the validity, appropriateness and usefulness of rapid reviews are undermined 2 . Research prioritisation plays a key role in minimising research waste by ensuring that research resources are targeted towards questions of the most potential benefit 15 . Research prioritisation involves identifying, prioritising and obtaining consensus on research needs and questions that are relevant and important to all relevant stakeholders for that topic 16 . The James Lind Alliance (JLA) is a non-profit organisation that brings multiple stakeholders, including patients, carers and clinicians, together in an equal, transparent and evidence-based Priority Setting Partnership (PSP) to determine the most important evidence uncertainties or unanswered research questions. Although commonly focused on knowledge gaps surrounding the effects of treatments, the approach has been broadly applied to other areas such as diagnosis, prevention, and more recently, to identify methodology uncertainties in recruitment and retention within clinical trials 17,18 . This Priority III PSP aims to identify the top 10 unanswered research questions about how we plan, do and share rapid reviews, as identified and prioritised by contributors drawn from key stakeholder groups, including patients, the public, reviewers, researchers, clinicians, policymakers and funders. Evidence Synthesis Ireland is conducting the Priority III PSP in collaboration with the JLA. Evidence Synthesis Ireland is an all-Ireland initiative funded by the Health Research Board and the Health and Social Care, Research and Development Division of the Public Health Agency in Northern Ireland. It aims to make evidence syntheses better designed, conducted, reported, and more usable within health care policy and clinical practice decision-making by patients, the public, health care institutions and policymakers, clinicians and researchers. Ethical approval Ethical approval was granted by the National University of Ireland Galway Research Ethics Committee (reference: 20-Apr-02). The study employs a PSP based on the methods of the JLA and will be reported following the REporting guideline for PRIority SEtting of health research (REPRISE) criteria 16 . Following the JLA Guidebook for PSPs 19 , this PSP will take place in seven stages and build on modified JLA guidance 17,18 and previous PSPs in recruitment and retention within clinical trials 17,18 . The seven stages of the project are: establishing the steering group, identifying and inviting potential partners, gathering uncertainties, data processing and verifying uncertainties, interim priority setting; final prioritisation workshop; and disseminating findings. Establishing the Steering Group (6 months). The Priority III PSP will be led and managed by an international Steering Group who will coordinate, oversee and guide the PSP activities. The primary roles of the Steering Group will be to discuss and agree on the scope and remit of the project, enable access and reach to key stakeholder groups, contribute intellectually towards the study methods and interpretation, and ensure that all perspectives are captured and meaningfully included throughout. The Steering Group will also determine decisions about the processes used throughout as the project progresses. Membership of the Steering Group will include individuals and representatives from organisations, including patients and the public, reviewers, researchers, clinicians, policymakers and funders. The Steering Group will comprise of up 25 members across stakeholder groups. Potential members will be approached to participate via email. Patient and public participants will be paid for their time spent on Steering Group activities. Other participants (researchers/ reviewers, clinicians, policy makers, funders) will not be paid. Steering Group members will identify and engage additional appropriate partners through a process of peer knowledge and consultation, using the Steering Group members' respective networks. Organisations and individuals who can reach and advocate for key stakeholder groups will be invited to participate with the PSP as partners. We have not set a minimum or maximum number of partners we would like to reruit. Partners will be organisations or groups representative of diverse stakeholder perspectives (patients and the public, reviewers, researchers, clinicians, policymakers and funders) who will commit to supporting, participating and promoting the PSP among their stakeholder groups. As far as possible, the partners involved will seek to represent the interests of all stakeholders. All partners will be asked to confirm that they agree to support the PSP. Review Methodology PSP will gather uncertainties from patients and the public, reviewers, researchers, clinicians, policymakers and funders. This will be undertaken by conducting an initial survey with all relevant stakeholders. The survey will identify unanswered questions or uncertainties about how we plan, do and share rapid reviews. The survey will be designed and piloted by the Steering Group members. The survey will be created using QuestionPro 20 software and hosted on the Evidence Synthesis Ireland website. Participants will be asked to give their explicit content to take part in the survey using yes/no questions. The survey will contain four open-ended questions. Three of the questions will focus on different stages of the rapid review process, and participants will be asked to answer as few or as many as they wish. The fourth question asks for any additional questions or comments on the rapid review process. The four questions are; 4. Do you have any other questions or comments on how we plan, do and share the results of rapid reviews? Participant demographic data will also be collected to monitor responses of different stakeholder groups and help refine and target the promotion of the survey towards under-represented groups if necessary. In line with previous methodology-focused PSPs 17,18 , the survey will be open for four weeks. The survey will be advertised and distributed via the Steering Group and the PSP Partners. Specifically, members will distribute the survey via their networks, mailing lists, newsletters and social media. We do not have an a priori determined sample size and instead will distribute the survey widely with a view to reaching as wide an audience as possible. The survey and all other study materials can be found as Extended data 21 . The initial survey consultation process is expected to produce substantial 'raw' questions and comments indicating stakeholders' areas of uncertainty. The survey data will be downloaded from QuestionPro 20 . These raw questions will be categorised and refined by the NUI Galway research team (CB, CH, DD) into summary questions that are clear, addressable by research, and understandable to all. Similar or duplicate responses will be combined where appropriate. Questions will be considered out of scope if they do not relate to planning, doing and sharing the results of rapid reviews within the healthcare setting, asking for information or advice, or being too broad, unclear, unrelated or off topic. Questions deemed to be out of scope will be compiled separately and made available for future use upon request. The Steering Group will exercise oversight on data analysis and processing to ensure that the raw data are being interpreted appropriately. The summary questions will be developed in a reflective manner that is understandable to all audiences. The process will be conducted transparently to ensure that the finalised questions can be traced back to the raw response data. Each in-scope question will be checked against existing sources of evidence to determine which questions remain unanswered. A question will be verified as unanswered if a synthesis gap is apparent following a search of all relevant systematic reviews published within the past three years. We judged a review to be systematic when it involved explicit methods to search, select, critically appraise and synthesise individual studies. If a systematic review has been conducted to answer any of the questions completely, the quality of that systematic review will be appraised using the AMSTAR 2 22 tool to help inform decisions about the extent to which the question has been answered. Evidence checking will be completed by one researcher (CB) and verified by one other researcher (DD) Difference of opinion will be resolved by consultation with a third researcher if necessary. Existing sources of evidence will be identified through a search of the PubMed bibliographic database for systematic reviews published from 2018 to the time of searching (2021) using a search strategy developed specifically for use in this project with the support of an experienced information specialist (see Table 1 for search strategy and limits applied). The JLA Question Verification Form, which describes the process used to verify question uncertainty, will be completed. In line with JLA guidance, the Question Verification Form includes details of the types and sources of evidence used to check uncertainty. This form will be published on the JLA website to enable stakeholders to understand how the PSP decided that these questions are unanswered and any limitations of this process. The Steering Group will be asked to review and refine, as appropriate, the final list of summary questions for inclusion in the interim survey. Interim priority setting is where the long list of questions is reduced to a shorter list that can be taken to the final priority setting workshop. This shortlist is aimed at a wide audience and will be administered using an electronic survey (QuestionPro software 20 ) and hosted on the Evidence Synthesis Ireland website. It will be designed and piloted by the Steering Group members. In this survey, stakeholders will be asked to prioritise summary questions developed from the initial survey in order of importance. This interim priority setting process will help reduce the long list to a short, manageable set of approximately 20 indicative questions that are clear, addressable by research and understandable. The survey will be open for four weeks and will be advertised and distributed like the initial survey via the Steering Group and the PSP Partners. Participant demographic data will be collected to monitor response rates from different stakeholder groups to help refine and target the promotion of the survey towards under-represented groups. Questions prioritised in the interim survey will go forward for final prioritisation at a stakeholder workshop(s). Where the interim prioritisation does not produce a clear ranking or cut off point, the Steering Group will decide, and report, which questions are taken forward to the final prioritisation. The final priority setting stage will involve two half-day virtual workshops facilitated by the JLA to ensure transparency, accountability, fairness and appropriate representation. Based on the methods of the JLA, these workshops would ordinarily take place in person, but due to the Priority III study being conducted under restrictions contingent upon the global COVID-19 pandemic, the workshops will be held virtually. With input from the Steering Group, up to 24 contributors drawn from key stakeholder groups (patients, public, reviewers, researchers, clinicians, policymakers and funders) will be recruited to participate in a day of group discussion, plenary sessions and ranking exercises to determine the top 10 questions for research. Four virtual breakout rooms will be used to facilitate smaller group discussions. All participants will declare their interests, enabling a diverse group of stakeholders to exchange knowledge, perspectives, and experiences and inform the decision-making process. A maximum of four Steering Group members will be invited to participate in the workshops, with a steering group member in each breakout room to participate and answer any questions about the Priority III process. Additional Steering Group members who wish to attend will do so in an observer capacity. The outcome of the workshop will be consensus on a top 10 list of research priorities of unanswered questions about rapid review methodology. The results of the study will be disseminated through the JLA and Evidence Synthesis Ireland websites. The Steering Group will identify additional ) OR (("Rapid Reviews" OR "systematic reviews" OR "systematic literature reviews") AND "METHODS"[SH] AND ("literature search*" OR Information Storage and Retrieval)) OR (("Rapid Reviews" OR "systematic reviews" OR "systematic literature reviews") AND "METHODS"[SH] AND ("plain language summar*" OR "lay summar*")) OR (("Rapid Reviews" OR "systematic reviews" OR "systematic literature reviews") AND "METHODS"[SH] AND "strength of evidence")) OR (("Rapid Reviews" OR "systematic reviews" OR "systematic literature reviews") "METHODS"[SH] AND "study selection")) OR (("Rapid Reviews" OR "systematic reviews" OR "systematic literature reviews") AND "METHODS"[SH] AND "data extraction")) OR (("Rapid Reviews" OR "systematic reviews" OR "systematic literature reviews") AND "METHODS"[SH] AND "quality assessment")) OR (("Rapid Reviews" OR "systematic reviews" OR "systematic literature reviews") AND "METHODS"[SH] AND "critical appraisal"))) NOT Protocol appropriate audiences to engage when sharing the results, such as researchers, funders, and patient and clinical communities. The steering group will also identify opportunities to collaborate and contribute evidence to answer the top 10 list of research priorities. The Priority III study will identify a top 10 of unanswered research questions regarding rapid review methodology, guided by an adaptation of the James Lind Alliance PSP process. The findings of the study will contribute to minimising research waste in rapid review methodology, ensuring that research resources will be used to answer questions on this topic that have been prioritised by key stakeholders internationally, including patient and the public, reviewers, researchers, clinicians, policymakers and funders. At the time of submission, all stages up to the dissemination of findings had been completed. There are several factors that contributed to the submission of the protocol at this time: 24 key stakeholder members to participate in discussions (group discussion, plenary sessions, raking exercises, virtual breakout rooms). The outcome of the workshop will be consensus on a top 10 list of research priorities of unanswered questions about rapid review methodology. Disseminating findings -results will be shared through JLA and Evidence Synthesis Ireland websites 7. The paper is written in the future tense, but all steps except for 7 above, has already been completed per the authors. Throughout my reading, I am anticipating that this is a future study that has yet to be conducted. It comes as a surprise to me that at the very end, in the "study status," the authors reveal all the steps except the final one (disseminating findings), have been completed. Minor grammatical error: "Evidence suggests that rapid reviews are increasingly being commissioned, and the results used, to inform decision making by policymakers …." Please remove the comma after "results used." The sentence should read: "Evidence suggests that rapid reviews are increasingly being commissioned, and the results used to inform decision making by policymakers…" ○ Presumably the authors of the paper are on the steering group? This is not clearly stated. ○ NUI Galway needs to be spelt out for readers unfamiliar that it represents National University of Ireland, Galway. The authors do not state how much the partners will be reimbursed. The time frames for the initial steps listed above are inconsistent. Some are written to only include the duration, but others include both duration (e.g., 1 month, 6 months, 2 days) and the month (Oct 2020, April 2021, May 2021). My recommendation would be to stick with one approach and be consistent (e.g., include the time period with actual dates because the duration can be implied). This paper describes a proposed research project that consists of a series of focus groups and stakeholder meetings to define the most critical 10 research questions regarding rapid reviews. It is well-written with minimal grammatical and syntax errors, but I am unclear why it was written in the future tense. The authors describe a proposed study, when in fact, all steps of the project have been conducted except for the last one. When I get to the final section of the paper and realize the proposed steps have been completed, my natural questions include; How did the sessions go? Who was recruited to participate? How many were recruited and what was the breakdown? How much were the participants reimbursed for their time? What did the questions reveal? What are the next steps? But these questions are unanswered by the authors because the paper is written as a proposed project, despite the steps having already been completed. This issue needs to be resolved before I am fully comfortable with accepting in the current state. Thank you for the opportunity to review this paper. My recommendation would be to convert this into a dissemination paper, such as the PRioRiTy (Prioritising Recruitment in Randomised Trials) publications the authors cite (references 17 and 18). If one of the purposes of the project is to conduct research on the top 10 list of research priorities of unanswered questions about rapid review methodologies AND then disseminate those findings, then the authors may want to consider converting their paper into one that serves the latter purpose. Is the rationale for, and objectives of, the study clearly described? Yes Is the study design appropriate for the research question? Yes purpose of peer review was, except perhaps to ensure accuracy of reporting. While I imagine that there are good reasons for the protocol being submitted at this late stage, I think it would be more transparent (and helpful for the reader) that this is made clear from the beginning, along with the reason. Thus, I suggest that the protocol should be written in the past tense, where applicable, and dates added for all stages. The inclusion of the process for checking that questions were indeed still unanswered is a very valuable inclusion in the methods -I look forward to seeing the results of this process. I hope that the authors can include in the publication of the results the questions that are already answered and the sources of the existing evidence. I do have some concerns about the methodology here and suggest that the authors include a justification of their choice of methods or further clarification: The quality of the systematic reviews will be appraised using AMSTAR 2 but there is no description in the protocol as to how the score will be used in deciding if the question has been answered, e.g. does it need to have a minimum quality? ○ Only PubMed will be searched and only reviews published in English will be included. It is not clear how many reviewers will be involved in the quality assessment and why non-English and grey literature is not being included. If the authors were to evaluate their own review process using AMSTAR 2 I expect that it would not score highly. Thus, the authors should provide greater clarity and a justification for their choice of methods for this part of the protocol. In the paragraph headed 'Data processing and verifying uncertainties (5 months)' I noticed that the scope of the priority setting exercise is limited to the healthcare setting. This is a very important limitation but no explanation is given for why. Why not include public health or health system questions, which are equally (if not more) important for rapid reviews? It is not clear from the protocol if there was any attempt to include the views of people from lowto middle-income countries, which may be able to offer a broader perspective of the unanswered questions. Can this be clarified? When defining rapid reviews (in the abstract and introduction) I suggest noting that an important distinction from systematic reviews is that rapid reviews are generally conducted with the needs of decision-makers in mind -see for example: Normally I would expect the material in the current paper to be included in the final publication, rather than in a separate protocol paper that seems to have been written essentially contemporaneously with the final work. I will not second-guess this decision, but will instead comment as if the project were not now largely complete. There are a few places where I believe that the reader's understanding of the project could be improved by including either tables or supplementary material to improve the specificity of the document. The document states that: "Membership of the Steering Group will include individuals and representatives from organisations, including patients and the public, reviewers, researchers, clinicians, policymakers, and funders. The Steering Group will comprise of up 25 members across stakeholder groups." Either the membership of this Steering group should be provided, or the types of groups and organizations that were contacted to participate should be provided. Were these groups restricted to Ireland, or to Europe, or to English speaking countries, or to advanced economies? It seems likely that the ultimate product coming from a Eurocentric steering committee will be different than that which involves individuals from Asia, Africa, and South America. On the other hand, it would be difficult to adequately represent researchers, clinicians, policymakers, and funders across a wide geographic distribution. The areas of expertise associated with steering group members (statistics, survey methodology, etc.) should be described in more detail. Similarly, the nature of the "partners" should also be better defined, as should a bit about the total number of stakeholders expected to participate. The final report should include information on the nature and number of stakeholders that were contacted and who declined to participate. Is the rationale for, and objectives of, the study clearly described? Yes Are the datasets clearly presented in a useable and accessible format? Not applicable Competing Interests: No competing interests were disclosed. Systematic review The authors state that there is no minimum or maximum number of partners. Patient and public partners will be reimbursed (the authors do not state how much). No minimum or maximum number of participants are identified Gathering uncertainties (1 month -Oct 2020) from patients and the public, reviewers, researchers, clinicians, policymakers and funders (by conducting an initial 4 question survey with all relevant stakeholders) Data processing and verifying uncertainties (5 months) -raw questions will be categorized and refined into summary questions (downloaded from QuestionPro and then categorized and refined by a "NUI Galway Interim priority setting ( 1 month -April 2021) -long list of questions is reduced to shorter list. Stakeholders will be asked to prioritize summary questions Final prioritization workshop (2 days -May 2021) -two half day virtual workshops of up to EPC Methods: An Exploration of Methods and Context for the Production of Rapid Reviews What are the best methodologies for rapid reviews of the research evidence for evidence-informed decision making in health policy and practice: a rapid review Are the datasets clearly presented in a useable and accessible format? Not applicable This is a very interesting and valuable project. It has a well-described and well written methodology for priority setting. The results will provide an important contribution to the field. I do have some comments, however, for consideration: I was rather surprised to reach the end of the protocol and find out that the research had already been completed, except for the dissemination of the findings. It made me wonder what then the Competing Interests: I have a professional interest in rapid review methods and have written on the subject in both peer-reviewed journal papers and in a paid capacity for WHO/PAHO.Reviewer Expertise: I have experience in conducting both systematic reviews and rapid reviews. I have also been involved in the development and assessment of methodologies for rapid reviews.I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. The paper provides a well-structured and readable overview of a project intended to develop a better understanding of the strengths and limitations of "rapid reviews," and presumably provide Reviewer Expertise: I currently perform systematic reviews and meta-analysis for covid19-related diagnostics. I have previous experience in randomized clinical trials, genomic epidemiology, and preclinical research.I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.Reviewer Report 23 September 2021 https://doi.org/10.21956/hrbopenres.14503.r30257 © 2021 Staley K. This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. TwoCan Associates, Ross-on-Wye, UK This is a clear summary of the approach taken to identify priorities for research on rapid reviews. It is consistent with the standard JLA approach and is well-described. I do not have any concerns about this. I'm not an expert on rapid reviews, so I can't comment on that. Is the rationale for, and objectives of, the study clearly described? Yes Are the datasets clearly presented in a useable and accessible format? Not applicable Competing Interests: No competing interests were disclosed. I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.