key: cord-0066282-g1s2dqct authors: Dodd, Susanna; Fish, Rebecca; Gorst, Sarah; Hall, Deborah; Jacobsen, Pamela; Kirkham, Jamie; Main, Barry; Matvienko-Sikar, Karen; Saldanha, Ian J.; Trépel, Dominic; Williamson, Paula R. title: Representation of published core outcome sets for research in regulatory guidance: protocol date: 2021-06-09 journal: HRB Open Res DOI: 10.12688/hrbopenres.13139.2 sha: 18b91cee5ad6296a657f55d7bf4f4d5baa2f6a07 doc_id: 66282 cord_uid: g1s2dqct Background: The COMET Initiative promotes the development and use of ‘core outcome sets’ (COS), agreed standardised sets of outcomes that should be measured and reported in all studies in a particular clinical condition. COS are determined by consensus amongst key stakeholders, including health professionals, policymakers and patients, ensuring that the priorities and expertise of these representatives inform the choice of the most important outcomes to measure for a given condition. There is increased recognition of the need to integrate COS across the healthcare system and with existing regulatory apparatus, to ensure that outcomes being recorded are those of key relevance to important stakeholders. The aim of this study is to assess the degree of concordance between outcomes recommended in COS for research and in guidance provided by two key regulators: US Food and Drug Administration (FDA) and the European Medicines Agency (EMA). Methods: COS for research published during 2015-2019 with patient involvement and covering drug or device interventions will be compared against relevant regulatory guidelines, matched by condition. Guidance documents matching in scope (relating to intervention and population) to a COS for research will be scrutinised to identify all suggested outcomes for comparison against the core outcomes in the corresponding COS. Discussion: This study will identify variation between outcomes suggested in FDA and EMA regulatory guidance relative to outcomes included in published COS for research, thus demonstrating the degree of representation of COS in regulatory guidance and vice versa. We will share the study findings (in particular, highlighting any lack of concordance between COS and regulatory guidance overall or for particular disease areas) and will invite feedback from FDA and EMA; we will seek to highlight where findings support the recommendations towards using well-developed COS or will make recommendations to COS developers on outcomes of importance to these key regulators. involvement and covering drug or device interventions will be compared against relevant regulatory guidelines, matched by condition. Guidance documents matching in scope (relating to intervention and population) to a COS for research will be scrutinised to identify all suggested outcomes for comparison against the core outcomes in the corresponding COS. Discussion: This study will identify variation between outcomes suggested in FDA and EMA regulatory guidance relative to outcomes included in published COS for research, thus demonstrating the degree of representation of COS in regulatory guidance and vice versa. We will share the study findings (in particular, highlighting any lack of concordance between COS and regulatory guidance overall or for particular disease areas) and will invite feedback from FDA and EMA; we will seek to highlight where findings support the recommendations towards using well-developed COS or will make recommendations to COS developers on outcomes of importance to these key regulators. Measuring patient health outcomes helps to inform healthcare decisions that are made by patients, healthcare professionals and funders. The Core Outcome Measures in Effectiveness Trials (COMET) Initiative 1,2 brings together people and groups interested in the development and application of agreed standardised sets of outcomes, known as ''core outcome sets'' (COS). One of the successes of COMET has been the development of a publicly available searchable database of completed and ongoing COS development projects 3 . COS may be developed for research or clinical practice, and are determined by consensus amongst health professionals, researchers, policymakers and patients or their representatives, thus ensuring the priorities and expertise of these key stakeholders determine the most important outcomes to measure for a given condition. COS are increasingly being recommended for use by trial funders and healthcare organisations 4 . The Core Outcome Set-STAndards for Development (COS-STAD) 5 minimum standards was published in 2017, providing benchmarks against which to assess the quality of COS. COS-STAD covers 11 key features of COS development relating to three aspects of the COS development process: scope (health condition, population and intervention covered by the COS), stakeholder involvement (including patients, healthcare professionals and researchers) and consensus process (relating to the initial outcomes lists, scoring and consensus decisions, and unambiguous wording of outcomes). Healthcare regulators play an important role in quality improvement, and frameworks adopted by certain organisations rely on evidence on outcomes to inform decision-making 6-8 . Specifically, as an example, to support improvement in healthcare services in the UK, bodies such as the Healthcare Quality Improvement Partnership (HQIP) or UK National Institute for Health and Care Excellence (NICE), are recognising the relevance of considering COS for consistent measurements to inform their guidance 9, 10 . In 2018, NICE guidance on methods to determine relevant guideline outcomes was updated to indicate that COS should be used, if suitable based on quality and validity 9 . The HQIP tool describing key features of national clinical audits and registries states that the rationale for quality improvement objectives should take into account relevant evidence from the COMET database 10 . Similarly, the Swedish Agency for Health Technology Assessment and Assessment of Social Services (SBU) 11 endorse the use of well-developed COS to inform choice of outcomes in trials and systematic reviews. A number of research funding agencies (particularly those commissioning the use of pragmatic randomised control trial to inform policy and regulation) are increasingly recommending that applicants should consider using a COS if one exists 4 . For example, the international SPIRIT reporting guidelines endorse consulting the COMET database to identify relevant COS 12,13 , and in the UK, as an example, National Institute for Health Research Health Technology Assessment (NIHR HTA) programme refers applicants to the COMET database, suggesting that they include established core outcomes "unless there is good reason to do otherwise" 14 . The authors believe that uniformity in recommendations from NIHR and other public funders regarding use of COS would promote greater consistency in outcome collection globally. This benefit would have additional impact if the consistency in such recommendations extended to those from commercial sponsors. The NIHR is a unique health funding agency as Technology Assessment Review teams are funded to provide NICE with independent research to inform their guidance committees 15 . Whilst the relationship between funding research and regulating health varies internationally, regulators such as the US Food and Drug Administration (FDA) and European Medicines Agency (EMA) are influential in commissioning of research to help inform their decisions and it is important to assess the degree of concordance between patient outcomes suggested in FDA and EMA guidance and core outcomes included within COS for research, matched by condition. The US FDA publishes official Guidance Documents and other regulatory guidance 16 , covering topics such as biologics, drugs, medical devices and food, as well as general guidance on study design and outcomes, such as their guidance on the conduct of randomised trials during the COVID-19 pandemic 17 or on Patient-Reported Outcome Measures. 18 . These guidance documents describe the FDA's current opinion on regulatory issues but are not legally binding (unlike FDA regulations, which are federal laws 19 . Similarly, the EMA publishes scientific guidelines prepared in consultation with regulatory authorities in the European Union Member States to inform marketing authorisation applications for human medicines 20 , with full justification required for any deviations from these guidelines. This study will compare the outcomes suggested in these guidance documents against core outcomes included within COS, matched by condition, in order to progress this field by furthering our understanding of the similarities and the differences between COS and guidelines 21 . Changes were made to the text in response to the reviewer comments regarding the following points: clarification that we will endeavour to share the study results with the regulatory authorities, regardless of the findings; addition of an example of COS endorsement external to the UK; explanation regarding the selection of COS published between 2015-2019; additions to methods section regarding extraction of information on whether COS developers consulted FDA/EMA guidelines during the COS development process and whether participants from low/middle-income countries were involved in COS development; addition to analysis section that we will explore the impact of COS characteristics (for example, the number of core outcomes or the involvement of LMIC participants in COS development) on their concordance with corresponding EMA/FDA guidelines; and clarification that will endeavour to discuss, and ultimately produce guidance about, how researchers should determine key outcomes in the case of lack of concordance between COS and regulatory guidelines. The COMET database contains 108 COS for research published between 2015 and 2019 which involved patients in the consensus process. Selection of only those COS published in the last five years which involved patients will increase the number of COS-STAD (Core Outcome Set-STAndards for Development 5 ) standards met. (Note that this study includes COS identified as part of the annual COMET systematic reviews up to and including the systematic review conducted in 2020, which only included studies published up to the end of 2019.) The scope of the COS meeting these criteria will be assessed to ensure that they cover drugs or devices, and if not, they will be excluded from the cohort. For each COS for research, we will search the FDA 16 and EMA 20 websites to identify guidance covering the relevant disease/condition, using the key clinical terms as search terms. If necessary, we will refine these searches using the Google site-specific search facility e.g. searching for "diabetes site: fda.gov" or "diabetes site: ema.europa.eu" when searching for guidelines relating to diabetes on the FDA or EMA websites respectively. We will engage with COS developers if clinical input is required for guidance on appropriate search terms (e.g. to determine synonymous clinical terms to those used to describe the disease/condition under investigation in the COS) or if there is any ambiguity regarding whether identified guidance documents match the scope of the COS (in terms of the disease/condition or interventions). We will initially identify regulatory guidance/COS pairs where the scope is an exact match but will also consider situations where one of the pair may be more general than the other, based on an assessment of the descriptions of the population (i.e. clinical condition/disease) and intervention in the COS publications and regulatory guideline documents, using a previously-developed framework 22 (see Figure 1 ). Pairs which focused on different interventions or different populations were not considered to be a match (i.e. only matches corresponding to types a-c. e-g, i-k in Figure 1 were eligible for inclusion). Each reviewer independently applied this matching algorithm to each pair of FDA/EMA guideline and corresponding potentially relevant COS. Discrepancies were resolved through discussion. • COS eligibility: COS for research (including those intended for both research and practice) were included if published between 2015 and 2019, involved patients in the consensus process and related to drug or device interventions. • Regulatory guidance eligibility: EMA/FDA guidance were considered eligible for comparison against relevant COS for research if their scope (in terms of clinical condition/disease and intervention) matched at least generally with that of the corresponding COS for research (i.e. matches of type a-c. e-g, i-k in Figure 1 ). Data on the year of publication, disease name, specific condition and outcomes included in the eligible published COS for research will be exported from the COMET database and COS database into an Excel spreadsheet. We will record whether the COS developers consulted FDA and/or EMA guidelines as part of the COS development process, as detailed in the COS publication. We will also record from the COMET database whether participants from low/middle-income countries (LMIC) were involved in COS development. Once FDA and EMA guidance documents are identified which match in scope to a COS for research, the guidance documents will be scrutinised in order to identify all suggested outcomes (i.e. those outcomes which the guidelines state should/could/may/might be considered) relating to the specific COS population and intervention. Any additional caveats included in the guidelines about each of the recommended outcomes (e.g. relating to the age category or severity) will be recorded. If specific measurement tools (e.g. quality of life questionnaires) are recommended, the reviewer will search for and extract the individual items within these measurement tools, in order to assess whether these individual items correspond to any outcomes recommended by the corresponding COS/guidelines. Verbatim guidance document text regarding the suggested outcome measures will be recorded in tabular form for each COS. The matching between the scope of the COS and regulatory guidance will be classified as exact or general (e.g. COS is narrower/ broader) in relation to both the population with the condition and the interventions (as per Figure 1) , with input from clinical members of the research team and/or the COS developers, if necessary. Data extraction will be carried out by all researchers for the initial three COS/guidance pairs to ensure consistency of approach; subsequent data extraction will be carried out independently by two researchers. Disagreements will be resolved by discussion with SD/PW if necessary. Mapping between core outcomes and outcomes suggested in guidelines will be checked by the lead author (SD). The mapping of the verbatim extracted text from the EMA/FDA guidance to each of the core outcomes will be coded as specific (i.e. direct correspondence between the wording of the core outcome references in the guidance compared to the wording in COS) or general (i.e. only general alignment between the wording of text in the guidance relative to the wording in the COS), and this mapping will be summarised using a table as demonstrated for the type 2 diabetes SCORE-IT COS 23 in Appendix 1. Again, we will contact COS developers if clinical input is required regarding general or specific correspondence between core outcomes and those suggested in the guidance. We will use a tick to demonstrate specific correspondence between the wording of the core outcome references in the guidelines compared to the wording in COS, whereas a tick in brackets will be used to indicate general alignment (with further detail provided in a footnote) between the wording of suggested outcomes in the guidelines relative to the wording in the COS. For each COS, we will record the number (and percentage) of COS outcomes which were covered in the guidance (separately for FDA and EMA) in general or specific terms (separately) and either general/specific terms. The distribution of these percentages will be summarised across guidance documents as a whole, split by FDA and EMA, using descriptive statistics and graphical presentation, overall and split by disease category. We will also present results according to the breakdown of matching between scope of intervention and population between the COS and guidelines, as shown in the matrix in Appendix 2. Note that only results for highlighted cells a-c, e-g, i-k will be presented (i.e. those corresponding to at least a general match in both intervention and population between the COS and guidelines). In addition, by way of symmetry we will present the results above which instead compare how the core outcomes relate to those suggested in EMA and FDA guidelines, in order to identify the agreement of COS with outcomes suggested in corresponding guidelines; i.e. we will present two additional sets of tables/results, the first with outcomes suggested in the EMA guidelines, and the second with outcomes suggested in the FDA guidelines, as the index list of outcomes. We will explore the impact of COS characteristics (for example, the number of core outcomes or the involvement of LMIC participants in COS development) on their concordance with corresponding EMA/FDA guidelines. The findings of this study will be disseminated through publication in an open access peer-reviewed journal and presentation at both national and international conferences. Contact will be made with FDA and EMA colleagues and feedback on our findings requested. This study will identify any misalignment between outcomes suggested by EMA and FDA regulatory guidance relative to those included in published COS for research, thus demonstrating the degree of representation of core outcomes, which have been agreed by consensus by key stakeholders, within regulatory guidance, and vice versa. A lack of concordance between COS and regulatory guidance may highlight the opportunity for such guidelines to be better informed by COS and vice versa, and we will use the evidence obtained from this study to engage the relevant regulatory bodies in discussions accordingly. We will endeavour to discuss, and ultimately produce guidance about, how researchers should determine key outcomes in the case of lack of concordance between COS and regulatory guidelines. Details of working group ICMJE guidance will be followed with regards to publication policy. Guidance SCORE-IT core outcome not explicitly mentioned but covered by the following general terms EMA FDA Death from a diabetes related cause such as heart disease (✓) a a Cardiovascular disease/safety profile Open Peer Review or "globally". Regarding methods, it is stated that the studies should have been published between 2015-2019. Why is 2020 not included (as we are in 2021 now). Please explain why this is: because of COVID-19, nothing was published or any other reason. We currently think there is a lack of discussion on the potential limitations of the study. For example, limitations regarding relevance and context. It would be interesting to see whether a COS -set produced in a high-income country compared to a low-income country differ, and also, if one or the other has influenced EMA or FDA more. We understand that there could be difficulties in drawing conclusions out of this type of information, however it is important to discuss the potential importance of this implication which in the long run may mean that the production of COS will have impact on research outcomes of thus more importance to high-income countries. (Patients or other stakeholders may look upon the prioritization of outcomes differently depending on contextual factors). There is also a lack of discussion on the quality of the included COS. In the section on methods, we think the authors should add a paragraph on how they will assess quality of the included COS. This should be discussed regarding both the selection of participants, how the actual prioritization process is done as well as numbers of the outcomes included in the core outcome set. (Core outcome sets that include large number of outcomes will likely have more of a concordance with regulatory guidance than those with fewer outcomes. It is our opinion that a core set by definition should include less than 10 outcomes, but when it doesn't this needs to be addressed. It would be interesting if the authors note whether the included published core outcome sets had searched for FDA or EMA documentation regarding important outcomes before they started their study. Finally, we assume the authors will discuss how researchers should relate to core outcome sets that do not include all the requested outcomes requested by regulators. That will be an interesting discussion to read. Looking forward to see this study published! 7. Is the rationale for, and objectives of, the study clearly described? Yes Are the datasets clearly presented in a useable and accessible format? Yes Competing Interests: No competing interests were disclosed. We confirm that we have read this submission and believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however we have significant reservations, as outlined above. Author Response 01 Jun 2021 Susanna Dodd, University of Liverpool (a member of Liverpool Health Partners), Liverpool, UK Thank you for your comments. Our responses are given below: In the last sentence of the Discussion part in the abstract, the authors state that they will seek feedback from FDA and EMA "if there is a lack of concordance between COS and regulatory guidance overall or for particular disease areas." We think feedback from the regulatory authorities would be interesting regardless of what the results are. ○ Thank you for highlighting this point. We agree with your comment and will endeavour to share the results of this study to the regulatory authorities, regardless of the findings. As the COMET database in our understanding includes studies from different countries, we suggest that an example from another country than the UK as well, is included in the introduction to give a broader perspective. As it reads now, there seems to be only the UK or "globally". We currently think there is a lack of discussion on the potential limitations of the study. For example, limitations regarding relevance and context. It would be interesting to see whether a COS -set produced in a high-income country compared to a low-income country differ, and also, if one or the other has influenced EMA or FDA more. We understand that there could be difficulties in drawing conclusions out of this type of information, however it is important to discuss the potential importance of this implication which in the long run may mean that the production of COS will have impact on research outcomes of thus more importance to high-income countries. (Patients or other stakeholders may look upon the prioritization of outcomes differently depending on contextual factors). ○ Thank you for this suggestion. The vast majority of COS for research are led from a high-income country (HIC); a recent systematic review found only four COS for research that were led from low/middle-income countries (LMIC). As such it is unlikely that it will be possible to make meaningful comparisons between COS from a HIC/LMIC to determine their respective impact on regulatory guidelines. However, the COMET database contains information on whether LMIC participants were involved in COS development, and therefore we could assess whether the involvement of LMIC participants in COS development impacts on the concordance between COS and regulatory guidelines. This has been added to the data extraction and analysis sections of the manuscript. There is also a lack of discussion on the quality of the included COS. In the section on methods, we think the authors should add a paragraph on how they will assess quality of the included COS. This should be discussed regarding both the selection of participants, how the actual prioritization process is done as well as numbers of the outcomes included in the core outcome set. (Core outcome sets that include large number of outcomes will likely have more of a concordance with regulatory guidance than those with fewer outcomes. It is our opinion that a core set by definition should include less than 10 outcomes, but when it doesn't this needs to be addressed. ○ It is relatively difficult to define COS 'quality'; the COS-STAD standards were introduced in 2017, but we do not think it would be fair or helpful to apply COS-STAD retrospectively to studies that were published before 2017. We decided to include only those COS for research which were published in recent years (between 2015 and 2019) and which involved patients in the consensus process, in order to increase the likelihood that high standard COS are included. We have added to the analysis section that we will explore the impact of the number of core outcomes on the level of agreement between COS and corresponding EMA/FDA guidelines. It would be interesting if the authors note whether the included published core outcome sets had searched for FDA or EMA documentation regarding important outcomes before they started their study. ○ Thank you for this comment. This information is currently being extracted from each COS publication, so we have added this explicitly to the methods section. Finally, we assume the authors will discuss how researchers should relate to core outcome sets that do not include all the requested outcomes requested by regulators. That will be an interesting discussion to read. Looking forward to see this study published! ○ Thank you for this suggestion. This is an interesting point which we had not mentioned in the protocol, so we have now added it to the discussion. Competing Interests: I have no competing interests. 279. PubMed Abstract | Publisher Full Text | Free Full Text 3 Core Outcome Set-STAndards for Development: The COS-STAD recommendations healthtechnology-assessment-good-practices-recommendation-appendix. pdf?sfvrsn=350e6d96_4 Accessed Healthcare Quality Improvement Partnership UPCARE tool Swedish Agency for Health Technology Assessment and Assessment of Social Services SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials Conduct of Clinical Trials of Medical Products during COVID-19 Public Health Emergency Guidance for Industry: Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims How the US Food and Drug Administration evaluates the scientific evidence for health claims More than half of systematic reviews have relevant core outcome sets Selecting Core Outcomes for Randomised Effectiveness trials In Type 2 diabetes (SCORE-IT): a patient and healthcare professional consensus on a core outcome set for type 2 diabetes Core outcome sets through the healthcare ecosystem: the case of type 2 diabetes mellitus The authors have identified a question that is relevant for researchers, COS developers, and the broader health care community: to what extent do FDA and EMA guidance recommendations align with the results of published core outcome sets? The authors set out a well-conceived protocol for exploring this question in a way that makes relevant comparisons between COS and guidance and reports it coherently. I think this is a project that absolutely needs to be done.In the third paragraph the authors note that uniformity in recommendations from public funders like NIHR would promote greater consistency in outcomes, and "this benefit would have additional impact if the consistency in such recommendations extended to those from commercial sponsors." Do you mean "for" commercial sponsors? The statement doesn't make sense to me otherwise.The third paragraph notes that FDA and EMA are influential in commissioning research to help inform their decisions. Yes, but in this context it seems much less important to me than their powerful role as arbiters of evidence for commercialization of new products. This is a primary function of their guidance.Another thought on impact: FDA and EMA guidance are often not well aligned (to the great frustration of drug developers). So another possible impact of this work might be to facilitate alignment of international regulatory guidance (at least for outcomes) with COS as a common touchstone.Also, one technical mistake in the introduction. The fourth paragraph of the intro states that FDA guidance is not legally binding, which is true, but then adds "unlike FDA regulations, which are federal laws." Regulations are not laws. Laws can only be made by the U.S. Congress. Congress authorizes FDA to write regulations to work out the details of how the laws should be implemented. So regulations are legally binding, but they are not laws. (And there is a missing closed parenthesis at the end of the statement about federal laws.)Thank you, we have now checked the document and have really only one remaining comment for potential improvement.We still lack a discussion on the quality of the included COS. We understand if this may be out of scope for this study, but we still believe that it would be interesting to hold such a discussion as we already today are faced with only a little guidance on how to deal with obvious low quality COS. It would be a pity when conducting this otherwise great study, to not include these aspects, because they are already part of the current problems in addressing COS today. Please take this as advice only. We are happy to approve of the current version. Are sufficient details of the methods provided to allow replication by others? Not applicable Competing Interests: No competing interests were disclosed.We confirm that we have read this submission and believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. Reviewer Thank you for letting us review this interesting study. We think the research question is well chosen and will give interesting insights and results of the use and applicability of COS. We only have some minor comments which we hope will be helpful in improving the study further.In the last sentence of the Discussion part in the abstract, the authors state that they will seek feedback from FDA and EMA "if there is a lack of concordance between COS and regulatory guidance overall or for particular disease areas." We think feedback from the regulatory authorities would be interesting regardless of what the results are.1.As the COMET database in our understanding includes studies from different countries, we suggest that an example from another country than the UK as well, is included in the introduction to give a broader perspective. As it reads now, there seems to be only the UK 2.