key: cord-0864218-me27fndn authors: Paul, Jenine; Davidson, Randy; Johnstone, Cheryl; Loong, Margaret; Matecsa, John; Guttmann, Astrid; Schull, Michael J. title: Public engagement can change your research, but how can it change your research institution? ICES case study date: 2020-09-30 journal: International journal of population data science DOI: 10.23889/ijpds.v5i3.1364 sha: 6719547a372f2bdd5f6f2228efc2cffa930799b2 doc_id: 864218 cord_uid: me27fndn This article explores the approach that ICES (formerly the Institute for Clinical Evaluative Sciences) uses to encourage public engagement at both the research study and corporate level. ICES is an independent not-for-profit research institute in the province of Ontario, Canada. This article was co-written by ICES’ public engagement team and four members of the ICES Public Advisory Council (PAC). As part of the process of writing this article PAC members provided their reflections on why they got involved, what worked well and the limitations and challenges of ICES’ approach. ICES described the development of its public engagement strategy to inform how the institution would capture and incorporate the values of Ontarians in ICES activities and research. ICES provided details on two key elements of its strategy: the formation of a PAC to advise its leadership, and the creation of resources and supports to encourage researchers to incorporate public engagement in their projects. PAC members and ICES provided perspectives on what impact they perceive as a result of the public engagement strategy. PAC members expressed that ICES has demonstrated listening to and using their input, but it is too early to evaluate if their feedback has changed the way ICES conducts its work. ICES discussed the challenges and successes in building and implementing the public engagement strategy, including recruiting a diverse council, aligning with public priorities and creating a culture of engagement. As a result of public input, ICES has restructured the way the institution explains its privacy and cybersecurity approach to build trust and confidence. ICES has also seen an increase in researchers using public engagement resources, and early data suggests that in 2019 about 20% of scientists included some form of public engagement in their projects. ICES’ journey to public engagement resulted in important changes to processes and activities at the institution, but there is much more that needs to be done. PAC members advocate that public members should be engaged in health data research and hope that public input will be a core element in health data research in the future. ICES will continue its efforts to address public priorities and will seek to further evaluate the impact of public engagement across the organisation. This article was co-written by ICES' public engagement team and four members of the ICES Public Advisory Council. The content in this article is a synthesis of perspectives and reflections from both the institution and public members. Public involvement and engagement are gaining recognition as an important practice to improve the design, conduct, dissemination and impact of health research [1, 2] . Public involvement and engagement can occur within an individual research study and can also be undertaken more broadly by a article explores the approach that ICES (formerly the Institute for Clinical Evaluative Sciences) uses to encourage public engagement at both the research study and corporate level. ICES is an independent not-for-profit research institute in the province of Ontario, Canada. Since 1992, ICES has been entrusted with securely holding and analysing the administrative and other health data records collected by the provincial health system for 13.4 million Ontarians. ICES' community of almost 500 research, data and clinical experts access these data to improve policies and services that support the health of Ontarians. ICES also provides external access to these data for projects conducted by third party researchers and facilitates data analysis for other health system stakeholders and government organisations [6] . ICES began formal public engagement activities in 2017 to incorporate public input into the institution's decisions, activities and research. Prior to this, interaction with the public was largely limited to the dissemination of research findings through media releases and some focus groups [7] as ICES lacked any ongoing mechanism to engage the public in its work. With ICES' core work relying on the personal health information of all Ontario citizens, the institution's leadership recognised that they must understand Ontarians' values and perspectives to continue to use these data responsibly. Multiple factors within the research space, both within Canada and internationally, precipitated ICES' development of a strategy for public engagement. Within Canada, the Strategy for Patient-Oriented Research (SPOR), led by the Canadian Institutes of Health Research, began a national movement toward patient engagement in research. SPOR supports evidence-based health care by creating a continuum of research from studies to practice, part of which involves enhancing the role of patient input in research [8] . Currently, public engagement is not required by many of the prominent research funding organisations in Canada, however, grant review panels have begun noting that public engagement would be an asset in some research projects. Internationally, examples like care.data in England [9] demonstrated the need for governments to engage with the public to understand and operate within social licence [10] . ICES also had relationships with other data organisations that were actively engaging the public, including the Farr Institute in the United Kingdom and Population Data BC, and participated in the development of a consensus statement on public engagement [11] . In addition, with the broadening of research access to ICES' data platform, including the creation of a virtual access model for private sector researchers, ICES recognised a need to better understand and be guided by public values. Previously, ICES focused primarily on working within the research community, aiming the institution's efforts towards academic and government audiences. Given the lack of a public profile for ICES, one of the organisation's Public Advisory Council (PAC) members described it as "Ontario's Best Kept Secret". ICES' 2017 strategic plan was the first to identify public engagement as a priority, which led to ICES speaking more publicly about the institution and its role as a data steward. In 2018, ICES developed a public engagement strategy to inform how the institution would work with members of the public and address its new strategic priority. The public engagement strategy outlines a series of mechanisms to capture and incorporate the values of Ontarians in ICES activities and research at all levels of the organisation. The strategy was developed in line with both Canadian and international best practices for patient involvement and engagement, including those defined by the Canadian Institutes for Health Research, INVOLVE in the United Kingdom, and the International Association of Public Participation (IAP2) Core Values for the Practice of Public Participation [4, 5, 12] . Within the strategy, ICES outlines its principles for engagement which define the values which ICES uses to ensure meaningful public engagement ( Figure 1 ). Other frameworks influenced ICES' principles, including the SPOR Patient Engagement Framework guiding principles [13] and the Patient-Centered Outcomes Research Institute engagement principles [14] . ICES' public engagement team designed the strategy with the intent of having a two-tiered approach: public input to influence organisational decision-making and support for researchers to have smaller, project-specific engagement mechanisms. This focused ICES' public engagement activities into the four key pillars described below ( Figure 2 ). This article concentrates on ICES' approach to the first two of these pillars: ICES' PAC and resources for staff and researchers. The strategy also includes an evaluation framework which identifies ICES' intended public engagement outcomes and is used to evaluate public engagement activities and the impact of these activities on ICES' work. The evaluation framework connects the goal of ICES' public engagement strategy to the key pillars (activities) of the strategy. Each activity is related to a series of intended outcomes which have a defined metric or output that can be used to monitor and measure progress towards the defined goal ( Table 1) . The outcomes and metrics were influenced by frameworks and approaches for evaluating engagement from both Canada and the UK [15, 16, 17] . Metrics from this evaluation framework are reported annually to the ICES' Board of Directors and Executive to monitor progress of the public engagement strategy. ICES also created Key Performance Indicators to report annual benchmarks on the number of strategic decisions that include public input, and the number of research projects that include public engagement. The ICES Data Repository consists of coded and linkable health datasets that are captured through Ontario's health system and other administrative, demographic and social sector data; this means that all residents of the province of Ontario are potentially captured within ICES' datasets. For this reason, ICES chose to use the term "public engagement" instead of "patient engagement" as the •We will strive to create engagement strategies that capture the diversity of people living in Ontario. •We will co-design engagement mechanisms and build rela�onships between the public and the ICES team. •We will value the opinions of the public and reflect these opinions wherever possible. •We will communicate informa�on, including risks, in a truthful and accessible manner to support the public in making informed decisions. •We will report back on how feedback from the public is applied and explain when it is not. Solicit public perspec�ves to guide the organiza�on Build awareness of our work and our mission institution aimed to engage with all those who contribute to its Data Repository ( Figure 3 ). The ICES PAC also supported the use of the word "public", as they do not identify as patients except when they are actively receiving medical care. The ICES PAC is one of the key pillars of ICES' public engagement strategy. It is made up of 20 members of the public from across the province of Ontario • # of topics brought to the Public Advisory Council for feedback Better integration of public values in the ICES decision-making process. • # of instances where public feedback shaped an ICES decision Greater alignment of ICES activities with public values. • # of ICES organisation-wide activities that incorporate public perspectives Increased public engagement resources available for ICES researchers. • # pubic engagement resources sourced/created Increased understanding of public engagement in research and its benefits. • % increase of researchers using public engagement tools and resources Expanded public engagement activities within ICES research projects. • % increase of researchers doing public engagement in their projects Better recognition of ICES' commitment to public engagement. • # partnership opportunities involving public engagement Better public awareness and understanding of ICES and ICES research. • # communications materials created to explain ICES work to a public audience Better public awareness and understanding of ICES commitment to privacy and cybersecurity Council and other public members understanding of ICES privacy and cybersecurity procedures Increased number of public engagement activities at various levels of the organisation. • Total # of public engagement activities at all levels Members of the PAC were selected to reflect the diverse perspectives of people from across Ontario whose data are in the ICES Data Repository to ensure that the council represented the lived experiences of the community [18] . Any individual who resides in Ontario was invited to apply online to be part of the PAC. The call for applicants was shared through social media channels, local newspapers, in libraries and recreational centres, and by partner organisations. Recruitment was conducted through online application as an expectation that further communication via online access would be required in the role. The online application process did not appear to limit applicants by age or socio-economic status. Applications included demographic information such as city of residence, age, gender, race/ethnicity, education level, and specific questions related to self-identification such as identifying as French-Canadian or LGBTQ2. ICES consulted with Health Quality Ontario, the provincial organisation responsible for ensuring quality of health, to design the application form, based on their leadership in public engagement in health care [19] . The application also asked open-ended questions about their interest in health data research, why they wanted to get involved, a short autobiography and their understanding of the commitment. The Manager of Public Engagement and the Director of Human Resources interviewed shortlisted applicants by phone to assess if they would be an appropriate fit for a diverse and interactive council. Applicants were not required to have any educational qualifications or training in research, science, healthcare or data. Applicants who had professional roles in healthcare and health policy were excluded to ensure that institutional knowledge and biases about health data or policy did not influence public members perspectives. When creating the PAC, particular care was taken to capture perspectives of the Ontario public across sex, geography, age, ethnic and cultural diversity, education level and prior experience as a public advisor. Members were asked to participate for a two-year term, from March 2019 to March 2021. They were given an extensive orientation package with background information on ICES and health data research and were invited to a virtual orientation People with lived experience with a specific disease or disorder whose data you are using. session with a Question/Answer period with ICES senior staff. The PAC meets four times a year for two in-person meetings (5-6 hours in length) and two teleconference meetings (2 hours in length). The required commitment is 14-16 hours per year. Additional opportunities arise for PAC members to be involved, like project working groups or research projects, and they opt-in on a volunteer basis. Typically, ICES will have 5-10 additional opportunities for PAC involvement per year. PAC members are compensated $20 (CAD) per hour for time spent in meetings, based on Canadian public/patient advisor compensation standards. Their honorarium is provided in the form of gift cards from selected locations to allow for ease of distribution [20] . The PAC works directly with the Manager of Public Engagement, who reports to the CEO and Chief Science Officer of ICES. The Manager of Public Engagement, CEO and Chief Science Officer are present at all PAC meetings. This is to ensure that PAC feedback is received by the most senior staff of the organisation, who can then implement their feedback at the corporate level. Other ICES management and staff are often present at meetings to provide background information or guidance from their specific areas of expertise. Occasionally research projects seek feedback from the PAC, but due to the PAC's more broad and general perspective this is project dependent. In between meetings, ICES continues to communicate with the PAC via email communications and follow-ups on topic discussions. A member of ICES' Board of Directors attends most of the in-person meetings and acts as Board liaison. Summaries are prepared after each meeting and shared with the ICES Executive team and any other interested parties both within ICES and more broadly. For ICES to truly represent the values of Ontarians, public input at both the corporate and research project level is required. Within any given year, ICES typically has over 500 active research projects, each with distinct topics and objectives and thus equally different cross-sections of public data. Naturally, this poses challenges for encouraging public engagement within research projects, implying that each project team would need to develop an engagement plan that would be specific to their study population. At the onset of the public engagement strategy, ICES was aware of only five projects that had actively engaged members of the public and incorporated input into their research. Due to the nature of how ICES research is conducted, the institution chose not to make it mandatory for all projects to include public engagement. Instead, scientists could opt to include engagement activities in their projects and leverage the resources and expertise provided by the public engagement team as needed. Considerable effort was made to include public engagement in key corporate communications to raise the profile of the strategy and the resources available. ICES scientists and staff were asked to complete a survey to identify which public engagement topics they would most like to have resources for; the results of the survey are described below (Figure 4) . The survey consisted of an extensive list of topics and practices based on the typical steps of a public engagement mechanism or plan which were based on both suspected need and other Canadian and international resources [4, 5, 13] . Scientists were then asked to select their personal top five topics for which they desired resources, which in turn informed the most requested list. In response to these results, an internal resource hub was built to house public engagement tools for researchers. This resource hub includes ICES created tools and links to trusted international sources to guide scientists through the steps of engagement including best practices from SPOR in Canada, INVOLVE and the National Coordinating Centre for Public Engagement in the UK. Specific tools were designed to fit seamlessly into the existing research project pipeline to make it easier for scientists to include and report on engagement work. ICES scientists are offered the opportunity to do one-onone consultations with the Manager of Public Engagement and walk through how they might conduct engagement within their project given the study population, timelines and budget. How can public engagement change your research institute? Over the past two years ICES has seen growth, change and challenges as it embarked on its public engagement strategy. In some instances, the efforts to capture public values in ICES work were successful; in others they moved slowly and less effectively than anticipated. The section below provides reflections from both the ICES PAC and ICES itself on the successes and challenges of the public engagement strategy, and whether each party feels that change is evident in the organisation as a result. To best capture the strengths and limitations of the ICES PAC model, PAC members were asked to write their reflections on the process so far by answering the following questions. The discussion below was co-written by 4 PAC members and reviewed by all members. Each member had different reasons for getting involved with the ICES PAC. Some got involved because of an interest in healthcare or data analytics; others to bring a particular voice to the table and advocate for their community, such as rural or northern communities, specific ethnic groups or new immigrants. However, the overarching reason that members got involved was personal experience. Many members had personal experiences with health care, whether directly or as a caregiver for friends or family members. These personal experiences may have been positive or negative, but most importantly they were ones that members wanted to share. Being public advisors to ICES allows them to share their perspectives and contribute to relevant and important issues related to ICES' mission and priorities. What do you think has worked well from your experience with the PAC? The PAC selection process effectively reflected diversity by capturing the differing viewpoints found in the population, as consideration was given to ethnic, cultural and geographical differences. The current members of the PAC are engaged and contribute actively; each member brings skills, expertise and, most importantly, an open mind. As well, having an adept facilitator run the meetings is integral to achieving a positive experience for the PAC that yields meaningful results. The facilitator fosters a respectful environment in which all PAC members are encouraged to give candid opinions, whether brainstorming at in-person meetings or collaborating during a webinar. Agendas are written with detail and shared in advance, and meetings follow agendas closely. These combined attributes promote progression, idea-generation and cohesion, and optimise the experience for PAC members. Although the PAC is diverse, ICES can continue to better represent society within its membership. For example, an Most requested public engagement resources by ICES scien�sts and staff insufficient number of males and young adults applied, resulting in a lack of representation with respect to those voices. However, the size of the group must also be considered, as increasing membership to more than 20 may hinder group dynamics. All PAC members started their two-year term at the same time and some goals will take more than two years to complete. During the first year the members have been learning about the organisation and how to participate as part of the council. The second year will likely be where members can truly contribute to making change at the organisation. Some PAC members expressed that working inperson is ideal, but most meetings have had to rely on remote collaboration because of expense, geographic distribution, competing priorities and the COVID-19 pandemic. In some cases, activities or discussions lacked context or the context was not effectively communicated to the PAC. More information could be provided with how the PAC contributes to ICES' objectives at-large. Many of the priorities set by the PAC have pertained to ICES broadening its communications to the general public and building awareness of the institution and health data research. These priorities do not align with ICES' communication strategy, and the institution continues to focus on the research community. Also, ICES' departments are defined and separate which makes it challenging to have one central voice when engaging the general public and other audiences. ICES has demonstrated listening to and using PAC input, but one year of involvement is not enough time to evaluate if the feedback collected has changed the way ICES conducts its work. The PAC remains positive that there will be an impact and visible change to the way ICES operates in the future but acknowledges that due to the structure and nature of non-profit organisations, these changes might not always be immediately evident, and more time is required for shifts to occur. In addition to council meetings, PAC members also participate in smaller side projects, such as reviewing impact stories or evaluating information on the ICES website. These side projects allow PAC members to work with other ICES staff and departments, and there has been evidence of PAC input leading to changes in these areas. However, the results of these smaller side projects were not enough for PAC members to feel that their inputs have changed the way ICES does its work. This demonstrates that change needs to occur at a larger scale for members to feel their work has provided significant impact. The PAC feels that it will continue to grow in responsibility and importance. ICES had trepidations about beginning public engagement efforts as an organization that had been using public data for over 20 years out of the public view [10] . However, initial discussions with both focus groups and the PAC demonstrated a general public awareness and acceptance that the Ontario government provided public data for research in some capacity. This sentiment is in line with broader public opinions of linked heath data being used by third parties for research [2, 7] . The PAC specifically said that ICES' track record of publicly available research imbued trust in the organisation and its use of health data for research. Recruitment of a diverse, province-wide public council posed several challenges. It was particularly difficult to find male volunteers, and ICES had to adjust selection criteria to ensure that there was equal representation of both male and female members. Ontario is a large province and health care delivery differs significantly in the northern and southern parts of the province. ICES targeted recruitment specifically to northern Ontario by approaching media outlets in Sudbury, a major city centre that is connected to the more northern and rural parts of the province. The articles and radio interviews that resulted from this pitch led to a significant number of applicants from Sudbury and surrounding area. Overall, ICES received over 140 applications for the PAC, with social media being the most listed method of hearing about the application. ICES attributes this significant number of applications with having both targeted and broad sharing of the recruitment call through partners and the transparency and openness of the call. Recruitment materials made it clear that applications were open to anyone and stated the time commitment which was quite minimal. No mention of compensation or remuneration was mentioned on the recruitment flyer. ICES then individually interviewed each applicant to gauge their contributions to the PAC. Shortlisting applicants and conducting over 50 phone interviews was very time consuming. However, this targeted and two-stage recruitment assisted in creating a balanced and engaged PAC. The PAC collectively identified the health data research topics that were of most interest to them in their first meeting. These priority topics ensured that discussions are driven both by PAC and ICES interests. For example, the PAC identified that they are interested in privacy and cybersecurity and that the way ICES explains its privacy approaches is not effective in answering their questions. Historically, ICES described its privacy procedures in relation to privacy law to justify why the institution is allowed to access and use health data. The PAC felt that it would be more helpful to discuss how ICES keeps data safe, how data are used, and what would happen if there was a breach. As a result, ICES' Chief Privacy and Legal Officer is working with the PAC to restructure the way the institution explains its privacy and cybersecurity approach to build trust and confidence. ICES aims to continuously seek PAC feedback and then follow-up on how their input is used, or not used, on a timely basis. PAC members often give feedback multiple times throughout the progress of an initiative or decision to ensure the process is open and transparent. However, some PAC priorities do not align with departmental priorities and ICES is unable to act quickly on PAC feedback due to readiness, timing and funding. A key example of this is the issue of broader engagement of the general public and public awareness of ICES and ICES research. The PAC has expressed that all Ontarians should know about ICES and how administrative data is used for health data research. They feel that ICES should focus on building awareness of the organisation as a key communications priority. This criticism is an expected and welcomed part of seeking advice from the PAC, however ICES has not yet been able to address this PAC priority. As the institution is just beginning to develop its public-facing narratives, ICES must manage expectations for how quickly it can develop communications as it does not have the capacity or funding to do province-wide awareness campaigns. ICES continues to embed public perspectives across the organisation and has identified broadening its public engagement activities as an objective in its new strategic plan. Overall, the creation of the PAC resulted in ICES' leadership hearing from a new perspective, that of the public whose data are the core commodity of the organisation. The presence of ICES' leadership team and Board at PAC meetings elevates their input and perspectives to the level of decisionmaking. The PAC has changed the way the institution thinks about key issues, and public perspective has become a more consistent component of strategic decisions. Within their first year, ICES leadership team has brought five strategic issues to the PAC for input, three of which resulted in decisions that were shaped using their input. For example, the PAC played a key role in shaping ICES' new strategic plan, providing feedback on strategic priorities and commitments for the next three years across the organisation. In some cases, the PAC has repositioned the organisation's approach, validated the importance of a plan, or contradicted assumptions that the ICES leadership have held, as seen with the privacy example above. However, more work needs to be done and ICES needs to continue to demonstrate that it is listening and using public input to change the way the organisation does its work. Across the institution, the value of the PAC is becoming recognised as a crucial element to ensure that ICES can continue to use public health data for its work. Encouraging research teams to conduct engagement in their projects is an ongoing process. In Canada, most funding bodies do not require research projects to conduct public involvement or engagement. ICES scientists are funded by said research bodies and thus ICES chose not to make it a requirement for scientists to include public engagement in their project. Before the establishment of the ICES public engagement strategy, few ICES scientists had carried out any public engagement, and many expressed that they were unsure how to incorporate it into their projects. ICES researchers also described several hurdles to developing robust public engagement plans for their projects including the uncertainty of funding and the lag period between grant submission and receipt. Since many ICES projects leverage provincial population data without identifiers, researchers also expressed challenges to recruiting province-wide public perspectives. To mitigate these concerns, the public engagement team targeted their efforts to create tools that could be embedded into existing research processes and make it easy for scientists to begin engaging the public. For example, to assist scientists with their grants, ICES provides a modifiable boilerplate paragraph to include in grant proposals with fill-in fields for the specifics of the engagement plan. Scientists can also select a modified letter of support from the institution with a description of ICES-provided resources for public engagement. To aid in recruitment, ICES invited public members interested in participating in engagement activities to join a mailing list, which then acts as a distribution list for research team recruitment calls. Scientists can also bring their projects to the PAC to gain a more general public perspective. The resource hub is an effective way to provide tools for scientists who are already interested in engaging the public in their projects. However, passively offering support was not effective for helping those scientists who were just starting to consider public engagement and a more handson approach was needed. The public engagement team now works more closely with the research department to identify scientists or projects that are amenable to engaging the public. Those teams are then offered one-on-one consultations to receive specific and realistic guidance for how to build a public engagement mechanism based on the study population, budget, timelines and where the study team would like public input. Although growing a culture of engagement within ICES has taken more time than anticipated, there is a notable shift in the awareness of public engagement amongst the ICES research community. Public engagement is a frequent discussion at faculty and staff meetings, and ICES researchers are more readily accessing the supports that are offered. Beyond education and awareness, a key effort of this work was to develop mechanisms to capture and track public engagement across the organisation. ICES used the embedded tools and other existing science reporting structures to monitor the public engagement metrics from the evaluation framework and capture engagement work being done within research projects. For example, in 2019, 18 grants requested the modified letter of support with public engagement language. Thus far in 2020, 27 projects have requested the modified letter of support and 23 projects have indicated that they are including public engagement in their grant. Early data on science activity suggests that about 20% of scientists are including some form of public engagement in their projects. Whether it's a single public advisor or a 20-person advisory panel, this is demonstrative of a movement towards ICES scientists actively incorporating public input into their research. The responses below were co-written by 4 PAC members about their final perspectives on engagement in health data research. Why should the public be engaged in health data research? New and emerging technologies have made data collection more prevalent and the public has become more aware of their data. The public has a right to know that their data are being collected and what they are being used for. They should also be made aware of the stringent privacy measures that keep their data safe and the many ongoing data-informed research advances to health care. Public engagement on key issues can alleviate concerns around health data research and demonstrate that the use of these data produces significant benefits. Engagement also allows underrepresented voices an opportunity to influence the studies that lead to health policy development. What are your hopes for the future with regards to public engagement at ICES? In the future, the hope is that ICES will succeed in engaging the broader public and provide more opportunities for the public to be involved. ICES will broaden its communications activities and be more present on social media and blogs. Health data will become more open source to encourage discovery across the research community. As health data research continues to grow, scientists should consider the rights of the public and strive for more transparency and personal access. Health research will aim for proper representation of all voices, especially the underrepresented, and that public engagement will act as a safe space for the public to speak out. Ultimately, the hope is that health data research will continue to yield findings that enhance health care and positively impact health policy, with public input as a core element. ICES' journey to public engagement has resulted in important changes to processes and activities at the institution, though there is much more that needs to be done. To elevate the importance of the public voice, ICES has created a new public engagement strategy, a new department, and its PAC. The ICES research community is broadening its approach to accommodate for public input into their use of health data. These are still early phases of the public engagement strategy and the impacts of these changes are still to come. There is hope and intention that public engagement will continue to influence the work of the organisation. ICES has many plans for the future, including addressing the PAC's goal of building awareness of the institution and health data research. ICES is looking for ways to better integrate engagement across the organisation so that public input can be seen in decisionmaking at every level. As stated by one PAC member, "public engagement is not just a good idea, it's a necessary one". Patient engagement in research: a systematic review. BMC health services research Public responses to the sharing and linkage of health data for research purposes: a systematic review and thematic synthesis of qualitative studies. BMC medical ethics Canadian Institutes of Health Research. Strategy for Patient-Oriented Research -Patient Engagement Canadian Institutes of Health Research Briefing notes for researchers: involving the public in NHS, public health and social care research. London: INVOLVE ICES: Data, Discovery, Better Health. IJPDS Social licence and the general public's attitudes toward research based on linked administrative health data: a qualitative study Canadian Institutes of Health Research. Canada's Strategy for Patient-Oriented Research NHS to scrap single database of patients' medical details. Guardian Data Protection The social licence for research: why care.data ran into trouble Consensus statement on public involvement and engagement with data-intensive health research. IJPDS International Association for Public Participation. Core Values for the Practice of Public Participation Denver: IAP2 Strategy for Patient-Oriented Research -Patient Engagement Framework Patient-Centered Outcomes Research Institute. PCORI Engagement Rubric Making a Difference: A guide to evaluating public participation in central government. London: Involve and Shared Practice National Co-ordinating Centre for Public Engagement. How to evaluate public engagement projects and programmes Public and patient engagement evaluation tool (PPEET) Toronto: Health Quality Ontario Canadian Institutes of Health Research. Considerations when paying patient partners in research ICES would like to thank the members of the ICES Public Advisory Council who were involved in both writing and reviewing the article to ensure it best captured their perspectives as public advisors. ICES would also like to thank Alison Paprica who led the early quantitative studies on public perception of health data research and provided feedback and guidance on the paper. ICES is funded by an annual grant from the Ontario Ministry of Health (MOH). All authors have no conflicts of interest to declare. This manuscript did not involve primary research on human subjects and was not submitted for Research Ethics Board approval.