key: cord-0923712-ft6ew54c
authors: Small, Nicola; Ong, Bie Nio; Lewis, Annmarie; Allen, Dawn; Bagshaw, Nigel; Nahar, Papreen; Sanders, Caroline
title: Co-designing new tools for collecting, analysing and presenting patient experience data in NHS services: working in partnership with patients and carers
date: 2021-11-27
journal: Res Involv Engagem
DOI: 10.1186/s40900-021-00329-3
sha: 2f3a52e9ad8d2a23bca3173e42031e5035ed4d8b
doc_id: 923712
cord_uid: ft6ew54c

BACKGROUND: The way we collect and use patient experience data is vital to optimise the quality and safety of health services. Yet, some patients and carers do not give feedback because of the limited ways data is collected, analysed and presented. In this study, we worked together with researchers, staff, patient and carer participants, and patient and public involvement and engagement (PPIE) contributors, to co-design new tools for the collection and use of patient experience data in multiple health settings. This paper outlines how the range of PPIE and research activities enabled the co-design of new tools to collect patient experience data. METHODS: Eight public contributors represented a range of relevant patient and carer experiences in specialist services with varied levels of PPIE experience, and eleven members of Patient and Participation Groups (PPGs) from two general practices formed our PPIE group at the start of the study. Slide sets were used to trigger co-design discussions with staff, patient and carer research participants, and PPIE contributors. Feedback from PPIE contributors alongside verbatim quotes from staff, patient and carer research participants is presented in relation to the themes from the research data. RESULTS: PPIE insights from four themes: capturing experience data; adopting digital or non-digital tools; ensuring privacy and confidentiality; and co-design of a suite of new tools with guidance, informed joint decisions on the shaping of the tools and how these were implemented. Our PPIE contributors took different roles during co-design and testing of the new tools, which supported co-production of the study. CONCLUSIONS: Our experiences of developing multiple components of PPIE work for this complex study demonstrates the importance of tailoring PPIE to suit different settings, and to maximise individual strengths and capacity. Our study shows the value of bringing diverse experiences together, putting patients and carers at the heart of improving NHS services, and a shared approach to managing involvement in co-design, with the effects shown through the research process, outcomes and the partnership. We reflect on how we worked together to create a supportive environment when unforeseen challenges emerged (such as, sudden bereavement). SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s40900-021-00329-3.

National policy encourages the embedding of Patient and Public Involvement (PPI) in research as a means to improve both the relevance and meaningfulness of applied health services research in England [1] , and to ensure research improves the nation's health and wellbeing [2, 3] . Public involvement in research has previously been defined as 'research being carried out with or by members of the public' , including patients and carers, 'rather than to, about, or for' them (Pg. 1 [4] ). Patient involvement is commonly enabled through establishing a dedicated group to provide inputs for the study duration with public contributors sometimes included as coinvestigators to inform the study design from the earliest stage [5, 6] . However, concerns remain that PPI is exclusionary, challenging the relevance and meaningfulness of such work [7] . The National Institute for Health Research (NIHR) have produced a 'Values and Principles' of framework of evidence-based practice for PPI [8] , and the publication of UK Standards for Public Involvement [9] , has been designed to help to address recurring challenges to enable more inclusive and collaborative working, as well as supporting best practice [10] .

A number of studies have assessed how patients and carers can be meaningfully involved as contributors to research from pre-funding, through to the end of the study and beyond [11] [12] [13] , demonstrating the value of PPI and engagement (PPIE) input for shaping research, co-developing (that is, jointly developed), and delivering impactful outcomes [13, 14] . The number of PPIE Journals are increasing [15] , and the inclusion of the GRIPP2 reporting checklist and other tools have improved reporting, yet more needs to be done regarding impact assessment [16] [17] [18] [19] . Co-production is one approach of obtaining multiple perspectives and sharing views, reflection, learning, from the start to the end of the research [20] [21] [22] . This approach is rooted in participatory research, enabling people to become involved in the shaping, design and testing of new healthcare interventions that are patient-centred [23] . Co-design seems to be most effective as a method, when the importance of the relational aspect to involvement are emphasised [24] . Co-design involves close collaboration between PPIE contributors, and the research team [25, 26] . This happens through: sharing of power, perspectives and skills; respecting and valuing everyone's knowledge; and reciprocating, building and sustaining relationships [20, 22, 27] . Further features are essential: ongoing dialogue; joint ownership of key decisions; relationship building; valuing and evaluating [20, 28] , and working with PPIE contributors to encourage a reflective culture. When designing digital health interventions, these should equally be based on patient and carer experiences [29] [30] [31] . Finally, PPIE in co-design is context dependent [32, 33] and thus it is important to outline the approach taken in the co-design of an intervention.

Keywords: Patient and public involvement and engagement, Patient participatory groups, Patient experience, Co-design, Co-production, Digital participation, Qualitative health service research, Mental health services, Rheumatology, Primary care Plain English summary The way we collect and use patient experience data is important because of concern that patients and carers may be excluded by the limited ways it is currently done in NHS services. In this study, we worked in partnership with researchers, staff, patient and carer participants, and patient and public involvement and engagement contributors, to co-design new tools for the collection, analysis and presentation of patient experience data. We focused on services for people with musculoskeletal conditions and services for people with severe mental health conditions. Our PPIE group, formed at the start of the study, represented a range of relevant and diverse health experiences from patients and carers of specialist services, and primary care. The aim of this paper is to share our experiences from working in partnership with our PPIE contributors on the co-design work of the study. Illustrations of how the PPIE activities added crucial insights in the shaping of the tools are given alongside the research data from patients, carers and staff participants. We experienced some challenges during the project. We discuss how we managed to work together to create a supportive environment when unforeseen challenges emerged (such as, sudden bereavement). Our experiences of developing multiple components of PPIE work for this study demonstrates the importance of tailoring PPIE to suit different settings, and to complement people's strengths and capacity. It also shows the value of bringing diverse experiences together enabling a shared approach to co-design. Researchers and PPIE contributors wrote this paper jointly.

In this paper, we focus on the PPIE and co-design components of a project entitled 'Developing and Enhancing the Usefulness of Patient Experience and Narrative Data' (the DEPEND study: [34] ). While a dedicated PPIE group was active throughout the mixed-methods study, it was particularly important in the co-design of the new tools and associated guidance (the 'toolkit') and their evaluation, to ensure that the perspectives of patients and carers were at the heart of the study. This paper outlines in detail the significant PPIE component in the co-design approach in relation to three aspects: the process; the research components (i.e. the tools co-designed for collecting, analysing and presenting patient experience data); and the impact on the research partners, that is the research team and the PPIE contributors as advisers to the study.

The DEPEND study was developed in response to a dedicated call from the National Institute of Health Research whose brief posed the question: "What research is needed to make patient feedback data more credible and useful?" [35] . Patient experience data is currently routinely collected within the NHS using a range of methods, including the national Friends and Family Test (FFT) [36, 37] , the Picker survey [38] , and many condition-specific [39] , or organisation-specific online feedback surveys [40] , some of which are validated. While the attention paid to patient experiences of health care is welcome, a number of issues remain. Research has shown that health care staff are often sceptical of the relevance of patient experience data to local services [41] . Moreover, it is argued that the commonly used formats used to collect experience data tend to exclude patients and carers who might not be able to report their experiences of care.

The evidence suggests that patient experience should be measured in terms of the journey as experienced by the patient in order to capture transitions in care and continuity [42, 43] . This understanding of the dynamics of long-term health conditions, be they mental and/or physical, demands a re-think of what constitutes meaningful feedback [42] . Similarly, recent research on the measurement and improvement of patient experience in primary care [44] and specialist services [45] highlights that surveys may be insufficient to fully capture patient feedback. It has also been suggested that surveys should be made more useful and relevant to staff, and alternative feedback methods should be developed to suit context and staff roles [35, 37, 46] . Increasingly, NHS organisations collect large amounts of data, yet the analysis and reporting of patient feedback is variable because of inefficiencies, lack of expertise, capacity and resources [34] . This is particularly the case with narrative data and the extensive store of patient stories now available has prompted some studies to develop ways of synthesising this material so that it can be used for service improvement. Exploring alternative methods, such as adopting a machine learning classification approach has been reported in the literature [47] [48] [49] . The DEPEND study builds on the aforementioned literature and this is reported elsewhere [34] but in essence it used a collaborative approach to co-design by inviting staff, patient and carer research participants, and PPIE contributors, to share experiences, knowledge and resources, to co-design new tools for collecting, analysing and presenting patient experience data in NHS settings.

The aim of the involvement component to the DEPEND study was to inform the research process within each of the four parts ('Work streams') to the study to ensure integration of involvement into the intervention development process from the start (see Fig. 1 ). This paper will specifically report on the PPIE involvement in the codesign work stream 3 of the study to show how each PPIE co-design activity informed the shaping of the new tools, theoutput and the impact. We draw on exemplar data extracts from the staff, patient and carer research participants collected through focus groups and interviews, to illustrate how our PPIE group, informed the subsequent co-design of the intervention. Further, we also report how our PPIE contributors chose specific roles based on their expertise and capacity that impacted on their levels of involvement in co-design. Lastly, we reflect on our experiences as a partnership.

This paper focuses on the co-design element of the DEPEND study and has been guided by the GRIPP2 reporting guidance to improve the reporting of PPIE and transparency [16] . The GRIPP2 checklist (long format) was used as a guidance of reporting PPIE in this paper (See Additional file 1). Our paper is structured thematically illuminating how involvement derived insights informed the co-design process, analysis and outputs. It was jointly written with the researchers and three PPIE contributors (AML, DA, NB) to offer a shared reflection on the PPIE in co-design process, the research outputs, and the significant impact on the research partnership. [50] , which highlights key improvement areas for ensuring people have positive experiences of care in CMH services, OPT clinics, and GP services. The co-design approach adopted draws conceptually on Experience-Based Design (EBD: [51] ) theory that defines experience as a reconstruction, or a reconstitution of something that individuals have lived through. Meaning is given to that experience in a reflective and retrospective manner through words. This specific kind of knowledge is gained from close and direct personal observation, or contact, in order to design improved services and a better experience for patients or carers. It is conceived as a partnership, involving researcher, staff, patient and carer research participants and PPIE contributors, working as co-designers of services throughout the change process [51] .

Patient research participants were recruited through staff in the clinical sites; staff research participants were recuited via clinical research leads at each site. Research participants were invited to take part in either a focus group or a face-to-face individual interview, or both, as determined by participant preference. Qualitative data collected from staff, patients and carers participants from the first phase of data collection was summarised and discussed in follow-up co-design focus groups, with 57% of staff and patient and carer participants having previously taken part in the qualitative components in the first phase (see Table 1 ). We achieved a diverse sample of staff participants with roles in management and patient experience, clinical and information technology. The total number of people who participated in the three data collection phases for DEPEND is reported within the final report [34] . 

At the start of the DEPEND study, a PPIE group was created with three PPIE contributors who had experience of specialist services for musculoskeletal health conditions, and five PPIE contributors receiving treatment for serious mental health conditions; two of five PPIE contributors represented a dual perspective as carers and patients in the study. Two PPIE contributors were co-applicants to the study (AML and DA); they attended and contributed to the bi-monthly project management meetings, and the quarterly Study Steering Group, and favoured a more active role in the group. Both contributors were experienced in PPIE work having previously partnered with research teams to co-design digital interventions and disseminating the research findings on behalf of the research team [52, 53] . Representatives from two Patient Participatory Groups (PPGs) were next recruited as PPIE contributors to the study via the two primary care sites Lead clinicians (Site C1 and Site C2: [54] ). The PPG in Site C1 was a relatively small group with nine members, four of whom chose to support the DEPEND study as involvement contributors. Conversely, the PPG in primary care site C2 comprised of 40 + members: 7 members who regularly attended faceto-face PPG meetings chose to support the DEPEND study [55] . The Assistant Practice Manager managed the PPG, circulating the minutes produced by the PPG coordinator (and co-author of this paper, NB) and liaised with members.

All PPIE contributors to the study were reimbursed for their time (INVOLVE rates: [56] ) and travel. A PPIE document was developed and modified as the study progressed in line with an established Terms of Reference document developed by the Primary Care Research in Manchester Engagement Resource (PRIMER: an established PPIE group funded by the NIHR School for Primary Care Research at the University of Manchester [57] ). The document adhered to the principles of roles and responsibilities of INVOLVE [8] , and local Faculty guidance [58] .

Face-to-face involvement meetings were held in groups for each phase of the research study at the University or at each primary care site so PPIE insights could feed directly into the four work streams (see Fig. 1 ). Altogether, four PPIE group meetings were held at the University, and five PPG meetings at each site. Each PPIE/ PPG meeting was devoted to a specific topic that reflected the objective of each work streame in the DEPEND study [34] : (1) collation of current perspectives of collection, analysis and presentation of patient experience data; (2) co-design of the toolkit components and accompanying bespoke guidance; (3) testing out and evaluation of the new tools; and (4) PPIE-led dissemination activities. Research documents and an agenda were provided in advance of the meeting by email, or mail, and hard copies were available on the day. Each PPIE activity and contributions were recorded by the PPIE Co-ordinator and project researcher (NS), and circulated to everyone attending the meeting to check for accuracy. Insights from previous discussions were included in the prepared slide set to show how these informed the ongoing research data collection and qualitative analysis. Following the meetings, a working document was circulated where PPIE preferred roles and actions were outlined, and lessons learnt recorded. NS facilitated regular communication with the group via email, and ensured core documents were available for comment via Drop-Box for Business, email, telephone, or face-to-face discussion, enabling a good relationship with the researcher to be sustained.

During the co-design phase, two PPIE meetings, and two PPG meetings were held at each site. From the beginning, a modified EBD approach [51] created the right environment and ethos for the meetings to take place with our PPIE contributors as we adhered to the collective shared learning principles of co-design. The relationship that developed helped this to run smoothly. The PI, CS, chaired each meeting, with support from two members of the research team, NS and PN. Ahead of the meetings, we developed summaries of PPIE insights from phase 1 data collection with staff, patient and carer research participants, and presented these findings to the PPIE group using PowerPoint, to trigger co-design discussions. We tailored each co-design slide-set to reflect views from our PPIE partners and staff, patient and carer research participants from each of the sites. An example is shown in Additional file 2. We found that by including a slide on 'what we said, what you said' , helped to trigger discussions ahead of showing the research data collected from each of the sites.

Of note, co-design data was not collected for patients and carers using the services of Sites C1 and C2 as these were not collected in work stream 1. These perspectives were captured via Site A and B as these patients and carers used the services of Site A and Site B. The PPGs participation in the trigger discussions ensured that Site C1 and C2 could give their perspectives and interpretations. This influenced our co-design analysis.

Members of the research team, NS, PN and CS, and PPIE/ PPG contributors, individually and collectively shared thoughts about what might work in practice, challenged ideas and raised concerns together in ways that led to change (see Table 2 for an overview of these recommendations). Preferred roles in the co-design process and output were negotiated amongst the group members during each meeting; these roles evolved as the PPIE component progressed through the co-design to testing work stream. Some of the more active roles we outline in the reflective narrative to the paper.

The final step involved a written summary of each codesign discussion prepared by NS and emailed to the PPIE contributors, and wider research team for review.

All qualitative research data were transcribed, collated and analysed thematically drawing on a grounded theory approach [59] , and using NVivo11 qualitative analysis software. Thematic coding was conducted by the researchers (NS, PN, CS, BNO) producing distinct accounts for each research participant group (patients, carers and staff ) in each site. The co-design analysis on the research data and the PPIE insights was conducted in parallel to the research activity for work stream 1 and 2 (see Fig. 1 ), to enable issues and preliminary themes to be explored with participants and our PPIE contributors. We used trigger discussion slides showing the results of the research data collected from work stream 1 and the PPIE insights (from previous meetings) together with preliminary themes from the analysis; these insights were then incorporated in the specific co-design analysis following each meeting. The impact of our approach to the analysis was that the PPIE contributors experienced first-hand how their insights fed in to the co-design of the tools and the subsequent testing of these in multiple NHS sites. Detailed notes written following each co-design PPIE meeting were imported to each NVivo unit to support the main thematic analysis. Some more experienced PPIE contributors chose to email reflective thoughts on the minutes of each involvement meeting and these were also incorporated in the analysis. Preliminary themes from the analysis from work stream 1 were used as a basis for discussion at the co-design qualitative interviews and focus groups, and meetings with PPIE contributors, where links and distinctions across the multiple groups of participants and sites were explored (see Additional file 2). We now turn to the key themes that emerged from the co-design analysis.

As the FFT is a nationally mandated patient feedback tool [36, 37] , we sought specific feedback from our PPIE group on how to enhance the FFT question digitally by changing the response scale to capture emotional feedback collected via emoticons, and adapting the FFT free text question to invite more meaningful comments.

PPIE contributors liked that the proposed FFT question and how response could be enhanced to capture emotional feedback to encourage more people to use it. The issue of how best to capture people's feelings about the healthcare they received was tackled differently by PPIE contributors and research participants. In general, the PPG contributors in both primary care sites considered the use of emoticons appropriate, and feasible, as their practice was currently collecting feedback digitally in this way. Further, they also recommended that four instead of five options on the FFT response scale should be offered 'as people tended to go for the middle ranking where 4 makes the respondent go one way or the other' (researcher note). Likewise, the PPIE SMI contributors described making the FFT question and response via the interface simple, quick and friendly to use as 'some people like to use simple emoticons for feedback with an option for adding brief text' (researcher note). Some PPIE MSK contributors described preferring an alternative visual system, for example, traffic lights to express emotional feedback as it was felt that patients with MSK might experience discomfort to touch the iPad screen and that emoticons might not represent their pain. These views on appropriateness of using emoticons to collect emotional feedback resonated with the majority of staff participants working in mental health services (site B) where they talked about specific sensitivities of collecting feedback digitally in their context: 'I don't know whether necessarily in mental health they're always the best tool for measuring satisfaction, you know, for somebody who is suffering with an episode of depression, there's going to be nothing that makes them smile from ear to ear, so to see it as an emoticon to measure the most satisfied with a big smiley face, it's not something that's right at that time' (Site B CMHT, ID349, Staff FG).

As a result of these initial discussions, different options were explored by PPIE contributors, and in the subsequent co-design interviews with staff and patients. This led to a joint decision to test out emoticons with traffic light colours on the interface in four settings (Sites A, B OPT, C1 and C2: see Additional file 3], but not in the mental health setting, as these were viewed as inappropriate.

All PPIE contributors and research participants voiced how it would be most useful for a new digital tool to Our PPIE contributors also felt that capturing both positive and negative feedback from different service elements within each setting would be useful to examine:

'It was thought a good idea to capture the positive and negative feedback from different areas of the service e.g. experience at reception, experience with doctor etc. to explore the usefulness of feedback data.…' (PPG Site C2, researcher note).

As a result of these discussions the FFT free-text question was enhanced to capture positive and negative freetext comments via a digital interface to replicate the check-in process in place in three sites (B OPT, Sites C1 & C2: see Additional file 3).

All PPIE contributors and research participants could give views on the use of feedback via the following tools: iPad with self-standing kiosk; text message; dedicated phone line; face-to-face discussion; pen and paper; and the use of the site-specific website. This suite of FFT tools was based on the phase 1 data collected. The possibility of building in a feedback period was initially suggested by our PPIE SMI contributors, which would allow patients and carers to provide comments pre and post consultation in each NHS setting:

'Waiting time could be used for feedback. Pre and post feedback may be based on previous experience, or lack of experience, or expectations for the appointment. Subsequent visits can include the after-service experience' (researcher note).

This view was shared with our PPIE MSK contributors so that people could be invited to give feedback in their own time:

'There was group agreement that people should be invited to give feedback pre, during and post appointment and in their own time… Some contributors described people on medication with side effects that might make them confused… Pic-tures alongside each instruction might be of help' (researcher note).

In the PPG discussions, a few individuals asked if the digital Patient Access System that now is widely adopted in primary care could house our FFT feedback survey. Further alternatives were a phone app, providing a URL from the site website, or text message following a consultation. Some contributors liked the idea of enabling SMS text messages for collecting feedback: CMH services was co-designed to encourage the recording of verbal feedback from patients and carers that may have been excluded using current methods. In brief, each care coordinator would ask a patient a couple of trigger questions at the end of their home visit in order to allow comments on people's experience of the service. The care co-ordinator would then give a brief explanation to say that the team wants to improve information they collect about people's experiences of services. Responses to the trigger questions were to be recorded in a dedicated field within the electronic care record used by the CMHT (see Additional file 4). As a result of these insights and preferences, joint decisions were made on testing the new tools and they were placed in areas of privacy. They were monitored by the observation measures in place during the testing phase.

The majority of PPIE contributors and some patient and carer participants talked about using guidance, information and signposting to motivate patients and carers to give feedback using the new tools:

'The importance of presenting "feedback about the feedback" was met with group laughter but agreed to be a crucial element to the guidance produced to accompany the new tools' (PPI MSK, researcher note).

It was also voiced that any guidance could work to encourage not only more but also better quality feedback. This idea was adopted in the form of a colourful poster in four sites and later a larger poster to support use of the new tools in two sites (C1, C2: see Additional file 3):

'It was thought that more striking colours and something simple on the poster needed to be used-even something like "Please give us your feedback" with a big arrow pointing to the kiosk' (PPG Site C1).

The PPGs gave crucial insights on how to present and update the content of the poster ensuring it remained current amidst the plethora of information on display. Discussions about providing this information led to displaying examples of positive comments on a later version of the poster. Several PPIE contributors and research participants talked about the need to provide hands-on support, especially for older people or others who might be less confident in using digital devices: PPIE contributors and staff participants explored ways of providing support for patients and carers to use the kiosk. Staff reported that the majority of patients and carers would not use the kiosk unless they were asked to do so. Use increased once the larger colourful laminated poster with guidance notes was placed above the kiosk and a PPG member (PPIE contributor supporting the study) promoting the use of the kiosk was in post:

'The PPG was active in promoting the kiosk in the reception area. Five people walked up to the poster (this has never happened before with the smaller A4 landscape poster), read it for a few minutes, then used the kiosk without being prompted by either of us' (Site C2, observation note).

Two PPG members attached to Site C2 provided peer guidance on how to use the kiosk during the busiest clinics. Having this component in place allowed data capture on the acceptability of and continued engagement with digital feedback. We subsequently organised for a volunteer attached to Site B OPT to provide peer support during the testing phase. This role increased rates of ditial kiosk participation in Site C2 [60, 61] , and aided implementation in both settings as it responded to staff concerns about the impact of the new tools on their workload: Our recommendation that information and bespoke guidance should be provided ready for the set-up of the new tools was to be monitored and adapted to incorporate the feedback data throughout the testing period [34, 60, 61] . As part of the later phase of the co-design and implementation phase, visual feedback reports that included summaries of quantitative data and free text comments were split by sentiment (positive and negative comments: [34] ). In line with PPIE and staff preferences, these monthly reports were tailored to the different service contexts.

A summary of main points of feedback and recommendations made by the PPIE contributors to shape the research components by each theme through co-design is shown in Table 2 (Appendix). These crucial insights also helped us to tailor interview props for the co-design interviews (Additional file 2, Fig. 3 ).

A special PPIE dissemination and evaluation workshop in February 2018 was held, with presentations on the DEPEND study research findings. Nineteen members of the public attended the workshop, with representation from the study sites and multiple PPIE groups and networks across Greater Manchester. Members of our PPIE group co-presented at the workshop and co-facilitated discussions on our findings from DEPEND. Feedback on our reflections resonated with the public audience:

'I found that the way those presenting conveyed how much this project has meant to them, the emotions of it, was very moving and inspiring' (workshop attendee).

We also used this workshop to premier an animated video reporting the study findings [62] , and this was received positively:

We were able to use the comments and contributions to make final improvements to the animation prior to general release.

Following the PPIE workshop, two of our PPI contributors, AML and DA, co-presented with NS, our experiences of working together on DEPEND at an International PPIE conference [63] , and spoke about some of the challenges and successes. The presentation highlighted the importance of joint working between PPIE contributors, researchers and NHS staff in order to co-design new tools that have the best chance of working in practice.

Following the co-design and testing phases of DEPEND, we conducted a meeting with members of our PPIE group to hear collective and individual views on their experiences of the DEPEND study, and in particular the co-design model. Overall, the process and the outcome of PPIE in the DEPEND study was successful: strong and trusting relationships between PPIE partners and researchers had developed over the 2-year study. The PPIE partners made valuable contributions to, and provided important insights into each work package, ensuring that research priorities aligned with those of patients and carers and these enabled recommendations for delivering future PPIE work to be developed. All of our PPIE contributors commented on how much they had enjoyed being part of the team and that the experience had been rewarding. Participating in the DEPEND study had taken various forms for our PPIE contributors and examples are highlighted in a visual representation developed by one of our more experienced PPIE 'lived experience' coinvestigators, DA, a co-author on this paper (See Fig. 2) .

The model contains some personal involvement reflections and outcomes (green oval) and learning points (box) useful for future PPIE in co-design work from the mental health patient and carer perspective. The model emphasises the nuanced aspect to involvement in a study, what it may achieve for some more exprienced contributors and how. It is important to note that DA prefers the term 'lived experience' PPIE contributor; others in our PPIE group prefer the sole term 'contributor' to PPIE, and the PPGs prefer 'volunteer' or 'member' . These terms were captured in our Glossary. We have jointly written this paper using all preferred terms.

Peer relationships flourished with many PPIE contributors contacting one another in between meetings, and ongoing communication between the PPIE contributors and the research team enabled the partnership to be sustained during the funded 2-year period, and beyond. Other PPIE contributors to DEPEND, realtively new to PPIE, spoke of becoming more confident as they attended more groups at the University as the research progressed.

Specific roles were allocated to those PPIE contributors based on preferred activities. DA helped us to recruit carer participants within Site B using DA's Patient Expert Group networks. This enabled us to widen our recruitment with DA spending time on site with the researcher at carer events held by Site B and attending meetings with the carer lead at this site. During year 2 of the study, DA spent dedicated time working at the University with NS and contributed to the shaping of toolkit documents (see Additional files 3 and 4). Our PPIE co-applicant, AML, also spent a significant amount of time reviewing components for the toolkit. Other PPIE/ PPG contributors gave crucial practical support throughout the evaluation and dissemination phase to DEPEND. One PPG (Site C2) took a lead peer support role that helped to promote and sustain kiosk use during the evaluation of the tools phase. Of note, they witnessed how patients and carers mostly reacted positively to the new tools (these reactions are reported sepately [60, 61] ). They passed their observations on to the researchers via email and in-person; these insights were crucial to making sense of the unfolding analysis. Two PPIE contributors had a lead role in co-designing the animated film and co-delivered a public dissemination workshop.

The DEPEND study involved staff, patients and carers research participants from multiple sites co-producing the research with a core PPIE component embedded within each of the work streams [34] . We were able to adopt a shared approach to co-design through working in partnership with patients and carers, and research participants, to shape a suite of new tools to test the usefulness and relevance of patient experience data in four NHS settings. This paper reports on the most significant and innovative PPIE involvement, namely the co-design work: the enhancement of digital and non-digital methods to collect, analyse and present patient experience data by modifying the FFT question [36, 37] . The DEPEND study adopted a modified EBD approach and the deliberative discussion of the findings of the earlier research phases and the systematic application of shared interpretations allowed the tools to be critically reviewed, developed and refined by the PPIE contributors and research participants. Consequently, the co-design of a toolkit reflected key insights provided by our PPIE contributors alongside the shared analysis of the research data. The sensitivity to the different health conditions and service contexts ensured the adaptability and relevance of the tools across the four sites. Most importantly, the investment in the relationship between PPIE contributors and the research team helped to maintain mutual trust throughout the study. Our study makes a specific contribution to the evolving landscape of PPIE, co-design, and co-production. The novelty of our work lays with successfully embedding a multi-level PPIE strategy throughout the work streams for the duration of the study and beyond that enabled a partnership to evolve with significant impact on the research process, outcomes and the partnership members.

DEPEND was a complex multi-site study requiring significant PPIE action planning and input throughout each phase. Our PPIE in co-design experience in DEPEND highlights a number of important issues, which we discuss in turn with reference to the current guidance to practice [22] , and related research. Investment in the PPIE partnership with patient and carer contributors is key, from the pre-funding stage of research through to the dissemination, and beyond the study [5, 6, 22, 64, 65] . Our findings show the need to tailor PPIE to preferences and values of research partners, and consideration has to be given to how variation in individual needs affects ongoing participation [11, 12, 14, 66] . Having honest and open conversations and creating a glossary of terms at the start of our co-production journey helped us to tailor varied strands of PPIE work, matching specific roles to individuals and working flexibly to optimise participation in each work stream. Some contributors reviewed and contributed to core documents remotely via skype and dropbox, allowing inclusivity and personalised adaptation. The PPGs in Site C2 favoured a specific peer support role, matched to the way they want to engage in their general practice [67] . Recent coproduced research recommends implementing costed external training, especially with regard to establishing a relationship from the start [68] .Our PPIE group was diverse with patients and carers with differing levels of PPIE experience. We found this dynamic worked well in terms of assigning roles and responsibilities [22] , and tailoring varied strands of work to specific roles based on preferences enhanced the value of the PPIE in co-design as everyone was treated with equal importance [13, 14, 68, 69] . Establishing a joint understanding and clarity of roles was central to managing expectations, especially when sometimes we could not feasibly meet preferences for involvement. It was was hoped that 2 more active PPIE contributors would be involved in co-facilitating the interviews and focus groups but unfortunately this was not possible due to bureaucratic and budget constraits. All our contributors were well supported with comprehensive plans and actions and we demonstrated reciprocity in our approach of joint working by enabling focused discussions in groups [29, 30, 65] . On reflection, being transparent with our thinking helped us to build on the initial investment in the research partnership by sharing perspectives during the co-produced research [20, 21, 23, 24, 27, 28] . Having a dedicated PPIE coordinator with support from at least two members of the research team enabled facilitation of separate co-design meetings so that insights could feed into the unfolding data collection and analysis [25, 29] . We faced some major challenges and upsetting events during the 2-year study. The sudden passing of two of our PPIE contributors, Jane and Neal, made us reflect on the relationships developed during the course of the research and on how to manage difficult situations. Working closely together as researchers and PPIE contributors entails sharing personal experiences and building long-term relationships that are quite different from those that develop when researchers are carrying out short research projects. Researchers, and PPIE contributors, develop a sense of responsibility towards each other and managing expectations can become harder without the necessary support, tools or guidance in place. When a member of a group passes away suddenly, we found it might be difficult to know what to do to support other members of the group, who may already feel vulnerable. We supported each other individually and as a group, enabling people to express bereavement in their own way. Some of these learning outcomes are shown in DAs reflection model (Fig. 1) The research team also took time to individually and collectively reflect on what happened, and what could we do to make sure we all felt comfortable to continue with the research. Special peer relationships and friendships developed between PPIE contributors. For example, DA and NS, participated in the annual Community Festival at the University to champion their working partnership to the local community [70] .

The research team were offered individual counselling at the University, but we all felt a group intervention, where collectively we could continue to talk openly and honestly about what happened with our PPIE contributors would be more appropriate. The PI, CS, reached out to the clinician co-applicant on the research team, who facilitated a debriefing meeting with the researchers and the PPIE group at the University. The wider social responsibility community at the University also supported us all as a group by offering a platform for a co-delivered presentation on our PPIE experiences and to celebrate our achievements to date. Both these interventions helped us to move forward together illuminating the importance of sharing positive and challenging experiences and reciprocal learning. A posthumous award was conferred on our PPIE contributor in recognition of the exceptional PPIE work completed in the development of digital innovations for people with mental health conditions [71] .

The absence of guidance and support to deal with such emotionally upsetting events when working as a partnership led to further research funded by the NIHR to explore ways to address this issue given that PPIE is integral to research both in our local communities and beyond [72, 73] .

One of the benefits of co-design that emerged in this study was beginning the involvement and design process from the 'lived experience': from the reality of everyday work, rather than designing from theory, to ensure that we put patients and carers' perspectives at the heart of the research study. Having an extended PPIE group incorporating new and experienced contributors was conducive to co-design. The two PPGs exemplified the experiences and preferences of collecting feedback with long-term service use for our exemplar physical and mental health long-term conditions in primary care. The findings were crucial to capturing what tools might work in practice as they were on site. Published studies have similarly reported on the richness that emerges in the research and partnership from adopting PPIE in codesign [32, 33] . Our work resonates with the reciprocal learning reported emphasising the importance of building the foundations for relationships, fostering trust, respect, creating a safe space, to talk about the research and the effects on the partnership [74] .

Overall, both the process and the outcome of PPIE in the DEPEND study was seen to be successful by PPIE contributors and the research team, as indicated by the strong relationships that had been forged over the 2-year study. Our PPIE collaborators made valuable contributions to, and provided insights into, each work package, enabling us to develop recommendations for delivering future PPIE work. Issues to be considered in future PPIE activities described by the group included: retention of PPIE contributors; diversity; inclusion; choice of language; and measuring fear, or the stigma of becoming involved in PPIE activities [18, 19, [75] [76] [77] .

We recommend linking to external organisations who embed PPIE work at the heart of research, such as the NIHR School for Primary Care Research [3] , and the National Co-ordinating Centre for Public Engagement [78] , where guidance is kept current, to encourage a culture of fair, equitable and meaningful involvement. We further suggest researchers and PPIE contributors make use of the many established online PPIE toolkits, which have been co-produced with patients and carers [21, 24, 58] . Finally, the DEPEND study was conduced before the COVID-19 pandemic, however, the lessons learnt remain as challenges for health researchers worldwide [79] . It is imperative that we continue to share learning with the research community to continue to tackle challenges and to celebrate achievements when working with patients and carers in partnership [80] .

We focused on disseminating our PPIE learning throughout the study period through various avenues connected to the partnership, including: the national NIHR Patient Experience Learning Set [35] ; a focused PPIE International conference [63] , as well as specific events held promoting the use of feedback held at all sites, and at the University. It would have been impactful to hear more from the audience connected to Site B than was possible during the study about our novel tool of asking a trigger question to capture verbal feedback. We struggled to recruit carer research participants, despite having the assistance of a PPIE experienced contributor, DA, with access to many carer third sector organisations. Future research should move beyond recruiting via the organisations networks, which host the research, and invest in knowledge brokering activity with all stakeholders [81, 82] .

We have shown how PPIE in co-design shaped the research process and its materials by outlining specific ways in which it added value to the design and delivery of the toolkit. Our experiences of developing multiple components of PPIE work for this complex study across various health services, demonstrates the importance of tailoring PPIE to suit different settings, and individual strengths and capacity. The contribution of PPGs to PPIE in health services research work is under reported and remains to be explored further. Our study of PPIE in co-design and co-production shows the value of bringing diverse experiences together, and adopting continuous feedback loops that transparently show the way in which PPIE inputs shape the research. Adopting a shared approach to managing challenging situations is suggested as we encountered the need to develop psychological and practical solutions to partnership working with patients and carers. Our approach to PPIE in co-design was wide-ranging and iterative, and this paper includes important personal reflections on the emotional consequences of investing in PPIE for both PPIE contributors and the research team that need to be acknowledged for the process and outcome of meaningful PPIE to evolve and be impactful. 

Co-design of a suite of tools with guidance PPIE contributors thought a colourful poster would be the most effective way to advertise the use of the new tools in all sites PPG C2 gave examples of positive and negative experience data to include on the first iteration of the poster on the premise data would be replaced with real-time data captured by the new tools Contributors advised that the poster and other guidance to advertise the new tools would need to be near the kiosk; PPG C1 contributor suggested we might use an arrow pointing to the new tools; PPG C2 contributors felt that the new tools would need promoting by a volunteer on site Researchers should consider using to posters, large and small, co-designed with the PPIE group, to the requirements of each site (a, b outpatients, c1-c2); the toolkit guidance should be tailored to each site as follows:

show both positive and negative feedback kiosk data; researchers should refresh the data to avoid posters looking static, to correspond with the new feedback reports for staff (sites a, b outpatients, c1-c2); increase the size of the poster from A4 to A0 where possible and use an arrow to advertise the tools volunteers on site, where possible, to promote the use of the tools (sites c2, b OPT) The PPIE group to DEPEND whom kindly shared their experiences and gave their time, and invaluable insights to the study: Annmarie Lewis, Neal Sinclair, Dawn Allen, Kate Lurie, Helen Yeoman, Jane Reid-Peters, Annette Barber, Susan Moore, Nigel Bagshaw, and all of the PPG contributors (Sites C1 and C2). A volunteer attached to Site B Outpatients. The authors would like to thank all the patient, carer and staff research participants and members of our Study Steering Committee (Professor James Barlow (Chair), Professor Martin Knapp, Konstantina Poursanidou, Professor Nigel Collier, Simon Stones, Professor Carolyn Chew-Graham, Professor Tim Doran, Annmarie Lewis and Neal Sinclair). Sue Pargeter from the NIHR National Evaluation, Trials and Studies Coordinating Centre. Patrick Cahoon for his advisory and implementation role within one Trust. Cathy Lovatt for her advisory and implementation role within one Trust. We also acknowledge the assistance of Dr Brian McMillan on reviewing the final manuscript ahead of submission. We would also like to thank the reviewers' of this paper for their insightful contributions. We 

This paper is in memory of Neal Sinclair and Jane Reid Peters who sadly died during the study period. Neal was a co-applicant to DEPEND and leading member of our PPIE group. Jane was a member of our PPI group. We are grateful to both Neal and Jane who brought their energy, enthusiasm, expertise and experience to the project. Since we completed the study, we also learnt of the passing of a PPG contributor. Together, they leave a lasting and heartfelt influence on our work.

NS led the PPIE component to the study, co-led the qualitative research, data collection and analysis, and led the writing of the paper. BNO contributed to the qualitative data analysis, and to the writing of the paper. AML was a coapplicant and member of our PPIE group, and played a lead role in PPIE activities, contributed to discussions regarding research findings and interpretation, presented on behalf of the study team, worked with us on the co-design of information materials and contributed to the writing of the paper. DA was a member of our PPIE group and played a lead role in PPIE activities, contributed to discussions regarding research findings and interpretation, presented on behalf of the study team, worked with us on the co-design of information materials and supported the carer recruitment in Site B and contributed to the writing of the paper. NB is the PPG coordinator at Site C2, he played a lead role in coordinating the PPG activities by taking detailed minutes of discussions, contributed to discussions regarding research findings and interpretation, worked with us on the co-design of information materials to advertise the tools and commented via email on the writing of the paper. PN contributed to the PPIE component, contributed to the qualitative research, data collection and analysis, and to writing sections of the paper. CS led the overall design of the study and data collection, contributed to the analysis, and writing of the paper. All authors read and approved the final manuscript.

This project is funded by the National Institute for Health Research (NIHR) HS&DR programme, project 14/156/16. The views and opinions expressed are those of the authors and do not necessarily reflect those of the NIHR, the NHS or the Department of Health.

The data generated during or analysed during the current study are not publicly available due to ethical restrictions. All data queries and requests should be submitted to the corresponding author, Dr Nicola Small, SPCR Fellow, for consideration.

Supporting public involvement in NHS

Going the extra mile: improving the nation's health and wellbeing through public involvement in research

NIHR launches new Centre for Engagement and Dissemination

What is public involvement in research?

How the weather affects the pain of citizen scientists using a smartphone app. npj Digit Med

Patient and public involvement in doctoral research: impact, resources and recommendations. Health Expect

Being inclusive in public involvement (PI) in health research

Public involvement in research: values and principles framework

UK Standards for Public Involvement-one 'very different' year on: NIHR

Being Inclusive in Public Involvement (PI) in Health Research Manchester: Manchester University NHS Foundation Trust

Patient and public involvement in doctoral research: reflections and experiences of the PPI contributors and researcher

What makes it work? Exploring preferences of patient research partners and researchers involved in a long-term co-creative research collaboration

The role of patient and public involvement leads in facilitating feedback: "invisible work

Value and learning from carer involvement in a cluster randomised controlled trial and process evaluation-Organising Support for Carers of Stroke Survivors (OSCARSS)

Editorial briefing: involving people and patients in health care, health policy and research-across the board

GRIPP2 reporting checklists: tools to improve reporting of patient and public involvement in research

Reviewing progress in public involvement in NIHR research: developing and implementing a new vision for the future

Who defines the impact of research? A patient-centred opinion and call for action

Patient centred trials: patient and public involvement (PPI): The University of Manchester: Faculty of Biology

Guidance on co-producing a research project

A map of resources for co-producing research in health and social care: a guide for researchers, members of the public and health and social care practitioners

NIHR resource for public involvement-guidance on co-producing a research project

Bringing together co-production and community participatory research approaches: using first person reflective narrative to explore co-production and community involvement in mental health research. Health Expect

Improving health care services through experiencebased co-design

Testing accelerated experience-based co-design: a qualitative study of using a national archive of patient experience narrative interviews to promote rapid patient-centred service improvement

Reflecting on our co-produced study with young people with past mental health difficulties. Imperial College London17

Reflections, impact and recommendations of a co-produced qualitative study with young people who have experience of mental health difficulties. Health Expect

Empowering people to help speak up about safety in primary care: using codesign to involve patients and professionals in developing new interventions for patients with multimorbidity

Patients and staff as codesigners of healthcare services

Digital health at fifteen: more human (more needed)

Participatory codesign of patient involvement in a learning health system: how can data-driven care be patient-driven care? Health Expect

Van der Veer SN. Participatory co-design and normalisation process theory with staff and patients to implement digital ways of working into routine care: the example of electronic patient-reported outcomes in UK renal services

Digital methods to enhance the usefulness of patient experience data in services for long-term conditions: the DEPEND mixed-methods study. Health Services Delivery Res

Improving care by using patient feedback.: NIHR Dissemination Centre

Friends and Family Test

Using the Friends and Family Test to improve patient experience

NHS Surveys focused on patients

Patient experience of diabetes services

How do healthcare staff respond to patient experience feedback online? A typology of responses published on care opinion

Implementation and use of the Friends and Family Test as a tool for local service improvement in NHS general practice in England: Policy Innovation Research Unit

Rethinking policy approaches to measuring and improving patient experience

Editorials: friends and family test should no longer be mandatory

Improving patient experience in primary care: a multimethod programme of research on the measurement and improvement of patient experience. Program Grants Appl Res

The patient feedback reponse framework-understanding why UK hospital staff find it difficult to make improvements based on patient feedback; a qualitative study

Improving patient experience in hospital settings: assessing the role of toolkits and action research through a process evaluation of a complex intervention

Humans versus machines: text mining versus adapted grounded theory in the analysis of free text data from patient feedback surveys

Supervised machine learning algorithms can classify open-text feedback of doctor performance with human-level accuracy

Combining rules and machine learning for extraction of temporal expressions and events from clinical narratives

About the NHS Outcomes Framework (NHS OF) 2020

Bringing user experience to healthcare improvement: the concepts, methods and practices of experience-based design

Integrating mobile-phone based assessment for psychosis into people's everyday lives and clinical care: a qualitative study

Smartphone-enhanced symptom management in psychosis: open, randomized controlled trial

National Association for Patient Participation

Six ways general practice PPGs could use care opinion: update from care opinion: care opinion: what's your story

Budgeting for involvement: practical advice on budgeting for actively involving the public in research studies

PRIMER: The patient and public involvement and engagement group at The Centre for Primary Care and Health Services Research

PPIE toolkit: The University of Manchester

The discovery of grounded theory

Implementing a digital patient feedback system: an analysis using normalisation process theory

Exploring engagement with digital screens for collecting patient feedback in clinical waiting rooms: the role of touch and place. Health

DEPEND: use of digital methods for collection and use of patient experience data. NIHR Greater Manchester Patient Safety Translational Research Centre 2019. www. youtu be. com/ watch?v= BOYLx VJAzdI

Co-designing new tools for collecting, analysing and presenting patient feedback in NHS services: working in partnership with patients and carers

Maximizing engagement in mobile health studies

Developing a patient safety guide for primary care: a co-design approach involving patients, carers and clinicians

Drawing straight lines along blurred boundaries: qualitative research, patient and public involvement in medical research, co-production and co-design

Does patient and public involvement have an impact on general practice? A participatory action research study. SAPC ASM 2015-evidence and innovation in primary care

Bridging gaps: how we've managed digital exclusion during COVID-19 to improve access to healthcare for women who have experienced trauma-the Bridging Gaps group

The manifestation of participation within a co-design process involving patients, significant others and health-care professionals. Health Expect

Biology Medicine and Health, Patient and Public Involvement and Engagement. The 2017 PPIE Celebration Event Winners!

Managing difficult situations in patient and public involvement: workshop event 7th

Managing difficult situations in public involvement: a coproduction event. NIHR School for Pimary Care Research

Exploring patient and public involvement (PPI) and co-production approaches in mental health research: learning from the PARTNERS2 research programme

The dark side of coproduction: do the costs outweigh the benefits for health research?

Exploring the theory, barriers and enablers for patient and public involvement across health, social care and patient safety: a systematic review of reviews

Research Involvement and Engagement (2021) 7:85 • fast, convenient online submission • thorough peer review by experienced researchers in your field • rapid publication on acceptance • support for research data, including large and complex data types • gold Open Access which fosters wider collaboration and increased citations maximum visibility for your research: over 100M website views per year submit your research ? Choose BMC

We help universities engage with the public: how can we help you with public engagement? Bristol: NCCPE

Reflections from the COVID-19 pandemic on inequalities and patient and public involvement and engagemement (PPIE) in social care, health and public health research. Health Expect

Impact Accelerator Unit. Patient and public involvement and engagement in implementation

The role of patients and carers in diffusing a health-care innocation: a case study of "My Medication Passport

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations

See Table 2 . 

The online version contains supplementary material available at https:// doi. org/ 10. 1186/ s40900-021-00329-3.Additional file 1. GRIPP2 long form.Additional file 2. Example trigger discussion slides used in the co-design process.Additional file 3. The DEPEND study toolkit.

A new approach to collecting verbal feedback.