key: cord-1033948-2t0pwun2 authors: Gesell, Sabina B.; Prvu Bettger, Janet; Lawrence, Raymona H.; Li, Jing; Hoffman, Jeanne; Lutz, Barbara J.; Grudzen, Corita; Johnson, Anna M.; Krishnan, Jerry A.; Hsu, Lewis L.; Zwart, Dorien; Williams, Mark V.; Schnipper, Jeffrey L. title: Implementation of Complex Interventions: Lessons Learned From the Patient-Centered Outcomes Research Institute Transitional Care Portfolio date: 2021-07-08 journal: Med Care DOI: 10.1097/mlr.0000000000001591 sha: ca3dd6183d8c9cf4d3a22d57e773e67d5688af48 doc_id: 1033948 cord_uid: 2t0pwun2 Despite the well-documented risks to patient safety associated with transitions from one care setting to another, health care organizations struggle to identify which interventions to implement. Multiple strategies are often needed, and studying the effectiveness of these complex interventions is challenging. OBJECTIVE: The objective of this study was to present lessons learned in implementing and evaluating complex transitional care interventions in routine clinical care. RESEARCH DESIGN: Nine transitional care study teams share important common lessons in designing complex interventions with stakeholder engagement, implementation, and evaluation under pragmatic conditions (ie, using only existing resources), and disseminating findings in outlets that reach policy makers and the people who could ultimately benefit from the research. RESULTS: Lessons learned serve as a guide for future studies in 3 areas: (1) Delineating the function (intended purpose) versus form (prespecified modes of delivery of the intervention); (2) Evaluating both the processes supporting implementation and the impact of adaptations; and (3) Engaging stakeholders in the design and delivery of the intervention and dissemination of study results. CONCLUSION: These lessons can help guide future pragmatic studies of care transitions. transitional care, 9, 10 translation of effective strategies into routine care has not been realized for patients at high risk for fragmented care and poor outcomes. [11] [12] [13] Care processes that aim to bridge acute and postacute care settings vary widely and are fraught with challenges, including different information technology systems, siloed networks, and misaligned reimbursement schemes. Health care professionals responsible for coordinating care transitions recognize the value of closing gaps in care but often are unaware of best practices or how to translate evidence-based strategies to local situations. Frequent changes in institutional and health policies, patients' needs, and local resources create a challenging environment for addressing care transitions and studying interventions in ways that are both valid and generalizable. This paper presents lessons learned from the portfolio of care transition studies funded by the Patient-Centered Outcomes Research Institute (PCORI). Three major themes are examined through the review of case studies: (1) Delineating function (intended purpose) versus form (prespecified modes of delivery of the intervention); (2) Evaluating both the processes supporting implementation and the impact of adaptations; and (3) Engaging stakeholders in the design and delivery of the intervention and dissemination of study results. We highlight the complexities of conducting such studies, offer lessons learned from our experiences, and provide exemplars to inform future research. Investigators of 9 comparative effectiveness studies of care transitions funded by PCORI describe their experiences implementing and evaluating transitional care interventions. Two studies were observational. Seven studies were randomized controlled trials of transitional care interventions, mostly of pragmatic design. Unlike a traditional explanatory trial, a pragmatic trial is designed to test an intervention in everyday clinical settings with heterogenous patient populations to maximize applicability and generalizability. Thus, a pragmatic trial asks whether an intervention works in real life versus a tightly controlled trial environment (ie, effectiveness versus efficacy). 14 The 2 observational studies examined real-world practice. The 9 transitional care studies varied in their start dates, durations, target populations, clinical settings, and budgets (range: $1M−$16M), but all were studies of complex clinical interventions. The interventions were not funded by PCORI. Table 1 provides an overview of the case studies reviewed. Three major themes emerged when study investigators compared and contrasted their experiences. (1) Delineating the function versus form of the intervention and allowing for adaptation to the form while preserving fidelity to function. The complexity of care transitions interventions can make it difficult to compare findings or translate models from one context to another. PCORI's Methodology Committee uses 2 descriptive terms that help with this dilemma. 21, 22 Function refers to the core purposes of an intervention applied to meet an intended goal. Form refers to the strategies and activities needed to carry out the functions of the intervention, including the menu of intended modes of delivery, providers involved, materials or tools required, and intervention dose. 22 Too much, reactive, or the wrong kind of adaptation can weaken an intervention's effectiveness and make it difficult to understand what is being evaluated. Thus, it is critical to differentiate functions from forms, allow adaptations to the form while keeping core functions intact. (2) Evaluating both the processes supporting implementation and the impact of adaptations. The case studies describe planned and unplanned adaptations to transitional care interventions. They highlight the utility of analyzing process data to better understand adaptations as well as the difficulty in assessing the actual (intended and unintended) effects of adaptations. Findings highlighted the need to recognize and systematically measure adaptations in implementation; guided by the research design, existing data, and established frameworks; and systematically evaluated with quantitative and qualitative data collected during implementation. (3) PCORI's approach to engaging stakeholders in the design and delivery of the intervention and dissemination of study results appears to work. The philosophy of stakeholder engagement is now established in the scientific literature; however, investigators and funders often agree that the value of engagement it is not always realized due to the effort and resources required to implement and sustain it properly. Engagement can also cause problems in the community if it is poorly conceived, poorly conducted, or implemented without long-term commitment. The expectations PCORI placed on study teams to detail engagement plans as part of their contracts and to engage meaningfully with stakeholders, coupled with the flexibility afforded to teams to match engagement to their particular study needs, likely mitigated issues related to the possible negative implications of engagement. These themes are further discussed below. Hawe and colleagues called for designers of pragmatic complex interventions to standardize functions, while allowing the form of the intervention to be tailored to the participant or local conditions. 21, 23, 24 Similarly, the PRagmatic Explanatory Continuum Indicator Summary-2 25 framework recommends designing trials with the desired level of pragmatism in mind, and allowing flexible options for delivery, while maintaining appropriate fidelity to the intervention. An intervention's effectiveness is driven by its core functions. Thus, functions should not be adapted, and fidelity to core functions should be assessed. Functions can take on multiple forms, and these forms may be adapted as long as they fulfill the core function. The substitution of a particular form does not compromise the integrity of the intervention if a different form could be used to achieve the core function. 24 The concepts of forms and functions were developed by Hawe and colleagues and were recently elaborated on by Perez Jolles and colleagues, and Kirk and colleagues who recommend the use of these terms over previously used terms. Often used interchangeably are the terms: core components, core elements, core principles, active ingredients, key features, the theory of change process or logic model. 21, 24, [26] [27] [28] [29] Despite the shared understanding of their importance, intervention developers infrequently specify core functions and forms. Most protocols only specify forms (who does what, when, where). For example, a protocol might state "Nurse A asks patients who meet X, Y, Z inclusion criteria a specific screener question to determine if they are eligible for the transitional care program. If eligible based on the screener, Nurse A places note for physician in chart to initiate referral." The methodological concern is that most intervention protocols do not specify core functions. Thus, measuring fidelity to the functions is generally not possible using only the protocol. Core functions need to be specified first, describing why and how-why does what they are doing work and how does it work (mechanisms of action). Historically, the focus has been on fidelity to the intervention protocol, which means we have only measured fidelity to forms. In the example above, a study might be criticized for not having fidelity if Nurse A changed the wording of the question or if someone other than Nurse A asked the question. Instead, such adaptations need to be measured and evaluated in terms of their potential impact on core functions. To illustrate further, a nurse might spend 5 minutes teaching a patient about a medication at discharge, or the patient might watch a 5-minute video at home. However, these may lead to different levels of effectiveness despite serving the same purpose of patient education (function). Also, patient preferences for video or nurse-directed education may produce different levels of knowledge. Therefore, it is important to evaluate whether adaptations to form are associated with different implementation outcomes (eg, acceptability) and produce different patient outcomes. Historically, adaptations were considered a potential threat to implementation and the validity of studies that evaluated them. Yet adaptations may be necessary to address the needs of a particular patient population due to heterogeneity of disease severity or complications, other patient factors, or the local social context or environment-or because funding is insufficient to implement an intervention as originally designed. The goal should be to balance fidelity (to function) and context-aligned adaptation (to form) to maximize effectiveness, translation into routine clinical care, and sustainability. New work by Hawe, Perez Jolles and colleagues, and Kirk and colleagues suggests we should rethink how we consider fidelity entirely. Instead of focusing on protocol fidelity (which, given the way protocols are currently written, means we are measuring form fidelity only), we should instead focus on function fidelity. The advantages of disentangling functions from forms can be applied to interventions that include "just" clinical components as well as to interventions that are packages of clinical components and implementation strategies. Ideally, developers would specify how each intervention function is expected to generate an effect on the prespecified outcome in each patient population (or in specified patient subgroups). Core functions and forms can also be identified post hoc. 28 The idea of making causal pathways explicit is also reflected in PCORI's methodology standards for studies of complex interventions. 22 In the case studies below, we discuss the importance of delineating function versus form. Case Study: "Improving Transition from Acute to Post-Acute Care Following Traumatic Brain Injury-The BRITE Study" 30 is an ongoing multisite trial whose participants have highly variable ability to participate depending on the severity of their traumatic brain injury and their postdischarge community resources. The study compares the effectiveness of 2 approaches of transition from the hospital to the community. Flexibility in how the intervention is implemented depends on the individual's needs and the community resources available to meet those needs. 21, 24 The research team intended to focus their assessment of fidelity on monitoring function by specifically assessing which forms were used in the intervention (assessment of all needs, provision of specific resources, etc.), through listening to recorded contacts between interventionists and participants and utilizing a fidelity checklist. Ultimately, this approach to fidelity was not pragmatic on the PRagmatic Explanatory Continuum Indicator Summary-2 scale, 31 and it was more closely aligned with the idea of fidelity to form (ie, documentation of particular kinds of patient contact) rather than function. As the study evolved, the research team clarified its definition of function and emphasized fidelity to function while allowing flexibility in form. The process was further documented in a manual to embed flexibility in the intervention, and fidelity to function (eg, were needs assessed, were resources given, were contacts with participants made) is now tracked in a clinical database. Lesson Learned: Complex transitional care interventions should be flexible in form to maximize their effect on different patient subgroups and contexts. Case Study: The "Emergency Medicine Palliative Care Access" study is an ongoing pragmatic, multicenter randomized controlled trial with emergency department patients comparing 2 established models of palliative care: nurse-led telephonic case management versus specialty outpatient palliative care. The interventions were both complex, 32 with multiple core functions completed by a nurse or another provider. The form of delivery (telephonic versus face-to-face) was not originally designed to be adaptable, but the coronavirus disease of 2019 pandemic closed outpatient palliative care clinics temporarily, and all visits transitioned to a telehealth platform. However, the core functions of palliative care, such as assessing symptom burden and performing a goal of care conversation, remained. The research team will compare effectiveness of the mode of delivery as an additional exploratory analysis. Lesson Learned: Flexibility regarding an intervention's form allows necessary adaptation in response to external, real-world changes. Complex interventions may require adaptation under real-world conditions and will likely be adapted by future adopters. Case Study: In the PATient Navigator to rEduce Readmissions (PArTNER, Evaluating a Program to Improve Patient Experiences after Discharge from the Hospital) pragmatic trial, patients hospitalized at a minority-serving institution with a diagnosis of heart failure, pneumonia, chronic obstructive pulmonary disease, myocardial infarction, or sickle cell disease were randomly allocated to a Navigator intervention delivered by community health workers and peer coaches or to usual care. The Navigator intervention included (1) Assessment of barriers to safely transition from hospital to home during community health worker-led in-person visits in the hospital before discharge and at a home visit; and (2) Follow-up supports provided via phone by peer coaches from patient advocacy organizations over a 60-day period. In the Navigator group, completion rates of interventions varied across components (eg, > 90% had assessments of barriers to care,~80% had postdischarge home visits, but only 60% had 1 or more peer coaching calls). 33 Lesson Learned: Fidelity to form can vary greatly across various components of a complex intervention. The need for adaptations to form may therefore vary among intervention components and should be considered in the design phase of the study, evaluated, and specified for future adopters. Process evaluations inform researchers about expected effects and associated mechanisms of an intervention and can help explain differences between expected and observed outcomes. 34, 35 PCORI methodology standards for studies of complex interventions also emphasize the value of planning and conducting a full process evaluation. This information can also help elucidate how an intervention's effectiveness may be altered by adaptations to different settings, populations, changing contextual factors, or the external environment. Some events may be anticipated and adjustments made during trial implementation can be planned; others cannot be anticipated. The case studies below describe unplanned adaptations in 2 pragmatic trials. They highlight the utility of analyzing process data but did not evaluate the impact of these adaptations on implementation or outcomes. Case Study: "Relative Patient Benefits of a Hospital-Patient Centered Medical Home Collaboration within an Affordable Care Organization to Improve Care Transitions," was a cluster-randomized stepped wedge trial of a multicomponent intervention to improve care transitions among general medical and surgical adult patients. One component of the intervention, an inpatient discharge advocate, was delivered in different ways in the 2 participating hospitals, an unplanned adaptation in response to local contextual differences. In 1 hospital, a dedicated nurse practitioner served in this position, while in the other, this position was filled by expanding the roles of multiple existing "attending nurses." While the former had a high degree of quality control (fidelity to function) but limited ability to see all patients, the latter had more variable fidelity to function but greater ability to see more patients and a clearer path to sustainability. Lesson Learned: Mixed methods program evaluation can provide valuable lessons learned when circumstances lead to unplanned adaptations to the form of an intervention. Case Study: The "Comprehensive Post-Acute Stroke Services" study tested a comprehensive model of postacute stroke care in a cluster-randomized design including 40 hospitals and over 6000 participants. 15, 36 The protocol specified that a 2-person clinical team (a nurse and either a physician or a stroke-trained advanced practice provider) conduct a follow-up visit after hospital discharge and comprehensively assesses patient and caregiver needs. These data then informed an individualized care plan including secondary prevention, rehabilitation, recovery, and referral to community services. However, only 35% of eligible patients at intervention hospitals actually received the follow-up visit and individualized care plan within 30 days of discharge. As is common in pragmatic trials, fidelity to function was low. In the intention-to-treat analysis, no benefit on the primary outcome (patient-reported physical function) was detected. Self-reported blood pressure monitoring at 90 days was more prevalent in the intervention arm compared with usual care (72% vs. 64%) and was the only significant secondary outcome in the intention-to-treat analysis. The intervention produced a clinically meaningful improvement in home blood pressure monitoring which is critical for a population at high risk of future stoke. Clinical teams were trained to emphasize to patients at discharge the importance of home blood pressure monitoring. During implementation, hospital staff deemed it essential to provide their patients with blood pressure cuffs if telling them to monitor their blood pressure at home. This sentiment was common, and 12 (63%) intervention hospitals used their own resources to provide blood pressure cuffs to patients at discharge. This unplanned adaptation was not assessed. Lesson Learned: Evaluating processes that support implementation can elucidate the hypothesized causal pathway and can inform future attempts to adapt the model to other settings and populations. Evaluating adaptations generates evidence to support what should and should not be incorporated into the manualized intervention. Kirk and colleagues 27 developed a model and decision aid for understanding intended and unintended impacts of adaptations. Rather than just describing adaptations, they recommend a systematic assessment of relationships between constructs. Their model for adaptation design and impact recommends researchers consider potential causal pathways of adaptations (eg, mediators and moderators) and adaptations' intended and unintended impacts on outcomes. Model for adaptation design and impact recommends practitioners to design adaptations in a way that anticipates intended and unintended impacts and leverages best practice from research. Deciding which adaptations are permissible for an intervention should be guided by an established framework. Escoffrey et al 37 Currently, there are very few rigorously conducted studies evaluating implementation factors that are associated with higher fidelity to function. For future research, especially pragmatic trials, we recommend an approach to implementation in which adaptation is encouraged, recognized, respected; guided explicitly by the research design, existing data, and established frameworks; and systematically evaluated with quantitative and qualitative data collected during implementation. We recommend specifying which adaptations to the intervention's form are allowable and monitoring for potential intended and unintended impacts of these adaptations. We also recommend developers provide manuals describing the core functions and the menu of forms available for each function and, ideally, guidance on which forms to select for which situations; with decisions supported by data. While the benefits of stakeholder engagement are often discussed, 40, 41 few studies describe the challenges that accompany engagement, 42 particularly as they relate to the communities being served. "Engagement" is a term often used by researchers to describe interactions with the community. However, the quality of those interactions often hover on the lower ends of the community engagement continuum (eg, outreach and consultation), lack true leadership and bidirectional learning from communities (which occur at higher levels such as collaboration/shared leadership), and are rarely robust. In addition, there are few patient-centered, ecologically focused definitions of "community." 43 Therefore, researchers and community members often enter into partnerships not understanding the operationalized definition of community which lends the work of the partnership to a unilateral focus and further exclusion within the community. 44 The benefits of engagement begin early. Early initiation of high-quality, clearly defined community/patient engagement can mitigate mistrust, especially among marginalized communities, enhance participant enrollment and implementation of findings, and eliminate some of the difficulties with recruitment. 45 Therefore, we must engage the target patient population and health system stakeholders providers early in the intervention and study design phases to understand their needs and determine the implications of not addressing those needs within the study. Flexibility and the ability to adjust strategies are essential to engage community partners most effectively. While investigators and funders often recognize the value of engagement, it is not always well-implemented or sustained due to the effort and resources required. Engagement that is poorly designed, poorly implemented, or not sustained can increase rather than decrease mistrust in the community. The standards and milestones that PCORI requires of funded projects-from study design and recruitment to the dissemination of results-help to ensure that study teams engage meaningfully with stakeholders throughout the life of the study. Furthermore, PCORI recognizes the importance of allowing for flexibility of form so long as fidelity of function is preserved, enabling teams to match engagement to their particular study and stakeholder needs and to modify engagement form, as needed, to best cultivate trust in the stakeholder community. Engagement is essential in transitions of care given the complexity of different levels of care and heterogeneity of the populations of interest who are in transition. Particularly challenging, care transitions research involves the study of processes that encompass different providers, settings, and systems with the patient as the only consistent factor. Furthermore, it frequently involves an intervention that is itself complex and involves multiple layers (functions). In addition, while the professionals asked to execute the intervention may recognize its importance, they may lack awareness or agency to ensure the delivery of what is needed in the next step in the care chain. Factors such as these may explain the low fidelity to function observed in some studies. The case studies presented here represent a broad range of community stakeholders. All provide examples of partnerships that exemplify characteristics of higher levels of the community engagement continuum. In the case study below, the researcher-stakeholder partnerships developed a common definition of "community," and each demonstrated ways to account for the complexities of community in their implementation and evaluation processes. This case study illustrates the importance of understanding stakeholders' capacity for engagement and how best to achieve meaningful, bidirectional engagement. Case Study: "Sickle Cell Trevor Thompson Transition Project study: The Comparative Effectiveness of Peer Mentoring vs. Structured Education-Based Transition Programming for the Management of Care Transitions in Emerging Adults with Sickle Cell Disease" includes 14 sites in 7 states. The study examines the effectiveness of peer support in reducing acute care visits and improving quality of life among youth with sickle cell disease as they transition to adult care. Each site is represented by a "triad": a pediatric and adult sickle cell disease provider, a community-based organization, and the patient. Early on, the team noted that community relationships and capacity for community engagement varied significantly among sites. The study team worked iteratively with patient stakeholders and community partners to operationally define engagement and a purpose statement. These tools helped sites explore the concept of engagement within their own unique contexts. Creating a unique engagement goal also fostered a sense of ownership within each site and provided a foundation for sustainability of efforts beyond the study. Engagement pitfalls abound when researchers' values (eg, publications) misalign with community values (eg, quality of life). Therefore, co-developing the engagement goals fostered greater involvement in the process as well as the outcomes of the research. Lesson Learned: Developing a common definition of community allows partners to collectively recognize the heterogeneity of the community and to eliminate division that may inadvertently occur within the community due to the mistrust of the partnership among members of the community not directly involved in the partnership. Effective engagement begins with approaching all introductions without hierarchy or positions of power assists collaborators in dissolving pre-established power structures that may exist. Allowing sites to customize engagement approaches within the context and capacity of each site can enhance communication among stakeholders and support both implementation and sustainability. Often, the outcomes that matter most to patients, such as engagement in the discharge process, ability to carry out the discharge plan, physical and cognitive function, and caregiver burden, are harder to measure, lack validated instruments, and/or are deemed less rigorous by the larger scientific community. 46 Obtaining stakeholder input allows researchers to assess benefits and harms of interventions to enhance informed decision-making, account for heterogeneity in participants and settings, and clearly articulate all possible outcomes to stakeholders. 47 Patient-centered outcomes can often be proxy measures for important downstream variables including health maintenance and reduction of high-cost service use (eg, emergency department visits). In the case study below, we discuss how the target population can influence expectations for intervention success and therefore the outcomes most important to them after discharge. Case Study: In "Relative Patient Benefits of a Hospital-Patient Centered Medical Home Collaboration within an Affordable Care Organization to Improve Care Transitions," the team chose postdischarge adverse events as the primary outcome. Their reasons were: (1) these were felt to be important to the patients and caregivers in the study's Patient and Family Advisory Council and to other stakeholders; and (2) unlike readmissions, postdischarge adverse events are more directly related to quality of transitional care via a clear causal pathway. Lesson Learned: Postdischarge adverse events could only be measured via intensive postdischarge follow-up with patients, followed by physician adjudication. Yet the outcome was important to patients and the team found this outcome much more amenable to change than readmission rates. 18 Investigators have an ethical obligation to disseminate research findings. The first path for dissemination is often peer-reviewed abstracts for conferences and followed by journal publication. Yet, this strategy has well-known disadvantages: journal reviewers may be biased about the value of efficacy research over effectiveness studies, journals may not publish studies that have not demonstrated benefit despite the rigor of the design or significance of the findings as a whole, the process of reviews and revisions is lengthy, and paywalls (ie, for journal access) can prevent findings from being widely accessible. 48 We found that pragmatic trials of transitions of care interventions were difficult to publish in top-tier scientific journals, with the most common issue being that reviewers appeared to use traditional randomized controlled trial design criteria to evaluate them. This may change as more studies specify methodology standards used, are recognized for their methodologic rigor, and as the scientific community becomes accustomed to pragmatic trials. We recommend further educating reviewers to recognize the rigor and value of pragmatic, patient-centered studies of complex interventions (see Reeves 36 as an example). Stakeholders in each PCORI-funded study can help identify audiences who would benefit from being informed of the findings. Those who help develop each dissemination plan can also facilitate entrée into their own networks as vehicles to share findings with audiences who would directly benefit from and use that information. Dissemination of study results should begin with choosing stakeholders to codevelop and support plans for messaging, channels for dissemination, reach, and intent. There is now a trend to include stakeholders as coauthors on papers and presentations, as in community-based participatory research. However, educating stakeholders to take on this role typically requires intensive coaching and time. We recommend budgeting for stakeholders to help disseminate findings; they bring the research to life for decision makers. In the case studies below, we illustrate how nontraditional research partners can enhance dissemination of study results Case Study: In "An Emergency Department-to-Home Intervention to Improve Quality of Life and Reduce Hospital Use," involvement of a broad stakeholder group was critical to shaping every step of the research study, including dissemination. Patient and caregiver stakeholders created videos, presented at an annual PCORI meeting and on the PCORI website, outlining their contributions, why patient participation is critical to research, and what the contribution meant to them. Lesson Learned: Patients, clinicians, and other stakeholders engaged in research can provide powerful voices for change and strengthen research dissemination. Case Study: The development and publication of interim findings emerging from patient and caregiver focus groups of Project ACHIEVE offered an opportunity to pilot test principles and steps in their dissemination plan. This included leveraging Project ACHIEVE's extensive network of research partners and stakeholders and customizing messages for targeted end users-press releases, drafted tweets and 1-page summaries-for targeted end users (eg, patients and family caregivers, clinical providers and policy makers). Importantly, health literacy experts and patient/family caregiver representatives reviewed the patient-focused materials to ensure they were understandable and conveyed meaningful, actionable messaging. In addition, major hospital and advocacy organizations including the American Hospital Association, America's Essential Hospitals and Joint Commission Resources reviewed materials with incorporation of their suggested modifications for clinical providers, hospitals and policy makers. These efforts resulted in the peer-reviewed publication obtaining an Altmetric score of 226, placing it among the top 5% of research articles rated by the platform. Over 40 national news outlets covered the findings. One-page summaries remain readily available for download on the Project ACHIEVE website (https://achieve.med.uky.edu/achieve-news). Lessons Learned: Stakeholders are valuable active partners for dissemination of study results to users and policy makers; the research team needs to ensure that the message is compelling, direct, and succinct for end users. Importantly, dissemination as an interactive, iterative process, not a one-time activity. When policy reforms are proposed, transitional care studies are not always completed or teams are not always able to share findings. Dissemination can be challenging to align with policy reform and professional societies as demonstrated in this next case study. Case Study: The "Comparative Effectiveness Study of Post-Acute Care Options for Survivors of Stroke" was a retrospective study using clinical registry, survey, and claims data. Preliminary findings comparing outcomes for stroke patients who received postacute care in inpatient rehabilitation versus skilled nursing facilities were disseminated to the Medicare Payment Advisory Commission and key stakeholders because changes to payment were being considered that would impact delivery of inpatient rehabilitation care. The study's results illustrated how clinical data and patient-reported outcomes enhanced information in Medicare claims and improved the ability to assess effectiveness of post-acute care versus claims data alone. 49, 50 The Commission concluded that site-neutral payments for inpatient rehabilitation and skilled nursing facilities should exclude stroke patients, because clinical and patientreported outcomes showed the benefits of inpatient rehabilitation. Although disseminating directly to decision makers to influence policy reduced significance for peer-reviewed publications, findings were incorporated in the American Heart Association clinical guidelines that continue to recommend stroke patients go to an inpatient rehabilitation (versus skilled nursing) facility after hospital discharge. 51 Lesson Learned: Engaging stakeholders in dissemination before publication can lead to important advancements measured outside of peer-reviewed publications. This engagement approach also stands in contrast to press releases and publishing on websites like www. kff.org or https://evidencebasedprograms.org/ that enable policy officials and others to quickly evaluate the evidence underlying health and social programs. There is scant literature on how different stakeholders can be directly engaged with one another to drive translation of research into practice. In particular, while there evidence on ways that researchers can deliver study results to policy makers, little research has been conducted on how to engage patients and caregivers in this process. Heckert et al 52 acknowledge the inherent tension between the time it takes to meaningfully involve stakeholders and the time pressures to share timely results to inform care. The evidence-based framework designed by Lavis et al 53 may provide 1 way for researchers to more effectively and efficiently engage patients and caregivers in the transfer of knowledge to decision makers. This framework is rooted in 5 questions: (1) What should be transferred to decision makers? (2) To whom should research knowledge be transferred? (3) By whom? (4) How? And (5) With what effect? Table 2 illustrates stakeholder-driven strategies that researchers can use to successfully convey their study results to policy makers. Interventions that engage stakeholders, comprehensively address gaps in care, and improve the stakeholder experience and outcomes are complex. These complexities are magnified when evaluating novel transitional care interventions under real-world conditions. Transfer actionable messages from a body of research knowledge, not simply a single research report or the results of a single study to influence decision making To whom should research knowledge be transferred? Use multiple audience-specific messages. Identify the audience and tailor the message to the types of decisions they face and the types of decision-making environments in which they live or work (which can be time-consuming) By whom? Using credible messengers or trusted intermediaries (the messenger becomes the message) How? Interactive engagement (2-way exchanges) is more effective than passive processes (1-way push efforts by the researchers) With what effect? Knowledge transfer might inform debate, change awareness, or change a course of action to have an impact on decision-making around health Adverse events among medical patients after discharge from hospital The incidence and severity of adverse events affecting patients after discharge from the hospital Deficits in communication and information transfer between hospital-based and primary care physicians: implications for patient safety and continuity of care Lost in transition: challenges and opportunities for improving the quality of transitional care Role of pharmacist counseling in preventing adverse drug events after hospitalization Patient safety concerns arising from test results that return after hospital discharge Tying up loose ends: discharging patients with unresolved medical issues Rehospitalizations among patients in the Medicare fee-for-service program World Stroke Organization global stroke services guidelines and action plan Recommendations for the establishment of stroke systems of care: a 2019 update Effect of patient-centered transitional care services on clinical outcomes in patients hospitalized for heart failure: The PACT-HF Randomized Clinical Trial Stroke survivors' experiences transitioning from hospital to home Unmet needs of US acute stroke survivors enrolled in a transitional care intervention trial A pragmatic view on pragmatic trials Randomized pragmatic trial of stroke transitional care: the COMPASS study Clinical Effectiveness of Rehabilitation Services for Survivors of an Acute Ischemic Stroke Design of the patient navigator to Reduce Readmissions (PArTNER) study: a pragmatic clinical effectiveness trial The effects of a multi-faceted intervention to improve care transitions within an accountable care organization: results of a stepped-wedge cluster-randomized trial Improving evidence-based grouping of transitional care strategies in hospital implementation using statistical tools and expert review Impact of an emergency department-to-home transitional care intervention on health service use in medicare beneficiaries: a mixed methods study Complex interventions: how "out of control" can a randomised controlled trial be? Available at: www.pcori.org/sites/default/files/PCORI-Methodology-Standards Lessons from complex interventions to improve health Core functions and forms of complex health interventions: a patient-centered medical home illustration The PRECIS-2 tool: designing trials that are fit for purpose Translating research into new instructional technologies for higher education: the active ingredient process Towards a comprehensive model for understanding adaptations' impact: the model for adaptation design and impact (MADI) A case study of a theory-based method for identifying and reporting core functions and forms of evidence-based interventions Using research evidence to inform and evaluate early childhood intervention practices Available at: www.pcori.org/research-results/2017/comparingstrategies-help-people-traumatic-brain-injury-continue-recovering PRECIS-2: PRagmatic Explanatory Continuum Indicator Summary Developing and evaluating complex interventions: new guidance The FRAME: an expanded framework for reporting adaptations and modifications to evidence-based interventions Process evaluation of complex interventions: Medical Research Council guidance How to perform a preplanned process evaluation for complex interventions in geriatric medicine: exemplified with the process evaluation of a complex falls-prevention program for community-dwelling frail older fallers COMPASS trial in transitional stroke care: navigating towards true north A systematic review of adaptations of evidence-based public health interventions globally A scoping study of frameworks for adapting public health evidence-based interventions Adaptations of the evidence-based transitional care model in the U.S Critical issues in developing and following CBPR principles Methods for Community-Based Participatory Research for Health Methods guiding stakeholder engagement in planning a pragmatic study on changing stroke systems of care Defining "community" from the perspectives of individuals with sickle cell disease in rural Georgia Critiquing community engagement Why we need community engagement in medical research Measurement tools and outcome measures used in transitional patient safety; a systematic review Available at: www.pcori.org/research-results/ about-our-research/patient-centered-outcomes-research Available at: www.nytimes. com/2018/11/05/upshot/peer-review-the-worst-way-to-judge-researchexcept-for-all-the-others Unexplained variation for hospitals' use of inpatient rehabilitation and skilled nursing facilities after an acute ischemic stroke Assessment of home-time after acute ischemic stroke in medicare beneficiaries Guidelines for adult stroke rehabilitation and recovery: a guideline for healthcare professionals from the Researchers, patients, and other stakeholders' perspectives on challenges to and strategies for engagement How can research organizations more effectively transfer research knowledge to decision makers? The authors thank Karen Klein, MA, ELS, GPC, MWC, for editorial comments. The authors thank Sarah Birken, PhD, for content expertise.