key: cord-0739368-2ycqa7nm
authors: Richards, Julie E; Simon, Gregory E; Boggs, Jennifer M; Beidas, Rinad; Yarborough, Bobbi Jo H; Coleman, Karen J; Sterling, Stacy A; Beck, Arne; Flores, Jean P; Bruschke, Cambria; Grumet, Julie Goldstein; Stewart, Christine C; Schoenbaum, Michael; Westphal, Joslyn; Ahmedani, Brian K
title: An implementation evaluation of “Zero Suicide” using normalization process theory to support high-quality care for patients at risk of suicide
date: 2021-05-24
journal: Implement Res Pract
DOI: 10.1177/26334895211011769
sha: 0ba288c6eb27f4a7bbf4a7d243624058c8406280
doc_id: 739368
cord_uid: 2ycqa7nm

BACKGROUND: Suicide rates continue to rise across the United States, galvanizing the need for increased suicide prevention and intervention efforts. The Zero Suicide (ZS) model was developed in response to this need and highlights four key clinical functions of high-quality health care for patients at risk of suicide. The goal of this quality improvement study was to understand how six large health care systems operationalized practices to support these functions—identification, engagement, treatment and care transitions. METHODS: Using a key informant interview guide and data collection template, researchers who were embedded in each health care system cataloged and summarized current and future practices supporting ZS, including, (1) the function addressed; (2) a description of practice intent and mechanism of intervention; (3) the target patient population and service setting; (4) when/how the practice was (or will be) implemented; and (5) whether/how the practice was documented and/or measured. Normalization process theory (NPT), an implementation evaluation framework, was applied to help understand how ZS had been operationalized in routine clinical practices and, specifically, what ZS practices were described by key informants (coherence), the current state of norms/conventions supporting these practices (cognitive participation), how health care teams performed these practices (collective action), and whether/how practices were measured when they occurred (reflexive monitoring). RESULTS: The most well-defined and consistently measured ZS practices (current and future) focused on the identification of patients at high risk of suicide. Stakeholders also described numerous engagement and treatment practices, and some practices intended to support care transitions. However, few engagement and transition practices were systematically measured, and few treatment practices were designed specifically for patients at risk of suicide. CONCLUSIONS: The findings from this study will support large-scale evaluation of the effectiveness of ZS implementation and inform recommendations for implementation of high-quality suicide-related care in health care systems nationwide.

Many individuals see a health care provider prior to death by suicide, therefore health care organizations have an important role to play in suicide prevention. The Zero Suicide model is designed to address four key functions of high-quality care for patients at risk of suicide:

(1) identification of suicide risk via routine screening/assessment practices, (2) engagement of patients at risk in care, (3) effective treatment, and (4) care transition support, particularly after hospitalizations for suicide attempts. Researchers embedded in six large health care systems, together caring for nearly 11.5 million patients, are evaluating the effectiveness of the Zero Suicide model for suicide prevention. This evaluation focused on understanding how these systems had implemented clinical practices supporting Zero Suicide. Researchers collected qualitative data from providers, administrators, and support staff in each system who were responsible for implementation of practices supporting Zero Suicide. Normalization process theory, an implementation evaluation framework, was applied following data collection to: (A) help researchers catalog all Zero Suicide practices described, (B) describe the norms/conventions supporting these practices, (C) describe how health care teams were performing these practices, and (D) describe how practices were being measured. The findings from this evaluation will be vital for measuring the effectiveness of different Zero Suicide practices. This work will also Introduction Suicide rates have continued to rise across the United States for the past two decades, thus galvanizing a need for increased prevention and intervention efforts across settings (Caine, 2013; Hogan & Grumet, 2016) . Health care systems have a key role to play in suicide prevention, because many individuals see a health care provider prior to death by suicide-nearly a third in the week prior to suicide , half in the month prior, and the vast majority in the year prior Luoma et al., 2002) . The Zero Suicide (ZS) model is a quality improvement initiative for safer and more effective suicide care in health care settings that came out of a "call to action" developed by the Office of the US Surgeon General and the National Action Alliance for Suicide Prevention (U.S. Department of Health and Human Services Office of the Surgeon General and National Action Alliance for Suicide Prevention, 2012). The ZS model is not a formal validated treatment protocol with prescribed interventions, but rather a framework of evidence-based practices and implementation strategies designed to transform organizational culture (Hogan & Grumet, 2016 ). The ZS model strategy defines four essential clinical functions of high-quality care for patients at risk of suicide: identification of high-risk patients, engagement and care management, effective treatment, and supportive care transitions (see Table 1 for Terms and Definitions) (Brodsky et al., 2018; Education Development Center, 2020) . Each of these functions is supported by the application of a procedure or practice intended to support a specific function (Jolles et al., 2019) . Identifying high-risk patients involves screening and risk assessment, for example, using the nine-item Patient Health Questionnaire (PHQ-9)] (Kroenke et al., 2001) , and the Columbia Suicide Severity Rating Scale (C-SSRS) (Posner et al., 2011) . Engagement practices may include collaborative crisis response and safety planning interventions to help patients identify coping strategies and other resources during suicidal crises (Bryan et al., 2018; Stanley & Brown, 2012) . Effective treatments may include evidence-based therapies targeting intensity and/or frequency of suicidal thoughts and behaviors (Brown & Jager-Hyman, 2014) . Supportive care transitions ensure that patients receive uninterrupted care as they move between care settings, particularly from inpatient to outpatient settings following suicide attempt (Chung et al., 2017) . Although the ZS model has great potential to improve the way health care systems care for patients at risk of suicide, and has been widely adopted, there is limited evidence of the effectiveness of a collection of linked, mutually supporting clinical practices for prevention of suicide attempts (fatal and non-fatal). For example, the identification function is supported by research demonstrating that disclosure of suicidal thoughts on the PHQ-9 depression assessment is a strong predictor of subsequent suicide attempts (Louzon et al., 2016; Simon et al., 2013 Simon et al., , 2016 , but the link between identification of patients experiencing suicidal thoughts and the ability of health care systems to intervene effectively and help prevent suicide attempts is unclear. Similarly, though randomized clinical trials have demonstrated efficacy of some psychotherapeutic interventions for reducing suicide attempts, little is known about the effectiveness of these practices for suicide prevention in real-world settings or among people who have not volunteered to participate in research (Brown & Jager-Hyman, 2014) . Moreover, though prior research has demonstrated that systemic, bundled suicide prevention practices are more effective than any single practice (M. J. Coffey et al., 2015; Knox et al., 2003; Martin et al., 2016; While et al., 2012) , it is unclear which specific practices, or combination of practices, supporting different ZS functions are most effective (Ahmedani & Vannoy, 2014) .

Early examples of implementation of clinical practices related to the ZS model have been encouraging, such as the substantial reductions seen in annual suicide rates following implementation of the Perfect Depression Care program at the Henry Ford Health System (C. E. Coffey et al., 2013; Hampton, 2010) . Nevertheless, more rigorous study is needed to demonstrate how specific ZS clinical practices are associated with important patient outcomes, specifically suicide attempt and mortality. Therefore, researchers in six large health care systems are investigating the effectiveness of different clinical practices (and combinations of practices) for preventing suicide attempts and deaths (Yarborough et al., 2019) . At the time of this evaluation, these six health care systems had all implemented varying combinations of suicide prevention practices and were routinely collaborating via a formal suicide prevention learning collaborative (Bruschke & Flores, 2020) . Therefore, in support of the learning collaborative and the planned investigation of ZS effectiveness (Figure 1 ), this research team sought to build a blueprint of the practices supporting the four key clinical functions (identify, engage, treat, transition) of the ZS model across health care systems.

Thus, this evaluation focused on understanding how these six health care systems had implemented (or planned to implement) clinical practices supporting the four key clinical functions highlighted in the ZS Model, in support of a broader ZS evaluation. Building a thorough working knowledge of practice variation across systems would be essential for measuring and comparing the effectiveness of different ZS practices ( Figure 1 ). Therefore, we used normalization process theory (NPT), an implementation evaluation framework (McEvoy et al., 2014; Nilsen, 2015) , to help explain which individual and interrelated practices had been implemented and how those practices had been operationalized across different health care systems to support ZS. NPT was selected because the framework focuses on the organic process of how complex interventions become normalized in practice, shaped by four determinants-coherence (how the intervention is understood), cognitive participation (how users engage in the intervention), collective action (how the intervention is performed), and reflexive monitoring (how the intervention is monitored over time) ). Specifically, these four NPT determinants were applied following key informant data collection to help explain how different clinical practices supporting ZS had become normalized across organizations and care settings.

Six large health care systems, together caring for nearly 11.5 million members in California, Colorado, Michigan, Oregon, and Washington, are implementing clinical practices supporting the ZS model and agreed to participate in the implementation evaluation. All participating health care systems provide both comprehensive health care and insurance coverage to a defined patient population. Each system is described here and additional details are presented in Table 2 , using data compiled from 10/2018-9/2019 (Henry Ford Health System, 2019; Kaiser Permanente, 2019). The participating Kaiser Permanente (KP) health care systems in California together served approximately 9 million patient members, followed by Henry Ford Health System in Michigan serving 1.2 million, and KP health care systems in Oregon, Washington and Colorado similarly serving between 600,000-700,00 members each. The Institutional Review Board at each system approved its participation in the ZS evaluation.

The characteristics of the different health care system patient populations vary by sociodemographic characteristics associated with suicide, including age, race/ethnicity, income, and education (Nock et al., 2008; Rehkopf & Buka, 2006) . For example, at the time of this evaluation, Henry Ford provided care to the oldest patient population and had the largest proportion insured by Medicare (23.5%), while KP Southern California served the youngest population and had the largest proportion insured by Medicaid (10.4%). Henry Ford served the largest proportion of Black/African American patients (28.2%), KP Southern California the largest proportion of Hispanic/Latinx patients (40.9%), and KP Northern California the largest proportion of Asian patients (20.1%).

In 2016, Kaiser Permanente Care Management Institute (CMI) , convened a Suicide Prevention Learning Collaborative workgroup, which included clinical, operational, and patient safety stakeholders, as well as peer advisors (KP health plan members and advocates with lived experience from national mental health advocacy organizations), and researchers, including those from all six health care systems participating in the ZS evaluation. The charter of this workgroup included cross-regional collaboration on implementation of best practices for suicide prevention (Bruschke & Flores, 2020) . With oversight from operational mental health leadership, CMI works with members from the suicide prevention learning collaborative to facilitate development new electronic health record (EHR)-based tools, trainings and metrics to support key ZS practices which build from the work described in this evaluation.

A key informant meeting guide (see Supplement A) was developed to assist researchers embedded in each health care system (JER, GES, JMB, AB, BHY, KJC, SAS, JW, BKA) to collaborate with key informants and document clinical practices the systems had implemented (or planned to implement) across the different clinical functions of ZS. Prior to data collection, embedded researchers all had some knowledge about suicide prevention initiatives and clinical practices in their health care systems, so data collection was designed to enhance that knowledge. Key informants included local leaders and staff with knowledge of implementation of suicide prevention related processes and protocols in the various clinical settings. A structured data collection template (Miles et al., 2014) was designed to help the researchers catalog details about current and planned suicide prevention practices across care settings (Supplement B). The data collection template did not explicitly assess the domains of the ZS Model focused on implementation strategies (training, leadership, and continuous quality improvement) (Education Development Center, 2020), but these domains were implicitly addressed in questions about how the practices had/or would be implemented. The template was originally developed and tested at KP Colorado and subsequently tested at two other health care systems, with the finalized version made available for widespread use in March 2018. Templates were populated and reviewed iteratively by the embedded researcher and team members at each participating health care system. In parallel, templates were also uploaded to a private web-based file storage platform available to Health care System Research Network members (Health Care Systems Research Network, 2019) and each reviewed in detail by members of the full study team online during routine all-site team meetings (December 2018 through April 2019). Following these presentations, some researchers made additional edits and updates to their templates to reflect changing ZS practices across their systems and uploaded current versions to the shared private website. In August 2019, a health services research (JER) used the most recent version of the template from each health care system to summarize ZS practices across health care systems, including (1) the function addressed; (2) a brief description of the practice intent and mechanism of intervention (e.g., EHR-based clinical decision support tools), standard workflows/processes and health care system policies; (3) the target patient population and service setting; (4) when/how the practice was (or will be) implemented; and (5) whether/how the practice was documented/measured.

NPT was applied following key informant data collection to help understand how health care systems had implemented different clinical practices, and combinations of practices supporting the ZS model . NPT assumes the way that practices become routinely embedded and integrated into their social contexts-how they become normalized-is not a structured sequence of events, but instead is an organic process shaped by four determinants: coherence, cognitive participation, collective action, and reflexive monitoring Nilsen, 2015) . To support this evaluation, we first developed a working interpretation of the NPT determinants for application to ZS implementation, guided by two prior studies evaluating implementation of depression care in primary care (Franx et al., 2012; Gunn et al., 2010) . Specifically, like Gunn et al. (2010) , we developed working definitions for how each of the NPT determinants would apply to health care system implementation of practices (and combinations of practices) supporting the ZS model (Table 3) . Specifically, we used NPT to catalog and connect the descriptions of current and future clinical practices to ZS implementation (i.e., creating coherence) across the health care systems, and to facilitate the specification of the workflows/processes (cognitive participation; collective action) and approaches to measuring these workflows/ processes (reflexive monitoring). This approach involved using NPT pragmatically to define specific products resulting from our application of NPT to the key informant data (Table  3) to support the goals of the broader ZS evaluation (Figure 1) . Specifically, to describe coherence, a health services research (JER), first used the templated key informant data to create a thematic network (Attride-Stirling, 2001) (i.e., affinity diagram) of the existing practices supporting the four key clinical functions of ZS across all six participating health care systems. Embedded researchers from all participating health care systems, responsible for data collection, participated in several rounds of iterative review and refinement to ensure ZS practices were comprehensively and accurately represented in the final version of the thematic network. Second, we summarized the current state of practices supporting ZS across health care systems and created a list of tools (e.g., screeners, assessments, templates) supporting those practices to describe cognitive participation and collective action. Third, to describe reflexive monitoring we built a working knowledge of whether and how ZS practices were being measured and how different practices and combinations of practices could be measured over time. This work also involved conceptually mapping a suicide care continuum, similar to treatment cascades used to evaluate care gaps for patients living with HIV (Kay et al., 2016) .

Embedded health care system researchers had all begun populating the implementation templates in collaboration with local key informants by June 2018. An average of 14 key informants per health care system provided data to help populate each template (range 7-26). Key informants included providers, administrators, EHR programmers, and project managers (often known to embedded researchers) who were responsible for supporting implementation of practices supporting ZS in different ways. All health care systems had participated in virtual team review and updated their templates at least once before the designated health services researcher (JER) begin reviewing template data in August 2019. Prior to application of NPT, the health services researcher organized the practices summarized in the templates by function. In some cases, the function was changed (e.g., "treat" to "engage") to be consistent with the ZS framework definitions, in collaboration with a ZS expert (JGG).

Each product associated with application of NPT to the templated data (Table 3) was created consecutively and is described in detail below, including (1) the catalog of practices understood to support ZS (coherence); (2) the summary of the current state of norms/ conventions supporting these practices and how health care teams performed these practices (cognitive participation and collective action); and (3) a list of tools available to measure practices across systems and a model describing combinations of practices available to measure the continuum of suicide care across settings (reflexive monitoring).

In Figure 2 , we present the thematic network of clinical practices key informants understood to support ZS (coherence), derived from the implementation templates completed by an embedded researcher at each site. The number of current/planned clinical practices described by key informants varied across health care systems (average per health system = 32, range 19-55), as did the level of detail in the description of each practice (Supplement B). All templates included fairly detailed descriptions of practices supporting the identify function (i.e., standard workflows and tools), but practices supporting the engage, treat and transition functions were often less clearly defined. Moreover, often the clinical, operational, and patient safety stakeholders did not explicitly reference the ZS model when they described the current state of health care system practices supporting suicide prevention. Some templates also included descriptions of practices that were not specifically designed to support suicide prevention, but rather mental health and addiction care more generally. For example, substance use disorder screening/assessment, social support, care coordination/ outreach, and types of psychotherapy. Relatedly, some of these practices had been in place at most health care systems more than 5 years prior to this evaluation (like depression severity assessment with the PHQ-9 and different types of psychotherapy), while others had been implemented more recently (suicide severity assessment with the C-SSRS).

Identification of high-risk patients.-Practices described by key informants used to support the identification function of ZS included standard clinical decision support tools, implemented in the EHR, to support suicide risk screening followed by more comprehensive risk assessment. At the time of this evaluation, all health care systems monitor depression symptom severity, including suicidal ideation, among adult patients receiving care in the outpatient mental health specialty setting. All systems also had screening/assessment practices in place for substance use disorders, although the norms/conventions supporting these screening practices varied. Workflow variation across care settings was common and the health care systems had different screening and assessment approaches (e.g., varying criteria for deciding which patients received the PHQ-9). In the primary care setting, population-based screening and suicide risk assessment were less common; screening and assessment were more often done at the discretion of primary care providers and many systems were engaged in planning future implementation of different approaches for more universal screening/assessment. Some systems were also in the process of planning implementation of suicide risk prediction algorithms (Simon et al., 2018) for purposes of enhancing suicide risk assessment and engagement in suicide-related care.

Engagement and care management.-Practices designed to support the engagement function of the ZS model consistently included references to development of a safety plan or crisis response plan (Bryan et al., 2018; Stanley & Brown, 2012) , which usually included lethal means assessment (i.e., discussions about prescription medications, firearms, and planning how to limit access). Researchers cataloged varying safety planning workflows and documentation practices across sites and providers (Yarborough et al., 2019) ; however, a shift toward practice standardization was underway. CMI was facilitating the standardization of safety planning by promoting use of a common EHR-based template across health care systems (i.e., KP regions). Key informants also described integrating social support, referral practices, and care coordination programs to engage patients in care for suicidality. The norms/conventions supporting these practices, however, were often not well-defined.

Effective treatment.-Embedded health system researchers documented a variety of practices across health care systems that addressed the treatment function of ZS. These included pharmacotherapy; different types of evidence-based psychotherapy available for depression and other mental health disorders, including psychoeducational group-based therapy and cognitive behavioral therapy (CBT) (Hofmann et al., 2012) , electroconvulsive therapy (Pagnin et al., 2004; UK ECT Review Group, 2003) ; and psychiatric hospitalization, which is the current standard of care for patients at high risk of suicide (Brown & Jager-Hyman, 2014) . Researchers also cataloged evidence-based psychotherapies being used in some health care systems that were designed specifically for patients at high risk of suicide, including CBT for suicide prevention (CBT-SP) (Stanley et al., 2009 ) and dialectical behavior therapy (DBT) (Linehan et al., 2006) . Generally, mental health specialty providers in all systems were responsible for treatment of patients at risk of suicide, but the specific treatments providers were using were unknown in some health care systems and difficult to discern in others (e.g., differentiating CBT from CBT-SP, and DBT groups from other group-based therapies). Moreover, it was not possible to know exactly what parts of specific therapies patients were receiving (e.g., specific DBT/CBT skills/strategies) (Bryan, 2019; Koerner, 2013) .

Supportive care transitions.-The practice consistently documented across health care systems designed to support the transition function of ZS was follow-up care after discharge from inpatient psychiatric settings. However, follow-up practices after discharge from the emergency department (ED) setting (without inpatient admission) were varied. Other practices designed to support care transitions included intensive case management programs typically designed to support patients with severe and persistent mental illnesses in outpatient settings (Dieterich et al., 2017) . Finally, key informants at two health care systems reported sending caring messages (Motto, 1976; Motto & Bostrom, 2001) following inpatient discharge which addressed the ZS transition function, but the norms/conventions supporting this practices were not well-defined.

The data from key informants were also used to conceptualize how different practices and combinations of practices could be measured over time to support development of standard process measures and evaluation of ZS effectiveness (Figure 1 ). First, we summarized the common screening tools/assessments and health care utilization codes that could be extracted from EHRs across systems (Table 4) . Specifically, at the time of this evaluation all health care systems were using the PHQ-9 to measure depression severity (Kroenke et al., 2001) and most participating health care systems also implemented a standardized tool for suicide risk assessment (most commonly the C-SSRS) (Madan et al., 2016; Posner et al., 2011) in their mental health and addiction medicine settings. No systems were using the same tools for drug use disorder screening/assessment, but several systems were using a common screening tool for alcohol use disorders (AUDIT-C) (Bradley et al., 2007; Bush et al., 1998) . All systems were also in various stages of implementing new EHR tools designed to support consistent documentation of the key components of a safety plan (Bottomley, 2019; Stanley & Brown, 2012) . Providers in all systems also had routine procedures for documenting treatment in patient EHRs via diagnosis and procedure codes (i.e., ICD and CPT) associated with health care encounters (CMS.gov Centers for Medicare & Medicaid Services, 2019; World Health Organization, 1992) . Finally, follow-up within 7-day and 30-day post-discharge from inpatient psychiatric settings was consistently measured across all health care systems in accordance with national quality metrics measuring (National Committee for Quality Assurance [NCQA], 2019).

Next, we defined a conceptual care continuum (i.e., treatment cascade) (Figure 3 ) to measure combinations of measurable practices supporting the ZS model. In this continuum, primary care patients enter the ZS care pathway at the point of population-based depression screening. Patients receiving mental health specialty care enter the care pathway via routine depression and/or suicide risk severity assessment. Patients seen in emergency or urgent care settings for mental health-related concerns also enter at the point of depression/suicide risk severity assessment. Patients identified as being at risk of suicide via screening/assessment may be engaged in collaborative safety planning and receive one or more types of treatment (e.g., pharmacotherapy, psychotherapy). Finally, patients who received care for suicidality in an inpatient or ED setting receive proactive outreach following discharge.

This evaluation used NPT (McEvoy et al., 2014; Nilsen, 2015) to build a fundamental understanding of how six health care systems had implemented ZS practices to support four key clinical functions of high-quality care for patients at risk of suicide, and lays the foundation for health systems nationwide to understand the impact of implementing these practices. Our innovative application of NPT to data from key informants enabled us to comprehensively catalog practices understood to support ZS (coherence); summarize the current state of norms/conventions supporting these practices (cognitive participation) and how health care teams performed these practices (collective action); and define how we know ZS practices are happening and how to measure when they occur (reflexive monitoring). Embedded researchers who all had some knowledge about suicide prevention initiatives and clinical practices in their health care systems strengthened data collection and interpretation. The most well-defined practices focused on the identification functionspecifically, the use of common screening/assessment tools (Kroenke et al., 2001; Posner et al., 2011) to identify patients at high risk of suicide attempt. Many potential treatment practice options were also defined, but few treatment practices identified were designed specifically for patients at risk of suicide and it was difficult to differentiate whether and how providers were using various therapy options. In addition, engagement and transition practices were often inconsistently used and measured within health care settings; but health care systems were consistently measuring health care encounters documentation following psychiatric hospitalization (NCQA, 2019) and a shift toward standardized safety planning was underway (Bruschke & Flores, 2020; Stanley & Brown, 2012) .

This working knowledge of ZS practice variation will be used to support efforts to continue to improve suicide care across health care systems nationwide (Bruschke & Flores, 2020) and large-scale evaluation of the effectiveness of ZS practices, both alone and as part of a continuum of care for patients at risk of suicide. Specifically, defining an approach for measuring different practices (and combinations of practices) over time via development of standard process metrics (i.e., reflexive monitoring) have enabled participating health care systems, and will enable others nationwide, to undertake their own appraisal processes in support of ZS implementation. CMI worked in collaboration with mental health leaders to develop a standardized set of metrics to track implementation of ZS practices and suicide prevention outcomes (Bruschke & Flores, 2020) . These metrics will enable clinical, operational, and patient safety stakeholders to measure whether identification and engagement practices are occurring when expected (i.e., continuous quality improvement) within their own health care systems. The working knowledge of the variation of ZS practice implementation across health care systems will also allow researchers to compare the effectiveness of different approaches (i.e., workflows) used for identification, engagement, and transitioning patients at high risk of suicide. This evaluation will also support treatment metric development; however, as this work demonstrated, the investigation of the real-world effectiveness of specific psychotherapies for suicide prevention may be limited until we are able to more accurately define ways to measure variation in these treatment practices.

This project can provide a framework to support ongoing ZS implementation and evaluation in health care systems across the country. Specifically, this evaluation suggests important considerations for integrating or normalizing new practices to support the ZS model. For example, as suicide risk prediction algorithms (Kessler et al., 2017; Simon et al., 2018) are implemented to improve identification and engagement of patients at risk (U.S. Department of Veterans Affairs, 2017), it will be important to consider how to support this practice within the context of existing norms/conventions, workflows, and tools. This evaluation also underscored the potential utility of using standardized definitions and tools to measure existing engagement and transition practices, such as care coordination/outreach and caring message programs. Caring contact programs in particular have potential to be a simple and effective ZS practice, scalable across different types of health care systems (Carter et al., 2005 (Carter et al., , 2007 Comtois et al., 2019; Hassanian-Moghaddam et al., 2011; Luxton et al., 2012; Motto, 1976; Motto & Bostrom, 2001) .

The work presented here will also provide a vital foundation for understanding potential adaptation of practices supporting ZS during environmental disruption and evaluating those changes. For example, in response to the coronavirus disease 2019 (COVID-19) pandemic many US health care organizations rapidly converted to providing many services, including mental health and addiction medicine care, virtually (Wosik et al., 2020) . The processes used to identify patients at risk of suicide via virtual screening/assessment practices may be different than prior in-person practices. Moreover, reduced reliance on emergency and inpatient psychiatric settings during the COVID-19 response may also impact suicide rates and our understanding of the effectiveness of mental health care provided in these settings (Loch, 2014; Rabinowitz et al., 1994) . When the effects of COVID-19 on suicide become clearer (Reger et al., 2020) , understanding how ZS implementation adapted and changed during this time across these health care organizations will be vital for understanding which practices may have helped mitigate potential unintended negative effects of social isolation on suicide outcomes (Ahmed et al., 2020; Fatke et al., 2020; Vieira et al., 2020) .

This evaluation did not explicitly address the implementation strategy domains of the ZS model-training, leadership, and continuous quality improvement (Education Development Center, 2020) . These domains are critical to ensuring that practices supporting the four key clinical functions of ZS we evaluated result in high-quality suicide care. For example, providing training in specific evidence-based interventions provides clinicians with valuable skills and enhances fidelity to the intervention. The presence of a completed safety plan in an EHR does not necessarily mean that the safety plan was completed collaboratively in a way that was meaningful to the patient. Training, leadership, and continuous improvement are implementation strategies critical for ensuring consistent delivery of high-quality care and will be important considerations for the future evaluation of the effectiveness of different ZS practices (and combinations of practices).

This evaluation also did not take variations in practice maturity into account or consider all the ways that clinical ZS practices interact with related interventions in health care systems. For example, screening and assessment for substance use disorders was identified as a ZS identification practice, which makes sense given the strong association between suicide and substance use disorders (Espinet et al., 2019; Wilcox et al., 2004) , particularly alcohol use disorders (Bagge et al., 2013; Caetano et al., 2013; Cherpitel et al., 2004; Lejoyeux et al., 2008; Powell et al., 2001; Richards et al., 2020b) . However, we did not consider whether and how to measure engagement in care for alcohol and drug use disorders (NCQA, 2020), which could also be useful mechanism for improving suicide-related care (Richards et al., 2020a (Richards et al., , 2020b or addressing suicidal ideation and risky patterns of alcohol use together for purposes of suicide prevention (Kalk et al., 2019) .

Last, health care systems participating in this evaluation provide both integrated, comprehensive health care and insurance coverage to a defined member/patient population. This enabled us to define a conceptual care continuum for patients at risk of suicide across care settings (e.g., primary care/mental health specialty, inpatient/outpatient), but this care continuum may not be generalizable to organizations without the same responsibility to provide comprehensive health care to a defined population of members/patients across care settings. Health care organizations may also not have embedded researchers to assist with monitoring ZS implementation progress. However, many organizations have stakeholders invested in suicide prevention (like the clinical, operational, and patient stakeholders who participate in the suicide prevention collaborative within our health care systems) who can utilize the products of this evaluation (e.g., practice descriptions, measurement tools) to inform their own quality improvement processes.

This novel evaluation used NPT in combination with data from key informants to improve our understanding of clinical practices supporting the ZS model and inform recommendations for implementation of high-quality care for patients at risk of suicide in health care systems nationwide. Specifically, NPT was applied to create a catalog and description of practices supporting the four key clinical functions of ZS and model how to measure specific practices. These schemas will be used to inform a large-scale evaluation of the effectiveness of different ZS practices (and combinations of practices) and provide a blueprint to support suicide prevention practice implementation across health care systems, service settings, and patient populations.

Refer to Web version on PubMed Central for supplementary material.

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This evaluation was supported by the National Institute of Mental Health U01 MH114087 and U19 MH092201. Portions of these findings were presented as a poster at the 2019 12th Annual Conference on the Science of Dissemination and Implementation in Health, Arlington, VA, United States. Suicide risk care continuum. color coding corresponds to the ZS function defined in Figure 2 (blue = identification, green = engagement, orange = treatment, purple = transition). Richards et al. Page 22   Table 1 .

Determinant A factor that enables or hinders the clinical practice from achieving the desired effect.

The purpose of a practice. The core 4 clinical functions of high-quality suicide care defined in the ZS Model include identification, engagement, treatment and transition.

Process or event through which a clinical practice operates to affect outcomes.

The application of a procedure intended to support a specific function (aka "form following function"). This evaluation focuses on clinical practices intended to support the ZS Model.

A framework designed to support system-wide, organizational commitment to high-quality suicide care in healthcare.

ZS: Zero Suicide. Implement Res Pract. Author manuscript; available in PMC 2022 January 01. 

Epidemic of COVID-19 in China and associated psychological problems

Health care contacts in the year before suicide death

National pathways for suicide prevention and health services research

Variation in patterns of health care before suicide: A population case-control study

Thematic networks: An analytic tool for qualitative research

Alcohol as an acute risk factor for recent suicide attempts: A case-crossover analysis

Identifying suicide on the front lines of health

AUDIT-C as a brief screen for alcohol misuse in primary care

The Zero Suicide model: Applying evidence based suicide prevention practices to clinical care

Evidence-based psychotherapies for suicide prevention: Future directions

Zero Suicide research: Kaiser Permanente

Cognitive behavioral therapy for suicide prevention (CBT-SP): Implications for meeting standard of care expectations with suicidal patients

Use of crisis management interventions among suicidal patients: Results of a randomized controlled trial

The AUDIT alcohol consumption questions (AUDIT-C): An effective brief screening test for problem drinking. Ambulatory Care Quality Improvement Project (ACQUIP)

Acute alcohol intoxication and suicide among United States ethnic/racial groups: Findings from the national violent death reporting system

Forging an agenda for suicide prevention in the United States

Postcards from the EDge project: Randomised controlled trial of an intervention using postcards to reduce repetition of hospital treated deliberate self poisoning

Postcards from the EDge: 24 month outcomes of a randomised controlled trial for hospital-treated self-poisoning

Acute alcohol use and suicidal behavior: A review of the literature

Suicide rates after discharge from psychiatric facilities: A systematic review and meta-analysis

ICD-10-CM, ICD-10-PCS, CPT, and HCPCS code sets

An update on perfect depression care

Suicide in a health maintenance organization population

Effect of augmenting standard care for military personnel with brief caring text messages for suicide prevention: A randomized clinical trial

Predisposing and protective factors influencing suicide ideation, attempt, and death in patients accessing substance use treatment: A systematic review and meta-analysis protocol

Psychische Probleme in der Pandemie-Beobachtungen wahrend der COVID-19-Krise

Implementing a stepped-care approach in primary care: Results of a qualitative study

Embedding effective depression care: Using theory for primary care organisational and systems change

Depression care effort brings dramatic drop in large HMO population's suicide rate

Postcards in Persia: Randomised controlled trial to reduce suicidal behaviours 12 months after hospital-treated self-poisoning

Who we are

The efficacy of cognitive behavioral therapy: A review of meta-analyses

Suicide prevention: An emerging priority for health care

Correction to: Core functions and forms of complex health interventions: A patient-centered medical home illustration

Kaiser Permanente fast facts

Addressing substance misuse: A missed opportunity in suicide prevention

The HIV treatment cascade and care continuum: Updates, goals, and recommendations for the future

Developing a practical suicide risk prediction model for targeting high-risk patients in the Veterans health Administration

Risk of suicide and related adverse outcomes after exposure to a suicide prevention programme in the US Air Force: Cohort study

What must you know and do to get good outcomes with DBT? Behavior Therapy

The PHQ-9: Validity of a brief depression severity measure

Characteristics of suicide attempts preceded by alcohol consumption

Two-year randomized controlled trial and follow-up of dialectical behavior therapy vs therapy by experts for suicidal behaviors and borderline personality disorder

Discharged from a mental health admission ward: Is it safe to go home? A review on the negative outcomes of psychiatric hospitalization

Does suicidal ideation as measured by the PHQ-9 predict suicide among VA patients?

Contact with mental health and primary care providers before suicide: A review of the evidence

Caring Letters Project: A military suicide-prevention pilot program

Psychometric reevaluation of the Columbia-Suicide Severity Rating Scale: Findings from a prospective, inpatient cohort of severely mentally ill adults

Mates in construction suicide prevention program: A five year review

Implementing, embedding, and integrating practices: An outline of normalization process theory

Development of a theory of implementation and integration: Normalization process theory

A qualitative systematic review of studies using the normalization process theory to research implementation processes

Qualitative data analysis: A methods sourcebook

Suicide prevention for high-risk persons who refuse treatment

Initiation and engagement of alcohol and other drug abuse or dependence treatment (IET)

Making sense of implementation theories, models and frameworks

Suicide and suicidal behavior

Efficacy of ECT in depression: A meta-analytic review

The Columbia-Suicide Severity Rating Scale: Initial validity and internal consistency findings from three multisite studies with adolescents and adults

Alcohol consumption and nearly lethal suicide attempts. Suicide and Life-Threatening Behavior

Understanding who improves after psychiatric hospitalization

Suicide mortality and coronavirus disease 2019 -A perfect storm?

The association between suicide and the socio-economic characteristics of geographical areas: A systematic review

Association between patterns of alcohol use and short-term risk of suicide attempt among patients with and without reported suicidal ideation

Short-term risk of suicide attempt associated with patterns of patient-reported alcohol use determined by routine AUDIT-C among adults receiving mental healthcare

Risk of suicide attempt and suicide death following completion of the Patient Health Questionnaire depression module in community practice

Predicting suicide attempts and suicide deaths following outpatient visits using electronic health records

Does response on the PHQ-9 Depression Questionnaire predict subsequent suicide attempt or suicide death?

Cognitive-behavioral therapy for suicide prevention (CBT-SP): Treatment model, feasibility, and acceptability

Safety planning intervention: A brief intervention to mitigate suicide risk

Efficacy and safety of electroconvulsive therapy in depressive disorders: A systematic review and meta-analysis. The Lancet

Department of Health and Human Services Office of the Surgeon General and National Action Alliance for Suicide Prevention

VA REACH VET initiative helps save veterans lives: Program signals when more help is needed for at-risk Veterans

Isolamento social e o aumento da violencia domestica: o que isso nos revela? [The increase in domestic violence during the social isolation: What does it reveals?

Implementation of mental health service recommendations in England and Wales and suicide rates, 1997-2006: A cross-sectional and before-and-after observational study

Association of alcohol and drug use disorders and completed suicide: An empirical review of cohort studies. Drug and Alcohol Dependence

The ICD-10 classification of mental and behavioural disorders: Clinical descriptions and diagnostic guidelines

Telehealth transformation: COVID-19 and the rise of virtual care

Challenges of population-based measurement of suicide prevention activities across multiple health systems

Coherence Do people know what the work is? What clinical practices support the four clinical functions of ZS? Thematic network Cognitive Participation Do people join in to the work? What norms/conventions support ZS practices? Summary of current workflows supporting ZS practices Collective Action Skillset Workability Contextual Integration Interactional Workability Relational Integration How do people do the work? What tools are used to support ZS practices? How? Summary of clinical decision support tools used across healthcare systems Reflexive Monitoring How do we know the work is happening? How should we measure ZS practices?

manuscript; available in PMC

Social Support Programs X X Referrals/"Warm" hand-offs X X X X X X X Care Coordination/Outreach Programs X X X X X X X Care Management Programs X X X X X X X Treatment Safety Plan updates/monitoring X X X X X X X b Safety Plan Psychotherapy X X X X X X X ICD-10 Pharmacotherapy X X X X X X X NDC CodeElectroconvulsive Therapy X X X X X X X

Psychiatric Hospitalization X X X X X X Defined as more than 1 health care system presently or in the process of being implemented.b Discrete data element(s) implemented to capture EHR documentation.c Healthcare encounter (in-person, phone, etc.) EHR documentation.