key: cord-0734001-c7qitwuq
authors: Silfee, Valerie; Williams, Kelly; Leber, Brett; Kogan, Jane; Nikolajski, Cara; Szigethy, Eva; Serio, Catherine
title: Health Care Provider Perspectives on the Use of a Digital Behavioral Health App to Support Patients: Qualitative Study
date: 2021-09-28
journal: JMIR Form Res
DOI: 10.2196/28538
sha: e061f7f243b3141232d8610c430613c8770c388a
doc_id: 734001
cord_uid: c7qitwuq

BACKGROUND: Despite the growing evidence indicating the efficacy of digital cognitive behavioral interventions (dCBIs) for behavioral health (BH) treatment, broad and consistent use of such interventions has been limited by knowledge obtained in real-world settings, including factors that impact provider uptake/referral. Engaging providers early in the implementation process offers an opportunity to explore their needs and behaviors, integrate interventions into workflows, and better understand provider setting capabilities. OBJECTIVE: This study assessed providers’ views on the feasibility and acceptability of delivering a cognitive behavioral therapy (CBT)-based mobile app in multiple care settings. METHODS: Participating providers included BH and physical health (PH) providers from a women’s health center, an outpatient BH clinic, and both rural/urban primary care settings. All participating providers cocreated workflows through facilitated workshops, including establishing feedback loops between the project team and providers and identifying clinical champions at each site. Over a 12-week period, the providers referred adult patients experiencing anxiety or depression to a mobile app-based dCBI, RxWell, and provided other indicated treatments as part of usual care. Referrals were completed by the providers through the electronic medical record. To better understand facilitators of and challenges in integrating RxWell into routine practice and perceptions of sustainability, a series of qualitative interviews was conducted. Interview data were analyzed to identify major themes using an inductive content analysis approach. RESULTS: A total of 19 provider interviews were conducted to discover motivators and barriers for referring RxWell. The providers benefited from a focused discussion on how to incorporate the referral process into their workflow, and knowing the app content was rooted in evidence. Although the providers believed engaging in experiential learning was important, they indicated that more education on the digital health coach role and how to monitor patient progress is needed. The providers thought patient engagement may be impacted by motivation, a lack of comfort using a smartphone, or preference for in-person therapy. The providers also expressed enthusiasm in continuing to refer the app. They liked the ability to provide patients with support between sessions, to have an extra treatment option that teaches BH exercises, and to have a CBT treatment option that overcomes barriers (eg, wait times, copays, travel) to traditional therapy modalities. CONCLUSIONS: Digital intervention success in health care settings relies heavily on engagement of key stakeholders, such as providers, in both design and implementation of the intervention and focused evaluation within intended care setting(s). Scaling digital interventions to meet the mental health needs of patients in usual care settings leans on thoughtfully constructed and streamlined workflows to enable seamless referral of patients by providers. Our findings strongly suggest that providers are supportive of digital tool integration to support the mental health of patients and endorse its use within their routine workflow.

Approximately 1 in 5 adults in the United States lives with a mental illness [1] , with 40 million affected by anxiety and 16 million affected by depression annually [2, 3] . Coronavirus disease 2019 (COVID-19) has intensified symptom expression, with more than half of Americans reporting worsening mental health due to COVID-19 and 1 in 5 anticipating the pandemic to majorly impact their overall functioning [4] . Although effective treatments are available to manage anxiety and depression, including medication [5] and psychotherapy (including cognitive behavioral therapy [CBT]) [6] [7] [8] , many barriers exist in providing and receiving mental health care, leading to less than half of those experiencing mental health issues receiving treatment [1] .

Barriers to receiving mental health care include stigma [9, 10] , high costs for care [11] , limits in health insurance access or in-network care [12] , and limited access to mental health care professionals [13] . There are shortages of mental health professionals across the United States, with data from 2018 estimating that only one-quarter of the nation's need for a mental health provider are being met [12] . Provider shortages leave patients with extended wait times and longer distances to travel for care [12] , with 1 study finding patients waiting for up to 90 days for an initial psychiatric visit [14] . Moreover, two-thirds of primary care physicians (PCPs) struggle to obtain timely outpatient mental health services for patients, citing lack of mental health care providers, health plan barriers, and inadequate health insurance coverage as important challenges [15] . PCPs often serve as the main source of treatment for patients with common mental health concerns, and workforce shortages in primary care settings present yet another barrier to accessing mental health care [16] .

Digital cognitive behavioral interventions (dCBIs) can address barriers and care gaps in mental health care by increasing treatment accessibility and reducing the time it takes to receive care. It is estimated that over 10,000 of the 325,000 commercially available health and wellness digital apps specifically target mental or behavioral health (BH) [17] , and over the past several years, significant investments of up to $9.4 billion in the United Sates (2020) have been made to develop, validate, and regulate these apps for clinical use [18, 19] . Undoubtedly, the demands of COVID-19 on our health care system have accelerated the opportunity to improve access to mental health services through digital health technologies [20] .

There is a growing interest among providers to have access to digital tools [21] . Mobile app-based dCBIs have the potential to effectively reduce stress, anxiety, and depression [22] [23] [24] [25] , and improve care delivery for providers seeking to scale their mental health treatment options, streamline their workload, improve patient experience, and lower cost [26, 27] . Unfortunately, providers are infrequently referring and encouraging the use of a dCBI in their practices due to barriers such as a lack of integrated workflows (eg, prescribing and monitoring through the electronic medical record), low confidence in the app content or security, high costs, and limited bandwidth to identify, from thousands of commercial apps, the right one for their patients [18, 28, 29] .

There are many factors that influence the uptake and scalability of dCBIs in provider settings. Engaging providers early in the implementation process offers an opportunity to explore their needs and behaviors, integrate interventions into workflows, and better understand capabilities and limitations of routine care settings [30, 31] . Learning more about provider perspectives is an important step in addressing the complex issues around the effective implementation and scalability of digital tools [17, 31, 32] . Our study provides results from a system-level quality improvement initiative that includes an assessment of providers' views on the feasibility and acceptability of delivering a dCBI, RxWell, in multiple routine care settings.

UPMC is a large integrated delivery and finance system (IDFS) headquartered in Western Pennsylvania, offering both medical care and health care coverage to more than 3.9 million members, while supporting and improving care across the continuum [33] . Participating care settings were identified from the UPMC provider network based on interest in participating in a quality improvement initiative focused on improving uptake of RxWell, a dCBI developed by UPMC Health Plan. Partnering provider sites included both behavioral and physical health (PH) providers: 1 women's health center with integrated BH, 1 outpatient BH clinic, and 2 rural/urban primary care and family medicine practices. Over a 12-week period, the providers referred adult patients experiencing depression or anxiety to RxWell, while providing other indicated treatments (medication/therapy referrals) as part of usual care. Referrals were completed by the providers through the electronic medical record. The app was offered to patients at no additional cost. A series of interviews was conducted with both behavioral (eg, clinical psychologist) and physical (eg, PCP) health care providers across 4 health care settings at the conclusion of the initiative. This quality improvement project was approved by the UPMC Quality Improvement Review Committee.

As part of an IDFS, UPMC integrates more than 40 hospitals and 800 doctors' offices and outpatient sites. These connections supported a targeted approach to clinical site identification based on senior leadership support and site needs for BH support. Initial in-person visits to each care setting were conducted to learn more about provider needs and the capacity to participate as well as to share an overview of RxWell and the evaluation design/plan. Subsequently, all participating partner sites engaged in a facilitated workshop to receive instructional and educational training materials as well as generate and align on a revised clinical workflow that integrated RxWell. Collaborative decisions included the timing and manner of how the app would be introduced and referred to patients, the degree to which providers would follow up on patient progress, and the feedback loop between clinical, research, and coaching staff. Although feedback loop processes varied by site, communication modalities remained consistent (eg, email and phone calls). During these conversations, a gatekeeper/physician champion was identified at each site to support communication throughout the quality improvement project period.

Across all project sites, we engaged 56 providers to refer RxWell. During the project period, a total of 449 patients were referred to RxWell and 164 individuals enrolled in the app, 63 from BH providers and 101 from PH providers. Referring providers had access to a dashboard in the electronic medical record that enabled them to track individual patient progress, including technique completion, number of messages to the digital health coach, and anxiety and depression scores. In addition, the project team emailed aggregate reports to providers at a cadence based on site preference that detailed the number of app referrals and downloads by the provider/site, as well as de-identified patient-level engagement and anxiety/depression score data.

RxWell was developed from evidence-based approaches to BH: CBT and mindfulness. The app combines these approaches with the support of a digital health coach to deliver anxiety or depression programming to patients [34] . The dCBI was designed to be used on its own, as part of a stepped care model, or alongside face-to-face therapy to extend the reach of care, support skill practice, and monitor outcomes. Key app components include audio-and text-based techniques, a digital health coach, and assessment of anxiety/depression symptoms. Patients chose which program to enroll in, with guidance from their provider being available, and had access to RxWell for 6 months. The anxiety and depression programs include 53 and 40 techniques, respectively, and patients can complete up to 5 new techniques per day. See Table 1 for a sample of technique descriptions. In addition, RxWell use metrics are collected through a secure app database, including the number of referrals, number of patients enrolled, patient demographics (age, gender), technique completion, coach messages, and anxiety and depression scores measured by the Generalized Anxiety Disorder 7-item scale (GAD-7) [35] and the Patient Health Questionnaire 8-item scale (PHQ-8) [36] .

Digital health coaches, employed by UPMC, provide in-app support by guiding patients through setting goals, building intrinsic motivation, and recognizing successes. Coaches are college graduates, trained in CBT, mindfulness, and motivational interviewing and supervised by a licensed mental health care provider. All communication between patients and health coaches occurs via an in-app text-based messaging function, with health coaches replying to messages during typical business hours. See Figure 1 for an example interaction between a digital health coach and a patient. 

Helps the patient relax their muscles by focusing on specific muscle groups. Audio-based Feeling physically tense Grounding practice used during emotional distress. The technique asks the patient to observe an object, which they can carry throughout the day.

Helps the patient catch negative cognitive distortions, identify "hot" thoughts, and reframe thoughts for a more balanced alternative.

Focuses the patient on slowing down and paying attention to the present moment and teaches the patient to observe feelings and thoughts without any judgment or opinion.

Educates the patient on antecedents, behaviors, and consequences and teaches them to connect A, B, and C to one another.

Tunes the patient into their 5 senses. The patient chooses a sense to work with by identifying 1 thing they can do to engage that sense.

The patient imagines a pleasant scene that can have positive physical or mental benefits. Options include a beach, a forest, or a lake.

Educates the patient on sleep and sleep habits, the body clock, and the sleep drive. The patient is given choices for solving specific sleep problems.

Text-based Better sleep 

A convenience sampling approach, across all sites, was used to identify individual providers for participation. Two experienced interviewers (authors KW and BL) conducted semistructured telephone interviews with both BH and PH providers in April-July 2020 to understand facilitators of and challenges in integrating RxWell into their practice as well as their perceptions around sustainability. Interviews were conducted after the 12-week implementation period. Each interview lasted for approximately 30 min. The interview guide (see Multimedia Appendix 1) was developed with input from key stakeholders, including app product designers, a provider, and a qualitative expert.

Interview data were analyzed to identify major themes using an inductive content analysis approach. All interviews were audio-recorded and transcribed verbatim. Transcripts were then reviewed to gain a contextual understanding of the data. Inductive data saturation was assessed to understand whether new themes were described by providers that were not already described in the data [37] . Open coding was conducted on 7 transcripts to allow for initial categorization of relevant themes/concepts. Codes were further created and defined through an iterative process of reviewing 4 additional transcripts, thereby formulating the final global codebook. Dovetail coding software was used for the organization and coding of the data. Two trained independent coders (KW and BL) applied codes to all 19 transcripts. The coders met to review and adjudicate inconsistencies in code application.

A total of 19 interviews were conducted with health care providers (8 BH providers and 11 PH providers) to discover barriers to and motivators for referring RxWell to their patients. The providers were predominantly 30-49 years old, and most had less than 20 years of clinical practice experience ( Table 2) . Each interview lasted for approximately 30 min. Four primary thematic categories emerged: benefits and complexities of incorporating the app into the provider's treatment toolkit, factors that influence both provider and patient engagement, challenges and opportunities with current provider monitoring capabilities, and potential for sustaining app referrals in clinical settings. 

The providers discussed several factors that drive their use/referral of RxWell as a treatment option as well as factors that they believe support patient engagement in the app; often, these factors overlapped. The providers believed they are more apt to trust and refer a patient to RxWell because the content is rooted in a strong, evidence-based treatment theory: The providers offered insights into why patients may not engage or stay engaged with the app, including believing that patients needed to have a certain level of motivation and emotional capacity to engage with the app, had a busy lifestyle, or had a lower interest in obtaining support: "The big thing I heard was the motivation factor. Actually, there sometimes would be a desire to use it but they did not actually get over the hump and use it" (BH provider). Motivation was not seen as an app-specific problem, as some providers noted patient motivation can be a challenge with any form of BH treatment.

The providers also commonly mentioned patients forgetting to download the app or speculated that patients forgot to use it. Other barriers to engagement that the providers perceived included patients not owning or feeling comfortable using a smartphone, preferring in-person therapy, and finding provider and app reminders bothersome: "I think it is potentially a little harder for stereotypically older individuals [who] are less tech-savvy individuals who find their phone is merely a communication device rather than an all-in-one type of device" (PH provider).

The providers benefited from having a focused discussion on how to incorporate the app referral process into their existing workflow, including having access to instructional materials. A provider stated, "I very much like to talk things out with people and have those kinds of planning and brainstorming sessions collaboratively…I like being able to walk through the steps of that so I can understand it really well so I can appropriately explain it to someone else" (BH provider). Being able to engage in experiential learning (ie, using the app) was seen as an important experience for providers to partake in during implementation development "because I was able to use it some myself and talk to patients about it from a more personal experience, which is how I like to do therapy. You do such a better job explaining things if you have actually tried it yourself and can really speak to what you liked and did not like" (BH provider). Even providers who did not test the app due to time limitations acknowledged that doing so would have supported their ability to refer patients.

In terms of an addition to usual care services provided, BH and PH providers offered the app to patients for fundamentally different reasons. BH providers approached the app as a supplement to therapy or as a resource that would reinforce, to the patient, what they were already teaching them during in-person therapeutic sessions: "I say, this is a great way to kind of [have] somebody helping you to remember to do skills and that I think it could really add to what they are already getting" (BH provider). PH providers tended to offer the app as 1 of the treatment options available to the patient and often referred patients to the app and provided a medication prescription or a referral to in-person therapy: "I offer CBT on some level to most of my patients with anxiety and depression, and so it becomes an option whether they want to do that or not. Usually I talk about the option of CBT first, and if they are interested in that, then we kind of discuss whether face-to-face or [the app]" (PH provider).

Across provider types, patients were more likely to be referred if they were early in treatment, could use extra educational support, were resistant to other treatment modalities, or were waiting for an initial BH appointment: The providers also shared several reasons for why they would be hesitant or unlikely to refer certain patients to the app, including if the patient was older (eg concern over technology skillset), was happy with their current treatment, preferred in-person therapy, had severe symptoms (eg suicidality, escalating symptoms), or had conditions where the app did not align with treatment goals (eg dysregulation, socially avoidant behaviors): Almost all providers described how they typically introduce the app to patients as a treatment option. This initial conversation consisted of a brief overview of the app, often including specific details about the educational components, such as CBT, mindfulness, sleep, and in-the-moment relief options:

Most providers discussed referring and ordering the app for patients during or right after the appointment. BH providers often were able to provide this support: "So, they would get the text [with a link to download the app] and I would have them download it with me and then I would have them go through the process with me of signing up" (BH provider). PH providers were unable to support patients in downloading the app due to time constraints.

Monitoring patients after referral of the app was an important activity for the providers. Yet many providers were unaware of the in-depth monitoring capabilities (eg GAD-7 and PHQ-8 scores, number of techniques completed, number of messages sent to the digital health coach) within the electronic health record, even though this information was covered during their initial training sessions:

I usually follow up with people; I will have the office staff talk with them within a couple of weeks…so it would be a great additional option to be able to reference part of the chart and see if they were active with something like this. Anything that is objective like that is helpful. [PH provider] The providers commented on key monitoring features that they would like to access directly, including notifications of app progress or non-use, symptom severity alerts, and a population summary of all patients referred to the app, features not currently available in the electronic health record:

Further, even though all patients were assigned a digital health coach when they first accessed the app, the providers lacked clarity on the digital health coach role and on how coaches communicated with patients: "I think there was still some lack of clarity about that role exactly, and what that involves, and how people were utilizing that role" (BH provider). However, nearly all providers predicted that it would be helpful for patients to have access to a coach who could support them with tool navigation: "I think that, again, I think it is a great idea. Any kind of relationship like that, even if it is a digital one is certainly, again, another way to maybe connect with someone or just kind of supervise how they are doing, even if it is from afar" (PH provider).

The providers discussed their ability to see an impact on patients. Although most providers shared that they had not yet received feedback from patients or their patients had not used the app long enough for them to see any impact, there were some indications of early success. In addition, some providers believed the evidence-based principles behind the app would likely yield patient improvements, as traditional CBT has been shown to produce positive impacts: "Well, I do not really know, but I just know overall the skills I am teaching, with the anxiety especially, I see great improvement for the people who want to follow through and learn these things. And I just feel that [the app] will support that" (BH provider). Three providers reported hearing or noticing positive changes in patients and attributed improvements to the app as well as other co-occurring treatments:

…I had 1 young girl. It is interesting because initially when I saw her, and her symptoms were fairly mild and somewhat situational. She really found it was helpful in terms of just checking in with her mood. And then I called her first and she said, "Oh, I have done it some and I really feel good." We had started meds and then I just did a check in with her last week. And she said, "I have not really done it much, but I know it is there." She was transitioning and going back to college. She said, "When I get back, I am going to try to do it more." So, she thought it was helpful and she kind of used it as...In the old days when we used to have people who carry around their Xanax or Klonopin just in case. She sort of felt it was a tool that she had should her symptoms re-escalate.

Several providers wanted to understand how use of the app impacted their patients, and mentioned that knowing patient-level outcomes would likely increase their or their patients' engagement:

Almost all providers expressed enthusiasm and interest in continuing to refer patients to the app: I think it would be fantastic. I would love to be able to continue to use modalities like this that are kind of pushed to right where patients are looking to where patients would have the information wherever they are going. They will have it at home, and they will have it while they are waiting in line at a store and they could do it. I think that the downside is pretty low and it is actually, it would be very difficult for me to find significant downsides to it, especially whenever there [ 

The purpose of this paper was to understand providers' views on the feasibility and acceptability of delivering RxWell in multiple routine care settings. Overall, we found that providers valued access to RxWell for their patients, believed this tool may serve to overcome traditional barriers related to in-person therapy (ie, talk therapy resistance, time, scheduling, travel, and costs), and viewed having a trained digital health coach support patients in their use of the CBT-based digital app as an added value to care. The providers also shared some hesitancy around prescribing the app to patients with certain characteristics and offered some insights into how to improve patient monitoring.

Although there is an increasing interest among providers to prescribe dCBIs such as RxWell, existing research highlights several concerns and hesitancies among providers around integrating dCBIs into clinical practice, including a lack of integration with clinical workflows; low confidence in the validation and evidence behind app content; concerns over high cost, security, and privacy; and the importance of peer endorsement from other providers [18, 28, 38] . Some of these concerns can be mitigated through implementation strategies designed to better understand the care settings and contexts [30, 39] . Examples of effective strategies to address known challenges include understanding the needs of the target setting and population, engaging relevant stakeholders early in the adaptation and planning process, and obtaining feedback to support an iterative approach [39, 40] . As part of this paper, we focused on cocreating workflows through facilitated workshops that served to establish feedback loops between the project team and providers, provide instructional and educational training materials, and identify site-level clinical champions. Our results support these efforts to overcome challenges cited in the literature, as providers believed that engaging in experiential learning (eg, using the app) and having a focused discussion and access to instructional materials are beneficial. They also highlighted the importance of knowing the app content is rooted in evidence and discussing how to incorporate the referral process into their workflow.

Research has also suggested that implementing dCBIs within clinical settings can be better facilitated by establishing organizational support, increasing education and awareness around the apps, and integrating the apps into clinical workflows, such as embedded referrals in the electronic medical record [18, 38] . The median visit length for a primary care visit that covers about 6 topics is 15 min [41] . Given these time constraints, it is important to integrate digital app prescription within the provider workflow, and the interviewed providers endorsed the ability to deliver patients RxWell through a 2-click workflow in the electronic medical record and adoption of this digital tool into their daily workflow. However, despite the streamlined workflow, some providers were unable to support patients in downloading the app due to time constraints. Further, the providers discussed not feeling sufficiently educated about how they could monitor user progress within the app and were interested in having a population view of patients they referred, notifications in the electronic medical record related to use or milestones, and patient-level outcome data to support care delivery.

Finally, the providers identified specific factors that influence their decision to refer, including perceptions of the app not working well for patients who are older or patients with severe behavioral symptoms. They were more likely to refer patients who were early in treatment, needed extra educational support, were resistant to other treatment modalities, or were waiting for an initial BH appointment. Although the literature is limited in terms of factors influencing referrals to a dCBI, other studies exploring the decision-making process among providers referring patients to BH solutions have identified comfort with patient diagnoses, the level of familiarity with other BH resources, and patient characteristics as important facilitators of how and when a provider refers a patient to care [41, 42] .

By leveraging the unique position and commitment of an integrated payer-provider health system and its quality improvement efforts to enhance patient and provider experience, scale digital interventions, and ultimately improve health [33] , we were able to illuminate key barriers and facilitators from the perspective of providers regarding the referral and use of RxWell. Results from this study will serve as an important early step toward supporting and sustaining provider engagement in digital health. Given our finding that providers are hesitant to refer some patients (ie, patients who are older, are happy with their current treatment, prefer other treatment modalities, or have severe symptoms), future efforts are needed to explore how providers can support both initial and sustained engagement across different patient populations. These efforts may also include strategies such as generating a data-driven list of situations or patient characteristics (eg, geographical locale, technology skill level, severity diagnosis) for which RxWell would be most appropriate, learning more about older adult populations' challenges in accessing and using digital tools, and continuing to share supportive materials for providers to use as a reference when introducing the app to patients.

The providers also expressed wanting to know more patient-level outcomes among patients they did refer and cited limitations in their capacity to monitor patients over time. Knowing whether a patient has downloaded and used the app, and for how long, offers feedback to the provider about the patient's engagement in the treatment modality, while information about BH assessment outcomes signals program impact. The ability to scale dCBIs depends upon efforts to make already existing and integrated monitoring functions even more streamlined for providers who have limited time and resources. It is critical to design clinical workflows and referral pathways that meet the needs of the clinical setting and environmental circumstances (ie, . Future efforts are needed to explore workflows that provide better visibility into patient-level outcomes, which may increase provider engagement and provide insight into how digital health coaches can further support treatment and therapeutic alliances. Additionally, qualitative evaluations focused on patient and other key stakeholder perspectives are needed to provide a more comprehensive picture of the barriers to and facilitators of successful digital app engagement.

There are several study limitations that are important to acknowledge. Based on our convenience sampling method, results may have limited generalizability due to the nonrepresentative sample. Further, self-selection bias may have influenced the findings and themes, as providers who participated in the interviews may have different perspectives on their experiences referring patients to RxWell than providers who did not respond to the interview invitation. Further, although several interviewee demographics were presented (gender, age, provider type, and years in practice), race/ethnicity data were not collected from interviewees. Finally, the COVID-19 pandemic began amid the study period, which may have impacted provider use of the digital app and uniquely influenced their perceptions. Qualitative interviews illuminated implementation adaptations that providers made during this time, and such adaptations often facilitated the providers' ability to continue referrals during remote patient visits.

Our results strongly suggest that providers are supportive of digital tool integration and endorse the use of dCBIs within their workflow. The ability to scale digital interventions to meet the mental health needs of patients relies on streamlined workflows that enable BH and PH providers to easily refer patients to evidence-based interventions. To further enhance provider acceptance of dCBIs, more information is needed about which patients benefit most from such digital tools. Finally, although this paper focuses on factors influencing provider engagement, it is important to gather the perspectives of other critical stakeholders to ensure broad uptake and use of digital interventions that support the BH needs of patients.

Key Substance Use and Mental Health Indicators in the United States: Results from the 2016 National Survey on Drug Use and Health (HHS Publication No. SMA 17-5044, NSDUH Series H-52)

Substance Abuse and Mental Health Services Administration

The implications of COVID-19 for mental health and substance use.: Kaiser Family Foundation

Anxiety and depression: optimizing treatments

Types of Therapy: Helpful Guide to Different Therapy Options

The effects of cognitive and behavioural therapies for anxiety disorders on depression: a meta-analysis

A meta-analysis of nonrandomized effectiveness studies on outpatient cognitive behavioral therapy for adult anxiety disorders

Perceived barriers and facilitators to mental health help-seeking in young people: a systematic review

The impact of mental illness stigma on seeking and participating in mental health care

Access and cost barriers to mental health care, by insurance status

Key Barriers Limiting Patient Access to Mental Healthcare

Global health workforce labor market projections for 2030. Hum Resour Health

Availability of outpatient care from psychiatrists: a simulated-patient study in three U.S. cities

Beyond parity: primary care physicians' perspectives on access to mental health care

The rise of primary care physicians in the provision of US mental health care

By the numbers: ratings and utilization of behavioral health mobile applications

Beyond validation: getting health apps into clinical practice

Q3 2020: A New Annual Record for Digital Health (Already)

How mental health care should change as a consequence of the COVID-19 pandemic

Towards a consensus around standards for smartphone apps and digital mental health

Mobile mental health: review of the emerging field and proof of concept study

Smartphones for smarter delivery of mental health programs: a systematic review

The efficacy of smartphone-based mental health interventions for depressive symptoms: a meta-analysis of randomized controlled trials

Can smartphone mental health interventions reduce symptoms of anxiety? A meta-analysis of randomized controlled trials

Developing and evaluating digital interventions to promote behavior change in health and health care: recommendations resulting from an international workshop

Delivering the Benefits of Digital Health Care. London: Nuffield Trust

The role of health-care providers in mHealth adoption

Digital Tools Can Help Patients, but Prescribing the Right App Is No Simple Feat

Three problems with current digital mental health research . . . and three things we can do about them

Factors that influence the implementation of e-health: a systematic review of systematic reviews (an update)

Achieving change in primary care-causes of the evidence to practice gap: systematic reviews of reviews

UPMC's population health management strategy: a road map to high-value health care

A study protocol for a non-randomised comparison trial evaluating the feasibility and effectiveness of a mobile cognitive-behavioural programme with integrated coaching for anxious adults in primary care

A brief measure for assessing generalized anxiety disorder: the GAD-7

Comparison of the usefulness of the PHQ-8 and PHQ-9 for screening for major depressive disorder: analysis of psychiatric outpatient data

Provider-and patient-related barriers to and facilitators of digital health technology adoption for hypertension management: scoping review. JMIR Cardio

Saturation in qualitative research: exploring its conceptualization and operationalization

Developing an implementation strategy for a digital health intervention: an example in routine healthcare

Effective implementation of change in healthcare: a systematic approach

Time allocation in primary care office visits

When behavioral health concerns present in pediatric primary care: factors influencing provider decision-making

Barriers and facilitators for referrals of primary care patients to blended internet-based psychotherapy for depression: mixed methods study of general practitioners' views. JMIR Ment Health

), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information

We thank all provider sites, clinical leadership, and interviewees for their collaboration and input into each stage of this quality improvement project. We also want to acknowledge Nicole Beswick, MS, NBC-HWC, for her support in conducting this study.

VS, BL, CS, KW, JK, and CN are employees of the UPMC Insurance Services Division; author ES is a consultant of UPMC Health Plan.

Provider interview guide. [