key: cord-0907781-nnvren4t authors: Parisi, Kathryn E.; Dopp, Alex R.; Quetsch, Lauren B. title: Practitioner use of and attitudes towards videoconferencing for the delivery of evidence-based telemental health interventions: A mixed methods study date: 2021-10-15 journal: Internet Interv DOI: 10.1016/j.invent.2021.100470 sha: 572692b0a49771312dc54cda076ba95f9f893a99 doc_id: 907781 cord_uid: nnvren4t The implementation of evidence-based psychosocial interventions using video-conference delivery (VCD) has the potential to increase accessibility to effective treatments, although its use remains limited and understudied. This study employed a mixed methods approach in surveying mental health practitioners about their attitudes regarding VCD of interventions that are considered evidence-based (i.e., have been shown to improve targeted outcomes in rigorous research). One hundred and eleven practitioners were sampled from several national and regional U.S. practice organizations and were administered quantitative surveys about their use of and attitudes towards VCD of evidence-based interventions (EBI). We examined the relationship between practitioner-level technology access, experience, and training with technology fluency and acceptability of using VCD. Quantitative results indicated the most frequently used adaptation for VCD was Tailoring and that practitioner education predicted attitudes towards EBIs. A subset (n = 20) of respondents were then purposively selected for qualitative interviews to further investigate accessibility, appropriateness, and feasibility of delivering EBIs via video conference. A conventional content analysis revealed that VCD was appropriate and acceptable for EBIs; however, many practitioners also described barriers related to feasibility of implementation. The results of this study have important implications for telemental health dissemination efforts which seek to extend services to populations not served well by traditional, in-person mental health services. The use of video-conference technologies to deliver evidence-based interventions (EBIs) has gained traction in recent years, touted for its potential to reach a broader range of individuals (e.g., rural underserved communities, communities with limited numbers of practitioners; Myers, 2019) than with traditional, clinic-based methods (Langarizadeh et al., 2017; Shigekawa et al., 2018) . Indeed, real-time, remote-delivery of EBIs even has the potential to match outcomes seen in outpatient, inperson approaches to treatment delivery (Pruitt et al., 2014; Hilty et al., 2013; Berryhill et al., 2019) . Reviews of EBIs delivered via videoconference have found results supporting the effectiveness of these interventions, whether implemented in university training sites or community clinics. Interventions for internalizing disorders appear to have the highest efficacy (Spates et al., 2016; Zhao et al., 2017) , including those for adolescents (Reyes-Portillo et al., 2014) as well as caregivers and families (Chi and Demiris, 2015) . Although rates of telemental health are increasing even more rapidly now in response to the COVID-19 pandemic (Wind et al., 2020) , research exploring factors influencing successful video-conference delivery (VCD) for EBIs is lacking. As end-users of telemental health tools, practitioner attitudes and interests towards the implementation of EBIs via VCD are important to consider when planning implementation and dissemination efforts. Mixed (i.e., qualitative and quantitative) method studies have unveiled the nuances and challenges of molding and formatting existing EBIs into programs for VCD (Caffery et al., 2017) ; however, these studies have also revealed how much is still unknown about the role of practitioners in VCD EBI implementation. While telemental health includes videoconferencing, online and mobile application-based self-help resources, tele-assessment, and psychiatric interventions (Myers and Turvey, 2012; Rice et al., 2020) , the current study will only focus on the use of evidenced-based, psychosocial interventions for mental health care delivered via (real-time) videoconferencing. This method offers unique advantages, as it most closely approximates in-person services than other methods; however, it also has unique challenges. Practitioner utilization of telemental health boasts efficacious outcomes and positive practitioner attitudes (prior to the COVID-19 crisis), however, growth in the use of telemental health has not been sufficient to reduce disparities (Patel et al., 2020) . For practitioners in private practice, additional challenges may impede their ability to implement EBIs through telemental health such as lack of access to technology and beliefs that using computers or technology could interfere in therapy (e.g., rapport, treatment implementation, increased practitioner workload) thus negatively affecting client outcomes (Becker and Jensen-Doss, 2013; Schueller et al., 2016) . In fact, fewer practical barriers have predicted more positive practitioner attitudes of VCD. Also, practitioners often prefer in-person contact over VCD. Prior training and current practice environment (e.g., private practice, academic medical center, university or community clinic) seem to shape VCD use more than other factors like practitioner theoretical orientation (Becker and Jensen-Doss, 2013 ); yet, these relationships need to be explored further. Availability and access to equipment may change rapidly in healthcare settings with the advent of new technologies. Still, additional research is needed that focuses on EBIs and examines the acceptability of utilizing VCD among community practitioners, particularly among practitioners who provide services to rural and minority populations as well as youth and families. Because VCD requires a higher level of practitioner involvement than with other telemental health methods (e.g., computer-based, practitioner-assisted interventions), practitioners' views on use of, accessibility to, and feasibility of telemental health are especially important to consider. The current investigation focused on VCD of psychosocial EBIs including whether practitioners make modifications to existing treatment models when implementing EBIs in a new context (i.e., VCD). Fidelity refers to the ability of practitioners to maintain consistent adherence and quality when delivering a treatment protocol or manual so that the intended outcome of the intervention can be reliably achieved (Bellg et al., 2004; Moncher and Prinz, 1991) . High fidelity to treatment models has been associated with increased positive outcomes such as preserving the mechanisms of behavior change associated with interventions (Henggeler et al., 1997; Miller and Rollnick, 2014) . In contrast, some research has also shown that therapists' adherence to a treatment protocol does not predict the outcomes of psychotherapy (Webb et al., 2010) and perhaps a rigid emphasis on fidelity is not of paramount importance for large-scale dissemination (Kendall and Frank, 2018) . When implementing treatments within the community, it is more practical or feasible to expect an interventionist will maintain minimum adequate levels of fidelity while also making context-specific adaptations (Stirman et al., 2013) . Therefore, some degree of modification is to be expected (Chu and Leino, 2017; Cook et al., 2014) . When considering telehealth options, adaptations to treatment models may be expected and beneficial (especially when originally developed for inperson delivery), but little is known about the frequency and types of adaptations. To gather this type of explorative information, mixed methods (including broad surveys and in-depth interviews) are useful. The current study thus aimed to fill this gap in the existing literature on the balance between fidelity and adaptations in the context of VCD. Thus, the current study seeks to specifically explore modification use. Previous research which informs the current study hypotheses has found that for practitioners that used telemental health services more frequently, training in the platform, years of experience, and perceived ease of use of technology have been among the top predictors of use (Simms et al., 2011) . The aims of the current study were as follows: 1) examine the frequency and predictors (e.g., demographics such as training, years of experience, technology access, practice setting, and orientation) of attitudes towards the use of VCD for EBIs; 2) examine the frequency and predictors (e.g., attitudes towards evidence-based practice; computer fluency, comfort, and efficacy) of use of VCD for EBIs, including any adaptations that have been made to the EBI to deliver it via video-conference; and 3) explore practitioner perspectives regarding VCD of EBIs in their professional experiences. To achieve these varied research aims, we utilized both quantitative and qualitative data (i.e., mixed methods). Specifically, we collected a larger quantitative sample (QUAN) before selecting a smaller subset of participants for qualitative interviews (Qual). The QUAN ➔ Qual data was finally integrated to provide a deeply detailed response to our exploratory question about practitioners' experiences with VCD of EBIs. The primary hypotheses therefore include: 1a) having access to technology and training in telemental health strategies will correlate with increased technology fluency and increased acceptability of conducting therapy via telecommunication, 1b) increased practitioner years of experience will correlate with decreased technology fluency and decreased acceptability of conducting therapy via VCD; 2) qualitative interview themes related to adaptation of EBIs for VCD will inform understanding of the frequency of adaptations reported by the quantitative survey, and 3) practitioners will have diverse and varied responses about how they are approaching the delivery of EBIs via VCD. The current investigation addressed the research aims in two major steps. First, by developing an exploratory survey, quantitative data about adoption of and barriers to implementing EBIs via VCD was collected. Following the initial review of responses, a subset of respondents was recruited for qualitative interviews. Institutional Review Board approval was granted for the project as well as internal funding for participant compensation. To obtain a diverse group of practitioners, recruitment used both purposive and snowball sampling methods (Teddlie and Yu, 2007) . Purposively, practitioners (i.e., clinical and counseling psychologists, clinical social workers, marriage and family practitioners) were identified through various national mental health professional organizations in the United States (as described more under Procedures). These organizations varied in their focus (i.e., treatment modality, disorderspecific) to capture a broad, nationally representative sample reflective of the different foci of therapists in the United States in general. This included a diverse representation of theoretical orientations (e.g., cognitive-behavioral, psychodynamic, humanistic). Participants from a wide variety of professional backgrounds (e.g., psychology, social work, counseling) and levels of training (e.g., master's, PsyD, PhD) were recruited. We required participants to self-report that they were mental health practitioners who had experience with VCD. Study eligibility criteria included: a) being a licensed mental health practitioner who has completed their graduate education and b) having used both EBIs and VCD for service delivery in the past 12 months. Upon completion of the survey portion of the study, participants were asked to nominate up to five colleagues who might be interested in participating. These individuals were then recruited for the study (i.e., snowball sampling). Fifty-two participants agreed to be contacted for a follow-up interview (46% of entire sample), of which twenty practitioners (38% of those consenting to be interviewed) were selected via stratified sampling to reflect the demographic makeup of the quantitative sample and invited to participate in qualitative interviews. In other words, recruitment of the qualitative sample focused on obtaining a comparable number of masters-level therapists as those who completed the quantitative survey. Purposive recruitment began with an email advertisement with the link to the quantitative survey; this email was sent to the Listservs of several U.S. national professional practice organizations (e.g., Association for Behavioral and Cognitive Therapies). Moreover, private practitioners and agencies through Psychology Today's practitioner directory were identified by state, filtered by "Video Counseling" specialty, and contacted if their email information was available. In this manner, the sample includes practitioners from 40 different states across all regions of the US including Alaska and Hawaii. The authors also advertised the study at an international conference and on social media (e.g., Twitter). Individuals who followed the link could provide electronic informed consent, attest that they met inclusion criteria, complete the survey (modal time survey was open = 14 min; range = 5 min -8 h, SD = 1 h), and indicate whether they were willing to be contacted for a follow-up interview. One-hundred ninety-seven individual IP addresses were logged as having navigated to the survey, of which 113 respondents completed all measures in the study (completion rate = 57%). Responses were collected via secure, anonymous data collection software (i.e., Qualtrics). Interview participants were then randomly selected from those survey respondents who agreed to be contacted, with random selection stratified by demographic characteristics including gender, race, geographic region, and degree level (e.g., Masters versus Doctorate). Selected participants were contacted to arrange a remote interview via secure web conferencing software (i.e., Cisco WebEx). Verbal informed consent from each participant was obtained before completing the interview (M interview length = 31 min; SD = 10.33). Interviews were audio-recorded, transcribed, reviewed, coded, and later analyzed by the first author and a team of trained research assistants. Access. A demographic questionnaire from another research study (i.e., Becker and Jensen-Doss, 2013) was modified for the current investigation to evaluate predictors (e.g., training, orientation, practice setting, years of experience) of practitioner access to and use of different kinds of VCD technology. Practitioners were asked (yes/no) about the availability of support and resources necessary for telemental health service delivery and their frequency of use of telemental health strategies (i.e., voice calls, text messaging, email, VCD) on a five-point Likert scale (0 = never to 4 = always). Computer-Email-Web Fluency Scale (CEW). The 21-item CEW (Bunz, 2004) was developed to assess an individual's ability to perform basic computing functions necessary for checking email and using an internet browser. Individuals were asked dichotomous (yes/no) questions about whether they could carry out various activities using a computer, computer applications, and the internet. Internal consistency of the CEW is good (Cronbach's α = 0.82-0.89) for the four subscales (i. e., Computer fluency, Email fluency, Web navigation, Web editing). Factor analysis of the CEW confirmed the 4-factor structure contributed over 67% of variance in scores (Bunz, 2004) . We added another question asking practitioners about their fluency with using a web-camera. Reliability analyses for the current study were sufficient (α = 0.83). Computer-Assisted Therapy Attitudes Scale (CATAS). The CATAS (Becker and Jensen-Doss, 2013 ) is an 8-item self-report questionnaire which has a two-factor structure measuring a) Efficacy, or belief in selfefficacy with technology; and b) Comfort with using technology. In the original study in which the CATAS was developed, fair to good internal consistency was demonstrated by the Comfort subscale (α = 0.59) and Efficacy subscale (α = 0.84), respectively. In the present study, minor modifications to the questionnaire items were made to make them appropriate for professionals already engaged in VCD. Reliability analyses for the current study revealed α = 0.79 for the overall scale, α = 0.70 for the Comfort subscale, and α = 0.57 for the Efficacy subscale. Evidence-Based Practice Attitudes Scale (EBPAS). The 15-item Evidence-Based Practice Attitudes Scale (EBPAS; Aarons, 2004) was developed to assess practitioners' attitudes towards evidenced-based practices on a five-point Likert scale from 0 (not at all) to 4 (to a very great extent). It produces four subscales: Requirements (practitioner use of new intervention if it was an agency or state requirement), Appeal (practitioner use of new treatment if makes intuitive sense or if had colleague approval), Openness (practitioner willingness to use new treatment), and Divergence (extent practitioner does not find EBIs important/useful). Ecological validity of the EBPAS has been supported by the diverse disciplinary (e.g., social work, psychology) and educational backgrounds (e.g., some college/graduate work, PhD/MD) of practitioners in the original sample (Aarons, 2004) . The factor structure of these subscales and the overall EBPAS factor structure support its reliability (Aarons et al., 2007) . Reliability for the current sample was acceptable (Requirements: α = 0.94; Appeal: α = 0.77; Openness: α = 0.77; and Divergence: α = 0.67.) Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM), and Feasibility of Intervention Measure (FIM). This collection of three pragmatic measures ask about an intervention's relative acceptability, appropriateness, and feasibility and can be tailored to specific interventions (Weiner et al., 2017) . Higher scores indicate greater acceptability, appropriateness, and feasibility. VCD of EBIs was inserted for the intervention in these measures and psychometric properties were examined and reported for this modification (Weiner et al., 2017) . Reliability analyses for the three measures in the current study were high (AIM: α = 0.94; IAM: α = 0.95; FIM: α = 0.90). Survey on VCD Use. An exploratory questionnaire was developed to survey practitioner delivery of EBIs via VCD. This survey aimed to understand the frequency and mode of VCD use, the type of EBIs used via VCD and for what percentage of their client base, and frequency and strategies used to adapt these EBIs for VCD delivery. The list of VCD platforms was adapted from a national telepractice survey (Behl and Kahn, 2015) ; the list of adult-focused EBIs was obtained from APA Division 12 (Society of Clinical Psychology, 2016); the list of youthfocused EBIs was obtained from a series of "Evidence Base Reviews" and "Evidence Base Updates" published in the Journal of Clinical Child and Adolescent Psychology (see Southam-Gerow and Prinstein, 2014 for an overview of this publication series); and the list of adaptation strategies was taken from an existing framework and coding system (Stirman et al., 2013 (Stirman et al., , 2017 . To understand broad acceptability, practitioners were also asked to rate their satisfaction with VCD of EBIs on a 5-point Likert scale from 0 (not at all) to 4 (I am very satisfied with it). Qualitative interview. Questions in the qualitative interview were designed to elicit practitioner perspectives on the acceptability, appropriateness, and feasibility (Weiner et al., 2017) of use of VCD and delivery of EBIs via those platforms. The interview guide began with background questions (e.g., educational background, current professional position) and a "grand tour" question about experience with VCD for EBIs in general. Subsequent questions addressed acceptability (e.g., "What are the advantages of video-conferencing? The disadvantages?"), appropriateness (e.g., "What adaptations or modifications have you made to EBIs when delivering them via VDC and how does that compare to in-person delivery?"), and feasibility (e.g., "How easy is it to use video-conferencing for treatment? What has been most helpful?") of VCD for EBIs. Finally, practitioners were asked an open-ended question about what would be helpful for implementing telemental health in future efforts. Linear regressions were used to examine the predictors of attitudes towards VCD of EBIs. Variables included previous training, years of experience, EBPAS, CATAS, and CEW. Results from added questions about videoconferencing were examined separately to maintain the psychometric integrity of the CEW. Descriptive statistics were utilized to determine the frequency of use of VCD for EBIs, including adaptations that were made to the EBI to deliver it via videoconference technology. Correlations between the frequency of use and adaptations with the subscales of the CATAS and CEW were examined. The relationship between practitioner demographic characteristics (i.e., level of training, years of experience, orientation, practice setting), their technological fluency, and their acceptability of conducting therapy via VCD was analyzed using a series of regressions. Analyses were performed with SPSS software, Version 24.0 (IBM Corp., Armonk, NY). Statistical significance was defined as p < .05 with appropriate controls for experiment-wise error rate (e.g., Bonferroni correction). A priori power analysis for the quantitative measures was conducted using G*Power Version 3.0.10 (Erdfelder et al., 1996) . A point biserial correlation revealed that 111 subjects would be needed to achieve a medium effect (r = 0.3) with power to detect an effect of β = 0.95. We analyzed interview responses using a conventional content analysis coding method (Hsieh and Shannon, 2005) , in which emergent themes (consistent patterns in content of responses to interview questions across participants) were identified from interview transcripts through an immersive review process. The first author systematically grouped responses into various consistent themes about use and adaptation of EBIs for VCD. Secondary independent coding was not available for the current study. The research team justified proceeding with an individual coder for several reasons. First, coding was restricted to a conventional content analysis of broad themes within a pre-selected framework, which made the codes relatively straightforward for a single interviewer. Secondly, all interviews were conducted and analyzed by the same researcher, which helped to make the subsequent analysis relatively straightforward. Finally, coding decisions and themes were discussed as a research team to achieve clarity and consensus. Three major categories of qualitative themes were deductively applied from Weiner et al.'s (2017) framework: acceptability, appropriateness, and feasibility. Originally defined by Proctor et al. (2011) , acceptability, appropriateness, and feasibility have been conceptualized as constructs describing how well an EBI fits or matches with practitioners' preferences, needs, and capabilities (respectively). Inductive coding was then applied to identify emergent subthemes under the three major themes. Coding of interviews was completed using NVivo 12 software. The coding manual was developed iteratively before initial coding. After finalizing the coding scheme, the first author developed narrative summaries of the information. As a validity check, we then solicited feedback from a subset of participants (20%, n = 4) who each reviewed and gave us comments on the accuracy, comprehensiveness, and credibility of the summaries. Based on their feedback, the summaries were refined and clarified. In reviewing summaries, we noted that coding had reached saturation; meaning themes were recurring and it appeared unlikely that additional participants would expand on those themes. Table 1 summarizes participant demographic data. Surveyed practitioners (N = 113) were primarily female (74%) and White (76%; Multiethnic = 9%; Black = 7%; Hispanic/Latinx = 6%), averaged 41 years of age (SD = 10.64), held a master's degree (65%) or a doctoral degree (35%), and identified their professional discipline as clinical psychology (39%; vs. counseling psychology = 28%; social work = 21%; substance abuse/mental health counseling = 10%; marriage and family therapy = 9%). Most participants worked in private practice (71%), followed by academic medical centers (10%), then university/collegebased clinics (4%), Veterans Affairs medical centers (4%), or other setting (11%). These practice settings were primarily located in urban areas (46%), suburban areas (35%), or rural areas (18%). The majority of telemental health clients were located in rural areas (40%) but suburban (30%) and urban (30%) were also common. Nearly half (49%) of participants identified their theoretical orientation as cognitive behavioral, followed by eclectic (15%), behavioral (10%), family/systems (9%), and humanistic/interpersonal (8%). Notes. EBI = evidence-based intervention, VCD = video-conference delivery. Table 3 presents statistics on the relations between demographic predictors and survey responses. Divergence on the EBPAS was significantly predicted by education level (β = − 0.307, p = .005), such that more education (i.e., a doctoral degree) was related to lower Divergence. All other relations were not significant. Use of VCD. On average, practitioners reported using VCD for 16% of their client base; however, VCD use was not related to the frequency of adaptations, computer fluency, or computer self-efficacy (ps > 0.05). Overall, practitioners reported frequent EBI use in both face-to-face sessions and when using VCD (85% and 86% of respective sessions). Practitioners reported using a diverse range of EBIs via VCD (most frequent: Individual Cognitive Behavior Therapy = 38% of participants reporting; see Table 4 ). As a measure of broad acceptability, participants indicated their satisfaction with VCD of EBIs with the modal response, "I am satisfied with it" (n = 37; 40%) followed by an equal number (ns = 27 each; 29%) of respondents reporting that they were "very satisfied with it" and "somewhat satisfied with it." Adaptations. Table 5 provides summary statistics for type and frequency of EBI modification. Practitioners reported modifying EBIs during VCD at a rate of 30% of overall sessions, although there was considerable variability among responses (SD = 33%). The most Note. EBIs = Evidence-Based Intervention. Note. EBIs = Evidence-Based Intervention. CBT = Cognitive Behavior Therapy. PCIT = Parent-Child Interaction Therapy, EMDR = Eye Movement Desensitization and Reprocessing, TF-CBT = Trauma-Focused CBT. Notes. EBI = evidence-based intervention, VCD = video-conference delivery. K.E. Parisi et al. frequently used modification reported by practitioners was Tailoring (M = 1.95, SD = 1.10), which involves any minor change to the intervention that leaves all of the major intervention principles and techniques intact while making the intervention more appropriate, applicable, or acceptable (Stirman et al., 2013; Stirman et al., 2017) . The next most frequent adaptations made to EBIs to facilitate VCD across sessions were: Integrating with another approach ( Using an EBI in VCD was associated with general EBI use (r = 0.66, p < .001) and with the frequency of using specific modifications, including Removing elements (r = − 0.20, p = .05) and Shortening/ Condensing parts of the treatment protocol (r = − 0.22, p = .04). Modifications were associated with increased webcam fluency (r = 0.21, p = .04) as well. Increased frequency of delivering an EBI, both in person (r = 0.39) and via VCD (r = 0.40), were significantly related to comfort with using technology in treatment (ps < 0.001). A total of 20 practitioners were recruited to report on their personal experiences using EBIs via VCD. Interview participants were 25% male (n = 5), 50% held a master's degree while the other half held a doctoral degree (n = 10), and 25% (n = 5) held primarily academic positions while the rest were primarily clinical practitioners. In terms of theoretical orientation, 40% identified as cognitive-behavioral (n = 8), 20% (n = 4) identified as behavioral, 20% (n = 4) as systems, and the rest (n = 4) identified as having other primary theoretical orientations including gestalt, person-centered feminist, clinical mindfulness, and psychodynamic. Most interview participants (70%) worked in private practice (n = 14); others were distributed across outpatient or university clinics or veteran's affairs (VA) medical centers. A conventional content analysis revealed six major themes: appropriateness, feasibility, facilitators, acceptability, use, and barriers. See Table 6 for example quotes for each theme and major subtheme. The most discussed theme among interview participants was appropriateness, which participants described as compatibility of VCD for evidence-based practice and the ability of this format to adequately address client needs. An equal number of references to the appropriateness and the lack of appropriateness of VCD for EBIs were made during interviews. Describing its appropriateness, many participants described how similar VCD is to traditional in-person treatment, stating that they rarely needed to modify treatment for video format. Participants also described how they were able to learn about VCD, whether by receiving training at their workplace or by engaging in online platforms (e.g., communities, informational pages) to read and discuss strategies with others. Additional subthemes frequently referenced were the ability of VCD to address traditional barriers, appropriate for children and adolescents, comparable course or outcomes, and comparable rapport. For example, practitioners frequently stated that VCD addressed lack of access and disparities for those who have been traditionally underserved (e.g., rural clients for whom transportation is a barrier to accessing services). The most common subtheme within the reference of "is not appropriate" was the concern of engagement challenges. The second most frequently discussed theme was feasibility, under which three subthemes emerged: references to when VCD is feasible or is not feasible (with the former being much more common), and resources to enhance feasibility. Feasibility was defined as the ease of implementing EBIs via VCD. Lack of feasibility included administrative barriers and difficulties billing for telehealth. Participants referenced the feasibility of VCD with children and adolescents, and to how it is often more advantageous to use telehealth over in-person treatment because attendance is easier via telehealth. In terms of work with adolescents, participants referred to the match of technology with the younger generation. One of the most frequently discussed tools to enhance feasibility included reliable internet service for both clients and practitioners. Most practitioners were successful without additional support; however, participants also spoke about having dedicated support staff, both administrative and technical, to facilitate VCD practice. Lack of feasibility was less frequently discussed and usually involved factors on the client side, such as unreliable internet service, but equipment was not reported as a barrier since clients frequently used their smartphones to participate in therapy. The third most discussed theme was facilitators, which is defined as things that increase VCD use or make it easier to implement. Seven subthemes related to Facilitators emerged: techniques, adaptations, technology, practitioner interest, collaborators, and supervision. Examples of techniques include things that practitioners have learned through experience, not leaning into the camera or gesturing too much with their hands. Participants provided creative resources which they Table 6 Coding nodes and corresponding example quotes from interviews. Example of quote for code Acceptability Yesterday I was doing telehealth on three continents…I'm a big fan. The other thing I kind of like about telehealth is you can get an idea, especially if you're doing home-based telehealth, you can get an idea of how people are living which provides a lot of information. Is not acceptable I think as a practitioner I just still-it's hard when you're not seeing someone and they just stop answering their phone and, stop responding to you and you never met them in person. It just feels like it's harder sometimes to track people down or be certain that they're safe and doing okay. Appropriateness Patient alliance I think is similar [to in-person treatment] as well in that how they feel towards me. We're seeing kids in some of these extremely rural areas that have-their trauma may have been ten years ago, and they've never seen a practitioner. Not for lack of needing one, but just lack of access. And so that's been really nice to have that ability to kind of address those disparities. Is not appropriate Individuals will show up to session, like without a shirt on orcause they're very comfortable because they're in their homeor drinking or drunk or high or smoking marijuana or whatever. Individuals will be drunk and waving around a gun and there's really nothing you can do. Dealing with [the trauma] narrative, going through lots of written work and a lot of that was contingent on whether or not I received their homework via the person at the front desk who was supposed to fax it over and if they didn't give it to me then I just had to have them read it to me. Facilitators I do spend a good bit of time on the front end kind of trouble shooting with all of my patients on like this is what happens like if the video doesn't show up, this is how you-, let's do this together, let's figure out some backups to make sure we can work it out together. And I think that helps in the long run because they feel more comfortable with the equipment. Feasibility It takes prep work but it's certainly doable. Is feasible I think that [telehealth] was pretty standard and really well accompanied. We did a good job by augmenting prolonged exposure with the VA [Veteran Affairs] PE coach. We don't have great ways of billing yet for these services. Use It's almost weird to see a patient not through telehealth for me just because I do telehealth so much. Note. EBIs = Evidence-Based Intervention. K. E. Parisi et al. use to enhance feasibility: dedicated staff members for technical support, "chat bots," and the use of headphones and speakers to improve reception. Many interview participants discussed digital facilitators such as the creation of phone applications for homework and the use of computer programs to play games with their child and adolescent clients. Many spoke about being able to provide services when they were across states lines, which helped when they relocated because they did not have to build up a new client base. Participants shared that staff such as administrators, research coordinators, and school nurses (present at the client's location) were critical to the facilitation of VCD implementation. A less commonly discussed theme included acceptability, which is the perception among implementation stakeholders that a given treatment, service, practice, or innovation is agreeable, palatable, or satisfactory (Weiner et al., 2017) . Relatively equal proportions of references were made to acceptability versus lack of acceptability of VCD for EBIs. Acceptability was reflected by frequent use among some participants who said they were using VCD exclusively. Lack of acceptability was underscored by "practitioner frustration;" yet, it should be noted that statements such as these were infrequent. Several participants discussed feeling overwhelmed or annoyed with technical difficulties and lack of support to resolve issues. Use was also a less commonly discussed theme, including references to use of VCD with EBIs and use with underserved populations. The most frequently discussed EBIs used via VCD included Dialectical Behavior Therapy (DBT), Trauma-Focused Cognitive Behavior Therapy (TF-CBT), Prolonged Exposure Therapy (PE), Cognitive Processing Therapy (CPT), Cognitive Behavior Therapy (CBT), and references to using EBIs in general. References to use with underserved populations spoke to rural, remotely located individuals/families as well as those with limited access to transportation. Several participants endorsed exclusive use or using VCD for most of their clients because it was convenient and preferable for them. Barriers was the least frequently discussed theme, which includes things that decrease, impair, or hinder the successful implementation of EBIs (Kazdin et al., 1997) . Barriers related to the delivery of EBIs with VCD were technological, financial, collaborative, and those related to documentation (e.g., consent forms, routine outcome measures, homework forms). Technological barriers included those related to equipment, connectivity, user error or learning curve, and visibility (e.g., harder to "Read emotions on screen," glasses obscure eye contact and can create glare). Financial barriers were related to the lack of ability to bill health insurance companies for VCD. Collaborative barriers were related to the need for support from other professionals or health care facilities, including logistical coordination of transmitting record and homework completion when working with a remote facilitator. Barriers related to documents overlapped with collaborative barriers. This investigation utilized a mixed-methods approach to understand practitioner perspectives on the delivery of EBIs via VCD. Participants represented a diverse group of practitioners who were dispersed across professional disciplines (i.e., clinical and counseling psychology, social work, mental health counseling). Nearly half (49%) of participants identified their theoretical orientation as cognitive behavioral as is representative of the national trends in training (Heatherington et al., 2012) ; also, more practitioners in this study held master's degrees than doctoral degrees. Most participants (71%) worked in private practice settings which likely reflected the recruitment strategies taken by the research team (e.g., Psychology Today professional directory audience). Our results therefore may generalize most readily to telemental health practitioners working in private practice in the U.S., but broadly were drawn from a nationally representative sample. Taking both quantitative and qualitative data together to develop a rich understanding of this specialized practice approach, our results suggest that surveyed practitioners were highly fluent, comfortable, and felt moderately efficacious with using computers in treatment. Higher levels of education predicted participants' adherence to evidence-based approaches; however, their adaptation of protocols for VCD remained essential for translatability. Findings indicated that the further out from training a practitioner was, the less open they were to new or manualized treatments and more likely to use an EBI for reasons other than it being required of them. These findings point to the salience of training in practitioner decision-making and suggests the need for continuing education to focus on innovative updates to practice such as the incorporation of technology (Daniels and Walter, 2002) . Furthermore, participants endorsed the need for knowledge-sharing and training for practitioners in VCD. Participants spoke to the usefulness of social media in gaining knowledge about VCD implementation, which serves as a supplement to formal continuing education programs. Given the dramatic uptake of VCD to deliver treatment and EBIs during the COVID-19 crisis, these findings are especially relevant to explore and understand, although we collected these data prior to the pandemic. The first aim of this study was to examine the frequency and predictors of practitioner attitudes towards the use of VCD for EBIs. Previous literature has shown low rates of VCD utilization, despite practitioner attitudes towards its use being relatively positive (Lindsay et al., 2015; Perry et al., 2020) . Findings from the current study found similarly low rates of utilization, as well as positive attitudes. In the current study, the leading predictor of whether practitioners were more likely to use an EBI or a manualized treatment was if it appealed to them, followed next by if they were open to trying something new. Importantly, participants did not seem to be primarily motivated to adopt a manualized EBI because it was a requirement provided by their supervisors/administration. These data suggest a lack of leadership or organizational emphasis on using EBIs, which may be less relevant for private practitioners. Divergence on the EBPAS was significantly predicted by education level, such that more education (e.g., a doctoral degree) was related to less Divergence among participants. Previous research has shown that practitioners who prepare for clinical practice careers (e.g., master's-level practitioners) have less favorable perceptions of evidence-based approaches than those who prepare for a research career (e.g., PhD-level practitioners; Luebbe et al., 2007) . This is an essential factor to consider as qualitative data from interviews suggests that training in graduate school is a critical predictor of evidence-based practice, even many years after graduation. This finding is also supported by previous work in telepsychology training which indicates that experiences garnered in university training clinics with VCD can bridge the gap from training programs in mental health to the larger outside context of technology-facilitated service delivery (Dopp et al., 2017) . Considering the potential for training programs to normalize VCD use as a versatile tool among their clinical trainees and reduce usage barriers, the results of the current study support making VCD an essential component of graduate education in mental health service delivery. Previous research has also shown that training in the platform, years of experience, and perceived ease of use of technology have been among the top predictors of use (Simms et al., 2011) . Enhanced education therefore may be able to decrease barriers such as negative beliefs and attitudes about computers interfering with rapport or making treatment more difficult. Additional research should follow the trajectories of trainees from programs with and without exposure to VCD to explore their use of technology in treatment. The COVID-19 crisis has necessitated the move to VCD for many practitioners, including trainees, thus providing a rich opportunity to understand more about VCD delivery and its utilization. It is possible that the requirement to utilize VCD due to COVID-19 may negatively impact practitioner perceptions (as was found in the current analyses); however, because VCD may be considered a safer option or the only option in many facilities, practitioners may now be even more motivated to make VCD work, at least while some clinics remain closed. Additionally, how they have made it feasible is important for future implementation efforts (Wind et al., 2020) . This study contributed to the literature on facilitators related to VCD implementation, including tools and approaches that may not otherwise enter treatment. With increased rates of VCD utilization for EBIs, a tension between EBI fidelity versus adaptation for VCD is created. Participants in the current study were surveyed to determine the level of drift they had away from the standardized treatment protocol (i.e., adaptation). Outcomes indicated participants adapted treatment for VCD at a rate of 30% of overall sessions; although, there was considerable variability among responses (SD = 33%). This frequency is in sharp contrast to what was reported during qualitative interviews, in which most practitioners stated that there was little to no need to adapt treatment at all. Results of the current study indicate that modifications such as tailoring and integration may be perceived by practitioners as a normal part of implementation necessary to facilitate the best care (Borkovec and Sharpless, 2004; Moree and Davis, 2010) , regardless of the modality of delivery (Norcross and Wampold, 2011) . In turn, this normalization may increase frequency of this modification for VCD. To provide more context to these responses, practitioners involved in the interview process stated that frequent modifications were required surrounding technological barriers and documentation. This finding that practitioners modify treatment is consistent with studies which have examined practitioner-level barriers to the use of computer-assisted therapies (Becker and Jensen-Doss, 2013) . However, creative adaptations may, in fact, enhance services rather than simply detract from the protocol due to the unique aspects of VCD (Griner and Smith, 2006) . Social desirability may also account for the discrepancy between actual and perceived adaption frequency among practitioners in this study. For example, interviewed practitioners may not only have participated because they were more interested in VCD practice but may want to assert that fidelity is still possible. Therefore, they underestimate their frequency of modification use. This tension between trying to maintain fidelity while making sure treatment can translate to a nontraditional setting effectively seems like a challenge for practitioners (Wiltsey Stirman et al., 2015) . Modifications to allow for treatment implementation should be planned and benefit client care, and practitioners should avoid making them with the sole goal of personal comfort with treatment delivery (Lee et al., 2008) . Future research may seek to explore modification use more discretely and ideally tie this use to client outcomes and practitioner characteristics. Results of this study point to the need to meet practitioners at their current skill level and market interventions and strategies to practitioners using techniques such as user-centered design (UCD; Dopp et al., 2019) . While client outcomes are of paramount importance in clinical decision-making, dissemination and implementation scientists cannot forget that practitioners are also end-users of EBIs and their individual preferences are important factors in what kinds of services reach individuals in need. The results of the current study provided insights into practitioner motivations for utilizing VCD. A primary motivation included the ability to reach clients at great distances, including those in rural communities, across states lines, or those for whom transportation is a barrier. Practice settings in this study were primarily located in urban areas (46%), while the majority of telemental health clients were in rural areas (40%). Thus, participants considered VCD of EBIs as highly appropriate, especially for clients in rural areas or even for those in metropolitan centers like New York City where public transportation takes several hours. Studies exploring barriers to treatment often cite travel as a significant limitation in accessing and remaining in services over time, especially in the treatment of children and adolescents (Kazdin et al., 1997) . Results from the qualitative interviews further confirm the benefits to both parties, including convenience for the practitioner. VCD provides the ability to save time because, "I don't have to get up from my desk and walk to the waiting room to get the client." Interviewed practitioners consistently expressed positive attitudes regarding this model. A major limitation of the recruited sample was the response bias of early adopters (Rogers, 2003) which may not be generalizable to the large array of practitioners now utilizing VCD in their practices due to COVID-19 concerns. Additionally, the use of self-report measures only (as opposed to the measurement of behavioral outcomes) may have allowed for practitioners to overestimate their skill-level, underestimate their drift from treatment protocol, and overestimate their effectiveness in delivering these services (Kosmerly et al., 2015; Waller and Turner, 2016) . Future studies may seek to use other forms of measurement to assess for accuracy of reports on use of EBIs, VCD, and modification of treatment. While computer literacy is now a basic skill essential for everyday professionalism, the high ratings of comfort and efficacy with using computers in treatment reported by the study sample may have been skewed by the recruitment techniques (e.g., practitioners who listed themselves as providing "Online Counseling" on Psychology Today). Participants were also self-selected into this study and many of them found VCD as highly acceptable, which was reflected by their satisfaction scores. This self-selection process may have biased reports from both survey participants and interviewees as other practitioners who incorporate various levels of VCD use may have divergent opinions and experiences to those reported in this study (Braver and Bay, 1992) . Finally, secondary coding of qualitative transcripts was not conducted in this study, which may have introduced bias in the qualitative analysis. Despite these limitations, this investigation provides evidence for efforts which aim to increase training, access, and resources for VCD or EBIs. This study was the first of its kind to utilize a mixed-methods approach to survey practitioners about their use of EBIs via VCD. In recent years, delivering treatments via VCD was starting to gain ground (Simms et al., 2011; Wind et al., 2020) . However, the presence of COVID-19 has forced VCD to become an even more significant method of mental health delivery than ever before (e.g., Hayhurst, 2020; Reay et al., 2020 ). Yet, little research has been conducted to understand where and how VCD was being utilized for EBI delivery prior to the global pandemic. The present study took the first steps in addressing this gap in the literature by providing evidence that practitioner attitudes towards and acceptability of specific and novel modes of delivery are an important consideration in efforts to increase EBI dissemination and improve access to services. Even with this addition, there are still more questions and barriers to be considered. A larger effort may be necessary to organize effective dissemination across community settings. This work was supported by the University of Arkansas Marie Wilson Howell's Fund, University of Arkansas, Department of Psychological Science (PI: Parisi, #1901.03S). The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Computer-assisted therapies: examination of practitioner-level barriers to their use Practitioner perspectives on telepractice for serving families of children who are deaf or hard of hearing Enhancing treatment fidelity in health behavior change studies: best practices and recommendations from the NIH behavior change consortium Videoconferencing psychotherapy and depression: a systematic review Generalized anxiety disorder: bringing cognitivebehavioral therapy into the valued present Assessing and compensating for self-selection bias (nonrepresentativeness) of the family research sample The Computer-Email-Web (CEW) Fluency Scale-development and validation Mixed methods for telehealth research A systematic review of telehealth tools and interventions to support family caregivers Advancement in the maturing science of cultural adaptations of evidence-based interventions Changes in implementation of two evidence-based psychotherapies for PTSD in VA residential treatment programs: a national investigation Current issues in continuing education for contemporary behavioral health practice Implementation of telepsychology services in a university training clinic: process and lessons learned Integrating implementation and user-centered design strategies to enhance the impact of health services: protocol from a concept mapping study GPOWER: a general power analysis program Culturally adapted mental health intervention: a metaanalytic review. Psychotherapy: theory, research, practice A turning point for telehealth: COVID-19 spurs rapid uptake of connected care The narrowing of theoretical orientations in clinical psychology doctoral training Multisystemic therapy with violent and chronic juvenile offenders and their families: the role of treatment fidelity in successful dissemination The effectiveness of telemental health: a 2013 review Three approaches to qualitative content analysis Barriers to treatment participation scale: evaluation and validation in the context of child outpatient treatment Implementing evidence-based treatment protocols: flexibility within fidelity Practitioner adherence to guidelines in the delivery of family-based therapy for eating disorders Telemental health care, an effective alternative to conventional mental care: a systematic review Using a planned adaptation approach to implement evidence-based program in new contexts Implementation of video telehealth to improve access to evidence-based psychotherapy for posttraumatic stress disorder Evidencebased practice in psychology: perceptions of graduate students in scientist-practitioner programs The effectiveness and ineffectiveness of complex behavioral interventions: impact of treatment fidelity Treatment fidelity in outcome studies Cognitive-behavioral therapy for anxiety in children diagnosed with autism spectrum disorders: modification trends Using telehealth to remediate rural mental health and healthcare disparities Telemental Health: Clinical, Technical, and Administrative Foundations for Evidence-based Practice What works for whom: tailoring psychotherapy to the person Telemental health and US rural-urban differences in specialty mental health use Identifying and addressing mental health providers' perceived barriers to clinical video telehealth utilization Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda Additional clinical benefits of home-based telemental health treatments Telehealth mental health services during COVID-19: summary of evidence and clinical practice Web-based interventions for youth internalizing problems: a systematic review Leveraging the social network for treatment of social anxiety: pilot study of a youth-specific digital intervention with a focus on engagement of young men Diffusion of Innovations Exploring mental health practitioners' interest in using web and mobile-based tools in their practices The current state of telehealth evidence: a rapid review To use or not to use: practitioners' perceptions of telemental health Evidence base updates: the evolution of the evaluation of psychological treatments for children and adolescents A review of web-based technology in behavioural activation Development of a framework and coding system for modifications and adaptations of evidence-based interventions Empirical examinations of modifications and adaptations to evidence-based psychotherapies: methodologies, impact, and future directions Mixed methods sampling: a typology with examples Practitioner drift redux: why well-meaning practitioners fail to deliver evidence-based therapy, and how to get back on track Therapist adherence/competence and treatment outcome: a meta-analytic review Psychometric assessment of three newly developed implementation outcome measures Relationships between clinician-level attributes and fidelity-consistent and fidelity-inconsistent modifications to an evidence-based psychotherapy The COVID-19 pandemic: the 'black swan' for mental health care and a turning point for e-health Systematic review of the information and communication technology features of web-and mobile-based psychoeducational interventions for depression We would like to thank (names redacted for blind review), and the research assistants of the REACH and FCI labs.