key: cord-0739585-2q9rejz5 authors: Garcia, Dainelys; Blizzard, Angela M.; Peskin, Abigail; Rothenberg, W. Andrew; Schmidt, Ellyn; Piscitello, Jennifer; Espinosa, Natalie; Salem, Hanan; Rodriguez, Gabriela M.; Sherman, Jamie A.; Parlade, Meaghan V.; Landa, Alexis L.; Davis, Eileen M.; Weinstein, Allison; Garcia, Angela; Perez, Camille; Rivera, Jessica M.; Martinez, Chary; Jent, Jason F. title: Rapid, Full-Scale Change to Virtual PCIT During the COVID-19 Pandemic: Implementation and Clinical Implications date: 2021-02-14 journal: Prev Sci DOI: 10.1007/s11121-021-01211-0 sha: 13628ff43fd61311ee36b041d6626cda5faccbf8 doc_id: 739585 cord_uid: 2q9rejz5 Health agencies call for the immediate mobilization of existing interventions in response to numerous child and family mental health concerns that have arisen as result of the COVID-19 pandemic. Answering this call, this pilot study describes the rapid, full-scale change from a primarily clinic-based Parent–Child Interaction Therapy (PCIT) model to a virtual service model (i.e., I-PCIT) in an academic and community-based program in Miami, Florida. First, we describe the virtual service training model our program developed and its implementation with 17 therapists (M(Age) = 32.35, 88.2% female, 47.1% Hispanic) to enable our clinic to shift from providing virtual services to a small portion of the families served (29.1%) to all of the families served. Second, we examine the effect of I-PCIT on child and caregiver outcomes during the 2-month stay-at-home period between March 16, 2020, and May 16, 2020, in 86 families (M(ChildAge) = 4.75, 71% Hispanic). Due to the rapid nature of the current study, all active participants were transferred to virtual services, and therefore there was no comparison or control group, and outcomes represent the most recently available scores and not treatment completion. Results reveal that I-PCIT reduced child externalizing and internalizing problems and caregiver stress, and increased parenting skills and child compliance with medium to large effects even in the midst of the COVID-19 pandemic. Finally, the study examined components of our virtual service training model associated with the greatest improvements in child and caregiver outcomes. Preliminary findings revealed that locally and collaboratively developed strategies (e.g., online communities of practice, training videos and guides) had the strongest association with child and caregiver outcomes. Implications for virtual service delivery, implementation, and practice in the midst of the COVID-19 pandemic are discussed. The 2019 coronavirus disease (COVID-19) is a major public health crisis that has resulted in increased fear, stress, risk for child maltreatment, and potential long-term mental health implications for children and families (Golberstein et al. 2020) . During the COVID-19 pandemic, children and families have encountered significant life disruptions, including widespread school closures, remote learning transitions, and job losses that contribute to concerns about child and family mental health (Golberstein et al. 2020 ). Children exposed to traumatic and/or stressful events, such as COVID-19, are at greater risk for experiencing emotional and behavioral problems (Rubens et al. 2018) . Elevated levels of caregiver distress following stressful events also have been shown to negatively impact parenting practices and the caregiver-child relationship (Kelley et al. 2010) . Families with children with preexisting emotional and behavioral problems and/or caregivers with preexisting psychological concerns may be particularly vulnerable to COVID-19-related stressors. In fact, recent work has highlighted the negative impact of the COVID-19 pandemic on level of stress and mental health of caregivers of children with chronic conditions (Tilburg et al. 2020) . This is especially concerning for immigrant and minority communities that have been shown to be disproportionately impacted by COVID-19 (Clark et al. 2020) . Therefore, interventions that foster positive caregiver-child interactions and reduce child behavioral problems and caregiver distress are needed in response to COVID-19, particularly for populations at greatest risk for negative outcomes. Parenting interventions represent one such promising set of interventions. Evidence-based face-to-face parenting interventions have historically been considered the gold standard approach for improving the caregiver-child relationship and child disruptive behaviors (Kaminski and Klaussen, 2017; Pantin et al. 2009; Piquero et al. 2016 ; Thomas et al. 2017) . Given the effectiveness of these parenting interventions, efforts have been made to disseminate these interventions and evaluate what strategies are associated with successful implementation. However, face-to-face formats of service delivery often limit accessibility and availability of services (Danko et al. 2016; Nock and Ferriter, 2005; Owens et al. 2002) . There have been multiple calls to diversify the service delivery formats due to the significant gap between the children who need services and the children who actually receive and complete services, especially among racial and ethnic minorities (Kazdin and Blase, 2011; McGoron and Ondersma, 2015; Ramos and Chavira, 2019) . In an effort to increase accessibility (e.g., broader geographic reach, services occur in family's natural environment) and reduce treatment barriers (e.g., lack of transportation, childcare, or schedule availability; McGoron and Ondersma, 2015) , synchronous virtual delivery formats of parenting interventions have been developed. Emerging research supports the effectiveness of virtual delivery of parenting interventions in improving treatment engagement, child disruptive behavior, and caregiver-child relationships (e.g., Dadds et al. 2019; Comer et al. 2017; Estrada et al. 2019; Prado et al. 2019) . As it relates to virtual delivery of services for younger children, Parent-Child Interaction Therapy (PCIT), one of the most supported face-to-face parenting interventions (Kaminski and Claussen, 2017) , has been examined via videoconferencing format (i.e., Internet-delivered PCIT [I-PCIT] ) and found to be associated with improvements in child behavior and parenting skills in small open trials (Fleming et al. 2021; Kolhoff et al. 2020) , and to have comparable treatment outcomes to face-to-face PCIT in a randomized controlled trial (RCT; Comer et al. 2017) . While the emerging evidence for I-PCIT supports that this intervention is appropriate for children and families during COVID-19 (Gurwitch et al. 2020) , little is known about what elements of virtual service delivery training promote therapist proficiency in this delivery format. In particular, evaluating principles of training and virtual service delivery will enhance understanding of how to improve the ecological validity of virtual parenting interventions for children during a pandemic (Comer and Myers, 2016) . Within PCIT's emerging dissemination and training literature, within-agency trainers have been identified as key personnel for growing and sustaining delivery of PCIT services within organizations (Barnett et al. in press; Brabson et al. 2020) . Didactics, behavioral rehearsal activities, fidelity checks, consultation, and supervision, when indicated, are endorsed as strategies to incorporate into therapist training (Beveridge et al. 2015; Brabson et al. 2020) . Utilization of virtual synchronous video conference consultation from an expert trainer has been associated with modest improvements in child disruptive behavior above and beyond the effects of face-to-face PCIT services (Funderburk et al. 2015) . Although there is emergent literature on strategies used to train therapists in face-toface PCIT, there remains a need to examine what training strategies are appropriate for virtual service delivery and understand how these training strategies relate to familylevel outcomes (Beveridge et al. 2015; Brabson et al. 2020 ). More generally, research on how to train therapists to deliver virtual services effectively, compared with other aspects of virtual service delivery, such as feasibility and cost effectiveness remains underdeveloped (Edirippulige and Armfield, 2017) . Existing research describes disciplinespecific training models delivered over extended periods of time (i.e., months) and often in context of an academic program (Felker et al. 2020; Traube et al. 2020) . Considering the COVID-19 global pandemic, researchers have called for an acceleration in workforce capacity-building related to virtual services (Torous et al. 2020) . Existing programs needed to be able to quickly and effectively deliver services virtually to continue to serve patients. To do so, training and capacity-building strategies for therapists should be effective in building competencies, amenable to therapists of different disciplines, and resource efficient. The Coalition for Technology in Behavioral Science (CTiBS) evidence-based framework for telebehavioral health competencies describes competencies that can be leveraged to advance workforce capacity-building in virtual service delivery (Maheu et al. 2017 ). This CTiBS framework guides supervisors and practitioners to develop training in virtual service delivery. Because the framework is interprofessional, measurable, and comprehensive, it may serve as a guide for how to support programs or therapists to become competent in delivering services virtually. A review of studies on training and education to support virtual clinical services found that online learning was the most common approach and that there were few models which included hands-on practice opportunities (Edirippulige and Armfield, 2017) . Yet, research consistently shows that didactic training alone is not sufficient to support effective implementation (Edmunds et al. 2013) Ongoing and additional supports, such as coaching and consultation, enhance the likelihood that an innovation will be implemented effectively (Edmunds et al. 2013) . A recent study by Traube and colleagues examined a comprehensive virtual service training model for therapists implementing the Parents as Teachers home visiting model (Traube et al.2020) . Therapists received 120 h of synchronous and asynchronous training in virtual service delivery, as well as role plays. Therapists identified the comprehensive training and availability of supervisors as facilitators to implementation and emphasized the importance of hands-on practice opportunities (Traube et al. 2020 ). This training model was developed and delivered as a package, over an extended period. The model provides a foundation of how to train therapists in virtual service delivery, but it is important to examine which supports within multi-component training models are most effective and how to quickly train therapists. In response to the evolving pandemic and to minimize disruptions to care while mitigating exposure, our team utilized multiple strategies to build the capacity of PCIT therapists to rapidly shift to implementation of virtual services on March 16, 2020, when the stay-at-home order was instituted in Miami, a diverse city disproportionately impacted by COVID-19. The term virtual service delivery, as used in the current study, refers to the provision of synchronous telemental healthcare remotely, including secure videoconferencing and/or phone calls (Chou et al. 2016; Jent et al. 2018 ). In the current study, we present preliminary data to describe the rapid, full-scale change from a primarily clinic-based PCIT model to a virtual delivery model (i.e., I-PCIT), in response to the COVID-19 pandemic. Specifically, our program shifted from providing I-PCIT services to a small portion of our target population (with 17% of therapists providing services to 29.1% of program families) to providing I-PCIT services to all families. First, we describe the virtual service training model and accompanying strategies our program developed, which were guided by the CTiBS framework, to implement this full-scale change to a virtual delivery model. Second, we examined the preliminary effect of I-PCIT on child and caregiver outcomes during the period between March 16, 2020, and May 16, 2020 (when the stay-at-home order was first lifted). Specifically, we expand on previous research examining the impact of I-PCIT on child behavior and caregiver burden (Comer et al. 2017 ) by examining outcomes in the following domains: (a) child externalizing and internalizing behavior and compliance; (b) parenting skills; and (c) caregiver stress. Finally, we examined what training strategies were associated with greatest improvements in these outcomes. We analyzed the training strategies in an exploratory fashion. Given the encouraging research findings supporting the use of consultation as a strategy to effectively train and support therapists delivering PCIT, and research that suggests practice opportunities are critical to effective training in virtual service delivery, we expected that therapist use of consultation would be associated with the greatest improvements in child and caregiver outcomes (Edirippulige and Armfield, 2017; Funderburk et al. 2015; Jackson et al. 2018) . Importantly, we use the term "outcomes" to describe child and caregiver behaviors at the end of the initial 2-month stay-at-home order. However, as noted below, these outcomes were measured after most families had received, on average, about half of all treatment sessions. We decided to collect and report on data over the course of this 2-month period to understand exactly how training effects took hold, and family outcomes changed, over the course of the pandemic stay-at-home order period. Results presented here are preliminary and should be interpreted with caution. Future studies are planned (and data are being collected currently) to follow-up with this sample after their full courses of treatment are completed and several months thereafter. We offer this caveat at the beginning of this manuscript in the interest of full transparency. The academic and community-based program consists of six PCIT clinics, four that are embedded within the community and two that are embedded within the university hospital complex. The program was established in 2011 and is grantfunded to provide PCIT services free of charge to over 300 children aged 2 to 8 years and their families annually. Prior to March 2020, our program offered a hybrid approach to behavioral parenting interventions. Families self-selected into either (1) in-person, clinic-based, or (2) virtual services (i.e., I-PCIT). The program's six clinics consist of 17 master's-and doctoral-level therapists with backgrounds in clinical psychology and mental health counseling. The program includes staff therapists (76.5%) and trainees (23.5%) who complete PCIT as an elective training rotation. Either a staff therapist or a trainee was the therapist of record, and analyses reflect the therapist of record. Therapists received PCIT training and weekly supervision with a Certified PCIT Trainer. Therapists had a mean age of 32.35 years (SD = 6.76 years; 88.2% female; 47.1% Hispanic, 88.2% White). All therapists had a master's (64.7%) or doctorate (35.3%) degree, and 64.70% were Certified PCIT Therapists (i.e., PCIT International standards). Four therapists (23.5%) delivered virtual services prior to COVID-19. Patient participants included 86 children aged 2 to 8 years and their primary caregivers who were actively receiving services between March 16, 2020, and May 16, 2020. Demographic information is presented in Table 1 . Screening Families were screened for services through a phone eligibility screening, which included providing verbal consent and completing a background form and behavioral questionnaires in English or Spanish. Inclusion criteria included (a) a child 2-8 years old, (b) primary caregiver fluent in English or Spanish, (c) elevated child disruptive behavior on the Eyberg Child Behavior Inventory Intensity Scale (ECBI; raw score ≥ 131; Eyberg and Pincus, 1999) , or the Externalizing Problems subscales or composite of the Behavior Assessment System for Children, Third Edition (BASC-3 T score ≥ 60; Reynolds and Kamphaus, 2015) . Assessments If inclusion criteria were met, families completed the intake assessment, which included written consent, completion of a clinical interview, and completion of questionnaires about their child's functioning via REDCap. Families also participated in a 20-min behavioral observation to assess parenting skills and child compliance. Following the intake process, families began PCIT, which consisted of weekly assessment of child behavior and parenting skills taught to caregivers during two phases: Child-Directed Interaction (CDI) and Parent-Directed Interaction (PDI), over the course of 18 weeks. Therapists coach caregivers on their parenting skills from behind a one-way mirror via a wireless headset (for in-person services) or via videoconferencing (for virtual services). The academic and community-based PCIT program initially established I-PCIT as a subspecialty practice following brief consultation with Comer and colleagues (Comer et al. 2017) . Given the promising findings of their RCT, we established a similar service delivery model with one exception (our PCIT program utilized internet-based audio for visits, not separate phone calls). Developing a Virtual Service Training Model: Strategic Use of Workgroups On March 13, 2020, our program decided to transition all in-clinic services to virtual starting March 16, 2020. To expedite the transition, program leadership established seven workgroups of two-to-four therapists with the goal of developing a virtual service training model. Workgroups were led by program champions, including therapists with prior experience delivering services virtually. Each workgroup was assigned a topic (e.g., developing a consultation framework for supporting therapists in virtual service delivery) and conducted extensive literature reviews, discussed the development and/or refinement of training protocols and administrative and clinical procedures, assessed therapist needs, and identified potential challenges to implementation. The workgroups used the competency domains of the CTiBS framework for telebehavioral health competencies to guide the development of the strategies and accompanying resources. For example, the Virtual Environment and Telepresence competency domain informed the development of the Skills Practice support (Support 4), in which the therapist completed practice scenarios to help support a family in setting up an optimal virtual environment. The workgroups identified eleven strategies (described in detail in Table 2 ) within 72 h of the transition to virtual services. The workgroups met during a weekly team meeting to discuss each workgroup's recommendation and identify a plan for implementing the training strategies. On the same day, the full team was notified about the training strategies available to support them in the transition to virtual service delivery. Weekly workgroup meetings supported continued refinement of training strategies based on ongoing feedback. Staff therapists were required to, at minimum, view locally and collaboratively developed recorded trainings on virtual service delivery (strategy 2), review a Frequently Asked Questions document during an existing team meeting (strategy 7), and attend an Online Community Practice (CoP) held during an existing team meeting (strategy 8). PCIT trainees completing an elective PCIT rotation were encouraged to utilize as many of the strategies available that would be helpful for their training but not required to complete minimum requirements in order to maintain and respect professional limits to their time allocated to the rotation. However, PCIT trainees were co-therapists with staff therapists who were exposed to at least minimum requirements. It was expected that staff therapists would model skills and share knowledge during co-therapy. Data on therapists' use of the 11 training strategies were collected using a self-reported online survey collected via REDCap on the week that the first stay-at-home order in Miami was lifted. The survey asked therapists to indicate (a) which of the 11 training strategies they used during the 2-month period wherein the stay-at-home order was in effect (from March 16, 2020, to May 16, 2020) and (b) the total estimated time they spent using the 11 strategies. Notifying Participants At the time data were abstracted, 85 families were actively receiving PCIT services. Out of the 85 families, 87.1% (n = 74) began receiving services pre-COVID-19. On March 16, therapists contacted the 74 families, 60 (81.1%) of whom were receiving in-clinic PCIT and 14 (18.9%) of whom were receiving I-PCIT pre-COVID-19, to inform them of the transition to virtual services and obtain information regarding their preference for continuing care. Families selected from the following options: (a) transition to I-PCIT or (b) transition to biweekly phone support. Out of the 60 families who were actively receiving in-clinic PCIT, 83.3% (n = 50) selected to transition to I-PCIT and 11.7% (n = 7) selected to transition to biweekly phone support. By May 2020, all but one family who initially selected biweekly phone support transitioned to I-PCIT. Families who were receiving I-PCIT pre-COVID-19 were given the option to continue I-PCIT or transition to biweekly phone support. All 14 families that were actively receiving I-PCIT pre-COVID-19 continued with I-PCIT. Transitioning Participants to Virtual For families who were actively receiving I-PCIT services pre-COVID-19, PCIT services remained relatively unchanged, except for a brief weekly check-in related to COVID-19 stressors. However, families who transitioned to I-PCIT from in-clinic services were required to complete a consent form for virtual services and a technology set-up session (i.e., test equipment and setting up home environment) prior to starting I-PCIT. Caregivers completed all I-PCIT sessions online through a HIPAA-compliant Zoom session. Families used a computer with a webcam, tablet, or smartphone (angled to capture the play area) to broadcast live to the therapist, who provided live feedback and coaching on the caregiver's skills during play-based interactions via headphones. Eyberg Child Behavior Inventory (ECBI; Eyberg and Pincus, 1999) The ECBI is a 36-item caregiver-report measure of disruptive behavior in children 2-16 years. The ECBI has demonstrated acceptable levels of reliability and validity in racially/ethnically diverse populations (α = 0.94, testretest = 0.75; Gross et al. 2007 ) as well as stability over time and sensitivity to treatment change. Raw scores ≥ 131 (T score ≥ 60) on the Intensity Scale are considered clinically significant. The baseline ECBI Intensity Scale raw score was used to establish program eligibility and the most recent score was used as an outcome measure. Behavior Assessment System for Children, Third Edition, Caregiver Rating Scale (BASC-3 PRS; Reynolds and Kamphaus, 2015) The BASC-3 PRS is a caregiverreport measure of emotional, behavioral, and adaptive functioning of children aged 2-21 years. BASC-3 has well-established validity and reliability (α = 0.83-0.96, test-retest = 0.87-0.92; Reynolds and Kamphaus, 2015) . T scores ≥ 70 are clinically significant and T scores 60-69 are considered at-risk. The Internalizing Problems composite score was used as an outcome measure. The DPICS-IV is a behavioral observation coding system that measures the quality of caregiver-child social interaction during 5-min play situations. Numerous studies have documented the reliability and validity of DPICS coding categories (Eyberg et al. 2013) . To examine changes in parenting skills, caregiver verbalizations were coded for the frequency of positive statements (i.e., "Do Skills": labeled praises, behavior descriptions, and reflections), and negative statements (i.e., "Don't Skills": questions, commands, and criticisms). Caregiver-led situations were coded for the rate of caregiver effective commands, the rate of caregiver correct follow-through on effective commands (i.e., praises for child compliance or time out warnings following noncompliance), and the rate of child compliance to effective commands. Coding was conducted live by PCIT therapists who were trained to 80% coding reliability. The DPICS-IV was used as an outcome measure of caregiver skill acquisition. The PSI-4: SF is a well-validated 36-item caregiver-report measure of parenting stress that has good internal consistency (α = 0.96; Abidin, 2012) . The caregiver percentile score on the PSI-4: SF Total Parenting Stress Scale was used as an outcome measure of caregiver stress. Paired-sample t tests were used to investigate whether PCIT improved child and caregiver outcomes between March 16, 2020, and May 16, 2020. These t tests identify whether there was a significant difference between baseline scores ("pre-COVID-19" scores) and most recently available scores prior to the May 16, 2020 ("during-COVID-19" scores; M = 13.19 weeks since baseline) in the following domains: (a) child behavior (i.e., ECBI Intensity Scale raw score, DPICS-IV Child Compliance Rate, BASC-3 Internalizing Subscale); (b) parenting skills (i.e., DPICS-IV Do and Don't Skills, Effective Command Rate, Correct Follow-Through Rate); and (c) caregiver stress (i.e., PSI-4: SF Total Stress Scale). We next conducted a series of iterative path analyses in MPlus Version 8.3 to understand what training strategies were associated with treatment outcomes and caregiver skill acquisition (Muthén and Muthén, 2017) . These analyses proceeded in three steps. First, study covariates (i.e., child gender, race, ethnicity, and age, caregiver gender, ethnicity, race, and education, total number of treatment weeks and treatment sessions, and number of pre-COVID-19 treatment sessions) were entered into models predicting each during-COVID-19 dependent variable. To preserve study power and parsimony, only significant covariate associations at p < .05 were retained in further analyses. Second, pre-COVID-19 measures of the dependent variable were entered into the model. Third, variables representing therapist-reported use of each of the 11 training strategies were entered into each model predicting each of the 8 during-COVID-19 dependent variables. These were the final 8 models reported in "Results" (Table 3) . In accordance with expert recommendations (Muthén and Muthén, 2017) , full information maximum likelihood (FIML) estimation procedures were utilized to account for data missingness, and nesting of families within therapist was accounted for by adjusting standard errors of estimates utilizing the type = COMPLEX MPlus algorithm. Controlling for nesting of families within therapist controls for therapistlevel effects on during-COVID-19 outcomes (i.e., differences in baseline therapist skill level, comfort with technology, experience with virtual service delivery, etc.). Controlling for prior number of weeks, treatment sessions, and pre-COVID-19 treatment sessions protects against the concern that significant differences in during-COVID-19 measures were due to a greater number of pre-COVID-19 sessions attended by families. Controlling for pre-COVID-19 measures of each dependent variable ensures that any virtual service training effects emerge above and beyond baseline levels of behavior, as recommended by intervention scientists (e.g., De Los Reyes, 2017). Participant missing data on any during-COVID-19-dependent variable did not significantly differ from participants with complete data on any covariate, baseline treatment outcome or caregiver skill variable, training strategy variable, or specific therapist. Thus, no problematic missingness emerged in these data. Descriptive data on all study variables are reported in Tables 1 and 2, and correlations between study variables are reported in Supplemental Table 1 . Notably, when the stay-at-home order took effect, families had completed, on average, 5.21 total sessions. However, the first two to three sessions are intake sessions, and the third (or fourth) session is a "teach" session where caregivers didactically learn skills from the therapist. Therefore, on average families completed 2.65 actual treatment sessions (wherein caregiver, therapist, and child were all present and the caregiver was actively practicing the skills) before the pandemic stay-at-home order came down. Over the course of the stay-at-home order, families completed, on average, an additional 6.59 sessions (Table 1) . Therefore, most families in this study had recently started treatment as the PCIT program as the stay-at-home order was put into place. Consequently, the child and adult "outcomes" presented here demonstrate approximately midtreatment progress and should be considered preliminary and interpreted with caution. Paired-sample t tests revealed that during-COVID-19 measures demonstrated significant improvement compared with pre-COVID-19 measures on all child and caregiver outcomes (Table 2) . Specifically, with regards to child outcomes, child ECBI scores improved by an average of 34.59 points (t(83) = 10.19, p < .01), child compliance rate increased by an average of 50.15 percentage points (t(32) = 6.13, p < .01), and child BASC-3 Internalizing Problems Composite t scores improved by an average of 4.36 points (t(53) = 4.42, p < .01). With regard to caregiver outcomes, PSI-4 Total Stress percentile scores improved by an average of 18.16 points (t(59) = 18.17, p < .01). Caregiver skills also improved. Total caregiver positive statements ("Do Skills") increased by an average of 18.75 skills (t(83) = 14.71, p < .01), caregiver negative statements ("Don't Skills") decreased by an average of 23.43 skills (t(83) = 12.05, p < .01), caregiver effective commands rate increased by an average of 53.30 percentage points (t(33) = 9.49, p < .01), and caregiver correct followthrough rate increased by an average of 63.33 percentage points (t(53) = 10.54, p < .01). The effect sizes of all child and caregiver outcomes were medium-to-large (see Hedges' g section of Table 2 ; Cohen, 1988) . Next, we tried to ascertain whether specific strategies of the virtual service training model were associated with observed Table 3 as standardized parameter estimates. All results control for pre-COVID-19 levels of each outcome and nesting of families within therapists. Larger patterns are reported in prose. For detailed results refer to Table 3 . Notably, all therapists completed at least one strategy, 71.6% of therapists completed three or more strategies, and cumulatively, therapists reported that they completed 4.82 h of virtual service training. During-COVID-19 ECBI Scores Therapists viewing locally and collaboratively developed recorded trainings on virtual service delivery, participating in a CoP, and utilizing a locally developed I-PCIT guide were all associated with significantly decreased ECBI scores, even after controlling for pre-COVID-19 scores and caregiver gender (Table 3) . However, therapists' viewing web-based trainings developed by outside agencies, role-playing skills, reviewing videos of prior cases, and utilizing a FAQ document were all associated with significantly increased ECBI scores after controlling for participation in other training strategies. Therapists viewing webbased trainings developed by outside agencies, reviewing videos of prior cases, and using a locally developed I-PCIT guide were all associated with higher child compliance. Scores Therapists viewing locally and collaboratively developed recorded trainings and participating in a CoP were each associated with significantly decreased BASC-3 Internalizing scores. However, therapist viewing web-based trainings developed by outside agencies, reviewing videos of prior cases, and using a FAQ document were associated with increased BASC-3 Internalizing scores (Table 3) . During-COVID-19 PSI-4: SF Total Parenting Stress Scale Scores Therapists viewing locally and collaboratively developed recorded trainings and participating in a CoP were both associated with significantly lower Total Parenting Stress scores, whereas therapists viewing webbased trainings from outside agencies and utilizing a FAQ document were both associated with significant increases in these scores (Table 3) . Therapists viewing locally and collaboratively developed recorded trainings and utilizing a locally developed I-PCIT guide were associated with significant increases in caregiver Do Skills. However, consultations, role-plays, and use of a FAQ document were associated with significantly lower caregiver Do Skills (Table 3) . Therapists viewing locally and collaboratively developed recorded trainings and role-playing their skills was related to significant decreases in Don't Skills. However, therapists viewing web-based trainings from outside agencies, participating in consultations, reviewing videos of prior cases, and receiving live observation and feedback were associated with significantly higher Don't Skills. * p < .05, N/A indicates parameter estimate was not included in the final model because it did not significantly (p < .05) predict the outcome in an initial model only including covariates (i.e., child ethnicity, age, parent gender, ethnicity, education, number of treatment weeks, number of treatment sessions, number of pre-COVID-19 treatment sessions). Covariates not significant in this initial model were trimmed from final model in interest of model parsimony and to ensure final model was estimable. Support 5 is marked as N/A in two models where it was not estimable because participants whose therapists participated in that training had not completed that measure yet Therapist participation in a CoP, use of a locally developed I-PCIT guide, and in-session co-therapist support were associated with significantly higher correct follow-through rates. However, use of a FAQ document and live observation and feedback were associated with lower correct follow-through rates. Summary Across All Outcomes Specific training strategies appeared to be broadly beneficial across all during-COVID-19 outcomes. Therapists viewing locally and collaboratively developed recorded trainings was associated with improvements in all caregiver and child treatment outcomes, and 2 out of 5 parenting skills. Similarly, therapists participating in a locally coordinated CoP was associated with significant improvements in all treatment outcomes. Finally, therapist use of a locally developed I-PCIT guide was associated with improved ECBI scores, caregiver correct follow-through rate, and child compliance rate. Unfortunately, other training strategies were associated with negative treatment outcomes. Therapist use of web-based trainings developed outside of the local environment was associated with worsening of all three treatment outcomes. Therapist use of a FAQ document also was associated with increases in all three treatment outcomes as well as decreases in Do Skills and Correct Follow-Through rates. In an effort to answer calls for immediate service mobilization (Holmes et al. 2020) , while aligning with the CTiBS framework (Maheu et al. 2017 ) and identified implementation strategies (Powell et al. 2014) , we expanded the virtual service delivery capacity of our team to meet the mental health needs of children and their families in our high-risk community during the COVID-19 pandemic. While researchers have called for an acceleration in workforce capacity building related to delivering services virtually (Torous et al. 2020) , little is known about what strategies are most effective at promoting successful rapid implementation and how these methods may impact patient outcomes. The current study is a first step in addressing this gap (e.g., Edmunds et al. 2013) . Findings revealed that children demonstrated large reduction in caregiver-reported disruptive behaviors (Table 2 ). This magnitude of change is similar to that observed by Comer et al. (2017) for children receiving both clinic-based and I-PCIT (i.e., large to very large effects), with the current study being the first to provide evidence for the effectiveness of I-PCIT for families who transitioned from in-clinic to virtual services and, importantly, who made this transition during the COVID-19 pandemic. Of note, the second time point for the current study occurred prior to the end of the PCIT protocol, with families on average being about halfway through treatment, and should therefore be interpreted with caution. Consistent with caregiver-reported changes in child externalizing behavior, children were also observed to be significantly more compliant, and this effect size was similarly large. Additionally, significant decreases were found for internalizing behavior problems, with medium effect size. Findings are consistent with Chase and Eyberg (2008) , with reductions in internalizing symptoms following in-clinic PCIT, even among children with baseline subclinical internalizing symptoms. The current study adds to the existing literature by demonstrating that even without adaptations specifically geared to addressing internalizing symptoms, I-PCIT can still positively impact internalizing symptoms. This is especially important in the context of COVID-19, as children often experience increases in both externalizing and internalizing symptoms during stressful and/or traumatic events (Rubens, Felix and Hambrick, 2018) . Additionally, caregivers reported significant and large reductions in parenting stress and showed improvements in their parenting practices. Specifically, caregivers were observed to make more positive statements and fewer negative statements toward their children at the second time point, and they were observed to issue more effective commands with appropriate follow through. These large effects highlight PCIT's potential to impact both actors in the bidirectional caregiver-child relationship (Thomas et al. 2017) . Not only does PCIT improve child behavior and parenting skills, but also our data demonstrate it also significantly reduces parenting stress during times of unprecedented uncertainty, such as the COVID-19 pandemic. The aforementioned encouraging findings provide preliminary support for the use of PCIT in helping vulnerable families adapt to COVID-19-related challenges (e.g., sickness, working from home, school closures, routine disruption) that otherwise may have significant and deleterious effects on family mental health. Moreover, preliminary findings related to the efficacy of I-PCIT in the midst of a global pandemic fare well relative to other clinicbased PCIT studies of children with behavior problems (McNeil and Hembree-Kigin, 2010) . In terms of the association between the virtual service training model and outcomes, findings suggest locally and collaboratively developed strategies (i.e., recorded trainings on virtual service delivery, CoP, and I-PCIT guide), two of which were required of staff therapists, appeared to have the most consistent strong association with family-level outcomes. However, other training strategies appeared to have varying impacts on treatment outcomes. This was especially evident for training strategies that were developed outside of the local environment, which were at times found to have a negative association with family-level outcomes. These conflicting findings highlight the importance of examining the implementation processes in relation to the larger system in which they are embedded (Glisson and Williams 2015) . Our results are consistent with research that suggests innersetting factors (e.g., co-creating, collaborating) play a key role in training transfer (Jackson et al. 2019) . These findings also are consistent with Baldwin and Ford's training transfer model (1988) , which hypothesizes that training transfer occurs as a result of three factors: training design (e.g., content of training), individual trainee characteristics (e.g., demographics, skills), and work environment (e.g., organizational climate, support, opportunities to utilize skills). In the current study, therapists played a critical role in the design and organization of certain training strategies, which also likely contributed to maintaining a supportive and collaborative organizational climate. This may in part explain why locally and collaboratively developed strategies were more likely to have contributed to training transfer and ultimately to positive effects on outcomes. Another possible explanation for the conflicting findings is that training strategies that were externally developed were less adapted to local context and less practical in nature. Taken together, results highlight the importance of co-creating and collaborating in the context of training and professional development. Despite the positive effect associated with locally and collaboratively developed strategies, some locally and collaboratively developed strategies also were found to have varying associations with treatment outcomes. These strategies tended to focus on providing additional practice and resources (e.g., skill practice, case review, consultation, FAQ). It is possible that conflicting findings may be related to the process by which therapists engaged with these supports, as conclusions involving the temporal direction of training strategy utilization cannot be made. Above and beyond the required training strategies for staff therapists, both staff therapists and trainees self-selected the training strategies. It is possible that findings may be attributable to therapists utilizing training strategies in a tiered manner, where locally and collaboratively developed strategies served as a core component of initial training and other strategies related to ongoing support were utilized if additional support was needed. It is important to interpret the findings in light of several limitations. First, the virtual service training model was developed and implemented within a university-based clinic setting where there was an existing established virtual service delivery structure, which likely contributed to the success in rapidly transitioning our program to full-scale virtual services. It is possible that findings may not generalize to community and other settings, where the same level of implementation support may not be feasible, and the extent of existing infrastructure for virtual services may vary. Additionally, the sample predominantly identified as Hispanic. Despite limitations with regard to generalizability to other ethnic groups, findings contribute to the existing literature as Hispanic children are often understudied in psychosocial intervention research (La Greca, Silverman, and Lochman, 2009) . Therefore, it is important for future research to examine whether these findings extend to other settings and ethnic groups. Second, the sample size was relatively small given that only participants who were actively receiving services both in-clinic and virtually during the 2-month period between March 16, 2020, and May 16, 2020, were included. Therefore, the data presented should be interpreted as preliminary and in need of replication in larger samples. Third, the lack of a control group limits the ability to make inferences about changes in child and caregiver functioning. Changes in child and caregiver outcomes may be attributable to regression to the mean or a social desirability bias in responding rather than intervention effects. However, given research supporting higher levels of externalizing and internalizing behavior problems among children exposed to stress, trauma, and/or disaster (Rubens et al. 2018) , it is unlikely that symptoms would have resolved naturally. Fourth, data analyzed were from the primary caregiver and relied on caregiver report, particularly for child internalizing behavior. Thus, future studies should incorporate multimodal assessments that include multiple informants allowing for a more comprehensive evaluation of child behavior. Fifth, due to the rapid nature of the current study and timing at which data was analyzed, during-COVID-19 scores represent the most recently available scores approximately halfway through treatment for most families. Thus, it is unclear whether intervention effects will be maintained over time. Nonetheless, this study was an important first step in understanding the effectiveness of PCIT in improving child and caregiver outcomes during a global health crisis. Another limitation was the rare therapist use of the "shadowing cases" training support (only 1.16% of families had therapists who used this support). Though we continued to include this training support in study analyses to ensure results are reported transparently, null findings concerning this training support should be interpreted extremely cautiously and future studies with larger samples are needed to investigate this support's efficacy. A final limitation was that the current study was underpowered to examine potential mediating processes that might explain how therapist participation in training strategies led to better or worse family outcomes. Future studies could utilize the Integrated Model of Program Implementation (Berkel, Mauricio, Schoenfelder, and Sandler, 2011) to examine indicators of parent responsiveness (e.g., attendance rate, active participation, home practice, and satisfaction) as potential mediators. Despite these limitations, the present study is the first to outline the implementation process required to rapidly transitioning PCIT to virtual service delivery and examine the strategies associated with greatest improvements in family-level outcomes. Moreover, our training approach is aligned with the CTiBS framework, allowing for replicability of our model. The current study is also among the first to link training strategy use with both reported and observed child and caregiver outcomes, allowing for examination of the translation of training to therapeutic effect. Finally, our study captured child and family functioning at pre-pandemic baseline levels, allowing us to assess the impact of our program in the midst of a pandemic. The current study provides preliminary findings with important clinical implications for the rapid, effective mobilization of resources to provide widely accessible mental health services to children and families during a global pandemic. An overwhelming amount of research indicates that training alone is insufficient to support successful implementation (Edmunds et al. 2013) . The current study adds to this work by outlining the training strategies found most effective in producing positive change in domains commonly affected by stressful and/ or traumatic events. Given the varying associations with treatment outcomes for some training strategies, it is recommended that future training efforts promote a hierarchal approach to training, where universally beneficial training strategies that have been shown to have the strongest association with positive family-level outcomes (e.g., locally and collaboratively developed strategies) are used to build a foundation of knowledge related to virtual service delivery and situationally beneficial strategies (e.g., skill practice, case review, consultation, FAQ) are used when more specific supports are needed. Taken together, significant improvements in externalizing and internalizing child behavior as well as caregiver stress and parenting skills over time and in the context of a global pandemic build on the existing PCIT dissemination and training literature and highlight the potential for I-PCIT to effectively address the mental health needs of children and their families. Continued efforts are needed to leverage technology innovations to rapidly increase access to much-needed interventions such as PCIT, particularly in the context of a global pandemic. The online version contains supplementary material available at https ://doi.org/10.1007/s1112 1-021-01211 -0. Funding Clinical services were funded by a grant from The Children's Trust of Miami, Florida. Conflict of Interest The authors declare that they have no conflict of interest. The study was performed in accordance with the ethical standards as laid down in the 1964 Declaration of Helsinki and its later amendments or comparable ethical standards. The study was approved by the University of Miami IRB. All study participants provided written consent to participate. Parenting stress index-fourth edition (PSI-4) Transfer of training: A review and directions for future research Train-to-sustain: Predictors of sustainment in a large-scale implementation of Parent-Child Interaction Therapy. Evidence-based Practice in Child & Adolescent Mental Health Putting the pieces together: An integrated model of program implementation State-wide dissemination and implementation of parent-child interaction therapy (PCIT): Application of theory Understanding the Effectiveness of the Cascading Model to Implement Parent-Child Interaction Therapy Clinical presentation and treatment outcome for children with comorbid externalizing and internalizing symptoms Technological considerations for the delivery of real-time child telemental healthcare Disproportionate impact of the COVID-19 pandemic on immigrant communities in the United States Statistical power analysis for the behavioral sciences Future directions in the use of telemental health to improve the accessibility and quality of children's mental health services Remotely delivering real-time parent training to the home: An initial randomized trial of Internet-delivered parent-child interaction therapy (I-PCIT) Therapist-assisted online treatment for child conduct problems in rural and urban families: Two randomized controlled trials Outcomes of Parent-Child Interaction Therapy in an urban community clinic: A comparison of treatment completers and dropouts Inaugural editorial: Making the Journal of Clinical Child & Adolescent Psychology your "home journal Dissemination and implementation of evidence-based practices: Training and consultation as implementation strategies Education and training to support the use of clinical telehealth: a review of the literature eHealth Familias Unidas: Efficacy trial of an evidencebased intervention adapted for use on the internet with Hispanic families DPICS-IV) comprehensive manual for research and training Eyberg child behavior inventory & Sutter-Eyberg student behavior inventory-revised: Professional manual Implementation of a telemental health training program across a mental health department An Effectiveness Open Trial of Internet-Delivered ParentTraining for Young Children With Conduct Problems Living in Regional and Rural Australia Comparing client outcomes for two evidence-based treatment consultation strategies Assessing and changing organizational social contexts for effective mental health services Coronavirus Disease 2019 (COVID-19) and Mental Health for Children and Adolescents Reliability and validity of the Eyberg Child Behavior Inventory with African-American and Latino parents of young children Leveraging parent-child interaction therapy and telehealth capacities to address the unique needs of young children during the COVID-19 public health crisis Multidisciplinary research priorities for the COVID-19 pandemic: a call for action for mental health science Training transfer: A systematic review of the impact of inner setting factors Using technology to expand the reach of PCIT Evidence base update for psychosocial treatments for disruptive behaviors in children Rebooting psychotherapy research and practice to reduce the burden of mental illness Predicting posttraumatic stress symptoms in children following Hurricane Katrina: a prospective analysis of the effect of parental distress and parenting practices Parent-Child Interaction Therapy with Toddlers: A Communitybased Moving beyond efficacy and effectiveness in child and adolescent intervention research An interprofessional framework for telebehavioral health competencies Reviewing the need for technological and other expansions of evidence-based parent training for young children Parent-child interaction therapy Mplus user's guide Parent management of attendance and adherence in child and adolescent therapy: a conceptual and empirical review Barriers to children's mental health services A randomized controlled trial of Familias Unidas for Hispanic adolescents with behavior problems A metaanalysis update on the effects of early family/parent training programs on antisocial behavior and delinquency A systematic review of strategies for implementing empirically supported mental health interventions Rationale and design for eHealth Familias Unidas Primary Care: A drug use, sexual risk behavior, and STI preventive intervention for Hispanicyouth in pediatric primary care clinics Use of technology to provide mental health care for racial and ethnic minorities: Evidence, promise, and challenges Behavior assessment system for children A meta-analysis of the impact of natural disasters on internalizing and externalizing problems in youth Parent-child interaction therapy: a meta-analysis Telehealth training and provider experience of delivering behavioral health services Digital mental health and COVID-19: Using technology today to accelerate the curve on access and quality tomorrow High levels of stress due to the SARS-CoV-2 pandemic among parents of children with and without chronic conditions across the USA