key: cord-0056992-g1bcf9bu
authors: Dopp, Alex R; Mapes, Ayla R; Wolkowicz, Noah R; McCord, Carly E; Feldner, Matthew T
title: Incorporating telehealth into health service psychology training: A mixed-method study of student perspectives
date: 2021-02-24
journal: Digit Health
DOI: 10.1177/2055207620980222
sha: abda0761717bc1ef561999ec3eb4038ca804249b
doc_id: 56992
cord_uid: g1bcf9bu

OBJECTIVE: Telehealth is increasingly recognized as an avenue for enhancing psychologists’ capacities to meet the mental health needs of a diverse and underserved (due to barriers e.g., distance, transportation) public. The present study sought to inform training in telepsychology (i.e., telehealth delivery of psychological services) by using both quantitative and qualitative methods to explore the perspectives of doctoral students who have already been involved in such training. METHOD: A total of 19 predoctoral students from two universities, with at least some experience in telepsychology training, provided their perspectives on two complementary research questions: (1) How do students perceive their level of competence in various domains of telepsychology?; and (2) What are students’ perspectives on the process of telepsychology competency development during their doctoral training? RESULTS: The results of our study provide early evidence that doctoral trainees are able to develop telepsychology competencies and suggest that a supportive, training-oriented environment and fit between telepsychology and existing programmatic areas of emphasis are likely key to success. CONCLUSIONS: Continued efforts to enhance training in providing telepsychology services should focus on how to best define, measure, and promote competency development in this emerging specialty area.

therapy, diagnostic interviewing and assessment, supervision, ethical decision-making, and respect for cultural diversity. 8, 9 Research has consistently found that telepsychology has tremendous potential for improvements and innovations in care across a variety of clinical problems and demographic characteristics. [10] [11] [12] It offers an important avenue for making psychological services accessible to diverse populations who experience barriers (e.g., distance, transportation, work schedule) to in-person care, such as the 110 million Americans who live in mental health professional shortage areas 13 and people whose work or family obligations preclude regular office visits during typical business hours. In fact, numerous experts have identified telepsychology as a critical area for future growth in the field. [14] [15] [16] This is further represented by an increasingly supportive policy environment, with emerging avenues that permit the practice of telepsychology across state lines without requiring licensure in each state. 17, 18 The present study explored the experiences of individuals who could soon join the growing telepsychology work force: doctoral students in health service psychology (i.e., clinical, counseling, and school) programs. Given the rapid growth of the telepsychology market, some programs have begun preparing students to provide telepsychology services through dedicated training efforts. 19, 20 Unfortunately, such training remains rare as a part of formal coursework or supervised clinical experiences. 21, 22 Most psychologists are limited to the predoctoral internship, a one-year placement that serves as the capstone clinical experience of the doctoral student's graduate program, or postdoctoral and post-licensure training in telepsychology. Furthermore, these trainings often rely on circumscribed didactics or continuing education programs (e.g., workshops, online courses) and lack expert supervision. 21 Without greater attention to training in the clinical, ethical, and legal issues involved, 23 the number of psychologists who develop sufficient competence for independent practice in telepsychology will likely remain low -in turn limiting its tremendous potential impact.

Efforts to understand telepsychology training are best grounded in the broader understanding of competency development for telehealth, which has emerged over the past several decades. In this context, "competencies" refer to domains of knowledge, skills, values, and/or attitudes that must be used habitually and judiciously in effective professional practice. 24 Various health service fields such as medicine, dentistry, and nursing have each proposed sets of telehealth competencies; however, in order to provide a standard of care in the context of telehealth, common training and competencies are imperative. The skills and knowledge required for virtual interaction, compared to in-person care, can necessitate the addition or adjustment of professional behaviors. Such adjustments may include, for example, understanding and implementing changes for rapport building strategies, such as alternative behaviors for a handshake or increasing intentionality for providing eye contact through the video system. Numerous researchers have provided frameworks for defining telehealth care competencies. [25] [26] [27] [28] For example, Hilty et al. 27 provided a summary and comparison of the telebehavioral competencies across professions, with nine domains: patient care, communication, systems-based practice, professionalism, practice-based learning, knowledge, technology, education/training, and other technologies (e.g., social media, apps, e-mail). Hilty et al. noted that various fields place more emphasis on different domains but that these categories provide a framework for understanding the implementation, measurement, and evaluation of competencies.

Articulation of specific telepsychology competencies is less advanced. Various guidelines for telepsychology are now available (see APA 7 ) but each suggests a slightly different set of required competencies. As a step toward more unified and comprehensive guidance in this area, McCord and colleagues 29 synthesized all five available telepsychology guidelines into a Cube Model that identified 34 key foci of telepsychology training, organized into 9 practice domains, and their intersections with various modalities and settings of service delivery (all described in detail later). This model offers a useful descriptive framework for understanding and evaluating telepsychology training efforts. Now, there is a need for research that moves beyond identifying telepsychology training foci to understanding the processes by which training can be most successful. Such research will be especially useful for informing doctoral training efforts, but has implications for competency development in telepsychology at later stages of practice as well, given that many practicing psychologists never received training in telepsychology during their formal training years.

To that end, the present study explored factors that influence telepsychology training from the perspective of doctoral students who had already sought out such training. Students who are early adopters or "lead users," such as those who are already learning to provide care via video-conference, often have valuable insights into a topic based on their high level of interest and motivation. 30 We collected and integrated data using mixed methods (i.e., quantitative and qualitative; see Palinkas et al. 31 ) to answer two related, exploratory research questions: (1) How do students perceive their level of competence in various domains of telepsychology?; and (2) What are students' perspectives on the process of telepsychology competency development during their doctoral training? More specifically, we took a QUAL þ Quan approach involving simultaneous collection of qualitative and quantitative data for primarily qualitative, exploratory purposes (see Palinkas et al. 31 ). Such research is meant to produce rich, detailed understanding of a phenomenon that can inform and/or complement more quantitatively focused research. [31] [32] [33] Method Participants Participants were 19 students who were enrolled in doctoral training programs in health service psychology at one of two universities with whom the authors were affiliated: the University of Arkansas or Texas A&M University. As only four U.S. doctoral programs are known to offer dedicated telepsychology training, 19 this represents a considerable proportion of those programs. To be eligible for the study, students needed to have received some didactic and/or experiential training in telepsychology (i.e., they did not necessarily need to have delivered clinical services via telepsychology), be enrolled in their doctoral program, and not yet have started their predoctoral internship. Participants were 74% female and 26% male (ns ¼ 14 and 5); identified as Non-Hispanic White/Caucasian (n ¼ 12; 63%), Hispanic/Latinx (n ¼ 4; 21%), or Asian/Pacific Islander (n ¼ 3; 16%); and tended to be similar in age (M ¼ 27.1; SD ¼ 2.46). They were enrolled in PhD programs in clinical psychology (n ¼ 10; 53%) or counseling psychology (n ¼ 9; 47%). The majority (n ¼ 10; 53%) of students had applied to predoctoral internship sites through the U.S. match system, which determines where they will complete the final phase of their doctoral clinical training, and all of those students who applied had matched at an internship training site. We do not provide additional details (e.g., breakdowns by training program) to protect participant confidentiality, given the small size of the sample.

There were important similarities and differences between the telepsychology training experiences offered at the two universities. The similarities included didactic training focused on telepsychology-related knowledge; on-site supervision, consultation, and technical assistance; and opportunities for supervised clinical service delivery using telepsychology modalities. Moreover, student trainees in both settings were enrolled in APA accredited doctoral programs with scientist-practitioner orientations.

The University of Arkansas is located within a predominantly rural service area. It offered telepsychology training in its standard clinical practicum for clinical psychology doctoral students. All activities took place in the program's dedicated training clinic and were only available to students in that program. Prior to providing telepsychology services, students needed to complete didactic and general practicum (i.e., basic assessment and intervention skills) training requirements and obtain supervisor approval. Telepsychology services used a home-based delivery model, in which the client received services at their residence via synchronous video-conference. Individual and group supervision were conducted together for in-person and telepsychology services. The relationships between authors of this paper and the University of Arkansas are as follows: the first author was a supervising psychologist in the training clinic with primary oversight for telepsychology services (though other faculty also provided supervision of telepsychology cases and the first author was not a direct clinical or research supervisor for any student at the time of their participation) and the lead researcher for this study; the second and third authors were doctoral students in clinical psychology who provided administrative support during implementation of telepsychology services; and the fifth author was the clinic director.

In contrast, Texas A&M University was located in a predominantly urban area and offered a standalone, advanced training clinic that only provided telepsychology services. Practicum training at that clinic was available to students from all doctoral programs in health service psychology within the nearby geographic region of the university. Students applied and interviewed for placement at the clinic after having completed at least one semester of practicum training with their home institution. Admitted students received ongoing didactic training and individual/group supervision during the practicum. The telepsychology clinic primarily used a "hub-andspoke" model, in which the client presented at a remote site (e.g., a community clinic or health center located in a rural area) where they received services via synchronous video-conference. However, clinicians also used telephone counseling when necessary (e.g., if the client was unable to travel to the remote site), which could be delivered anywhere in the state -including the client's home. Supervision was provided in-person by the clinic director and a postdoctoral fellow, as well as via video-conference by off-site tele-supervisors (e.g., when a Spanish-speaking supervisor was needed for services provided in Spanish). The fourth author of this paper was the director of the clinic at Texas A&M.

The Institutional Review Board of the University of Arkansas approved all recruitment and data collection procedures for the present study. For recruitment, the authors generated an initial list of students who were eligible based on their participation in practicum training experiences in telepsychology. The lead researcher sent email invitations to 31 eligible students. The recruitment emails emphasized that students' participation was optional, that their decision to participate would not affect their standing in their programs, and that study data would not be used for student evaluations or otherwise shared with program faculty. Of those invited, 61% (n ¼ 19) ultimately completed the study. Data collection lasted from March to June 2018.

Data collection consisted of a single session, conducted by the lead researcher either in-person (n ¼ 13; 68%) or via a secure video-conferencing program (n ¼ 6; 32%). During the research session, the participant first completed a verbal informed consent process that emphasized the same points noted above regarding participation. They then completed a demographic questionnaire and telepsychology competency rating scale via Qualtrics; and completed a qualitative interview with the lead researcher. The interviews (but not the remainder of the research sessions) were audio recorded for later transcription and analysis, and the researcher also took notes during interviews. Participants' study materials were all identified using a unique, anonymous participant identification number to maximize confidentiality; only the first author had access to identifiable data (i.e., audio recordings) which were destroyed once they were no longer needed. Participants each received a $25 electronic gift card for their time.

Our use of mixed methods balanced breadth of information (quantitative measures) and depth of information (qualitative measures), allowing for a more comprehensive understanding of students' experiences with telepsychology training than either method would provide alone. Under the taxonomy of mixed-method research proposed by Palinkas et al., 31 the present study simultaneously examined quantitative and qualitative data (QUAL þ Quan), such that the data served complementary functions by elaborating on different aspects of the same topics.

Quan -Demographic questionnaire. On this questionnaire, participants provided the basic demographic characteristics reported previously and provided specific information about their previous experiences with telepsychology. First, participants rated their level of experience with telepsychology service delivery during their graduate training across various modalities (i.e., video-conference [including home-based vs. remote-clinic delivery], telephone, textbased, email, mobile application, and Web/internetbased) and settings (i.e., university counseling centers, hospitals, community clinics, private practice, schools, prisons and jails, military/Veterans Administration facilities) specified in the McCord et al. 29 Cube Model. Experience was rated as none, exposure only, minimal, moderate, or extensive (with behavioral anchors provided to guide ratings). The questionnaire also asked about hours of supervised experience with telepsychology (options were none, 1-50, 51-200, or >200), and the percentage of that experience involving services for children and adolescents. Finally, to consider the implications of telepsychology training for subsequent career stages, participants used a 5-point Likert scale (from Very Unlikely to Very Likely) to rate their likelihood of completing telepsychology training experiences during internship/postdoctoral training or use telepsychology in their future career.

Quan -Telepsychology Competency Rating Scale. Although some researchers continue to focus on defining competencies, others have highlighted the need for developing metrics to assess competencies. 34 There is no wellestablished rating scale for use with behavioral health providers to assess telehealth competencies. For this study, we developed a new 34-item, standardized rating scale for doctoral students to self-assess their telepsychology competencies, called the Telepsychology Competency Rating Scale (TCRS). A copy of the measure is available as Online Supplemental Material for this article. The TCRS was organized by the nine domains of McCord et al.'s Cube Model: service (6 items), supervision (1 item), technical skills (5 items), administrative skills (8 items), multicultural competence (2 items), research and evaluation (3 items), risk assessment (2 items), ethics and law (5 items), and assessment (2 items). For each item, the scale provided a name and brief description (all taken directly from the Cube Model), and then solicited a "general or average" rating of the student's competence in that area (i.e., across all telepsychology training experiences). Ratings were completed using a 5-point Likert scale (0 ¼ None, 1¼ Low, 2¼ Moderate, 3¼ High, 4 ¼ Expert); we also provided behavioral anchors for each rating option (e.g., Moderate: "extensive experience and independence, but still requires regular supervision"). The lead researcher was available to answer participant questions during completion of the TCRS. As the TCRS was newly developed for this study, its psychometric quality is currently unknown, although we did conduct a preliminary examination of internal consistency.

Due to an error that occurred when transferring the TCRS into Qualtrics, the single item for supervision was inadvertently left out of data collection for the present study. Thus, we unfortunately did not collect participants' ratings of that domain and therefore were unable to consider it when calculating average ratings of competence across all domains on the TCRS.

QUAL -Interview. We used a qualitative, semi-structured interview to elicit information about participants' experiences with telepsychology training. The interview began by asking the participant what their experience has been like learning about and (if applicable) delivering telepsychology services. Next, a series of questions requested specific information regarding what was most helpful and most challenging in developing competency for each of the nine competency domains from the TCRS, as well as overall. A list of the domains, including relevant training foci (with definitions) as listed in the TCRS, was available to aid the participant in reflecting on each domain. Finally, the 10 students who had applied to internships were asked how their telepsychology training related to that next stage of their training and competency development. Interviews lasted an average of 59 min (SD ¼ 11.25).

Quantitative data analysis. We calculated descriptive statistics (e.g., Ms, SDs, frequencies, ranges) for items on the demographic questionnaire and TCRS. Our small sample size precluded the use of more complex inferential statistics (e.g., testing for differences in TCRS ratings across domains or between subgroups). For the TCRS, we calculated average ratings for the items in each domain and an overall average rating across domains. The scale had high internal consistency (a ¼ .946), although this is very preliminary evidence for reliability given the small sample and aforementioned missing item. Internal consistencies for most of the domain subscales of the TCRS also seemed acceptable (as ¼ .769-.921), except for two of the two-item subscales: multicultural competence (a ¼ .491) and assessment (a ¼ .603).

Rapid coding of qualitative data. Immediately following the completion of each qualitative interview, the lead researcher completed a set of brief, focused codes using a Rapid Evaluation and Assessment Method. 35 These codes summarized each participant's perspective on three topics related to developing telepsychology competencies: what was most helpful, what was most challenging, and (when applicable) the relevance for internship. The lead researcher created three to five summary codes for each topic based on each interview, and organized those codes by participant within a deidentified matrix. As much as possible, the lead author recorded codes in participants' own words to maximize validity and help to identify exemplar quotes.

Once all interviews were complete, we used an iterative and collaborative process to extract general themes summarizing the content of responses across participants (i.e., conventional content analysis) 36 from the matrix of codes. Initially, the doctoral student co-authors independently reviewed the matrix of codes and each generated a preliminary set of themes. When identifying a theme, they considered its cohesiveness and prevalence across participant responses, but also incorporated quotes and perspectives that were inconsistent (i.e., negative case analysis). They then combined their preliminary themes into a single set of themes through additional discussion and consensus. They also consulted with the lead researcher for feedback as needed during the analytic process. It should be noted that the coders explored for the presence of different themes across subgroups of training sites (e.g., model of telepsychology services, rural vs. urban service area) and trainees (e.g., amount of telepsychology experience), but found remarkable consistency in trainees' perspectives and therefore focused on themes for the entire sample. The three authors agreed that the interviews had reached saturation (i.e., additional interviews were no longer resulting in meaningful incremental changes to the themes) and, therefore, further data collection was unnecessary. The two clinic director coauthors reviewed the initial set of themes and provided feedback, which the other authors addressed through additional refinements to produce a final set of themes. Once we had finalized themes, the lead author selected an exemplar quote to illustrate each theme and transcribed it verbatim from the audio recordings; the remaining authors reviewed and agreed to all selected quotes.

Participants reported varied amounts of supervised experience with telepsychology. Over half (53%) reported 50 hours or less, with one reporting zero; six (32%) reported 51-200 hours; and three (16%) reported over 200 hours (M ¼ 317 hr). The majority of students' hours of experience were typically devoted to work with adults. On average, 10.5% of hours were with children and adolescents, though this was highly variable (SD ¼ 23.22%; range ¼ 0-80%). Over half (n ¼ 11; 58%) of participants reported no telepsychology experience with youth.

In terms of the types of telepsychology training, Table 1 presents the proportion of the sample reporting a given level of experience, by modality and setting. Among modalities, participants reported the most experience with video-conferencing (remote clinic delivery), telephone/audio call, and video-conferencing (home-based delivery). The proportions reporting moderate or extensive experience were 64%, 63%, and 26%, respectively. When considering settings, the most prominent was community clinics: 53% of the sample endorsed moderate or extensive experience there. Counseling centers were the next most common with 37% reporting some (i.e., minimal or greater) experience. Most participants had no experience providing telepsychology in the other modalities or settings.

Regarding the likelihood of using their telepsychology training in internship and postdoctoral training, the average participant rating was 3.1 out of 5 (in the neither likely nor unlikely range). A considerable minority of students (n ¼ 6; 32%) thought that this was somewhat or very unlikely. However, ratings for the likelihood of using their telepsychology training in their future career averaged a much higher 4.1 (in the somewhat likely range). Five participants (32%) rated this item as very likely; none rated it as somewhat or very unlikely. Table 2 summarizes the participants' ratings of their telepsychology competencies on the TCRS. The highest average rating was for multicultural competence (M ¼ 2.4); that and four other domains (service, research and evaluation, risk assessment, ethics and law) averaged in the moderate range. The average overall competence score on the TCRS was in the same range (M ¼ 2.0). The lowest average rating was for administrative skills (M ¼ 1.2); that and two other domains (technical skills, assessment) averaged in the low range. Table 3 , which also provides an illustrative quote for each theme, and we describe each theme in detail next. Within each major topic area, themes are reported from most to least impactful (based on our assessment of their salience, prevalence, etc.) in the text and table. We refer to participants as "students" throughout this section to emphasize their role and context.

Supervisory and environmental support. Breadth and depth in supervisory expertise, within a structured and wellresourced training environment, led to supportive climates in which trainees felt comfortable working on their knowledge and skills. Students also received more informal consultation and support from their peers. Students noted the importance of policies and procedures that were clear and specific, informed by student input, and disseminated in multiple ways (e.g., didactic training, clinic newsletter). They also appreciated the availability of program administrative staff who served as points of contact and helped solve challenges (related to technology and interpreting New and rapidly changing area of practice "Because it's a new area, that so much of it requires us to kind of brainstorm and find solutions.

. . I think someday, hopefully, the guidelines and the ethics will all be more ironed out." Domain-specific lack of exposure "How to make technology work. . .I realize that I don't know all those steps cause they have been dealt with by other folks. . .my knowledge definitely ends at a certain point."

Overextension of cultural competence "We're serving a greater area, every rural community is different.

. .we talk to the community members so that we understand the values and culture more."

Tension between accessibility and appropriateness of care "I think client appropriateness is, that's sometimes a challenge in itself, especially if you're considering the fact that there are no other resources. . .They might not be the best fit, but it's like, are you going to just deny them services altogether?"

Relevance of training to internship Influenced their training goals and interests "I think it's been career-altering if I can describe it that way.

. . I've been learning so much about the potential that it has and, uh, the impact that it can have on my career moving forward."

Sites viewed their experiences with telepsychology favorably "Having this experience is a huge asset. It provides a really unique training opportunity.

. .really unique in terms of client population that you serve and service delivery. So I think it really kind of made you stand out." policies, for example). Despite the largely positive feedback about program supports, some students found that supervisors in their program were variable in their level of telepsychology knowledge. Also, a few students described community-based training sites as providing little telepsychology-specific training or support.

Learning by experience. Students perceived that they were able to learn firsthand through repeated practice -and that their supervisors encouraged them to approach the learning process in flexible, creative, and exploratory ways. Students said that this learning orientation helped them feel comfortable and supported (and, in turn, more open and willing to be vulnerable) during the learning process. Even students who had limited experience with telepsychology, such as working with a single case or indirect exposure (e.g., in group supervision), identified that experience as contributing to their initial competency development. Importantly, however, those same students still acknowledged their limited exposure as a barrier to competency development.

Students' doctoral programs placed considerable importance on training students to work with diverse, underserved populations and on increasing access to mental health care. Thus, students found that their general clinical training backgrounds prepared them well for telepsychology; for example, they already had strong backgrounds in relevant multicultural, ethical/ legal, and risk assessment domains. Students also described how their own values were aligned with their programs' and, thus, they were a good fit for telepsychology on a personal level.

Expansion of existing competencies. Students described much of telepsychology as building on competencies for in-person service delivery, rather than representing brand-new learning. Many described being surprised to find that telepsychology services were "really similar" to in-person services in most ways, and thus they were able to build on foundational knowledge and skills from a variety of training areas (e.g., service delivery, administrative skills, assessment, research). Students described these competencies as common to doctoral training in psychology generally, though there some were program-specific (see "congruence with training program"). Furthermore, students noted that telepsychology training expanded their general competencies in important ways, such as using existing risk assessment skills over telephone or video-conference, as well as leading them to develop new competencies, such as technical skills.

Integration of research and practice. Several students stated appreciation for the central importance of data collection and analysis in their "data-driven" training clinic. A few also described how their telepsychology training influenced their later research interests or viceversa.

Use of technology. Many students noted that using technology during service delivery would solve some issues, such as a client's lack of transportation, but also "adds a layer" of new issues. Examples included unfamiliarity or difficulty with the technology (i.e., lack of technical fluency among clinicians or clients); technological failures (e.g., loss of connectivity); and limited capabilities of the technology used (e.g., remote sites needing to collect paper forms from clients and fax them to the clinic). Despite the issues introduced by technology, students agreed that the benefits of telepsychology outweighed those challenges. Moreover, several students (especially one who said they "love technology") noted that the prevalence of technology in daily life made telepsychology more relevant to them and their clients; they described use of technology as helpful, not challenging.

New and rapidly changing area of practice. Students discussed how even telepsychology experts, such as supervisors, are typically still learning the best ways to deliver services and provide training in this area. Research to inform telepsychology is also limited, and often unfamiliar or difficult to locate (e.g., published outside of mainstream psychology journals). Moreover, students noted how critical aspects of telepsychology -such as the technology used or the legal and policy contexts in which services are deliveredchanged frequently, causing their knowledge to quickly become obsolete. Given all of these factors, students often felt unsure about how to proceed with telepsychology services and required frequent supervision and consultation. They reported the need for patience, creativity, and flexibility while developing competencies in telepsychology.

Domain-specific lack of exposure. A significant minority of students described how they had not pursued intensive exposure in certain domains due to lack of interest, particularly in the technical skills, administrative skills, and research/evaluation domains. Among the students who were interested in developing those domains, they also noted limited opportunities for exposure due to other individuals holding primary responsibility (e.g., service coordinators or Information Technology staff addressed serious technical issues; office administrative staff billed clients). That division of labor did allow students to focus on their clinical role, which they acknowledged as helpful for training. It is notable that students who worked in both service coordinator and clinician roles did not describe the same lack of exposure. Also, all students had limited exposure to domains that were less relevant to their settings (e.g., neither clinic accepted insurance).

Overextension of cultural competence. Some students expressed how telepsychology allowed them to engage more diverse individuals in services, and thus required the ability to effectively serve those diverse individuals. Given the large catchment areas served by the participating clinics, students were sometimes unfamiliar with their clients' communities and cultural backgrounds, despite strong generalist training in multicultural competence. There is also the need to keep up with locally available resources in clients' communities. In addition, students noted the limited availability of bilingual supervisors as a barrier, with services in Spanish remaining limited despite a high prevalence of Spanish-speaking clients.

Tension between accessibility and appropriateness of care. Students reported finding it difficult to make decisions about whether to initiate services with a client who was not a good fit for outpatient telepsychology services but also had no alternatives available in their community. Examples of factors contributing to a poor fit between the client and services included level of risk, severity of psychopathology, or the need for specific types of assessment that were not available for telepsychology administration. Related to the previous challenge on cultural competence, they noted how efforts to engage members of underserved communities in services are more likely to result in such dilemmas.

These themes summarize a subset of students' (n ¼ 10) views on the relevance of telepsychology training to predoctoral psychology internships.

Exposure to telepsychology often shifted or focused students' long-term career interests, which were reflected in the criteria they used to search for and evaluate the fit of internship sites. Settings for which they saw telepsychology as relevant included correctional/forensic sites (especially those that included community services) and a site offering Dialectical Behavior Therapy with phone coaching. Some students made additional telepsychology experience one of their highest priorities for internship training, as they wanted to either make telepsychology a central focus of their career or expand on the training received in their doctoral programs. Others focused on different training goals (e.g., ability to work with underserved populations) but still considered telepsychology to be a valuable training experience that was consistent with those training goals. It is worth noting that some students found it challenging to identify telepsychologyfocused internships, or did not pursue such internships because they perceived their level of experience and competence as too limited to be competitive.

Internship sites viewed their experiences with telepsychology favorably. Students received indications of this theme directly from internship faculty, most often during interviews. Generally, students perceived that internship sites considered telepsychology training to be a unique, valuable experience that increased the student's readiness for internship. Students reported sites indicated to them that such experience was a "huge asset," "exciting," and "very relevant." This was true even of sites with no telepsychology training for interns, as the students thought that sites viewed telepsychology training as an indicator of other desirable student qualities (e.g., multicultural orientation, openness to different training experiences) related to readiness for internship. Even students with very minimal exposure to telepsychology reported receiving similarly positive reactions. However, students did note that telepsychology was not emphasized at some sites and that they had to communicate the value of their training to some internship faculty who were less familiar with telepsychology.

In order to meet the growing need for a telehealth workforce, [14] [15] [16] doctoral programs in health service psychology must expand upon the currently limited training available in telehealth. 21, 22 The need for expanded training has increased substantially this year, as telepsychology has been central to the mental health services response to the COVID-19 global pandemic. 37 The present study sought to inform such expansion efforts by capturing the perspectives of clinical psychology doctoral students regarding two complementary topics, using a QUAL þ Quan exploratory approach 31 to produce rich understanding of telepsychology training experiences. 32, 33 First, participants tended to rate their telepsychology competencies as Moderate overall (M ¼ 2.0) for the majority of domains (66.67%). Second, when describing the process of telepsychology competency development during their doctoral training, participants identified learning by experience, congruence with training program goals and values, and expansion of existing competencies as keys to their success; they also described high levels of interest, and success, in pursuing telepsychologyrelated experience in the next phase of their training (i.e., the predoctoral internship). The study had several methodological strengths, including (a) inclusion of "early adopter" 30 29 Despite the exploratory nature of this study, doctoral programs interested in developing students' competencies in telepsychology, and telehealth more broadly, may find our results informative in their efforts to implement specific telehealth training experiences, either inhouse or at external training sites. Our results suggest that telehealth training can be successful at the doctoral level, where moderate levels of competence are likely sufficient for a specialty area of practice (i.e., not expected of all program graduates). 38 It will be important for training programs to consider their ability to address areas identified as most helpful (e.g., "supervisory and environmental support," "congruence with training program goals and values") and challenging (e.g., "new and rapidly changing area of practice") when deciding whether telehealth training fits well into their program. Dopp and colleagues 19 describe a useful process for implementing telepsychology services in a doctoral training clinic and offer guidance on practical issues (e.g., development of policies and procedures, effective use of consultation, selection of equipment, ensuring HIPAA compliance) from an administrative perspective; while the article focused on telepsychology services, much of the guidance applies to implementation of telehealth training generally. Many doctoral training clinics in health services psychology programs recently began offering telepsychology services in response to COVID-19, 37,39 but additional efforts are likely necessary for those programs to develop associated training practices. Thus, the findings from this study could help guide the development of formalized telepsychology training efforts around the services that doctoral programs rapidly implemented in response to this global health crisis.

Programs that wish to provide focused training in telehealth, and telepsychology especially, should draw on existing literature for competency development, given that participants described the development of telepsychology competencies as sharing many characteristics with high-quality training more broadly. For example, psychology programs might consider the development of a variety of assessment tools (e.g., direct observation, written tests, role plays, selfassessment) that can triangulate evaluation of knowledge, skills, and attitudes involved in telehealth competencies, as with core/foundational competency assessment. 8 It will also be necessary to establish expected levels for various telepsychology competencies (e.g., "minimum levels of achievement" in the language of the APA Commission on Accreditation), 40 especially those that are more unique to the specialty (e.g., technical skills). The TCRS has the potential to develop into one such self-assessment tool, but it is currently limited by unclear psychometric quality and a lack of detailed norms for different modalities and settings (e.g., our participants might have higher competencies for video-conferencing and telephone counseling in general outpatient settings, the areas in which they had the most experience) and different stages of doctoral training. The tendency for self-assessments of competence to be inflated also suggests that the TCRS should not be used in isolation. 41 Literature on telehealth competency assessment could help inform improvement of the TCRS, although such research remains rare. In a study by Goran, 42 the author examined competency assessment of tele-ICU nurses and found the most common method used was performance appraisal and observation, although peer review and self-assessment also were used. For behavioral telehealth, there is even less research on the assessment of competencies, but existing studies have used therapist fidelity measures (e.g., Gellis, Kenaley, & Have) 43 or video vignettes and self-report surveys (i.e., Giffords, Niles, Rivkin, Koverola, & Polaha). 44 Our results also suggest that, as in many specialties, doctoral training in telepsychology can usefully prepare students to pursue more advanced training opportunities. Across qualitative and quantitative (i.e., demographic) data, students described positive experiences with, and interest in, applying their telepsychology training during subsequent career stages (e.g., predoctoral internship, postdoctoral fellowship). Continued development in telepsychology is likely a necessary and desirable outcome of doctoral-level exposure to this specialty. Additional training experiences should be developed at the predoctoral internship and postdoctoral fellowship levels, as well as for mid/latecareer psychologists wishing to respecialize. It would also be useful for that research to examine the specific settings (e.g., VA, prisons) where telepsychology training and practice are most applicable. Moreover, it will be useful to examine patterns of telehealth training in other health service professions outside psychology, as trainees in those professions could similarly benefit from telehealth experiences that are tailored to different career stages and settings.

Our findings generally support the benefits of doctoral-level training in telepsychology, yet they also suggest critical remaining challenges that also merit consideration for telehealth training more broadly. In particular, our participants tended to rate their competency as "low" in three domains on the TCRS -technical skills, administrative skills, and assessment -and described in the qualitative interviews how the "use of technology" introduced new problems and they had a "lack of domain-specific exposure." The achievement of well-rounded telepsychology training may require doctoral students to serve in multiple roles with various emphases on clinical, administrative, and technical skills. Furthermore, it is important to keep in mind that some students appreciated the narrower focus on service-related competencies in their telepsychology training; it may not be necessary, or practical, for all telepsychology trainees to develop equivalent competency in all possible domains. These remaining questions illustrate how exploratory, primarily qualitative findings can inform future research; for example, some students described how the use of technology in their personal life was beneficial for developing technical competencies, which could inform efforts to develop (and test) predictive models of the characteristics of successful telepsychology trainees.

In contrast, addressing the lack of exposure to psychological assessment presents a different challenge due to the limited number of assessment tools that can be reliably and validly administered via telehealth. 45 Fortunately, computerized versions of widely used assessments, such as the Weschler Adult Intelligence Scale, are currently in development and show considerable initial promise for remote administration. 46 Tele-assessment thus provides an excellent example of telepsychology being a "new and rapidly changing area of practice." Faculty supervisors or those in leadership roles, such as training directors and clinic directors, should strive to stay abreast of such developments in telepsychology so that the latest science can be incorporated into student training. At the systems level, professional organizations and state and federal funding agencies can invest in keeping up-to-date, researchbased recommendations available for clinicians and training programs, and can serve a vital role in advancing competent and ethical telehealth service delivery.

The present results should be interpreted in the context of several limitations. First, given that this study focused on depth of understanding, 32,33 our small sample from two training sites limits the generalizability of the findings. It would also be useful for future research to examine complementary, more quantitative issues around prediction and broad generalizability, for example by collecting data on telepsychology competencies from a large, diverse sample of trainees in health service psychology programs. Caution must also be taken when generalizing these findings beyond the place (we focused on programs in the United States) and time (we conducted this study prior to the COVID-19 pandemic) in which they were collected. Second, we may have reached different conclusions if we had collected information from others involved in telepsychology training efforts, such as supervisors, clinic directors, internship faculty, or the clients served by trainees. The research team represented some of these alternative perspectives, but we could not incorporate our own viewpoints into the analysis (e.g., during qualitative coding) without jeopardizing the validity of our findings. It will be important for future research to consider these different perspectives, as well as trends and factors influencing the attitudes of training faculty and clients toward telepsychology. Moreover, the disparity between the roles of the participants (primarily clinician roles) and research team (primarily administrative roles) may have limited our understanding of the participants' perspectives, though we ameliorated this concern somewhat by including student coders and co-authors.

Third, doctoral students in health service psychology programs who are not "early adopters" might have different training needs (e.g., more didactic education about telepsychology) and challenges (e.g., lower motivation to develop telepsychology competencies). It is also possible that some students from the two universities in the study had negative experiences with telepsychology training, but declined our invitation to participate in this study. Fourth, our rapid coding procedure limited the depth of information obtained from qualitative interviews for this initial analysis, and limited the initial codes to a single researcher's perspective (i.e., we could not examine interrater reliability or have participants engage in member-checking review of findings). In the future, we plan to conduct in-depth content analysis of the themes discussed by participants for each domain of telepsychology competence and to more fully explore subgroup differences. Fifth, the results of the TCRS should be interpreted with caution until additional research is able to examine the questionnaire's psychometric strength, beyond the preliminary internal consistency examined in this study. Sixth, we were unable to make firm conclusions about telesupervision due to our inadvertent exclusion of the relevant item from the TCRS. Research specifically focused on tele-supervision models and techniques will be important in the future because that is another rapidly changing area of telepsychology practice. 47, 48 Finally, although we took extensive efforts to avoid bias through our recruitment and consent procedures, it will be important for independent researchers (i.e., not supervisors or clinic directors in the training programs being studied) to verify our findings in the absence of potential demand characteristics.

In conclusion, the results of our study provide early evidence about the conditions under which telepsychology training can be feasible at the doctoral level. A supportive, training-oriented environment and fit between telepsychology and existing programmatic areas of emphasis will likely be key to success. We hope that a growing number of programs will heed our call to provide such training. It will also be important to consider how supportive learning environments can be created for internship and postdoctoral training, as well as mid/late-career psychologists who seek to develop telepsychology competencies outside of a formal training environment. Moreover, continued research is needed to better understand how we can best define, measure, and promote telepsychology competency development. Finally, it is worth noting that understanding of telepsychology training should also advance efforts to articulate general competencies in telebehavioral health across a variety of professions (see e.g., Hilty et al.), 27 thus contributing to the development of a diverse and well-rounded telebehavioral health workforce as well as a clearly defined role for psychologists within that workforce.

Contributorship: AD designed the study, with input from all other authors, and AD collected all study data. AD, AM, and NW conducted data analyses. CM and MF contributed to interpretation of findings. AD wrote the first draft of the manuscript. All authors reviewed and edited the manuscript and approved the final version of the manuscript.

Declaration of conflicting interests: The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Telemedicine 2020 and the next decade

Telehealth for global emergencies: implications for coronavirus disease 2019 (COVID-19)

Report: telehealth will be a billion-dollar industry by

American telemedicine association practice guidelines for telemental health with children and adolescents

ATA practice guidelines for video-based online mental health services

Guidelines for the practice of telepsychology

Competency assessment toolkit for professional psychology

The competency movement within psychology: an historical perspective

The effectiveness of telemental health: a 2013 review

Innovations in exposure therapy for PTSD treatment

Telepsychology outcome research with children and adolescents: a review of the literature

Department of Health and Human Services

Future directions in the use of telemental health to improve the accessibility and quality of children's mental health services

Future directions in the design, development, and investigation of technology as a service delivery vehicle

Psychotherapy in 2022: a Delphi poll on its future

Association of State and Provincial Psychology Boards. PSYPACT: advancing the interjurisdictional practice of psychology: legislative updates

Department of Veterans Affairs. VA expands telehealth by allowing health care providers to treat patients across state lines

Implementation of telepsychology services in a university training clinic: process and lessons learned

Training the next generation of counseling psychologists in the practice of telepsychology

Looking to the future: integrating telemental health therapy into psychologist training

Ethical concerns for telemental health therapy amidst governmental surveillance

The practice of tele-mental health: ethical, legal, and clinical issues for practitioners

Competencies in professional psychology

Toward a common taxonomy of competency domains for the health professions and competencies for physicians

Necessary skills and knowledge for staff providing telehealth services

The need to implement and evaluate telehealth competency frameworks to ensure quality care across behavioral health professions

It's not just FaceTime: core competencies for the medical virtualist

A consolidated model for telepsychology practice

Lead users: a source of novel product concepts

Mixedmethods designs in mental health services research: a review

Using thematic analysis in psychology

Qualitative research in psychology: expanding perspectives in methodology and design

A framework of interprofessional telebehavioral health competencies: Implementation and challenges moving forward

Methods of rapid evaluation, assessment, and appraisal

Three approaches to qualitative content analysis

The COVID-19 telepsychology revolution: a national study of pandemicbased changes in U.S. mental health care delivery

A cube model for competency development: Implications for psychology educators and regulators

Rapid telepsychology deployment during the COVID-19 pandemic: a special issue commentary and lessons from primary care psychology training

Commission on Accreditation. Standards of accreditation for health service psychology

Flawed selfassessment: Implications for health, education, and the workplace

A new view: tele-intensive care unit competencies

Integrated telehealth care for chronic illness and depression in geriatric home care patients: the integrated telehealth education and activation of mood (I-TEAM) study

Continuing education training focused on the development of behavioral telehealth competencies in behavioral healthcare providers

Best practices for remote psychological assessment via telehealth technologies

Can a computer administer a Wechsler intelligence test?

Telesupervision: building bridges in a digital era

New developments in technology-assisted supervision and training: a practical overview

Supplemental material: Supplemental material for this article is available online.