key: cord-0941152-gw0bhlri
authors: Wagner, Liliana; Weitlauf, Amy S.; Hine, Jeffrey; Corona, Laura L.; Berman, Anna F.; Nicholson, Amy; Allen, William; Black, Michelle; Warren, Zachary
title: Transitioning to Telemedicine During COVID-19: Impact on Perceptions and Use of Telemedicine Procedures for the Diagnosis of Autism in Toddlers
date: 2021-06-04
journal: J Autism Dev Disord
DOI: 10.1007/s10803-021-05112-7
sha: df387059c565b2c3ced4ccbf239f4bdd52c626f6
doc_id: 941152
cord_uid: gw0bhlri

The COVID-19 pandemic has increased the use of telemedicine as an avenue to address the need for diagnostic clarification in young children at risk for autism spectrum disorder (ASD). Although preliminary research has supported the use of telemedicine for identifying ASD in toddlers, little is known about the experiences of practitioners attempting direct-to-home tele-assessment. We surveyed diagnostic providers regarding changes in practice behavior in the initial months of the COVID-19 pandemic and their perceptions of ASD tele-assessment. We also examined the use of the TELE-ASD-PEDS, a novel tool for ASD tele-assessment, in response to COVID-19 at seven sites across the country. Results support the clinical acceptability and diagnostic utility of ASD tele-assessment while also highlighting critical avenues of future investigation.

Disruptions related to COVID-19 have had far-reaching impacts on the families of individuals with intellectual and developmental disabilities (Baweja et al., 2021; Constantino et al., 2020a, b; Rose et al., 2020) . In a survey of over 800 caregivers, the majority of families polled reported a loss of at least some educational or healthcare service since the start of the pandemic (Jeste et al., 2020) . Many states, organizations, and providers have responded by offering telehealth services (Esper et al., 2020; Grossman et al., 2020; Perez et al., 2020) , including providers in the field of developmental and behavioral pediatrics (Fung & Ricci, 2020; Jang et al., 2021) . Specifically, telehealth solutions have been offered to address the pressing need for diagnostic clarification in young children deemed at risk for autism spectrum disorder (ASD), without which they may not be able to access critical early intervention services (Stahmer & Mandell, 2007) . Although recommendations from professional organizations, test publishers, and government agencies have been mixed regarding use of tele-assessment for ASD during COVID-19 (Krach et al., 2020) , other recent work supports the feasibility and acceptability of its implementation across circumscribed regions and groups (Alfuraydan, 2020; Dahiya et al., 2021; Juárez et al., 2018; Nazneen et al., 2015; Smith et al., 2017; Stainbrook et al., 2019) . To date there have been few studies of practitioners currently engaged in tele-assessment of ASD (henceforth referred to as 'ASD tele-assessment') and how their practice behavior has changed as a result of the pandemic. In this study, we strive to better understand provider perceptions and use of ASD tele-assessment during the pandemic.

ASD is characterized by deficits in social communication and social interaction, together with the presence of restricted, repetitive behaviors (American Psychiatric Association, 2013) . Traditional best practice models of assessment of ASD in young children commonly include face-to-face, play-based interactions and a comprehensive assessment of developmental functioning that may span several hours, if not days (Penner et al., 2018) . However, investigation of alternate ASD assessment modalities indicates that young children with clear features of ASD may be readily, accurately diagnosed using streamlined assessment models, including tele-assessment (Corona et al., 2020a, b; Juárez et al., 2018) . Prior to COVID-19, barriers such as geographic distance, lack of qualified providers, and cost in terms of time and financial resources (Bishop-Fitzpatrick & Kind, 2017; Mazurek et al., 2014) prompted some groups to investigate tele-assessment options. The onset of the COVID-19 pandemic, including suspension of in-person appointments and ongoing social distancing requirements in clinic settings, hastened tele-assessment uptake (Berger et al., 2021; Jang et al., 2021; Wagner et al., 2020) . Alfuraydan (2020) reviewed 10 studies investigating ASD tele-assessment from 2000 to 2019. Findings suggest that there is potential for telehealth models to improve access to assessment and diagnosis of ASD, especially in those individuals with clear symptoms. Other reviews have examined studies evaluating ASD tele-assessment specifically with young children located in low-income, underserved communities (Dahiya et al., 2020) as well as more broadly with children from 0 to 12 years of age (Dahiya et al., 2021) . Reviewed studies reported positive outcomes, including high levels of satisfaction with telehealth technology, sensitivity and specificity of ASD diagnosis, and interrater agreement.

Although past work documents the promise of ASD teleassessment, the majority of studies included have examined its use within well-controlled research settings. Given the potentially broad range of children's presenting concerns, as well as environmental factors that could affect telemedicine in clinical practice (such as a stable internet connection, a quiet home environment for assessment, and so on), there is a need to examine and describe the utility of ASD tele-assessment when it is deployed in real time, across a range of providers, within the context of everyday clinical practice. In an initial feasibility study in a direct-to-home setting, Talbott et al. (2020) adapted an in-person, lab-based screening measure for infants into a Level 2 screening protocol called the Telehealth Evaluation of Development for Infants (TEDI). Parents of 11 infants (ages 6-12 months) were coached to administer semi-structured behavioral probes over telehealth using a kit of play materials mailed to families before the evaluation. Results provided preliminary evidence for the feasibility, reliability, and acceptability of a novel screening tool designed to identify symptoms of ASD in very young children via telehealth.

In a second direct-to-home study, following suspension of in-person appointments in response to COVID-19, our team converted face-to-face diagnostic evaluations for toddlers into telemedicine-based assessments using a novel instrument specifically designed for identifying symptoms of ASD in young children via telemedicine (TELE-ASD-PEDS [TAP]; (Corona et al., 2020a Wagner et al., 2021) . The TAP was designed to elicit symptoms of ASD in children under 36 months of age using widely available materials within approximately 20 min of time. The TAP includes eight discrete, caregiver-led activities or social bids, including opportunities for interactive play, physical play routines, and requesting activities. It is intended to augment diagnostic decision-making (yes/no/maybe ASD) by expert clinicians within the context of a developmental evaluation.

From March-June 2020, internal data were analyzed across 204 telemedicine evaluations of toddlers between 16-36 months of age to better understand the use of the TAP as a measure for clinical use in direct-to-home ASD teleassessment (Wagner et al., 2020; Warren, 2020) . Findings provided preliminary support for use of the measure to "rule in" a diagnosis of ASD in a sample of clinically referred young children. Providers also reported high levels of comfort and satisfaction using the TAP for tele-assessment.

Although promising, these results reflect data from one institution from which the measure was developed, across a homogeneous group of expert diagnostic psychologists (i.e., ADOS-2 research reliable clinicians with previous experience with telehealth, across a single federated health system). Even as the TAP was being used and researched within its home institution, it underwent broad clinical distribution beginning in March 2020. Due to the widespread and pressing need for tools that could be used within clinical tele-assessment, we shared information on the development, administration, and scoring of the TAP through a series of no-cost, open-access, web-based trainings. Additionally, administration guidelines, record forms, informational brochures, and preliminary information regarding tool psychometrics (clinicaltrials.gov, NCT03847337) were made freely available on the institution's website. However, it was unclear how often the TAP was actually used in practice, with whom providers chose to use it, or how providers perceived its value as a clinical instrument.

The present study includes data from two sources. First, we surveyed a voluntary sample of providers who had participated in our initial TAP trainings on the tele-assessment of ASD. Survey topics included assessment procedures prior to and during the COVID-19 pandemic and provider comfort with tele-assessment. Second, we collected deidentified tele-assessment data from a subset of providers. These data included demographics of patients seen using the TAP, diagnoses provided, and provider diagnostic certainty. The aims of this study were to (1) document providers' use of tele-assessment prior to and during the COVID-19 pandemic, (2) understand providers' comfort with and perceptions of tele-assessment and the TAP, and (3) begin to examine clinical data from providers' use of the TAP to better understand with whom the TAP is being used, what diagnoses are given following the use of the TAP, and how frequently further assessment is recommended following use of the TAP. Given the primarily descriptive nature of this study, we did not have specific a priori hypotheses. We present these results below and discuss findings, limitations, and recommended future directions for understanding teleassessment of ASD during COVID-19 and beyond.

Participants. All individuals who registered to attend a TAP webinar (n = 2,403) were invited to complete the provider survey. A total of 202 providers completed the survey (8.4% response rate) between June 2020 and November 2020. Providers completing the survey included licensed clinical psychology providers, medical providers, experts in child development and behavior (speech language pathologists, behavior analysts), school psychologists, and graduate students (see Table 1 ). Clinicians had an average of fifteen years of experience (range = 1-40 years) working in a pediatric setting with children with ASD.

Measure. The provider survey was developed explicitly for the purposes of this study. Survey questions asked about clinical practice before and during COVID-19, as well as provider use and perceptions of tele-assessment and the TAP. Open-ended questions asked about perceived benefits and challenges related to tele-assessment.

Procedure. Two months after the initial TAP training webinars, all individuals who registered for the webinar (n = 2,403) were emailed an invitation to complete the online survey. No identifying information was collected as part of the survey. This project was approved by the medical center Institutional Review Board.

Data analysis. Provider perceptions of tele-assessment and the TAP were summarized using frequency counts. We examined changes to providers' autism evaluations during the pandemic by analyzing responses to a Redcap survey asking about specific practice behaviors before and during COVID-19. Open-ended responses on the benefits and challenges of tele-assessment and the TAP were analyzed qualitatively. The first author coded responses using themes reported in our prior work (Corona et al., 2020a, b; Wagner et al., 2020) . Responses were assigned to multiple categories, if multiple themes were present in the responses (e.g., if a provider mentioned both technology and distractions within the home environment as challenges related to tele-assessment). New themes were generated after review of responses that did not fit into existing categories.

Participating providers. Among those who attended initial TAP webinars, a subset of providers shared de-identified TAP rating forms from their unique clinical sites. A total of 22 providers across seven sites (located in OH, IN, VA, TN, NM, MN, CA) provided data from 197 telemedicine evaluations using the TAP from approximately June-October 2020. One provider from each site was responsible for collecting and uploading TAP rating forms. Provider-level demographic data were not collected. Measure. The TAP rating form allows providers to rate child behaviors observed during the administration of the TAP (described above). Following administration, providers rate seven behavioral symptom categories using both dichotomous (yes/no) and Likert (3 = behaviors characteristic of ASD clearly present; 2 = behaviors characteristic of ASD present at subclinical levels; 1 = behaviors characteristic of ASD not present) scoring procedures. A total score is obtained by summing the child's Likert-style scores for each key behavior. The TAP rating form also allows providers to document (1) clinician diagnostic impression (ASD vs. no ASD vs. unsure), (2) the diagnosis assigned following the appointment (open-ended), (3) a Likert rating of clinician diagnostic certainty (1 = completely uncertain, 2 = somewhat uncertain, 3 = somewhat certain, 4 = completely certain), and (4) whether further in-person testing was recommended (yes vs. no).

Procedure. Several providers who had attended one or more TAP webinars contacted the first, third, or final author and offered to share deidentified clinical data. Providers self-selected into this process, rather than being specifically recruited to share data. To facilitate data sharing, providers were sent individualized links to upload deidentified TAP forms into a HIPAA-compliant institutional folder accessible only to the provider and research team. Materials uploaded included only TAP rating forms, without identifying patient information.

Data analysis. Descriptive statistics were used to report (1) provider diagnostic impressions, (2) provider diagnostic certainty, and (3) referral for further assessment following tele-assessment using the TAP. One-way analysis of variance (ANOVA) was used to examine differences in TAP scores and provider diagnostic certainty among children as a function of provider diagnostic impression. Multiple regression was used to examine relations among child age, TAP score, and provider diagnostic certainty.

The majority of clinicians reported that they were actively engaged in making clinical diagnoses of ASD prior to COVID-19, with 96% of the sample reporting expertise in the assessment of children age birth to three years and 94% of the sample in children four to six years. Thirty-five providers (17%) reported using telemedicine in their clinical practice prior to the suspension of in-person appointments, for a combination of assessment (n = 12), therapy (n = 18), and research-related activities (n = 2).

We examined self-reported changes in provider practice behavior pre-and during-COVID-19. Most providers (78%) reported using telemedicine for diagnostic services during COVID-19, an increase from the 6% who conducted teleassessment prior to COVID-19. An exact McNemar's test determined that there was a statistically significant difference in the proportion of individuals conducting tele-assessment before and during COVID-19 (p < 0.0001). Providers also reported changes regarding the measures used as part of their diagnostic evaluations before and during COVID-19. For example, before COVID-19, the majority of providers reported using cognitive assessments (76%), measures of adaptive functioning (82%), emotional and behavioral rating scales (66%), and paper-and-pencil screeners (55%) as part of their assessments. However, fewer providers reported using these measures during COVID-19 (see Table 2 ). Further, 92% of providers reported using the Autism Diagnostic Observation Schedule, Second Edition (ADOS-2) as part of their evaluations prior to COVID-19. Requirements for standardized activities and materials preclude the use of the ADOS-2 for remote assessment (Marshall, 2020) .

Finally, providers were asked to report the number of visits included within their diagnostic appointments before and during COVID-19. Eighteen percent of providers reported conducting their diagnostic evaluations in one visit during COVID-19 (vs. 21% before COVID-19) and 37% of providers reported conducting their diagnostic evaluations across one to three visits during COVID-19, compared to 30% before COVID-19.

The majority of providers reported feeling comfortable completing tele-assessments with toddlers, making diagnoses of ASD following tele-assessment, coaching caregivers through administration of the TAP, and providing feedback and recommendations to families via telemedicine (See Table 3 ). Most providers also reported feeling that it is appropriate for a toddler to receive a diagnosis of ASD following teleassessment. Ninety-three percent of providers reported the TAP was either very helpful or helpful in providing information to guide clinical decision-making. Of the providers using the TAP for diagnostic evaluations, 85% plan to continue using the instrument after normal clinical practice resumes.

Providers also provided qualitative feedback on various aspects of tele-assessment, including perceived benefits and challenges of tele-assessment in general and specific feedback on the TAP. Several themes emerged with regard to the benefits of tele-assessment, many of which were consistent with themes identified in our earlier work (Wagner et al., 2020) . For example, the most frequently reported benefit of tele-assessment was the increased access for families who previously faced multiple barriers to attending a traditional, in-person evaluation (n = 85). Many providers also reported that evaluating children in their home environment yielded valuable clinical information (n = 75), increased child comfort (n = 22), and allowed for increased caregiver involvement with the evaluation process (n = 12). Providers also acknowledged the benefit of continuing to provide services during COVID-19-related restrictions (n = 20). Novel themes not reported in prior work included mention of logistical benefits of tele-assessment, including increased flexibility with scheduling, increased efficiency, and the opportunity to have multiple providers attend the visit (n = 17). Many of the themes that emerged related to tele-assessment challenges were also consistent with our earlier findings (Wagner et al., 2020) . Many providers (n = 87) reported technology-related challenges, including poor wireless connection, dropped calls, and inconsistent audio, all leading to difficulty obtaining adequate clinical observations. Providers also reported caregiver-related challenges, saying some caregivers had difficulty following directions during assessment activities, keeping the child in the frame, or appeared anxious or unsure of the tele-assessment process (n = 59). Some providers also reported challenges related to the home environment (e.g., distractions, presence of siblings, availability of play materials; n = 21).

New themes emerging from providers' responses included challenges related to child factors (n = 27). Specifically, providers reported that tele-assessment was difficult when assessing children who were older, medically complex, had flexible language, or more subtle symptoms of ASD. Finally, some providers reported that it was challenging to adjust their typical evaluation process (e.g., forgo a physical exam or cognitive assessment, adjust feedback) and find new, evidence-based tools to use (n = 33). Providers also identified some challenges specific to the TAP, including the failure of some insurance companies to recognize the test (n = 3), insufficient information on scoring (n = 5), and caregiver difficulty following the prompts (n = 33). A small number of providers expressed that the TAP is too brief for the child to show sufficient symptoms (n = 2) and does not probe for or measure a sufficient range of skills and ASDrelated symptoms (e.g., imitation, pretend play, restricted and repetitive interests and behaviors, n = 3).

Providers from the seven sites described above uploaded TAP data for 197 children ranging from 14 to 60 months of age (mean age = 31.70 months, SD = 8.60 months; see Table 4 ). Race and ethnicity were provided for a subsample of the children included in the analysis (74% and 27%, respectively). No other child-level data were collected, nor was information collected regarding referral processes.

Of the 197 children on whom we received tele-assessment evaluation data from the seven external sites, 40% received diagnoses of ASD following evaluation (ASD Diagnosed). For a small percentage of children (4%), providers indicated that children likely met criteria for ASD based on diagnostic impression (i.e., selected 'ASD' in forced choice diagnostic impression), but deferred a diagnosis pending further, in-person testing (ASD Suspected). Providers reported being unsure of whether children met criteria for ASD in 11% of evaluations, in all but one case deferring a diagnosis and referring for in-person testing (Diagnosis Uncertain). Finally, 46% of children did not meet criteria for ASD (No ASD) and were assigned diagnoses of Developmental Delay or speech and language delay (n = 33), ADHD (n = 1), Unspecified Neurodevelopmental Disorder (n = 4), or disruptive behavior disorders (n = 4). Of these children, 24% were recommended for further in-person evaluation for diagnostic clarification. Mean TAP scores and providers' mean ratings of diagnostic certainty are presented in Table 5 .

One-way analysis of variance (ANOVA) was used to compare TAP total scores across diagnostic groups. Due to violation of the assumption of homogeneity of variances (Levene's statistic = 3.426, p < 0.05), Welch's ANOVA was used (Delacre et al., 2019) . The omnibus test of the model indicated that there were significant differences among TAP scores (Welch's statistic (3, 24.7) = 180.62, p < 0.05). Post-hoc comparisons using the Games-Howell procedure indicated significant differences in TAP scores across groups (all p-values < 0.01; See Table 6 ), with the exception of ASD Diagnosed vs. ASD Suspected. This is unsurprising, as children in the ASD Suspected group presumably also showed a high level of ASD symptoms during testing. Children who received an ASD diagnosis following tele-assessment received the highest TAP scores (m = 16.67, SD = 2.62), followed closely by children for whom providers suspected ASD (m = 16.57, SD = 1.99), then children for whom there was diagnostic uncertainty (m = 12.71, SD = 2.49). Children who did not receive a diagnosis of ASD received the lowest TAP scores (m = 8.50, SD = 1.95).

A second ANOVA was conducted to examine providers' diagnostic certainty across diagnostic group (see Table 7 ). Again, Welch's ANOVA was used due to violation of the homogeneity of variance assumption (Levene's statistic = 5.61, p < 0.05). Welch's test indicated significant differences in diagnostic certainty (Welch's statistic (2, .80) = 28.32, p < 0.01). The Games-Howell procedure was used to examine post-hoc comparisons across three of the groups. Comparisons were not examined for the ASD Suspected group, as there was no variance in the diagnostic certainty scores across the providers. Post-hoc tests showed that providers reported the highest levels of diagnostic certainty when ruling out a diagnosis of ASD (m = 3.57, SD = 0.56) than for all other groups (all p-values < 0.05). Providers reported higher diagnostic certainty for children who received an ASD diagnosis (m = 3.28, SD = 0.88) than children in the Diagnosis Uncertain group (m = 2.24, SD = 0.77). Finally, multiple linear regression was used to examine the association of child sex, age, and TAP scores (sum of Likert score, score ≥ 11) with provider diagnostic certainty. The model accounted for 17% of the variance in provider diagnostic certainty [F (4,190) = 9.94, p < 0.01].

The landscape of ASD-related diagnostic services has undoubtedly changed in the context of restrictions put in place due to COVID-19. Many providers have turned to telemedicine as an avenue for continued service delivery despite limited standardized guidance regarding the remote elicitation and observation of ASD symptoms. Although preliminary research has supported the use of telemedicine for identifying ASD in toddlers (Juárez et al., 2018; Nazneen et al., 2015; Smith et al., 2017; Stainbrook et al., 2019) , many of these studies have been deployed in the context of controlled laboratory settings as opposed to direct-to-home models, and little is known about the experiences of community practitioners attempting direct-to-home tele-assessment.

In the current paper, we first present our findings from a survey of a diverse group of diagnostic providers who were polled about specific changes in their practice behavior in the initial months of the COVID-19 pandemic. Providers were also asked to provide their perceptions regarding the feasibility, appropriateness, strengths, and challenges of ASD tele-assessment. In addition, we built upon existing research on the TELE-ASD-PEDS (TAP; Corona et al., 2020a, b; Wagner et al., 2020) and examined how it was used within the community in response to COVID-19 restrictions across 197 tele-evaluations from seven unique sites across the country. Results provide further support for the clinical acceptability and diagnostic utility of tele-assessment for ASD while also highlighting critical avenues of future investigation.

The onset of COVID-19 resulted in a significant increase in the use of tele-assessment of ASD. Provider use of tele-assessment for ASD diagnostic evaluations rose from 6% pre-pandemic to 78% at the time of the survey.

Tele-assessments were also spread over more sessions, with more providers seeing children 2 or 3 times rather than one. It is possible that this change is provider-driven, with providers wanting to increase their unique observations of the child to accommodate any potential loss of information that might have occurred with the transition away from in-person visits. However, it is equally possible that caregivers, many of whom are managing working from home and caring for multiple children, are unable to devote sufficient time for a single uninterrupted evaluation. Understanding the child and environmental variables that influence session structure and length represents an important target for future work.

Providers also reported changes to their assessment batteries during COVID-19. Fewer providers reported using measures of cognitive functioning, adaptive skills, language, and emotional/behavioral functioning during teleassessments. This change could be attributed to a lack of guidance and/or comfort surrounding use of these measures via telehealth (Krach et al., 2020) , limited access to validated telehealth tools assessing such within early childhood (Talbott et al., 2020) , or characteristics of the patients that providers chose to see via telehealth (rather than postponing for in-person evaluation) (Sutherland et al., 2018) . Further, most providers reported using the ADOS-2 in their diagnostic evaluations prior to COVID-19. Because the ADOS-2 cannot be administered via telemedicine due to its highly standardized administration procedures and materials, providers by necessity have had to seek alternatives, particularly during the first few months of adjustment to safer-at-home orders and safety protocols within clinical spaces. In addition to the TAP, other tele-assessment tools have emerged and gained prominence during COVID-19, due to their ability to be administered over a telehealth platform [BOSA, (Lord et al., 2020) ; MIGDAS-2, (Monteiro & Stegall, 2018) ]. Encouragingly, these options give providers important choices and flexibility when designing tele-assessment procedures for future use. Despite the changes to their diagnostic evaluations, most providers reported high levels of comfort and satisfaction with all aspects of the tele-assessment process, including areas that might be more unfamiliar, such as coaching caregivers through task administration and using novel measures like the TAP.

Over half of the providers (56%) reported using the TAP in their ASD tele-assessments during COVID-19, nearly all of whom reported that the measure provided valuable information to guide their decision-making. Further, 85% of the providers using the TAP plan to continue to use the measure as restrictions related to COVID-19 are lifted. These high levels of comfort and satisfaction with the TAP from a novel, more heterogeneous sample of providers is consistent with the findings from the providers polled at our institution and lends further support for TAP use and value as a clinical assessment tool.

Providers reported several benefits to tele-assessment. Many providers felt the opportunity to observe children in their homes, interacting with familiar caregivers and play materials, yielded unique and valuable clinical information and allowed caregivers to be more involved than in traditional, in-person evaluations. Previous research suggests that caregivers enjoy taking an active role in assessment (Corona et al., 2020a, b) , although additional research is needed to assess caregiver perceptions of tele-assessment conducted in the home. Some providers reported that their visits were more productive and efficient over telemedicine as the toddlers did not have to tolerate multiple transitions or adjust to a new environment. Providers also reported increased flexibility with respect to scheduling and the ability to easily include multiple providers in a visit without overwhelming the family or child. Finally, many providers reported that tele-assessment increased access to diagnostic services for families who difficulty attending in-person evaluations before COVID-19. This includes, for example, families with transportation barriers, those living in rural areas, or families who had difficulty taking time off work or securing childcare for siblings during the appointment. This has significant implications for the use of tele-assessment even beyond the pandemic.

Importantly, providers also identified challenges with tele-assessment. The most frequently reported challenges were related to technology problems, including dropped calls, spotty audio and video, and caregiver difficulty operating their devices or keeping their child on the screen. These difficulties sometimes impacted the quality of the observations, with providers unable to adequately assess more subtle behaviors such as coordinated eye contact. Providers also reported challenges regarding distractions in the home environment and caregiver difficulty following their instructions and prompts. These challenges are consistent with prior research examining barriers related to tele-assessment (Juárez et al., 2018; Wagner et al., 2020) . Some providers suggested that telemedicine-based assessment was more challenging with children who were older and had more subtle or complex presentations. Finally, a subset of providers expressed that it was challenging to adjust their evaluations to identify and learn new measures and forgo certain aspects they had relied upon during in-person evaluations. This challenge likely reflects the rapid response to COVID-19, with many providers quickly adjusting their practice with little precedent or guidance, rather than reflecting an inherent barrier in tele-assessment.

Analysis of deidentified TAP data revealed that, relative to our initial research-based TAP validation sample (Corona et al., 2020a, b) and patients seen within our institution during the pandemic (Wagner et al., 2020) , the current sample of children was more evenly distributed across diagnostic category, with the majority of children not receiving a diagnosis of ASD. Mirroring findings from our internal data, TAP scores were highest for those children diagnosed with ASD during tele-assessment, followed closely by children with whom ASD was suspected but deferred pending further, in-person testing. This finding is perhaps unsurprising, as the observations gained through the TAP activities were used to inform diagnostic decision-making. Further, TAP scores were associated with providers' diagnostic certainty. Providers were most confident in their diagnostic impressions when ruling out a diagnosis of ASD, only recommending in-person testing for 24% of those children. This is a departure from findings from our internal data, in which providers reported the highest levels of certainty when assigning an ASD diagnosis. This finding provides preliminary support for use of this measure to not only rule in but also rule out a diagnosis of ASD. On a broader scale, these findings suggest that, in the context of typical models of ASDrelated care, providers were able to classify a large number of children with certainty via tele-assessment. Thus, models of care utilizing this mechanism could be conceptualized to more accurately route families to services and offer direct tele-consultation as a meaningful option for many families.

While this study provides the first systematic evaluation of provider use and potential clinical accuracy of the TAP, there are several major limitations that are important to acknowledge to best understand the contribution of this work. First, this was not a planned research study. The COVID-19-related restrictions and the lack of validated measures for the tele-assessment of ASD prompted us to deploy the TAP for clinical use while it was still in development. Thus, materials had not been finalized; data from initial trials were not yet published; and many characteristics of optimal rigor were not present. There was ad hoc recruitment of providers and a somewhat limited response rate relative to the total number of providers who attended the initial trainings on the TAP (i.e., it is likely those with positive experiences using the TAP were more likely to respond). As a result, value to the broader community remains in question.

Second, we have limited data from families regarding clinical outcomes following tele-assessment. Longitudinal studies are needed to ensure that those families who receive a diagnosis over telemedicine are accessing ASD-related services at the same pace and level as those receiving their diagnoses in-person. Finally, we still do not understand all the characteristics of the instrument. It will be important to examine the measure's use in a racially, ethnically, and geographically diverse sample to continue to understand for whom the measure works best. In addition, though the TAP was designed and has been studied primarily with younger children, providers in the current study reported using the TAP with children up to 60 months of age. Additional work is needed to determine the clinical utility and psychometric properties of the TAP within the pre-school age range.

The importance of understanding provider perceptions of the benefits and barriers to ASD tele-assessment extends beyond the immediate and relatively short-term impacts of COVID-19. Delays in accessing ASD-specific diagnostic and intervention services, specifically for underserved and under-resourced populations, were present long before the pandemic (Constantino et al., 2020a, b; Durkin et al., 2017; Wallis et al., 2020; Zuckerman et al., 2017 Zuckerman et al., , 2018 and will undoubtedly continue after its cessation without significant, systemic intervention. Telemedicine has the potential for increasing access to care for those families who might otherwise face delays. The rapid onset and pervasive impact of COVID-19 presented a unique opportunity to evaluate changes in practice behavior and utilization of novel tools and technologies. Going forward, understanding the impact, relevance, and appropriateness of tele-assessment across phenotypes, regions, and providers will be an important target of research and dialogue within the diagnostic community.

Use of telehealth for facilitating the diagnostic assessment of autism spectrum disorder (ASD): A scoping review

Diagnostic and statistical manual of mental disorders

COVID-19 pandemic and impact on patients with autism spectrum disorder

Characterizing available tools for synchronous virtual assessment of toddlers with suspected autism spectrum disorder: A brief report

A scoping review of health disparities in autism spectrum disorder

Timing of the diagnosis of autism in African American children

The impact of COVID-19 on individuals with intellectual and developmental disabilities: Clinical and scientific priorities

TELE-ASD-PEDS: A telemedicine-based ASD evaluation tool for toddlers and young children

Toward novel tools for autism identification: Fusing computational and clinical expertise

Parent perceptions of caregiver-mediated telemedicine tools for assessing autism risk in toddlers

A systematic review of technological approaches for autism spectrum disorder assessment in children: Implications for the COVID-19 pandemic

A systematic review of remote telehealth assessments for early signs of autism spectrum disorder: Video and mobile applications

Taking parametric assumptions seriously: Arguments for the use of Welch's F-test instead of the classical F-test in one-way ANOVA

Autism spectrum disorder among US children (2002-2010): Socioeconomic, racial, and ethnic disparities

Rapid systemwide implementation of outpatient telehealth in response to the COVID-19 pandemic

Rethinking 'essential' and 'nonessential': the developmental paediatrician's COVID-19 response

Rapid implementation of virtual neurology in response to the COVID-19 pandemic

Diagnostic evaluations of autism spectrum disorder during the COVID-19 pandemic

Changes in access to educational and healthcare services for individuals with intellectual and developmental disabilities during COVID-19 restrictions

Early identification of ASD through telemedicine: potential value for underserved populations

Testing our children when the world shuts down: Analyzing recommendations for adapted tele-assessment during COVID-19

Brief Observation of Symptoms of Autism (BOSA)

Remote diagnosis, support could aid families during lockdown. Spectrum

Age at first autism spectrum disorder diagnosis

MIGDAS-2) Monteiro Interview Guidelines for Diagnosing the Autism Spectrum

A Novel system for supporting autism diagnosis using home videos: Iterative development and evaluation of system design

Systematic review of clinical guidance documents for autism spectrum disorder diagnostic assessment in select regions

Telemedicine in behavioral neurology-neuropsychiatry: Opportunities and challenges catalyzed by COVID-19

The effect on and experience of families with a member who has intellectual and developmental disabilities of the COVID-19 pandemic in the UK: developing an investigation

Investigating the accuracy of a novel telehealth diagnostic approach for autism spectrum disorder

State infant/toddler program policies for eligibility and services provision for young children with autism. Administration and Policy in Mental Health

Measuring the service system impact of a novel telediagnostic service program for young children with autism spectrum disorder

Telehealth and autism: A systematic search and review of the literature

Brief Report: Preliminary feasibility of the TEDI: A novel parent-administered telehealth assessment for autism spectrum disorder symptoms in the first year of life

Use of the TELE-ASD-PEDS for autism evaluations in response to COVID-19: Preliminary outcomes and clinician acceptability

Unpublished manuscript

Adherence to screening and referral guidelines for autism spectrum disorder in toddlers in pediatric primary care

Autism Spectrum Disorder Assessment in Toddlers

Disparities in familiarity with developmental disabilities among low-income parents

Disparities in diagnosis and treatment of autism in Latino and Non-Latino white families

The authors wish to acknowledge and thank the parents and children whose data contributed to this work, as well as the research staff at the Vanderbilt Kennedy Center Treatment and Research Institute for Autism Spectrum Disorders.

Author Contributions LW wrote the manuscript with support from LLC, ASW, JH, and ZW. LW performed the statistical analyses. AFB and AN created and maintained the database for patient and provider data. WA and MB contributed data from telehealth evaluations and provided their perceptions of the tele-assessment model. Ethical Approval All procedures performed in this study were in accordance with the ethical standards of the institutional research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. Analysis of existing clinical data was approved by the Institutional Review Board at Vanderbilt University Medical Center.