key: cord-0714366-kvhypwwx
authors: Reisinger, Debra L.; Hines, Elesia; Raches, Christine; Tang, Qing; James, Cristina; Keehn, Rebecca McNally
title: Provider and Caregiver Satisfaction with Telehealth Evaluation of Autism Spectrum Disorder in Young Children During the COVID-19 Pandemic
date: 2022-05-17
journal: J Autism Dev Disord
DOI: 10.1007/s10803-022-05576-1
sha: d2b13e7d57813ca5c43bc3cd4ce5d56fc1b4f5f6
doc_id: 714366
cord_uid: kvhypwwx

The present study examines provider and caregiver satisfaction with telehealth evaluation of autism spectrum disorder (ASD) in young children during the coronavirus SARS-CoV-2 (COVID-19) pandemic. A telehealth model of ASD evaluation was implemented with 308 children ages 14 to 78 months between May 2020 to June 2021. Data were gathered from electronic health records, autism-specific telehealth diagnostic tools, and post-evaluation surveys. Overall, the majority of providers and caregivers were satisfied with telehealth ASD evaluation. Multiple variables were associated with ratings of satisfaction, differing by providers and caregivers. Findings have important implications for the feasibility and acceptability of telehealth ASD evaluations, in addition to key factors to consider in optimizing and sustaining telehealth evaluations beyond the COVID-19 pandemic.

telehealth assessments, providing ASD diagnoses based on telehealth evaluation results, and giving feedback and recommendations to families via telehealth (Wagner et al., 2020 (Wagner et al., , 2021 . The use of telehealth during the pandemic has also resulted in greater appreciation for the value of evaluating a child's behavior in their home environment (Jang et al., 2021) and increasing caregiver involvement in the evaluation process (Wagner et al., 2021) .

Utilization of telehealth for ASD evaluations has the potential to provide significant advantages to families compared to in-person evaluations. For example, previous work has reported a potential cost savings of over thirtyfive dollars for families related to travel alone, not considering additional expenses such as meals, hotels, childcare, or potential lost wages for families in rural or underserved areas (Reese et al., 2015) . More recently, a provider survey indicated telehealth increases access to ASD diagnostic evaluations for children and families who may have had difficulty attending in-person appointments prior to COVID-19 due to transportation and other barriers such as taking time off work and finding childcare for siblings (Wagner et al., 2021) . Notably, parents shared that if telehealth were not an option, they would have waited to travel to a diagnostic center, which was likely to delay the diagnosis, subsequently impacting entry into early intervention services (Juárez et al., 2018) .

Caregivers have also reported high levels of satisfaction with ASD and neurodevelopmental telehealth diagnostic evaluations Corona et al., 2021; Juárez et al., 2018; McNally Keehn et al., in press; Reese et al., 2013; Talbott et al., 2020) . Prior to the pandemic, Juárez et al., (2018) examined caregiver satisfaction related to receiving a diagnosis of ASD via telehealth and found parents reported high levels of satisfaction with the telehealth evaluation method. In a recent study utilizing the TELE-ASD-PEDS and TELE-STAT, caregivers found the telehealth screening process to be acceptable and comfortable . The majority of parents provided positive feedback about their involvement in the assessment and indicated that their child may have felt more comfortable playing with a parent compared to an unfamiliar adult. Further, most parents reported the provider's instructions were easy to follow, they understood what they were asked to do, and they felt comfortable being a part of the evaluation. Other factors associated with caregiver satisfaction include convenience, privacy, and reduced travel burden (Juárez et al., 2018; McNally Keehn et al., 2021) . These findings contrast with some previous literature on caregiver satisfaction with receiving an ASD diagnosis via traditional, in-person evaluations (Crane et al., 2016; Howlin & Moore, 1997) . Specifically, caregiver dissatisfaction with the traditional process has been linked to factors such as the wait time for receiving a diagnosis, Reese et al., 2015; Stainbrook et al., 2019) . Three primary methods of remote assessment of ASD symptoms have been developed and studied in recent years. One method includes asynchronous or "store-and-forward" telehealth, in which caregivers record in-home videos of their child's behavior and upload the videos for providers to view as part of the diagnostic assessment (Nazneen et al., 2015; Kanne et al., 2018; Sutantio et al., 2021) . Another method utilizes audiovisual technology allowing a remote provider (i.e., a clinician with expertise in ASD) to interview caregivers directly and observe a trained on-site provider administer a standardized ASD screening tool (e.g., Screening Tool for Autism in Toddlers and Young Children [STAT; Stone et al., 2008] ). For example, Juárez et al., (2018) used telehealth to observe the administration of the STAT and found that trained providers correctly identified 80% of children who later received a diagnosis of ASD via a comprehensive in-person evaluation. The third method of telehealth assessment of ASD symptoms focuses on real-time, remote administration of caregiver mediated ASD assessment tools in which the provider coaches caregivers to complete interactive activities with their child as part of a standardized assessment protocol. For example, the TELE-ASD-PEDS is a clinical assessment tool that was developed pre-pandemic for caregiver-mediated telehealth evaluations of ASD and has been a valuable tool during the COVID-19 pandemic (Wagner et al., 2020 (Wagner et al., , 2021 .

Findings from the aforementioned studies provide preliminary support for the feasibility, utility, and accuracy of telehealth ASD diagnostic evaluations for young children. A recent review examining various telehealth approaches used in ASD diagnostic evaluations suggests the use of telehealth for ASD diagnosis was acceptable to both families and providers (Alfuraydan et al., 2020) . However, the acceptance of telehealth diagnostic evaluations for ASD and the development of robust ASD telehealth diagnostic tools remains limited. This is likely due to a myriad of reasons, including legal and ethical considerations (Quigley et al., 2019) , the difficulties associated with making a diagnosis in a highly heterogenous population, and the necessity of observation and interaction with an experienced provider (McNally Keehn et al., 2021) . Nonetheless, research conducted since the onset of the COVID-19 pandemic continues to support the feasibility and acceptability of telehealth ASD screening and evaluations in young children (Talbott et al., 2020; Wagner et al., 2020) .

Studies conducted during the pandemic examining provider satisfaction with neurodevelopmental telehealth evaluation in children have revealed that most providers are satisfied and comfortable with the process (McNally Keehn et al., 2021; Wagner et al., 2021) . Specifically, providers have reported feeling comfortable with completing experience. Specifically, caregivers and providers have cited technology barriers with telehealth, including audio/ video distortions and internet connection issues Wagner et al., 2020) . Further, some caregivers have indicated concern about the small size of their device screen and their child's interest in touching the camera and/ or screen . In addition, some caregivers encounter difficulty operating their devices during the evaluation or keeping their child on the screen while engaging in assessment activities (Wagner et al., 2021) . Providers have also reported challenges related to the family's home environment, including lack of play materials and distractions. While providers regularly experience barriers related to technology and family challenges, a recent provider survey indicated that these problems were rarely rated as disruptive to the evaluation process (McNally Keehn et al., 2021) .

At the onset of the pandemic, our large outpatient clinic at Children's Hospital quickly shifted to providing telehealth diagnostic ASD evaluations in order to continue serving patients and maintain access to ASD diagnostic services. The present study seeks to expand the current literature on provider and caregiver satisfaction with telehealth ASD diagnostic evaluations completed with young children and their families during the COVID-19 pandemic. Specifically, we examine caregiver and provider satisfaction, their relationship, and separately by provider (e.g., psychologist/ pediatrician). Based on previous work, we hypothesized that both caregivers and providers would be satisfied with telehealth ASD evaluations, satisfaction ratings between caregivers and providers to be positively correlated, and caregiver satisfaction to relate similarly regardless of provider type. Further, we explored associations between caregiver and provider satisfaction and child demographics, diagnostic outcomes, and telehealth evaluation factors. Although the present literature on satisfaction with ASD telehealth evaluations is limited, we hypothesized that caregiver satisfaction would be related to travel time saved, diagnostic outcome, and technical or setup difficulties. Similarly, we hypothesized that provider satisfaction would be related to child age, diagnostic outcome, and technical and setup difficulties. This study expands on the growing literature using a large sample across both caregivers and providers, including both psychologists and pediatricians, in addition to exploring a broader range of factors related to satisfaction to build on the feasibility, utility, and acceptability of telehealth methodology for diagnostic ASD evaluations in young children during an unprecedented global pandemic and in the future. the way in which providers deliver an ASD diagnosis, and perceived lack of support during and after a diagnosis is provided (Crane et al., 2016) . Although ASD telehealth diagnostic evaluations may improve the wait time for receiving a diagnosis, additional work is needed to explore the impact on satisfaction with providers and perceived support families obtain post-diagnosis following a telehealth diagnostic evaluation.

It is important to note that studies reporting high levels of satisfaction with ASD telehealth diagnostic evaluations have also highlighted potential drawbacks of the telehealth (42-124) Note. ASD = autism spectrum disorder; ASD Present = an ASD diagnosis was given; ASD Absent = an ASD diagnosis was not given; ASD Unsure = the clinician was not sure about whether the child had ASD or not at the time of the evaluation; Vineland-3 scores are reported as standard scores (SS) of telehealth for neurodevelopmental evaluation; however, all were trained in the TELE-ASD-PEDS prior to the start of the study. Specifically, all providers participated in the online TELE-ASD-PEDS training webinars offered through Vanderbilt University (https://vkc.vumc.org/vkc/triad/ tele-asd-peds). Further, providers participated in regularly scheduled peer supervision and consultation meetings led by a clinical psychologist (and Certified Autism Diagnostic Observation Schedule™, Second Edition [ADOS-2; Lord et al., 2012] Independent Trainer) who engaged in additional, ongoing trainings and consultation with the TELE-ASD-PEDS developers.

Provider Telehealth Evaluation Survey. A survey to capture provider experiences with providing telehealth neurodevelopmental evaluations was developed for this study. Survey items included information about evaluation components (including use of assessment tools), whether any

A total of 308 children between the ages of 14 and 78 months (M = 37.9, SD = 10.9) referred by their primary care provider (PCP) or specialist within the Riley Hosptial for Children completed a telehealth evaluation. Data were drawn from a larger study offering telehealth neurodevelopmental evaluations between May 2020 and June 2021 at the Riley Hospital for Childrenoutpatient neurodevelopmental evaluation clinic. Children and their families that required a language interpreter were excluded. See Table 1 for child and family demographics.

Providers included a total of seven licensed psychologists and four pediatricians with neurodevelopmental training who completed the neurodevelopmental telehealth evaluations. All providers had significant experience in the evaluation of children with neurodevelopmental disabilities (mean years of experience = 13; range = 2-35 years). Prior to COVID-19, none of the providers had experience in the use 4.6 (0.8) 1 (0.71%) 4 (2.84%) 11 (7.80%) 25 (17.73%) 100 (70.92%) Note. 1 N=287; Items rated on Likert scale (1 = Not satisfied; 5 = Very satisfied); 2 N=141; Items rated on Likert scale (1 = Strongly Disagree; 5 = Strongly Agree history and ASD symptom-specific interview). The trained provider leads the child's caregiver through a series of eight discrete activities or social bids, including opportunities for interactive play, physical play routines, and requests using toys readily available in the child's home. Following administration, providers complete a rating form on seven different behaviors (e.g., socially directed speech and sounds, eye contact, unusual vocalizations, unusual or repetitive play or body movements, unusual sensory exploration) using both dichotomous (yes/no) and Likert (3 = behaviors characteristic of ASD clearly present; 2 = behaviors characteristic of ASD present at subclinical levels; 1 = behaviors characteristic of ASD not present) scoring procedures. A total score is calculated by summing the seven Likert scale item scores. This total score was utilized in the present study to quantify ASD symptom severity in the participants, with higher scores being indicative of greater ASD symptoms.

Adaptive Behavior. The Vineland Adaptive Behavior Scales, Third Edition -Comprehensive Parent/Caregiver Form (Vineland-3) was used to assess overall adaptive functioning of child participants (Sparrow et al., 2016) . The Vineland-3 was administered via remote link emailed to caregivers prior to their evaluation to obtain the child's overall adaptive functioning from four subdomains (communication, daily living skills, socialization, and motor skills). The Vineland-3 provides an adaptive behavior composite score (mean of 100 and standard deviation of 15) that was used in the current study in addition to the subdomain standard scores. A subset of caregivers completed the questionnaire due to adding the measure several months into the start of the study.

Due to COVID-19, children previously scheduled or on the waitlist for an in-person ASD evaluation who were less than 48 months of age were contacted by the clinic scheduling team to offer the telehealth evaluation. For the present study, we also included children outside of the TELE-ASD-PEDS age range (up to age 6:11) who presented with language deficits ranging from no functional word use to occasional simple 2-word phrases. If a child was between 4 and 6 years of age, a language screening was conducted by a speech-language pathologist to determine if the child met the language inclusion criteria. A verbal script (similar to the examples provided in the TELE-ASD-PEDS training: https://vkc.vumc.org/vkc/triad/tele-asd-peds) was developed for the scheduling team in order to ensure standardized delivery of information regarding telehealth evaluation appointments. Caregivers who consented to telehealth were provided verbal and written instructions on how to prepare diagnosis was communicated to the family (e.g., ASD, developmental delay, language/communication delay [e.g., mixed receptive-expressive language disorder], unspecified behavioral/emotional disorder, etc.), ASD diagnostic outcomes and provider certainty, evaluation next steps, technology (e.g., audio distortions/limitations, video distortions/ freezing) and family/home setup barriers (e.g., child not compliant, distracting family members/siblings/pets/home environment, placement of the video camera, inadequate toys/materials) encountered, and ratings of satisfaction with the telehealth evaluation. Two provider satisfaction items were rated on a dichotomous yes/no scale and two items were rated on a five-point Likert scale (e.g., Not satisfied [1] -Very satisfied [5]; see Table 2 for the list of satisfaction questions asked). These two scores were averaged to create an overall provider satisfaction score. Some survey items were adapted from Juárez et al., (2018) . Due to small sample size in some cells, provider ratings of diagnostic certainty (completely certain, somewhat certain, somewhat uncertain, or completely uncertain) were dichotomized (certain or uncertain) for analysis.

Caregiver Telehealth Satisfaction Survey. A survey measuring caregiver experience and satisfaction with their child's telehealth evaluation was adapted from Juárez et al., (2018) for the present study. Ten satisfaction items were rated on a five-point Likert scale [e.g., Strongly Disagree (1) -Strongly Agree (5)]). A summary score was calculated by averaging the ratings from all ten items (range 0-5) to represent overall satisfaction (See Table 2 for the list of satisfaction questions asked). Further, to examine differences in satisfaction with the provider versus the method of evaluation, two subscales were created from the ten caregiver satisfaction items: overall provider satisfaction (mean of four items) and overall satisfaction with the telehealth evaluation provider (mean of five items). Additional items measured caregiver report of travel time saved, preferred method of evaluation (e.g., in-person or telehealth), and child/family demographic information. Due to small sample size in some cells, caregiver ratings of their preference for their visit (in-person with another specialist in my region, in-person with a specialist at Children's Hospital, telehealth visit with specialist doctors, or telehealth visit with a specialist at my primary care office) were dichotomized (in-person or telehealth) for analysis.

ASD Diagnostic Assessment. The TELE-ASD-PEDS ) is a newly developed remote assessment tool designed to aid in the screening and diagnosis of ASD symptoms and behaviors in young children ages 14 to 36 months (clinicaltrials.gov, NCT03847337). The TELE-ASD-PEDS takes approximately 15-20 min to administer and is led by a trained provider in conjunction with a comprehensive diagnostic interview (e.g., developmental scores. Means (standard deviation) were reported for continuous variables and counts (percentages) were reported for categorical variables.

Next, generalized mixed effect models with random effects were utilized to examine the association between provider satisfaction and caregiver overall satisfaction, caregiver satisfaction with the provider, and caregiver satisfaction with the telehealth evaluation. Psychologist and pediatrician satisfaction scores were analyzed separately in relation to caregiver overall satisfaction, caregiver satisfaction with the telehealth evaluation, and caregiver satisfaction with the provider. Similarly, generalized mixed effect models with random effects were utilized to examine caregiver overall satisfaction in relation to child demographics, diagnostic outcomes, and telehealth evaluation factors. Lastly, generalized mixed effect models with random effects were utilized to examine provider satisfaction in relation to child demographics, diagnostic outcomes, and telehealth evaluation factors. Generalized mixed effect models with random effects were utilized to account for correlations within the same provider. Data analysis was performed with SAS version 9.4 (SAS Institute, Inc., Cary, NC). p values < 0.05 were considered statistically significant. Multiple comparisons were not controlled for in the analyses since we focused on a few specific a-priori hypotheses and reported the individual p-values for each; therefore, it was deemed unnecessary to adjust the p-values. Missing data were considered to be random due to caregivers or providers not completing the questionnaires or the information not being available in the child's medical record. Using the demographic data collected on all participants, we examined group differences between caregivers that completed the survey and those who did not complete the survey. No significant group differences were found across gender, type of insurance, ASD diagnostic outcome, or ASD symptom severity. The average age between the two groups of respondents was significantly different (p = 0.013), with the children of caregivers who completed the survey being younger (M = 36.25, SD = 10.27) than those who did not complete the survey (M = 39.32, SD = 11.22).

Of the 308 evaluations that were conducted, 287 provider satisfaction surveys (93.19%) and 141 caregiver satisfaction surveys (45.78%) were collected. Of the 287 provider satisfaction surveys completed, 235 were completed by psychologists and 52 were completed by pediatricians. Table 2 details the mean, SD, and proportions for each of the satisfaction questions that were asked to both providers and caregivers. Overall, the majority of caregivers and providers for their evaluation, including accessing the virtual Zoom Health platform, and were scheduled for a 90-to 120-minute telehealth diagnostic evaluation led by a psychologist or a pediatrician. The necessary speed and bandwidth of internet access needed was not explicitly outlined to caregivers; however, if the internet connection was found to be insufficient during the evaluation, a follow-up in-personevaluation was recommended. Caregivers who declined telehealth were informed their child would remain on the waitlist for in-person evaluation and would be contacted when scheduling resumed. A specific wait time for families who declined a telehealth evaluation was not provided given the unknown timeline regarding safe return to in-person evaluations and associated hospital policies and guidelines. Data on the differences between wait times for in-person and telehealth evaluations was not available.

Members of the research team gathered data from the Electronic Health Record (EHR) on child age, sex, home zip code, insurance type, and primary referral question (e.g., categorized into ASD, developmental delay/intellectual disability/language delay, and other neurodevelopmental disability [e.g., ADHD, learning disability, behavioral or emotional problem]) and entered it into a secure research database. Prior to the day of the evaluation, caregivers were emailed the link to complete the Vineland-3 via remote administration. On the day of evaluations, providers conducted a comprehensive diagnostic interview, administered the TELE-ASD-PEDS via Zoom Health with the child and their family, completed the TELE-ASD-PEDS rating form, and provided recommendations and next steps to the caregivers. A diagnosis was communicated to a family if the provider was able to determine the appropriate diagnosis during the telehealth evaluation; however, a diagnosis was not always provided for various reasons (e.g., technical difficulties, diagnostic uncertainty, etc.). Providers were also sent a secure link via email to complete an individual survey for each evaluation. On the day following their child's evaluation, caregivers received a secure link via email (unless they requested another method) to complete a satisfaction survey. Caregivers were also provided remote follow-up care navigation from one of two social workers staffed in our clinic approximately three to four weeks after their telehealth evaluation. Providers completing surveys for > 85% of scheduled evaluations were provided a $25 gift card for each month of participation. Caregivers were provided a $25 gift card for completed surveys.

Descriptive statistics were used to analyze demographics, diagnostic outcomes, and Vineland-3 and TELE-ASD-PEDS did not receive an ASD diagnosis (M = 4.0, SD = 0.7) or the provider was unsure (e.g., could not rule in or out) of the diagnosis (M = 4.3, SD = 0.5; p = 0.017). Caregiver satisfaction was also significantly related to provider diagnostic certainty (p = 0.02). Specifically, caregivers were more satisfied with providers who were certain about their ASD diagnosis (M = 4.6, SD = 0.6) in comparison to those who were uncertain (M = 4.1, SD = 0.8). Caregiver satisfaction was also significantly related to their visit preference (p < 0.001), with caregivers preferring telehealth visits (M = 4.8, SD = 0.3) being more satisfied than those who preferred an in-person visit (M = 4.3, SD = 0.7).

Caregiver satisfaction was not related to child race, child age, ethnicity, household income, household size, type of insurance, travel time saved, whether or not the provider could provide any type of diagnosis, provider preference for in-person or telehealth visit, or technical and setup difficulties during the appointment (ps > 0.05).

Provider overall satisfaction ratings were examined in relation to child demographics, diagnostic outcomes, and telehealth evaluation factors (see Table 4 ). Provider overall satisfaction was significantly related to whether or not the provider could make a diagnostic determination in the telehealth evaluation (p < 0.001). Specifically, providers who were able to provide any diagnosis during the evaluation (M = 4.2, SD = 0.6) were more satisfied than those who were not (M = 3.5, SD = 1.0). Similarly, providers who were able to provide a diagnosis of ASD (M = 4.4, SD = 0.5) in the telehealth evaluation were more satisfied than those who were unsure of the diagnosis (M = 3.6, SD = 0.9) or the diagnosis was ruled out (M = 4.2, SD = 0.6; p < 0.001). Similarly, providers who were certain about their diagnosis (M = 4.3, SD = 0.6) were more satisfied in comparison to those who were uncertain (M = 3.4, SD = 1.1; p < 0.001). Provider satisfaction was also significantly related to the provider's preference for an in-person visit, (p < 0.001). Specifically, providers who preferred an in-person visit (M = 3.9, SD = 0.5) were less satisfied than those who did not (M = 4.8, SD = 0.5). Lastly, technical difficulties (p = 0.007) and setup difficulties (p < 0.001) were significantly related to provider satisfaction, with providers who did not report any technical or setup difficulties having higher satisfaction ratings. Lastly, provider satisfaction was significantly related to child TELE-ASD-PEDS Total scores (p < 0.001), with providers rating higher satisfaction for evaluation of children with greater ASD symptom severity. indicated they were satisfied (4) or very satisfied (5) with the telehealth evaluation.

Overall, both providers (M = 4.2, SD = 0.7) and caregivers (M = 4.6, SD = 0.6) reported on average being satisfied with the telehealth ASD evaluations. Provider satisfaction was not significantly associated with caregiver overall satisfaction (t = 0.69, p = 0.494), caregiver satisfaction with the provider (t = 1.70, p = 0.091), or caregiver satisfaction with the telehealth appointment (t = 0.11, p = 0.914).

Both psychologists (M = 4.1, SD = 0.7) and pediatricians (M = 4.6, SD = 0.6) indicated they were on average satisfied with the telehealth ASD evaluations overall. Psychologist satisfaction was not significantly related to caregiver overall satisfaction (t = 0.77, p = 0.444) or caregiver satisfaction with the telehealth appointment (t = 0.06, p = 0.951); however, it was significantly related to caregiver satisfaction with the provider (t = 2.06, p = 0.042). Pediatrician satisfaction was not significantly related to caregiver overall satisfaction (t=-0.30, p = 0.771), caregiver satisfaction with the provider (t=-0.76, p = 0.458), or caregiver satisfaction with the telehealth appointment (t = 0.08, p = 0.936).

Caregiver overall satisfaction ratings were examined in relation to child demographics, diagnostic outcomes, and telehealth evaluation factors (Table 3) . Caregiver overall satisfaction was significantly related to child adaptive functioning (Vineland-3 ABC, Communication skills, Daily Living Skills, and Socialization Standard Scores). Specifically, results suggest caregivers of children with lower adaptive skills were more satisfied with the telehealth evaluation (ps < 0.024). The relationship between caregiver satisfaction and child sex approached significance (p = 0.052), with caregivers of female children (M = 4.7, SD = 0.6) reporting slightly higher satisfaction than caregivers of male children (M = 4.5, SD = 0.6). Caregiver satisfaction was also significantly related to ASD diagnostic outcomes. Specifically, caregiver satisfaction was significantly related to the child's TELE-ASD-PEDS Total score (p = 0.014); caregivers of children with greater ASD symptom severity were more satisfied with their telehealth evaluation. Further, caregivers of children diagnosed with ASD reported higher satisfaction ratings (M = 4.7, SD = 0.5) in comparison to those who child age, adaptive skills, race, ethnicity, gender, household Provider satisfaction was not significantly related to Caregiver satisfaction in relation to child demographics, diagnostic outcomes, and telehealth evaluation factors may be an ideal targeted population who could benefit most from the option of a telehealth evaluation. It is important to note that child demographics such as race, age, and ethnicity were not related to caregiver satisfaction in the present study; however, this study did exclude non-English speaking families and the majority of our sample included non-Hispanic participants. With regard to ASD diagnostic outcomes, caregiver satisfaction was increased if their child received an ASD diagnosis at the end of the telehealth evaluation and if the provider was certain about their diagnosis. As noted above, there are several barriers that can occur during a telehealth evaluation including technical difficulties and issues with the home setup/environment Wagner et al., 2020) . Aside from these barriers potentially occurring, proper and extensive training of providers in conducting ASD telehealth evaluations is critical. This will likely increase their confidence to provide a diagnosis with certainty, ultimately increasing caregiver satisfaction, and likely bolstering acceptability of ASD telehealth evaluations in both caregivers and providers beyond the unexpected circumstances of the COVID-19 pandemic.

Although telehealth evaluations can increase access to care for some families, it may not be the preferred evaluation method by all families that would likely benefit (e.g., rural or underserved populations), especially for families who may have limited access to adequate audio/video technology, access to internet, or experience with technology. Therefore, it is important to always consider the caregivers' preference when offering a telehealth or in-person evaluation. Specifics regarding the necessary speed and bandwidth of the caregiver's internet was not explicitly outlined during the time of scheduling in order to obtain an appointment; however, poor internet connection can play a critical role in the ability to observe the child's behaviors and provide an accurate diagnosis during a telehealth evaluation. Approximately 20% of caregivers in our sample reported technical difficulties; however, there were no differences found in their ratings of satisfaction between those who experienced technical difficulties and those who did not. Unsurprisingly, we found that caregivers who preferred the telehealth evaluation were more satisfied than those who would have opted for an in-person visit, underscoring the importance of considering caregiver preference. Similarly, Corona and colleagues (2021) also found that some parents appreciated income, household size, type of insurance, travel time saved, or caregiver preference for an in-person or telehealth visit.

The present study builds on the current literature showcasing the feasibility, utility, and satisfaction with ASD telehealth evaluations during the unprecedented COVID-19 pandemic (Alfuraydan et al., 2020; Corona et al., 2021; Juárez et al., 2018; Wagner et al., 2020 Wagner et al., , 2021 . Given that the current literature on ASD telehealth evaluations is limited, additional research is needed to further our understanding of the variables related to satisfaction with telehealth evaluations, especially with the increased emphasis on telehealth utilization since COVID-19. Utilizing a recently developed telehealth assessment tool specifically designed for remote ASD diagnostic evaluations, our results indicate both caregivers and providers were highly satisfied with the telehealth evaluation overall. Further, provider and caregiver satisfaction were related to different factors.

Approximately 88% of caregivers who completed the survey in the present study reported they were satisfied with the telehealth evaluations (e.g., agreed or strongly agreed they were satisfied). In addition to caregivers being satisfied with the telehealth evaluations overall, there were a few specific factors related to their high satisfaction. Regarding child characteristics, caregivers of children with lower adaptive skills, including communication, daily living skills, and socialization skills, and higher autism symptomology scores reported higher rates of satisfaction. This is important with regard to the utility of ASD telehealth evaluations. Specifically, for children who are more impaired developmentally and who have higher ASD symptoms, a telehealth evaluation likely sped up the wait time for accessing an evaluation and diagnosis, in addition to opening doors for these children to access early interventions and supports. Furthermore, these findings may demonstrate increased caregiver confidence in the use of telehealth ASD evaluations for children who present with this type of developmental profile and higher ASD symptom severity. Additional research into caregiver decision making when choosing to have their child evaluated via telehealth could help further delineate how child characteristics are related to caregiver satisfaction. Moreover, children referred with these characteristics that includes not only how to administer and interpret the diagnostic assessment tools reliably and accurately, but also to manage and troubleshoot technical and/or setup/home environment difficulties.

Similar to caregivers, providers also reported high rates of satisfaction overall with the telehealth evaluations, with 94% of all providers being satisfied or very satisfied with the evaluations conducted. Providers in the study included both psychologists with an expertise in ASD evaluation and pediatricians trained in ASD assessment and evaluation. Incorporation of data from both psychologists and pediatricians is consistent with previous literature focused on ASD diagnostic telehealth evaluations ; notably, the present study was novel with the majority of providers being psychologists. When collapsing all providers into one group, provider satisfaction was not significantly related to caregiver satisfaction; however, differences between disciplines (psychologists versus pediatricians) were found. Specifically, caregiver satisfaction with the provider was positively related to psychologist satisfaction, but this relationship was not found with pediatricians. Of note, our sample of evaluations conducted by pediatricians was much smaller than those conducted by psychologists, and this could be one reason for this finding. Overall, both pediatricians and psychologists reported high rates of satisfaction, providing preliminary evidence for the feasibly of telehealth evaluations using the TELE-ASD-PEDS in both provider types. Future studies could examine differences in caregiver satisfaction with telehealth ASD evaluations across a variety of different providers (e.g., pediatricians, psychologists, developmental pediatricians).

Provider satisfaction increased when they were able to provide any diagnosis (e.g., ASD, developmental delay, speech/language delay, behavior, etc.) as a result of telehealth evaluation. Similar to caregivers, provider satisfaction was also higher if they were able to provide an ASD diagnosis and they were certain about their diagnosis. As noted above, investing in proper training, education, and supervision to empower providers with the confidence to make a diagnostic determination is important to increasing acceptance of telehealth evaluations for ASD. Further, best practices in the evaluation of ASD include the combination of caregiver interview and observational assessment (Hyman et al., 2020) . Having well developed, evidencebased diagnostic and screening observation tools, like the TELE-ASD-PEDS (Wagner et al., 2020 (Wagner et al., , 2021 is necessary to increase the ability of providers to offer effective telehealth evaluations, ultimately leading to an appropriate diagnosis.

Not surprisingly, provider satisfaction was directly related to the provider's preferred method of evaluation; those who preferred telehealth reported higher levels of satisfaction. a telehealth appointment as an initial step in the diagnostic procedure but also wanted to be seen in-person during the diagnostic process. Together, these findings suggest a hybrid model may be appropriate and ideal for some families and providers, particularly when a diagnosis is not clear through a telehealth evaluation.

Unexpectedly, caregiver satisfaction was not related to travel time saved. This is contrary to our hypotheses and previous findings showing families appreciated telehealth services due to time and money saved related to travel Reese et al., 2015) . These findings have implications for increased acceptance and utility of telehealth evaluations for children and their families regardless of their distance from the provider. For some caregivers who may be within close proximity to the provider's location, they may still prefer and be highly satisfied with a telehealth evaluation. This inconsistent finding may also be closely related to the COVID-19 pandemic. Specifically, caregivers may have been more focused on and were satisfied with the ability to still obtain access to services during these unprecedented times rather than their concern with the amount of travel time saved. Further, these findings could be related to general feelings of safety during the pandemic regardless of travel time saved. Additional work is needed to further clarify and understand the relationship between access to care and telehealth ASD evaluations.

Importantly, socioeconomic factors such as household income or the type of insurance (Medicaid versus private insurance) a child had at the time of the evaluation did not impact caregiver satisfaction. This may provide preliminary evidence that individuals at all economic levels who can benefit from ASD telehealth evaluations are reporting similar satisfaction levels. However, given that we excluded non-English speaking families and our reduced survey completion rate by caregivers (approximately 46% completion rate), those who are non-English speaking and who did not complete the post-survey may be more vulnerable to these socioeconomic impacts, potentially affecting caregiver satisfaction with the telehealth diagnostic model. Therefore, additional research is needed to examine if telehealth evaluations may be an optimal option for leveling access to high quality ASD evaluations for all children regardless of their socioeconomic status. Lastly, caregiver satisfaction was not related to provider reported technical or setup difficulties. This could potentially be due to caregivers being unaware that there were any difficulties with the audio/video or their current home setup for the evaluation, especially since providers reported technical or setup difficulties in less than 20% of all visits. Further, provider skill in managing technical difficulties or setup issues could also have played an important role. As noted above, it is imperative for providers to receive extensive training in telehealth ASD evaluations also important to consider. Although this was not analyzed Similar to caregivers, provider preference for evaluation is Furthermore, assessing access to the necessary technology needed for a telehealth visit at the time of triage, including the details of what to expect with a telehealth visit, what materials/toys are needed, and how the appropriate technology setup (e.g., access to reliable internet, a device with a camera and audio) and ideal home environment are important to a successful telehealth visit. It would also be important to gauge caregiver preference for an in-person or telehealth appointment. As previously highlighted, the utility of a hybrid model including both telehealth and inperson evaluation may be ideal. For example, the telehealth evaluation could be the first step in the diagnostic process followed by an in-person evaluation; however, if the telehealth evaluation is conclusive, it may not be necessary to bring the patient in for an evaluation which could minimize the delay in getting access to the appropriate diagnosis and subsequent treatment.

Lastly, clinics and hospitals should consider investing in appropriate and extensive training for their providers in telehealth evaluation and the use evidenced-based ASD telehealth diagnostic tools. As demonstrated by our results, provider confidence and certainty to make an accurate diagnosis, including ruling in or out ASD, is important to provider and caregiver satisfaction. Given the unprecedented circumstances, the need for telehealth has shifted and continues to remain at the forefront for many providers and caregivers, but the utility, feasibility, and benefits that telehealth evaluations has to offer suggests it is an important tool we should continue utilizing into the future.

Although the present study expands and builds on the current literature with provider and caregiver satisfaction of telehealth ASD evaluation, there are limitations that need to be considered. This study did not collect data from families who were unsuccessful in accessing their telehealth visit (did not attend the scheduled appointment), from families more extensively in the present study, it is likely that diagnostic uncertainty played a role in provider preference for an in-person visit. While telehealth may work well for identifying and diagnosing some children with ASD, it may not be ideal for all. Our findings contrast with those reported by Becevic et al., (2015) where providers reported that they could adequately deliver services via telehealth, yet t preferred in-person visits. It is unclear from their findings as to why this preference was indicated given their overall high ratings of satisfaction with using telehealth. One potential reason for differential findings when compared to the present study is that they collected data regarding general preferences for telehealth versus in-person evaluations, while our study assessed preferences for evaluation modality for each telehealth assessment conducted. As noted above, offering a hybrid model for some families that allows both telehealth and an in-person option, if the telehealth evaluation does not lead to a clear diagnostic determination, may be helpful for providers in the future.

Consistent with previous findings (Wagner et al., 2020 (Wagner et al., , 2021 , providers who reported experiencing no technological or setup difficulties during the telehealth appointment reported higher levels of satisfaction. Technology is unfortunately fallible; however, as highlighted above, technical and setup difficulties occurred in less than 20% of the telehealth evaluations, adding to the feasibly of utilizing telehealth to diagnosis ASD. Notably, despite these technical and setup difficulties being reported by the providers, caregiver satisfaction was unrelated and not impacted. This suggests telehealth ASD evaluations can still be a useful tool for providers despite the limitations to come with using technology. Further exploration is needed into how the different technical and setup difficulties may impact caregiver and provider satisfaction.

From a clinical utility perspective, several key considerations from these data emerge with regards to how other clinics or hospitals may consider provider and clinician satisfaction in the utilization of telehealth ASD evaluations moving forward and beyond the current pandemic. More specifically, many of the variables examined in the present study could be screened for at the time of referral to better prioritize triaging children into a telehealth ASD evaluation. For example, clinics could consider prioritizing children who are more impaired developmentally and with high ASD symptoms severity by administering a questionnaire asked At the time of the COVID-19 pandemic, access to diagnostic evaluations for children referred for concerns of ASD took a significant hit. There were limited tools developed for telehealth ASD evaluations, limited use of telehealth diagnostic evaluations in this population, and the evidencebased literature supporting the feasibility and utility is still in its infancy. Families were given the option of transitioning to virtual appointments or waiting for in-person assessments to resume, with no clear indication of when that may occur. Overall, our results build on the feasibility and utility of telehealth ASD evaluations with the large majority of providers and caregivers being satisfied and their satisfaction being related to different child characteristics, diagnostic outcomes, and aspects of the telehealth evaluation. Specifically, caregivers reported higher rates of satisfaction for children with lower adaptive skills (communication skills, daily living skills, and socialization skills) and greater ASD symptom severity, if they had a preference for telehealth evaluation, and if the provider was certain about their child's diagnosis. For providers, satisfaction was higher for children presenting with greater ASD symptom severity, there were no technical or setup difficulties, if they provided an ASD diagnosis, and if they were certain about their diagnosis. The feedback collected by providers and caregivers highlight both acceptability of the telehealth model in addition to key factors to be considered for optimizing the use of telehealth in this population. Future work is needed to validate our results and continue the exploration of these, and other variables related to provider and caregiver satisfaction with telehealth evaluations for ASD. who declined a telehealth visit when offered, or the differences in wait times for in-person and telehealth visits. Gathering this information could increase our understanding of the families who felt comfortable choosing telehealth and those who did not, which may help providers and families in the decision-making process. This may also help with triaging patients appropriately to an in-person or virtual evaluation at the time of referral and better understand accessibility of an ASD evaluation based on wait times for either a telehealth or in-person evaluation.

Our results are also limited and should interpreted with caution for patients who are Hispanic/Latinx due to low sample size for this population, as well as exclusion of non-English speaking families. Additionally, caregiver response to the follow-up survey was 46%. This return rate exceeded those documented in a recent systematic review of patient health care and professional survey return rates (Meyer et al., 2020) . However, it is important to note that our findings may not be representative of all families who engage in telehealth ASD evaluation and should be considered when interpreting our findings. A notable strength of our study is the larger sample size as compared to previous telehealth ASD studies Juárez et al., 2018) . Additional work using large samples is critical for moving the field of telehealth evaluations forward in addition to exploring other methods of increasing caregiver participation in post-evaluation survey collection. Further, important questions remain surrounding the longitudinal impact of telehealth ASD evaluations on children and their families, including the ability to access services and interventions. Lastly, future work should include the exploration of mediating factors, such as demographic variables or ASD symptom severity, on caregiver and provider satisfaction with ASD telehealth evaluations.

Our sample of providers was also unbalanced with there being substantially more psychologist-led telehealth evaluations in comparison to pediatricians; however, this was a novelty in comparison to previous work Becevic et al., 2015) . Further, we did not control for multiple comparisons due to the focused nature of our analyses and individual reporting of p-values; however, our findings between psychology providers and caregiver satisfaction may not have survived error correction prompting caution when interpreting these results. Lastly, additional research is needed to better understand the technology and setup barriers that providers encounter during telehealth evaluations. For example, are there certain barriers that are more impactful to provider satisfaction and if so, can these be identified at the triaging process to exclude patients from being scheduled for telehealth evaluations and route them directly into an in-person only visit.

Evaluating a novel mobile-health screening tool

Prevalence of autism spectrum disorder among children aged 8 Years -Autism and Developmental Disabilities Monitoring Network, 11 Sites

Tackling healthcare access barriers for individuals with autism from diagnosis to adulthood. Pediatric Research. Advance online publication

Telehealth evaluation of pediatric neurodevelopmental disabilities during the COVID-19 pandemic: Clinician and caregiver perspectives

A novel system for supporting autism diagnosis using home videos: Iterative development and evaluation of system design

An evaluation of explicit ethical statements in telehealth research with individuals with autism spectrum disorder

Evaluation of the Diagnostic Stability of the Early Autism Spectrum Disorder Phenotype in the General Population Starting at 12 Months

Modified Checklist for Autism in Toddlers, revised with follow-up (M-CHAT-R/F)

Preliminary evidence for the integrated systems using telemedicine

Evaluating interactive videoconferencing for assessing symptoms of autism

Measuring the service system impact of a novel telediagnostic service program for young children with autism spectrum disorder

Use of telehealth for facilitating the diagnostic assessment of autism spectrum disorder (ASD): A scoping review

Rural trends in diagnosis and services for autism spectrum disorder

User satisfaction with telehealth: Study of patients, providers, and coordinators. The Health Care Manager

TELE-ASD-PEDS: A Telemedicine-based ASD Evaluation Tool for Toddlers and Young Children

Parent perceptions of caregivermediated telemedicine tools for assessing autism risk in toddlers

Experiences of autism diagnosis: A survey of over 1000 parents in the United Kingdom

Whittling down the wait time: Exploring models to minimize the delay from initial concern to diagnosis and treatment of autism spectrum disorder

Diagnosis in Autism: A survey of over 1200 patients in the UK

Family and practitioner perspectives on telehealth for services to young children with autism

Diagnostic evaluations of autism spectrum disorder during the COVID-19 pandemic

Early identification of ASD through telemedicine: Potential value for underserved populations

Use of the TELE-ASD-PEDS for autism evaluations in response to COVID-19: Preliminary outcomes and clinician acceptability

Transitioning to telemedicine during COVID-19: Impact on perceptions and use of telemedicine procedures for the diagnosis of autism in toddlers

Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations

Validity of telemedicine for diagnosing autism spectrum disorder: Protocol-guided video recording evaluation

Brief report: Preliminary feasibility of the TEDI: A novel parent-administered telehealth assessment for autism spectrum disorder symptoms in the first year of life

We are most grateful to the children and families who participated in both the clinical care and research portions of this study. We are also very grateful to members of our team that supported this project, including our director (Angela Tomlin, PhD.) and our social work team (Carrie Fitterling, LCSW & Margo Ramaker, LCSW). Additionally, we are grateful to the many providers and LEND (Leadership Education in Neurodevelopmental Disabilities) trainees in the Division of Developmental Medicine, including Angela Paxton, Leah Bode, Allison Etling, Brett Enneking, Maura Rouse, Tybytha Ryan, and Alyssa Jones for their contributions to study development and data collection and management.Author Contributions DLR, EH, and CR contributed to the manuscript preparation and revisions. DLR contributed to study aims, planned analyses, and interpreted the data. QT contributed to study analyses and interpretation. CJ contributed to data extraction and preparation, and to manuscript revisions. RMK conceptualized and designed the study and contributed to data collection and manuscript revisions. All authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

This research was supported by the Robert & Helen Haddad Family Foundation and Riley Children's Foundation (award to RMK) and T73MC00015-27-01 (HRSA; PI: Tomlin). The authors have no relevant financial or non-financial interests to disclose.