key: cord-0954237-vki4d8qg
authors: Burbach, Frank R; Stiles, Katie M
title: Digital Mental Health and Neurodevelopmental Services: Case-Based Realist Evaluation
date: 2021-09-17
journal: JMIR Form Res
DOI: 10.2196/29845
sha: 0421229000d8ad088d5453acdf3b7326185b0990
doc_id: 954237
cord_uid: vki4d8qg

BACKGROUND: The rapid movement of mental health services on the internet following the onset of the COVID-19 pandemic has demonstrated the potential advantages of digital delivery and has highlighted the need to learn from prepandemic digital services. OBJECTIVE: The aim of this study is to explore the different elements of interconnected digital mental health and neurodevelopmental services of a well-established provider to the UK National Health Service and how web-based delivery enables young people and their families to access high-quality assessments and interventions in a more timely, flexible, and person-centered manner than in-person delivery. METHODS: A realist evaluation multiple case–study design was used, with 9 pediatric cases (aged 8-15 years) identified as representative of the services provided by Healios. Presenting concerns included autism and ADHD, anxiety and panic attacks, low self-esteem, anger and self-harm. The research literature was used to define the program theory and six context-mechanism-outcome (CMO) statements. The CMOs formed the basis for the initial data extraction, with novel elements added via an iterative process. RESULTS: We identified 10 key elements of web-based services: flexible delivery and timely response, personalized care to the individual, comprehensive care enabled by multiple interconnected services, effective client engagement and productive therapeutic alliances, use of multiple communication tools, client satisfaction with the service, good clinical outcomes, ease of family involvement throughout sessions or from different locations, facilitation of multi-agency working and integration with National Health Services, and management of risk and safeguarding. These elements supported the six CMOs; there was clear evidence that young people and their families valued the responsiveness and flexibility of the web-based mental health service and, in particular, how quickly they were seen. There was also clear evidence of individual needs being met, good therapeutic alliances, and client satisfaction. Multiple communication tools appeared to maximize engagement and working digitally facilitated multi-agency communication and delivery of safe care. The abovementioned factors may be related to the finding of good clinical outcomes, but the methodology of this study does not allow any conclusions to be drawn regarding causality. CONCLUSIONS: This study demonstrates the effectiveness of interconnected digital mental health and neurodevelopmental services as well as how web-based delivery enables young people and their families to access assessments and interventions in a more timely, flexible, and person-centered manner than in-person delivery. The 10 key elements of web-based service delivery identified through the 9 case studies suggest the potential advantages of web-based work. These elements can inform future research and aid in the delivery of high-quality digital services.

The onset of the COVID-19 pandemic has forced services to be delivered on the web and given impetus to digital health care as a solution to access gaps by expanding and leveraging existing technologies [1, 2] . In particular, although people with existing mental health problems have been resilient during the pandemic [3] , many children and young people (CYP) have experienced low levels of anxiety and depression [4] , exacerbating ongoing problems in accessing services [5] . Hence, a "digital revolution" [6] has been accelerated, with calls for the consolidation of gains [7] and the need to learn from prepandemic digital services [8] . A review of prepandemic literature concluded that "tele-mental health has potential to be an effective and acceptable form of service delivery" and called for future digital mental health implementation to use a combination of previous evidence and COVID-19 experiences [8] . Nicholas et al [9] reviewed children and adolescent mental health services (CAMHS) during the pandemic in Australia and noted that young people reported web-based services to have improved service quality, whereas the clinicians were significantly less positive.

The implications of web-based mental health services have previously been examined within the rubric of the ethics of digital delivery. In a narrative review of mostly North American literature, advantages were identified, including convenience, increased acceptance and adherence, cost efficiencies, enhancements in communication, and other therapy benefits [10] . Furthermore, factors contributing to the positive evaluation of web-based care by clients included meeting needs in a timely and effective manner [11] , a more egalitarian therapeutic alliance [12] , and an increased sense of being in control [9, 13] .

This paper contributes to the literature on the effectiveness of digital service delivery by Healios, which has been providing a range of web-based CYP mental health and neurodevelopmental (ND) services to the UK National Health Service (NHS) since 2015. We purposely adopted a realist evaluation (RE) approach [14, 15] to assess whether our service works in a particular context. In this case, we designed a digital service to maximize the advantages of providing web-based mental health services, and, in line with RE practice, we initially examined those case examples that involved different services and service combinations. The context is the provision of web-based youth mental health and ND services before the pandemic to inform development in the future.

In line with the RE methodology, we developed an initial program theory and context-mechanism-outcome (CMO) statements to enable data collection to focus on testing the different elements of the program theory. Our RE mid-range theory is that providing services digitally allows young people and their families to access mental health and ND services in a more timely, flexible, and person-and family-centered manner than in-person delivery. All the existing qualities of in-person service delivery, including the formation of a therapeutic alliance, can be delivered safely and effectively via the internet; clinical outcomes are equivalent to or better than in-person care, and additional efficiencies can be achieved. Digital delivery also improves outcome measurements and interagency communication.

Using themes primarily from recent reviews [9, 10] and previous research suggesting that mental health service outcomes are associated with therapeutic alliance and client satisfaction [16, 17] , the authors and colleagues created CMO statements for Healios' web-based services:

1. Timely provision of mental health services leads to better engagement and outcomes. 2. Personalized and flexible care leads to successfully building therapeutic alliance and client satisfaction. 3 . Access to multiple interconnected services leads to more comprehensive care that meets the individual's needs. 4 . Digital mental health provides a more egalitarian experience than in-person mental health and leads to the empowerment of the client and better outcomes. 5. Using multiple digital communication tools (eg, videoconferencing, therapeutic information, interactive whiteboards, rating scales, and outcome measures) can enhance engagement and the therapeutic alliance. 6. Web-based communication facilitates safe care, networking, and access to therapeutic resources: particularly, information, support from peers and family, and liaison with schools and other agencies.

The goals of this study are (1) to analyze selected cases to explore the CMO statements and (2) to obtain insights into the delivery of high-quality web-based mental health care and ND services.

Healios is a fully remote company using a bespoke electronic patient record system and remote delivery system, Panacea, which uses a cloud-based Ruby on Rails application hosted by Amazon Web Services. This specialized, secure platform (ISO [International Organization for Standardization] 27001 certified) has been developed over the past 8 years. It facilitates referral and client management, including a portal for the NHS to securely make referrals and monitor the progress of cases, the delivery of interventions and recording of sessions.

Accredited clinicians deliver all assessments and interventions through Panacea via split-screen videoconferencing, facilitating interaction with a clinician and interactive slides. Although clinicians adapt sessions to meet client needs, slide sets are provided to structure all clinical sessions. For example, a 10-session cognitive behavioral therapy (CBT) deck provides structure and ensures fidelity to the National Institute for Health and Care Excellence-endorsed (2017) CYP CBT manual, but additional sessions can be offered and some sessions can be omitted. The British Association for Behavioral and Cognitive Psychotherapies-accredited therapists can choose interactive CBT decks covering panic disorder, generalized anxiety, obsessive compulsive disorder, and depression. The other therapy package is goal-based intervention (GBI; maximum 6 sessions) delivered by CAMHS professionals or psychological well-being practitioners for mild-moderate presentations, especially where the young person is more practically oriented. In some cases, getting help or getting more help assessments (CAMHS tier 2 or 3 initial assessments) are offered to young people to ascertain whether CBT or GBI would be appropriate.

The Healios autism assessment service involves an extensive 5+ hours-long digital assessment process involving 2 clinical professionals conducting 3 videoconferencing clinical interviews with the young person and their parents based on 2 gold standard-validated assessment tools (Autism Diagnostic Interview-Revised and Autism Diagnostic Observation Schedule) [18] and clinical information obtained from schools, NHS, and social services, as appropriate. A video-conferencing multidisciplinary team, comprising a minimum of 3 clinicians, makes a diagnostic decision using the Diagnostic and Statistical Manual of Mental Disorders-5 criteria, followed by feedback to parents with advice on responding to their child's needs and a report with guidance for teachers and other professionals. The attention-deficit/hyperactivity disorder (ADHD) assessment service is similar, and both may be preceded by an initial screening appointment. The postdiagnostic intervention (PDI) comprises up to 10 sessions of tailored content adapted to the needs of the young people and their families.

In addition to the abovementioned CYP services, Healios also offers adult ND services, including autism and ADHD assessments, along with postdiagnostic support. Adult mental health services include perinatal CBT, family interventions, and mental health assessments. However, this study will only focus on CYP services.

As this is a small case series study that uses fully anonymized clinical record data, research ethics committee approval was not required.

This study included 9 cases seen at Healios between April 2018 and May 2020. Of these, 6 were female and 3 were male, and they were aged 8-15 years (mean 11.67, SD 2.83; Table 1 ). Although this represents a higher proportion of females than referrals from 2018 to 2020 (female: 5763/10,763, 53.54%; male: 5000/10,763, 46.46%), gender balance was not expected to have any bearing on the results, as cases were selected to be representative of different service lines available to CYP at Healios. Further details on case selection can be seen in the section Case Selection and Procedure. The journeys of the 9 cases through Healios' services are shown in Figure 1 . 

We used a multiple case-study design with 9 cases identified using convenience sampling as representative of the service lines offered by Healios [19] . To examine the different pathways that clients could take through Healios, researchers identified contracts where multiple service lines had been commissioned, and team leads were approached to think of relevant cases. Inclusion criteria were patients who were aged <16 years, had completed their full course of treatment or assessment, and whose case had been closed by Healios (ie, had completed feedback questionnaires and, where appropriate, had completed routine outcome measures [ROMS]). Cases were randomly selected from relevant NHS contracts, and case notes were checked against the inclusion criteria. Once cases that covered a range of the services available to CYP at Healios as well as the different pathways that could be taken were identified, no further cases were sought. The relevant outcomes and process data were then extracted. Key themes and elements of web-based work were identified from Stoll et al [10] and Nicholas et al [9] to provide structure for the data extraction during which new themes were added and existing themes reanalyzed and recategorized. Following this iterative process, 10 factors were identified (Textbox 1).

Textbox 1. The 10 key elements identified for data extraction. 

The 10 key elements (KEs) identified for the web-based service provision are presented below. These have been identified through previous literature [9, 10] , with novel themes added and existing themes recategorized, as appropriate during data extraction.

Healios offers services between 9 AM and 8 PM, Monday-Sunday, giving young people and their families maximum flexibility in scheduling sessions via their web-based portal. Waiting time from referral to initial appointment ranged from 4 to 51 days (mean 22.67, SD 14.59). For comparison, in-house NHS autism assessments have an average waiting time of 352 days from referral to diagnosis [20] , and some local CAMHS mental health services report a median waiting time of 82-182 days with an overall average waiting time of 56 days for treatment [21] . The total time from referral to discharge for the case-study group was between 18 and 51 weeks (mean 26.35, SD 11.56).

Gemma had her first appointment booked on the day her referral was accepted and was seen 4 days later. Similarly, Nadia was booked in for her first appointment within a day of her referral being accepted and was seen 9 days later (see Table 1 for all wait times from referral to first appointment). Lily booked the majority of her sessions for weekday evenings and stated in her first session that part of the appeal of Healios was that she could still attend school as usual with time for therapy in the evenings. Alex and Arun also scheduled evening sessions; Alex did so to coordinate the appointments with his father's working hours. 

All session content was adapted to suit the young person, and this was evident in all 9 exemplar cases. Gemma was referred for a getting help assessment, where she disclosed severe low mood and social anxiety symptoms, and was referred on to GBI. She made progress but required further support and was stepped up to CBT. She was initially anxious about being on camera, and the clinician suggested using a post-it note to block the camera. After further sessions, she was more comfortable with the camera switched on and felt that she was able to appear on screen. Alex was initially referred to the ND team but showed symptoms of anxiety and reported intrusive thoughts during her ND screening and autism assessment; she was referred to the Mental Health CBT Pathway before starting her PDI sessions. Phoebe was seen for a getting help assessment and was booked for her first session of GBI in early May, but this was postponed until the end of June because of exam commitments. Viv said,

It was just amazing they understood me.

[Viv]

Healios offers multiple service lines and, where commissioned, can move referrals between these services as appropriate. Hence, young people experiencing multiple difficulties can receive comprehensive care through interconnected digital pathways, and 6 of the 9 cases received multiple services. Arun was initially referred for a getting more help assessment and subsequent CBT, but because of lack of engagement in the early sessions, he was internally referred for ND screening. Similarly, Alex was referred to Healios for an ND screening and was assessed for both autism and ADHD. However, during these sessions, she presented with symptoms of anxiety and intrusive thoughts and was also referred to within Healios for CBT sessions in addition to PDI for autism.

As part of Healios' mental health services, getting help and getting more help assessments are offered before treatment to explore current difficulties, early life experience and development, and to assess risk. Both Phoebe and Gemma were referred to Healios for a getting help assessment, following which both were offered 6 sessions of GBI, with Gemma being further stepped up to CBT on completion of GBI. 

Effective client engagement and therapeutic alliance was examined using Healios postsession ratings (HPSR). At the end of each session, clients rated 4 statements on a scale of 0-100.

In 6 cases, HPSR was used. With Nadia and Alex, HPSR was used for every session of CBT, and with Viv, it was used for all sessions of PDI for ADHD; however, with Gemma and Arun, it was only used sporadically. Table 2 summarizes the average scores for each statement given by the individual cases in which the HPSR was used.

For these cases, the average score given on the overall measure of HPSR was 92.72 (SD 12.63; range 68.53-100), indicating that sessions are appropriate, and the average subscale scores of >90 suggest productive therapeutic alliances. In particular, Viv and Alex consistently rated their sessions as 100, indicating that they formed strong therapeutic alliances with their clinicians and were engaged with their sessions and treatment. This was reflected in the feedback given by Viv's mother, "Listen, gave constructive advice, didn't shy away from giving tough messages." Nadia also commented on the therapeutic alliance formed, "They listened to me, took interest in what I had to say and always had a positive response."

They spoke directly to my child making him feel valued. [Viv's mother] 

All sessions take place on Healios' clinical platform and involve interactive activities, including completing formulations, therapeutic exercises, rating scales, and using a whiteboard. The use of these features was evident in all 9 cases. Phoebe used the interactive whiteboards during her getting help assessment to describe her current feelings and how she found school, friendships, and home life using emojis and pictures. Lily used the mood diary between sessions to document her thoughts and feelings, capturing when she had felt worried, which were subsequently shared with her clinician in the next session.

In addition to HPSR, the interactive platform includes sliding scale measures for in-session self-rating of mood and the severity of difficulties. The platform also allows clinicians to share other resources in real time (eg, looking at websites) and for Arun, this included playing a web-based, interactive version of Connect4 at the beginning of his autism assessment to promote engagement. Question 12 of the Healios Experience of Service Questionnaire (HESQ)-"The online interactive activities during my session were helpful"-provides some feedback on the interactive communication tools. Unfortunately, 2 of the 5 young people who completed the HESQ answered "Don't know" but 2 answered "Certainly true" and 1 answered "Partly true." Furthermore, 4 of the 5 parents who completed this question responded "Certainly true" and 1 responded "Don't know." We do not have data from the other cases because of a technical hitch with this questionnaire, but it is worth noting that in response to this question, for the service as a whole, 66.7% (1917/2874) of parents and young people responded "Certainly true;" 18.93% (544/2874), "Partly true;" 2.75% (79/2874), "Not true;" and 11.62% (334/2874), "Don't know."

The staff were nice. I liked the part where we had 1:1 sessions and enjoyed the activities that we did.

[Rachel]

All cases reported satisfaction with the services received. Feedback was routinely collected from young people and their families at the end of assessment or treatment using the NHS Friends and Family Test (FFT) and the HESQ. The FFT asked if they would recommend the service to others, with responses recorded on a 5-point scale (young person version: 0="I disagree a lot" and 4="I agree a lot"; parent version: 0="Extremely unlikely" and 4="Extremely likely"). The HESQ comprised 12 items, 9 relating to satisfaction with the care received and 3 to the web-based processes or environment, all being rated on a 3-point scale (0="Not true," 1="Partly true," and 2="Certainly true"). Both questionnaires also invited comments on things that went well and suggestions for improvement. In all cases, either the FFT or HESQ had been completed by the young person and/or their parents, and in some cases completed multiple times after different services. In 6 cases, both the FFT and HESQ were completed (Table 3) .

Sam answered 8 HESQ questions on care and 2 on environment, answering "Certainly true" for all questions except "It was easy to talk to the people who saw me and If a friend needed this sort of help, do you think they should come here?" to which he answered, "Partly true." Both Sam and his mother responded to the same 10 of the 12 questions, not answering "Were you given enough explanation about the help available here?" and "The online activities in my sessions were helpful." Sam's mother responded "Certainly true" to all the other questions. It is not clear why they did not answer 2 of the HESQ questions, but it is interesting to note that Sam's mother also completed the FFT and responded that it would be "Extremely likely" that she will recommend the service to others. She also wrote an unsolicited letter in which she thanked the clinician for her "professionalism and kind heart," saying that it was "like having a genie in a bottle: she recommended so many improvements as if she'd known my son all his life." Gemma answered 4 HESQ care questions, rating "I have been given enough explanation about the help available here" as "Certainly true" and "I feel that the people who saw me listened to me, My views and worries were taken seriously," and "Overall, the help I have received here is good" as "Partly true." It is not clear why she did not answer the other questions, but in the free text she added that "Being understood, non-judgemental" was what she liked about the care. Gemma did not complete the HESQ following her CBT and GBI, but her mother completed the HESQ for all services and responded "Certainly true" to all questions. 

Standardized assessment measures were used as part of clinical and diagnostic assessments and to measure therapeutic progress. The Revised Children's Anxiety and Depression Scale (RCADS) and Strengths and Difficulties Questionnaire (SDQ) was used with all the cases, except with Sam, where only the SDQ was completed because of technical problems. In GBI and CBT cases, the RCADS [22, 23] and SDQ [24] were completed before and after the course of intervention as outcome measures, with any missing data being related to parental noninvolvement after initial assessment in the case of older children. There is also clear evidence that session-by-session outcome measures, Young Person Clinical Outcomes in Routine Evaluation (YP-CORE) [25] , and goal-based outcomes (GBOs) [26] were successfully used to complement the RCADS and SDQ, where there was a risk of incomplete data. GBOs assessed progress toward idiosyncratic therapeutic goals using a 0-10 scale, whereas the YP-CORE, a 10-item self-report questionnaire, covered anxiety, depression, trauma, physical problems, functioning, and risk to self. This is scored on a 5-point scale (0="not at all" and 4="most or all of the time"). YP-CORE total scores up to 5 are considered healthy; 6-10, low level; 11-15, mild; 16-20, moderate; 21-25, moderately severe; and above 25, severe problems (Table 4 ). In most cases, the outcome measures detected reliable and/or clinical improvement (reliable improvement was considered as statistically significant change and clinical improvement as scores changing from above to below the clinical cut-off). Gemma was initially referred for a getting help assessment, because she struggled with significantly low mood, anxiety and insomnia related to long standing social relationship difficulties and bullying at school. Her initial score of 36 on the YP-CORE indicated severe problems, but following the available six sessions of GBI, her scores had only slightly improved, and she was stepped up to CBT. When she started her CBT treatment a fortnight later, Gemma completed the RCADS and SDQ, both of which indicated clinically significant problems. Following 10 sessions of CBT that focused on understanding and coping with anxiety, especially in group situations at school, all of her outcome measures showed clinically significant improvement: her YP-CORE scores had reduced to the moderate level, the RCADS changed from clinical and fell within the normal range and SDQ scores reduced from very high to raised.

Alex was initially referred to the ND pathway for a screening consultation and was diagnosed with autism. During the assessment, she disclosed intrusive thoughts and feelings of anxiety, which was reflected in the anxiety subscale of the RCADS. She was referred for 10 sessions of CBT that focused on psychoeducation around anxiety and exposure to situations as well as sleep hygiene and thought challenging. At the end of treatment, Alex's scores on the RCADS had reduced overall and the anxiety subscale score fell within the normal range, showing clinical improvement on this measure. Although Alex's father's score on the RCADS had reduced, they still fell within the clinical range. No improvement was seen on the SDQ, which was also completed by him. YP-CORE scores also remained consistent, suggesting that Alex was still experiencing moderate problems. This may be related to her autism and, therefore, was addressed in her subsequent PDI.

Idiographic measures such as GBOs are highly sensitive to therapeutic change [27] , and an improvement of 2.45 points represents a reliable change [28] . Phoebe set 2 goals in GBI session 1 and made steady progress throughout the 6 sessions. Her first goal, "Be more confident," increased from a score of 3 to 8, and her second goal, "To self-harm less," increased from a score of 2 to 7, indicating reliable change on both by session 5 (given that the primary focus was on relapse prevention, goals were not scored in the final session). Phoebe's improvement on GBOs was mirrored by her steadily decreasing scores on the YP-CORE from 18 (moderate problems) to 8 (low-level problems). In contrast, Lily set 2 goals in her fifth session of CBT, and her goal ratings fluctuated during treatment. Her first goal was to challenge her thoughts, and although this was initially rated as 6, it decreased to 4 in her eighth session, during which she reported that she had experienced exam-related stress. During her final 13th session, she rated this goal as 9. Her second goal, "to be less self-critical," was initially rated 7 and remained steady before being rated 6 at week 12 and then 8 in her final session. Therefore, Lily exhibited reliable change on her first goal only while her RCADS and SDQ scores showed significant clinical improvement, that is being within the normal range at the end of treatment.

Nadia was also referred to Healios for CBT and set goals in session 3. Her first goal was to be more proactive after school and on weekends. This goal was scored 0 when set but it increased throughout the course of CBT and was scored 10 (fully achieved) in her final (10th) session. Her second goal was to get herself ready for mathematics lessons; this goal, which was initially scored 4, was fully achieved by session 5. Nadia also aimed to have multiple horse-riding lessons and set this as her third goal; this goal was scored a 0 when set and increased to 3 in her final session. Therefore, Nadia showed reliable change on all her goals. Despite Nadia's significant school-related difficulties detected in the first session, her YP-CORE score was 11, putting her in the mild problems range. Her RCADS and SDQ scores indicated clinically significant problems, which were mirrored by her mother's scores (Table  4) . Although her YP-CORE scores fluctuated between 2 (healthy) and 8 (low-level problems) week after week between sessions 2 and 9, they dropped to 1 in her fourth session. This session took place over the Christmas holidays and Nadia reported reduced anxiety because she did not have to attend school. Despite her YP-CORE scores increasing again after the Christmas period, by the end of treatment, she had returned to school and her YP-CORE score was 0; her RCADS and SDQ scores were in the normal range.

It is such a relief to finally have some answers as to why 'Arun' has been struggling so much. [Arun's father]

In all 9 cases, there was evidence that parents were involved in some or all of the sessions. Some of the older children had their parent or parents attending either the beginning of the sessions or just their first session. There was also evidence in all cases of parents completing ROMS and feedback questionnaires.

Both cases 4 and 6 had parental involvement throughout their CBT, with Nadia's mother joining her throughout the 10 sessions of CBT. In contrast, Lily was joined by her mother for only 3 of her 13 CBT sessions. This demonstrates the collaborative nature of the sessions, with Lily being able to choose when her mother could join her sessions. In both cases, however, family was found to be a protective factor. Both the young people reported that when they were struggling between sessions, they felt that they were able to speak to their mothers for support.

In Arun's case, his father's involvement appeared integral, and he attended all the sessions. Arun was initially referred for CBT, following a getting more help assessment; however, after the first few sessions, it became apparent that Arun was not engaging with treatment. Despite this, Arun's father attended the sessions and spoke about the impact that Arun's current difficulties were having on the whole family. After talking this through with their Healios CBT therapist, it was decided to refer Arun to the ND pathway for assessment. Had it not been for Arun's father's involvement, it may have been more challenging to understand his difficulties and what services he required and following his disengagement from CBT, he may have been discharged without further assessment. When Gemma's mother joined for her first session of GBI, Gemma felt comfortable to attend the subsequent sessions on her own. For sessions 2-6, her mother only joined for the first few minutes to give the clinician an update on their week before leaving Gemma and the clinician to work together. 

Web-based working facilitates multi-agency working and the ability to integrate with NHS services. There was evidence of integration with NHS services in all 9 cases of this study. In all the cases, detailed reports were written on completion of the service and shared directly with the referring CAMHS services and general practitioners.

For autism and ADHD services, clinicians benefit from the digital clinical records platform, which pulls the assessment scores into a template report. This means that clinicians can input assessment scores once only; this prevents errors in the writing of the bespoke report. Once completed, the report is automatically shared with the referring services through their referral portal and the family through their own portal. This process was followed in cases 1, 2, 3, 8, and 9.

In cases where multiple services had been received, integration with the NHS was vital to ensure that local teams were kept up to date with the progress of the young person. For example, in Arun's case, detailed reports were written up following his getting more help assessment, CBT, ND screening consultation, and autism assessment. These contained an overview of what had been discussed and the agreed upon next steps. All 4 of these reports were shared with the referring CAMHS services, so they were aware and up to date on Arun's care. Arun and his parents could also access these reports through their portal.

Integration with the NHS is also important to ensure that care continues seamlessly with local teams where necessary. This was important in Gemma's care, as after receiving both GBI and CBT, her clinician thought that it might be helpful for her to be offered further intervention to build on the progress that she had made. The local CAMHS team was updated on her treatment through a comprehensive summary report and they agreed that a medical review appointment would be useful to inform the next steps and ensure that Gemma continued to receive the support she needed.

School input is sought as part of the ND services provided by Healios. This input from teachers and special educational needs coordinators offers crucial insight into young people's behavior while at school. This was particularly important in Alex's ADHD assessment. Although her parents reported her to be very hyperactive at home with difficulties sleeping, her teachers described her as more withdrawn and shy at school. One of her teachers also completed the Conners-3 questionnaire, giving her average scores on all subscale measures of inattention, hyperactivity or impulsivity, learning problems, defiance or aggression, and relationships. Consequently, the patient did not meet the criteria for a diagnosis of ADHD.

As part of Viv's autism assessment, his school was sent a questionnaire to provide information on their current concerns and insights into his peer relationships, behavior, communication, and academic attainment. As his assessment occurred during the school summer holidays, the school did not submit their feedback until the term started again. This meant that there was a delay in his autism assessment outcomes. Once this feedback was received, the multidisciplinary team was able to meet to discuss all the information and arrive at a conclusion. The school then also provided input for Viv's ADHD assessment and completed the Conners-3 questionnaire. Both of these assessment reports were written, including the school's input, and shared with the referring CAMHS service as well as with Viv and his family. Families are encouraged to share these reports with the school to ensure that they can support the young people.

A commonly expressed concern regarding web-based assessments and treatment is that clinicians might not be able to accurately assess and effectively manage risk and safeguarding concerns. Healios follows strict risk and safeguarding protocols to manage any risks arising during sessions and communicates closely with the referring NHS and safeguarding teams when a local response is required.

All 9 cases underwent a risk assessment, but only Lily presented with concerns that required further action. She disclosed previous self-harm and suicidal thoughts and a current plan to end her life but no intent to do so. Due to the historical nature of suicidal ideation, no action was taken at this stage, but the risk was monitored in each session. In the middle of Lily's treatment, her mood dropped and her mother joined a session to disclose worries of risk to self over the previous week. Lily said that she had been having thoughts of being worthless and suicidal ideation; however, she reported no intent to act on these thoughts. During this session, a crisis management plan was formed, and protective factors and distraction techniques were discussed for Lily to use when feeling low. The Healios clinician was able to share the number for the local CAMHS crisis team if Lily's mother had any concerns about Lily's well-being between sessions. A risk assessment form was completed, the Head of Mental Health Services at Healios was informed, and the risk assessment and management plan was shared with the local CAMHS service. Risk was then monitored closely for the remaining 8 sessions.

In cases 1, 5, 7, and 9, lower-level concerns around risk were identified during risk screening and crisis phone numbers were shared for families to use if they had any concerns between sessions. In addition, a risk capture form was completed and attached to the session summary so that it could be accessed by the referring CAMHS team.

The purpose of a RE is to assess whether a program or service works in a particular context. In this study, we analyzed 9 cases that had received a digital mental health or ND service via the Healios web-based platform to verify our mid-range theory and our CMO statements.

We identified 10 KEs: These elements supported the 6 CMOs: There was clear evidence that young people and their families valued the responsiveness and flexibility of the web-based mental health and ND services and, in particular, how quickly they were seen. There was also clear evidence of individual needs being met, good therapeutic alliances, and client satisfaction. Multiple communication tools appeared to maximize engagement and working on the internet facilitated multi-agency communication and delivery of safe care. The above factors may be related to the finding of good clinical outcomes, but the methodology of this study does not allow any conclusions to be drawn regarding causality. Nonetheless, this study has verified our program theory regarding digital mental health and ND services and provides useful insights for the further development of digital services.

As this is qualitative research using a convenience sample, albeit cases selected in a random fashion to provide data in line with the aims of the study, these results are not necessarily representative of the service as a whole, and the results cannot be generalized to other digital services. For example, this study does not explore nonengagement or dropout from digital services [29] [30] [31] , and other digital providers using other technical platforms and procedures may not realize equivalent advantages. In addition, all cases explored were CYP; therefore, the mechanisms detailed here may not be applicable to adult populations. Studies of this type are potentially open to selection bias, as we only report 9 cases. However, this methodology was appropriate to identify the KE of web-based service delivery and explore whether interconnected digital mental health and ND services can be effectively delivered.

The 9 cases demonstrate the potential advantages of web-based service delivery for the collection of outcome and satisfaction data. Although there were some missing data, which appeared to be the result of a range of factors including system failures, the data available were more comprehensive than that available in most in-person services. It is suggested that information technology infrastructures would help with this problem [32, 33] . Collecting ROMS is quick and easy to perform on the web.

Clients can complete measures in privacy in the electronic waiting room before the sessions, and interactive measures can also be used during sessions. Digital delivery maximizes the potential for meaningful integration of outcomes in the clinical process and the delivery of feedback-informed treatment [34] .

These cases demonstrate that the web-based delivery of services also enables the appropriate involvement of family members or significant others, with people being able to attend from different locations. In all 9 cases, parents attended one or more sessions as decided by the young person, which enhanced their sense of autonomy. Parents appreciated the way in which they were able to join sessions, often to suit their work commitments. There was also evidence of the sessions being scheduled to fit around the young person's school commitments, although some other young people had chosen to have their sessions while at school (with their school's support). Although none of these cases involved peers in the sessions, this would be another potential advantage of web-based services.

It appears that web-based services have the potential to engage young people who might not otherwise be able to access in-person services and that the provision of having web-based services has many advantages. Another advantage that these particular cases have clearly demonstrated is the ability to offer interconnected digital services and to integrate services with the NHS. Working digitally enables seamless coordination and efficient delivery of care.

Although the effectiveness and acceptance of digital mental health solutions have been widely researched [7, 10, 11] , this has mainly been with adults, and there have been calls for further research into this method of delivery with CYP [8] . There have been no reports of digitally native mental health and ND services.

Initial data extraction was informed by previous studies [9, 10] identifying various elements of digital service delivery. Our study confirmed these elements within Healios' services and identified additional key themes, such as multi-agency working and integration with the NHS. This study also highlighted the advantages of interconnected digital services and digital delivery of ROMS.

This study demonstrates the effectiveness of interconnected digital mental health and ND services, and how web-based delivery enables young people and their families to access assessments and interventions in a more timely, flexible, and person-centered manner than in-person delivery. These 9 cases from the established digital mental health and ND services provided by Healios illustrated the potential afforded by web-based delivery beyond COVID-19 restrictions. We hope that the 10 elements identified will inform future research and facilitate the delivery of high-quality digital mental health care and ND services.

Using telehealth to remediate rural mental health and healthcare disparities

How mental health care should change as a consequence of the COVID-19 pandemic

COVID-19 Mental Health Policy Research Unit Group. Early impacts of the COVID-19 pandemic on mental health care and on people with mental health conditions: framework synthesis of international experiences and responses

CO-SPACE Study Report 04: Changes in children and young people's emotional and behavioural difficulties through lockdown. Co-Space Study

The NHS long term plan

The digital mental health revolution transforming care through innovation and scale-up. World Innovation Summit for Health

Opportunities from the coronavirus disease 2019 pandemic for transforming psychiatric care with telehealth

Implementation of Telemental Health Services Before COVID-19: Rapid umbrella review of systematic reviews

Implementation lessons from the transition to telehealth during COVID-19: a survey of clinicians and young people from youth mental health services

Ethical issues in online psychotherapy: a narrative review. Front Psychiatry

mHealth: a mechanism to deliver more accessible, more effective mental health care

Living in a Zoom world': survey mapping how COVID-19 is changing family therapy practice in the UK

The working alliance in online therapy with young people: preliminary findings

Realistic Evaluation

Realist impact evaluation: an introduction. Overseas Development Institute, London

Examining change in therapeutic alliance to predict youth mental health outcomes

Parent and adolescent satisfaction with mental health services: does it relate to youth diagnosis, age, gender, or treatment outcome?

Diagnostic procedures in autism spectrum disorders: a systematic literature review

Case study research

Autism waiting time statistics -Quarter

Access to child and adolescent mental health services in 2019. Education Policy Institute

The Revised Child Anxiety and Depression Scale-Short Version: scale reduction via exploratory bifactor modeling of the broad anxiety factor

The revised child anxiety and depression scale 25-parent version: scale development and validation in a school-based and clinical sample

The strengths and difficulties questionnaire: a research note

The young person's CORE: development of a brief outcome measure for young people

Goals and Goal Based Outcomes (GBOS), Third Ed

Youth top problems: using idiographic, consumer-guided assessment to identify treatment needs and to track change during psychotherapy

Interpreting standardized and idiographic outcome measures in CAMHS: what does change mean and how does it relate to functioning and experience? Child Adolesc Ment Health

Dropout from internet-based treatment for psychological disorders

Defining dropout from youth psychotherapy: how definitions shape the prevalence and predictors of attrition

Supporting family engagement with child and adolescent mental health services: a scoping review

Implementing routine outcome measures in child and adolescent mental health services: from present to future practice

Child-And Parent-Reported Outcomes And Experience From Child And Young People's Mental Health Services

Beyond measures and monitoring: realizing the potential of feedback-informed treatment

Young Person Clinical Outcomes in Routine Evaluation Edited by G Eysenbach; submitted 30.04.21; peer-reviewed by MDG Pimentel

Please cite as: Burbach FR

), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information

No funding was received to assist with the preparation of this manuscript. The authors would like to thank their colleagues, in particular Dougal Hare, for reviewing the manuscript.

Authors FRB and KMS are both employees of Healios Ltd.