key: cord-0026617-6fhazgvt authors: Lo, Brian; Shi, Jenny; Wong, Howard W.; Abi-Jaoudé, Alexxa; Johnson, Andrew; Hollenberg, Elisa; Chaim, Gloria; Cleverley, Kristin; Henderson, Joanna; Levinson, Andrea; Robb, Janine; Sanches, Marcos; Voineskos, Aristotle; Wiljer, David title: Considerations for evaluating digital mental health tools remotely- reflections after a randomized trial of Thought Spot date: 2021-02-28 journal: Gen Hosp Psychiatry DOI: 10.1016/j.genhosppsych.2021.02.010 sha: 5b474def67cbf470f8e990314da9306718765a61 doc_id: 26617 cord_uid: 6fhazgvt nan Physical distancing restrictions amidst the COVID-19 pandemic have significantly hampered the ability to conduct in-person research [1] . The pandemic has also create challenges in mental health service delivery, exacerbated existing mental health concerns, and accelerated the need for digital mental health tools [2] . Such tools, including mental health apps and virtual care platforms, can potentially offer improved access to services and provide mental health support from afar [2] . However, while digital mental health tools are promising, it is still imperative to evaluate the efficacy, safety, and usability of these solutions before widespread implementation. Through an iterative process, our team developed and evaluated Thought Spot [3] ; a mobile health app designed to support transitionaged youth (17 to 29 years old) in seeking mental health and wellness resources. The platform features an interactive map where users can learn and conduct filtered searches for resources that are relevant to their mental health needs. Users can also participate in crowdsourcing, in which they can add additional mental health resources or review those they have used. Thought Spot was the result of a co-design project where transition-aged youth played an active role in the design, development, and evaluation of the mental health app. Currently, there are few guidelines for conducting remote evaluations of digital mental health tools [4] [5] [6] . The current letter shares five considerations from a recently completed randomized controlled trial (RCT) of Thought Spot [3] . Although the evaluation was completed prior to the pandemic, the considerations described in this letter could apply to other remote evaluations. During the RCT, many administrative procedures and study activities were conducted remotely with participants. For example, eligible participants could choose between a conventional in-person consent process or the remote "e-consent" process, which was delivered through an electronic data capture tool, REDCap. Implementing e-consent is timesaving and can accommodate participants who are unable sign the consent form in person. However, using e-consent involves additional confidentiality, privacy, and data security considerations in order to comply with policies for human subject research [7] . Currently, there are few published guidelines on how to best implement e-consent processes, warranting more research dedicated to understanding the risks and benefits of this approach. For the RCT, a detailed suicide and risk management protocol (SRMP) [8] that outlined step-by-step instructions on handling mental health emergencies was implemented. It included detailed workflows involving research team members and mental health personnel at each study site. Additionally, moderation plans may be necessary to help mitigate risks such as inappropriate content or triggering language in mental health apps that allow participants to communicate and interact with each other. For example, during the RCT, a research team member served as a moderator and reviewed any new user contributions (e.g comments & reviews) for appropriateness. Overall, having a SRMP can be integral for capturing, preventing, and responding to potential adverse events during an evaluation. Analyzing usage data can be useful for understanding how participants engaged with the app without requiring in-person observation. Usage data can also be used to compare the intervention's impact on users who used the app versus those who did not [3, 9] . In depth analysis of usage data can be further augmented with qualitative research methods to help contextualize findings and improve our understanding of user behaviour. While in-person usability testing remains the gold standard [10] , over-the-phone interviews are still capable of generating valuable insights about users' expectations and challenges. When engaging with participants during the development of Thought Spot, over half of these individuals preferred to participate in usability and user experience interviews by phone. However, during these phone interviews many participants wanted to elaborate their experiences with the app in hand. Thus, it can be beneficial to adopt video-conferencing tools to allow app users to easily share their screen with researchers. Responding to unexpected technical issues in a timely manner was a key challenge during the RCT. Since technologies evolve and receive constant updates, technical issues are inevitable and must be addressed. As technical issues can potentially affect users' experience and the outcomes of remote evaluations, it is crucial to establish a process that encourages efficient identification, reporting, and resolution of issues. It is also valuable to outline and report the types of updates or fixes permitted during a trial. Tracking all software changes is important because it can affect data analysis and interpretation. Overall, as research practices evolve due to the pandemic, remote evaluations of digital mental health tools will become a necessity. While the considerations in this article are not exhaustive, they can inform the development of a safer and more effective remote evaluation plan. The Thought Spot project was funded by the Ontario Ministry of Training, Colleges and Universities through the Mental Health Innovation Fund and the eHealth Innovation Partnership Program (eHIPP) grant from the Canadian Institute for Health Research (EH1-143558). There are no conflicts of interest to disclose. Impact of the COVID-19 pandemic on clinical research Digital mental health and COVID-19: using technology today to accelerate the curve on access and quality tomorrow Effects of a mobile and web app (thought spot) on mental health help-seeking among college and university students: randomized controlled trial RCTs: development of a smartphone app for fully automated nutrition-labeling intervention trials Participant recruitment and retention in remote eHealth intervention trials: methods and lessons learned from a large randomized controlled trial of two web-based smoking interventions Remote methods for conducting tobacco-focused clinical trials Research Electronic Data Capture (REDCap) electronic Informed Consent Form (eICF) is compliant and feasible in a clinical research setting Core components and strategies for suicide and risk management protocols in mental health research: a scoping review Intention-to-treat and per-protocol analysis Cognitive and usability engineering methods for the evaluation of clinical information systems We thank all students who participated in the trial and in our youth engagement activities. We also thank all staff, collaborators, youth advisory committee members and practicum students who were involved and supported the Thought Spot project, including: Daryl Boshart, Almerinda Collela, Paulysha De Gannes, Dr. Gunther Eysenbach, Genevieve Ferguson, Olivia Heffernan, Tyson Herzog, Dr. Jeffrey Hoch, Dr. Huan Jiang, Mandee Kaur, Dr. Marcus Law, Julia Roy, Shehab Sennah, Sarah Sharpe, Angie Solomon, Chelsea Stunden, Vivetha Thambinathan, Tim Tripp, Nicole VanHeerwaarden, and Julie Zasada. We thank the CAMH REDCap Operations Committee for their assistance in implementing the e-consent process and survey delivery procedures. We also thank Laura Pastore for copyediting the manuscript. QoC Health served as the software developers of the Thought Spot platform.