key: cord-0914540-w9mz1i7s authors: Amiri, Parham; Karahanna, Elena title: Chatbot use cases in the Covid-19 public health response date: 2022-02-07 journal: J Am Med Inform Assoc DOI: 10.1093/jamia/ocac014 sha: 80ca359a7778ca77ee04869bfcd22c40c0e51583 doc_id: 914540 cord_uid: w9mz1i7s OBJECTIVE: To identify chatbot use cases deployed for public health response activities during the Covid-19 pandemic. MATERIAL AND METHODS: We searched PubMed/MEDLINE, Web of Knowledge, and Google Scholar in October 2020 and performed a follow-up search in July 2021. We screened articles based on their abstracts and keywords in their text, reviewed potentially relevant articles, and screened their references to (a) assess whether the article met inclusion criteria and (b) identify additional articles. Chatbots, their use cases, and chatbot design characteristics were extracted from the articles and information from other sources and by accessing those chatbots that were publicly accessible. RESULTS: Our search returned 3334 articles, 61 articles met our inclusion criteria, and 61 chatbots deployed in 30 countries were identified. We categorized chatbots based on their public health response use case(s) and design. Six categories of public health response use cases emerged comprising 15 distinct use cases: risk assessment, information dissemination, surveillance, post-Covid eligibility screening, distributed coordination, and vaccine scheduler. Design-wise, chatbots were relatively simple, implemented using decision-tree structures and predetermined response options, and focused on a narrow set of simple tasks, presumably due to need for quick deployment. CONCLUSION: Chatbots’ scalability, wide accessibility, ease of use, and fast information dissemination provide complementary functionality that augments public health workers in public health response activities, addressing capacity constraints, social distancing requirements, and misinformation. Additional use cases, more sophisticated chatbot designs, and opportunities for synergies in chatbot development should be explored. Covid-19 imposed significant demands on healthcare systems, making effective use of available resources and capacity expansion vital. The social distancing requirements to slow virus transmission imposed further restrictions on using traditional in-person services and required solutions that accommodated social distancing. Furthermore, the fast-shifting landscape of information about the virus that included reversal of policies, 1 and the rapid diffusion of disinformation and misinformation created anxiety and confusion. 2 This resulted in a surge in the number of calls to helplines. 3 One response to these issues involved the deployment of chatbots as a scalable, easy to use, quick to deploy, social-distanced solution. Chatbots are algorithms that interact with users using natural language, either text or voice, 4, 5 with their distinguishing feature being natural language conversational interactions. Given the use of chatbots in a variety of settings prior to Covid-19, the existing infrastruc-ture and abundance of prebuilt modules resulted in their rapid development and deployment to address Covid-19 needs. In addition, offthe-shelf Covid-19 chatbot solutions quickly became available. [6] [7] [8] [9] [10] [11] These offer a wide range of functionality that can be easily customized for each organization, including risk assessment of patients and employees, information dissemination by answering common questions, surveillance, and integration with existing infrastructure. We systematically searched the literature to identify chatbots deployed in the Covid-19 public health response. We gathered information on these to (a) derive a comprehensive set of chatbot public health response use cases and (b) identify their design characteristics. Our study adds to an emerging literature on the use of chatbots in healthcare in general (see Car et al. 12 and Montenegro et al. 13 for reviews), and in the Covid-19 response in particular (see Almalki and Azeez 14 for a review). We conducted a scoping review of the scientific literature to identify studies (journal articles, conference proceedings, and preprints) reporting or proposing the use of chatbots in the Covid-19 public health response. We limited our search to articles in English published before October 10, 2020 and conducted a follow-up search on July 29, 2021. Our search terms included all combinations of terms for "chatbots" (e.g., conversational agents) and Covid-19. Figure 1 presents our search terms and summarizes the article selection process which is based on the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. 15 We searched 3 sources (PubMed/MEDLINE, Web of Knowledge, Google Scholar) and engaged in a 2-stage screening process to identify relevant articles. First, we reviewed the title and abstract of articles matching our search terms to identify papers that met the minimum inclusion criteria. Second, we searched the papers' relevant citations using the same process. We used the following inclusion criteria: (a) the paper focused on the implementation, use, or potential use of a chatbot and (b) was used in the Covid-19 public health response. We excluded articles where (a) chatbots were tangential to the paper, such as papers discussing use of AI or telemedicine during the pandemic, (b) only included a general discussion of chatbot benefits without referring to a specific chatbot use case, or (c) chatbots were used for other purposes during the pandemic (e.g., domestic abuse victims support 16, 17 ). Our search yielded 57 academic articles and 50 unique implemented chatbots. We excluded proposed chatbots from our sample but discuss additional use cases identified from these articles in our discussion section. We then used Google search to gather additional information on each of the 50 chatbots. An additional 17 chatbots were incidentally identified in this search (e.g., a news arti- cle mentioning the chatbot we had identified mentioned additional chatbots). We included these as our objective was to identify a comprehensive set of chatbot public health response use cases. Our data collection was supplemented by accessing these chatbots to gather more information about their design and use. We gained access to 47 out of the 67 chatbots. For chatbots not conversing in English, we used Google Translate to understand the interaction. We could not access chatbots that required organizational credentials, customer or patient accounts, local phone numbers (except for the USA), or national identification numbers for access. Instead, we relied on the literature and news articles for information on these. Therefore, our analysis of design characteristics has an overrepresentation of publicly accessible chatbots. This does not influence our use cases since chatbot objectives were described in the articles. We excluded 9 cases from our sample since our analysis revealed that they were not chatbots. We performed another search on July 29, 2021 for any additional use cases. We identified 3 new chatbots that focused on vaccination, bringing our final sample to 61 chatbots and resulting in 1 additional use-case category and 1 new use case. The 61 chatbots reflect a global sample of chatbots deployed in more than 30 countries. These include 33 chatbots that conversed in 45 languages other than (or in addition to) English. Tables 1 and 2 in Appendix 1 provide background information on each chatbot, its use cases, and design features. References to case numbers below refer to the corresponding chatbots in Appendix 1. We categorized these chatbots based on (a) their use case which reflects the public health response activity they supported and (b) their design characteristics. We used qualitative methods to allow our use cases and use-case categories to emerge from our data. Specifically, both authors engaged in open coding (see Miles and Huberman 18 ) where we identified the public health response activities that the chatbots supported. We iterated between our data and our codes to identify use cases. We then engaged in axial coding to aggregate use cases into use-case categories. Finally, we independently categorized the chatbots based on their use case(s) and design features. We resolved differences through discussion. We were unable to assess some chatbots on some attributes because of variations in available information. Further, as a chatbot could belong to multiple categories (e.g., delivered multiple use cases), our numbers do not always add up to 61. We identified 6 use-case categories that captured how chatbots were deployed in the Covid-19 public health response: risk assessment (34 cases), disease surveillance (12 cases), information dissemination (42 cases), post-Covid eligibility screening (2 cases), distributed coordination (1 case), and vaccine scheduler (1 case). These 6 categories consist of 15 distinct chatbot public health response use cases described in Table 1 . Risk-assessment chatbots (N ¼ 34) triage users based on their Covid-19 symptoms and exposure risk and make behavioral and clinical (e.g., go to emergency room) recommendations. They enable social distancing by reducing footfall in healthcare facilities, which results in capacity expansion and better capacity utilization of the healthcare system. Such chatbots have been developed by countries to help reduce the burden on their healthcare systems [19] [20] [21] and by healthcare providers and institutions for their patients, customers, students, or employees. 22, 23 Risk-assessment chatbots ask a series of questions to triage users. While many chatbots leverage risk-assessment criteria from official sources (WHO, CDC, or other government health agency), the questions asked vary significantly across chatbots, and as does the order in which they are asked. Some ask general questions about exposure and symptoms (e.g., Case 7), whereas others also check for preexisting conditions to assess high-risk users (e.g., Case 1). Based on the assessed risk, the chatbot makes behavioral recommendations (e.g., self-monitor, quarantine, etc.). In cases of Covid-19 exposure combined with symptoms, recommendations across chatbots vary. Chatbots from healthcare facilities provide links to establish a video call or make an appointment or to initiate a telemedicine session (e.g., Case 4). Other chatbots ask users to call an emergency number or their physician and provide links to official resources (e.g., Case 5). Two chatbots direct users to another chatbot for a more detailed screening (Cases 8 and 29). Although not claiming to diagnose, a few chatbots also try to eliminate differential diagnoses by asking more detailed questions (e.g., Case 41) . The chatbots also differ in how they present their assessment. Most present the user's exposure risk and next steps. A smaller group (3 cases) provides a report and explains the reasons behind their recommendation (Cases 15, 22, and 36). Data collected by some chatbots are used for disease surveillance (7 cases), which is a separate use case. They do so by periodically checking with users for symptoms or Covid-19 exposure. The joint use of risk assessment and disease surveillance use cases is a prevalent combination for chatbots by organizations and schools which monitor disease among their employees or students (3 of 5 cases). 10, 23, 24 Surveillance chatbots We identified 4 use cases for surveillance activities. The first use case, disease surveillance (7 cases), involves gathering disease prevalence data which enables early assessment of virus transmission and prediction of clusters and outbreaks. Early detection allows officials to formulate a prompt and appropriate response, including the nature and timing of nonpharmaceutical interventions such as lockdowns and travel restrictions. 25 Some chatbots also collect concerns about the pandemic from the public to inform policy, 21 which illustrates a second surveillance use case-public reporting (3 cases). For example, some allow users to report rumors and misinformation (e.g., Case 57) which enables officials to quickly respond to curb misinformation. 26 Another (Case 19) also allows users to report guideline violations, which enables prompt investigation and violations reduction. 21 The third use case, Covid-19 symptoms surveillance (2 cases), collects data on Covid-19 symptoms experienced by those who have or had the virus. This enables the mapping of the multitude of Covid-19 symptoms, their frequency, and the sequence in which they manifest. Cross-location comparison of symptoms and sequences allows assessment of how the virus affects different communities. 27, 28 The final use case, proactive monitoring (3 cases), involves proactively monitoring at-risk populations, such as the elderly, 28-31 by checking whether users are experiencing symptoms or have been exposed to the virus. Unlike disease surveillance chatbots where the user initiates the interaction, these chatbots initiate contact with the users and ask questions about symptoms. At the onset of the pandemic little was known about Covid-19 and information and guidelines were in constant flux. The public had many questions and concerns regarding the virus which overwhelmed health providers and helplines. As a response, chatbots were deployed to provide current and reliable information and to educate the public about the coronavirus, its symptoms, transmission, and more recently, vaccines; to fight misinformation and disinformation (passively or proactively); to promote nonpharmaceutical interventions to slow transmission (e.g., social distancing, wearing masks); to inform the public about the state of the virus at local and national levels; and to provide information regarding available resources including vaccinations. We were able to assess the type of information provided for 37 of the 42 information dissemination chatbots (see Table 2 in Appendix 1). Based on the information they provided, we identified 7 use cases for information dissemination (see Figure 2 ). Many chatbots provided multiple types of information (26 cases). Education and information about nonpharmaceutical interventions was the most common combination with 15 chatbots providing both, but as Table 2 shows, chatbots exhibited a wide range of configurations of information dissemination use cases. Post-Covid eligibility screening chatbots A different screening use-case category involves screening recovered Covid-19 patients for eligible volunteers for an activity such as plasma donation (Cases 2 and 9). 32 These chatbots first ask if the user has been diagnosed with Covid-19 and check if they have been symptom-free long enough to be eligible for the activity. An interesting use case-mHero 33,34 -involves facilitating coordination between distributed frontline healthcare workers and health organizations or the Ministry of Health in areas of poor technology infrastructure (1 case). The chatbot can gather real-time data from frontline workers to enable provision of essential support, answer their questions, and provide them with real-time information. Originally developed in response to the Ebola outbreak to reach frontline workers with basic text and audio messages, 33 it can now also be implemented in WhatsApp and Facebook messenger. Its impact on fighting the pandemic in countries with less developed technology and healthcare infrastructures makes this an important use case, 33 and unlike our other use cases that are public-facing, this illustrates how chatbots can be used to support frontline healthcare worker needs. Chatbots were also used for scheduling vaccine appointments (1 case). 35 The chatbot searches for appointment availability across various locations and automates the appointment scheduling process. This enables more efficient utilization of available vaccines, reduces wait times in vaccine centers, and allows users to easily find available appointments. Target audience for use cases Chatbots were designed either for the general population (35 cases) or for a specific population (17 cases). The general population audience could be as broad as the world (e.g., the WHO chatbot) or a country (e.g., the CDC chatbot in the United States). Many state or regional governments also developed their own chatbots; for instance, Spain has 9 different chatbots for different regions. Specific population audiences were more diverse. Some focused on at-risk groups. For instance, in Brazil, a chatbot designed to support LGBT teenagers added Covid-19 information. 36 Others were developed by healthcare institutions and a diverse range of organizations for their customers, patients, students, and employees. Many of these screen employees (and students) to assess whether they are safe to report to work, and some provide a badge to "cleared" employees so they can report to work. [21] [22] [23] [24] While the majority of the specific population chatbots are developed for organizational use, an example of a countrywide-specific audience chatbot is the mHero platform deployed in several African countries, which connects frontline healthcare workers to the ministry of health. 34 Table 4 shows how the chatbot target audience varies across use cases. As shown in Figure 3 , the chatbots in our sample varied in their design along a number of attributes. Thirty-six chatbots delivered use cases in a single use-case category (which we term single purpose), and 25 delivered use cases across multiple categories (which we term multipurpose). The most common single-purpose chatbots were for information dissemination (21 cases) and risk assessment (12 cases). The most common categories to be combined were risk assessment (22 cases) and information dissemination (21 cases), with the most common multipurpose chatbot combination being these 2 categories (18 co-occurrences). Five chatbots delivered use cases in 3 or more categories (see Table 3 ). Appendix 2 shows the chatbot use-case combinations for the 15 use cases we identified. Chatbots were deployed on a variety of platforms, the most common being web-based (34 cases) and social media (22 cases). Less common were SMS (6 cases), phone call (4 cases), and standalone or healthcare apps (8 cases). Thirty chatbots were embedded within a specific organization's platform (e.g., Case 1, Clara on the CDC's website). Embedding a chatbot within a high-traffic platform can enhance its visibility and discoverability and reduce the effort required to engage with it. Chatbots varied in the degree of privacy afforded the user. While some do not require user identifying information (23 cases), one asks the user to voluntarily disclose identifying information (e.g., email) to enable the chatbot to follow-up (Case 34), while the rest require identifying information for access (29 cases) (e.g., telephone number, national identification number, social media account, institutional credentials). Most chatbots (70%) lacked anthropomorphism. The most common anthropomorphic feature was gender with 9 chatbots being female, 5 male, and 1 transgender. In addition, 1 chatbot had its gender randomly assigned for each interaction (Case 22) and 1 gave the user the option to choose (Case 28). Most chatbots were text-based (42 cases), 4 were voice-based, and 4 had both text and voice options. Further, most chatbots (35 cases) interacted via predetermined choice and response options (i.e., use system-directed initiative), 4 used free-form questions and responses (i.e., use user-directive initiative), and 9 used a hybrid dialog style. Finally, interactions with chatbots were primarily designed to be user-initiated, with only 3 chatbots initiating conversations (Cases 29, 34, and 51). Only 3 chatbots were designed to initiate follow-up (Japan's Prefecture Line chatbots (e.g., COOPERA) and CareCall), or recurring conversation (Alexa-My day for seniors skill) (Cases 34, 51, and 29). Another, at the University of California San Francisco (Case 12), sends an email or text to employees to remind them to use the chatbot to get an entry pass (e.g., prior to their shift) that clears them to report to work. Table 4 summarizes the chatbot design features for each chatbot use-case category and chatbot use case. Chatbots are being used as a complement to healthcare and public health workers during the pandemic to augment the public health response. Their scalability, wide accessibility, fast information dissemination, and substitution for in-person contact provide the functionality required to address the capacity expansion, social distancing requirements, and quick accurate information transmission needs of the public health response. The chatbots' ability to automate simple, repetitive tasks and to directly communicate with users enables quick response to multiple inquiries simultaneously, directs users to resources, and guide their actions. This frees up healthcare and public health workers to deal with more critical and complicated tasks and addresses capacity bottlenecks and constraints. Our findings extend those of a prior study on Covid-19 chatbots 14 that, based on a review of 9 studies, identified 5 chatbot application domains: disseminating health information and knowledge, self-triage and personal risk assessment, monitoring exposure and notifications, tracking Covid-19 symptoms and health aspects, and combating misinformation and fake news. Leveraging a larger sample of 61 chatbots, we add to these findings by (a) identifying additional use-case categories (post-Covid-19 eligibility screening, distributed coordination, and vaccine scheduler); (b) identifying specific use cases based on the variability of chatbot functionality in each use-case category; (c) identifying use-case combinations; and (d) examining the chatbots' design features. The majority of our chatbots focused on risk assessment or information dissemination and about half involved multiple use cases, mostly combining risk assessment and information dissemination together or with other use cases. While use cases were combined in many distinct combinations, which of these are most effective is an open question. A few observations from our results suggest future opportunities. Most risk assessment and disease surveillance chatbots did not fol-low-up on symptomatic users. Privacy concerns and regulations may have precluded this since following up requires that chatbots capture identifying information. Further, while surveillance could provide insights into the state of the pandemic, fewer chatbots were deployed for surveillance, and all but one combined surveillance with other use cases, suggesting that surveillance functionality was often a by-product. Moreover, though many chatbots leveraged risk-assessment criteria from official sources (e.g., CDC), there was variability in criteria across chatbots. A comparison of symptom-checker tools indicated great variability in effectiveness in terms of their sensitivity and specificity, 37 with some outperforming the CDC symptomchecker. Therefore, while utilizing official sources is a prudent practice, especially for off-the-shelf solutions and for non-healthcare organizations, more work is required to understand best practices. In a rapidly changing information environment (e.g., frequently updated guidelines, guideline reversal), information dissemination chatbots face the critical challenge of keeping their knowledge base up-to-date. Automating knowledge extraction of the latest information from reliable sources to automatically create question and answer pairs and update the chatbot's knowledge base is a viable solution to this challenge. 38, 39 With one exception, chatbots were designed for use by the public. The mHero use case for distributed coordination between frontline healthcare workers and healthcare organizations or the Ministry of Health highlights the potential for chatbots to support and gather timely information from frontline healthcare personnel. The development of multiple such use cases, including surveillance and logistics, would be especially beneficial as a frugal solution to bridge the digital divide in areas of poor infrastructure. The provision of behavior support is another promising area for chatbot use cases. While chatbots were commonly used to provide information, there is a gap between knowledge and action that chatbots could help bridge through repetition, monitoring and interventions, and by providing real-time suggestions to users. 4, 40 For instance, they could automatically suggest a route to avoid walking through crowded areas. Such use cases are more sophisticated and would require the use of sensor or geolocation data. More broadly, while the Covid-19 pandemic crisis had agencies and organizations scrambling to create their own chatbot solutions to address their needs, it may also be important to step Given that we were unable to assess all chatbots on all categories, and that some categories are not mutually exclusive, the numbers do not always add up to 61. b Several chatbots were deployed on multiple platforms. c There are also one transgender chatbot, one where gender is randomly assigned, and one where the user can choose the gender. back and understand where and how synergies and efficiencies of effort can be achieved. Differences in language even within the same country, differences in local information and guidelines, and differences in popularity of different social media platforms across countries may limit the scope of such efforts. Similarly, local development efforts may be necessary for chatbots that deliver combinations of use cases, some more universal (e.g., risk assessment) but others requiring local solutions (e.g., providing an entry pass after risk assessment (Case 12)). Nonetheless, economies of effort can occur (as we have observed) through off-the shelf solutions from vendors that organizations can customize to their needs. The 15 use cases that we have identified provide a basis for identifying functionality and customization options for different organizations or constituents. A set of guidelines and best practices to chatbot development vendors and to organizations by agencies such as the CDC can aid in coordinating efforts and in preparedness for the next pandemic. Design-wise, ease of development, ease of access, and platform penetration have likely contributed to the prevalence of web-based and social media chatbots. For example, the built-in modules for chatbot design on social media platforms allow for fast and easy development. Coupled with the popularity of social media, this fueled deployment on social media apps such as Line, WhatsApp, Facebook Messenger, WeChat, and Telegram. Furthermore, accessibility via both smartphones and personal computers makes such chatbots widely available. Though a minority, we highlight the importance of SMS-based and phone-call-based chatbots to bridge the digital divide and reach people who lack access to smartphones or reliable internet connections or lack the skills to use technology. Further, because of the vital need for a quick response to mitigate the surge in the healthcare system, most chatbots were developed very quickly. As such, their design is relatively simple using decision-tree structures, system-directed initiatives, and focused on a narrow set of simple tasks. More sophisticated designs based on machine learning and sensor data are fruitful directions for new or enhanced public health response use cases. Tighter integration with organizational processes may also be beneficial in some cases. 14, 41 Importantly, a digital divide persists among vulnerable populations worldwide. [42] [43] [44] For example, tech savvy individuals can use chatbots to quickly find available vaccine appointments broadening the vaccine gap among such populations. While the chatbot conversational interface and voice modalities are well suited to bridge the digital-skill divide, increased attention to bridging the digital divide with respect to platform accessibility can help broaden use, reduce access disparities, and help ameliorate pandemic health disparities. 45 We identified 6 broad use-case categories and 15 use cases where chatbots were deployed in the Covid-19 public health response. Chatbots are scalable, enable social distancing, augment the capacity of healthcare and public health workers, broadly disseminate information, and gather real-time information from a broad audience to inform public health interventions. They can easily be deployed on different platforms and have easy-to-use conversational interfaces that enable broad reach and access to different demographics. Chatbots are most commonly used for information dissemination and risk assessment, which are critical to public health response. Scant attention has been devoted to chatbot deployment to support or gather data from frontline healthcare workers which is suggestive of future work. Finally, the vast majority of chatbots do not followup on users beyond an initial interaction or users need to initiate contact. This highlights a potential tension between privacy and functionality, and balancing these could benefit use cases where follow-up or proactive contact may be useful. Our study did not examine factors related to chatbot effectiveness across different use cases or use-case combinations, and given alternative digital health tools (e.g., form-based symptom-checkers for risk assessment), whether and under what conditions chatbots offer a comparative advantage. These are fruitful directions for research. None. WHO reverses position on face masks as coronavirus cases climb Trump suggested 'injecting' disinfectant to cure coronavirus? We're not surprised Medical Helpline Calls Soar as COVID-19 Spreads Chatbots in the fight against the COVID-19 pandemic On the design of and interaction with conversational agents: An organizing and assessing review of human-computer interaction research Delivering Information and Eliminating Bottlenecks with CDC's COVID-19 Assessment Bot HealthConnect for COVID-19. Secondary HealthConnect for COVID-19 Tech's Role in Tracking, Testing, Treating COVID-19. Secondary Roundup: Tech's Role in Tracking, Testing, Treating COVID-19 Scale Customer Service with Watson Assistant and the Expertise of IBM. Secondary Scale Customer Service with Watson Assistant and the Expertise of IBM Conversational agents in health care: scoping review and conceptual analysis Survey of conversational agents in health Health chatbots for fighting COVID-19: a scoping review PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation Domestic abuse during the UK's Covid-19 lockdown: from normal to new normal and what survivors' experiences might teach us Women are using code words at pharmacies to escape domestic violence during lockdown Qualitative Data Analysis: An Expanded Sourcebook Federal actions to support and strengthen local efforts to combat COVID-19: primary health care (PHC) in the driver's seat Digital response during the COVID-19 pandemic in Saudi Arabia Designing a competent chatbot to counter the COVID-19 pandemic and empower risk communication in an emergency response system How Hospitals Are Using AI to Battle Covid-19 Implementation of a digital chatbot to screen health system employees during the COVID-19 pandemic Women's College Hospital Mental health of family, friends, and co-workers of COVID-19 patients in Japan Covid-19 does not look like what you are looking for: clustering symptoms by nation and multi-morbidities reveal substantial differences to the classical symptom triad Syndromic surveillance insights from a symptom assessment app before and during COVID-19 measures in Germany and the United Kingdom: results from repeated crosssectional analyses Carecall: a call-based active monitoring dialog agent for managing covid-19 pandemic Early SNS-based monitoring system for the COVID-19 outbreak in Japan: a population-level observational study The relationship between fever rate and telework implementation as a social distancing measure against the COVID-19 pandemic in Japan A semantic web framework for automated smart assistants: COVID-19 case study Three Early Digital Health COVID-19 Response Success Stories Chatbot for COVIDVaccine using deep learning The Brazilian PrEP1519 Study Group, et al. Adapting to the COVID-19 pandemic: continuing HIV prevention services for adolescents through telemonitoring Diagnostic accuracy of web-based COVID-19 symptom checkers: comparison study A qualitative evaluation of language models on automatic question-answering for covid-19 Collecting Verified Covid-19 Question Answer Pairs Lost in knowledge translation: time for a map? A guide to chatbots for COVID-19 screening at pediatric health care facilities Digital Divide Persists Even as Lower-Income Americans Make Gains in Tech Adoption About a Quarter of Rural Americans Say Access to High-Speed Internet Is a Major Problem Digital Gap Between Rural and Nonrural America Persists Bridging the health disparity of African Americans through conversational agents The authors contributed equally to this work. Supplementary material is available at Journal of the American Medical Informatics Association online. None declared. The data underlying this article are available in the article and in its online supplementary material.