key: cord-1017929-bqu5780k
authors: Sipior, Janice C.
title: Considerations for Development and Use of AI in Response to COVID-19
date: 2020-06-09
journal: Int J Inf Manage
DOI: 10.1016/j.ijinfomgt.2020.102170
sha: 7be2979e96dae9ad6d37238923c3990ef17d24d5
doc_id: 1017929
cord_uid: bqu5780k

Artificial intelligence (AI) is playing a key supporting role in the fight against COVID-19 and perhaps will contribute to solutions quicker than we would otherwise achieve in many fields and applications. Since the outbreak of the pandemic, there has been an upsurge in the exploration and use of AI, and other data analytic tools, in a multitude of areas. This paper addresses some of the many considerations for managing the development and deployment of AI applications, including planning; unpredictable, unexpected, or biased results; repurposing; the importance of data; and diversity in AI team membership. We provide implications for research and for practice, according to each of the considerations. Finally we conclude that we need to plan and carefully consider the issues associated with the development and use of AI as we look for quick solutions.

Virtually everyone across the world has been impacted by the COVID-19 pandemic originating from Wuhan China and wants quick solutions. Vint Cerf, vice president and Chief Internet Evangelist at Google and past Association for Computing Machinery (ACM) president, sees an "awesome responsibility!" as "our profession and the products it creates will have a prominent role in shaping our post-COVID-19 society" (Cerf, 2020) . Let's hope we can quickly seize this responsibility. Indeed, the pandemic is expected to accelerate the rate and pace of technological change, particularly for the emerging AI revolution (Wladawsky-Berger, 2020). (Sabherwal, Sabherwal, Havakhor, and Steelman, 2019) . Of course, IT does not drive strategy. Rather what is needed is strategic leadership to develop unique insight to solve a pressing need and then drive change to realize a vision. For example, at biotech start-up Insilico Medicine, based in Hong Kong, while the focus is still on using machine learning to find drug treatments for cancer, fibrosis, and many other diseases, the Chief Executive Officer (CEO) prioritized several projects related specifically to COVID-19, and their platform has been repurposed for virus research (Field, 2020) .

For AI applications, companies may consider a Chief AI Officer (CAIO) to present opportunities for innovation which have the potential for dramatic impact, by contrast to the CIO who focuses on the delivery of IT services and the effect on firm performance. Davenport and Dasgupta (2019) recommend instituting competence centers (CC) or centers of excellence (COE) "to entrench AI" (p. 2). Coordinated, collaborative, centralized oversight of AI can be achieved with CCs or COEs, rather than taking a bottom-up approach to adopt AI applications, which is inefficient and unproductive (Davenport and Dasgupta, 2019) . The CAIO is a natural position to head up such centers, which could house AI talent. Thus, the main responsibility of the CAIO is to identify areas appropriate for the application of AI, across the entire company, to add value in meeting the goals and objectives of the business strategy. Additionally, the CAIO will, in coordination with internal and external partners and vendors, manage development teams, oversee implementation of applications, and continuously monitor the performance of those applications.

AI can encounter unpredictable or unexpected occurrences in the data, and correspondingly generate unpredictable or unexpected results, including biased output, because of the complex nature of AI solutions. Bias in AI systems can "affect the brand value of the organization" (Gartner, 2020, p. 12) . Thus, a consideration of bias as well as unpredictable or unexpected results in AI is essential. Machine learning-based systems currently in use may encounter vastly different data, reflecting the changed circumstances of life imposed by the coronavirus lockdowns (Heaven, 2020) . For example, detecting fraud in credit card transactions relies on rules and predictive models to interpret a consumer's spending pattern data and provide a fraud prevention payment score to approve the transaction. However, spending behavior has changed as people are not purchasing big-ticket items or spending in new places as they usually had done, requiring adjustments for the increase in purchases of garden equipment and power tools (Heaven, 2020) .

Degree of confidence in AI results, or in data sets, may spill over to trust in the results. Level of familiarity with the use of AI may also impact trust in the recommendations made. Sufficient trust is necessary for an individual to act based upon the recommendations (Engler, 2020) . What level of trust should patients and clinicians place in a system designed to predict which infected patients will become severely ill, to better prepare to deliver lifesaving medical treatment? In a 2019 survey, 2,000 consumers in the US were asked "When you think about AI, which feelings best describe your emotions?" (Davenport, 2019) . The results revealed the most common response (45%) to be "Interested," followed by "Concerned" (40.5%), "Skeptical" (40.1%), "Unsure" (39.1%), and "Suspicious" (29.8%). Davenport (2019) suggests that this deficiency of trust in AI is attributable to overhyping the claims regarding the capabilities of AI systems, a lack of transparency in revealing characteristics of the system and how it will be used, and a lack of external certification of the output produced. Engler (2020) regards the role AI is playing in response to COVID-19 to be overhyped in news article, in terms of tasks it performs, its effectiveness and scale, and disregard of the level of human involvement.

Biases may arise due to historical patterns of bias repeated in an AI application or learned by the application itself through machine learning. Specific instances of bias in machine learning algorithms have been reported (Yapo and Weiss, 2018) , contrary to the claim that the neural network basis of machine learning affords "speed, accuracy and lack of bias" (Marr, 2016, emphasis added) . Since "AI has not yet been impactful against COVID-19" (Naudé, 2020b, p. 1), we present unrelated examples to show that bias can occur in the use of AI. Among the biases uncovered in AI applications are gender bias, in Google's search (Carpenter, 2015) and speech recognition software (Larson, 2016) ; and racial bias in calculating scores to predict the likelihood of committing a crime (Angwin, Mattu, Larson, and Kirchner, 2016) and in face recognition software (Crawford, 2016) .

According to Weyerer and Langer (2019) , three stages contribute to AI-based bias: (1) AI input, (2) AI learning, and (3) AI outcome. Algorithmic bias appears to be the most frequently cited bias (Pandya, 2019) . This bias arises from "how the software is designed, developed, deployed and the quality, integrity, and representativeness of the underlying data sources" (Pandya, 2019, para. 9) . Data sets used to train neural network based systems can be inaccurate, biased, or incomplete. The neural network learns to recognize patterns in the data by weighting factors found within the data, thereby generating results that reflect imperfections within the data set. Often, historical hiring data from a company is used, with the intention of capturing attributes and traits of high-performing employees, which then may replicate prior institutional bias (Pandya, 2019) . To mitigate or manage bias, it may be more important to achieve diversity in development and deployment teams, than in the training data set (Shellenbarger, 2019) because these teams could undertake an audit of the AI system for bias. AI itself could assist in screening for bias patterns in the characteristics selected to serve as attributes, in the data set, or in the model itself.

Repurposed AI may be susceptible to producing unpredictable, unexpected, or biased results as the model was initially trained for the original purpose. Repurposed AI is not new, but may offer promise in an environment requiring rapid responses to find quick solutions. With a desire to find ways to speed up the development and implementation of promising COVID-19 applications, repurposing might be quicker because a similar existing application could be modified, rather than initiate the development of an entirely new system.

For example, consulting firm PwC developed an AI-based Automatic Contact Tracing tool, by repurposing an existing system initially used to track assets within a building (Sreeharsha, 2020) . Indoor geolocation is used to track contact among employees in an office environment. Ambient signals, which are the constant background field of radio signals, are captured. Millions of these wireless signals are aggregated into a graph to analyze how these signals interact to determine the time and proximity of employee interactions at work. Information from this contact tracing system can be used to determine with whom an infected employee has had contact to reduce the risk of COVID-19 cases spreading within a company. Reports describing this system reported no details about the repurposing.

In another example, Smartvid.io, headquartered in Cambridge, MA, USA, offers an AI software product that assesses risk and safety at work sites for construction companies (Sreeharsha, 2020) . This system uses a combination of cameras and various deep learning approaches for computer vision to analyze construction-focused images to look for specific indicators of risk in imagery. For example, is a worker on a ladder or working at heights? Aggregate data indicates the overall risk. Leveraging the capabilities of this software, Smartvid.io produced an AI-powered enterprise software product that detects if two individuals are closer than one-person length apart and if more than 10 individuals are gathered together. This system now also analyzes images to assist in enforcing social distancing on-site.

In one more example, Qure.ai Technologies Pvt. Ltd., a startup based in Mumbai, India, repurposed an AI system, originally developed to detect tuberculosis, to assist with diagnosing and assessing COVID-19 cases by checking computer tomography (CT) lung scans for signs and severity of the disease (Olson, 2020) . A team led by the head of radiology, Francesco De Cobelli, at San Raffaele Hospital in Milan, Italy, immediately found the system made the disease look worse than it was. However, this AI application is in the early stages of development. Dr. De Cobelli's team spoke daily with engineers from Qure.ai, who retrained the system using hundreds of X-rays of COVID-19 patients. Qure.ai CEO, Prashant Warier, said the early issues were resolved and that San Raffaele Hospital is now a paying customer. By comparison, it has taken five years to validate AI systems that can detect breast cancer. Skepticism remains as British radiologist and health-tech firm consultant Hugh Harvey commented, "There's no way anyone has done that specifically for Covid in the past three months" (Olson, 2020, para. 6) .

What can we learn from examples of repurposed AI? Little research has been done on repurposing information systems in general, or AI in particular. Mawson-Lee (2006) reported that a system designed to manage all requests made to the IT unit within a company, "can be easily repurposed… for a wide range of task management purposes" (p. 79), but with no explanation of considerations in doing so. Regarding repurposing of AI, Russell, Dewey, and Tegmark (2015) note the necessity to maintain control over system training and use. They argue that in repurposing, if that system is "selecting the actions that best allow it to complete a given task, then avoiding conditions that prevent the system from continuing to pursue the task is a natural subgoal" (p. 111). AI systems which can be controlled during repurposing are termed "corrigible" (Soares, Fallenstein, Yudkowsky, and Armstrong, 2015) . Santoni de Sio and van den Hoven (2018) identify two requirements for an autonomous system to remain under human control. The first, termed a "tracking" condition, specifies that the system should respond to both moral reasons for designing and deploying the system and relevant facts in the environment. The second, termed a "tracing" condition, stipulates that the system be designed to trace back the outcome of its operations. Bergstein (2020) notes that AI systems can be confused by new situations, inherent in being trained for one task and being taught to perform a different task. However in this repurposing process, the system may not retain all of the expertise gained in performing the original task, which is termed "catastrophic forgiving."

AI uses vast quantities of historical data, with known outcomes, to discern patterns in that data and make predictions about the future. The accuracy of those predictions depends upon the quality of the data set used to train the model, the quality of the data set used to produce the prediction, the design of the algorithm, and the performance of the system using real-world data (Silverman, 2020) . In real-world settings, the AI system must be trustworthy in its performance, which depends upon the data. For example, researchers have expressed skepticism about the real-world performance of models, developed to detect COVID-19 in the lungs, touted by developers as attaining a 90-96 percent accuracy rate (the percent of predictions correct during development), because the training data set is not representative of actual patients (Council, 2020; Olson, 2020) . The amount of data and the diversity of the populations it represents help boost the accuracy of the system (Council, 2020) . Nonetheless, 96 percent accuracy "is suspiciously high for any machine learning problem" and real-world deployment nearly always degrades system performance (Engler, 2020, para. 10) .

Wide-scale access to relevant big data is key. If training data is not sufficient and relevant, the output produced will be inferior or incorrect. Rather than repeat the old adage "Garbage in, garbage out," we instead agree that "There's another law of artificial intelligence and machine learning: The more data, the more accurate the results will be" (Kessler, 2020, para. 8) . A major barrier to success with AI is a lack of good data and a major risk to organizations is not understanding the context of that data (Zielinski, 2017) . The selection of the data set should be substantiated for its intended use in the model; and the quality and relevance of the data set should be subject to rigorous assessment (Board of Governors of the Federal Reserve System, 2011). Specifically, training data must be vetted for quality, integrity, and representativeness in terms of diverse characteristics of the underlying data sources (Pandya, 2019) as well as accuracy, relevance, and freshness (Roselli, Matthews, and Talagala, 2019) . Data used in production processing may differ from the training data set. Thus, production data should be monitored for completeness, the expected distribution anticipated, uncommon cases, and timeliness (Roselli et al., 2019) .

Much of the data being produced about COVID-19 is compiled in separate silos by the government, academia, and business (News Bites -Private Companies, 2020). With frequent new discoveries made about the virus, real-time updates must be made to the data and to the models (Council, 2020) . A cohesive data strategy, that lacks silos and enables deep learning technologies to create new insights, is necessary.

An example of an effort to provide for integrated data access is that of C3.ai, a leading enterprise AI software provider headquartered in Redwood City, CA, USA, which produced a vast open cloud-based repository, known as a data lake (Kessler, 2020) , A variety of large data sets from a multitude of different sources (see C3.ai, 2020) is provided, including location and confirmed case data from the World Health Organization and Johns Hopkins, genome sequences of COVID-19 samples from the National Center for Biotechnology Information Virus Database, global patient data such as symptoms and lab results, and the Milken Institute's database of treatment and vaccine trackers (News Bites -Private Companies, 2020). The C3.ai Data Lake, one of the largest integrated sources of COVID-19 data in the world, is publicly available, at no cost, to global research and scientific communities (C3.ai, 2020). Pre-established linkages enable researchers to easily navigate and explore all of the associations within and across the unified data sources, which are updated in real-time. The CEO of C3.ai, Tom Siebel, commented "This enables scientists to perform very advanced research using AI, accurately predict the spread of the disease, and evaluate the efficacy of social mitigation" (News Bites -Private Companies, 2020).

The use of data requires a consideration of data ethics and regulations, especially within the context of healthcare. Though the considerations are relatively new and vary from source to source, some principles are universal (The Institute of Internal Auditors, 2020). Regarding ownership and consent, it is individuals who own and therefore grant consent for the use of their personal data. Once consent is provided, transparency and disclosure about how the data is used must be provided. Privacy and security should be maintained through all reasonable efforts. In terms of currency, individual should be made aware of any financial transactions resulting from the use of their personal data. Compliance with regulations which govern and control the use of personal information must be maintained.

A diversity of disciplines is the key to success in AI, especially to minimize risk associated with rapid deployment of new technologies to respond to the coronavirus (Gartner, 2020) . Such teams enable alignment of "a company's culture, structure, and ways of working to support broad AI adoption" (Fountaine, McCarthy, and Saleh, 2019, p. 64) . Team membership should be wellrounded, from a wide range of backgrounds and skill sets, for complex problem-solving with innovative solutions and for recognizing the potential for bias. To address issues such as bias, ethics, and compliance, among others, roles such as an "AI Ethicist," attorney, and/or review board, may be added (Davenport and Dasgupta, 2019, p. 5) . "The biases that are implicit in one team member are clear to, and avoided by, another,… So it's really key to get people who aren't alike" (Shellenbarger, 2019, para. 10) . Engler (2020) underscores the importance of subjectmatter experts, especially for COVID-19 applications.

In our discussion, we highlighted the considerations for developing and deploying AI in the context of COVID-19 related applications. For these considerations we have formulated research questions.

In aligning IT strategy with business strategy, future research could evaluate actual common crisis-management successes and failures applicable to planning for and responding to pandemics. The results could be applied to the formulation of a conceptual framework framed specifically around pandemics that can be used to assist in strategy formulation.

There is very limited academic research focusing on understanding the impact of the use of AI applications applying rigorous methods and theories (Duan, Edwards, and Dwivedi, 2019) . With the potential for unpredictable, unexpected, and biased results in an environment within which development and deployment is hastened by necessity, future research to increase this understanding becomes critical.

AI can exacerbate existing biases and create new biases in judgment decision making (Angwin et al., 2016; Carpenter, 2015; Crawford, 2016; Larson, 2016; Marr, 2016; Yapo and Weiss, 2018) . Biases may arise due to biases encoded in an AI application itself, learned by the application through machine learning, or introduced by the users. Methods with which to mitigate them could be explored. For example, level of trust in cognitive technologies such as AI and machine learning, and their ability to produce the same judgment as human decision makers, may lead to bias. Degree of confidence in AI or in data gathered may spill over to confidence in interpreting the results. Level of familiarity with the use of AI may impact bias. If biases persist or emerge, new methods to mitigate bias may have to be developed. For example, mechanisms to mitigate bias may be built into the AI application itself, such as ensuring algorithm transparency by providing an explanation of the results or decision recommended.

Research on repurposing AI is clearly needed as it is an approach being used to hasten development during the COVID-19 pandemic, but within which little research has been done. Such research might yield insights beneficial for future pandemics and other crises requiring quick solutions. Similarities between the specific purposes of the original and repurposed systems could be explored to determine when this development approach is appropriate. The development techniques useful in repurposing could be defined and documented. The impact of repurposing on bias could be explored to discover whether new biases are created due to the change away from the original purpose. Additional research could further explore "corrigible" AI systems, "tracking" conditions, "tracing" conditions, and "catastrophic forgiving," among other characteristics.

Extensive research has been done on data quality focusing on the reach, solutions, and methods for data quality assessment (Madnick, Wang, Lee, and Zhu, 2009 ). However, little research examines the actual level of data quality within organizations (Nagle, Redman, and Sammon, 2020) . We concur with the dire necessity to assure data quality, especially in AI applications to support the healthcare industry. Nagle at al. (2020) propose a typology of data quality which could serve as a useful starting point to formulate preemptive actions companies could take against the most frequent types of data error.

Previous research has examined diversity in AI teams by considering whether the focus should be on the diversity of the team or on the strength of each member, and whether a team of diverse and weak members can outperform a uniform team of strong members (Marcolino, Jiang, and Tambe, 2013) . Such questions are particularly relevant when development and deployment efforts are to be undertaken quickly. Further, Shellenberger (2019) encourages diversity in the development team to mitigate or manage bias in AI systems, which may be particularly relevant to quickly built repurposed AI. Future research could examine the contributions and capabilities of both development and deployment team members according to the specific purpose of the AI system.

Companies are expected to ramp up automation, including AI application development, in the aftermath of the coronavirus outbreak (Loten, 2020) and in doing so, they must learn quicker than ever before (Wladawsky-Berger, 2020). AI is playing a key supporting role and key in using AI is thoughtful and informed planning and preparation which addresses many considerations.

Implementation requires leadership. Companies may consider a CAIO as the key role to determine how resources devoted to AI might be used for advantage in realizing business strategy. In coordination with CCs or COEs and/or internal and external partners and vendors, the organizational response can be rolled out and monitored by the CAIO.

Mechanisms for contending with unpredictable, unexpected, or biased data, and the potential for such results must be in place, especially in the case of repurposed systems. Remnants from the system's original purpose may remain, causing incorrect results.

Among the considerations addressed, it is repurposed AI which offers promise for rapid availability in real-world application. However, as previously cautioned, these systems must be fully tested for the new use before being deployed. Historical patterns of bias may be repeated in the repurposed system or new biases may arise. Loss of confidence in AI results may spill over to loss of trust in the results, which could ultimately lead to disuse of such an expensive resource.

An AI system should be used with caution because the accuracy rate attained during development may not be attained in real-life situations. Hence, wide-scale access to relevant quality data is key to thoroughly train the system before it is deployed. Additionally, data ethics and regulations, especially within the context of healthcare, must be addressed.

The goal of diverse AI team membership is to attain the best possible performance of the system in use, in compliance with data ethics and regulations. Thus, it is team member knowledge which is the focus in team member selection.

The goal of this paper is to provide considerations for organizations in the development and use of AI systems for COVID-19, and any future pandemics. AI has a role in assisting with many relevant questions, including diagnosis, contact tracing, social distancing, workplace safety, and more. Many considerations must be addressed for effective use in real-world application to ensure the solutions meet the needs of the business as mistakes could slow progress in dealing with the virus. Included among the many considerations are aligning information technology (IT) strategy with business strategy; considering unpredictable or unexpected results, including biased output; repurposing AI to find quick solutions; the importance of the role of data as the lifeblood in AI; and the critical necessity for diversity in team membership. Among the considerations addressed, it is repurposed AI which has received little research attention but offers promise for rapid availability of systems in real-world application. We hope for success, in this approach and others, in the quest for AI systems to benefit humanity as we respond to COVID-19 and beyond.

Machine bias: There's software used across the country to predict future criminals. And it's biased against blacks

Mar/Apr). AI still gets confused about how the world works

COVID-19 Puts Spotlight on Artificial Intelligence

Board of Governors of the Federal Reserve System

Google's algorithms show prestigious job ads to men, but not to women. Here's why that should worry you

Implications of the COVID-19 pandemic

NIH, Nvidia Use COVID-19 Patient Data to Build Diagnostic System

Artificial intelligence's white guy problem. The New York Times

Can we solve AI's 'trust problem'?

How to set up an AI center of excellence

Artificial intelligence for decision making in the era of Big Data-evolution, challenges and research agenda

A guide to healthy skepticism of artificial intelligence and coronavirus. The Brookings Institution

Inside a startup's pivot to have AI fight coronavirus

Building the AI-powered organization

Healthcare Technology Innovations for Identifying and Managing COVID-19 Patients. Gartner

Our weird behavior during the pandemic is messing with AI models

How Big Data Is Attacking the Coronavirus

AI transforming the enterprise 8 key AI adoption trends

Research shows gender bias in Google's voice recognition. The Daily Dot

AI for business: what's going wrong, and how to get it right

Businesses Expected to Lean on Automation for Recovery

Overview and framework for data and information quality research

Multi-agent team formation: Diversity beats strength?

What is the difference between artificial intelligence and machine learning? Forbes

The online effect: Transitioning from the legacy help desk to the online task management system

Artificial intelligence-enabled rapid diagnosis of patients with COVID-19

The ABCs of Assured Autonomy

Assessing data quality: A managerial call to action

Artificial Intelligence against COVID-19: An early review

Artificial intelligence vs COVID-19: Limitations, constraints and pitfalls

C3 IoT: A deep "data lake" for coronavirus information. News Bites -Private Companies

AI Software Gets Mixed Reviews for Tackling Coronavirus

Can artificial intelligence be biased? Forbes

Managing bias in AI. Companion Proceedings of The 2019 World Wide Web Conference

Research Priorities for Robust and Beneficial Artificial Intelligence

How does strategic alignment affect firm performance? The roles of information technology investment and environmental uncertainty

Meaningful human control over autonomous systems: A philosophical account

A crucial step for averting artificial intelligence disasters

Tackling COVID-19 requires better governance of AI and other frontier technologies -here's why

Artificial Intelligence and Ethics AAAI

Computer vision could help enforce social distancing in the workplace; with no coronavirus vaccine in sight, good behavior will be crucial to keeping people safe when companies start to reopen plants and offices

Global knowledge brief: Data ethics. The Institute of Internal Auditors

Garbage in, garbage out: The vicious cycle of ai-based discrimination in the public sector

What history tells us about the accelerating AI revolution

Ethical implications of bias in machine learning

Recruiting gets smart thanks to artificial intelligence