key: cord-0812997-9vfko5bn authors: Marescotti, Diego; Narayanamoorthy, Chandrasekaran; Bonjour, Filipe; Kuwae, Ken; Graber, Luc; Calvino-Martin, Florian; Ghosh, Samik; Hoeng, Julia title: AI-driven laboratory workflows enable operation in the age of social distancing date: 2021-12-17 journal: SLAS Technol DOI: 10.1016/j.slast.2021.12.001 sha: afa392ba778253716360c1c170a6f1c97b5c126d doc_id: 812997 cord_uid: 9vfko5bn The COVID-19 (Coronavirus disease 2019) global pandemic has upended the normal pace of society at multiple levels—from daily activities in personal and professional lives to the way the sciences operate. Many laboratories have reported shortage in vital supplies, change in standard operating protocols, suspension of operations because of social distancing and stay-at-home guidelines during the pandemic. This global crisis has opened opportunities to leverage internet of things, connectivity, and artificial intelligence (AI) to build a connected laboratory automation platform. However, laboratory operations involve complex, multicomponent systems. It is unrealistic to completely automate the entire diversity of laboratories and processes. Recently, AI technology, particularly, game simulation has made significant strides in modeling and learning complex, multicomponent systems. Here, we present a cloud-based laboratory management and automation platform which combines multilayer information on a simulation-driven inference engine to plan and optimize laboratory operations under various constraints of COVID-19 and risk scenarios. The platform was used to assess the execution of two cell-based assays with distinct parameters in a real-life high-content screening laboratory scenario. The results show that the platform can provide a systematic framework for assessing laboratory operation scenarios under different conditions, quantifying tradeoffs, and determining the performance impact of specific resources or constraints, thereby enabling decision-making in a cost-effective manner. We envisage the laboratory management and automation platform to be further expanded by connecting it with sensors, robotic equipment, and other components of scientific operations to provide an integrated, end-to-end platform for scientific laboratory automation. The COVID-19 global pandemic has upended the normal rhythm of society at multiple levels-from daily activities in personal and professional lives to the way businesses and the sciences operate. The disruption has affected various operations at different levels, from complete "lockdown" to "work from home" to partial or "socially distant" work procedures. The pandemic crisis has posed unprecedented challenges to the operation of scientific research laboratories across the world. Particularly, it has led to shortages in vital supplies, changes in standard operating protocols leading to suspension of operations because of social distancing and stay-at-home guidelines. These changes have led to limiting their operational capabilities causing delays and loss in productivity 1, 2, 3, 4, 5 . While the entire scientific community has acknowledged such social-distancing measures as a critical step to slowing down the pandemic, these measures have come at a cost, leading to a huge "loss of scientific progress" 6, 7, 8 . Thus, the pandemic has led to a growing need for enabling laboratory operations in the "new normal" 1, 2, 3 . Despite this awareness of the challenges and costs of "shelter in place" and social distancing on laboratory research, most responses have focused on identifying remote operational issues and developing re-entry roadmaps for risk and safety compliance checklists, with little to no use of digital technologies for redefining laboratory operations 1, 8 . Rapid progress in digital technologies-particularly sensing modalities for the internetof-things, artificial intelligence (AI), and robotics-provide promising avenues for digital transformation of laboratory operations. Particularly in the area of laboratory automation, the major focus has been on robotics 3 , specifically on using robotic arms to automate specific processes 9, 10 . Similarly, electronic laboratory notes 12 and cloud-based documentation and sharing platforms 13 have enabled process automation of laboratory functions. Recent developments in augmented or virtual reality, robotic process automation, and conversational interfaces (voice) have opened potential new opportunities for automating laboratory processes 14 . Laboratory operations involve complex, multicomponent systems involving equipment (robotics or devices), reagent supply chains, protocols, human resources, data handling and quality control, high-performance computing, and analytics. It is unrealistic to completely automate the entire diversity of laboratory processes. It would require significant investment in terms of equipment, facility planning and organization, and training of personnel. With recent developments, particularly in the role of game simulations for training machine learning algorithms, AI technologies have made significant strides in modeling and learning complex, multicomponent systems 15, 16, 17, 18 . Simulation models, which capture various layers and constraints, play a pivotal role prior to the building of actual systems, and they require significant cost and time resources. Game simulation models have been used extensively in AI development; for example, the DeepMind AlphaGo 19 program (from Google DeepMind) combined deep reinforcement learning to defeat the human champion in the ancient game of Go by generating millions of game moves to explore the parameter space of the gameplay and present novel moves. Various simulation techniques (e.g., agent-based simulations and Monte Carlo and discrete event simulations) are incorporated to study the interaction dynamics of individual components in a system, which are represented as "digital twins" or avatars 20 with different parameters. Further, the system can be augmented with reallife data from various activity monitors, equipment sensors, cameras, and robotic instruments to capture multiple layers of granularity. Thus, a platform which connects diverse laboratory processes and data sensing and monitoring modalities while leveraging the power of simulation-driven learning can provide the foundation for laboratory automation by flexibly combining human and machine intelligence to optimize performance and productivity under various constraints and risk scenarios as exemplified by the COVID-19 global pandemic. This paper outlines a laboratory management and automation platform customized and developed by SBX Corporation, based on the Garuda platform 21 . It combines multilayer information (physical layout, operational information, data, and computational information) on a simulation-driven learning engine to plan and optimize laboratory operations under various constraints and risk scenarios. Specifically, the platform provides: a) Integration of multidimensional information layers for laboratory management, b) Connectivity of the different information layers on a single, secured, and privacy-preserving private cloud-based dashboard and c) Simulation-driven learning engine, which combines game simulation dynamics with learning and optimization techniques for hypothetical scenario generation Based on the Garuda platform, a connectivity and automation technology, the laboratory management and automation platform were developed as a customized solution which provides an automation dashboard with different modules (called gadgets) for the different layers, as shown in Figure1 (details in Figure S1 ). The Garuda platform provides the underlying connectivity between the gadgets and allows configuration, customization, and development of new gadgets on the automation dashboard. The LM encapsulates the facility and equipment layers and provides the ability to capture the floor plan and layout of the laboratory room, allowing users to design, modify, and view the room configuration. The manager allows users to create a new laboratory floor plan layout with specific facility materials (furniture and equipment) and their configurations, including dimensions, orientation, and layout ( Figure S2a ). The catalog panel allows configuration of facility inventory, which can be used to select specific materials from the inventory to be configured for a specific laboratory. The design panel allows both 2D and 3D view of the layout as well as mapping of a layout to a specific laboratory to represent a specific laboratory ( Figure S2a , 2b, 2c). The LM module is connected to the other modules on the automation dashboard and serve as the baseline for the operational and information layers. The LM provides a flexible option for changing layouts to address needs arising from different operational and business conditions. It can also be used to create multiple layout configurations for a laboratory by changing the physical location of specific equipment or putting constraints on movement of people (social distancing). These layout configurations can be associated with a specific workflow and simulated on the platform to plan the optimal layout configuration for different operating conditions. The WM captures the operational information associated with the SOP for an experiment in a laboratory. Specifically, the WM is used to encode the details of an experimental protocol-the tasks, subtasks, detailed description of the tasks, associated devices (equipment as outlined in the LM), time of execution, and role of personnel associated with the workflow ( Figure S2d ). The workflow forms the baseline of an experimental protocol associated with a laboratory. A 8 laboratory can have multiple workflows assigned, depending on the experiment focus area, personnel, or devices used and can be duplicated or merged to create complex processes. Further, task details can be duplicated and reused across different workflows or assigned to different laboratory layouts ( Figure S2f It is important to note that the workflow analysis in WM is "static" i.e., the distributions are computed on the baseline workflow and does not consider the dynamics of changing the task dependencies, number of devices, personnel, or other constraints of operation. The SM and RM together form the core of the platform for HSG based on a specific laboratory layout and associated workflows. The SM allows simulation of a laboratory operation scenario under various operating conditions, constraints, and parameters. An HSG ( Figure S3a ) defines a combination of the laboratory layout, workflow, and operating conditions (parameters) for a simulation run in two key dimensions: Workflow parameters: Defines the specific workflow and laboratory layout for a simulation, together with the task dependencies in the specific workflow. The dependencies of the task establish the sequence of actions for the tasks and subtasks for a given workflow in a particular HSG. Simulation parameters: The parameter control panel ( Figure S3c The factors include, Parallelization using AI: automatic parallelization of tasks based on dependency during the HSG execution; Use alternative devices: enable the HSG to use alternative devices for a given task or subtask where applicable for expediting execution; Normalize factor: normalize the time of execution for a task or subtask to expedite the simulation run time. There are more advanced options which allow further granular control to the optimization engine on the HSG, including, Speed of personnel: Optimize the speed of personnel; All tasks are independent: allows hyper-parallelization of tasks, particularly for automated processes; Use same personnel as much as possible: allow control of personnel reuse to optimize resource utilization; give priority to longer duration: Allow the optimization engine to prioritize tasks based on their time distribution. In summary, the HSG represented in Figure S3a As mentioned earlier, the platform presented in this work builds a novel simulationdriven optimization engine, which uses the LM and WM to define and simulate different HSGs under various operating conditions. In this section, we delve into the core components of the simulation-driven optimization engine ( Figure S4 ). The engine consists of two main components, the details of which are explained below. The first is the Simulation processing engine, composed of the controller, configurator, and The complete flow of the simulation command-control is outlined in Figure S5 , including initiation of the simulation servers, connection to the LM and WM, loading of simulation parameters in the simulation engine, and creation of the task queue. The simulation and resource (player) engines are synchronized, with the simulation engine executing the sequence of tasks from the task queue and the player engine controlling the interaction dynamics of the resources and constraints and their states in the system. The process for task queue creation (green highlight in Figure S5 ) in the commandcontrol flow employs an optimization module for parallelization of tasks in the workflow. Specifically, the module employs a graph-based topological sort algorithm to generate a directed acyclic graph (DAG) associated with a given workflow-first, generation of a task graph, followed by identification of parallel tasks and subtasks (subgraph detection) and, finally, merging of tasks to generate the DAG through a breadth-first search routine on the topological sort results of the graph nodes. This optimization module is executed in the command-control flow when the "parallelize using AI" option is selected in the parameters panel of the SM. The engine is built on and customized on Garuda libraries for supporting various simulation and optimization algorithms and is implemented on the Unity Framework [24] for game engine simulations. As outlined in this section, the simulation-driven optimization engine is built in a modular and extensible manner on the Garuda platform to support addition of modules for learning (e.g., scheduling, spatial intelligence, sequence generation, and optimal route selection), real-life (sensor) data integration, etc. (see Figure S6 ). The laboratory management and automation platform presented in this work is a computeintensive process, particularly the simulation-driven optimization engine, which requires massive computation cycles for exploring the combinatorial space of different workflow and simulation parameters associated with an HSG. Further, different what-if scenarios may be executed for different workflow combinations, leading to multiple HSGs being generated for a given workflow. Thus, it is imperative to engineer the platform development and deployment for scale. In this section, we highlight some of the key system engineering design, technology architecture, and deployment frameworks for the platform. As detailed in the Methods section, the system was developed on the Garuda platform to leverage the modularity of components, connectivity, and automation features on a cloudbased architecture. Each of the components (gadgets) of the LM, WM, SM, and RM were developed as individual services but by using the same technology stack (the front-end and back-end stack technologies are described in Figure S7a However, the cloud-based approach together with the individual service-oriented architecture also enables deployment of the system on distributed multi-server systems ( Figure S8b ). Its support for distributed deployment is critical for scaling the simulation engine to leverage graphical processing unit (GPU) systems and other dedicated computation configurations for future development of machine learning and AI modules on the platform. The platform is delivered as a suite of Docker containers ( Figure S7a ) deployed on Linux servers by Ansible playbooks and roles covering all stages of the process, from basic configuration of the server to deployment and customization of the application modules. This approach is easily parametrizable, allowing reproduceable deployment on premise or on public, private, or hybrid cloud. Access details on the specific software tools are provided in Figure S7c . In summary, the platform design, development, and deployment architectural choices were governed by the goals of scalability, modularity, extensibility, and Continuous Integration, Continuous Deployment (CICD) to enable support for large-scale simulationdriven optimization of laboratory operation management and automation. A concept video demonstrating some key features of the platform is available as supplementary material. The execution of preclinical in vitro studies is a complex procedure which requires integration of multiple fundamental highly interconnected steps including: 1) study ideation, 2) laboratory activity preparation, 3) laboratory activity execution, and 4) data analysis and reporting. While The COVID-19 pandemic presents potential challenges to normal study lifecycle management and experimental execution, with the need for remote or work-from-home restrictions and communication being restricted to online conferences rather than in-person meetings. Figure 2 shows the changes in laboratory operations in the COVID-19 period relative to the normal pre-COVID-19 period. As can be seen from the illustration, these changes require planning and scheduling of laboratory operations to prioritize safety (ensuring social distancing and minimizing contact between technicians) while optimizing laboratory performance (in terms of the execution time of assays and multiplexing of multiple assays). As part of planning and scheduling experiments in the "new normal," the simulationdriven optimization platform presented in this work was used to assess the different workflow combinations under different operating conditions (with and without social-distancing restrictions). Figure 3b shows the key workflows and assessment criteria. Specifically, four workflows in increasing complexity of operating protocols were tested on the platform: a) Assay #1 1x: Baseline assay #1 workflow (color-coded in yellow); b) Assay #1 3x: Three assays of the Assay #1 workflow (for triplicate measurement; color-coded in green); c) Assay #1 3x + 1X Assay #2: Combination of three Assay #1 workflows with an additional Assay #2 workflow (color-coded in orange) and d) Assay #1 3x + 3X Assay #2: Multiplexing of three Assay #1 workflows with three Assay #2 workflows (color-coded in blue). The assessment criteria, as shown in Figure 3b The baseline workflow simulations (called Assay #1 1x, 1-5 in Supplementary Table 1 ) serve as the benchmark for establishing the ability of the platform to capture the normal working condition of the assay (approximately 2 days). As the baseline workflow has more device utilization than personnel utilization, increasing the number of personnel or the impact of social distance does not lead to a significant change in performance (execution time). For three assays of the HCS workflow in a 1-week time window (called Assay #1 3x Assay #1 Weekly, rows 6-10 in Supplementary Table 1) , the platform leverages the parallelization within the different task or subtasks of the workflow (intra-SOP parallelization) to enable performance gain (the workflow is executed in approximately 4 days and does not require three times the baseline for Assay #1 1x). The impact of personnel or social distancing is limited, as seen in the HCS 1x case, except in the case of three personnel with social-distancing constraints, where the execution time is increased (no. 10 in Table 1a ). Thus, for the Assay #1 3x workflow, operating with two people under social-distancing constraints allows optimal performance (no. 8 in Supplementary Table 1) . When the protocol is updated to include an additional Assay #2 assay with the 3x Assay #1 (called Assay #1 3x + Assay #2 1x Assay #1 Weekly and 1x RGA, rows 11-15 in Supplementary Table 1) , the platform can achieve parallelization efficiency between the two workflows. In fact, as shown in row no. 13 in Supplementary Table 1, the execution time is only increased by 1 h for the Assay #1 3x + 1x Assay #2 workflow relative to the Assay #1 3x assay workflow (from approximately 4 days 5 h to 4 days 6 h). To delve deeper into these results, the timing sequence of events generated by the simulation run for these two cases (nos. Table 1 ) are shown in Figure 4 . The Manhattan plots (refer to Figure 4 legend for details) of the timing sequence show that the system attains efficiency by scheduling "bursts" of simultaneous activities (taller bars) between periods of low parallelization, where mainly device-only tasks are executed. However, the sequence distribution shows that the platform distributes the tasks and subtasks of the combination workflow (Assay #1 3x + Assay #2 1x) across the entire week (orange boxes in the two workflows), thereby leveraging parallelization across the workflows and increasing performance efficiency. The final workflow combination, which mandates the multiplexing of three Assay #1 and three Assay #2 workflows (called Assay #1 3x + 3X Assay #2 Assay #1 Weekly + 3X Assay #2) and is shown in row no. 16-25 in Supplementary Table 1 . The role of optimal scheduling of the workflows is evident in this complex 3x + 3x case, as the performance is severely impacted in cases where parallelization of the workflows (both intra-and inter-SOP) is not optimized, irrespective of the number of personnel or social-distancing restrictions, as seen in cases no. workflows, optimization is critical prior to assignment of personnel and social-distancing constraints to achieve performance within a 5-day period (weekly time window). Table 1 While the results elucidated in this section are focused on the specific requirements for the HCS laboratory, the platform can be flexibly configured to assess different laboratory settings and conditions. This laboratory management and automation platform provides a foundation for enabling the planning, operation, and analysis of complex laboratory processes. As shown in this study, the various modules within the platform address specific components of the operational processes, from laboratory layout and inventory planning to workflow and protocol management to simulation of different conditions (HSGs) and interpretation of the results to enable decisionmaking. The current platform and its application in the use cases outlined in the Results section showcase the potential of the system to enable quantified decision-making for operational processes rather than making decisions only based on prior experience, intuition, or trial-anderror methods, which are expensive in terms of cost and risks. There are various dimensions in which the laboratory management and automation platform is limited in its application and can be enhanced and expanded, both in the context of laboratory operations and the broader areas of scientific operations and office work 25 . Specifically, one dimension for the platform is around the simulation engine (SM module), which does not capture the impact of equipment usage, inventory, layout, and position through a spatial intelligence component. This dimension can be enhanced with the addition of sensing modules (motion, temperature, pressure, or vibration sensors) that can capture real-time data on actual operations and can be used for simulation-driven inference 15, 22 and optimization, which is currently lacking in the platform. Further, the learning modules in the simulation engine are currently limited in their scope and can be developed to support different classes of learning algorithms-specifically, reinforcement learning (RL) modules 17, 18, 19 . While the current platform does not incorporate real-time data, addition of RL modules can learn and search for optimal solutions in real-time from different conditions (for example, find rescue-mode operations in case of a sudden change in operations involving equipment malfunction or accidental reagent spill, which can impact interdependent workflows). Combination of simulation-driven optimization and an RL module for adaptive learning in real-time based on sensor data can help build a robust platform for laboratory management. Another dimension of laboratory automation is the increasing use of robots and robotic equipment 3, 16, 25 . A synergistic combination of human-machine collaboration across a spectrum of operations provides a realistic use case for robotics, particularly in the post-pandemic era 25 . The current platform does not support the integration of robotics in the system. The platform outlined in this work, however, provides a modular architecture that can support the integration of robotic systems into the workflow, allowing users to first "simulate" their dynamics prior to physical deployment in efficient and cost-effective configurations. While the current work encapsulates the core modules for layout, workflow, simulation, and results, we envisage that the laboratory management and automation platform will further connect with other components of scientific operations, including quality management, inventory and logistics management, facilities management, training and data management, and financial management systems, to provide an integrated, end-to-end platform for scientific laboratory automation in the times of COVID and beyond. Philip Morris International is the sole source of funding and sponsor of this research. The lab automation platform was customized and developed based on a proprietary platform software developed by SBX Corporation, Japan. Covid-19 changed how the world does science, together COVID-19: Biomedical research in a world under social-distancing measures Using Robotics in Laboratories During the COVID-19 Outbreak: A Review Coronavirus: social distancing is delaying vital scientific research. The Conversation Coronavirus pandemic forces some research labs to shut down The COVID-19 pandemic and research shutdown: staying safe and productive Science-ing from home Robotic crowd biology with Maholo LabDroids The automation of science How to pick an electronic laboratory notebook Systematizing scientific laboratory work by a workflow and template for electronic laboratory notebooks. Education for Chemical Engineers do science! Voice-activated assistants hit the lab bench The frontier of simulation-based inference Robothor: An open simulation-to-real embodied ai platform Playing hide-and-seek, machines invent new tools. Quanta Magazine From chess and atari to starcraft and beyond: How game AI is driving the world of AI Google AI algorithm masters ancient game of Go Review of digital twin about concepts, technologies, and industrial applications Software for systems biology: from tools to integrated platforms Accelerating Simulation-based Inference with Emerging AI Hardware, 2020 International Conference on Rebooting Computing (ICRC) 25th International Conference on Automation and Computing (ICAC) Unity: A general platform for intelligent agents Soft Office: a human-robot collaborative system for adaptive spatial configuration We thank Sindhoora Bhargavi Gopala Reddy for language editing support and Stephanie Boue and Fabian Moine for figure design and preparation.