key: cord-0112633-883xvirz authors: Roelofs, Sean; Landry, Benoit; Jalil, Myra Kurosu; Martin, Adrian; Koppaka, Saisneha; K.Y.Tang, Sindy; Pavone, Marco title: Vision-based Autonomous Disinfection of High Touch Surfaces in Indoor Environments date: 2021-08-25 journal: nan DOI: nan sha: 34ba59faf9eda32c7251ba35bfb4c8c193ef3dbe doc_id: 112633 cord_uid: 883xvirz Autonomous systems have played an important role in response to the Covid-19 pandemic. Notably, there have been multiple attempts to leverage Unmanned Aerial Vehicles (UAVs) to disinfect surfaces. Although recent research suggests that surface transmission is less significant than airborne transmission in the spread of Covid-19, surfaces and fomites can play, and have played, critical roles in the transmission of Covid-19 and many other viruses, especially in settings such as child daycares, schools, offices, and hospitals. Employing UAVs for mass spray disinfection offers several potential advantages, including high-throughput application of disinfectant, large scale deployment, and the minimization of health risks to sanitation workers. Despite these potential benefits and preliminary usage of UAVs for disinfection, there has been little research into their design and effectiveness. In this work, we present an autonomous UAV capable of effectively disinfecting indoor surfaces. We identify relevant parameters such as disinfectant type and concentration, and application time and distance required of the UAV to disinfect high-touch surfaces such as door handles. Finally, we develop a robotic system that enables the fully autonomous disinfection of door handles in an unstructured and previously unknown environment. To our knowledge, this is the smallest untethered UAV ever built with both full autonomy and spraying capabilities, allowing it to operate in confined indoor settings, and the first autonomous UAV to specifically target high-touch surfaces on an individual basis with spray disinfectant, resulting in more efficient use of disinfectant UAVs have seen widespread use in response to the Covid-19 pandemic. They have delivered medical supplies, including personal protection equipment (PPE) & Covid-19 tests, enforced quarantine restrictions, tracked crowd sizes, and even delivered Girl Scout cookies. One particular use of UAVs that has garnered a lot of attention is mass spray disinfection. UAVs with disinfectant spraying capabilities have been used in countries including China, the United States, India, Spain, France, Indonesia, and Nigeria [1] [2] [3] [4] [5] [6] [7] . Many of these UAVs were originally built for spraying agricultural fields but were re-purposed during the pandemic. Recently, scientific research has shown that the risk of contracting SARS-CoV-2 (the virus that causes through contact with fomites [8] , defined as inanimate objects or surfaces contaminated with infectious agents, is relatively low. The United States Center for Diseases Control and Prevention (CDC) estimates that each contact with an infected surface has a less than 1 in 10,000 chance of resulting in an infection with SARS-CoV-2 [9] . Even so, recent modeling results suggest that fomites may be an important source of SARS-CoV-2 transmission, particularly in schools and child daycares, as the virus can persist on some surfaces for up to 72 hours [10] . Other human coronaviruses (e.g. CoV-229E, MERS-CoV and SARS-CoV) may also transmit via surface contact [11] [12] [13] . In addition to coronaviruses, many other viruses may spread via fomite transmission. Norovirus, the leading cause of acute gastroenteritis that causes 19-21 million infections and 570-800 deaths annually in the United States [14] , is known to spread due to surface contamination [11, 15, 16] . Many other pathogens, such as adenoviruses and influenza A H1N1 can also spread through contact with contaminated surfaces and objects [11, [17] [18] [19] [20] . Offices, restaurants, sports and music venues, and other public indoor spaces are prone to fomite transmission. Frequent surface disinfection could therefore reduce the spread of these pathogens. In a study that tested various surfaces inside a classroom for the presence of the human coronavirus CoV-229E, doorknobs and desk tops were frequently found to be contaminated [13] . We therefore focus our efforts on the disinfection of high-touch surfaces, such as door handles, to reduce the spread of pathogens, including SARS-CoV-2, via fomite transmission. There are several advantages to using UAVs to disinfect surfaces. UAVs that disinfect surfaces could be deployed en masse, spray high volumes of disinfectant, and limit sanitation workers' exposure to contaminated areas. Given the potential benefits and preliminary deployments, several researchers have conducted investigations into the use of UAVs for spray disinfection -especially since the start of the Covid-19 pandemic. However, the existing literature fails to investigate using UAVs to a) spray disinfectant indoors, b) spray disinfectant with a high level of autonomy, and c) target specific high-touch surfaces with disinfectant. These are important areas of research because, respectively, many pathogens last longer on indoor surfaces sheltered from the elements, Fig. 1 . The UAV is capable of autonomously identifying door handles (a high-touch surface) in unstructured environments and disinfecting them with an ethanol solution. The proposed system is an important step towards harnessing robotics to mitigate the spread of infectious diseases. unlocking the benefits of mass deployment depends on autonomous functionality, and targeting high-touch surfaces increases the efficiency of sprayed disinfectant. In order to address these shortcomings, this work makes the following important contributions: 1. we design, to the best of our knowledge, the first small scale and fully autonomous UAV with disinfectant spraying capabilities, 2. we verify, on a representative sample, that the sprayed disinfectant achieves the coverage necessary to neutralize many common viruses. Here we provide a brief overview of existing work related to the use of autonomous UAVs for disinfection. In the wake of the pandemic, there have been several limited investigations into semi-autonomous disinfectant robots and UAVs, and their hardware capabilities. [21, 22] present tele-operated ground robots equipped with sprayers. [23] [24] [25] present UAVs equipped with sprayers. Specifically, [25] and [23] look at the trade off between drone weight and volume of disinfectant carried. In both instances, there is mention of indoor flight, but no evidence that the UAVs flew indoors or could do so without direct manual control is provided. Moreover, none of those UAVs have autonomous capabilities greater than the ability to follow preset waypoints, and the effectiveness of the spraying method is not investigated. [26] and [27] investigate how spray parameters including flying height, flying speed, and spray rate impact a disinfectant-spraying UAV's ability to cover a surface. [26] takes a purely theoretical approach and calculates that varying these parameters can lead to disinfectant coverage between 30 and 0.24 g/m 2 . [26] also explores tradeoffs between area covered and flight time. [27] investigates how height and speed impact the coverage of a surface with an actual UAV. These lines of research are relevant to our work, because our approach also requires specific spraying parameters to adequately cover a target surface. However, unlike previous work, we propose spraying a relatively smaller target (as opposed to a large swath of ground), so the corresponding spraying parameters are different. Finally, most relevant to this work is existing literature on complete UAV spray systems. First, [28] presents PaintCopter, a fully autonomous UAV that spray paints patterns on surfaces. Notably, this UAV remains tethered during operation and therefore does not address some of the design considerations relevant to our problem. Second, [29] presents a UAV equipped with ozone disinfectant for use in outdoor settings. This research is unique because it analyzes the decision to use ozone as a disinfectant and shows that ozone has sterilizing properties against SARS-CoV-2. Similar to this work, we also present an analysis of our disinfectant choice and its sterilizing properties; however, we look at viruses beyond SARS-CoV-2 and use a more widely available disinfectant. Although quickly growing, the existing body of works on the utilization of autonomous UAVs for the disinfection of indoor surfaces remains incomplete. Most notably, all of the systems mentioned above lack at least one crucial component that would prevent their widespread adoption. Specifically, no existing spraying UAV has the ability to operate autonomously, apply disinfectant in a targeted manner, and safely maneuver indoors. Our work is unique in its attempt to simultaneously address all of these shortcomings, and represents a significant step forward towards the more widespread adoption of UAV systems in addressing concerns related to public health. In this section, we introduce the UAV and its hardware. The design was guided by the competing requirements to have full on-board autonomy and spraying capabilities while being small enough to fly indoors. We used a combination of commercially available, custom, and 3D printed parts. To our knowledge, this is the smallest untethered UAV ever built, either commercially or academically, with both autonomous and spraying capabilities, which allow it to operate in unstructured confined indoor settings. The UAV is built on a Holybro S500 quadrotor base. This base is small enough to fly indoors but large enough to mount the compute, sensors, spraying system, and other components. We use a 4-cell battery and 900kV 2812 motors to generate sufficient thrust. At full load, this power system allows the UAV to stay airborne for over 10 minutes. The UAV's compute is provided by a Nvidia Jetson Xavier NX. This lightweight edge computing device runs Ubuntu and has a 21 TOPS GPU, which is essential for running object detection on board in real time. The UAV also has a Pixhawk 4 flight controller for lower level motor control and radio communication. The UAV uses two visual sensors. The first is the Intel RealSense Depth Camera D435i. This camera provides 30fps 480 × 640 RGBD images. The D channel encodes the distance each pixel is from the camera. The second is the Intel RealSense Tracking Camera T265. This camera provides highly accurate real time pose estimation through use of internal visual odometry. Together, these cameras report the information needed to estimate the UAV's position and generate a 3D map of its environment. These cameras are mounted to the UAV with a custom 3D printed mount. This mount keeps the cameras at a known transformation relative to each other. Finally, there is a suite of integrated sensors including an IMU in the flight controller. The UAV is equipped with a spray system consisting of a tank, pump, and nozzle. The pump (Adafruit, Peristaltic Liquid Pump) draws disinfectant from a 250 mL fuel tank re-purposed to hold the disinfectant. The disinfectant travels through silicone tubing with 2 mm inner diameter and 4 mm outer diameter to a spray nozzle of diameter 80 µm (Victory Innovations, VP50). The whole system is mounted on the UAV with a custom 3D printed mount. The flow rate of water from the nozzle was measured to be 2.6 oz/min. There are many parameters that could change how the UAV sprays disinfectant on a target surface. For example, when detecting a door handle to disinfect, the UAV must pick a location relative to the handle from which to spray disinfectant, and that relative location could be modified. In this section we find the spraying parameters needed to neutralize virus particles on a surface with the UAV. To effectively disinfect a surface, we aim to select a disinfectant solution that targets a wide variety of pathogens and is able to inactivate them within a short period of time. The World Health Organization recommends 70% ethanol as a disinfectant as it inactivates a wide variety of pathogens [30] . Feline calicivirus (a norovirus surrogate), adenovirus, H1N1, and many coronaviruses are effectively inactivated by 70% ethanol within 1 minute [8, [31] [32] [33] [34] in carrier tests. Carrier tests determine the effectiveness of a disinfectant on a virus that has been dried on a surface and are more robust than other tests (e.g. suspension tests) in determining the effectiveness of a disinfectant [35] . For these reasons, we selected 70% ethanol as the chemical disinfectant solution for the system. For a spraying-capable UAV to disinfect a surface, it must generate a film of disinfectant on the target surface that persists long enough to inactivate the target pathogens. As discussed in the previous section, this means the UAV, using a 70% ethanol solution, must generate a film that persists for at least 60 seconds. There are several parameters on a spraying-capable UAV one could use to satisfy this constraint; however, we chose to focus only on spraying distance and spraying time. We chose these parameters because: 1. spraying-capable UAVs can readily adjust their spraying distance and time, 2. using these parameters does not assume any control over the spraying mechanism, 3. generating a film that persists for 60 seconds should always be possible by spraying closer to the surface for a longer duration. We considered two objectives, which compete with each other, when choosing the spraying distance. The first objective is to avoid collisions with obstacles, and the second is for a significant proportion of the droplets to reach the surface. Spray too close and the UAV will collide with walls and handles; spray too far away and the droplets (when sprayed horizontally) will miss the target due to gravity and downdraft caused by the propellers. Fig. 2 . Droplets are swept away by the propellers' downdraft. Spraying too far from a target surface will result in fewer droplets reaching the surface, but spraying too close will result in a greater chance of collision between the UAV and the surface. To quantify the effect of gravity and downdraft, we sprayed a known amount of disinfectant onto a vertical surface attached to a scale while the propellers were running. With 30 cm between the spraying nozzle and target surface, 73% of the droplets (by mass) landed on the surface. Past 30 cm, there was a significant decline in mass of droplets reaching the surface. Therefore, we determined 30 cm was the farthest distance from a surface we could spray while still obtaining a reasonable spray coverage We want the UAV to spray the minimal amount of time such that the 70% ethanol will maintain a film on the target surface for at least one minute to inactivate many common viruses. Since we are spraying from a UAV onto a vertical surface, we expect the ethanol coverage to reduce rapidly due to the combined effects of evaporation and gravity. In order to determine the minimal time required, we sprayed disinfectant onto a vertical surface (the door handle) for 1, 2, and 3 seconds and observed how long the film of ethanol remained on at least 90% of the exposed door handle surface. We conducted this test with the UAV immobilized (its landing gear was mounted to a chair) and its sprayer 30 cm from the door handle. Before spraying, we turned the UAV's propellers on to create the downdraft that pulls droplets away. We recorded a video of the ethanol solution as it evaporated off the door handle and quantified the coverage area in Figure 3 by manually tracing the portion of the area covered by ethanol in ImageJ immediately after spraying and one minute after spraying. Based on these results (Figure 3 ), we find that spraying for more than 1 second is required in order to create adequate (≥90% for 1 min) coverage of the door handle, and spraying for more than 2 seconds uses more disinfectant than necessary. Therefore, we chose 2 seconds for the UAV's spray duration. To autonomously spray door handles, the UAV must be able to detect, localize, and compute a spraying pose for previously unseen handles in real time. Our approach is inspired from [36] and uses the same YOLO [37] object detection model to provide bounding boxes of handles and doors. When a handle and door are detected, we generate a point cloud representation of each using their respective bonding boxes and the depth channel from the RGBD image. However, unlike [36] , our perception system has to work at a greater distance from doors to avoid collisions. To do so, we devised a novel method for localizing the handle and computing a spraying pose. The UAV detects handles from a relatively large distance to both reduce collisions and to keep most of the door associated with a handle in the camera's field of view. The downside of detecting from a large distance is that the point cloud representation of a handle region has a low point density. With a low point density and noisy depth channel, it becomes difficult to differentiate points belonging to the actual handle from points belonging to the nearby door surface or door frame. We can approximately localize a handle by computing the raw centroid of the handle region. However, this method results in a large variance in the direction perpendicular to the door due to the noise of the sensor, the ratio of actual handle points and door frame points captured in the handle region, and the potential to include some points from the door frame which may be recessed or protruding (see the red oval in Figure 4) . To address this, we fit a planar model to the door region using the RANSAC algorithm as implemented in the Point Cloud Library [38] . Then we project the raw centroid of the handle region to the door plane to reduce the variance in the direction perpendicular to the door. Finally, to calculate our best estimate of a handle's position we use a constant offset in the direction of the door's normal vector to account for a handle's protrusion from the door's surface. The whole process is summarized in Figure 4 . We compute the spraying position from the handle position and the door normal such that the when in the spraying position, the UAV's spraying nozzle points perpendicular to the door and is 30cm from the handle. See To autonomously spray door handles, the UAV must have several advanced software features including mapping, collision free path planning, path following, and a state machine. In this section we provide an overview of these software components. The UAV creates and updates a 3D map of its environment in the form of an voxel grid, a efficient volumetric representation of free, empty, and unexplored space. This map is created from point cloud measurements paired with the UAV's pose during measurement. Figure 5 shows a map of a hallway that the UAV generated. Given a start position, end position, and 3D map of the environment, we must plan a 3D path from start to end that will avoid all collisions. To do so, we use the OMPL [39] implementation of RRT* [40] , an efficient search algorithm in complex spaces. This algorithm requires the UAV to predict if certain positions will result in a collision, which it does by modeling itself as a rectangular cuboid and checking for collisions with the voxel grid. Figure 5 shows a path the UAV planned around an obstacle. Given a series of waypoints from the path planning module, the UAV sends these waypoints to the PX4 flight controller, which then follows them using a cascaded PID-based controller. Fig. 5 . A visual representation of the 3D occupancy map estimated using the UAV's depth and tracking cameras. Also pictured is a path computed using RRT* that avoids the the obstacles in the map. At a high level, the UAV can takeoff, explore towards a set final goal, detour to spray door handles as it sees them, and land when it reaches the final goal (follow along in Figure 6 ). In all stages of flight, the UAV continuously estimates its position using the tracking camera and builds a 3D map of the world using its position and the depth camera measurements. The UAV starts by taking off with a preset final goal. Once takeoff is complete, the UAV immediately enters explore mode. In explore mode, the UAV plans and follows a collision free path towards the final goal. The UAV assumes that following this path will lead it through a hallway that could contain handles on either side. While following this path, the UAV yaws side to side in order to scan both walls. When the UAV detects both a door and a handle on the door with its CNN, the UAV stops exploring and switches to spray mode. In spray mode, the UAV computes the spray pose as described in section 5.2 and travels towards this pose while keeping the handle in its field of view. Once the UAV is within one meter of the spray position, it rotates to aim the sprayer towards the handle, waits to be within a small threshold of the desired spray position, and then turns the sprayer on for 2 seconds. When spraying is finished, the UAV returns to the center of the hallway and continues in explore mode. If the UAV triggers a fail safe like low battery or inability to reach the final goal, it will land. If the UAV does reach the final goal, it will land having successfully completed its mission. Our code and a video demonstration of the UAV are available to view at github.com/StanfordASL/surfacedisinfect. Given the context in which our proposed system is meant to be deployed, it is critical that we validate its ability to consistently and accurately localize and spray chosen surfaces in its environment. Indeed, substitut- Fig. 6 . At all times, the UAV is estimating its position and building a map of the world. To spray door handles, the UAV first takes off and then transitions between searching for door handles in explore mode and spraying the handles in spray mode. In explore mode, the UAV travels towards a final goal while yawing to see handles on walls. In spray mode, the UAV computes a spraying pose for a detected handle and disinfects the handle. The UAV lands once it reaches its final goal or encounters a safety concern. ing manual disinfection for an automated solution similar to the one presented in this work could exacerbate the spread of harmful pathogens if the autonomous system chosen were unable to achieve a minimum level of reliability. To this effect, we tested our proposed system in a motion capture environment that contained a door and a door handle. The motion capture system allowed us to validate the performance of the autonomous system via ground truth information. First, we determined where the target spray pose should be by pointing the UAV's sprayer directly at the door handle at a distance of 30 cm and recording the position reported by the motion capture system. We then ran 10 trials where the UAV started at different positions and headed towards a final goal that would lead it past the door and door handle. In all 10 trials, the UAV autonomously found and sprayed the door handle. See Figure 7 for a top down view of each path the UAV took towards the handle. While the drone was spraying the handle, its average distance from the target spray pose was 6.8 cm. In Figure 8 we plot the difference between the spray nozzle's ideal position and its actual position during each run. Notice that the deviation in the nozzle's position during an individual trial is relatively small compared to the deviation between trials. This indicates that, relatively speaking, the drone is better at holding a position than it is at localizing door handles in its environment. This further suggests that overall performance improvement could be most easily attained by developing better localization strategies than the one presented in section 5.1 . In this work, we present a UAV that can autonomously find door handles and spray them with disinfectant in unstructured, novel indoor environments. We showed that this UAV can reliably find and target door handles with disinfectant spray. We also demonstrated that disinfectant spray from this UAV could neutralize many common viruses on the surface of a door handle without using excessive disinfectant. As this UAV system was novel in many regards, our work exposes several avenues for future work. First, we found that building a system that was small enough to operate in indoor environments (such as hallways) put significant limitations on the UAV's ability to carry large amounts of disinfectant. Indeed, in its current configu-ration, the UAV is carrying enough disinfectant to spray only about 20 door handles. Additionally, future iterations of the UAV would ideally be able to spray a larger variety of surfaces. Moreover, the flight time of the UAV is dramatically affected by the payload requirements of this task. One potentially promising solution would be to use a UAV tethered to a mobile ground robot such that the UAV's compute, power, and disinfectant could all be stored on the ground. This would allow for vastly increased disinfectant capacity and flight time, while maintaining the maneuverability and targeting benefits of a UAV. Finally, future iterations of this UAV could allow for full coverage of all surfaces on a target object, including any crevices or surfaces that are difficult to reach, by having an adjustable nozzle that allows the UAV to spray the object or surface from different directions. Although our proposed system suffers from some limitations, it helps lay the groundwork to develop future fleets of UAVs that reduce the spread of pathogens via fomites, and for a broader adoption of autonomous systems to tackle key societal challenges related to public health. 3 ways china is using drones to fight coronavirus EagleHawk Deploys Disinfectant Drones to Sanitize Facilities India uses drones to disinfect virus hotspot as cases surge Spain's military uses DJI agricultural drones to spray disinfectant in fight against Covid-19 First use of drones to fight coronavirus in France Coronavirus: Indonesia uses drones to disinfect dense areas COVID-19: Nigerian Startup Beat Drone deploys drones to disinfect communities Fomite Transmission, Physicochemical Origin of Virus-Surface Interactions, and Disinfection Strategies for Enveloped Viruses with Applications to SARS-CoV-2 Center For Disease Control and Prevention. Science Brief: SARS-CoV-2 and Surface (Fomite) Transmission for Indoor Community Environments Risk of fomite-mediated transmission of SARS-CoV-2 in child daycares, schools, and offices: a modeling study Routes of transmission of influenza A H1N1, SARS CoV, and norovirus in air cabin: Comparative analyses Probable transmission chains of Middle East respiratory syndrome coronavirus and the multiple generations of secondary infection in South Korea Isolation and identification of human coronavirus 229E from frequently touched environmental surfaces of a university classroom that is cleaned daily Norovirus Disease in the United States A Point-Source Norovirus Outbreak Caused by Exposure to Fomites Centers for Disease Control and Prevention Fomite-mediated transmission as a sufficient pathway: a comparative analysis across three viral pathogens Viability of human adenovirus from hospital fomites Centers for Disease Control and Prevention Design and Development of Spray Disinfection System to Combat Coronavirus (Covid-19) Using IoT Based Robotics Technology All-terrain mobile robot disinfectant sprayer to decrease the spread of COVID-19 in open area Disinfectant Spraying System with Quadcopter Type Unmanned Aerial Vehicle Technology as an Effort to Break the Chain of the COVID-19 Virus Sanitization using Hexacopter Autonomous Drone The effectiveness of disinfectant spraying based on drone technology Drone Application for Spraying Disinfection Liquid Fighting against Covid-19 Pandemic -Theoretic Approach Operational Study of Drone Spraying Application for the Disinfection of Surfaces against the COVID-19 Pandemic PaintCopter: An Autonomous UAV for Spray Painting on Three-Dimensional Surfaces Assessing the potential of unmanned aerial vehicle spraying of aqueous ozone as an outdoor disinfectant for SARS-CoV-2 In: Infection prevention and control of epidemic-and pandemic-prone acute respiratory infections in health care. World Health Organization Comparative efficacy of ethanol and isopropanol against feline calicivirus, a norovirus surrogate Inactivation of influenza A virus H1N1 by disinfection process Ethanol and isopropanol inactivation of human coronavirus on hard surfaces Chemical disinfection of nonporous inanimate surfaces experimentally contaminated with four human pathogenic viruses A disc-based quantitative carrier test method to assess the virucidal activity of chemical germicides Robust and Adaptive Door Operation with a Mobile Robot You Only Look Once: Unified, Real-Time Object Detection 3D is here: Point Cloud Library (PCL) The Open Motion Planning Library Sampling-based Algorithms for Optimal Motion Planning We thank the Stanford UAV club for their thoughtful advice on the propulsion system. This work was supported by a National Science Foundation grant CBET-2030390.