key: cord-0656473-w1e60xds authors: Gozes, Ophir; Frid-Adar, Maayan; Greenspan, Hayit; Browning, Patrick D.; Zhang, Huangqi; Ji, Wenbin; Bernheim, Adam; Siegel, Eliot title: Rapid AI Development Cycle for the Coronavirus (COVID-19) Pandemic: Initial Results for Automated Detection&Patient Monitoring using Deep Learning CT Image Analysis date: 2020-03-10 journal: nan DOI: nan sha: 00a407540a8bdd6d7425bd8a561eb21d69682511 doc_id: 656473 cord_uid: w1e60xds Purpose: Develop AI-based automated CT image analysis tools for detection, quantification, and tracking of Coronavirus; demonstrate they can differentiate coronavirus patients from non-patients. Materials and Methods: Multiple international datasets, including from Chinese disease-infected areas were included. We present a system that utilizes robust 2D and 3D deep learning models, modifying and adapting existing AI models and combining them with clinical understanding. We conducted multiple retrospective experiments to analyze the performance of the system in the detection of suspected COVID-19 thoracic CT features and to evaluate evolution of the disease in each patient over time using a 3D volume review, generating a Corona score. The study includes a testing set of 157 international patients (China and U.S). Results: Classification results for Coronavirus vs Non-coronavirus cases per thoracic CT studies were 0.996 AUC (95%CI: 0.989-1.00) ; on datasets of Chinese control and infected patients. Possible working point: 98.2% sensitivity, 92.2% specificity. For time analysis of Coronavirus patients, the system output enables quantitative measurements for smaller opacities (volume, diameter) and visualization of the larger opacities in a slice-based heat map or a 3D volume display. Our suggested Corona score measures the progression of disease over time. Conclusion: This initial study, which is currently being expanded to a larger population, demonstrated that rapidly developed AI-based image analysis can achieve high accuracy in detection of Coronavirus as well as quantification and tracking of disease burden. • For Coronavirus patients the system outputs quantitative opacity measurements and a visualization of the larger opacities in a slice-based "heat map" or a 3D volume display. A suggested "Corona score" measures the progression of patients over time. o The coronavirus infection surprised the world with its rapid spread and has had a major impact on the lives of billions of people. Non-contrast thoracic CT has been shown to be an effective tool in detection, quantification and follow-up of disease. Deep learning algorithms can be developed to assist in analyzing potentially large numbers of thoracic CT exams. • Purpose: To develop AI-based automated CT image analysis tools for detection, quantification, and tracking of Coronavirus and demonstrate that they can differentiate coronavirus patients from those who do not have the disease. • Materials and Methods: Multiple international datasets, including from Chinese disease-infected areas were included. We present a system that utilizes robust 2D and 3D deep learning models, modifying and adapting existing AI models and combining them with clinical understanding. We conducted multiple retrospective experiments to analyze the performance of the system in the detection of suspected COVID-19 thoracic CT features and to evaluate evolution of the disease in each patient over time using a 3D volume review, generating a "Corona score". The study includes a testing set of 157 international patients (China and U.S). • Results: Classification results for Coronavirus vs Non-coronavirus cases per thoracic CT studies were 0.996 AUC (95%CI: 0.989-1.00) ; on datasets of Chinese control and infected patients. Possible working point: 98.2% sensitivity, 92.2% specificity. For time analysis of Coronavirus patients, the system output enables quantitative measurements for smaller opacities (volume, diameter) and visualization of the larger opacities in a slice-based "heat map" or a 3D volume display. Our suggested "Corona score" measures the progression of disease over time. The coronavirus infection, COVID-19 has surprised the world with its rapid spread, potential virulence, with potential profound overall impact on the lives of billions of people from both a safety and an economic Once a decision has been made to use thoracic CT as these recent studies suggest for patient diagnosis or screening, a need immediately rises to rapidly evaluate potentially very large numbers of imaging studies. AI technology, in particular deep learning image analysis tools, can potentially be developed to support radiologists in the triage, quantification, trend analysis of the data. AI solutions have the potential to analyze multiple cases in parallel to detect whether chest CT reveals any abnormalities in the lung. If the software suggests a significantly increased likelihood of disease, the case can be flagged for further review by a radiologist or clinician for possible treatment/quarantine. Such systems, or variations thereof, once verified and testedcan become key contributors in the detection and control of patients with the virus. In a manner analogous to the way in which COVID-19 represents a new strain of coronavirus not previously found in humans and presumably representing a mutation of other coronaviruses, an AI algorithm can be rapidly created from one or more algorithms that perform a similar task. This is in contrast to the standard way of developing a DL algorithm, entailing several phases: I. Data-collection phase in which a large amount of data samples need to be collected from predefined categories; expert annotations are needed for ground-truthing the data; II. Training phase in which the collected data is used to train network models. Each category needs to be represented well enough so that the training can generalize to new cases that will be seen by the network in the testing phase. In this learning phase, the large number of network parameters (typically on the order of millions) are automatically generated; III. Testing phase in which an additional set of cases not used in training is presented to the network and the output of the network is tested statistically to determine its success of categorization. In the case of a new disease, such as the coronavirus, datasets are just now being identified and annotated. There are very limited data sources as well as limited expertise in labeling the data specific to this new strain of the virus in humans. Accordingly, it is not clear that there are enough examples to achieve clinically meaningful learning at this early stage of data collection despite the increasingly critical importance of this software, especially given fears of a pandemic. It is our hypothesis that AI-based tools can be rapidly developed leveraging the ability to modify and adapt existing AI models and combine them with initial clinical understanding to address the new challenges and new category of COVID-19. Our goal is to develop deep-learning based automated CT image analysis tools and demonstrate that they can enable differentiation of coronavirus patients from those who do not have the disease to provide support in the detection, measurements, and tracking of disease progression. The system we propose receives thoracic CT images and flags cases suspected with COVID-19 features. In addition, for cases classified as positive, the system outputs a lung abnormality localization map and measurements. Figure 1 shows a block diagram of the developed system. The system is comprised of several components and analyzes the CT case at two distinct levels: Subsystem A: 3D analysis of the case volume for nodules and focal opacities using existing, previously developed algorithms and Subsystem B: newly developed 2D analysis of each slice of the case to detect and localize larger-sized diffuse opacities including ground glass infiltrates which have been clinically described as representative of the coronavirus. For Subsystem A we use commercial off-the-shelf software that detects nodules and small opacities within a 3D lung volume (RADLogics Inc., Boston [5] ). This software was developed as a solution for lung Since current lung pathology detection solutions were built with a specific focus on the nodule detection task, they cannot be relied upon for detecting more diffuse global GG opacities. To address the additional disease-driven opacities, we proposed a data-driven solution on a per-slice basis, as shown in Figure 1 , Subsystem B. Working in the 2D (slice) space has several advantages for Deep-Learning based algorithms, in limited data scenarios. These include an increase in training samples (with many slices per single case), using pre-trained networks that are common in the 2D space, and easier annotation for segmentation purposes. In our solution (B), the first step is the Lung Crop stage: we extract the lung region of interest (ROI) using a lung segmentation module. The U-net architecture for image segmentation [6, 7] was trained using 6,150 CT slices of cases with lung abnormalities and their corresponding lung masks which were taken from a U.S based hospital (Table I : Dataset-6). The segmentation step enables the removal of image portions that are not relevant for the detection of within-lung disease making the learning process of the next step easier. In the following step, we focus on Detecting Coronavirus related abnormalities: We use a Resnet-50 -2D deep convolutional neural network architecture [8] ; The network is 50 layers deep and can classify images into 1000 categories. The network was pre-trained on more than a million images from the ImageNet database [9] . As commonly done in the medical imaging field, we further train the network parameters (fine-tune) to solve the problem at hand: suspected COVID-19 cases from several Chinese hospitals are used (Table I : Dataset-1). The cases were annotated per slice as normal (n=1036) vs abnormal (n=829). To overcome the limited amount of cases, we employ data augmentation techniques (image rotations, horizontal flips and cropping). In a follow-up abnormality localization step, given a new slice classified as positive, we extract "network-activation maps" which correspond to the areas most contributing to the network's decision. This is performed using the Grad-cam technique for producing visual explanations for network decisions [10] . Example results of four COVID-19 slices that we classified as abnormal by the network can be seen in Figure 2 (B). On top, the CT image is shown. In the bottom row, corresponding colored maps are provided. In red we see the strongest network output while blue is the weakest. We note the maps align well with the diffused opacities, providing a strong indication that the network managed to learn important characteristics associated with COVID-19 manifestations. To mark a case as COVID-19 positive, we calculate the ratio of positive detected slices out of the total slices of the lung (positive ratio). A positive case-decision is made if the positive ratio exceeds a pre-defined threshold. To provide a complete review of the case, we combine the output of Subsystem A -3D analysis and Subsystem B -2D slice-level. In Figure 3 we see a case of Coronavirus and the combined output findings map from the proposed system. We can see the nodular and focal diffuse opacity detections in green and the larger opacity detection in red. The two subsystems complement and, in some locations reinforce each other. In addition to the visualization, the system automatically extracts several outputs of interest, including per slice localization of opacities (2D), and a 3D volumetric presentation of the opacities throughout the lungs. We also propose a Corona score which is a volumetric measurement of the opacities burden. The corona score is computed by a volumetric summation of the network-activation maps. The score is robust to slice thickness and pixel spacing as it includes pixel volume. For patient-specific monitoring of disease progression, we suggest the Relative Corona score in which we normalize the corona score by the score computed at the first time point. A set of experiments is conducted next to demonstrate the performance of the automated analysis. We start with an evaluation of the ability to detect slice-level Coronavirus. The performance of this step is crucial for obtaining overall case wise detection. For the validation step, we used 10% of the slices from the development dataset comprised of cases from the Chinese population ( Both ROC curves are displayed in Figure 4A . In the ROC calculation, for Coronavirus patients that include multiple time points, the analysis was performed using the first time point in the series. In our final experiment, we evaluate patients that were imaged in time points for whom the first CT scan was obtained 1-4 days following the first signs of the virus (fever, cough). In the first example patient, we review a case with a single focal opacity and present volumetric measurements over time. The second case involves patient with multiple opacities and shows an overview of the patient recovery process with its corresponding Corona score over time. In this initial exploratory work, we show the capabilities of AI to assist in the efforts to accurately detect and track the progression or resolution of the Coronavirus. This is the first report to our knowledge in the literature of software specifically developed to detect, characterize and track the progression of COVID- Rapidly developed AI-based automated CT image analysis tools can achieve high accuracy in the detection A consistent and reproducible method for rapid evaluation of high volumes of screening or diagnostic thoracic CT studies using AI can assist in this crisis in several ways: Highly accurate systems can reliably exclude CTs which are negative for findings associated with the corona virus. This decreases the volume of cases passing through to the radiologist without overlooking positive cases. Progression and regression of findings could be monitored more quantitatively and consistently. This would allow a greater volume of patients being screened for Coronavirus, with earlier and more rapid detection of positive cases, which could lead to more effective identification and containment of early cases. As illustrated above, using standard machine learning techniques and innovative AI applications, in combination with an established pulmonary CT detection platform, an effective tool can be utilized for the screening and early detection of patients who may have contracted the COVID-19 pathogen. In individual patients who have contracted the virus and have the pulmonary abnormalities associated with it, the same methodologies can be used to accurately and more rapidly assess disease progression and guide therapy and patient management. [11] 50 abnormal thoracic CT scans (slice thickness, {5,7,8,9,10}mm) from China of patients that were diagnosed by a radiologist as suspicious for COVID-19 (from Jan-Feb 2020). The cases were extracted by querying a cloud PACS system for cases that were referred for laboratory testing following the scan. Cases were annotated for each slice as normal The Global Initiative on Sharing All Influenza Data (GISAID). Coronavirus COVID-19 Global Cases by Johns Hopkins CSSE Sensitivity of Chest CT for COVID-19: Comparison to RT-PCR Chest CT for typical 2019-nCoV pneumonia: relationship to negative RT-PCR testing COVID-19): Relationship to Duration of Infection U-net: Convolutional networks for biomedical image segmentation Improving the segmentation of anatomical structures in chest radiographs using u-net with an imagenet pretrained encoder Deep residual learning for image recognition Imagenet: A large-scale hierarchical image database Grad-cam: Visual explanations from deep networks via gradient-based localization The lung image database consortium (LIDC) and image database resource initiative (IDRI): a completed reference database of lung nodules on CT scans