key: cord-0817735-4b0fj8pn
authors: Yan, Tao; Wong, Pak Kin; Ren, Hao; Wang, Huaqiao; Wang, Jiangtao; Li, Yang
title: Automatic Distinction between COVID-19 and Common Pneumonia using Multi-Scale Convolutional Neural Network on Chest CT Scans
date: 2020-07-25
journal: Chaos Solitons Fractals
DOI: 10.1016/j.chaos.2020.110153
sha: a908b7ae23a7cb97bead8f690a7330549dc15c7f
doc_id: 817735
cord_uid: 4b0fj8pn

The COVID-19 pneumonia is a global threat since it emerged in early December 2019. Driven by the desire to develop a computer-aided system for the rapid diagnosis of COVID-19 to assist radiologists and clinicians to combat with this pandemic, we retrospectively collected 206 patients with positive reverse-transcription polymerase chain reaction (RT-PCR) for COVID-19 and their 416 chest computed tomography (CT) scans with abnormal findings from two hospitals, 412 non-COVID-19 pneumonia and their 412 chest CT scans with clear sign of pneumonia are also retrospectively selected from participating hospitals. Based on these CT scans, we design an artificial intelligence (AI) system that uses a multi-scale convolutional neural network (MSCNN) and evaluate its performance at both slice level and scan level. Experimental results show that the proposed AI has promising diagnostic performance in the detection of COVID-19 and differentiating it from other common pneumonia under limited number of training data, which has great potential to assist radiologists and physicians in performing a quick diagnosis and mitigate the heavy workload of them especially when the health system is overloaded. The data is publicly available for further research at https://data.mendeley.com/datasets/3y55vgckg6/1.

The 2019 novel coronavirus (SARS-CoV-2) is a global threat since it emerged in early December 2019 [1] . People infected with SARS-CoV-2 will experience fever, cough, myalgia, headache, and other flu-like symptoms [2] . These virus-induced diseases are collectively named COVID-19 by the World Health Organization [3] . Early detection and treatment of presumptive patients could significantly mitigate the spread of COVID-19 and reduce mortality [1, 2] . COVID-19 is typically confirmed by reverse-transcription polymerase chain reaction (RT-PCR) [4] . However, the RT-PCR test has problems such as the insufficient supply of RT-PCR kit, time-consuming and high false negatives, which may cause patients to fail to be diagnosed in time and enter standard treatment procedures [5, 6] . At present, many experts have proposed to use the chest computed tomography (CT) to diagnose suspected cases because initial chest CT may present abnormal findings indicating COVID-19 [7] . CT also has advantages of fast turnaround time, high positive rate and can provide more detailed information related to the pathology [8, 9] . Although chest CT has shown great potential for diagnosing COVID-19 pneumonia, manual identification of radiographic features including peripheral ground-glass opacities often has low specificity in distinguishing COVID-19 from other types of pneumonia such as viral pneumonia and bacterial pneumonia [10] . Besides, the rapid growth of COVID-19 patients and multiple CT scans (average 300 slices per scan) of each patient has produced a large number of CT images, which is a huge challenge for radiologists, especially in the epidemic area. A potential solution to identify COVID-19 from the massive number of CT slices quickly and accurately is to develop a computer-aided detection system using convolutional neural network (CNN), which is an emerging artificial intelligence (AI) technique.

CNN is one of deep learning algorithms and can automatically learn the most predictive representations in a manner of layer-by-layer feature combinations while it is also notorious for consuming large amounts of data [11] . The current structure of CNN has achieved much success in the detection of COVID-19 from X-Rays [12] , treatment, medication, screening and prediction for the Covid-19 [13] [14] , etc. To data, several work groups have implemented the use of CNN for automatic detection of COVID-19 on chest CT [15] [16] [17] [18] . Although these CNN-based AI systems have impressive specificity and sensitivity, challenges remain in the use of CNN to screen COVID-19. Firstly, most existing CNN-based methods are employed relatively large datasets. Although CNN can automatically learn the most predictive representations in a manner of layer-by-layer feature combinations while it is also notorious for consuming large amounts of well-annotated data [11] . Sufficient CT samples together with accurately annotated labels it is costly and hard to obtain, especially for some developing countries and small-scale hospitals. Secondly, few studies consider multi-scale features to cope with variations of the size and location of COVID-19 lesions. The infections of COVID-19 in CT images frequently distribute bilaterally, peripherally in lower zone predominant, and the infectious features can vary significantly in scale depending on the condition of the patients [5] [6] [7] [8] [9] [10] . For instance, in mild cases, the anomalies look small, while in severe cases they appear scattered and spread over a wide range.

From previous studies in the CT image analysis field, we have found that multi-scale inputs with different levels of contextual information can improve the performance in prediction and classification tasks particularly for complicated problems involving a limited number of images. For example, Wang et al. [19] adopted a new multiscale rotation-invariant CNN model for classifying various lung tissue types on CT images. The model employed a Gabor-local binary pattern that introduces a good property in image analysis-invariance to image scales and rotations. Liu et al. [20] used a multi-scale CNN for lung nodule classification and achieved an error rate of 5.41% and 13.91% for binary and ternary classifications, respectively. Yan et al. [21] proposed a lesion annotation network to extract multi-scale features, experiments show promising qualitative and quantitative results on lesion retrieval, clustering, and classification in CT images.

Thus, the purpose of this study is to develop an AI system based on a multi-scale convolutional neural network (MSCNN) for automatic differentiation of COVID-19 from other common pneumonia. Our major contributions are summarized as follows: 1) We publish a chest CT data set, which includes 416 COVID-19 positive CT scans and 412 common pneumonia (CP) CT scans. Compared with the existing open data set, our data has been double confirmed and more refined. COVID-19 patients were confirmed positive by RT-PCR. CP patients were laboratory-confirmed bacterial pneumonia, mycoplasma pneumonia, fungal pneumonia, and viral pneumonia. All the images contain lesions were re-confirmed by two experienced radiologists.

2) We proposed a novel MSCNN architecture that learns feature representations of multi-scale inputs that can achieve better performance without large-scale training data.

3) Through multi-scale spatial pyramid decomposition, data augmentation, transfer learning, and other strategies, our AI system achieves comparable diagnostic performance than experienced radiologists.

This section presents related works in terms of automated COVID-19 screening on chest CT images and involved methods of our work. Although the above studies have demonstrated promising results by using chest CT for the diagnosis of COVID-19, most existing methods are based on large well-annotated datasets.

The selection and labeling of data require considerable manpower, however, at such an outbreak situation radiologists have limited time to perform the tedious manual drawing, researchers have to wait a long time before they gather enough data to train a high-quality model. In this study, we will use a multi-scale convolutional neural network to overcome the lack of data.

One challenge in medical image domain is that regions of interest are often scale-invariant, i.e., visually similar patterns often appear in varying scales, multi-scale CNN has been successfully utilized to learn scale-invariant patterns in a variety of medical image analysis tasks, such as breast MRI malignancy classification [22] , cancer subtype classification from histopathological images [23] , macular optical coherence tomography image classification [24] , etc. Here we focus on review multi-scale CNN used in CT images [19] [20] [21] , [25] [26] [27] . Kim et al. [25] developed a multi-scale gradual integration CNN for the false-positive reduction in pulmonary nodule detection, in their experiments on the LUNA16 challenge datasets, the model achieved the highest performance. Liu et al. [26] proposed a novel multi-view multi-scale CNN for lung nodule type classification from CT images, the experimental results shown the promising classification performance even with complex ground-glass opacities and non-nodule types. Shen et al. [27] applied a hierarchical multi-scale CNN to capture nodule heterogeneity by extracting discriminative features from alternatingly stacked layers.

The Experimental results demonstrate the effectiveness of their method on classifying malignant and benign nodules without nodule segmentation.

Inspired by the above-mentioned researches, in this study we propose a novel multi-scale CNN for the diagnosis of COVID-19. To fully explore multiple features describing COVID-19 from multi-scales, we will be based on multi-scale spatial pyramid decomposition and the latest CNN algorithm to develop the MSCNN model.

This study was approved by the institutional review board of Xiangyang Central Hospital and Xiangyang No.1 People's Hospital in the Hubei province of China. Written informed consent was waived by the institutional review board for the retrospective study. We retrospectively acquired 416 three-dimensional (3D) chest CT scans with abnormal findings from 206 patients who were confirmed positive COVID-19 by RT-PCR between January 1 and May 1, 2020. The same patient underwent one or multiple CT scans at various timelines during the course of the disease (The average CT scan per patient is 2, with a range from 1 to 5). A total of 412 patients and their chest CT scans (one scan per patient) with laboratory-confirmed were retrospectively identified from the participating hospitals between January 1, 2018, and December 1, 2019. These CP patients were randomly selected and consisted of non-COVID-19 viral pneumonia (such as influenza virus), bacterial pneumonia, and fungal 11 pneumonia. The selected patients and their CT scans are randomly divided into 80% for training, 10% for validation, and 10% for testing. Furthermore, the test cases of CT scans are selected from patients who have not been included in the training stage. The demographics of the selected patients is summarized in Table 1 . No significant difference exists between COVID-19 and CP groups in terms of sex distribution (p-value>0.05). However, the average age of patients with COVID-19 is lower than that of patients with CP (p-value<0.001). Sample exclusion, inclusion, and distribution of the sample data are described in Figure 1 . Coronavirus disease 2019; CP: Common pneumonia.

Our idea of the multi-scale strategy is motivated by the clinical fact that radiographic features of COVID-19 have varied shapes, locations, and sizes. The peripheral ground-glass opacities in the early stage are very small in size and need to be analyzed at a finer scale, the pulmonary consolidation in the late stage can be visualized in a coarse-scale [5] [6] [7] [8] [9] [10] . Besides, from an information quantity perspective, it may be reasonable to use morphological and structural features in different scales and thus effectively integrating multi-scale contextual information [25, 27] .

According to the above motivations, we introduce the multi-scale spatial pyramid (MSSP) decomposition [28] to create multi-scale views of the CT image to capture key multi-scale information for better classification even though the training data is small. The MSSP creates an image pyramid by the reduced and Gaussian low pass filtered versions of the image of the previous level [28] . Although CNN algorithms benefit from spatial pooling for providing some inherent invariance to distorted, scaled, and translated inputs [29] , but in the ensembled MSCNN module, the MSSP decomposition can be used to reduce the time complexity and effective parameters of the overall model to reduce the chance of over-fitting and to obtain a promising performance in practice [24] . In this study, three levels of the l ∈{0, 1, 2}) for extracting scale specific information. The architecture of the proposed AI system is shown in Figure 3 . In this study, EfficientNetB0 is the backbone of the three CNNs. The latest EfficientNetB0 algorithm is pre-trained to achieve 93.5% (top-five) accuracy on 1,000 categories of ImageNet and its params size is just 5.3 Megabytes [30] . The main building block of EfficientNetB0 is the mobile inverted bottleneck which is tuned more efficiently by carefully balancing network depth, width, and resolution, resulting in better performance. To construct the MSCNN module, the last fully connected layer of each EfficientNetB0 is firstly dropped and a global average-pooling layer, which can capture more informative features by enforcing the correspondence between features and classes, is added.

Then, a dense layer with a dimension of 64 is added, followed by the addition of a dropout layer with a probability of 0.5 which is used to reduce overfitting. After this, the outputs of three CNNs are concatenated for better classification. Finally, a new fully-connected layer with one output node is added with a sigmoid activation function to generate continuous numbers between 0 and 1 which indicates the probability for COVID-19. To secure high sensitivity for COVID-19 diagnosis, the cut-off number is set to be 0.5. This means that if the final score is greater than 0.5, the slice is diagnosed as COVID-19.

To improve the interpretability of the AI system, the heat map calculated by output feature maps of the last convolutional layer of each 

Two rounds of binary classification experiments are carried out to evaluate the AI system. We start with an evaluation of the ability to detect slice level COVID-19 since the training is on 2D slice level and tuning of hyper-parameters is easier than 3D scan level and. In fact, we have also tried a 3D classification network, but the performance is not satisfactory due to the limited number of training CT scans and limited memory of GPU. At the slice level analysis, we will compare the diagnostic performance of different CNNs in the MSCNN module. Moreover, we also proceed with an evaluation of the detection ability at the scan level for COVID-19 vs CP patients because the radiologist diagnosis is on scan level, which is consistent with clinical practice. As one scan is COVID-19 positive when any one of its slices is also COVID-19 positive, the top 3 highest scores of all slices of a scan are averaged as the scan-level score [33] . As a result, though training and validation are done on the slice level, the AI system can take the whole 3D CT scan into account and generate a single prediction on the scan-level. Two radiologists with more than 15 years of chest CT diagnosis experience individually evaluate the CT scans in the test set and compare their average performance with the AI system.

The diagnostic performances of the AI system and radiologists are evaluated using sensitivity, specificity and accuracy together with their 95% confidence intervals. These three metrics are used to compare the performance difference between the AI system and human experts. A p-value less than 0.05 is considered statistically significant. Confusion matrixes are also plotted to make it easy for checking any confusion between two classes (i.e. mislabeling one as another). All statistical analyses are performed by using Python 3.7.6 and Sklearn 0.22.1. Figure 4 (a)~4(e) shows the confusion matrices of the three CNNs and AI system at slice level analysis. The aforesaid confusion matrices depict that there are still some cases for the AI system to misclassify CP as COVID-19. Nevertheless, experienced radiologists also have similar rates of errors. 

To further validate the effectiveness of the AUC 0.934 n/a n/a Note: The p-values are calculated by comparing the AI system with radiologists using a 2-sided McNemar test. *Per scan sensitivity is the ratio of true positive identified to all CT scans with COVID-19 lesions; **Per scan specificity is the ratio of true negative identified to all CT scans without COVID-19 lesions; ***Per patient accuracy is the ratio of all true values identified to all CT scans. AI: artificial intelligence; CI: confidence interval; AUC: area under the receiver operating characteristic curve. n/a: not applicable. 

Our pilot study on automatic distinction between COVID-19 and other common pneumonia on chest CT using MSCNN AI system demonstrates a promising result. Compared with single scale input, the MSCNN can achieve better performance.

COVID-19 pneumonia has affected the world with its rapid spread [1] . Many affected patients quickly develop acute respiratory failure with a very poor prognosis and a high mortality rate [2] . Driven by the desire to develop an AI system for the rapid diagnosis of COVID-19 to assist radiologists and clinicians to combat with this pandemic, we utilize two important strategies, MSSP and MSCNN to develop an AI system which does not require a large number of labeled CT data and can achieve satisfactory results. 84.1%, p-value = 0.804), respectively. Even though the three indexes of the AI system are slightly higher than the human experts, but all the p-values > 0.05, this means there is no significant difference in the diagnostic performance between the AI system and the experienced radiologists. Although the diagnostic accuracy of the AI system is not significantly larger than experienced radiologists, the AI system can diagnose a CT scan with a time of 0.17 minute (10 sec per scan), whereas experienced radiologists require an average of 10 minutes to read a scan. Besides, the AI system can localize the lesions or other key structures in the image when the diagnosis result is given. As shown in Figure 2 , the areas highlighted in the attention maps are the true areas that radiologists will consider the most appropriate to predict COVID-19 and CP. Therefore, the proposed AI system shows great potential to improve the diagnosis time and mitigate the heavy workload of radiologists for differentiating COVID-19 from other common pneumonia.

Nevertheless, our study still has some limitations. Firstly, although our AI system works well on the test dataset of 88 CT scans, it still needs to test on large CT dataset to prove its generalization. Another fundamental limitation arises from the black box, which is the nature of deep networks, although attention maps aid interpretation by highlighting the dominant areas, they are still not sufficient to visualize what unique features are used by the CNN algorithm to distinguish between COVID-19 and CP. Thirdly, this study just cares about the general pattern on chest CT, more detailed clinical information is excluded, further improvement and integration of multidisciplinary approaches are necessary for extending the application of the AI system. Finally, the training data only uses axial view CT slices, combining the lung regions in the axial direction with coronal and sagittal views for diagnosing more diseases together with their severities is a future work of our research.

In sum, this AI system shows a good diagnostic performance for the detection and differentiation of COVID-19 based on a small number of chest CT data. In many developing countries and small-scale hospitals, the number of chest CT scans of COVID-19 is limited, so the number of training samples for building a low-cost intelligent COVID-19 diagnosis system for their own use is always small. In this research, MSSP, MSCNN, and data augmentation are used together to alleviate the scarcity of training data to improve the diagnostic performance of the AI system. In order to defeat COVID-19 and encourage further research in this area, we have shared the dataset at https://data.mendeley.com/datasets/3y55vgckg6/1. It is believed that this system can provide valuable support for radiologists and physicians in performing a fast and accurate diagnosis in the initial screening of COVID-19 and mitigate the heavy workload of them especially when the health system is overloaded.

The authors declare no conflict of interest.

A Novel Coronavirus from Patients with Pneumonia in China

Clinical Characteristics of Coronavirus Disease 2019 in China

Coronavirus disease (COVID-19) pandemic. World Health Organization

Correlation of chest CT and RT-PCR testing in coronavirus disease 2019 (COVID-19) in China: a report of 1014 cases

Chest CT for typical 2019-nCoV pneumonia: relationship to negative RT-PCR testing

Sensitivity of chest CT for COVID-19: comparison to RT-PCR

Initial CT findings and temporal changes in patients with the novel coronavirus pneumonia (2019-nCoV): a study of 63 patients in Wuhan, China. European radiology

Performance of radiologists in differentiating COVID-19 from viral pneumonia on chest CT

Chest CT for detecting COVID-19: a systematic review and meta-analysis of diagnostic accuracy

Comparison of chest CT findings between COVID-19 pneumonia and other types of viral pneumonia: a two-center retrospective study. European radiology

Deep learning

Application of Deep Learning for Fast Detection of COVID-19 in X-Rays using nCOVnet

Applications of Machine Learning and Artificial Intelligence for Covid-19 (SARS-CoV-2) pandemic: A review

Review of artificial intelligence techniques in imaging data acquisition, segmentation and diagnosis for covid-19

Clinically Applicable AI System for Accurate Diagnosis, Quantitative Measurements, and Prognosis of COVID-19 Pneumonia Using Computed Tomography

Artificial intelligence distinguishes COVID-19 from community acquired pneumonia on chest CT

Dual-Sampling Attention Network for Diagnosis of COVID-19 from Community Acquired Pneumonia

Multisource Transfer Learning With Convolutional Neural Networks for Lung Pattern Analysis

Multiscale Rotation-Invariant Convolutional Neural Networks for Lung Texture Classification

Multiview convolutional neural networks for lung nodule classification

Holistic and comprehensive annotation of clinically significant findings on diverse CT images: learning from radiology reports and label ontology

Multi scale curriculum CNN for context-aware breast MRI malignancy classification. InInternational Conference on Medical Image Computing and Computer-Assisted Intervention

Multi-scale Domain-adversarial Multiple-instance CNN for Cancer Subtype Classification with Unannotated Histopathological Images

Macular OCT Classification Using a Multi-Scale Convolutional Neural Network Ensemble

Multi-scale gradual integration CNN for false positive reduction in pulmonary nodule detection

Multi-view multi-scale CNNs for lung nodule type classification from CT images

Multi-scale convolutional neural networks for lung nodule classification. InInternational Conference on Information Processing in Medical Imaging

The Laplacian Pyramid as a Compact Image Code

Learning Hierarchical Features for Scene Labeling

Rethinking model scaling for convolutional neural networks

The effectiveness of data augmentation in image classification using deep learning

Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization

Development and Evaluation of an AI System for COVID-19 Diagnosis. medRxiv

We thank all the patients involved in this study. We would also like to thank the doctors and