key: cord-0977884-akdjoeqk authors: Wu, Xiangjun; Hui, Hui; Niu, Meng; Li, Liang; Wang, Li; He, Bingxi; Yang, Xin; Li, Li; Li, Hongjun; Tian, Jie; Zha, Yunfei title: Deep learning-based multi-view fusion model for screening 2019 novel coronavirus pneumonia: a multicentre study date: 2020-05-05 journal: Eur J Radiol DOI: 10.1016/j.ejrad.2020.109041 sha: e8b9eaa883485ed777d2575608c44e92b0c106cc doc_id: 977884 cord_uid: akdjoeqk Abstract Purpose To develop a deep learning-based method to assist radiologists to fast and accurately identify patients with COVID-19 by CT images. Methods We retrospectively collected chest CT images of 495 patients from three hospitals in China. 495 datasets were randomly divided into 395 cases (80%, 294 of COVID-19, 101 of other pneumonia) of the training set, 50 cases (10%, 37 of COVID-19, 13 of other pneumonia) of the validation set and 50 cases (10%, 37 of COVID-19, 13 of other pneumonia) of the testing set. We trained a multi-view fusion model using deep learning network to screen patients with COVID-19 using CT images with the maximum lung regions in axial, coronal and sagittal views. The performance of the proposed model was evaluated by both the validation and testing sets. Results The multi-view deep learning fusion model achieved the area under the receiver-operating characteristics curve (AUC) of 0.732, accuracy of 0.700, sensitivity of 0.730 and specificity of 0.615 in validation set. In the testing set, we can achieve AUC, accuracy, sensitivity and specificity of 0.819, 0.760, 0.811 and 0.615 respectively. Conclusions Based on deep learning method, the proposed diagnosis model trained on multi-view images of chest CT images showed great potential to improve the efficacy of diagnosis and mitigate the heavy workload of radiologists for the initial screening of COVID-19 pneumonia. In December, 2019, a pneumonia associated with coronavirus disease 2019 pneumonia emerged in Wuhan, Hubei Province, China. COVID-19 can be transmitted through airborne, surface of objects, and especially respiratory droplets and close contacting with the infected ones [1] . Until March 15 th , 2020, COVID-19 has caused 81,048 infection cases and 3,204 deaths in China. It has caused 75,333 infection cases and 2,624 deaths outside China [2] . Currently, the real-time reverse-transcriptase-polymerase-chain-reaction (RT-PCR) test is widely used as the gold standard for confirming COVID-19 pneumonia. However, it has the nature of time consuming and high false negative [3] . Thus, it is urgent to find an effective and accurate method to screen COVID- 19 . Computed tomography (CT) imaging is a critical tool in the initial screening of COVID-19 pneumonia. Shi et al. studied the CT image data of 81 COVID-19 pneumonia patients and found that in problem, deep learning can achieve human-level accuracy, such as in pulmonary nodules diagnosis [6] , breast cancer diagnosis [7] , retina image segmentation [8] , etc. In clinical applications of lung diseases, Coudray et al. studied the use of pathological images of lung cancer tissues to predict lung cancer subtypes and gene mutations [9] . They trained the deep learning network inceptionV3 to establish prediction models, and achieved good performance in independent validation sets. Wang et al. used CT image data from lung regions to predict EGFR gene mutations using an end-to-end deep learning network, and achieved the area under the receiver-operating characteristics curve (AUC) of 0.81 in the validation set of 241 cases [10] . Hence, the deep learning-based methods can be used to assist radiologists to improve the diagnostic efficiency and accuracy in the diagnosis of COVID-19 pneumonia. In this study, we aimed to train a diagnostic model based on deep learning network on the chest CT images to screen the patients with COVID-19 pneumonia. The dataset and demographic characteristics of patients are detailed in Table 1. The chest examinations were performed with a CT scanner (Optima CT680, GE Medical Systems, USA;) in RHWU. The CT protocol was as follows: tube voltage, 120 kVp; tube current, 249mA; and mAs, 108.7. The images were acquired at lung (window width, 1500 HU; window level, -750 HU) settings. For patients in the 1 st HCMU, the examinations were performed with a CT scanner (Discovery, CT750 HD, GE Medical Systems, USA). The CT parameter was set as: tube voltage, 120 kVp; tube current, 219mA; and mAs, 95.6. The images were acquired at mediastinal (window width, 250 HU; window level, 40 HU) settings. LightSpeed VCT (GE Medical Systems, USA) scanner was used for BYH with the CT protocol: tube voltage, 120 kVp; tube current, 392 mA; and mAs, 238.9. The images were acquired at lung (window width, 1500 HU; window level, -750 HU) settings. All the images were standardized by mapping the pixel values in a range from 0-255 and reconstructed with a slice thickness of 0.625 mm or 5 mm. For each set of chest CT images, the lung region in each axial CT slices was firstly segmented using threshold segmentation and morphological optimization algorithms. The lung regions in coronal and sagittal CT slices were segmented accordingly. Subsequently, for each patient of chest CT images, the slice with the most pixels in the segmented lung area from each of the axial, coronal and sagittal views was selected as the inputs of the deep learning network. Finally, the three-view CT images of 495 patients were randomly divided into 395 cases (80%, 294 of COVID-19, 101 of other pneumonia) of the training set, 50 cases (10%, 37 of COVID-19, 13 of other pneumonia) of the validation set and 50 cases (10%, 37 of COVID-19, 13 of other pneumonia) of the testing set. Each image (512×512 pixels) was resized into 256×256 pixels for deep learning network training, validation and testing. We trained out deep learning network using the three-view images of maximum lung region. In previous deep learning-based studies on tumour diagnosis, it is believed that the largest slice of the tumour on CT images contains most of the information about the tumour [11] . Inspired by this, in this study, we selected the layer with the largest lung region as region of interest. As previous study of chest CT images has shown that images from multiple views can contribute to learning more useful information without introducing more redundant information [6] . To predict the classification of lung nodules by CT images, a multi-view deep learning model of lung nodules classification study showed good performance. Thus, we trained our deep learning network using the CT images from axial, coronal, and sagittal views of the lung regions for developing the diagnosis model to screen the COVID-19. To discriminate the patients with COVID-19 pneumonia and other pneumonia, we developed a multiview fusion model based on deep learning network as shown in Fig. 1 . Our model was constructed base on the modification of ResNet50 architecture [12] . We took the set of three-view images as the network input instead of the RGB three-channel image in typical ResNet50. We first trained our deep learning network using randomly selected 395 training set of three-view images. During the training phase, the validation set of three-view images were used to optimize the hyperparameters to achieve the best performance. The outputs of Res blocks were concatenated for each patient to facilitate input of Dense layer. The parameters of the deep network were updated through the training stage using an rmsprop optimizer with a learning rate of 1 × 10 −5 and batchsize of 4. The multi-view fusion deep learning network was implemented using Python (version, 3.6) language and based on the Keras (version, 2.1.6) framework. Deep learning network training and testing were performed using TITAN XP with server model. The performance indicators of the deep learning network are calculated in the R language (version, 3.5.1). We evaluated the classification performance using AUC, accuracy, sensitivity, and specificity. AUC is an index to measure the performance of the classifier. The AUC is calculated according to the area under the ROC curve. Accuracy is calculated by (true positive samples + true negative samples) / (all samples). Sensitivity is calculated by (true positive samples) / (true positive samples + false negative samples). Specificity is calculated by (true negative samples) / (false positive samples + true negative samples). To evaluate the performance of multi-view fusion model, we trained a single-view, model using the axial images with the maximum lung region as input for comparison. Delong test is used to test the differences between the AUC of single-view model and multi-view model. Mann-Whitney U test and Chi-square test are used to calculate the distribution differences in different sets of RHWU and 1 st HCMU. P values <0.05 is considered indicative of statistical significance. To illustrate the performance of the proposed model, we firstly evaluated the single-view model, in which only the axial-view CT images were used for training. We achieved AUC of 0.767, 0.642, and 0.634 in the training, validation, and testing set respectively. The accuracy was 0.640 and 0.620, sensitivity was 0.676 and 0.622 for the validation and testing sets. For the multi-view model, the AUC, accuracy, and J o u r n a l P r e -p r o o f sensitivity in the training set were 0.905 (p<0.001), 0.833 and 0.823 respectively. The performance was comparable with the AUC of 0.732 (P=0.423) and 0.819 (P=0.08), accuracy of 0.700 and 0.760, and sensitivity of 0.730 and 0.811 for the validation and testing set. The receiver-operating characteristics (ROC) curve and the confusion matrix of the single-view and multi-view models in the training, validation, and testing set are shown in Fig. 2 . The performance comparison of the single-view and multiview models is summarised in Table 2 . The results showed that we can achieve better performance in COVID-19 classification with multi-view model. To further verify the performance of the model, we conducted subgroup analysis for the two hospitals (RHWU and 1 st HCMU) that both have COVID-19 and other pneumonia data in terms of gender and age. Given the small datasets of subgroup for the validation and testing sets, we grouped them as one set for each hospital in subgroup analysis. Mann-Whitney U test and Chi-square test are used to calculate the distribution differences of age and gender groups in different sets for the two hospitals. The ROC curves of the gender and age groups are shown in Fig. 3 and Fig. 4 , respectively. In age group analysis, the distribution between the two hospitals is significantly different on the training set (P<0.001), but no significant different distribution is found on the validation and testing sets (P = 0.097). The performance is better in the group of age≥60 (AUC of 0.983) than in the group of age＜60 (AUC of 0.680) on the validation and testing sets in for RHWU. Whereas the performance is opposite Table 3 . It is noted that the statistical differences in both age and gender groups on the training set are partly due to the small dataset in different subgroups of the two hospitals. To visualize the diagnosis results, we presented the CT images of four patients in axial, coronal and sagittal views, including a 46-year-old male with COVID-19 pneumonia (Fig. 5a) , an 84-year-old female with bacterial pneumonia in validation set (Fig. 5b) , a 62-year-old male with COVID-19 pneumonia (Fig. 5c ) and a 52-year-old female with bacterial pneumonia in testing set (Fig. 5d ). As previous reports on CT features for COVID-19 pneumonia [4, 13, 14] , the characteristics of GGO were clearly seen in two classified COVID-19 patients (Fig. 5a , c, marked with red arrows). For these four patients, their risk scores with COVID-19 infections are 0.801, 0.461, 0.946, and 0.315 (range from 0-1), which were evaluated by the multi-view deep learning fusion model. In this study, we took the cut-off of the model as 0.653, according to Youden index. Normally, it takes more than ten minutes to examine the CT images of on patient and to identify the COVID-19 infected patient by a trained radiologist. In contrast, less than five seconds are taken to achieve the comparable screen result in COVID-19 pneumonia identification using multi-view deep learning model, which will mitigate over 100-fold workload of radiologist in the initial screening of patients with COVID-19 pneumonia. As the COVID-19 has rapidly spread global-wide [2] , it is very challenging for the radiologists to fast and accurately diagnose the pneumonias under tremendous workload. Artificial intelligence tools, more specifically, deep learning networks are the promising tools to assist radiologists in the initial screening J o u r n a l P r e -p r o o f of COVID-19. Eventually, it could mitigate the workload of radiologists, improve diagnosis efficiency and provide more timely and appropriate treatments for COVID-19 patients. In this study, we developed a multi-view deep learning fusion model to discriminate the patient with COVID-19 through CT images. Compared with the single-view model, the multi-view model used more information of CT images and showed better performance. For the single-view model, it is easy to be overfitting because there is less information to use. Moreover, if the training is conducted on the data of only one hospital, it is easy to fall into the local optimal solution, which is not necessarily effective to the other hospitals, so the performance of the model may fluctuate greatly among different hospitals. To date, some efforts on COVID-19 diagnostic through AI approaches have been reported [15] [16] [17] [18] . Chen et al. [16] developed and validated deep learning diagnostic model of COVID-19 pneumonia using single view CT slices of 106 cases patients from single center. They achieved the accuracy, sensitivity and specificity of 0.989, 0.943 and 0.992, respectively, in the testing set. In another two centers study, Wang et al. [17] used 99 patients CT images data to identify COVID-19 pneumonia by transfer learning method. The AUC, accuracy, sensitivity and specificity of their model in the testing set were 0.780, 0.731, 0.670 and 0.760, respectively. It is important to note that most of patients with COVID-19 are more likely to have CT abnormalities characterized by bilateral and subpleural ground glass opacities (GGO), vascular thickening, spider web, and crazy-paving patterns [19] [20] [21] , while patients with other pneumonia mainly show patchy or density increasing shadows [22] . The advanced AI-based algorithms can discriminate COVID-19 from other pneumonia by learning not only these typical CT image signs, but also some highdimensional features, such as texture, wavelet information etc., which are unable processed by radiologists. Consequently, these deep learning-based studies showed significant effect on the initial screening of COVID-19 pneumonia through CT images. However, due to the small datasets and less than two centers were involved in these studies, the CT slices were taken as samples for developing deep learning model, resulting in inter-slice fusion and time cost issues. In addition, the generalization performance on multicentre data remains to be verified. Here, we utilized the information from the automatic segmentation of multi-view of CT images to train our deep learning network, which avoided the heavy workload of manually sketching the features in CT images of patients with pneumonia by the radiologists. Therefore, we could take one patient as a sample, which means that one patient only needs to be tested once. In this study, we established and verified the performance of the multi-view deep learning fusion model on 495 multi-center data. Our results further demonstrated that our model has the great potential to be used for the diagnosis of COVID-19 pneumonia. In future studies, a large, multicentre, and prospective dataset collection is necessary for training and validating AI models. More advanced AI algorithms also should be developed to achieve more efficient and accurate COVID-19 diagnosis. Due to the capability of disease quantification, AI-based method has great potential to predict the severity of COVID-19. Moreover, the deep learning-based model could be used for treatment decision-making in the patient with COVID-19. As the medical resources are seriously insufficient due to the outbreak of COVID-19 worldwide, fast diagnosis and accurate prediction of This study has some limitations. First, we only took the slices of the maximum lung region in three views to build the training, validation, and testing set. More CT slices in different views could be added as input to improve the performance of the proposed model. Second, more CT images of COVID-19 patients from other hospitals throughout China, or even worldwide could be involved to validate the model. Finally, more detailed clinical information was unavailable for subgroup analysis, such as body mass index and the severity of COVID-19 infection. Nonetheless, the gender and age subgroup analyses in this study allowed a preliminary evaluation of the model. In summary, we developed a multi-view fusion model to initial screening of COVID-19 pneumonia. This model achieved a better performance with single-view model and subgroup analysis. It showed great potential to improve the efficiency of diagnosis and mitigate the workload of radiologists. We envision that the multi-view deep learning-based tools can be used to assistant radiologist to fast and accurately identify COVID19 pneumonia. and sagittal views that selected from the maximum lung region selection. The three branch networks output feature maps that aggregated to feed into a fully connected dense layer. Finally, the layer outputs Categorical data of age were presented as counts (range). Note: Delong test is used to test the differences between the AUC of single-view model and multi-view model. *Quantitative data were presented as value (95% confidence interval). A familial cluster of pneumonia associated with the 2019 novel coronavirus indicating person-toperson transmission: a study of a family cluster WHO, Coronavirus disease (COVID-2019) situation reports-54 Diagnosis of the Coronavirus disease (COVID-19): rRT-PCR or CT? Radiological findings from 81 patients with COVID-19 pneumonia in Wuhan, China: a descriptive study Correlation of Chest CT and RT-PCR Testing in Coronavirus Disease 2019 (COVID-19) in China: A Report of 1014 Cases Knowledge-based Collaborative Deep Learning for Benign-Malignant Lung Nodule Classification on Chest CT Association of Peritumoral Radiomics With Tumor Biology and Pathologic Response to Preoperative Targeted Therapy for HER2 (ERBB2)-Positive Breast Cancer Endto-End Adversarial Retinal Image Synthesis Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning Predicting EGFR mutation status in lung adenocarcinoma on computed tomography image using deep learning 2D and 3D CT Radiomics Features Prognostic Performance Comparison in Non-Small Cell Lung Cancer Deep Residual Learning for Image Recognition Conference on Computer Vision and Pattern Recognition (Cvpr) Imaging and clinical features of patients with 2019 novel coronavirus SARS-CoV-2 CT radiomics can help screen the coronavirus disease 2019 (COVID-19): a preliminary study Deep learning-based model for detecting 2019 novel coronavirus pneumonia on high-resolution computed tomography: a prospective study, medRxiv A deep learning algorithm using CT images to screen for Corona Virus Disease Performance of radiologists in differentiating COVID-19 from viral pneumonia on chest CT Chest CT Findings in 2019 Novel Coronavirus (2019-nCoV) Infections from Wuhan, China: Key Points for the Radiologist Performance of radiologists in differentiating COVID-19 from viral pneumonia on chest CT A comparative study on the clinical features of COVID-19 pneumonia to other pneumonias We thank all the patients involved in this study. We acknowledge Dr. Guoqing Xie and Dr. Peng The authors declare that they have no conflict of interest. All human and animal studies have been approved by the appropriate ethics committee and have therefore been performed in accordance with the ethical standards laid down in the 1964 Declaration of Helsinki and its later amendments or comparable ethical standards. Written informed consent was waived by institutional review board for retrospective study.