key: cord-0816855-h63afhr6
authors: Shan, Fei; Gao, Yaozong; Wang, Jun; Shi, Weiya; Shi, Nannan; Han, Miaofei; Xue, Zhong; Shen, Dinggang; Shi, Yuxin
title: Abnormal Lung Quantification in Chest CT Images of COVID‐19 Patients with Deep Learning and its Application to Severity Prediction
date: 2020-11-22
journal: Med Phys
DOI: 10.1002/mp.14609
sha: 26b958224d3c20f92628cb36954af09b1b1d88d6
doc_id: 816855
cord_uid: h63afhr6

OBJECTIVE: CT provides rich diagnosis and severity information of COVID‐19 in clinical practice. However, there is no computerized tool to automatically delineate COVID‐19 infection regions in chest CT scans for quantitative assessment in advanced applications such as severity prediction. The aim of this study is to develop a deep learning (DL) based method for automatic segmentation and quantification of infection regions as well as the entire lungs from chest CT scans. METHODS: The DL‐based segmentation method employs the “VB‐Net” neural network to segment COVID‐19 infection regions in CT scans. The developed DL‐based segmentation system is trained by CT scans from 249 COVID‐19 patients, and further validated by CT scans from other 300 COVID‐19 patients. To accelerate the manual delineation of CT scans for training, a human‐involved‐model‐iterations (HIMI) strategy is also adopted to assist radiologists to refine automatic annotation of each training case. To evaluate the performance of the DL‐based segmentation system, three metrics, i.e., Dice similarity coefficient, the differences of volume, and percentage of infection (POI), are calculated between automatic and manual segmentations on the validation set. Then, a clinical study on severity prediction is reported based on the quantitative infection assessment. RESULTS: The proposed DL‐based segmentation system yielded Dice similarity coefficients of 91.6%±10.0% between automatic and manual segmentations, and a mean POI estimation error of 0.3% for the whole lung on the validation dataset. Moreover, compared with the cases with fully manual delineation that often takes hours, the proposed HIMI training strategy can dramatically reduce the delineation time to 4 minutes after 3 iterations of model updating. Besides, the best accuracy of severity prediction was 73.4%±1.3% when the mass of infection (MOI) of multiple lung lobes and bronchopulmonary segments were used as features for severity prediction, indicating the potential clinical application of our quantification technique on severity prediction. CONCLUSIONS: A DL‐based segmentation system has been developed to automatically segment and quantify infection regions in CT scans of COVID‐19 patients. Quantitative evaluation indicated high accuracy in automatic infection delineation and severity prediction.

53 [8] . To date (July 23rd, 2020), there have been 14,765,256 confirmed cases reported all around the world [9] . Each suspected 54 case needs to be confirmed by the real-time polymerase chain reaction (RT-PCR) assay of the sputum [10] . Although it is the 55 gold standard for diagnosis, confirming COVID-19 patients using RT-PCR is time-consuming in many countries and has 56 been reported to suffer from high false negative rates. On the other hand, because chest computed tomography (CT) scans 57 collected from COVID-19 patients often show typical features such as bilateral multifocal patchy consolidation or ground 58 glass opacities (GGO) in the lung [11, 12] , it has been used as an important complementary indicator in COVID-19 59 screening due to high sensitivity [13, 14] .

Due to fast progression of COVID-19, a considerable proportion of COVID-19 patients will progress to severe or even 61 critically ill stage. According to a retrospective study on 138 hospitalized COVID-19 patients at Zhongnan Hospital of 62 Wuhan University [12] , 26 .1% of the patients were transferred to the intensive care unit (ICU) after enrollment. The median 63 time from first symptom to dyspnea was only 5.0 days, and to acute respiratory distress syndrome (ARDS) was only 8.0 64 days. Similar observations were also reported by Chen et al. [15] and Huang et al. [11] . Thus, timely identification of 65 patients who may progress to the severity stage at the early stage is pivotal for subsequent active intervention [16] . Although 66 CT provides rich imaging information, it only provides qualitative evaluation in the radiological reports owing to the lack of 67 computerized tools to accurately quantify the infection regions and their longitudinal changes. Besides, contouring infection 68 regions in the chest CT is necessary for quantitative assessment; however, manual contouring of lung lesions is a tedious and 69 time-consuming work, and inconsistent delineation could also lead to subsequent assessment discrepancies. Thus, a fast Accepted Article Besides COVID-19 infection regions, the whole lung, lung lobes and bronchopulmonary segments of each subject were 139 also segmented using our system. To acquire the ground-truth for bronchopulmonary segments, radiologists annotated 140 bronchopulmonary segments on CT images, which we used as the ground-truth for bronchopulmonary segmentation.

Specifically, radiologists first annotated lung vessels and bronchus using semi-automated tools on 3D Slicer [24] . 193 In our experiments of severity prediction, the 300 validation scans were used, where 97 cases were severe cases. We 194 performed 10-folded validation and repeated the experiments 20 times.

Since both severe and non-severe cases are highly mixed in the feature space, the classical linear classifiers cannot obtain 196 satisfactory results. We thus used the support vector machine (SVM) as the classifier for severity prediction. To handle data 197 imbalance between the severe and non-severe groups, we modified the classical C-SVM and constructed a cost-sensitive 198 learning criterion to learn the hyperplane of the classifier as follows:

where and are the sample numbers of the severe and non-severe groups, respectively, and is the total 

This article is protected by copyright. Fig. 6 shows the boxplots of these POIs calculated from 300 CT scans of COVID-19 patients in Shanghai.

242 Fig. 7(a) shows that the mean POIs of left and right lower lobes are higher than those of other lobes, which coincides with the 243 findings reported in [33, 34] . [37] .

Conclusively, chest CT has played a key role not only in the diagnosis and treatment of COVID-19 but also in evaluating 307 both disease progression and therapeutic efficacy [13, [38] [39] [40] . However, the role of CT in identifying COVID-19 is still 308 controversial. Some researchers gave a critical review and questioned the role of CT in identifying COVID-19 [41] . 320 Second, the system is developed to quantify COVID-19 infections only, and it may not be applicable for quantifying other 321 pneumonia, e.g., bacterial pneumonia. Fig. 9 shows some examples on the application of our model to other lung diseases.

From the figure, one may observe that the model trained with COVID-19 CT images is able to detect similar symptoms (i.e., 323 ground glass opacities) in CT images from other lung diseases. However, if a tumor or an infection contains a large portion 324 of homogenous consolidation, the model would fail to detect the complete contour as shown in Fig. 9 (b) (right lung) and Fig.   325 9(d) (large tumor in LIDC dataset [43] ). Since most of COVID-19 infections in CT consist of ground glass opacities and 326 sometimes a small portion of inhomogeneous consolidation, there is seldom a large portion of homogenous consolidation 327 associated with COVID-19. Therefore, the model fails to recognize this pattern, which often appears in bacterial pneumonia 328 or lung cancer.

Finally, in our future work, we will extend the system to quantify severity of other pneumonia using advanced machine 330 learning methods such as transfer learning and deep ensemble learning [44] .

One may argue that the typical cases in Fig. 5 indicate that the segmentation network may miss small infection in the lung, 332 implying that the proposed segmentation network would not be helpful for studying disease progression starting from early 333 stage. In our opinion, the failure cases of the segmentation model are due to the following two reasons. First, the GGO is 334 very light and small, and the contrast is insufficient for the algorithm to draw accurate contour of infection. Second, the cases 335 with small infection are minority in our dataset. As our algorithm is learning-based and data-driven, it did not do quite well 336 on such cases. In the future, such data will be specifically collected to address this problem.

With this automatic DL-based segmentation, many studies on quantifying imaging metrics and correlating them with 338 syndromes, epidemiology, and treatment responses could further reveal insights about imaging markers and findings Accepted Article Accepted Article Table 3 . Inter-rater variability analysis between two radiologists on randomly sampled 10 CT cases. The Dice coefficients, and POI difference in whole lung, lung lobes and bronchopulmonary segments, were estimated to serve as the reference for assessing the automatic segmentation accuracy. * indicates no significant difference between contouring results of two radiologists on the validation dataset according to paired t-test. 

A novel coronavirus from patients with pneumonia in China

A novel coronavirus genome identified in a cluster of pneumonia cases -Wuhan

Importation and human-to-human transmission of a novel coronavirus in Vietnam

First case of 2019 novel coronavirus in the United States

A novel coronavirus outbreak of global health concern

The response of Milan's Emergency Medical System to the COVID-19 outbreak in Italy

A new coronavirus associated with human respiratory disease in China

Severe acute respiratory syndrome-related coronavirus-The species and its viruses, a statement of the Coronavirus Study Group

Coronavirus disease 2019 (COVID-19)Situation Report -158

Chest CT for typical 2019-nCoV pneumonia: relationship to negative RT-PCR testing

Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China

Clinical characteristics of 138 hospitalized patients with 2019 novel coronavirus-infected pneumonia in Wuhan, China

The role of imaging in the detection and management of COVID-19: a review

Rapid AI development cycle for the coronavirus (COVID-19) pandemic: Initial results for automated detection & patient monitoring using deep learning CT image analysis

Epidemiological and clinical characteristics of 99 cases of 2019 novel coronavirus pneumonia in Wuhan, China: a descriptive study

AI-Driven CT-based quantification, staging and short-term outcome prediction of COVID-19 pneumonia

The First Affiliated Hospital, Ed. Handbook of COVID-19 Prevention and Treatment

Construction of multi-region-multi-reference atlases for neonatal brain MRI segmentation

Iterative multi-atlas-based multi-image segmentation with tree-based registration

V-net: Fully convolutional neural networks for volumetric medical image segmentation

Fourth International Conference on 3D Vision (3DV)

Deep residual learning for image recognition

Large-scale evaluation of V-Net for organ segmentation in image guided radiation therapy

Automatic MR kidney segmentation for autosomal dominant polycystic kidney disease

Automatic detection of subsolid pulmonary nodules in thoracic computed tomography images

U-Net: Convolutional networks for biomedical image segmentation

Longitudinal assessment of COVID-19 using a deep learning-based quantitative CT pipeline: Illustration of two cases

Serial quantitative chest CT assessment of COVID-19: Deep-Learning Approach

Volume and mass doubling times of persistent pulmonary subsolid nodules detected in patients without known malignancy

A prediction rule to identify low-risk patients with community-acquired pneumonia

Validity of pneumonia severity index and CURB-65 severity scoring systems in community acquired pneumonia in an Indian setting

Time course of lung changes on chest CT during recovery from 2019 novel coronavirus (COVID-19) pneumonia

Emerging coronavirus 2019-nCoV pneumonia

Chest CT Findings in Coronavirus Disease-19 (COVID-19): Relationship to Duration of Infection

Automatic Lung Segmentation Based on Texture and Deep Features of HRCT Images with Interstitial Lung Disease

Lung Segmentation on HRCT and Volumetric CT for Diffuse Interstitial Lung Disease Using Deep Convolutional Neural Networks

Deep Learning-Based Quantitative Computed Tomography Model in Predicting the Severity of COVID-19: A Retrospective Study in 196 Patients

Review of artificial intelligence techniques in imaging data acquisition, segmentation and diagnosis for covid-19

Dual-sampling attention network for diagnosis of COVID-19 from community acquired pneumonia

Diagnosis of coronavirus disease 2019 (covid-19) with structured latent multi-view representation learning

Chest CT and Coronavirus disease (COVID-19): A critical review of the literature to date

ACR recommendations for the use of chest radiography and computed tomography (CT) for suspected COVID-19 infection

LUNA16 -Grand Challenge

Deep ensemble learning of sparse regression models for brain disease diagnosis