key: cord-0115975-wuiyh15o
authors: Ramesh, Vignav; Rister, Blaine; Rubin, Daniel L.
title: COVID-19 Lung Lesion Segmentation Using a Sparsely Supervised Mask R-CNN on Chest X-rays Automatically Computed from Volumetric CTs
date: 2021-05-17
journal: nan
DOI: nan
sha: 9c58c85ec232094619476defe0ffa5d4f66ca40b
doc_id: 115975
cord_uid: wuiyh15o

Chest X-rays of coronavirus disease 2019 (COVID-19) patients are frequently obtained to determine the extent of lung disease and are a valuable source of data for creating artificial intelligence models. Most work to date assessing disease severity on chest imaging has focused on segmenting computed tomography (CT) images; however, given that CTs are performed much less frequently than chest X-rays for COVID-19 patients, automated lung lesion segmentation on chest X-rays could be clinically valuable. There currently exists a universal shortage of chest X-rays with ground truth COVID-19 lung lesion annotations, and manually contouring lung opacities is a tedious, labor-intensive task. To accelerate severity detection and augment the amount of publicly available chest X-ray training data for supervised deep learning (DL) models, we leverage existing annotated CT images to generate frontal projection"chest X-ray"images for training COVID-19 chest X-ray models. In this paper, we propose an automated pipeline for segmentation of COVID-19 lung lesions on chest X-rays comprised of a Mask R-CNN trained on a mixed dataset of open-source chest X-rays and coronal X-ray projections computed from annotated volumetric CTs. On a test set containing 40 chest X-rays of COVID-19 positive patients, our model achieved IoU scores of 0.81 $pm$ 0.03 and 0.79 $pm$ 0.03 when trained on a dataset of 60 chest X-rays and on a mixed dataset of 10 chest X-rays and 50 projections from CTs, respectively. Our model far outperforms current baselines with limited supervised training and may assist in automated COVID-19 severity quantification on chest X-rays.

Coronavirus disease 2019 (COVID-19), a febrile respiratory illness caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), was initially reported to the World Health Organization (WHO) in December 2019. 1 As of February 14, 2021, the WHO reported 108,153,741 worldwide cases and 2,381,295 confirmed deaths. 2,3 Chest X-rays are frequently obtained to determine the extent of lung disease in potential COVID-19 patients and are a valuable source of data for creating artificial intelligence models for both COVID-19 prediction and severity quantification.

Findings from the COVID-19 Immune Response Study suggest that early intervention is critical for COVID-19 treatment; patients with moderate illness have not yet developed end-organ damage, meaning that treatment is more effective. 4 Thus, clinicians currently utilize a variety of methods for early COVID-19 detection. Reverse transcription polymerase chain reaction (RT-PCR) is a standard diagnostic method that involves the extraction of nucleic acid from samples obtained by oropharyngeal swab, nasopharyngeal swab, bronchoalveolar lavage, or tracheal aspirate. 5 However, RT-PCR does not provide insight into the severity of COVID-19 lung infection; severity quantification is a task that image analysis is best suited to perform. 6, 7 Computed tomography (CT) imaging has been used to identify areas of consolidation or ground glass opacities in the lungs 8 and thus has value in visualizing the extent of COVID-19 infection (Yan et al., 2020; Fan et al., 2020; Oulefki et al., 2020; Akbari et al., 2020) . Most work to date assessing disease severity on chest imaging has focused on segmenting CT images; however, given that CT scans are performed much less frequently than chest X-rays for COVID-19 patients, automated lung lesion segmentation and severity quantification on chest X-rays could be clinically valuable. There currently exists a universal shortage of chest X-rays with ground truth COVID-19 lung lesion annotations, and manually contouring lung opacities on chest X-rays is a tedious and labor-intensive task. However, CT imaging is a modality that has been commonly obtained in countries other than the United States (namely China, Russia, and India), and thus, CTs with ground truth lung lesion annotations are publicly available. 9 To accelerate diagnosis and severity detection, increase access to treatment for a wider demographic of COVID-19 patients, and augment the amount of open-source chest X-ray training data for supervised deep learning (DL) models, an automated method of segmenting lesions on chest X-rays of COVID-19 patients that utilizes publicly available CT data is critically needed.

Few other published methods of segmenting COVID-19 lung lesions on chest X-rays exist. Tang, Sun, and Li proposed a U-Net with a ResNet-18 backbone for segmentation of opacity regions on chest X-rays. 11 While the model detected lung lesions fairly accurately, high-density anatomical structures such as bronchial trees were often perceived as additional opacity regions and thereby compromised model predictions, indicating that the network leaves room for improvement. Oh, Park, and Ye proposed a patch-based convolutional neural network approach with a relatively small number of trainable parameters for COVID-19 diagnosis inspired by statistical analysis of potential imaging biomarkers of chest X-ray radiographs. 12 They adopted an extended fully convolutional DenseNet103 with a ResNet-18 backbone for the segmentation task; however, this model only segments the lungs and heart on chest X-rays and does not provide any information on opacity regions beside saliency maps, which do not contain accurate lesion outlines and thereby are often difficult to interpret for clinical purposes.

We develop an automated pipeline for COVID-19 lung lesion segmentation on chest X-rays. Due to the lack of publicly available annotated chest X-ray data, we implement a pixel-based algorithm (a method operating at the pixel level) that generates coronal X-ray projections from annotated volumetric CTs to augment the training dataset. A Mask R-CNN framework is then trained on this mixed dataset. Our model achieves superior accuracy with only limited supervised training.

The main contributions of the paper can be summarized as follows:

1. We present the first publicly available, open-source chest X-ray dataset containing over 100 images, assembled from various public sources (see Section 2.1), with COVID-19 lung lesion annotations produced by our Mask R-CNN model.

We implement a pixel-based algorithm to compute a coronal X-ray projection (with overlaid segmentations) from an annotated CT volume. We also obtained 100 chest X-rays with ground truth lung lesion annotations from the following source:

• General Blockchain Inc. 23−25 : Compiled from a variety of public sources as well as through indirect collection from hospitals and physicians 2.2 CT to X-ray Conversion 

where Φ is an X × Y × Z 3D array denoting the volumetric CT and Θ is an X × Z matrix denoting the computed X-ray.

Besides being used to compute coronal X-ray projections, the CT to X-ray conversion algorithm is also used to generate the labels for each sample in the training dataset. Given a volume of ground truth mask slices in grayscale format, where a nonzero pixel value is considered part of an opacity region and vice versa, the CT to X-ray conversion algorithm is used to generate a coronal mask projection from the axial mask volume. A recursive floodfill 27 is then performed on the mask projection to generate sets of pixel values, where each set stores the coordinates of the pixel values within a single disjoint opacity region on the projection. Each set is then used to generate a concave hull 28, 29 , a data structure that computes a boundary polygon from a list of points by composing the edges of a series of triangles constructed from randomly chosen trios of points. After the coronal mask projection is constructed, each of its opacity regions has a corresponding concave hull that stores the boundary points of its representative polygon. These concave hulls are passed to the Mask R-CNN as labels for the X-ray projections in the training dataset.

We employ a naive implementation of the Mask R-CNN framework for the task of instance segmentation. 30 In a Mask R-CNN architecture, training samples are fed into a ResNet-101 backbone network (see Figure 2) , convolved, and passed to the Region Proposal Network (RPN) to generate a set of proposed regions possibly containing lung lesions. Anchors corresponding with each region of interest are then passed through a series of feature maps to generate masks outlining COVID-19 lung lesions on the input chest X-ray. Object classes and bounding boxes are computed via a series of fully connected layers. The task of COVID-19 lung lesion segmentation is posed as a problem of binary classification between the image background and lung lesions. The final output is a predicted mask corresponding with the input chest X-ray, which can then be overlaid on the input image for clinical use. Figure 3 provides a diagram of this network architecture. 

We implemented the Mask R-CNN architecture using TensorFlow and Keras. We trained for 30 Due to the paucity of publicly available chest X-ray data with COVID-19 lung lesion annotations, we also used data augmentation to improve model accuracy and reduce potential overfitting. The following augmentation techniques were applied to the training samples with the specified probabilities: 

We trained identical Mask R-CNN architectures (see Section 2.3) on two different training datasets of the same size for exactly 30 epochs with the same test dataset, allowing us to compare results across training datasets.

Dataset 1 (X-rays Only): The first training dataset consisted of 60 chest X-rays with ground truth annotations obtained from General Blockchain Inc.'s public dataset via random selection.

The second training dataset consisted of 10 chest X-rays randomly selected from the 60 chest X-rays in Dataset 1 as well as 50 randomly selected X-ray projections from CT volumes obtained from Kaggle, MosMedData, and the COVID-19-20 Lung CT Lesion Segmentation Grand Challenge (see Section 2.1).

Test Dataset: Our model was evaluated on the remaining 40 chest X-rays of COVID-19 positive patients from General Blockchain Inc's dataset.

To evaluate our model's performance on the test set (see Section 2.4), we use the Intersection over Union (IOU) metric. IOU is a similarity metric between the ground truth and the prediction segmentations. 34

Baseline * * Metric Dataset 1 (X-rays Only) * Dataset 2 (Mixed) * Dataset 1 (X-rays Only) *

0.8056 ± 0.0266 0.7937 ± 0.0291 0.3824 ± 0.0349 0.4870 ± 0.0322 * Margins of error obtained via a 1-sample t-test for population mean (µ) with 95% confidence * * Tang, Sun, and Li's U-Net segmentation model

When trained on Dataset 1 (X-rays only), the proposed model achieved an IOU score of 0.81 ± 0.03. When trained on Dataset 2 (mixed), it achieved an IOU score of 0.79 ± 0.03. The similarity between these results indicates that we can replace more than 83% of chest X-ray training images with X-ray projections generated from CTs while maintaining model accuracy. The following figures contain representative results (see Figure 4 for ground truth and predicted masks; see Figure 5 for predicted segmentations overlaid on chest X-rays from the test set). 

The above results far exceed the few existing published baselines. For instance, Tang, Sun, and Li's U-Net segmentation model described in Section 1 (the only published COVID-19 lung lesion segmentation framework with publicly available model schematics), achieved IOU scores of 0.38 ± 0.03 and 0.49 ± 0.03, respectively, both of which are significantly lower than our model's corresponding IOU scores of 0.81 ± 0.03 and 0.79 ± 0.03. Since we trained and tested our model and the baseline model on the same datasets, our Mask R-CNN likely outperformed Tang, Sun, and Li's U-Net segmentation architecture due to its structure as a series of recurring feature maps rather than contracting and expansive paths, the presence of the RPN, and its greater complexity in the form of a ResNet-101 backbone rather than a ResNet-18 backbone. (Note that in the domain of biomedical image segmentation, it is widely accepted that Mask R-CNN models are more robust than U-Net models. 35, 36 ) Furthermore, when the predicted masks are on the input images, our results can be utilized in clinical contexts to quantify the amount of lung disease on chest X-rays of COVID-19 patients and can thus assist in prognosis detection and treatment determination.

We propose a fully automated pipeline to segment COVID-19 lung opacities on chest X-rays. The majority of current work regarding COVID-19 opacity segmentation has been in CT imaging; however, since chest X-rays are much more commonly obtained than CTs, COVID-19 lung disease segmentation on chest X-ray images could enable the development of artificial intelligence applications that use these images to assess disease severity and evaluate or predict progression. Our model first utilizes a pixel-based summation algorithm to compute a coronal X-ray projection from axial CT and mask volumes. A floodfill-and hull-based approach is then employed to generate labels for each chest X-ray projection, which are used to train a Mask R-CNN for the task of instance segmentation. Our model achieved an IOU score of 0.81 ± 0.03 when trained on Dataset 1, and an IOU score of 0.79 ± 0.03 when trained on Dataset 2. A small number of chest X-rays compared to the number of frontal projections was chosen for Dataset 2 in order to demonstrate the potential for X-ray projections from CT volumes to replace actual chest X-rays as training samples for deep learning models.

A limitation of our study is that we used small amounts of publicly available data; however, our results still suggest that improved accuracy can be obtained by augmenting chest X-ray data with large numbers of frontal projections of public CT volumes. Training and testing our model on larger datasets could improve future results.

COVID-19 Chest CT Image Segmentation -A Deep Convolutional Neural Network Solution

WHO Coronavirus Disease (COVID-19) Dashboard. Available at

COVID-19) -Symptoms and Causes

Can Early Intervention Slow the Progression of COVID-19?

Performance of Radiologists in Differentiating COVID-19 from Non-COVID-19 Viral Pneumonia at Chest CT

Correlation of Chest CT and RT-PCR Testing for Coronavirus Disease 2019 (COVID-19) in China: A Report of 1014 Cases

Sensitivity of Chest CT for COVID-19: Comparison to RT-PCR

Clinical Characteristics of 138 Hospitalized Patients With 2019 Novel Coronavirus-Infected Pneumonia in

Review on Diagnosis of COVID-19 from Chest CT Images Using Artificial Intelligence

Recurrent Instance Segmentation

Segmentation Model of the Opacity Regions in the Chest X-Rays of the Covid-19 Patients in the US Rural Areas and the Application to the Disease Severity. medRxiv

Deep Learning COVID-19 Features on CXR Using Limited Training Data Sets

Mask R-CNN

COVID-19 CT Lung and Infection Segmentation Dataset. Zenodo

MosMedData: Chest CT Scans With COVID-19 Related Findings Dataset

CT Images in COVID-19

The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository

General Blockchain Inc: Covid-19 Chest Xray Segmentations Dataset

Predicting COVID-19 Pneumonia Severity on Chest X-Ray with Deep Learning

Levoy M: Efficient Ray Tracing of Volume Data

An Efficient Flood-Filling Algorithm

Fast Concave Hull Implementation in Python

Exploring DenseNets and a Comparison with Other Deep Architectures

Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers

Feature Pyramid Networks for Object Detection

Simple Understanding of Mask RCNN

Generalized Intersection over Union

Comparing Mask R-CNN and U-Net Architectures for Robust Automatic Segmentation of Immune Cells in Immunofluorescence Images of Lupus Nephritis Biopsies. Imaging, Manipulation, and Analysis of Biomolecules, Cells, and Tissues XIX

Mask-RCNN and U-Net Ensembled for Nuclei Segmentation