key: cord-0693856-5mpz69t6
authors: Nikolaou, Vasilis; Massaro, Sebastiano; Fakhimi, Masoud; Stergioulas, Lampros; Garn, Wolfgang
title: COVID-19 diagnosis from chest x-rays: developing a simple, fast, and accurate neural network
date: 2021-10-12
journal: Health Inf Sci Syst
DOI: 10.1007/s13755-021-00166-4
sha: b70020d77c2cafdde9fc105cf41cd1ca3afb38ef
doc_id: 693856
cord_uid: 5mpz69t6

PURPOSE: Chest x-rays are a fast and inexpensive test that may potentially diagnose COVID-19, the disease caused by the novel coronavirus. However, chest imaging is not a first-line test for COVID-19 due to low diagnostic accuracy and confounding with other viral pneumonias. Recent research using deep learning may help overcome this issue as convolutional neural networks (CNNs) have demonstrated high accuracy of COVID-19 diagnosis at an early stage. METHODS: We used the COVID-19 Radiography database [36], which contains x-ray images of COVID-19, other viral pneumonia, and normal lungs. We developed a CNN in which we added a dense layer on top of a pre-trained baseline CNN (EfficientNetB0), and we trained, validated, and tested the model on 15,153 X-ray images. We used data augmentation to avoid overfitting and address class imbalance; we used fine-tuning to improve the model’s performance. From the external test dataset, we calculated the model’s accuracy, sensitivity, specificity, positive predictive value, negative predictive value, and F1-score. RESULTS: Our model differentiated COVID-19 from normal lungs with 95% accuracy, 90% sensitivity, and 97% specificity; it differentiated COVID-19 from other viral pneumonia and normal lungs with 93% accuracy, 94% sensitivity, and 95% specificity. CONCLUSIONS: Our parsimonious CNN shows that it is possible to differentiate COVID-19 from other viral pneumonia and normal lungs on x-ray images with high accuracy. Our method may assist clinicians with making more accurate diagnostic decisions and support chest X-rays as a valuable screening tool for the early, rapid diagnosis of COVID-19. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s13755-021-00166-4.

COVID-19 is an infectious disease caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) [1]. The virus was first observed in Wuhan, China, in December 2019 and then spread globally. On March 11, 2020 , the World Health Organization (WHO) declared COVID-19 a pandemic [1] . The virus spreads via respiratory droplets or aerosol so that it can be transmitted into individuals' mouth, nose, or eyes of individuals in close proximity. The most common symptoms include a high fever, continuous cough, and breathlessness [2] .

COVID-19 is usually diagnosed by an RT-PCR test [3] and often is complemented by chest radiographs, including x-ray images and computed tomography (CT) scans [4] . X-ray machines are widely available worldwide and provide images quickly, so chest scans have been recommended, by some researchers [5] , for screening during the pandemic. Unlike the RT-PCR test, chest scans provide information about both the status of infection (i.e., presence or absence of the disease) and disease severity. Moreover, x-ray imaging is an efficient and cost-effective procedure. It requires relatively cheap equipment and can be performed rapidly in isolated rooms with a portable chest radiograph (CXR) device, thus reducing the risk of infection inside hospitals [6, 7] . Despite these benefits, the American College of Radiology (ACR) and the Centers for Disease Control and Prevention (CDC) have not endorsed chest imaging as a first-line test for COVID-19 [8] . In the diagnosis of COVID-19, chest CT scans are highly sensitive (97%) but far less specific (25%) than RT-PCR [9] . Recent research, however, suggests that deep learning techniques may improve the specificity of x-ray imaging [10] for the diagnosis of COVID-19 (see publications in Table 1) .

Many of the studies in Table 1 use convolutional neural networks (CNNs) that take advantage of transfer learning-that is, they are built on existing CNNs (e.g., Effi-cientNet, VGG-16, AlexNet, GoogLeNet, SqueezeNet, ResNet) that were trained on large-scale image-classification datasets (e.g., ImageNet [11] ). The size of the dataset and generic nature of the trained task (e.g., describe the general shapes of an object) make the features learned by the existing CNN useful for other computer-vision problems with other images. As shown in Table 1 , most of the studies reported high accuracy (above 87%) , ten studies reported high sensitivity (above 88%) [12, 13, 16-18, 29, 30, 33-35] , seven studies reported high specificity (above 78%) [14, 15, 17, 18, 28, 29, 31, 34, 35] , three studies reported high F1-scores (above 94%) [14, 15, 30] , and three studies reported above 94% precision [12] [13] [14] . All of the models were validated internally (i.e., trained and validated on different random splits of the same dataset; the proportions of the dataset for training versus validation vary). However, it remains unclear whether these models would perform as well on an external validation task (i.e., on a different dataset that was not involved in the model's development).

To address the need for external validation and advance the possibility of using x-ray technology to alleviate the impact of the global pandemic, we develop a CNN using a parsimonious yet powerful pre-trained CNN as the baseline model, and we assess its diagnostic accuracy on an independent (external) dataset. We compare the model's performance on two classification datasets: a) COVID-19 vs. normal lungs (two-class classification) and b) COVID-19 vs. other viral pneumonia vs. normal lungs (three-class classification).

We used the COVID-19 Radiography database [36] , a public database that contains 15,153 images (as of 3rd May 2021) across three cohorts: COVID-19, other viral pneumonia, and normal lungs. We randomly split each of the three cohorts into train (70%), validation (20%), and test (10%) subsets (Fig. 1) . Only the train and validation subsets contributed to the model's development; we kept the test subset separate for external validation.

We developed a hybrid CNN using a pre-trained Con-vNet called EfficientNetB0 [37] , which is the baseline model of a family of EfficientNet models (from Efficient-NetB0 to EfficientNetB7). These models use compound scaling, in which the image's dimensions (i.e., depth, width, and resolution) are scaled by a fixed amount at the same time. Models with compound scaling usually outperform other CNNs, as shown in Fig. 2 : the baseline B0 model starts at a higher accuracy than some other models (e.g., ResNet-50, Inception-v2), while the latest Effi-cientNet model (B7) achieves the highest accuracy of all (84%). Although the EfficientNets achieve high accuracy, they require fewer parameters (5 million for B0; 66 million for B7) and less computation time than most other models. In this work, we utilize the EfficientNet-B0 as our baseline model for the following reasons: (a) it has less parameters than the rest models (B1-B7) of the Efficient-Net family, (b) it is more cost-efficient for training and tuning than the more advanced EfficientNetB1-B7 model as it does not require much computational power (see Discussion section) and (c) it contributes to high accuracy in differentiating COVID-19 from non-COVID-19 viral pneumonia and healthy images (see Results section), thus satisfies the rational of our study for developing a parsimonious yet powerful convolution network.

On top of the baseline EfficientNetB0 network, we connected a fully dense layer of 32 neurons (Fig. 3 ). The features learned by the baseline model are run through our (two-class or three-class) classifier to extract new features from the sample.

Our CNN uses about 5 million parameters (Fig. 4) , which is considerably fewer than AlexNet (61 million) and GoogLeNet (7 million) [14] . Thus, our CNN was faster to train and less likely to overfit the training data, which leads to worse performance on external datasets. We also reduced the risk of overfitting by adding 20% and 50% (which outperformed the 20% drop-out rate for the threeclass model; results not shown) drop-out rates to our twoclass and three-class prediction models, respectively. By adding drop-out during training, a proportion of features (20% and 50% for the two-and three-class model respectively) is set to zero, whereas during validation, all features are used. This makes the model at validation more robust, leading to higher testing accuracy (Figs. 5 and 6). We applied Sigmoid and Softmax operations to model the two-and three-class classification outputs, respectively. We used an Adam optimizer with a learning rate of 0.001. Before training the model, we "froze" the convolution base (i.e., EfficientNetB0) to preserve the representations learned during the baseline model's original training. Subsequently, we trained only the weights in the two dense layers that we added to the convolution base.

We augmented the training data to eliminate the class size imbalance and avoid overfitting. First, we resized the images to 150 × 150 pixels to reduce the data volume, and we normalized them to the [0, 1] interval because neural networks are more efficient with small input values. Then, we augmented the training images through random transformations to increase the variety of the images. Specifically, we manipulated the parameters by (a) rotating the image by 40 degrees, (b) randomly shifted the height and width horizontally or vertically by 20% of the image's initial size, (c) randomly clipping the image by 20%, (d) randomly zooming in by 20%, (e) randomly flipping half of the images horizontally, and (f ) filling in the pixels that were created by a rotation or a height or width shift. Data augmentation is essential for avoiding overfitting in small samples because the additional, varied images prevent the CNN from being exposed to the same image twice.

We fine-tuned our CNN by "unfreezing" a few of the top layers (i.e., from block 7a onwards) in the convolution base and jointly training both the "unfrozen" layers and the two layers that we added (Fig. 7) . By training some of the top layers of the baseline CNN, we adjusted the presentations of the pre-trained model that were more abstract (in terms of shape and size) to make them more relevant and specific to our sample, thereby improving our model's performance.

We assessed the model's performance with the following metrics [14] :

(a) sensitivity (or recall): the percentage of positive cases that were predicted to be positive (b) specificity: the percentage of negative cases that were predicted to be negative Moreover, we constructed 95% confidence intervals for the above metrics using bootstrapping.

We include results for the two-class classification (COVID-19 vs. normal lungs) and the three-class classification (COVID-19 vs. other viral pneumonia vs. normal lungs). Table 2 reports the performance of the two-class classification model with feature extraction only and with fine-tuning. Figure 5 depicts the performance after finetuning on the train (blue line) and validation subsets (orange line). The model reached an accuracy of 93.8% on the validation subset after training for 10 epochs (i.e., runs through the train dataset).

As shown in Table 2 , the model's performance improved with fine-tuning, achieving a recall (i.e., sensitivity) of 90% (95% CI 88, 92), specificity of 97% (95% CI 96, 98), precision (i.e., PPV) of 91% (95% CI 89, 93), F1-score of 90% (95% CI 88, 92), and accuracy of 95% (95% CI 94, 96). The fine-tuned model misclassified only 32 normal images as COVID-19 (versus 70 misclassifications in the model with only feature extraction). Table 3 reports the performance of the three-class classification model with feature extraction only and with fine-tuning. Figure 6 depicts the performance after finetuning on the train (blue line) and validation subsets (orange line). The model reached an accuracy of 92.6% after 10 epochs.

As shown in Table 3 , the model's performance improved with fine-tuning, achieving an accuracy of 93% (95% CI 92, 95), recall (i.e., sensitivity) of 94% (95% CI 93, 96), precision (i.e., PPV) of 86% (95% CI 84, 88), and F1-score of 90% (95% CI 88, 92). The F1-score improved 

We implemented a hybrid CNN that combines a pretrained EfficientNetB0 network with a dense layer (32 neurons) to differentiate between x-ray images of COVID-19 and normal lungs (and other viral pneumonia, in the three-class classification). After feature extraction and fine-tuning, the model achieved 95% (95% CI 94, 96) accuracy for the two-class classification and 93% (95% CI 92, 95) accuracy for the three-class classification. This model's performance is comparable to existing models [10, 12, 13] , but it offers several other advantages. Methodologically, to the best of our knowledge, this is the first instance in which the pre-trained Efficient-NetB0 (the baseline model of the EfficientNet family, which uses compound scaling to achieve higher accuracy) with a dense layer(32) on top has been used to improve the accuracy of COVID-19 diagnosis from X-ray images. Chaudhary et al. [12] used an EfficientNet-B1 model to distinguish COVID-19 from non-COVID-19 and normal x-ray images with 95% accuracy and 100% sensitivity, while Luz et al. [13] used a range of EficientNetB0-B5 models with four dense layers on top. In the latter case, the EfficientNet-B0 with four layers on top achieved an accuracy of 90% and sensitivity of 93.5%, whereas the best performing EfficientNet-B3 with four layers on top achieved 94% accuracy and 96.8% sensitivity. In comparison, our model achieved better accuracy (93%) and slightly better sensitivity (94%) than the B0-X (where X denotes the four layers on top) with fewer parameters (5 million vs 5.3 million). It also achieved similar accuracy and sensitivity with the B1 and B3-X (6.6 and 12.3 million parameters respectively). Jiao et al. [38] also used EfficientNet as the baseline

ConvNet, but the prediction was different (COVID-19 severity: critical vs. non-critical), and the model was considerably more complex: it connected a convolutional layer (256 neurons) and a dense layer (32 neurons) on top of the baseline. Despite its complexity, the model did not perform as well on an external dataset: accuracy of 75% (95% CI 74, 77), sensitivity of 66% (95% CI 64, 68), and We also mitigated overfitting by augmenting the training images (i.e., randomly transforming existing images to generate new ones). Data augmentation also addressed the class imbalance (as there were unequal numbers of COVID-19, other viral pneumonia, and normal images). We fine-tuned the model by training some of the top layers of our baseline CNN to improve their specificity to the current problem.

Finally, all of the reviewed studies except for one [14] reported only point estimates for their performance metrics (accuracy, sensitivity, specificity, positive predictive value, negative predictive value, and F1-score). By including the 95% confidence intervals, we capture the uncertainty of our estimates and enable a more comprehensive appraisal of the model.

The main limitation of our study is the lack of patient data. We appreciate that the model built by Jiao et al. [38] included patient clinical data (e.g., age, sex, oxygen saturation, biomarkers, comorbidities), which slightly improved the accuracy of the image-based CNN. We recognize that this is an important avenue for future research. We further note that the training and tuning of specificity of 70% (95% CI 69, 71). Although the present model addresses a different problem, we are confident that it performs better on the two-class classification problem than the model built by Jiao et al. Given that a model trained on more parameters is more prone to overfitting, we believe that our relatively small number of parameters and data augmentation strategy contributed to our model's superior performance.

Another advantage of this study lies in its design. Previous works have used a variety of splits to create subsets of data for two purposes: training and validation. For instance, Pham [14] used several random splits (80% vs. 20%; 50% vs. 50%; 90% vs. 10%) of an older version of the COVID-19 Radiography database used here. Rahimzadeh et al. [16] used eight subsets of their dataset for training and another subset for testing; Panwar et al. [17] used 70% for training and 30% for validation. To the best of our knowledge, the present research is the first to cluster the dataset into three independent subsets: train (70%), validation (20%), and test (10%). Thus, we reduced overfitting by testing the model's performance on a subset of data that did not contribute to the model's development. our "light" CNN took about two hours on a conventional Macintosh computer with 16G RAM and one Terabyte of a hard disk. More computational power is instead needed to train more advanced versions of the EfficientNet (from B1 through B7), possibly increasing the barrier of entry for non-specialists users (i.e., clinicians).

This study uses a "light" CNN to discriminate COVID-19 from other viral pneumonia and healthy lungs in chest X-ray images. To the best of our knowledge, our model is at present the most parsimonious CNN used to address the demanding issue of COVID-19 diagnosis via chest x-ray. The model successfully overcame the issue of low specificity that has prevented the ACR and CDC from endorsing chest imaging to diagnose COVID-19 [8] .

Specifically, the fine-tuned model differentiated COVID-19 from normal lungs with a positive predictive value of 91% and specificity of 97% (95% CI 96, 98); it differentiated COVID-19 from other viral pneumonia and normal lungs with a positive predictive value of 86% and specificity of 95% (95% CI 94, 96). Both classifications had a negative predictive value of 95% (95% CI 94, 96), meaning that a negative COVID-19 classification would indicate a 95% chance of not having COVID-19. Moreover, as shown in Table 3 , just one image of other viral pneumonia was misclassified as COVID-19, proving a remarkable discriminatory ability given the overlap in the presentation of COVID-19 and other viral cases of pneumonia.

The insights presented in this study may help clinicians (namely, radiologists) accurately diagnose COVID-19 at an early stage by enabling the use of x-ray as a first-totest tool that can complement RT-PCR analyses. Moving forward, the validation of our CNN on other databases would increase our confidence in the use of neural networks to aid the early diagnosis of both COVID-19 and other life-threatening diseases that traditionally have been difficult to diagnose via imaging.

Reverse transcription polymerase chain reaction. https:// en. wikip edia. org/ wiki/ Rever se_ trans cript ion_ polym erase_ chain_ react ion

Imaging profile of the COVID-19 infection: radiologic findings and literature review

Severity of lung involvement on chest X-rays in SARS-coronavirus-2 infected patients as a possible tool to predict clinical progression: an observational retrospective analysis of the relationship between radiological, clinical, and laboratory data

The role of chest imaging in patient management during the COVID-19 pandemic: a multinational consensus statement from the Fleischner society

Advoc acy-and-Econo mics/ ACR-Posit ion-State ments/ Recom menda tions-for-Chest-Radio graphy-and-CT-for-Suspe cted-COVID 19-Infec tion

Correlation of chest CT and RT-PCR testing for coronavirus disease 2019 (COVID-19) in China: a report of 1014 cases

Deep learning in the detection and diagnosis of COVID-19 using radiology modalities: a systematic review

Imagenet: a large-scale hierarchical image database

Efficient-CovidNet: deep learning based COVID-19 detection from chest x-ray images

Towards an effective and efficient deep learning model for COVID-19 patterns detection in X-ray images

Classification of COVID-19 chest X-rays with deep learning: new models or fine tuning?

COVID-19 detection in chest X-ray images using a deep learning approach

A modified deep convolutional neural network for detecting COVID-19 and pneumonia from chest X-ray images based Page

on the concatenation of Xception and ResNet50V2

Application of deep learning for fast detection of COVID-19 in X-rays using nCOVnet

Radiology indispensable for tracking COVID-19

Explainable deep learning for pulmonary disease and coronavirus COVID-19 detection from X-rays

Within the lack of chest COVID-19 X-ray dataset: a novel detection model based on GAN and deep transfer learning

Automated detection of COVID-19 cases using deep neural networks with X-ray images

Using X-ray images and deep learning for automated detection of coronavirus disease

Social group optimization-assisted Kapur's entropy and morphological segmentation for automated detection of COVID-19 infection from computed tomography images

Deep learning COVID-19 detection bias: accuracy through artificial intelligence

COVIDiagnosis-Net: deep Bayes-SqueezeNet based diagnosis of the coronavirus disease 2019 (COVID-19) from X-ray images

COVID-19 detection using deep learning models to exploit social mimic optimization and structured chest X-ray images using fuzzy color and stacking approaches

CoroNet: a deep neural network for detection and diagnosis of COVID-19 from chest X-ray images

Performance evaluation of the NASNet convolutional network in the automatic identification of COVID-19

CovidGAN: data augmentation using auxiliary classifier GAN for improved COVID-19 detection

COVID-19 identification in chest X-ray images on flat and hierarchical classification scenarios

COVID-19: automatic detection from X-ray images utilizing transfer learning with convolutional neural networks

New machine learning method for image-based diagnosis of COVID-19

Detection of coronavirus disease (COVID-19) based on deep features and support vector machine

Generalizability of deep learning tuberculosis classifier to COVID-19 chest radiographs: new tricks for an old algorithm?

Truncated inception net: COVID-19 outbreak screening using chest X-rays

COVID-19 Radiography Database

Rethinking model scaling for convolutional neural networks. International Conference on Machine Learning

Prognostication of patients with COVID-19 using artificial intelligence based on chest X-rays and clinical data: a retrospective study. The Lancet Digital Health

The online version contains supplementary material available at https:// doi. org/ 10. 1007/ s13755-021-00166-4.Below is the link to the electronic supplementary material.Supplementary file1 (DOCX 40 kb)

The author received no funding for this work.

The publicly available COVID-19 Radiography Database [36] was used.

The python notebook for the two-class classification model is available here: http://localhost:8888/notebooks/OneDrive%20-%20University%20of%20 Surrey/Kaggla_data/Two-class_classification.ipynb. The python notebook for the three-class classification model is available here: http://localhost:8888/ notebooks/OneDrive%20-%20University%20of%20Surrey/Kaggla_data/Three-class_classification.ipynb.

The author declares no conflict of interest.