key: cord-0693363-dh53s60r
authors: Naronglerdrit, Prasitthichai; Mporas, Iosif; Sheikh-Akbari, Akbar
title: COVID-19 detection from chest X-rays using transfer learning with deep convolutional neural networks
date: 2021-05-21
journal: Data Science for COVID-19
DOI: 10.1016/b978-0-12-824536-1.00031-9
sha: 14ed2a7d8d355d986330e7baf3ba357045a17afd
doc_id: 693363
cord_uid: dh53s60r

Several well-known pretrained deep convolutional neural network models were evaluated on their ability to detect COVID-19 from chest X-ray images, following a transfer learning approach. The retrained models were tested on two different datasets containing COVID-19, normal, viral, and bacterial pneumonia cases. The best performing models among the evaluated ones were the MobileNet, DenseNet, and ResNet after transfer learning retraining with top performing classification accuracies varying from 96.76% to 100%, thus indicating the potential of detecting the new coronavirus from X-ray images.

On the March 11, 2020, the World Health Organization (WHO) officially announced the outbreak of the new coronavirus COVID-19 a pandemic, after spreading to more than a 100 countries and leading to several thousands of cases in its first few months, with WHO declaring a pandemic over a coronavirus for first time [1] . COVID-19, which is a highly infectious disease, is caused by the SARS-CoV-2 virus [2] . Relatively young subjects that have been affected by COVID-19 usually are asymptomatic or present mild symptoms like cough, headache, fatigue, and fever, which for the general population and especially elders and patients with chronic conditions may progress to more serious symptoms like diarrhea, dyspnea, pneumonia, and death [3] . Young and middle-aged subjects being diagnosed with COVID-19 are having significantly lower mortality rates comparing to elder subjects with COVID-19 which have higher risk to progress to severe condition [4, 5] . With a 3.4% mortality rate estimated by the WHO on March 3, 2020 [6] and as COVID-19 is highly infectious it can easily be spread from asymptomatic to vulnerable population. To tackle spreading of COVID-19 and thus protect vulnerable people several governments around the world have applied isolation measures like social distancing and lockdown, while in parallel they perform large scale or targeted to suspicious cases diagnostic tests.

The diagnosis of COVID-19 is performed by the reverse-transcription polymerase chain reaction (RT-PCR) test after collection of proper respiratory tract specimen, which is a laboratory-based test for detection and quantification of a targeted DNA molecule [7] . The RT-PCR test can be done only by laboratories having the necessary infrastructure to carry it out and subject to case test may need be repeated after one or two days while the cost of the equipment and the required PCR reagents is not low, thus making this diagnostic test expensive and sometimes time consuming without counting the need for specialized microbiologists to do the tests analyses and the laboratory protocols that need to be taken to keep staff safe [7] . Because of these difficulties in many countries, the number of diagnostic tests for the new coronavirus is performed to only suspicious and/or critical cases and governments have taken isolation measures, which are causing socioeconomical problems (e.g., increasing number of domestic abuse cases [8] , reduction of economic growth [9] , global trade [10] ). Based on the above-mentioned facts, the development of alternative, complementary, and low-cost tools for detection of COVID- 19 and for decision-making support is essential.

The development of powerful machine learning tools over the last decade and the existence of deep learning models for classification of images, trained from big data collections could offer support in the global effort against the COVID-19. In this paper, we investigate the use of existing convolutional neural network (CNN) models pretrained with large volumes of image databases on the detection of the new coronavirus using transfer learning. In detail, a number of well-known deep CNN models were retrained using two databases of chest X-ray images including COVID-19 examples. The remainder of this chapter is organized as follows: literature review is provided in Subsection 1.1; description of the methodology followed, datasets used, and pretrained deep CNN models evaluated is given in Section 2; evaluation results are provided in Section 3 and conclusions in Section 4.

An automated classification method for X-ray COVID-19 lung images was reported by Mahdy et al. [11] . The proposed technique takes a subject's X-ray lung image and increases its contrast by applying a median filter on it. The resulting image then undergoes a threshold based multilevel image segmentation using the Otsu objective function. The support vector machine (SVM) algorithm is then used to classify the COVID-19 positive images from the other images. The proposed algorithm was trained and tested using 40 contrast-enhanced lungs X-ray images of size 512 Â 512 in-plane resolution. This dataset includes 15 normal lung images and 25 infected lungs with COVID-19 images from the Montgomery County X-ray Set and covid-chest X-ray-dataset-master, respectively. They reported an average sensitivity, accuracy, and specificity of 95.76%, 97.48%, and 99.7% for their system, respectively.

Abbas et al. proposed a decompose, transfer, and compose (DeTraC) CNNs-based algorithm for classification of COVID-19 chest X-ray images in Ref. [12] . The proposed method first trains the backbone pretrained CNN model of the DeTraC to select deep local features of each input image and simplifies the distribution of the local build of the data using the class-decomposition layer of DeTraC. It then uses a sophisticated gradient descent optimization algorithm to complete the network's training. The proposed method finally uses the class-structure layer of DeTraC to polish the classification of the images. The authors used a combination of 80 samples of normal Chest X-ray images from Japanese Society of Radiological Technology (JSRT), 105 sample of COVID-19, and 11 SARS infected images from Cohen JP. COVID-19 image dataset [13] . They applied various data augmentation techniques like flipping, rotation, and translation to augment the amount of samples. They reported a performance accuracy of 95.12% with a sensitivity of 97.91%, a precision of 93.36%, and a specificity of 91.87%.

The application of the pretrained CNNs along with SVM classifier to detect the COVID-19 from chest X-ray images was reported in Ref. [14] . The proposed technique employs a pretrained CNN to extract deep features from the input chest X-ray image. The SVM classifier is then trained and used to recognize the COVID-19 cases. The performance of the presented approach was assessed using datasets online available in GitHub, Kaggle, and Open-i, which contain validated X-ray images. The proposed architecture, i.e., resnet50 plus SVM, attained COVID-19 detection accuracy, in terms of F1 score, False Positive Rate, MCC (Matthews Correlation Coefficient), and Kappa of 95.38%, 91.41%, 95.52%, and 90.76%, respectively (disregarding MERS, SARS, and ARDS).

Wang and Wong proposed a deep CNN architecture for recognition of subjects positive to COVID-19 from chest X-ray images, named COVID-Net in Ref. [15] . The authors used a human-machine cooperative strategy to design the COVID-Net. The proposed network is able to classify chest X-ray images into three groups: (a) normal (no infection), (b) non-COVID-19 contamination (e.g., bacterial, viral, etc.), and (c) COVID-19 viral contamination. This technique employed residual architecture design principles, introduced in Ref. [16] to enable reliable neural network architecture to be trained to its high performance. Moreover, they generated a chest X-ray image dataset to train the proposed COVID-Net, called: COVIDx. COVIDx dataset contains 13,800 Chest X-ray images from 13,725 patients, constructed as a mixture and alteration of three open access data repositories' images (i.e. [17, 18] , and [13] ). The proposed technique achieved 92.6% accuracy with 87.1% sensitivity and 96.4% Positive Predictive Value for detecting COVID-19 cases.

Li et al. introduced a three-dimensional (3D) deep learning framework to distinguish COVID-19 cases from chest X-ray images called: COVNet in Ref. [19] . The proposed method first extracts the lung region with the image as the region of interest (ROI) by a U-net [20] ebased segmentation algorithm. The resulting ROI part of the image was then fed to the COVNet for classification. The COVNet employs a ResNet-50, presented in Ref. [16] to extract features for the corresponding ROI. The proposed technique then performs a max-pooling operation to combine the features, generating a feature map. An entirely connected layer and softmax activation function are then used to determine the detection score for COVID-19, nonpneumonia and CAP case from the resulting feature map. The authors reported a performance of 90% and 96% in terms of sensitivity and specificity for the proposed framework on detecting COVID-19 cases from an independent testing dataset.

In Ref. [21] , an artificial intelligenceebased automatic computed tomography (CT) image examination tools for recognition, quantification, and tracking of COVID-19 cases from others were presented. The proposed method consists of two subsystems and analyses the CT case at two separate levels. For subsystem A, the authors used a salable off-the-shelf software to detect small opacities and nodules within a 3D lung volume (RADLogics Inc., Boston [22] ). This software generates quantitative measurements including axial measurements (RECIST), volumetric measurements, The Hounsfield Unit values, texture description, and calcification recognition for solid versus subsolid versus GG from the input image. They assumed that this software can detect Ground-Glass Opacities, which current studies showed is one of the important features for detecting COVID-19, within the input image. In subsystem B, the lung ROI is first extracted by a U-net architecture for image segmentation, presented in Refs. [20, 23] . It then uses a pretrained ResNet-50d2D deep CNN architecture, introduced in Ref. [16] , which has 50 layers and can categorize the image into 1000 types. The authors fine-tuned the network parameters by further training to solve the problem of having suspicious COVID-19 cases from different Chinese hospitals. The authors assessed the performance of the proposed system using CT images of 56 COVID-19 positive chines patients and for 51 chines non-Coronavirus patients from multiple institutions in China. The authors assessed the performance of the proposed system using CT images of 56 subjects with positive COVID-19 diagnosis and for 51 chines non-Coronavirus patients from multiple institutions in China. They reported an area under curve (AUC) of 0.996 (95%CI: 0.989e1.00), assuming the positive ratio as a decision feature.

Chowdhury et al. used a deep CNN-based transfer learning approach for automatic detection of COVID-19 pneumonia in Ref. [24] . The authors trained and tested four different popular CNN-based deep learning algorithms, called AlexNet [25] , ResNet-18 [26] , DenseNet-201 [26] , and SqueezeNet [27] , to classify normal and pneumonia patients using chest X-ray images. They considered two classification schemes: (a) COVID-19 pneumonia and normal and (b) COVID-19 pneumonia, viral, and normal. To generate experimental results, the author generated a public database that comprises a combination of 190 COVID-19, 1341 normal and 1345 viral pneumonia, chest X-ray images and conducted two experiments on two classification schemes: (i) two-and three-class classification using models trained without augmentation and (ii) two-and three-class classification using models trained with image augmentation. They used accuracy, sensitivity or recall, specificity, precision (PPV), AUC, F1 score measures to assess the performance of different networks. Their experimental results show that SqueezeNet outperforms other three different deep CNN networks for classifying images from normal and COVID-19 group and in normal, viral pneumonia and COVID-19 group, where its classification accuracy, sensitivity, specificity, and precision of normal and COVID-19 images, and normal, COVID-19 and viral pneumonia were (98.3%, 96.7%, 100%, 100%), and (98.3%, 96.7%, 99%, 100%), respectively. Moreover, the concept of transfer learning in deep learning framework was used by Vikash et al. [28] for the detection of pneumonia using pretrained ImageNet models [29] and their ensembles.

In this work, we relied on deep CNN models for image classification which have been pretrained from large volumes of images. Specifically, collections of chest X-ray images with known clinical diagnoses for each subject, including COVID-19, were used to retrain preexisting deep CNN models in a transfer learning approach. Before introducing the X-ray images to the deep CNN models to retrain them, image preprocessing was performed, consisting of image resizing and pixel values normalization to meet the input specifications of each pretrained deep CNN model. After retraining the models using the X-ray images, new chest X-rays with unknown clinical diagnosis labels were tested to automatically detect images of subjects with COVID-19. The block diagram of the transfer learning architecture used in the present evaluation for detecting COVID-19 patients from their chest X-ray images is shown in Fig. 13 .1.

In the following subsections, we describe the evaluation data and the pretrained deep CNN models used in the present evaluation. 

For the retraining of the preexisting deep CNNs and the evaluation of the new retrained models, we relied on two datasets available online. The first dataset [24] (Dataset-A) consists of grayscale chest X-ray images of size equal to 1024 Â 1024 pixels. The dataset has three classes and each of the X-ray images has been labeled as "COVID-19," "normal," or "viral pneumonia." The number of X-ray images per class of [24] is tabulated in Table 13 .1.

The second dataset [30] (Dataset-B) consists of grayscale chest X-ray images of size equal to 300 Â 400 pixels. The dataset has four classes and each of the X-ray images has been labeled as "COVID-19," "normal," "viral pneumonia," or "bacterial pneumonia." The number of X-ray images per class of [30] is tabulated in Table 13 .2.

Examples of chest X-ray images of subjects having been diagnosed as "normal," "COVID-19," "viral pneumonia," or "bacterial pneumonia" are shown in Fig. 13 .2.

During preprocessing of the X-ray images, they were resized to 224 Â 224 pixels, using bilinear interpolation, to fit to the pretrained deep models input size. The resized X-ray images' pixel values were then normalized to the range [0, 1] to the retraining of the deep CNN models to converge faster. For the preprocessing of the X-ray images, the computer vision and image processing library OpenCV [31] was used. 

The VVGNet has been constructed by the Visual Geometry Group of the University of Oxford [34] . The VGG network architecture was created using 3 Â 3 convolution filters which are packed as a stack of convolutional layers, with each stack then being connected to the max-pooling layer. Two fully connected layers are before the classification layer. The VGGNet can have different depth which is indicated by the number after the VGGNet name. In this evaluation, the VGGNet-16 which has 16 weight layers was used. The "vanilla" VGGNet was trained using the ImageNet [35] with 224 Â 224 input size and 1000-class outputs. The retrained VGGNet-16 for classification of X-ray images has 2, 3, or 4 class outputs. The VGGNet-16 architecture is tabulated in Table 13 .3. 

The MobileNet [33] is a CNN model in which convolutional layers can be replaced by depthwise separable convolutions and together with 1 Â 1 kernels are used for pointwise convolutions form a depthwise separable convolution block. The advantage of MobileNet's architecture is the reduced number of computations needed, both during the training of the CNN model as well as during online testing. The MobileNet has been constructed as a light-weight architecture for mobile and embedded vision applications. The "vanilla" MobileNet was trained using the ImageNet [35] with 224 Â 224 input size and 1000-class outputs. The retrained MobileNet for classification of X-ray images has 2, 3, or 4 class outputs. The architecture of MobileNet is tabulated in Table 13 .4.

ResNet [16] stands for Residual Network and is based on the residual learning framework. In residual networks, shortcut connections between stacks of convolutional layers are inserted. The result of the inserted shortcuts into the plain network is to address 

The Dense Convolutional Network (DenseNet) [32] is the CNN architecture that has the connection from the previous layer to every other next layer. Unlike ResNet which has the shortcut connection inserted into the plain network, DenseNet has connections inserted to other layers which are called the dense block, and each block is connected to the next block in cascade with a convolutional layer and pooling layer in between them. The "vanilla" DenseNet models were trained using the ImageNet [35] with 224 Â 224 input size and 1000-class outputs. The retrained DenseNet models for classification of Xray images have 2, 3, or 4 class outputs. The architectures of DenseNet-121, DenseNet-169, and DenseNet-201 are tabulated in Tables 13.8e13.10, respectively.

The transfer learning architecture described in the previous section was evaluated using the four deep CNN models described above for COVID-19 detection from X-ray images. The performance of the evaluated CNN models was measured in terms of classification accuracy, i.e.,

where TP is the number of true positives, TN is the number of true negatives, FP is the number of false positives, and FN is the number of false negatives of the classified dermatoscopic images. To avoid overlap between the training and testing subsets, a 10fold cross validation protocol was used. 

The X-ray image classification results for Dataset-A [24] for all evaluated deep CNN models after retraining them are tabulated in Table 13 .11. The best performing model and classification accuracy are indicated in bold. The best performing model and classification accuracy are indicated in bold.

As can be seen in Table 13 .11, the best performing model after transfer learning is the MobileNet, followed by ResNet-50, DenseNet-201, and DenseNet-121 all having accuracy more than 99%. In addition, we performed binary classification (i.e., COVID-19 vs. non-COVID- 19) , and the binary classification results are tabulated in Table 13 .12.

As can be seen in Table 13 .12, the best performing model in binary classification is also the MobileNet, followed by ResNet-101, DenseNet-201, and ResNet-152 all having accuracy more than 99%. To investigate the classification accuracy per class, the confusion matrices for the multiclass and the binary classification of Dataset-A are shown in Tables 13.13 and 13.14, respectively. As shown in Tables 13.13 and 13 .14, the best performing retrained MobileNet model can detect the COVID-19 class with accuracy at or close to 100% both in multiclass and binary classification setups.

The X-ray image classification results for Dataset-B [30] for all evaluated deep CNN models after retraining them are tabulated in Table 13 . 15 . The best performing model and classification accuracy are indicated in bold.

As can be seen in Table 13 .15, the best performing model after transfer learning is the DenseNet-201, followed by ResNet-50, with both having accuracy above than 96%. In addition, we performed binary classification (i.e., COVID-19 vs. non-COVID- 19) , and the binary classification results are tabulated in Table 13 .16. As can be seen in Table 13 . 16 , the best performing model in binary classification is the ResNet-50 with accuracy 100% followed by DenseNet-201 with accuracy 99.9%, DenseNet-201 and ResNet-152 both having accuracy 99.85%. To investigate the classification accuracy per class, the confusion matrices for the multiclass and the binary classification of Dataset-B are shown in Tables 13.17 and 13.18, respectively.

As shown in Tables 13.13 and 13.14, the best performing retrained models, i.e., DenseNet-201 model in the multiclass classification setup and the ResNet-50 model in the binary classification setup, are detecting COVID-19 class without any false rejections. In the case of multiclass classification, 0.23% of the normal class X-ray images and 0.31% of the bacterial pneumonia class X-ray images were false positives to COVID-19 class.

To summarize the above-presented results and focusing on COVID-19 detection (i.e., the binary setup), we present the detection accuracy in percentages for each of the evaluated deep CNN models for both X-ray datasets in Fig. 13 .3. As can be seen in Fig. 13 .3, the models that achieved competitive detection accuracy in both datasets are the DenseNet-201, MobileNet, ResNet-152, and ResNet-101. 

COVID-19 outbreak has already caused thousands of deaths as well as serious effects in world's health, social life, and economy. Because of the huge number of cases, the fact that the virus has not yet well-studied and because of the difficulties of the diagnostic tests, it is essential to investigate the possibility of having alternative, complementary, and supportive tools to assist medical staff in decision making and thus tackling this pandemic. Following a transfer learning approach, we evaluated several well-known pretrained deep CNN models for detection of subjects with COVID-19 from their chest X-ray images, which included normal, viral, and bacterial pneumonia cases, except the new COVID-19 coronavirus. The retrained models were tested on two different datasets, and the best performing models were the MobileNet, DenseNet, and ResNet with top performing classification accuracies varying from 96.76% to 100%. The evaluation results indicated the potential of detecting the new coronavirus from X-ray images. The collection of more chest X-ray data from diagnosed cases with COVID-19 will allow the use of data mining analysis to discover COVID-19 markers in the X-ray images and subsequently allow the development of such computer-aided diagnostic tools.

World Health Organization declares global emergency: a review of the 2019 novel coronavirus (COVID-19)

The proximal origin of SARS-CoV-2

3 COVID-19 detection accuracy (binary setup) for the two chest X-ray datasets and for all evaluated deep convolutional neural network models

The epidemiology and pathogenesis of coronavirus disease (COVID-19) outbreak

Clinical features of COVID-19 in elderly patients: A comparison with young and middle-aged patients

Contopoulos-Ioannidis, Population-level COVID-19 mortality risk for non-elderly individuals overall and for non-elderly individuals without underlying diseases in pandemic epicenters, medRxiv (2020)

WHO Director-General's Opening Remarks at the Media Briefing on COVID-19 -3 March 2020 -World Health Organization

The laboratory diagnosis of COVID-19 infection: current issues and challenges

Family violence and COVID-19: increased vulnerability and reduced options for support

The Great Lockdown

Chief Economist Team, DG Trade, European Commission

Automatic X-ray COVID-19 lung image classification system based on multi-level thresholding and support vector machine, medRxiv

Classification of COVID-19 in chest X-ray images using DeTraC deep convolutional neural network, medRxiv

COVID-19 Image Data Collection, 2020. Avaliable at

Detection of coronavirus disease (COVID-19) based on deep features

COVID-net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from

Deep residual learning for image recognition

Avaliable at

Figure 1 COVID-19 Chest X-Ray Data Initiative

Artificial intelligence distinguishes COVID-19 from community acquired pneumonia on chest CT, RSNA Radiol

U-net: convolutional networks for biomedical image segmentation

Rapid AI Development Cycle for the Coronavirus (COVID-19) Pandemic: Initial Results for Automated Detection & Patient Monitoring Using Deep Learning CT Image Analysis

Improving the segmentation of anatomical structures in chest radiographs using u-net with an imagenet pretrained encoder

Can AI Help in Screening Viral and COVID-19 Pneumonia?, arXiv Preprint

Imagenet classification with deep convolutional neural networks

Inception: Understanding Various Architectures of Convolutional Networks, CV-Tricks.comLearn Machine Learning, AI & Computer Vision

Proceedings of 2010 IEEE International Symposium on Circuits and Systems

A novel transfer learning based approach for pneumonia detection in chest X-ray images

The Data that Transformed AI ResearchdAnd Possibly the World

COVID-19 Detection X-Ray Dataset

The opencv library, Dr Dobb's

Paper Presented at the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

Mobilenets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Very Deep Convolutional Networks for Large-Scale Image Recognition

Paper Presented at the 2009 IEEE Conference on Computer Vision and Pattern Recognition