key: cord-0808923-i9c49rnn
authors: Heidari, Morteza; Mirniaharikandehei, Seyedehnafiseh; Khuzani, Abolfazl Zargari; Danala, Gopichandh; Qiu, Yuchen; Zheng, Bin
title: Improving the performance of CNN to predict the likelihood of COVID-19 using chest X-ray images with preprocessing algorithms
date: 2020-09-23
journal: Int J Med Inform
DOI: 10.1016/j.ijmedinf.2020.104284
sha: 89bf0bc0389f4be5fcac65dcab288aa62de6853b
doc_id: 808923
cord_uid: i9c49rnn

OBJECTIVE: This study aims to develop and test a new computer-aided diagnosis (CAD) scheme of chest X-ray images to detect coronavirus (COVID-19) infected pneumonia. METHOD: CAD scheme first applies two image preprocessing steps to remove the majority of diaphragm regions, process the original image using a histogram equalization algorithm, and a bilateral low-pass filter. Then, the original image and two filtered images are used to form a pseudo color image. This image is fed into three input channels of a transfer learning-based convolutional neural network (CNN) model to classify chest X-ray images into 3 classes of COVID-19 infected pneumonia, other community-acquired no-COVID-19 infected pneumonia, and normal (non-pneumonia) cases. To build and test the CNN model, a publicly available dataset involving 8,474 chest X-ray images is used, which includes 415, 5,179 and 2,880 cases in three classes, respectively. Dataset is randomly divided into 3 subsets namely, training, validation, and testing with respect to the same frequency of cases in each class to train and test the CNN model. RESULTS: The CNN-based CAD scheme yields an overall accuracy of 94.5% (2404/2544) with a 95% confidence interval of [0.93,0.96] in classifying 3 classes. CAD also yields 98.4% sensitivity (124/126) and 98.0% specificity (2,371/2,418) in classifying cases with and without COVID-19 infection. However, without using two preprocessing steps, CAD yields a lower classification accuracy of 88.0% (2239/2544). CONCLUSION: This study demonstrates that adding two image preprocessing steps and generating a pseudo color image plays an important role in developing a deep learning CAD scheme of chest X-ray images to improve accuracy in detecting COVID-19 infected pneumonia.

From the end of 2019, a new coronavirus namely COVID-19, was confirmed in human bodies as a new category of diseases that cause dangerous respiratory problems, heart infection, and even death.

To more effectively control COVID-19 spread and treat patients to reduce mortality rate, medical images can play an important role [1] . In current clinical practice, chest X-ray radiography and computed tomography (CT) are two imaging modalities to detect COVID-19, assess its severity, and monitor its prognosis (or response to treatment). Although CT can achieve higher detection sensitivity, chest X-ray radiography is more commonly used in clinical practice due to the advantages, including J o u r n a l P r e -p r o o f low cost, low radiation dose, easy-to-operate and wide accessibility in the general or community hospitals [2] . However, pneumonia can be caused by many different types of viruses and bacterial.

Thus, it may be time-consuming and challenging for general radiologists in the community hospitals to read a high volume of chest X-ray images to detect subtle COVID-19 infected pneumonia and distinguish it from other community-acquired non-COVID-19 infected pneumonia. It is because there are many similarities between pneumonia infected by COVID-19 and other types of viruses or bacteria.

Thus, this is a clinical challenge faced by the radiologists in this pandemic [3] .

To address this challenge, developing computer-aided detection or diagnosis (CAD) schemes based on medical image processing and machine learning has been attracting broad research interest, which aims to automatically analyze disease characteristics and provide radiologists valuable decisionmaking supporting tools for more accurate or efficient detection and diagnosis of COVID-19 infected pneumonia. To this aim, studies may involve following steps of preprocessing images, segmenting regions of interest (ROIs) related to the targeted diseases, computing and identifying effective image features, and building multiple-feature fusion-based machine learning models to detect and classify cases. For example, one study [4] computed 961 image features from the segmented ROIs depicting chest X-ray images. After applying a feature selection algorithm, a KNN classification model was built and yielded an accuracy of 96.1% to classify between COVID-19 and non-COVID-19 cases.

However, due to the difficulty in identifying and segmenting subtle pneumonia-related disease patterns or ROIs on chest X-ray images, recent studies have demonstrated that developing CAD schemes based on deep learning algorithms without segmentation of suspicious ROIs and computing handcrafted image features is more efficient and reliable than the use of the classical machine learning methods. As a result, many deep learning models have been reported recently in the literature to detect and classify COVID-19 cases [2, [5] [6] [7] [8] [9] [10] [11] [12] [13] . Although some deep learning convolution neural network (CNN) models are applied to CT images [5, 6] , more studies applied CNN models to detect and classify COVID-19 cases using chest X-ray images. They include different existing CNN models (i.e., Resnet50 [2, 7] , MobileNetV2 [8] , CoroNet [9] , Xception+ResNet50V2 [10] ) and several new special CNN models (i.e., DarkCovidNet [11] , COVID-Net [12] and COVIDX-Net [13] ). These studies used different image datasets with a varying number of COVID-19 cases (i.e., from 25 to 224) among the total number of cases from 50 to 11,302. The reported sensitivity to detect COVID-19 cases ranged from 79.0% to 98.6%.

Despite the promising results reported in previous studies, many issues have not been well investigated regarding how to train deep learning models optimally. For instance, whether applying image preprocessing algorithms can help to improve the performance and robustness of the deep learning models. To better address some of the challenges or technical issues, we in this study develop J o u r n a l P r e -p r o o f and test a new deep learning based CAD scheme of chest X-ray radiography images. The scheme can detect and classify images into 3 classes namely, COVID-19 infected pneumonia, the other community-acquired non-COVID-19 infected pneumonia, and normal (non-pneumonia) cases. The hypothesis in this study is that instead of directly using the original chest X-ray images to train deep learning models, we can apply image processing algorithms to remove the majority of diaphragm regions, normalize image contrast and reduce image noise, and generate a pseudo color image to feed in 3 input channels of the existing deep learning models that were pre-trained using color (RGB) images in the transfer learning process. It may help significantly improve model performance and robustness in detecting COVID-19 cases and distinguishing them from other community-acquired non-COVID-19 infected pneumonia cases. To test this study hypothesis and demonstrate the potential advantages of new approaches, we assemble a relatively large chest X-ray image dataset with 3 class cases. Then, we select a well-trained VGG16 based CNN model as a transfer learning model used in our CAD scheme. The details of the study design and data analysis results are reported in the following sections of this article.

In this study, we utilize and assemble a dataset of chest X-ray radiography (CXR) images that are acquired from several different publicly available medical repositories [14] [15] [16] [17] [18] . These repositories were initially created and examined by the Allen Institute for AI in partnership with the Chan Zuckerberg Initiative, Georgetown University's Center for Security and Emerging Technology, Microsoft Research, and the National Library of Medicine -National Institutes of Health, in coordination with The White House Office of Science and Technology Policy. Specifically, the dataset used in this study includes 8,474 2D X-ray images in the posteroanterior (PA) chest view. Among them, 415 images depict with the confirmed COVID-19 disease, 5,179 with other community-acquired non-COVID-19 infected pneumonia, and 2,880 normal (non-pneumonia) cases. Figure 1 shows examples of three chest X-ray images acquired in three classes of normal, community-acquired non-COVID-19 infected pneumonia and COVID-19 pneumonia cases (from top to bottom). It shows that the bottom part of images includes a diaphragm region with high-intensity (or bright pixels), which may have a negative effect on distinguishing and quantifying lung disease patterns using deep learning models. Hence, an image pre-processing algorithm is applied to identify and remove diaphragm regions. Specifically, the algorithm detects the maximum (the brightest -) and minimum (the darkest -) pixel value of the image, then uses a threshold = + 0.9 × ( − ) to segment the original image into a binary image as shown in Figure 1 In the next step, we convert the segmented grayscale images ( ) to 3-channel images suitable for fine-tuning an existing CNN model pre-trained using color (RGB) images. To do so, we apply an image noise filtering method and a contrast normalization method to preprocess the image after removing the diaphragm region. First, since the X-ray images often include additive noise, we apply a bilateral low-pass filter ( ) to . This filter is a non-linear filter and highly effective at noise removal while preserving textural information compared to the other low pass filters. In other words, this filter J o u r n a l P r e -p r o o f analyzes intensity values locally and considers the intensity variation of the local area to replace the intensity value of each pixel with the averaged intensity value of the pixels in the local area. To calculate the weights, we apply a Gaussian low-pass filter in the space domain. This step generates a noise-reduction image. Based on our experimental results, we select the following parameters in the bilateral filtering ( = 9 and = 75). Second, chest X-ray images may have different image contrast or brightness due to the difference in patient body size and/or variation of X-ray dose. To compensate such a potentially negative impact, we apply a histogram equalization ( ) method to normalize images. This filter can enhance lung tissue patterns and characteristics associated with COVID-19

infection. Then, as shown in Figure 2 , three preprocessed images namely, , = ( ) and = ( ) form a pseudo color image that is fed into 3 input (RGB) channels of the CNN model.

In this study, we adopt a transfer learning approach since the previous studies have shown in order to avoid either overfitting or underfitting consequences using a small training dataset, a better approach is to take advantage of a CNN initially trained using a large-scale dataset [19] . Currently, many CNN models have been previously developed and are available for different engineering applications. In this study, we select a VGG16 model, which was pre-trained on the ImageNet Large Scale Visual

Recognition Challenge (ILSVRC) using a large dataset with 14 million images [20] . J o u r n a l P r e -p r o o f ILSVRC challenge [21] . As shown in Figure 3 , the VGG16 model has 13 convolutions, 5 max pooling and 3 fully connection layers in 6 blocks, which include over 138 million trainable parameters.

In our transfer learning, the weights between all connected nodes in front or low layers of the VGG16 based CNN model maintain unchanged (blocks 1 to 5 as shown in Figure 3 ). Next, block 6 in the model is modified by replacing with one flatten layer and two fully connected layers, which include 256 and 128 nodes, respectively. In these layers, the rectified linear unit (ReLU) [22] is used as their activation function. Then, all trainable weights in all connection nodes of the whole modified VGG16 model are fine-tuned using chest X-ray image data. In this fine-tuning process, a small learning rate (learning rate = 10 −5 ) is used to make a small variation to the pre-trained parameters. In this way, we will preserve the valuable parameters as much as possible by avoiding dramatic changes on the pre- Table 1 shows the complete architecture of the transfer learning VGG16 model built in this study. 

First, the original chest X-ray image has 1,024×1,024 pixels, while the VGG16 model was pretrained using images of 224×224 pixels. and normal cases, the case partition or assignment is done on three classes independently. Table 2 shows the number of cases in each subset.

J o u r n a l P r e -p r o o f Second, there are different available techniques to deal with imbalanced data [24] . In this study, the class weight technique, as one possible way, is applied during training to reduce the potential consequences of imbalance data. In the class weigh technique, we adjust weights inversely proportional to class frequencies in the input data [25] . The weight, in class is computed using the following equation.

The weights of the classes are utilized while fitting the model. Hence, in the loss function, we assign higher values to the instances of smaller classes. Therefore, the calculated loss will be a weighted average, where the weights of each sample corresponding to each class during loss calculation are specified with .

Additionally, in the training data of minority cases (COVID-19 cases), a common augmentation technique [26] is applied to increase the training sample size. First, using shearing factors (≤0.2), image intensity is sheared based on the shearing angle in a counter-clockwise direction. Second, using zooming factors (≤0.2), images are randomly magnified. Third, using rotation factors (within ±20°), To reduce the risk of potential bias in data partition into three subsets of training, validation, and testing, we repeat this model training and testing process three times by randomly dividing all cases into training, validation and testing subsets three times using the same case ratios or numbers as shown in Table 2 . In addition, during these three times of case partition, the cases assigned to the validation and testing subsets are totally different (no duplication). Three trained models are tested using totally different testing cases. Thus, the total number of testing cases increases (as shown in Table 2 ) to 2,544 (848× 3). Figure 4 shows a schematic diagram that illustrates the complete architecture of this VGG16 transfer learning CNN model, as well as the training, validation, and testing phase.

We perform experiments to analyze two different accuracies. The first one is accuracy for a threeclass classification to distinguish between COVID-19 infected pneumonia, community-acquired pneumonia, and normal (non-pneumonia) cases. We compute accuracy values in detecting images in 3 classes. We also calculate (1) a macro averaging, which is the average of 3 accuracy values of 3 Then, The remaining part is randomly split to 90% train and 10% validation. First, based on three confusion matrices as shown in Figure 5(a-c) To further evaluate the performance of our CAD scheme in detecting the COVID19 infected pneumonia cases using chest X-ray images, we place both normal and community-acquired pneumonia images into the negative class and COVID-19 infected pneumonia cases into the positive class.

Combining the data in the confusion matrix, as shown in Figure 5 (d), the CAD scheme yields 98.4% detection sensitivity (124/126) and 98.0% specificity (2,371/2,418). The overall accuracy is 98.1%

(2,495/2,544).

Next, Table 4 shows and compares (1) confusion matrixes generated by four models trained and tested using different input images and three data subsets generated from the data partition, as well as In addition, Table 5 compares our transfer learning VGG16 based CNN model and 10 state-of-art models recently reported in the literature to detect and classify COVID-19 cases. The Table shows the number of cases in the training and testing data subsets, imaging modality (CT or X-ray radiography), and reported classification performance including either 3-class or 2-class classification for these studies. Although the reported performance of these studies cannot be directly compared due to the use of different image dataset and testing methods, the presented data clearly demonstrate that our model is tested using relatively large dataset and yields very comparable classification performance as comparing to the state-of-art models developed and tested in this research field. preprocessing approaches can also be adopted to develop new deep learning models for other medical images to detect and classify other types of diseases (i.e., cancers [30, 31] ).

Despite encouraging results, this study also has limitations. First, although we used a publicly available dataset of 8,474 cases, including 415 COVID-19 cases, due to the diversity or heterogeneity of COVID-19 cases, the performance and robustness of this CAD scheme need to be further tested and validated using other large and diverse image databases. Second, this study only investigates and tests two image preprocessing methods to generate two filtered images, which may not be the best or optimal methods. New methods should also be investigated and compared in future studies. Third, to further improve model performance and robustness, it also needs to develop new image processing and segmentation algorithms to more accurately remove the diaphragm and other regions outside lung areas in the images. Therefore, more research work is needed to overcome these limitations in the future studies.

In this study, we proposed and investigated several new approaches to develop a transfer deep learning CNN model to detect and classify COVID-19 cases using chest X-ray images. Study results demonstrate the added value of performing image preprocessing to generate better input image data to build deep learning models. These include removing irrelevant regions, normalizing image contrastto-noise ratio, and generating pseudo color images to feed into all three channels of the CNN models in applying the transfer learning method. The reported high classification performance is also promising, which provides a solid foundation to further optimize the deep learning models to detect COVID-19 cases and validate its performance and robustness using large and diverse image datasets in future studies.

What was Already Known on the topic What this study adds to our knowledge  Due to the low cost, low radiation, wide accessibility, chest X-ray radiography is a good imaging modality to detect COVID-19. However, its sensitivity is lower than CT.

 Developing deep learning model based CAD schemes of chest X-ray images may play a useful role in facilitating detection and diagnosis of COVID-19.

 A few deep learning models using chest X-ray images to detect COVID-19 have been  Due to the diversity of image contrast and noise, adding image preprocessing steps is important and can help improve deep learning model performance.

 In transfer learning, one should not only use original images. It should add two additional filtered images to fill in 3 input channels of the deep learning model, which can enhance information learning and improve model performance.

Clinical and computed tomographic (CT) images characteristics in the patients with COVID-19 infection: What should radiologists need to know

Automatic detection of coronavirus disease (covid-19) using x-ray images and deep convolutional neural networks

CT imaging and differential diagnosis of COVID-19

New machine learning method for image-based diagnosis of COVID-19

A deep learning algorithm using CT images to screen for corona virus disease (COVID-19)

Deep learning enables accurate diagnosis of novel coronavirus (COVID-19) with CT images

Detection of coronavirus disease (covid-19) based on deep features

Covid-19: automatic detection from x-ray images utilizing transfer learning with convolutional neural networks

Coronet: A deep neural network for detection and diagnosis of COVID-19 from chest x-ray images

A modified deep convolutional neural network for detecting COVID-19 and pneumonia from chest X-ray images based on the concatenation of Xception and ResNet50V2

Automated detection of COVID-19 cases using deep neural networks with X-ray images

COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-Ray Images

Covidx-net: A framework of deep learning classifiers to diagnose covid-19 in x-ray images

Large dataset of labeled optical coherence tomography (OCT) and chest X-Ray images

Can AI help in screening viral and COVID-19 pneumonia

Epidemiological and clinical characteristics of 99 cases of 2019 novel coronavirus pneumonia in Wuhan, China: a descriptive study

COVID-19 image data collection

Identifying medical diagnoses and treatable diseases by image-based deep learning

A survey on transfer learning

Imagenet large scale visual recognition challenge

A review of deep learning in image recognition

Rectified linear units improve restricted boltzmann machines

Adam: A method for stochastic optimization

Skin lesions classification into eight classes for ISIC 2019 using deep convolutional neural network and transfer learning

Logistic regression in rare events data

The effectiveness of data augmentation in image classification using deep learning

Interrater reliability: the kappa statistic

Prediction of breast cancer risk using a machine learning approach embedded with a locality preserving projection algorithm

Development and Assessment of a New Global Mammographic Image Feature Analysis Scheme to Predict Likelihood of Malignant Cases

Deep CNN models for pulmonary nodule classification: model modification, model integration, and transfer learning

A dual-mode deep learning transfer learning (D2TL) system for breast cancer detection using contrast enhanced digital mammograms

This work is support in part by the grant from National Cancer Institute (R01 CA197150). The authors also thank the research support from Stephenson Cancer Center, University of Oklahoma, which helps establishment of our Computer-aided Diagnosis Laboratory.J o u r n a l P r e -p r o o f

reported using small datasets. The models were trained using the original images only. The deep learning CAD scheme can achieve high performance in detecting and classifying not only between COVID-19 cases and healthy (non-pneumonia) cases, and also between COVID-19 infected pneumonia and other community-acquired non-COVID-19 infected pneumonia cases. Our model is tested using a larger dataset as comparing to previous studies reported in the literature, which further supports the feasibility of this CAD approach.

Morteza Heidari and Bin Zheng design the study. Morteza Heidari implements the idea and write the required computer scheme coding link to VGG16 model. Abolfazl Zargari and SeyedehnafisehMirniaharikandehei collect the dataset and help test image filtering and normalization algorithms.Gopichandh Danala helps designing and testing image pre-processing and blob discovery strategy.Morteza Heidari drafts the manuscript. Bin Zheng and Yuchen Qiu review and revise the manuscript.

None of the authors have any conflict of interest to report.