key: cord-0939700-7vtssku9
authors: Bargshady, Ghazal; Zhou, Xujuan; Barua, Prabal Datta; Gururajan, Raj; Li, Yuefeng; Acharya, U.Rajendra
title: Application of CycleGAN and transfer learning techniques for automated detection of COVID-19 using X-ray images
date: 2021-12-03
journal: Pattern Recognit Lett
DOI: 10.1016/j.patrec.2021.11.020
sha: f0bc20fda6527783dda9b86dbb969bb59f7d03da
doc_id: 939700
cord_uid: 7vtssku9

Coronavirus (which is also known as COVID-19) is severely impacting the wellness and lives of many across the globe. There are several methods currently to detect and monitor the progress of the disease such as radiological image from patients’ chests, measuring the symptoms and applying polymerase chain reaction (RT-PCR) test. X-ray imaging is one of the popular techniques used to visualise the impact of the virus on the lungs. Although manual detection of this disease using radiology images is more popular, it can be time-consuming, and is prone to human errors. Hence, automated detection of lung pathologies due to COVID-19 utilising deep learning (Bowles et al.) techniques can assist with yielding accurate results for huge databases. Large volumes of data are needed to achieve generalizable DL models; however, there are very few public databases available for detecting COVID-19 disease pathologies automatically. Standard data augmentation method can be used to enhance the models’ generalizability. In this research, the Extensive COVID-19 X-ray and CT Chest Images Dataset has been used and generative adversarial network (GAN) coupled with trained, semi-supervised CycleGAN (SSA- CycleGAN) has been applied to augment the training dataset. Then a newly designed and finetuned Inception V3 transfer learning model has been developed to train the algorithm for detecting COVID-19 pandemic. The obtained results from the proposed Inception-CycleGAN model indicated Accuracy = 94.2%, Area under Curve = 92.2%, Mean Squared Error = 0.27, Mean Absolute Error = 0.16. The developed Inception-CycleGAN framework is ready to be tested with further COVID-19 X-Ray images of the chest.

Coronavirus-2 (SARS-CoV-2) causes lung disease and by fast spreading affects the well-being of the global population. Identifying on-time the COVID-19 cases and monitoring the disease progress could prevent spreading it in the society and speed up treatment process (Apostolopoulos & Mpesiana, 2020) . Recently, polymerase chain reaction (RT-PCR) testing method employed to detect COVID-19 and identify SARS-CoV-2 ribonucleic acid (RNA) from respiratory specimens. The RT-PCR testing currently is the one of the top standards in detecting COVID-19, but it is very time-consuming, painful, and bit complicated manual process. Healthcare staff are required to engage in close contact with individuals to perform RT-PCR tests which may increase the risk of COVID-19 virus infection (Apostolopoulos & Mpesiana, 2020) . There are some other disadvantages in applying RT-PCR as well including, insufficient numbers of available test kits, costs, threats to the healthcare staff safety, and waiting time of test results ).

An alternative method in identifying COVID-19 are chest imaging investigations, such as X-ray or computed tomography (CT) imaging which are analyzed by specialist radiologists to identify visual indicators associated with COVID-19 biological infection . Early studies showed patients showed abnormalities in patients' chest radiology images in COVID-19 infection such as ground-glass opacity, bilateral or interstitial abnormalities in chest X-Ray and CT scans. The radiology scans required to analysis by radiologist to identify COVID-19 from the images. Artificial intelligence (AI) based tools could improve the performance of the radiologist analysis. The AI and machine learning algorithms recently demonstrate remarkable progress in image-recognition tasks . Wang et al. (2020) have developed the COVID-Net, which is a deep convolutional neural network (DCNN) method to identify patients with COVID-19 from X-ray imaging. The applied chest radiography images dataset contains COVID-19, viral and bacterial pneumonia, and other infection classes. The overall accuracy obtained by their method is 83.5% in four classes and 92.4% overall accuracy for binary classes (i.e., COVID-19, and non-COVID groups). Hemdan et al. (2020) have developed the COVIDX-Net AI algorithm to assist radiologists in identifying COVID-19 through chest X-rays. They introduced seven deep learning models involving VGG19, MobileNetV2, DenseNet201, ResNetV2, InceptionResNetV2, Xception and InceptionV3 applied these models in the COVIDX-Net algorithm. The results indicated that the effectiveness of the VGG19 and DenseNet was similar in terms of measuring F 1 -scores (0.91 and 0.89 respectively). On the other hand, Sethy and Behera (2020) proposed a deep learning technique in X-ray scans database for the same issue by achieving an accuracy of 95.38%. Ozturk et al. (2020) presented DarkNet or DarkCOVIDNet model and the proposed model has provided diagnoses for 3 classes, including positive COVID-19 cases, all types of pneumonia, and healthy classes. The model has been trained for binary classification, including COVID-19 and normal cases as well. The accuracy of 3-class classification is 87.02% and 98.08% for two classes.

CoroNet consists of a convolution neural network (CNN) has been introduced by Khan, Shah, et al. (2020) for detecting COVID-19. The experimental results have an overall accuracy of 89.6% from X-ray and 95% from CT radiology images for four classes including bacterial pneumonia, COVID-19, other pneumonias, and normal. It also trained and evaluated three classes including pneumonia, and normal (or benign) cases. There is some other research in the same area that based on CT scans image analysis. For example, COVNet introduced by Li et al. (2020) (Afshar et al., 2020; Barua, P.D. et.al, 2021; Ismael & Şengür, 2021; Sharifrazi, D. et.al, 2021; Shoeibi, A. et.al; Zhang et al., 2020; Zhang et al., 2021) , it is still challenging to detect it accurately using x-ray images alone. The current main issue in applying deep learning methods is that there are limited number of publicly available COVID-19 chest image databases available. The big data is needed to train the deep convolutional networks and generalize the developed models. To solve the limited availability of COVID-19 chest X-ray databases and for considering use of deep learning models to detect COVID-19, a new CycleGAN-Inception approach is proposed in this paper. Figure 1 shows the proposed model's framework in COVID-19 automated detection. The Extensive COVID-19 Xray and CT Chest Images Dataset (Walid & Fathi, 2020) has been used for validation of the proposed model however for the training set we used the dataset in addition augmented data by CycleGAN has added to training dataset to train the proposed model. The database has collected from several COVID-19 radiological databases including the "Extensive-COVID-19-Xray-and-CT-Chest-Images Dataset" (Khan, Sohail, et al., 2020) and the "Chest-Xray-Dataset" (Cohen et al., 2020) .

For better performance of the proposed model, CycleGAN (Zhu et al., 2017) has been used to generate and augment data by translating COVID-19 images to normal images and normal to COVID-19 images. The GAN methods have been previously used to produce new training images (Antoniou et al., 2017) , refine synthetic image (Shrivastava et al., 2017) and improve brain segmentation (Bowles et al., 2018) .

As shown in Figure 1 , data flow has four steps. Stream (x) presents all COVID-19 original X-ray image data from collected databases and stream (y) provides all non-COVID-19 original Xray image data from the selected databases. The Dx and Dy are the generated translated data by CycleGAN. All original and generated X-ray image data are fed to the pre-processing section of the proposed model to undergo normalization, resizing, and centralization. The pre-processing section processes data to make it more suitable to feed the data to the proposed pre-trained Inception V3. The proposed Inception V3 pre-trained model is modified and fine-tuned by freezing fully connected layers and adding two fine-tuned fully contacted layers. The primary utility and significance of this work are listed as follows:

 A novel data augmentation approach is developed to generate images with COVID-19 and normal characteristic using CycleGAN.  The developed novel model is trained and evaluated using original and translated image data using CycleGAN.  The newly designed model can identify positive COVID-19 cases from chest X-ray radiological images effectively and with high accuracy.

The rest of the paper is organized as the following: In section 2 the structure and concepts of the proposed CycleGAN-Inception model is presented. Section 3 provides the experimental configuration, evaluation metrics, and collected data. Section 4 describes discussion and results of the trained and evaluated the CGAN-Inception proposed model and demonstrates its effectiveness. Finally, Section 5 presents the summary of the paper. and non-COVID-19 (stream Y) X-ray images are from the selected databases. Dx and Dy data flow are the translated data by CycleGAN to deal with the data limitation problem.

The CycleGAN-Inception model is developed to categorize chest X-ray images with or without COVID-19 characteristics ( Figure 1 ). The proposed approach has two steps: (i) data augmentation technique, the model generates extra X-ray images by applying CycleGAN introduced by (Zhu et al., 2017) . Then all data transfer to image pre-processing section for normalization and resizing input data. In the next step, all data transfer to the proposed and modified Inception V3 pre-trained model. More details of the components of the proposed model were described in the following subsections.

A CycleGAN was introduced by (Zhu et al., 2017) as a model of training the deep CNNs in an image-to-image translation format. It also identifies dissimilar other generative adversarial networks developed by (Goodfellow et al., 2014) for image translation, allowing the CycleGAN to learn mapping between one image domain and another through an unsupervised approach (Zhu et al., 2017) . The CycleGANs facilitates learning mapping from one area (X) to another area (Y), without identifying the matched training pairs. It is also often used in data augmentation. Sandfort et al. (2019) evaluated the CycleGAN for data augmentation purpose by training CycleGAN in converting contrast CT scan images into non-contrast scans. GANs have been used in the past for data extension to produce unique training images set for classification purpose (Antoniou et al., 2017) , refine synthetic images (Shrivastava et al., 2017) and improve brain segmentation (Bowles et al., 2018) .

Since labelled medical imaging data can be both difficult and expensive to obtain, access to big medical data is essential in achieving an accurate and robust deep learning algorithm. Standard data augmentation is a routinely performed process which is used to increase generalizability. However, the GANs could offer a novel approach of data extension. Therefore, the newly trained CycleGAN with COVID-19 X-ray radiological images is applied in this work to augment existing models using non-contrast images. Zhu et al. (2017) and Godard et al. (2017) used cycle consistency loss to supervise CNN training. Zhu et al. (2017) have introduced a parallel loss to drive G and F with same coherence. Their model consists of mapping G: X→Y and F: Y→X by introducing adversarial discriminators of D x and D y . To distinguish between images {x} and translated images {F(y)}, D x and to discriminate between {y} and {G(x)} D y are applied. For more information about the principles and concepts of it the reader could refer to the Zhu et al. (2017) . As a summary the definitions of components are describing mathematically as follows.

The adversarial losses are definition in Equation (1): (1)

In This formula, G generates images G(x) that appears like images from field Y and D y observes between translated samples G(x) and original samples y.

The cycle consistency loss is defined as following in Equation (2) as per an image y from domain Y, G and F are satisfying backward cycle consistency:

(2)

The cycle consistency loss is formulated (see Equation 3) as follow:

(3) Therefore,

where ℷ conducts the relation of the two objectives. CycleGAN has been applied in the proposed model to translation of non-COVID-19 to COVID-19 images and vice versa as following:

The discriminators D x and D y are CNNs that read an image and classify it as true or false (real or fake). "True" is indicated by an output close to 1 and "false" as close to 0. The proposed architecture consists of five convolutional layers, which generate a single logit to detect whether the image is real or not. The architecture does not include a fully connected layer. The convolutional layers are then supported by batch normalization, and the rectified linear activation function (ReLU) for hidden units.

In reality, the datasets are often insufficient to train the deep learning algorithms and obtaining the labelled data can be costly. Transfer learning techniques are used to achieve higher performance of the machine learning algorithms by using labelled data or getting knowledge from similar domains (Pan & Yang, 2009) . It can implement the knowledge that has been realized in earlier settings. There are also two popular transfer learning strategies: deep feature extraction and fine-tuning (Bargshady et al., 2020b; Kaya et al., 2019) . Transfer learning trains a task similar to the deep neural network. The initial layers of the pre-training network are modified if needed. The fine-tuning parameters can be applied to the model's final layers for learning the features of the new dataset. The pre-trained model retrains with a new, smaller, dataset, and the weights of the model are refined according to the new tasks. Fine-tuning takes place by backpropagation with labels. The learning to transmit approach is more efficient than the effort required to train an original, new neural network. The parameters in the newly developed neural network are not established from scratch. DL algorithms can achieve a higher function or performance for many issues -however, they rely upon large amounts of data and a longer training timeframe (Bargshady et al., 2020a; Soar et al., 2018) . Therefore, reusing pre-trained models for similar tasks can be very beneficial. Accordingly, the Inception V3 (Szegedy et al., 2015; Szegedy et al., 2016) has been used as a pre-trained model and was finetuned with the proposed dataset in this work. A finetuned Inception V3 is modified and applied as a pre-trained Inception to classify COVID-19 images from collected and generated data by CycleGAN. Inception V3 assembles the sparse convolution kernels into fewer dense sub-convolution kernel groups. The structure of the Inception V3 and core unit of Inception is shown in Figure 2 and The proposed model architecture is shown in Figure 4 . We removed the Softmax layers and fully connected layers in Inception V3 and froze trainable layers. Then two fully connected layers of size 92 and 1028, and activation ReLU are included to train the model. 

The algorithm used in our proposed model is described below: 

We have performed the experiments using an Intel Core i7 computer with 3.3 GHz and 16 GB memory. The suggested algorithm was implemented by Python software v3.6 (Sanner, 1999) with Keras (Ketkar, 2017) and TensorFlow (Abadi et al., 2016) libraries. Using Keras library allows us to develop the prototypes quickly and the library supports both the CNN and the recurrent neural networks.

The "Extensive COVID-19 X-ray and CT Chest Images" Database were used in this work to train and develop our model (Walid & Fathi, 2020) . The database consists of two folders: 1) 5,500 normal and 4,044 COVID-19 X-ray chest scans. 2) 2,628 normal and 5,427 COVID-19 CT scans. Only the X-ray images from the database were used in our study. Samples of COVID-19 and non-COVID-19 X-rays images are presented in Figures 6  and 7 . The augmented data have been used to train the model in this work.

Samples of non-COVID-19 X-ray images (Walid & Fathi, 2020 ).

Samples of COVID-19 X-ray images (Walid & Fathi, 2020) .

Evaluation of obtained results is an important step in machine learning development. Estimation of the effectiveness and efficiency of the proposed model is done by training and testing datasets which are divided into 10 folds according to k-fold cross validation (Bengio & Grandvalet, 2004) . Cross validation is computationally intensive. It is used to develop the automated model using the training, testing and validation data. The dataset D is equally divided into k disjoint subsets. It uses (k-1) dataset for training purposes and a dataset for the purpose of testing by repeatedly training the algorithm k times.

The performances of the new model were evaluated using accuracy, area under curve, mean squared error, mean absolute error based on the confusion Matrix (Ouchicha et al.; Bradley, 1997) .

The confusion matrix includes four parts, true positive (TP), false positive (FP), false negative (FN), and true negative (TN).

The accuracy (ACC) is defined as correct predictions split by the total number of input data in testing and training sets.

Mean absolute error (Ismael & Şengür) represents the disparity between original scores and predicted scores. It judges how removed the predictions are from the true output. Mean squared error (MSE) is the average of the square disparity between the original scores and predicted scores.

The area under curve (AUC) refers to the area under the receiver operating characteristic curve (ROC) of FP rate versus TP rate at various points from 0 to 1 (Powers, 2011) .

The obtained results indicate that our proposed CycleGAN-Inception can accurately detect positive COVID-19 cases and differentiate between COVID-19 andnon-COVID-19 cases using radiographic images. The sample X-ray images generated by CycleGAN: (a) a sample of real and generated images from class non-covid19 to covid19. (b) a sample of real and generated images from class covid19 to non-covid19 is shown in Figure 8 . The ACC, AUC, MSE and MAE obtained by the proposed CycleGAN-Inception V3 model with k-fold cross-validation (k=10) method is displayed in Table 1 . The proposed model is trained and tested with 50 epochs. We have obtained an average accuracy of 94.2% and AUC of 92.2%. In Figure 9 , the TP, TN, FP, and FN of the test set have been calculated and the average number of TP, FN, TN, FP images for ten runs are displayed in the figure. From the total number of 9,544 images of the selected original dataset 954 images have been selected as test dataset based on the K-fold cross (K=10) validation. The developed model has compared with other transfer learning models namely Inception V3, VGG16 and finetuned Inception V3 using the same database. The results displayed in Table 2 demonstrate that our model outperformed the previous baseline models. The ROC curves obtained for four models including the proposed CycleGAN-Inception, Inception V3, Finetuned Inception V3, and VGG16 are shown in Fig. 10 .

The comparison of our technique with the gold-standard approach for automated detection of COVID-19 from chest X-ray radiological scans is shown in Table 3 . It illustrates that we have accomplished the highest accuracy and AUCs. The advantages of our method are as follows:

1. Proposed novel augmentation method. Hence, can be employed for smaller database also.

2. Obtained highest classification performance (Accuracy = 94.2% and AUC = 92.2%). 3. Developed model is accurate and robust as we have employed ten-fold cross-validation strategy. 4. Generated system is simple and faster as we have used transfer learning method.

The limitation of our method is that we have used only one public database. Our model needs to be tested with more diverse database. 

The aim of this paper was to develop a novel robust AI algorithm to detect COVID-19 using x-ray images automatically. Our proposed approach is simple, robust and more accurate. For doing this matter, we divided the model in two sections. In the first phase we applied CycleGAN technology for COVID-19 images which was the first time applied for COVID-19 X-ray image to translate COVID-19 X-ray image to normal once and vice versa. In the second phase we developed new deep learning model to classify the images in two normal and problematic classes. Therefore, we identified a new COVID-19 diagnosis system using deep learning technique and new framework was developed by integrating CycleGAN and finetuned Inception V3 X-ray transfer learning model. The proposed algorithm has trained, tested, and evaluated using the "Extensive COVID-19 Xray Chest Images" Dataset. A CycleGAN is applied to the proposed model as an unsupervised technique for data augmentation. The pre-trained Inception V3 deep convolutional network is modified by removing fully connected layer and adding two new fully connecting layers. The whole process is trained and evaluated. The obtained results demonstrate the effectiveness of the algorithm with MSE = 0.27, MAE = 0.16,

Tensorflow: A system for large-scale machine learning

Covid-caps: A capsule network-based framework for identification of covid-19 cases from x-ray images

Data augmentation generative adversarial networks

Covid-19: automatic detection from x-ray images utilizing transfer learning with convolutional neural networks

Enhanced deep learning algorithm development to detect pain intensity from facial expression images

The modeling of human facial pain intensity based on Temporal Convolutional Networks trained with video frames in HSV color space

Automatic COVID-19 Detection Using Exemplar Hybrid Deep Features with X-ray Images

No unbiased estimator of the variance of k-fold cross-validation

Gan augmentation: Augmenting training data using generative adversarial networks

Covid-19 image data collection: Prospective predictions are the future

Unsupervised monocular depth estimation with left-right consistency

Generative adversarial networks

Covidx-net: A framework of deep learning classifiers to diagnose covid-19 in xray images

Deep learning approaches for COVID-19 detection based on chest X-ray images

Analysis of transfer learning for deep neural network based plant classification models. Computers and electronics in agriculture

Introduction to keras

Coronet: A deep neural network for detection and diagnosis of COVID-19 from chest xray images

COVID-19 Detection in Chest X-Ray Images using a New Channel Boosted CNN

Artificial intelligence distinguishes COVID-19 from community acquired pneumonia on chest CT

Automated detection of COVID-19 cases using deep neural networks with X-ray images

A survey on transfer learning

Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation

Data augmentation using generative adversarial networks (CycleGAN) to improve generalizability in CT segmentation tasks

Python: a programming language for software integration and development

Detection of coronavirus disease (covid-19) based on deep features

Fusion of convolution neural network, support vector machine and Sobel filter for accurate detection of COVID-19 patients using X-ray images

Automated detection and forecasting of covid-19 using deep learning techniques: A review

Learning from simulated and unsupervised images through adversarial training

Rethinking the inception architecture for computer vision

Extensive COVID-19 X-Ray and CT Chest Images Dataset

Covid-net: A tailored deep convolutional neural network design for detection of covid-19 cases from chest x-ray images

Covid-19 screening on chest x-ray images using deep learning based anomaly detection

MIDCAN: A multiple input deep convolutional attention network for Covid-19 diagnosis based on chest CT and chest X-ray

Unpaired image-toimage translation using cycle-consistent adversarial networks

AUC = 92.2% and accuracy = 94.2%. Future work could validate the proposed model with more diverse big datasets and evaluate the performance in COVID-19 diagnosis. In future we may plan to explore the possibility of using our developed model to diagnose pulmonary edema, asthma, pericarditis, heart failure and pneumonia in addition to COVID-19 using chest x-ray images. Our new model can be used to diagnose other diseases using other imaging modalities as well.

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.