key: cord-0736335-4nyisjrm
authors: Shah, Pir Masoom; Ullah, Hamid; Ullah, Rahim; Shah, Dilawar; Wang, Yulin; Islam, Saif ul; Gani, Abdullah; Rodrigues, Joel J. P. C.
title: DC‐GAN‐based synthetic X‐ray images augmentation for increasing the performance of EfficientNet for COVID‐19 detection
date: 2021-10-19
journal: Expert Syst
DOI: 10.1111/exsy.12823
sha: 2f2dd4b946b755dfd13c2ea2732544bae5cb757d
doc_id: 736335
cord_uid: 4nyisjrm

Currently, many deep learning models are being used to classify COVID‐19 and normal cases from chest X‐rays. However, the available data (X‐rays) for COVID‐19 is limited to train a robust deep‐learning model. Researchers have used data augmentation techniques to tackle this issue by increasing the numbers of samples through flipping, translation, and rotation. However, by adopting this strategy, the model compromises for the learning of high‐dimensional features for a given problem. Hence, there are high chances of overfitting. In this paper, we used deep‐convolutional generative adversarial networks algorithm to address this issue, which generates synthetic images for all the classes (Normal, Pneumonia, and COVID‐19). To validate whether the generated images are accurate, we used the k‐mean clustering technique with three clusters (Normal, Pneumonia, and COVID‐19). We only selected the X‐ray images classified in the correct clusters for training. In this way, we formed a synthetic dataset with three classes. The generated dataset was then fed to The EfficientNetB4 for training. The experiments achieved promising results of 95% in terms of area under the curve (AUC). To validate that our network has learned discriminated features associated with lung in the X‐rays, we used the Grad‐CAM technique to visualize the underlying pattern, which leads the network to its final decision.

. The pandemic is spreading throughout the world at a very high speed, which has never been experienced in any infectious disease.

The WHO suggests social distancing as an effective technique to control the spread of the virus. Accurate and useful screening and testing is an essential step in this way so that the infected person may get proper treatment and may be isolated to halt the spread of the virus. Stat of the art method for COVID-19 detection and measurement of antibodies caused by infection are serology and reverse transcription-polymer chain reaction (RT-PCR) (W. . Another method is nucleic acid test (NAT) (D. . It is very much challenging to detect COVID-19 using testing kits because of their limited availability. Moreover, these tests take some time (few hours to a few days) to generate results that make it quite difficult, tedious, and time-consuming. Apart from these, the results are prone to errors, and the false positive rate leads to dissatisfaction. A fast, reliable, accurate, and consistent testing technique is required urgently to satisfy the situation's need in this direction.

Researchers in artificial intelligence (AI) have turned their focus towards the identification of novel coronavirus from images like computed tomography (CT-scans) and X-ray images using medical imaging techniques (Ng et al., 2020; Xu et al., 2020) . The use of chest radiography in the epidemic area for initial screening of COVID-19 is proposed recently (Ai et al., 2020) . Hence, screening by radiographic images can be used to substitute the NAT and PCR methods with their better sensitivity in some situations. The study (Yasin & Gouda, 2020) claims that chest X-rays met laboratory measures for the diagnosis of novel COVID-19. The rate of abnormal X-rays images in different hospitals increases during the pick of COVID-19 around the globe. Therefore, the researchers came to the conclusion (Stephens, 2020) that chest X-rays can be used for diagnosis of COVID-19. Detection with X-rays technique comes with several advantages over the PCR testing, like availability, timesaving, and so forth. However, this requires enough radiologists to engage the increasing population of COVID-19 patients, which may not be possible in the current era. Therefore, there is a need for a Computer Aided Diagnosis (CAD) base system that can automatically interpret x-rays or assist radiologists in decision-making.

Limited, annotated data and privacy are the major concerns in the medical research. On the other hand, deep-learning models have revolutionized many domains by achieving human-level accuracy on image classification tasks. In the context of COVID-19, we are dealing with a limited amount of data. Different deep learning models have already been applied to the COVID-19 with the limitation for classification. However, an effective model can only be obtained if the model is trained on a rich dataset. To tackle this issue of limited data in the context of COVID-19. Waheed et al. (Waheed et al., 2020) proposed a model; they called it covidgan, which generates synthetic x-ray images for COVID-19. However, the generated x-ray images were not validated. Therefore, in this paper, we propose a framework with which we generate synthetic images for the classes Normal, and Pneumonia) . We used the k-mean clustering algorithm to validate all the synthetic images. Our whole scheme is based on DC-GAN, K-mean clustering, and EfficientNetB4 algorithms. DC-GAN generates synthetic images for all the classes, and then the k-mean clustering algorithm is applied to the combination of all the classes to validate and annotate the images. At last, the EfficientNetB4 is be applied for the classification propose.To ensure that the classified results are accurate. We used CAM-Gard to visualize the network decision on the image.

Radiologists require an assistant tool to manage the growing population of COVID-19 patients. The development of such a tool can only be possible to have a huge enough dataset for training a deep learning model. In this regard, an attempt is carried out in this study with the following contributions:

• This paper extends the COVID-19 dataset using DC-GAN.

• The validation and annotation of the generated dataset is carried out in this study.

• This work provides the facilitation to train a robust model which can be used as a radiologist tool.

• Finally, the learned features of CNN are visualized and explored.

The rest of the paper is organized as follows: next section presents related work and Section 3 provides the details on the methodology. Section 4 demonstrate the obtained results and provides the discussion. Finally, Section 5 concludes the paper. Brunese et al. in (Brunese et al., 2020) worked on the detection of COVID-19 pneumonia from X-ray images using deep learning. According to the authors, a dataset of about 6523 X-ray images has been used to find the disease's positive cases. The process is divided into three stages. In the first stage, pneumonia is detected in the X-ray images. In the second stage, common pneumonia is differentiated from and COVID-19, while in the third stage, the infected area in the X-ray image is localized. Experiments are carried out using a deep learning algorithm based on VGG-16 by exploiting transfer learning.

In response to the inefficiency, inaccuracy, and unavailability of PCR machine for detecting COVID-19, Chowdhury et al. in (Chowdhury et al., 2020 ) experimented on relatively a big dataset of chest X-ray to identify COVID-19 cases. According to the author, the process is performed in two folds; the first model determines COVID-19 and normal X-ray image. The second model was trained to differentiate between viral pneumonia and COVID-19 pneumonia. The authors in this paper have focused on collecting many datasets used in different papers at Kaggle to make a reasonably big dataset for deep learning. Different variants of CNN models are used in experiments with and without augmentation using transfer learning. About 99% accuracy has been achieved in these experiments.

The early detection of COVID-19 is essential for the timed isolation of patient may stop the spreading of the virus from further spreading. In practice, methods are slow and costly. Therefore, there is a need for automatic detection. Detection of COVID-19 from X-ray images has been performed by Apostolopoulos et al. in (Apostolopoulos & Mpesiana, 2020) . They utilized two datasets with 1427 images and 1442 images. These datasets are collected from publically available repositories. Accuracy, sensitivity and specificity of the system using deep learning with transfer learning are 96%, 98%, and 96%, respectively. According to the authors, the detection of COVID-19 via X-ray is a useful addition to the traditional testing methods. It can help in maximizing the speed, efficiency, and accuracy of the tests performed in conventional ways.

Ozturk et al. in (Ozturk et al., 2020) , elaborated the importance of the early recovery of the COVID-19 positive patients and then discussed the methods for the detection of the said disease. The detection of COVID-19 in patients through CT and X-ray has been discussed in details and it is found that these are useful for timed detection. According to this paper, the detection is first performed in binary form that is COVID versus non-COVID. In the second approach, the detection is a multiclass classification which is COVID versus Non-COVID versus pneumonia. They have used a dataset of 125 X-ray images for their experiments and have obtained the accuracy of 98% for binary while 87% for the detection of COVID-19 disease respectively.

RT-PCR is expensive and slow method for covid19 detection, however, fortunately, X-ray of COVID-19 infected patients have certain pattern through which it can also be detected. COVID-19 detection from X-ray is difficult by normal eye, but deep learning algorithms can diagnose such pattern accurately. EfficientNet family of deep learning is used with a large data set of 13,569 X-ray images of three classes that is healthy, non-COVID-19 pneumonia, and COVID-19 patients. The system is evaluated by 231 images of the said classes. The overall accuracy of the system is 93% for COVID-19, while sensitivity is 96%. According to the authors in (Luz et al., 2020) , still there is need of a big data set for the evaluation purposes before applying in practice.

The CT and clinical features were examined for pregnant women and children by Lui in Chine in . Authors in this papers have InceptionV3 are used in experiments. Overall, 100% accuracy is achieved for different factors. According to Salman et al in (Salman et al., 2020) , this will help radiologists to release the pressure in peak time, improve diagnose on time, help in isolation and treatment on time, thus it will help in the control of COVID-19 pandemic.

Detection of COVID-19 based on deep features is presented by Sethy et al in (Sethy et al., 2020) . This papers has used a different approach. Maghdid et al. in (Maghdid et al., 2020) has published diagnosing of COVID-19 pneumonia from X-ray images and CT scan by deep learning and transfer learning techniques. The importance of AI in the detection of COVID-19 is discussed and stated that it needs a pre-processed dataset. Hence, the main focus of the paper is to develop a dataset for AI algorithms. A dataset is generated from multiple sources first, afterward, simple CNN is used to perform experiments on the dataset prepared. Furthermore, to accelerate and evaluate the accuracy, a modified version, and pre-trained CNN has been used.

Usefulness of chest X-ray images for diagnosing COVID-19 has been explored by Hall et al in (Hall et al., 2020) . According to the authors, testing of COVID-19 on RT-PCR is not available as needed, its false negative rate is up to 30% and it takes some time. As the result is required in a very short time with high accuracy so that the spread of virus may be stopped on time, therefore there is need of another testing system. X-ray images of chest has some patterns which can help in the diagnose of the said disease on time and X-ray machines are widely available. In 135 chest X-ray images with COVID-19 positive and 320 chest X-rays of pneumonia and viral bacteria are used in experiments. ResNet50 has been used for experiments in 10-fold cross validation manner and 89.2% accuracy has been achieved.

Radiologists require an assistant tool to manage the growing population of COVID-19 patients. The development of such a tool can only be possible to have a huge enough dataset for training a deep learning model.

The framework of this work consists of several phases. In the first phase, DC-GAN is used to generate synthetic images. DC-GANs were trained for all classes (COVID-19, Pneumonia, and Normal) separately. In the next phase, generated images for all the classes were merged with the original data to validate the correctness of the generated images. Three clusters were formed using K-mean algorithm. For 3-classes, the value of k is set to 3. In this process, 92% of the images were correctly placed in their respective clusters and the rest 8% were discarded.

In the last phase, EffecientnetB4 with some enhancements in the last layers was used for classification. As the data classes are created on assumptions basis, therefore, validation of the classifier is needed. Attenuation map was used to visualize the decision confidence on each image.

The high intention of attenuation map on lung indicates that our network has learned the relevant features. The illustration of this methodology can be seen in Figure 1 .

Since we discussed earlier in the context of COVID-19, we have minimal data available. Therefore, we approach two different datasets, Joseph Paul Cohen (Cohen et al., 2020) and Kaggle repository (Mooney, 2018) . To avoid the class imbalance issue, we acquired an equal number of samples for each class. Since the class COVID-19 consisted of the minimal number of 141 samples only, the rest of the two classes were also set to 141 samples each. The combined dataset consists of three classes (COVID-19, Pneumonia, and Normal) with 141 Â 3 = 423 instances. Furthermore, we divided the dataset into two subsets, train and test, with the ratio of 90% and 10%, respectively. The test set was kept completely separate from the training DC-GAN as well as EffiecentNET. 

In GANs, two networks are trained at the same time in which first focuses on the generation of images while the second on discrimination (Yi et al., 2019) . It is gaining importance in academia and industry because of its ineffective image generation and counteracting domain shift. GANs have obtained good performance in many image generation tasks such as super-resolution (Ledig et al., 2017) , text-to-image synthesis (Yang et al., 2017) , and image-to-image translation (Zhu et al., 2017) . According to rules, the patient's consent is mandatory when the diagnostic images are to be published in public domains (Clinical Practice Committee, 2000) . GANs are used widely to generate synthesis images that avoid privacy and provide sufficient images for analysis. Lack of experts is another challenge to annotate medical images in supervised learning. However, some efforts are made among many healthcare agencies to build large publically available datasets. Such as, The cancer imaging archives, Biobank, radiologist society of North America, and national biomedical imaging archive, the issue is still a big challenge. Typically, training samples can be enlarged by rotation, flipping, scaling, and elastic deformation (Clinical Practice Committee, 2000) . However, these do not provide sufficient variations which can be found in true samples.

GANs offer data samples more synthetic and have similar attributes to actual data. It has been used in several papers for augmenting images dataset with good performance. The GANs' first invariant Deep Convolutional GAN (DC-GAN) was initially proposed by Radford et al. (Radford et al., 2015) where both generative and discriminator are deep-CNN. DC-GAN is the stable version for training and modification of GAN proposed by Goodfellow et al. (Goodfellow et al., 2014) , which is foundation for many recent GANs (Odena et al., 2017) , (Yeh et al., 2017) , (Salimans et al., 2016) . The model has two neural networks where both are trained at the same time. Figure 2 shows an illustration of a typical GAN with both the networks. The first one is discriminator (known as D), which discriminates between real and fake images. It takes x as input and gives D(x). The second network is the generator (Known as G) which synthesizes the images declared as real with high probability by D. G takes input images z 1 ð Þ, …,z m ð Þ from simple distribution Pz, which is a uniform distribution and maps G z ð Þ to the image space of Pg. The main purpose of G is to succeed in getting Pg ¼ Pdata. Networks are trained in such a way that the following loss function may be optimized.

The training is such that the discriminator maximize D(x) for samples with x $ Pdata and minimize D(x)Â!$Pdata. The generator creates samples G (z) to for D in such a way that D will consider the images as real images. Based on this training, the generator improves its ability to create more realistic images while the discriminator enhances its ability to separate real images from synthesized image samples.

The network input is a vector of 100 random numbers obtained from a uniform distribution and gives a 64 Â 64Â 1 image as shown in Figure 3 . The network (Radford et al., 2015) has a fully connected layer remodelled to 4Â 4Â 1024 and four fractionally-strided convolutional to make a sample image 5Â 5 kernel size. The fractional strided convolutional (deconvolution) expands pixels by adding a zero pixel in between, which enlarges the inputted image. Each layer has batch normalization except the output layer, which stabilizes the GAN network and avoids the generation collapsing to a single point (Ioffe & Szegedy, 2015) . ReLU is the activation function in all the layers except the last layer, which has a tanh activation function.

It has a typical CNN that accepts input 64 Â 64 Â 1 (X-ray) and makes a binary decision: is the X-ray real or fake? The network has four convolutional layers with a kernel size of 5 Â 5 and a fully connected layer. Spatial dimensionality is reduced by applying strided convolution instead of pooling layers. Each layer has used Batch-normalization except input and output layers. Each layer has leaky ReLU activation function f(x)=max (x, leakÂx) while the last layer uses Sigmoid function for the likelihood probability (0,1) score of the image sample.

F I G U R E 2 A generative adversarial network illustration

Our real images dataset is taken from different sources. Therefore, their resolutions were different. To decrease the GPU processing, we resampled the images to 64 Â 64 pixels. We then trained the network for each class separately. However, the parameter setting is kept the same for all the training experiments. We performed all the experiments with 500 epochs, whereas the DC-GAN started producing a realistic X-ray image after 50th during the all class's during training period. Figure 4 shows the sample of synthetically generated X-ray images.

To validate the generated images by DC-GAN, we used the K-mean clustering algorithm. Since we were dealing with three classes, therfore we set the k ¼ 3. This algorithm is trained on the data with the ratio of 70:30 synthetic data and original data, respectively and in contrast, evaluated on 30:70 synthetic data and original data, respectively. To evaluate the performance of the clustering algorithm, we used accuracy, homogeneity, and inertia. We performed some other experiments with the k setting by decreasing the value of k and checked the results. However, by doing this, our accuracy, homogeneity, and inertia are compromised. Figure 5 shows the initial experiments with different k settings. It can be seen in the Figure 5 that higher results are obtained when the value of k is set to 3.

Generally, any variant of CNN can be easily fitted in our framework, but we used the EfficentNetB4. EffieicentNet has the advantage of highly 

In this experiment, we use EfferententB4. We started the model training from scratch on the available dataset (real images). This experiment training set comprises 296 number of instances, whereas the test set has 127 instances. The training process was continued up to 100 epochs. However, it needs to mention here that the number of epochs for all the experiments has remained the same (i-e 100 epochs). In this experiment, we recorded AUC of 89%.

In this experiment, we used the same number of instances for training and testing; however, we fine-tune our previous EfficientnetB4 model. By utilizing the technique, the model convergence faster than the previous experiment. The AUC on the test set is recorded 92%.

The number of training samples is increased by adding the synthetic images. The ratio of real and synthetic images has remained the same as 1:1.

Moreover used followed the strategy of transfer learning here also. By training the model on synthetic images, the AUC on the test-set is increased by 3% than experiment second.

F I G U R E 5 Experimental results of clustering with K means SHAH ET AL. 7 of 13 4.4 | Experiment 4

The experiment fourth was performed with the same setting as experiment third. However, we increased the training set by 1:2 real and synthetic images, respectively. This method affects the results positively of 96%, in terms of AUC.

The 2:1 respectively for the 4th experiment, which is shown in Table 2 row 4. All the experiments demonstrate that the AUC increases when transfer learning and synthetic images are added in training, confirming that the synthetic images have significant insights related to all the classes.

Developing a more robust understanding of deep learning models is an essential field of study. Deep convolutional neural networks are also known as black-box models due to the lack of information about their internal actions. In order to create explainable deep learning models, several researchers have recently proposed methods for using class activation maps (CAMs) that display deep-learning predictions to assist human experts in developing understandable deep learning models. A more descriptive input picture relating to the final model prediction for each class is emphasized in the author's proposed methods for gradient-based CAM (i.e., Grad-CAM) production. The availability of such information, along 

There are still several flaws in this study, like GAN architecture and training may be tuned by applying better techniques. Likewise, we selected a tiny dataset due to time constraints. In the same way, there are certain obstacles to get and add more labeled data into the learning process of F I G U R E 7 COVID-19: True-positive samples from the test set along with Grad-CAM SHAH ET AL. 9 of 13

Correlation of chest CT and RT-PCR testing for coronavirus disease 2019 (COVID-19) in China: A report of 1014 cases

Covid-19: automatic detection from x-ray images utilizing transfer learning with convolutional neural networks

Explainable deep learning for pulmonary disease and coronavirus covid-19 detection from xrays

Can ai help in screening viral and covid-19 pneumonia?

Coronavirus covid-19 global cases by the center for systems science and engineering (csse)

Generative adversarial networks. arXiv

Batch normalization: Accelerating deep network training by reducing internal covariate shift. International Conference on Machine Learning

Photo-realistic single image super-resolution using a generative adversarial network

Clinical and ct imaging features of the covid-19 pneumonia: Focus on pregnant women and children

Towards an effective and efficient deep learning model for covid-19 patterns detection in x-ray images

Diagnosing covid-19 pneumonia from x-ray and ct images using deep learning and transfer learning algorithms

Imaging profile of the covid-19 infection: Radiologic findings and literature review

Conditional image synthesis with auxiliary classifier gans. International conference on machine learning

Automated detection of covid-19 cases using deep neural networks with x-ray images

Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv, 6434. Salimans

Artificial intelligence for the detection of COVID-19 pneumonia on chest CT using multinational datasets

Detection of coronavirus disease (covid-19) based on deep features and support vector machine

Deep GRU-CNN model for COVID-19 detection from chest X-rays data

Radiologists find chest x-rays highly predictive of covid-19

Covidgan: Data augmentation using auxiliary classifier Gan for improved covid-19 detection

Clinicalcharacteristicsof138hospitalizedpatientswith2019novelcoronavirus-infected pneumonia in

Detection of sars-cov-2 in different types of clinical specimens

speeches/detail/who-director-general-s-opening-remarks-at-the-media-briefing-on-covid

A deep learning system to screen novel coronavirus disease

Automatic liver segmentation using an adversarial image-to-image network

Chest x-ray findings monitoring covid-19 disease course and severity

Semantic image inpainting with deep generative models

Generative adversarial network in medical imaging: A review

Unpaired image-to-image translation using cycle-consistent adversarial networks

His research interests include medicine, bioinformatics, machine learning, computer vision, ad-hoc networking. He has worked in industry for small startups, large corporations, research labs, as well as been involved in projects sponsored by KP Police

Hamid Ullah is working as lecturer in the Department of computer science

Rahim Ullah is working as lecturer in Govt DIR KP College. His research area is Neural networks for Medical images

He did his MS from the University of Agriculture Peshawar. He started his career as faculty member at Islamia College, Peshawar in 1998

Later on, the campus was upgraded to a fledged University named as Bacha Khan University, Charsadda. Since then he is working as Head of the department at Computer Science Department. His research interest includes medical data mining

Yulin Wang is a full professor and PhD supervisor in International School of Software, Wuhan University, China. He got PhD degree

He got his master and bachelor degree in 1990 and 1987 respectively from XiDian University, and Huazhong University of Science and Technology (HUST), both in China. His research interests include digital rights management, digital watermarking, multimedia and network security, and signal processing

He is currently an Assistant Professor with the Department of Computer Science, Institute of Space Technology, Islamabad. He has been part of the European Union-funded research projects during his Ph.D. studies. He was a focal person of a research team at COMSATS University, working in the O2 Project in collaboration with CERN, Switzerland. His research interests include resource and energy management in large-scale distributed systems

He is currently a Professor and the Dean Faculty of computing and informatics, University Malaysia Sabah. He is also an Honory Professor with the Department of

He received the Academic Title of Aggregated Professor in informatics engineering from UBI, the Habilitation in computer science and engineering from the University of Haute Alsace, France, a PhD degree in informatics engineering and an MSc degree from the UBI, and a five-year BSc degree (licentiate) in informatics engineering from the University of Coimbra, Portugal. His main research interests include IoT and sensor networks, e-health technologies vehicular communications, and mobile and ubiquitous computing. Prof. Rodrigues is the leader of the Next Generation Networks and Applications research group (CNPq)

He has authored or coauthored over 850 papers in refereed international journals and conferences, 3 books, 2 patents, and 1 ITU-T Recommendation. He had been awarded several Outstanding Leadership and Outstanding Service Awards by IEEE Communications Society and several best papers awards

Rodrigues is a licensed professional engineer (as senior member), member of the Internet Society, a senior member of ACM, and an IEEE Fellow

-based synthetic X-ray images augmentation for increasing the performance of EfficientNet for COVID-19 detection. Expert Systems, e12823

This work is partially supported by FCT/MCTES through national funds and when applicable co-funded EU funds under the project UIDB/50008/2020; and by Brazilian National Council for Scientific and Technological Development (CNPq) via Grant No. 313036/2020-9.

The authors declare that there is no conflict of interest regarding the publication of research work carried out in this paper and about the order of the authors in the manuscript.

The data that support the findings of this study are available from the corresponding author upon reasonable request. Saif ul Islam https://orcid.org/0000-0002-9546-4195