key: cord-0682954-i2ntme9c
authors: Acar, Erdi; Şahin, Engin; Yılmaz, İhsan
title: Improving effectiveness of different deep learning-based models for detecting COVID-19 from computed tomography (CT) images
date: 2021-07-29
journal: Neural Comput Appl
DOI: 10.1007/s00521-021-06344-5
sha: 1e0a7c85d0152368e6d3f70c1e3f2926414ae7dd
doc_id: 682954
cord_uid: i2ntme9c

COVID-19 has caused a pandemic crisis that threatens the world in many areas, especially in public health. For the diagnosis of COVID-19, computed tomography has a prognostic role in the early diagnosis of COVID-19 as it provides both rapid and accurate results. This is crucial to assist clinicians in making decisions for rapid isolation and appropriate patient treatment. Therefore, many researchers have shown that the accuracy of COVID-19 patient detection from chest CT images using various deep learning systems is extremely optimistic. Deep learning networks such as convolutional neural networks (CNNs) require substantial training data. One of the biggest problems for researchers is accessing a significant amount of training data. In this work, we combine methods such as segmentation, data augmentation and generative adversarial network (GAN) to increase the effectiveness of deep learning models. We propose a method that generates synthetic chest CT images using the GAN method from a limited number of CT images. We test the performance of experiments (with and without GAN) on internal and external dataset. When the CNN is trained on real images and synthetic images, a slight increase in accuracy and other results are observed in the internal dataset, but between [Formula: see text] and [Formula: see text] in the external dataset. It is promising according to the performance results that the proposed method will accelerate the detection of COVID-19 and lead to more robust systems.

COVID-19 is an infectious disease with a high mortality rate caused by the mutant SARS-CoV-2 virus. Following the identification of the first case in Wuhan, China, in December 2019, it quickly spread all over the world and was declared a pandemic by WHO on March 11, 2020. COVID-19 has caused major public health problems in the international community due to its rapid spread all over the world [1] . It is extremely important to be able to diagnose COVID-19 in an infected patient during the pandemic process. Although polymerase chain reaction (PCR) testing is the standard for confirming COVID-19 positive patients, medical imaging such as X-ray and non-contrast computed tomography (CT) plays an important role in the detection of COVID-19. Since COVID-19 can be detected in the early stages with the presence of lung ground-glass opacities in CT images, which are clearer and more accurate than X-ray images, the diagnosis of COVID-19 from CT will help clinicians make quick decisions for rapid isolation and appropriate patient treatment [2] .

Recently, computer vision has led to extraordinary developments with the advancement of artificial intelligence technology and especially the development of convolutional neural networks (CNN). It is widely used in the medical field and provides support for medical diagnosis [3] [4] [5] . Additionally, due to the heavy workload of large numbers of infected patients and healthcare professionals during the pandemic crisis, the AI-based computer-aided system could speed up the diagnostic process. These systems can provide effective diagnosis of the disease in a shorter time in cases such as radiologist insufficiency and deficiency. It can also reduce the need for human surveillance and identify details invisible to the human eye. In this context, many deep learning models have also begun to be developed for the diagnosis of COVID-19.

There is currently a great deal of interest in data-driven approaches to diagnosing COVID-19. Many researchers have used deep convolutional neural networks (CNN) such as VGG, ResNet, Inception, Xception and DenseNet, with well-known and proven performance, to diagnose COVID-19. However, deep learning networks such as convolutional neural networks require large amounts of training data. In addition, because CNNs have a large number of parameters, overfitting can easily occur in small datasets. However, medical imaging data are scarce, expensive and fraught with legal concerns over patient privacy, so datasets are small in size. A quick and easy method, data augmentation, can be used to overcome this problem [6] . The dataset is artificially expanded using techniques such as image rotation, transformation, scaling, brightness or contrast enhancement. However, it cannot be said that completely different images are produced since the same data are displayed differently with this method. In addition, the dataset can be enlarged by generating unique data by producing synthetic data with GAN, which is an advanced technique [7, 8] .

The main contributions of this article can be summarized as follows:

• It is difficult to identify COVID-19 as findings on CT images differ in both position and shape in different patients due to infections caused by COVID- 19 . In addition, we present a method for generating synthetic CT images by developing a GAN-based model to overcome the problem of overfitting in CNNs. In addition, we demonstrate the effectiveness of GANgenerated synthetic images in improving CNN's generalization ability for COVID-19 detection in the context of performance criteria in internal and external datasets. • We use a segmentation model based on ConvLSTMU-Net architecture and graph-cut image processing for segmentation of the relevant lung region from CT images. • We use data enlargement techniques such as random distortion, rotation, flip and zoom with an in-place / onthe-fly augmentation approach to increase the effectiveness of the model during model training.

The rest of the paper is organized as follows. In Sect. 2, brief information about the studies related to our study is given. The materials and methods used in the study are presented in Sect. 3. Section 4 presents the results of different analyzes for different deep learning algorithms in the proposed frameworks on internal and external datasets. The comparisons with the other related studies and the discussion of the results are given in Sect. 5. Section 6 presents the conclusion of this study.

Recently, many studies have been conducted to perform data augmentation on medical images. Zhoe et al. [9] developed the forward and backward GAN (F&BGAN) method to create synthetic images of lung nodules in lung cancer and used the VGG16 network for benign and malignant classification. Chuquicusma et al. [10] produced realistic lung nodules using DCGAN for the training of radiologists. Frid-Adar et al. [11] proposed a magnification scheme for advanced liver lesion classification based on a combination of standard image perturbation and synthetic liver lesion generation using GAN. Guibas et al. [12] proposed a two-stage pipeline for generating synthetic medical images from a pair of GANs, which was tested in practice on retinal fundi images. Shin et al. [13] proposed a method to generate synthetic abnormal MRI images with brain tumors by training a GAN using two publicly available datasets of brain MRI. Nowadays, there is intense interest in diagnosing COVID-19 using deep learning methods. Wu et al. [14] aimed to diagnose COVID-19 cases with the concept of multi-view fusion using the ResNet50 architecture. The datasets they used in their studies included 622 CT images collected from two different hospitals in China. They resized their images in the datasets to 256 Â 256. The system they developed achieved 76% accuracy, 81:1% sensitivity and 61:5% specificity. Xu et al. [15] collected the data they used in their study from 3 different hospitals in China. It achieved 86:7% accuracy, 81:5% sensitivity, 80:8% accuracy and 81:1% F1-score with pretrained ResNet18 architecture using 618 CT images. Jin et al. [ a system for detecting  COVID-19 using CNN architectures such as AlexNet,  VGG-16, VGG-19 , SqueezeNet, GoogleNet, MobileNetv2, ResNet18, ResNet50, ResNet101 and Xception. The dataset they use contains 1020 CT images in total. Among the 10 networks, ResNet101 and Xception performed better than the others according to their performance values. Chen et al. [19] used the U-Net?? segmentation model together with the ResNet50 model to diagnose COVID-19 using a very large and non-public dataset of 106 patients from the Renmin Hospital of Wuhan University. The results of COVID-19 classification are 95:2% accuracy, 100% sensitivity and 93:6% specificity. The U-Net and 3D CNN models are used for lung segmentation and diagnosing of COVID-19, respectively, in Ref. [20] . The results of the model are 90:7% sensitivity, 91:1% specificity and 95:9% AUC. Most of the previous works on COVID-19 are given in Refs. [21] [22] [23] .

We use two different datasets in our study:

1. The dataset, which was collected from different open sources [24] , contains 1607 COVID-19 and 1667 normal CT images. We randomly allocate 80% of the dataset for training and 20% for testing. We will refer to this dataset as the internal dataset throughout the study. 2. The external dataset in Ref. [25] contains 4001 COVID-19 and 9575 normal CT images. We use this dataset to evaluate the performance of the trained models.

Since the internal dataset is collected from different sources, it contains CT images with a resolution of min 220 Â 200 -max 623 Â 421. For these reasons, the images in the internal dataset were resized to 224 Â 224 resolution. In the external dataset, all CT images are 512 Â 512 high-resolution images. We also reduce the size of the CT images in this dataset to 224 Â 224 to be standard and perform central zooming.

Since distinctive information is in the lungs on CT images, we apply lung segmentation to obtain only the lung area. For segmentation, we applied a similar approach to Bidi-rectionalConvLSTMU-Net (BDCU-Net) [26] architecture. A total of 1667 normal CT data and masks in the internal dataset were used for training the lung segmentation architecture. We reserve 80% of the dataset for training and 20% of the dataset for validation. We initially started with 1e À 4 learning rate and training epochs of 30. We dynamically reduced the learning rate when there was no improvement in validation error for 4 consecutive training periods. When there is no improvement in validation error for 8 consecutive training epochs, it is taken as the criterion for early stopping of training. The Adam stochastic optimization algorithm, which was developed as a solution to the disappearing gradient problem, was used for parameter optimization. At the end of the 26th epoch, an early stop occurs, and accuracy: 0.9675 is obtained. The architecture of obtaining the mask images from the CT images by using lung segmentation model is given in Fig. 1 . The sample images and obtained mask images by using the lung segmentation model are given in Fig. 2 . The example images for the relevant region obtained as a result of Graphcut image processing [27] are given in Fig. 3 .

The GAN framework is a widely used modern method for generating synthetic data from many domains [28, 29] . Examples of data generation with GANs include text-toimage synthesis, superimage resolution, style transfer and symbolic music production [30] .

The working principle of GANs is simply; GAN is a min-max game between two hostile networks, generator G(z) and discriminator D(x). The generator tries to convert the random noise into observations that appear to be sampled from the original dataset, and the discriminator tries to guess whether an observation comes from the original dataset or is one of the generator's fakes. At the start of the process, generator G(z) takes a z-point (random noise) from a latent space and outputs noisy images and tries to fool the discriminator D(x). If the discriminator is D(x), the generator tries to distinguish whether the images from GðzÞ ¼ x are real or fake. The key of GANs lies in how we change the training of the two networks so that as the generator becomes more adept at deceiving the discriminator, the discriminator must adapt to maintain its ability to accurately identify which observations are fake. Thus, the generator tries to find new ways to fool the discriminator so that the loop continues between the two networks. The purpose of G(z) is to minimize the cost function V(D, G) and to maximize the D(x) by training both G(z) and D(x) at the same time:

The architecture of GAN applied in the study is given in Fig. 4 

Initially, the parameters of the discriminator network are set to non-trainable. The output of the generator network feeds the discriminator and the generator is updated through the discriminator, so the generator is stacked on the discriminator. The hyperparameter of LeakyReLu layers used in discriminator and generator networks is alpha ¼ 0:2, and momentum value of batch normalization layer used in generator is 0.8. The detailed designs of the discriminator and generator network used in the study are given in Fig. 5 .

80% of the internal dataset reserved for training iz used in GAN training, i.e., 1286 COVID-19 and 1333 normal CT images. Test data are not included in the training. The pixel values of the CT images are normalized from [0, 255] to ½À1; 1 as a preprocessing. In the training of both networks, Adam optimization algorithm with learning rate ¼ 0:0001 and momentum value beta ¼ 0:9 is used as hyperparameters. The least-squares function is used as the loss function. In addition, the number of training periods is determined as 40000 and the batch size is 64. The latent space dimension where the generator is fed, is taken as 100.

A total of 2314 normal and 2267 COVID-19 synthetic images are generated by GAN. The samples of the COVID-19 and normal synthetic images produced after the GAN training are given in Fig. 6 .

Deep CNN architectures pretrained on 9 well-known ImageNet [31] datasets were used in this study: VGG16, VGG19, Xception, ResNet50, ResNet50v2, Inceptionv3, InceptionResNetv2, DenseNet121 and DenseNet169. [32] . VGG16 is a convolutional neural network architecture named after the Visual Geometry Group of Oxford developed. VGG-16 is a convolutional neural network with a depth of 16 layers. The model loads a pretrained set of weights on ImageNet and reaches 90% accuracy on Ima-geNet, a dataset of more than 14 million images belonging to 1000 classes. VGG-19 is the version of VGG with a depth of 19 layers. VGG16 has 138, 357, 544 parameters while VGG19 has 143, 667, 240 parameters.

He et al. [33] developed the architecture called residual network (ResNet) to solve the vanishing/exploding gradient problem. ResNet is created by adding the blocks that feed the residual values to the next layers into the model. This is also called residual learning. ResNet-50 has 16 residual blocks. The main difference between ResNet50 and ResNet50v2 is now the arrangement of layers in blocks. ResNet50 and ResNet50v2 achieved 92:1% and 93% accuracy on the ImageNet dataset, respectively.

Inception [34] and InceptionResNet [35] are developed by Google. The concept of Inception represents Inception modules within the network. This is a simple and powerful architectural unit that also allows the model to learn parallel filters of different sizes and multiple scales. The main difference between Inception and InceptionResNet is that InceptionResNet uses residual layers. Inceptionv3 using the Inception modules has 23, 851, 784 while Incep-tionResNet has 55, 873, 736. Inceptionv3 and Incep-tionResNetv2 achieved 93:7% and 95:3% accuracy on the ImageNet dataset, respectively. Xception [36] is a deep convolutional neural network architecture that includes deeply separable convolutions and developed by Google researchers. Google presented an interpretation of the Initiation modules in convolutional neural networks as an intermediate step between regular convolution and deeply separable convolution processing. Xception has 22, 910, 480 parameters and achieved 94:5% accuracy on the ImageNet dataset.

DenseNet [37] is a type of convolutional neural network that uses dense connections between layers through dense blocks that connect all layers directly. Each layer receives additional inputs from all previous layers and transmits its own feature maps to all subsequent layers. The difference between DenseNet121 and DenseNet169 is the number of layers. DenseNet121 has 8, 062, 504 parameters while DenseNet169 has 14, 307, 880 parameters. DenseNet121 and DenseNet169 achieved 92:3% and 93:2% accuracy on the ImageNet dataset, respectively.

In this study, two types of experimental studies (with GAN and without GAN) are conducted using the real images from the internal dataset and incorporating synthetic images generated by the GAN into the internal dataset. Details of the dataset used in the two studies are given in Table 1 .

The 1667 normal class images in the internal dataset used in the training of models are the same as the images used in the BConvLSTM U-Net training. All networks in both frameworks, the pixel values of the CT images, were normalized from [0, 255] to [0, 1] as a preprocessing. In both experiments, Adam optimization algorithm with learning rate ¼ 0:001 and momentum value beta ¼ 0:9 was used as hyperparameter. Binary cross-entropy function is used as loss function. In addition, the number of training epochs is determined as 40 and the batch size was 16. The learning rate is reduced when there is no improvement in validation error for 5 consecutive training periods. It was taken as the criterion for early discontinuation of training when there is no improvement in validation error for 10 consecutive training periods. A fivefold cross-validation is performed and tested on a randomly selected 20% of the internal dataset and the external dataset.

In both experiments, pretrained deep CNN models VGG16, VGG19, Xception, ResNet50, ResNet50V2, DenseNet121, DenseNet169, InceptionV3 and Incep-tionResNetV2 were used and fine-tuned. Dense(512, relu), BatchNormalization (0.9), Dense(256, relu), Dense (1, Sigmoid) layers are used, respectively, in the fully connected layer for classification. The working principle of this study is given in Fig. 7 .

The performance of deep learning neural networks is generally directly proportional to the amount of data available. Data augmentation is referred to as a technique of artificially deriving data from existing training data through much more image processing such as rotating, panning, zooming, flipping [6] . The aim is to increase the generalization performance of the model while expanding the training dataset with new examples. Thus, the trained model constantly sees new, different versions of the input data, and the model can learn more robust features.

There are three types of data augmentation methods: first, expanding the existing dataset. The problem with this approach is that it does not fully increase the generalization ability of the model. The second is in-place/on-the-fly data augmentation. The network is trained to see new variations of the data in each epoch by using this type of data augmentation. This method increases the generalization ability of the model. Finally, a hybrid approach combines the first and the second approaches.

In our study, we use only in-place/on-the-fly augmentation approach in models trained with real data; in the second experiment, we increase the size of the dataset with synthetic data produced with GAN, and we adopt a hybrid approach with in-place/on-the-fly augmentation during the training of models. The applied data augmentation processes and sample images are given in Table 2 and Fig. 8 , respectively. Random distortion allows you to distort the image while maintaining elastically the image aspect ratio. The magnitude parameter determines how much it degrades. Flip left-right acts as a mirror on images. Rotation was used to rotate left and right a certain degree. Zoom was used to enlarge and reduce the image centrally. 

All training and testing processes are performed using AMD Ryzen 3970X CPU with 128GB RAM and Nvidia RTX 3080 GPU with 10GB memory. The Keras [38] deep learning library is used for processes. Estimation performances of the methods in this study are measured with metrics such as accuracy, precision, recall and F1-score. The confusion matrix used to explain the performance of the classification model consists of true positive (T P ), true negative (TN), false positive (FP) and false negatives (FN) .

Dividing the number of correctly classified cases into the total number of test images shows the accuracy and is calculated as follows.

where T P is the number of instances that correctly predicted, T P is the number of instances that incorrectly predicted, T N is the number of negative instances that correctly predicted, T N is the number of negative instances that incorrectly predicted. The recall is used to measure correctly classified COVID-19 cases. Recall is calculated as follows.

The percentage of correctly classified labels in truly positive patients is defined as the precision and is calculated as follows.

Specificity is the proportion of people who test negatively among those who actually do not have the disease and is defined as: Max factor=1.20

Specificity ¼

The F1-score is defined as the weighted average of precision and recall combining both precision and recall and is calculated as follows.

The same real images are used in the tests of the both experiments on both internal and external datasets. Information on test sets size is given in Table 3 . The comparison of performance results in internal and external datasets for both experiments is given in Tables 4 and 5 , respectively. Confusion values and performance results for each model are given in Section Appendix.

In this paper, two experiments with and without GAN were tested on two different datasets (655 internal and 14545 external) with different deep learning methods. Table 4 analyzes the performance results of models with and without GAN on internal dataset. We clearly see that in all deep learning models with GAN, a slightly better performance ratio is achieved in all of the accuracy, precision, recall, specificity and F1-score values compared to the models without GAN. The best detection accuracy rate obtained in the InceptionV3 model with GAN on the internal dataset is 99:51% and the F1-score rate is 99:5%. Table 5 analyzes the performance results of models with and without GAN on different external dataset that the system has never seen before. We clearly see that in all deep learning models with GAN, a better performance rate of 3% to 9% is achieved in all of the accuracy, precision, recall, specificity and F1-score values compared to the models without GAN. The best detection accuracy rate obtained in the InceptionV3 model with GAN on the external dataset is 94:98% and the F1-score rate is 91:73%.

The results of this study reveal that synthetic augments produced with GAN make an important contribution to the improvement of generalization performance, especially in external dataset.

In this research, we developed a GAN-based model to overcome the overfitting problem in CNNs and improve their generalization performance and proposed a model in which the dataset can be artificially amplified with synthetic CT images.

An internal dataset containing 3274 CT images was used for training two different experiments (with and without GAN). The improvement in the generalization ability of CNN models was investigated using synthetic data augmentation technique with GAN on this limited dataset. Both experiments were tested on an external dataset. This study uses two classes (COVID-19 and normal) for classification. Synthetic data augmentation with GAN expands the dataset, providing more variability. When the CNN is trained on real images and synthetic images, a slight increase in accuracy and the other results is observed in the internal dataset, but between 3% and 9% in the external dataset. Performance results show that synthetic images produced with GAN make a significant contribution to the detection of COVID-19, especially in external dataset that the system has not seen before.

In conclusion, we proposed a method to increase the accuracy of COVID-19 detection with minimal dataset by producing synthetic images of CT images. The proposed method will improve the generalization performance of CNNs and lead to more robust systems. Acknowledgements We would like to thank referees for valuable suggestions.

Author Contributions All authors contributed to the study conception and design. Erdi Acar, Engin Ş ahin and İhsan Yılmaz contributed to conceptualization; Erdi Acar contributed to data curation; Erdi Acar and Engin Ş ahin contributed to formal analysis; Erdi Acar, Engin Ş ahin and İhsan Yılmaz contributed to investigation; Erdi Acar and İhsan Yılmaz contributed to methodology; Erdi Acar contributed to resources and software; İhsan Yılmaz supervised the study; Engin Ş ahin contributed to visualization, writing-original draft preparation 

Conflicts of interest The authors declare that they have no conflict of interest.

Funding Not applicable.

Estimates of the severity of coronavirus disease 2019: a model-based analysis

Synergistic learning of lung lobe segmentation and hierarchical multi-instance classification for automated severity assessment of COVID-19 in CT images

Deep learning-enabled medical computer vision

Deep learning and medical diagnosis: a review of literature

Deep learning and medical diagnosis

A survey on image data augmentation for deep learning

Ganai: Standardizing ct images using generative adversarial network with alternative improvement

Data augmentation using generative adversarial networks (CycleGAN) to improve generalizability in CT segmentation tasks

Synthetic medical images using

How to fool radiologists with generative adversarial networks? a visual turing test for lung cancer diagnosis

Synthetic data augmentation using GAN for improved liver lesion classification

Synthetic medical images from dual generative adversarial networks

Medical image synthesis for data augmentation and anonymization using generative adversarial networks

Deep learningbased multi-view fusion model for screening 2019 novel coronavirus pneumonia: a multicentre study

A deep learning system to screen novel coronavirus disease 2019 pneumonia. Engineering

AIassisted CT imaging analysis for COVID-19 screening: building and deploying a medical AI system

Ai-corona: radiologist-assistant deep learning framework for covid-19 diagnosis in chest ct scans

Application of deep learning technique to manage COVID-19 in routine clinical practice using CT images: Results of 10 convolutional neural networks

Deep learning-based model for detecting 2019 novel coronavirus pneumonia on high-resolution computed tomography

Deep learning-based detection for COVID-19 from chest CT using weak label

Review of artificial intelligence techniques in imaging data acquisition, segmentation and diagnosis for covid-19

Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans

Applications of artificial intelligence in battling against Covid-19: a literature review

COVID-19: a survey on public medical imaging data resources

Open resource of clinical data from patients with pneumonia for the prediction of COVID-19 outcomes via deep learning

Bidirectional ConvLSTM U-Net with densley connected convolutions

GrabCut interactive foreground extraction using iterated graph cuts

Generative adversarial networks: an overview

A review on generative adversarial networks: Algorithms, theory, and applications

Imagenet: A large-scale hierarchical image database

Very deep convolutional networks for large-scale image recognition

Deep residual learning for image recognition

Neural Computing and Applications

Rethinking the inception architecture for computer vision

Inception-v4, inception-resnet and the impact of residual connections on learning

Xception: Deep learning with depthwise separable convolutions

Densely connected convolutional networks

Keras: Deep Learning for Humans

Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations

Neural Computing and Applications