key: cord-0575627-8ypdyywh
authors: Ghassemi, Navid; Shoeibi, Afshin; Khodatars, Marjane; Heras, Jonathan; Rahimi, Alireza; Zare, Assef; Pachori, Ram Bilas; Gorriz, J. Manuel
title: Automatic Diagnosis of COVID-19 from CT Images using CycleGAN and Transfer Learning
date: 2021-04-24
journal: nan
DOI: nan
sha: 0c037e1e2e05ca25083f870302ca87dd4f084e6b
doc_id: 575627
cord_uid: 8ypdyywh

The outbreak of the corona virus disease (COVID-19) has changed the lives of most people on Earth. Given the high prevalence of this disease, its correct diagnosis in order to quarantine patients is of the utmost importance in steps of fighting this pandemic. Among the various modalities used for diagnosis, medical imaging, especially computed tomography (CT) imaging, has been the focus of many previous studies due to its accuracy and availability. In addition, automation of diagnostic methods can be of great help to physicians. In this paper, a method based on pre-trained deep neural networks is presented, which, by taking advantage of a cyclic generative adversarial net (CycleGAN) model for data augmentation, has reached state-of-the-art performance for the task at hand, i.e., 99.60% accuracy. Also, in order to evaluate the method, a dataset containing 3163 images from 189 patients has been collected and labeled by physicians. Unlike prior datasets, normal data have been collected from people suspected of having COVID-19 disease and not from data from other diseases, and this database is made available publicly.

based on AI using CT-Scan or X-Ray images for precise diagnosis of has been highly regarded by researchers [32, 33, 34] . Deep learning (DL) is one of the fields of AI, and many research papers have been published on their application for diagnosing COVID-19 [35, 36] .

In this paper, a new method of diagnosing COVID-19 from CT-Scan images using DL is presented. First, CT-Scan images of people with COVID-19 and normal people were recorded in Gonabad Hospital (Iran). Next, three expert radiologists have labeled the patients' images. They have also selected informative slices from each scan. Then, after preprocessing data with a Gaussian filter, various deep learning networks were trained in order to separate COVID-19 from healthy patients. In this step, a CycleGAN [37, 38] architecture was first used for data augmentation of CT-Scan data; after that, a number of pretrained deep networks [39] such as DenseNet [40] , ResNet [41] , ResNest [42] , and ViT [43] have been used to classify CT-Scan images. Figure 1 shows the block diagram of method. The results show that the proposed method of this study has promising results in detecting COVID-19 from CT-scan images of the lung. The rest of the paper is organized as follows. In the next section, we present a review of previous research on the diagnosis of COVID-19 from CT-Scan images using DL techniques. In Section 3, the proposed method of this research is presented. In Section 4, the evaluation process and the results of the proposed method are presented. Section 5 includes the discussion of paper and finally, 4 the paper ends with the conclusion and future directions.

Prior research papers on the diagnosis of the COVID-19 disease using machine learning can be divided according to the algorithms used or the underlying modalities. Figure 2 shows various types of methods that can be used for diagnosis of COVID-19. As can be seen in this figure, the methods based on medical imaging can be divided into two groups: CT scan and X-ray. The focus of this article is on CT scan modality. Also, machine learning algorithms can be divided into two categories: DL [44, 45] and conventional machine learning methods [46, 47, 48] . Due to the large number of machine learning papers for diagnosing COVID-19 disease from CT modality, we have only reviewed papers that have used deep learning methods for this imaging modality. Table 1 provides an overview of these papers, the datasets used by them, the components of methods, and finally, their performance. 

This section of the paper is devoted to discussing the applied method and its components. In this paper, we have firstly collected a new CT scan dataset from COVID-19 patients; then, from each scan, the informative slices were selected by physicians. After that, several convolutional neural networks pre-trained on the ImageNet dataset [79] were fine-tuned to the task at hand. Here, we trained a Resnet-50 architecture [41] , an EfficientNet B3 architecture [80] , a Densenet-121 architecture [40] , a ResNest-50 architecture [42] , and a ViT architecture [43] . Several data augmentation techniques alongside a CycleGAN model were applied to improve the performance of each network further.

Here, the details of each step are presented; firstly, an explanation is given on the applied dataset. Specifications of each applied deep neural network (DNN) are discussed afterward. Finally, CycleGAN is explained in the last part alongside the overall proposed method.

In this paper, a new CT scans dataset of COVID-19 patients was collected from Gonabad Hospital in Iran; all data were recorded by radiologists between June 2020 and December 2020. The number of subjects with COVID-19 is 90, and 99 of the subjects are normal. It is noteworthy to mention that the normal subjects are patients with suspicious symptoms and not merely a control group;

this makes this dataset unique compared to its prior ones, as they usually have 

ResNet [41] architecture was introduced in 2015 with a depth of 152 layers;

it is known as the deepest architecture up to that year and still is considered as one of the deepest. There are various versions of the architecture with different depths that are used depending on the need. This network's main idea was to use a block called residual block, which tried to solve the problem of vanishing gradients, allowing the network to go deeper without reducing performance.

Proving its capabilities by winning the Image Net Challenge in 2015; the ideas of this network have been applied in many others ever since. In this paper, a version of this network with a depth of 50 has been used, which is a wise choice given the considerably smaller amount of data compared to the ImageNet database.

Three different criteria must be tested to design a convolutional neural network: the depth, width, and resolution of the input images. Choosing the proper values of the three criteria in such a way that they form a suitable network together is a challenging task. Increasing the depth can lead to finding complex patterns, but it can also cause problems such as vanishing gradients.

More width can increase the quality of the features learned, but accuracy for such network tends to quickly saturate. Also, high image quality can have a detrimental effect on accuracy. The network was introduced in [80] with a study on how to scale the network in all three criteria properly. Using a step-by-step scheme, the network first finds the best structure for a small dataset and then scales that structure according to the activity. The network has been used for many tasks, including diagnosing autism [81] and schizophrenia [82] .

Introduced by Huang et al. [40] , DenseNet, densely connected convolutional networks, has improved the baseline performance on benchmark computer vision task and shown its efficiency. Utilizing residuals in a better approach has allowed this network to exploit fewer parameters and go deeper. Also, by feature reuse, the number of parameters is reduced dramatically. Its building blocks are dense blocks and transition layers. Compared to ResNet, DenseNet uses concatenation in residuals rather than summing them up. To make this possible, each feature vector of each layer is chosen to have the same size for each dense block; also, training these networks has been shown to be easier than prior ones [40] . This is arguably due to the implicit deep supervision where the gradient is flowing back more quickly. The capability to have thin layers is another remarkable difference in DenseNet compared to other state-of-the-art techniques. The parameter K, the growth rate, determines the number of features for each layer's dense block.

These feature vectors are then concatenated with the preceding ones and given as input to the subsequent layer. Eliminating optimization difficulties for scaling up to hundreds of layers is another DenseNet superiority.

Arguably, the main problem with convolutional neural networks is their failure in encoding relative spatial information. In order to overcome this issue, researchers in [43] have adopted the self-attention mechanism from natural lan- shows not only superior results but also significantly reduced training time and also less demand for hardware resources [43] .

Developed by researchers from Amazon and UC Davis, ResNest [42] is also another attention-based neural network that has also adopted the ideas behind ResNet structure. In its first appearance, this network has shown significant performance improvement without a large increase in the number of parameters, surpassing prior adaptations of ResNet such as ResNeXt and SEnet. In their paper, they have proposed a modular Split-Attention block that can distribute attention to several feature-map groups. The split-attention block is made of the feature-map group and split-attention operations; then, by stacking those split-attention blocks similar to ResNet, researchers were able to produce this new variant. The novelties of their paper are not merely introducing a new structure, but they also introduced a number of training strategies.

Generative adversarial networks were first introduced in 2014 [83] and found their way into various fields shortly after [44] . They have also been used as a method for data augmentation, and network pretraining [84] previously as well.

A particular type of these networks is CycleGAN [37] , a network created mainly for unpaired image-to-image translation. In this particular form of image-toimage translation, there is no need for a dataset containing paired images, which is itself a challenging task. The CycleGAN comprises of training of two generator discriminators simultaneously. One generator uses the first group of images as input and creates data for the second group, and the other generator does the opposite. Discriminator models are then utilized to distinguish the generated data from real ones and feed the gradients to generators subsequently.

The CycleGAN used in this paper has a similar structure to the one presented in the main paper [37] . Compared to other GAN paradigms, CycleGAN uses image-to-image translation, which simplifies the training process, especially where training data is limited, which also helps to create data of the desired class easily. However, using other GAN paradigms, such as conditional GAN [85] , one can also create data of a specific class, yet training those methods is more complicated. A diagram of the CycleGAN is presented in Figure 4 , and also a few samples of generated data are illustrated in Figure 5 .

In this paper, to train the networks properly, first, we preprocessed images by applying a Gaussian filter. Then, we applied several data augmentation techniques [86] , namely, by using random flips, rotations, zooms, warps, lighting transforms, and also presizing [87] . We also studied our models' performance by training them using an augmented dataset generated by means of a CycleGAN model implemented using the UPIT library [88] . 

All models were trained using the FastAI library [87] and applying finetuning to the pre-trained models available at the timm repository [89] using a GPU Nvidia RTX 2080 Ti with 11 GB of RAM. As for the CycleGAN implementation, the UPIT library [88] was used. To find the best hyperparameters, such as the learning rate for the task at hand, and to evaluate our models properly, we divided the data into three parts: the first one for training, the second one for validation, and the last one for testing. This division was done using a 70/15/15 scheme, and also, no two slices of any patient are presented in two different parts simultaneously to make the results trustworthy. To set the learning rate for the architectures, we employed the two-stage procedure similar to the one presented in [80] ; lastly, we applied early stopping in all the architectures to avoid overfitting. The final selected values for batch size and hyperparameters are all available in Table 2 .

13 Figure 5 : Examples of generated data, the first column shows normal data from the main dataset. The second column shows the generated abnormal data from those images. The third column shows abnormal data from the main dataset. Lastly, the fourth column shows the generated normal data from those images.

The evaluation of each network's performance is measured by several different statistical metrics, considering that merely relying on one measure of accuracy, it is not possible to measure all the different aspects of the performance of a network. The metrics used in this article are accuracy, precision, recall, F1-score, and area under receiver operating characteristic (ROC) curve (AUC) [90] . How to calculate these metrics is also shown in Table 2 . In this table, TP shows the number of positive cases that have been correctly classified, TN has shown the number of negative cases that have been correctly classified, and FP and FN are the numbers of positive and negative cases that have been misclassified, respectively. In addition, for each network, a learning curve is plotted that shows the speed of learning and how to converge. 

Parameter Mathematical Equation

T P +T N F P +F N +T P +T N Precision T P F P +T P Recall T P F N +T P F1-Score 2 P rec×Sens P rec+Sens

This part of the paper is dedicated to showing the results of networks. Each network is first trained without using the CycleGAN, and then the effect of adding CycleGAN is measured. Tables 4 and 5 demonstrate the network results  without and with CycleGAN, and Figures 6 and 7 also show the networks' learn- ing curves. To make the results reliable, each network is evaluated ten times, and then the mean of performances, with confidence intervals, are presented.

As observable in these tables, CycleGAN has improved the performance of Ef-ficientNet, Resnet, and ResNeSt dramatically. Nevertheless, the ViT results

show no sign of improvement in the presence of CycleGAN; this is arguably due to its robustness or indistinguishability of wrongly classified samples from the other class. ROC curve for one run of the networks is also plotted in Figure 8 . 

In recent years, convolutional neural networks have revolutionized the field of image processing. Medical diagnoses are no exception, and today in numerous research papers in this field, the use of these networks to achieve the best accuracy is seen. Diagnosis of COVID-19 disease from CT images is also one of the applications of these networks. In this article, the performance of different networks in this task was examined, and also by applying a new method, an attempt was made to improve the performance of these networks. The networks used in this paper were Resnet, EfficientNet, Densenet, ViT, and ResNest, and the data augmentation method was based on CycleGAN. Table 6 summarizes the proposed method of previous papers. By comparing this table with our current work, the advantages of our work can be listed as using ViT, a transformer-based architecture that has achieved state-of-the-art performances; collecting a new dataset; and finally using CycleGAN for data augmentation. Eventually, the ViT network reached an accuracy of 99.60%, which shows its state-of-the-art performance and proves that it can be used as the heart of a CADS. By comparing the performances of our method compared to previous works in table 1, our methods' superiority is quite observable. The advantages of adding CycleGAN were also clearly displayed, and it was shown that this method could be used for this task by data augmentation to improve the performance of most deep neural networks. Finally, this article's achievements can be summarized in: first, introducing a new database and its public release, second, examining the performance of various neural networks on this database, and finally evaluating the use of CycleGAN for data augmentation and its impact on networks performances. Additionally, the performance of ViT was never previously studied for this task, which was investigated in this paper as well.

To evaluate the method, a CT scan dataset was collected by physicians, which we also made available to researchers in public. Also, due to the fact that this dataset was collected from people suspected of having COVID-19, normal class data, unlike many previous datasets in this field, were collected from patients with suspicious symptoms and not from other diseases.

In the past year or so, nearly all people have found their lives changed due to the COVID-19 outbreak. Researchers in image processing and machine learning have not been an exception, considering many research papers that have been published on a variety of automatic diagnostic methods using medical imaging modalities and machine learning methods. Building an accurate diagnostic system in these pandemic conditions can relieve many of the burdens on physicians and also help to improve the situation. In this paper, the use of convolutional neural networks for the task at hand was investigated, and also the effect of adding CycleGAN for data augmentation was examined as another novelty of the paper. Finally, our method reached state-of-the-art performances and also have outperformed prior works, which shows its superiority.

For future work, several different paths can be considered; first, more com-plicated methods in deep neural networks can be used, such as deep metric, few-shot learning, or feature fusion solutions. Also, the combination of different datasets to improve the accuracy and evaluate its impact on the training of different networks can be examined. Finally, combining different modalities to increase accuracy can also be a direction for future research.

Artificial intelligence and covid-19: deep learning approaches for diagnosis and treatment

Deep learning covid-19 detection bias: accuracy through artificial intelligence

Validation of sars-cov-2 detection across multiple specimen types

Classification and specific primer design for accurate detection of sars-cov-2 using deep learning

Automated detection and forecasting of covid-19 using deep learning techniques: A review

A review of modern technologies for tackling covid-19 pandemic

A hierarchical classification method for breast tumor detection

Systematic review of artificial intelligence techniques in the detection and classification of covid-19 medical images in terms of evaluation and benchmarking: Taxonomy analysis, challenges, future solutions and methodological aspects

Hospital outbreak of middle east respiratory syndrome coronavirus

Towards a novel peptide vaccine for middle east respiratory syndrome coronavirus and its possible use against pandemic covid-19

Applications of machine learning and artificial intelligence for covid-19 (sars-cov-2) pandemic: A review

Deep 23 learning analysis improves specificity of sars-cov-2 real time pcr

Sars-cov-2 diagnosis using medical imaging techniques and artificial intelligence: A review

Estimating uncertainty and interpretability in deep learning for coronavirus (covid-19) detection

Fusion of convolution neural network, support vector machine and sobel filter for accurate detection of covid-19 patients using x-ray images

Role of machine learning techniques to tackle the covid-19 crisis: Systematic review

The role of imaging in the detection and management of covid-19: a review

Role of biological data mining and machine learning techniques in detecting and diagnosing the novel coronavirus (covid-19): a systematic review

Real-time rt-pcr in covid-19 detection: issues affecting the results

Positive rt-pcr test results in patients recovered from covid-19

Sensitivity of chest ct for covid-19: comparison to rt-pcr

Automated detection of covid-19 cases using deep neural networks with x-ray images

Diagnosis of covid-19 using ct scan images and deep learning techniques

Application of deep learning techniques for detection of covid-19 cases using chest x-ray images: A comprehensive study

Lightweight deep learning models for detecting covid-19 from chest x-ray images

Determination of disease severity in covid-19 patients using deep learning in chest x-ray images

Corodet: A deep learning based classification for covid-19 detection using chest x-ray images

Integrating deep learning ct-scan model, biological and clinical variables to predict severity of covid-19 patients

Uncertainty-aware semi-supervised method using large unlabelled and limited labeled covid-19 data

A deep learning algorithm using ct images to screen for corona virus disease (covid-19)

Weakly supervised deep learning for covid-19 infection detection and classification from ct images

A review on deep learning techniques for the diagnosis of novel coronavirus (covid-19)

Review on machine and deep learning models for the detection and prediction of coronavirus

Automatic diagnosis of covid-19 and pneumonia using fbd method

Role of intelligent computing in covid-19 prognosis: A state-of-the-art review

Review of artificial intelligence techniques in imaging data acquisition, segmentation and diagnosis for covid-19

Unpaired image-to-image translation using cycle-consistent adversarial networks

Improved cyclegan with application to covid-19 classification

Material recognition for automated progress monitoring using deep learning methods

Densely connected convolutional networks

Deep residual learning for image recognition

Resnest: Split-attention networks

An image is worth 16x16 words: Transformers for image recognition at scale

Deep learning

Artificial intelligence within the interplay between natural and artificial computation: Advances in data science, trends and applications

Pattern recognition and machine learning

Risk factors prediction, clinical outcomes, and mortality in covid-19 patients

Deep learning in current neuroimaging: a multivariate approach with power and type i error control but arguable generalization ability

Coronavirus (covid-19) classification using deep features fusion and ranking technique

A light cnn for detecting covid-19 from ct scans of the chest

End-to-end covid-19 screening with 3d deep learning on chest computed tomography

The ensemble deep learning model for novel covid-19 on ct images

Densely connected convolutional networksbased covid-19 screening model, Applied Intelligence (2021) 1

End-to-end automatic differentiation of the coronavirus disease 2019 (covid-19) from viral pneumonia based on chest ct

Resgnet-c: A graph convolutional neural network for detection of covid-19

Covid-19 detection system using chest ct images and multiple kernels-extreme learning machine based on deep neural network

accurate diagnosis and lesion segmentation of covid-19 using ct images

Covid-19 classification by fgcnet with deep feature fusion from graph convolutional network and convolutional neural network

A deep learning algorithm using ct images to screen for corona virus disease (covid-19)

Dual-sampling attention network for diagnosis of covid-19 from community acquired pneumonia

Covidnet-ct: A tailored deep convolutional neural network design for detection of covid-19 cases from chest ct images, Frontiers in medicine 7

Deep transfer learning-based automated detection of covid-19 from lung ct scan slices

A deep transfer learning model with classical data augmentation and cgan to detect covid-19 from chest ct radiography digital images

Any unique image biomarkers associated with covid-19?

Automatic screening of covid-19 using an optimized generative adversarial network

Classification of covid-19 by compressed chest ct image through deep learning on a large patients cohort

Mohammadi, Application of deep learning technique to manage covid-19 in routine clinical practice using ct images: Results of 10 convolutional neural networks

Deep transfer learning based classification model for covid-19 disease

Weakly supervised deep learning for covid-19 infection detection and classification from ct images

Multi-task deep learning based ct imaging analysis for covid-19 pneumonia: Classification and segmentation

The detection of covid-19 in ct medical images: A deep learning approach

A deep learning interpretable model for novel coronavirus disease (covid-19) screening with chest ct images

Classification of covid-19 in chest ct images using convolutional support vector machines

A classification-detection approach of covid-19 based on chest x-ray and ct by using keras pretrained deep learning models

Automatic identification of covid-19 regions on ct-images using deep learning

Coronavirus disease (covid-19) detection using deep features learning

Covid ct-net: A deep learning framework for covid-19 prognosis using ct images

Diagnosis of covid-19 in ct image using cnn and xgboost

Cnn features off-the-shelf: an astounding baseline for recognition

Rethinking model scaling for convolutional neural networks

Deep learning for neuroimaging-based diagnosis and rehabilitation of autism spectrum disorder: A review

An overview on artificial intelligence techniques for diagnosis of schizophrenia based on magnetic resonance imaging modalities: Methods, challenges, and future works

Generative adversarial networks

Deep neural network with generative adversarial networks pre-training for brain tumor classification based on mr images

Conditional generative adversarial nets

Best practices for convolutional neural networks applied to visual document analysis

Deep Learning for Coders with fastai and PyTorch

Unpaired image-to-image translation

Pytorch image models

A comprehensive comparison of handcrafted features and convolutional autoencoders for epileptic seizures detection in eeg signals