key: cord-0859756-pj74k1i8
authors: Verma, Poonam; Tripathi, Vikas; Pant, Bhaskar
title: Comparison of Different Optimizers Implemented on the Deep Learning Architectures for COVID-19 Classification
date: 2021-02-23
journal: Mater Today Proc
DOI: 10.1016/j.matpr.2021.02.244
sha: b54e9264d47318d0df4c337c288634e8bb95a65f
doc_id: 859756
cord_uid: pj74k1i8

COVID-19 is the present-day pandemic around the globe. WHO has estimated that approx 15% of the world's population may have been infected with coronavirus with a large number of population on the verge of being infected. It is quite difficult to break the virus chain since asymptomatic patients can result in the spreading of the infection apart from the seriously infected patients. COVID-19 has many similar symptoms to SARS-D however, the symptoms can worsen depending on the immunity power of the patients. It is necessary to be able to find the infected patients even with no symptoms to be able to break the spread of the chain. In this paper, the comparison table describes the accuracy of deep learning architectures by the implementation of different optimizers with different learning rates. In order to remove the overfitting issue, different learning rate has been experimented. Further in this paper, we have proposed the classification of the COVID-19 images using the ensemble of 2 layered Convolutional Neural Network with the Transfer learning method which consumed lesser time for classification and attained an accuracy of nearly 90.45%.

COVID-19 has been a global health emergency and it has spread across the globe with more than 45 million confirmed cases and 1.2 million deaths with the vast majority of the world remains at the risk. The cases of COVID-19 have been rising again with a steep increase with the onset of the winters in the northern hemisphere. During this COVID-19 pandemic, prognostic predictions are required to improve and manage patient care. COVID-19 is caused by the virus due to which antibiotics are unable to treat the patients thereby complicating their infection if the body immunity is low. There exist large public datasets of more typical chest X-rays from the NIH [ [1, 2] .Similarly better software tools can help the doctors arrange for better analysis and have a second opinion that can act as the confirmation of the diagnosis by the radiologist and treatment procedures can then be decided thereby improving the patient survival rate [3] . If datasets of images related to the COVID-19 can be added with only the important features then better models with higher prediction accuracy can be developed to differentiate between the different types of virus-infected problems. The major challenge for this problem statement is the Unavailability of a large dataset of images of different types of the severity of the infection. In the present paper, we have made use of the COVIDGR 1.0 dataset that consists of images of COVID-19 tested patients that had an equal number of images of positive and negative COVID-19 patients. Another major issue is that the COVID-19 infection has many symptoms similar to the other flu infections. Being similar to other lung infections, it becomes a major challenge to classify it into accurate class.

In this paper, we have proposed a deep learning framework that constitutes the ensemble of 2 layered CNN for the classification of the COVID-19 images making use of transfer learning. This paper is organized as research works carried out on the data by other authors in Section 2, where the Literature Review of the COVID-19 is being discussed. Deep Learning Architectures compared and proposed have been discussed in Section 3. Section 4 defines the Methodology of the classifiers with an overview of the architecture of the proposed approach. Section 5 with results and comparison analysis with other classifiers and finally we conclude the open research challenges in Section 6.

Due to the severe upsurge of the infection of COVID-19, a lot of research is going on that has been focussed on various aspects of the COVID-19 ranging from the genome comparison of the COVID-19 with the genomes of malaria, SARS, etc, effect of the geographical factors on the infection and image analysis of CT scans and X-rays of the positive and negative patients. Since the last decade, Image Processing has proved as an important tool to classify the images of cancer and other diseases for diagnosis purposes. Further, the advancement of Machine Learning has helped in the easy and accurate Image Analysis. Large numbers of researches are being presently conducted for the Image analysis of the CT scans and X-rays of the lungs with better classification results that can decide the path of the treatment for the patients that can increase the survival rate of the patients. The advancements in the computing power of the systems and GPUs have helped to carry out complicated tasks are possible.

The proposed schema helps not only in the improvement of image classification but also in analysis with better symptomatic accuracies. Further, we have presented the comparison of the 4 state-of-the-art predesigned networks like VGG-16, ResNet-50, InceptionV3 & EfficientNet apart from the transfer learning implemented on the CNN. All these networks were fine-tuned with the transfer learning and multiple experiments were conducted on the acquired datasets using multiple optimizers and learning rates. Recent studies on the CT and X-Ray images of the COVID-19 patients reported that patients present abnormalities in chest CT images with most having bilateral involvement. Bilateral multiple lobular and subsegmental areas of consolidation constitute the typical findings in chest CT images of intensive care unit (ICU) patients on admission. In comparison, non-ICU patients show bilateral ground-glass opacity and subsegmental areas of consolidation in their chest CT images. In these patients, later chest CT images display bilateral ground-glass opacity with resolved consolidation [4, 5, 6] . In order to detect whether the CXR images of the patient have an infection or not, it is not possible to implement traditional machine learning algorithms since the input has a fixed dimension. Hence we have utilized deep learning networks for image classification.

Deep learning Architectures have been used for heavy computational tasks and these architectures have been able to provide the output with reduced time complexity. One of the first neural networks has been Convolutional Neural Network (CNN). CNN is termed as the group of deep neural networks that can help to classify the different classes of images. It reduces the preprocessing of the input data by assigning the weights to the most important features of the input data as the input data moves deeper into the layers for further processing. CNN is based on the multilayer perceptron. It utilizes a large number of neurons in the layers. CNN has been successfully used for the application of the classification of images. [7] . The network ingests images as separate layers of 3 colors overlapping each other. When images are fed into the CNN, and the images are expressed as multidimensional matrices. The square patch of pixels has a definite pattern and filters are used to further extract these patterns. Further Activation Maps are used to describe the number of steps for filtering the images [7, 8] . The images are augmented by slicing and rotating at 15° , 30°, and 45° , and then, rotated and flip by 90° , 180° , and 270° . Further, a Random Gaussian variable was added to the pixel values to make the noised applied in each presence of the infection. If the input image had no presence of the infection in the pixels. Those images will be removed and if the infection is present in the pixels, the images would be stored.

The standard way to model a neuron is determined by the following equation-

whose processing is steady [9] . Thus, the RELU function which describes the "non-saturating nonlinearity" is used and its equation is-

VGG-16: This deep learning architecture consists of 13 Convolutional layers with 3 layers for fully connected layers. The final output layer consists of the 1000 classes of the images for an input of 224 x 224 images. All the layers are not followed by the convolutional layers and max-pooling layers [11] . The activation function used in the hidden layers is RELU.

ResNet : ResNet stands for the Residual Network and it is an Artificial Neural Network that builds pyramidal cells using skip connections and contains nonlinearities [12] . The skip connections are carried out to overcome the problem of the vanishing gradients and further makes the network more simple [13, 14] . ResNet-50 has 50 layers where 48 convolution layers are present. It has a large number of tunable parameters. It carries out the 3.8 x 109 operations.

Inception V3: Inception is made up of many building blocks consisting of convolutions, average pooling, dropouts, and fully connected layers with softmax acting as the loss function [15, 16] . The input images are required to be resized to the dimension of 299 x 299 x 3. To improve the accuracy of the classification, it is required that the training samples are repeatedly passed for the training and it is termed as steps of an epoch. RMSProp is considered to be one of the best default optimizers that makes use of decay and momentum variables to achieve the best accuracy of the image classification. With the increase in the volume of the training dataset, it requires the model to learn gradually about the training samples, hence the implementation of the learning rate is helpful. Furthermore, the learning rate also avoids the issue of overfitting.

EfficientNet : This Deep Learning was proposed for scaling all the dimensions of the images with a stable coefficient. Those coefficients have been added to the baseline network to improve the performance of the network and increase the classification accuracy and time complexity. There are B0 to B7 variants of this model with the resolution ranging from 224 to 600 [17, 18, 19] . 

The proposed Methodology consisted of the feature extraction with the help of Transfer Learning techniques implemented on the ensemble 2 layered network of the CNN. The classification of the images starts with the preprocessing of the images that are required to be converted into the same channel so that the deep learning models can process the data. The COVID-19 has created a drastic issue over the world, however, a very less amount of images of the patients are available for open public domain access. To scale up the volume of the images, different data augmentation techniques including rotation, pixel swapping, etc can be used. In the second stage, models were defined with the different layers overlapping [20] . These layers were further enhanced with the different training parameters like number of epochs, number of folds per epoch, batch size, different optimizers, different learning rates. Further, the classification was carried out and the models defined were validated.

This section brings a series of statistical plots (Table 1) with the final presentation of the results achieved in the deep learning architectures described in the above sections. We considered a total of five algorithms for accuracy assessment. It was seen that the accuracy of the proposed approach (Transfer learning on the CNN) was the highest (85.81%) with a specificity of 0.732 and a sensitivity of 0.812. The performance of CNN is found to be comparable to the state of the art machine learning methods with only little pre-processing and fine-tuning [21] . These results were obtained using the network on the X-Ray images dataset of COVID-19. Fine-tuning of the CNN architecture with our own dataset was also done and then the results were finally computed.

CNN gives better results owning to better training of model for different classes, deep features of CNN provide much more information. Instead of using hand-crafted features, CNN has the potential to extract features automatically. Therefore, deep models are of great benefit to the ability to learn very complex tasks, which we hardly understand. In addition, intensive learning models can exploit non-linear relationships. It can be easily implemented in a neural network model with various activation functions. Another advantage is that there are several associated parameters in it and their tuning is known as hyper-parameter tuning which has generalization capability. Among the problems associated with CNN, one problem is that the images are high-dimensional due to which the process becomes more time-consuming.

The input of the data consists of the dataset described as above. It has images of the lungs of 426 positive and 426 negative patients. The models selected are VGG-16, ResNet-50, Inception V3, and EfficientNet. All four were experimented with two different optimizers RMSProp and Adam. Apart from the change in the optimizers, two different learning rates were used for the model training to avoid overfitting.

Each model consisted of the training run which had an epoch of 10 and steps per epochs to be 50. Features were extracted before the training procedure in the preprocessing phase of the images. In each validation fold out of 101 images, 40 images were randomly used for the validation tests. These training and testing simulations of all the specified models were performed on Google Colab using the 32 GB RAM.

Since the diagnosis through the medical images is a sensitive issue, therefore the results were validated using different techniques recursively. Major concern if of False Negative where not only the patient but also people around the patient tested false negative will be in grave danger. The term Accuracy can help to measure the samples that have been classified correctly [22, 23] . Precision can help to define the ratio of the true positive cases to the total positive observations predicted. CNN implemented with Transfer Learning synthesized the highest accuracy of approx 91% with " Adam" Optimizer at a learning rate of 1 e-4. VGG-16 showed balanced results with both the optimizers reaching a maximum of output efficiency of 65% on "Adam" and "RMSProp" optimizers with the learning rate of 4e-4. ResNet-50 was able to achieve an accuracy of approx. 85% with a learning rate of 1e-4. It can be clearly observed that transfer learning applied on the CNN with RMSProp optimizer and learning rate of 1e-4 has achieved the highest accuracy with reduced time complexity. Table 2 describes the comparison of the time complexity of the different Architectures over different optimizers. 

Since the beginning of the pandemic, numerous research studies are taking place. In our paper, the X-Ray images of the patients were used. Zhang et al. [24] made use of the pre-trained neural Networks of ResNet-50 to perform the classification of the COVID-19 infected patients. Numerous studies have been performed on the detection of COVID-19 symptoms via different techniques. Shan et al. [25] used VB-net for the image segmentations of patients.

A study similar to ours was conducted in [26] where they achieved 98% accuracy. But, their results could be prone to overfitting as they did not use multiple optimizers or different learning rates and only used three transfer learning methods. Zhang et al. [27] performed X-Ray image classification with the help of ResNet. Wang and Wong [28] adopted a convolutional neural network method for the classification of X-Ray images which achieved successfully 83.5% accuracy. A very famous transfer learning model "inception" was used by Wang et al. [29] to predict COVID. The COVIDX-Net includes seven different architectures of deep convolutional neural network models, such as modified Visual Geometry Group Network (VGG19) and the second version of Google MobileNet [30] . Popular Deep Learning Architectures such as Recurrent Neural Network (RNN), Long short-term memory (LSTM), Bidirectional LSTM (BiLSTM), Gated recurrent units (GRUs) and Variational AutoEncoder (VAE) are compared for the new and the recovered cases [31] .

In this paper, we have assessed that the Transfer Learning has improved the accuracy of the classification of the Images, although the dataset was limited in nature. The proposed Methodology pre-processed the COVID-19 X-Ray Images and input was provided into 5 different deep learning models of VGG-16, ResNet-50, Inception V3, EfficientNet and Transfer Learning on CNN. These models were evaluated for accuracy of classification of the Images to distinguish COVID-19 Positive and Negative Patients. These models were evaluated with the optimizers RMSProp and Adam with learning rate of 0.0001 and 0.0004. From all the models assessed the best accuracy was achieved by the Transfer Learning with CNN i.e. 90.45% with an optimizer RMSProp and Learning Rate of 0.0001. ResNet-50 model also achieved a good overall accuracy. In order to improve the accuracy of the ResNet-50, further Transfer Learning on ResNet-50 can be implemented over the COVID-19 X-Ray Images.

Simplifying decision trees. International journal of man-machine studies

Decision trees as probabilistic classifiers

Site distribution of different types of skin cancer: new aetiological clues

Clinical features of patients infected with 2019 novel coronavirus in Wuhan

A novel coronavirus from patients with pneumonia in China

A pneumonia outbreak associated with a new coronavirus of probable bat origin

Deep convolutional neural network for image deconvolution

Imagenet classification with deep convolutional neural networks

CNN-based segmentation of medical imaging data

Chest CT findings in coronavirus disease-19 (COVID-19): relationship to duration of infection

Attention-based VGG-16 model for COVID-19 chest X-ray image classification

Covid-resnet: A deep learning framework for screening of covid19 from radiographs

Comparative performance analysis of the resnet backbones of mask rcnn to segment the signs of covid-19 in chest ct scans

Detecting pathological brain via ResNet and randomized neural networks

Automatic detection of coronavirus disease (covid-19) using x-ray images and deep convolutional neural networks

Deep net model for detection of covid-19 using radiographs based on roc analysis

Automated medical diagnosis of COVID-19 through EfficientNet convolutional neural network

ECOVNet: An Ensemble of Deep Convolutional Neural Networks Based on EfficientNet to Detect COVID-19 From Chest X-rays

Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning

A new image classification method using CNN transfer learning and web data augmentation

Fine-tuning CNN image retrieval with no human annotation

Imagenet classification with deep convolutional neural networks

A deep learning approach to characterize 2019 coronavirus disease (COVID-19) pneumonia in chest CT images

Bag of tricks for image classification with convolutional neural networks

Open Database of COVID-19 Cases with chest X-Ray or CT images

Deep convolutional neural networks for image classification: A comprehensive review

Recent advances in convolutional neural networks. Pattern Recognition

Temporal dynamics in viral shedding and transmissibility of COVID-19

Pulmonary image classification based on inception-v3 transfer learning model

Covidx-net: A framework of deep learning classifiers to diagnose covid-19 in x-ray images

Deep learning methods for forecasting COVID-19 time-Series data: A Comparative study

Manuscript title:" Comparison of Different Optimizers implemented on the Deep Learning Architectures for COVID-19 Classification" All persons who meet authorship criteria are listed as authors, and all authors certify that they have participated sufficiently in the work to take public responsibility for the content, including participation in the concept, design, analysis, writing, or revision of the manuscript. Furthermore, each author certifies that this material or similar material has not been and will not be submitted to or published in any other publication before its appearance in the Hong Kong Journal of Occupational Therapy.

Please indicate the specific contributions made by each author (list the authors' initials followed by their surnames, e.g., Y.L. Cheung). The name of each author must appear at least once in each of the three categories below. Approval of the version of the manuscript to be published (the names of all authors must be listed): Poonam Verma; Dr. Bhaskar Pant & Dr. Vikas Tripathi Acknowledgements All persons who have made substantial contributions to the work reported in the manuscript (e.g., technical help, writing and editing assistance, general support), but who do not meet the criteria for authorship, are named in the Acknowledgements and have given us their written permission to be named. If we have not included an Acknowledgements, then that indicates that we have not received substantial contributions from nonauthors. Author's signature Date Poonam Verma ; 14 th Jan 2021

Potential conflict of interest exists:We wish to draw the attention of the Editor to the following facts, which may be considered as potential conflicts of interest, and to significant financial contributions to this work:

We wish to confirm that there are no known conflicts of interest associated with this publication and there has been no significant financial support for this work that could have influenced its outcome.

No Funding was received for this work.

Corresponding Author