key: cord-1027148-gkbfbh00 authors: Islam, Md. Zabirul; Islam, Md. Milon; Asraf, Amanullah title: A Combined Deep CNN-LSTM Network for the Detection of Novel Coronavirus (COVID-19) Using X-ray Images date: 2020-08-15 journal: Inform Med Unlocked DOI: 10.1016/j.imu.2020.100412 sha: 97f3a273777952d74620c7f3dd66cb863463af0c doc_id: 1027148 cord_uid: gkbfbh00 Nowadays, automatic disease detection has become a crucial issue in medical science due to rapid population growth. An automatic disease detection framework assists doctors in the diagnosis of disease and provides exact, consistent, and fast results and reduces the death rate. Coronavirus (COVID-19) has become one of the most severe and acute diseases in recent times and has spread globally. Therefore, an automated detection system, as the fastest diagnostic option, should be implemented to impede COVID-19 from spreading. This paper aims to introduce a deep learning technique based on the combination of a convolutional neural network (CNN) and long short-term memory (LSTM) to diagnose COVID-19 automatically from X-ray images. In this system, CNN is used for deep feature extraction and LSTM is used for detection using the extracted feature. A collection of 4575 X-ray images, including 1525 images of COVID-19, were used as a dataset in this system. The experimental results show that our proposed system achieved an accuracy of 99.4%, AUC of 99.9%, specificity of 99.2%, sensitivity of 99.3%, and F1-score of 98.9%. The system achieved desired results on the currently available dataset, which can be further improved when more COVID-19 images become available. The proposed system can help doc-tors to diagnose and treat COVID-19 patients easily. 2 COVID-19 can range from cold to fever, shortness of breath, and acute respiratory syndrome [3] . In comparison to SARS, the kidneys and liver are affected by coronavirus as well as the respiratory system [4] . Coronavirus detection at an early stage plays a vital role in controlling COVID-19 due to its high transmissibility. The diagnosis of coronavirus by gene sequencing for respiratory or blood samples should be confirmed as the main pointer for reverse transcription-polymerase chain reaction (RT-PCR), according to the guidelines of the Chinese government [5] . The process of RT-PCR takes 4-6 hours to get results, which takes a long time compared to COVID-19's rapid spread rate. RT-PCR test kits are in huge shortage, in addition to being inefficient [6] . As a result, many infected patients cannot be detected in time and tend to unknowingly infect others. With the detection of this disease at an early stage, the prevalence of COVID-19 disease will decrease [7] . To alleviate the inefficiency and scarcity of current COVID-19 tests, a lot of effort has been made to look for alternative test methods. Another visualization method is to diagnose COVID-19 infections using radiological images such as X-rays or computed tomography (CT). Earlier works have shown that anomalies can be found in COVID-19 patients in chest CT scans in the shape of ground-glass opacities [8] . The researchers have claimed that a system based on chest CT scans can be an important method for diagnosis and quantifying of COVID-19 cases [9] . Many researchers have demonstrated various approaches to detect COVID-19 utilizing X-ray images. Recently, computer vision [10] , machine learning [11, 12, 13] , and deep learning [14, 15] have been used to automatically diagnose several diverse ailments in the human body, which ensures smart healthcare [16, 17] . The deep learning method is used as a feature extractor that enhances classification accuracies [18] . The detection of tumor regions in the lungs, X-ray bone suppression, diabetic retinopathy, prostate segmentation, skin lesions, and the presence of the myocardium in coronary CT scans are examples of the contributions [19, 20] of deep learning. Therefore, this paper aims to propose a deep learning based system that combines the CNN and LSTM networks to automatically detect COVID-19 from X-ray images. In the proposed system, CNN is used for feature extraction and LSTM is used to classify COVID-19 based on those features. The LSTM network has an internal memory that is capable of learning from imperative experiences with long-term states. In fully connected networks, the layers are fully connected and the nodes between layers are connectionless and process only one input. In the case of LSTM, the nodes are connected from a directed graph along a temporal sequence that is considered an input with a specific order [21] . Hence, the 2-D CNN and LSTM layout feature combination improves classification greatly. The dataset used for this paper was collected from multiple sources and a preprocessing was performed to reduce the noise. In the following, the contributions of this research are summarized. J o u r n a l P r e -p r o o f 3 a) Developing a combined deep CNN-LSTM network to automatically assist the early diagnosis of patients with COVID-19 efficiently. b) To detect COVID-19 using chest X-rays using a dataset formed comprising 4575 images. c) A detailed experimental analysis is provided in terms of accuracy, sensitivity, specificity, F1-score, a confusion matrix, and AUC using receiver operating characteristic (ROC) to measure the performance of the proposed system. The paper is organized as follows: A review of recent scholarly works related to this study is described in Section 2. A description of the proposed system, including dataset collection and preparation, is presented in Section 3. The experimental results and comparative performance of the proposed deep learning system are provided in Section 4. The discussion is given in Section 5. Section 6 concludes the paper. To address the COVID-19 epidemic, researchers have developed deep learning techniques to diagnose COVID-19 based on clinical images, CT scans, and X-rays of the chest. This review describes the recently developed systems that have applied deep learning techniques in the field of COVID-19 detection. Rahimzadeh et al. [22] developed a concatenated CNN based on Xception and Res-Net50V2 models that classified COVID-19 cases using chest X-rays. Fig. 1 illustrated the overall system for the detection of COVID-19 consisting of several phases. Raw X-ray images were first passed through the preprocessing pipeline. Data resizing, shuffling, and normalization were done in the preprocessing pipeline. The preprocessed data set was then partitioned into a training set and testing set, and we trained the CNN and CNN-LSTM architecture using the training data. After each epoch, the training accuracy and loss were determined. At the same time, using 5-fold cross-validation, validation accuracy and loss were also obtained. The performance of the proposed system was measured with the following evaluation metrics: confusion matrix, accuracy, AUC using ROC, specificity, sensitivity, and F1-score. As the emergence of COVID-19 is very recent, none of the large repositories contain any COVID-19 labeled data, thereby requiring us to rely on different sources of images of normal, pneumonia, and COVID-19 cases. First, 613 X-ray images of COVID-19 cases were collected from the following websites: GitHub [35, 36] , Radiopaedia [37], The Cancer Imaging Archive (TCIA) [38] , and the Italian Society of Radiology (SIRM) [39] . Then, instead of data being independently augmented, a dataset containing 912 already augmented images was collected from Mendeley [40] . Finally, 1525 images of pneumonia cases and 1525 X-ray images of normal cases were collected from the Kaggle repository [41] and NIH dataset [42] . The main objective of the dataset selection was to make it available to the public so that it is accessible and extensible to researchers. The use of this dataset in further studies may also enable more efficient diagnoses of COVID-19 patients. We resized the images to ones with a resolution of 224 × 224 pixels. The number of X-ray images of each set was partitioned in Table 1 . The visualization of X-ray images of each class is shown in Fig. 2 . The proposed architecture was developed with a combination of a convolutional neural network (CNN) and a long short-term memory (LSTM) network, which are briefly described as follows. J o u r n a l P r e -p r o o f 7 A particular type of multilayer perceptron is a CNN, but a simple neural network cannot learn complex features, unlike a deep learning architecture. CNNs have shown excellent performance in many applications [43, 44] , such as image classification, object detection, and medical image analysis. The main idea behind a CNN is that it can obtain local features from high layer inputs and transfer them to lower layers for more complex features. A CNN comprises convolutional, pooling, and fully connected (FC) layers. A typical CNN architecture along with these layers is depicted in Fig. 3 The convolutional layer encompasses a set of kernels [45] for determining a tensor of feature maps. These kernels convolve an entire input using "stride(s)" so that the dimensions of an output volume become integers [46] . The dimensions of an input volume decrease after the convolutional layer is used to execute the striding process. Therefore, zero padding [47] is required to pad an input volume with zeros and maintain the dimensions of an input volume with low-level features. The operation of the convolutional layer is given as: where I refers to the input matrix, K denotes a 2D filter of size m × n, and F represents the output of a 2D feature map. The operation of the convolutional layer is denoted by I*K. To increase nonlinearity in feature maps, the rectified linear unit (ReLU) layer is used [48] . ReLU computes activation by keeping the threshold input at zero. It is mathematically expressed as follows: The pooling layer [49] performs a downsampling of a given input dimension to reduce the number of parameters. Max pooling is the most common method, which produces the maximum value in an input region. The FC layer [50] is used as a classifier that makes a decision on the basis of features obtained from the convolutional and pooling layers. . Long short-term memory is an improvement of recurrent neural networks (RNNs). LSTM proposes memory blocks instead of conventional RNN units in solving the vanishing and exploding gradient problem [51] . It then adds a cell state to save longterm states, which is its main difference from RNNs. An LSTM network can remember and connect previous information to data obtained in the present [52] . LSTM is combined with three gates, such as an input gate, a "forget" gate, and an output gate, where ‫ݐݔ‬ refers to the current input; ‫ܥ‬ t and ‫ݐܥ‬ −1 denote the new and previous cell states, respectively; and h t and ℎ‫ݐ‬ −1 are the current and previous outputs, respectively. J o u r n a l P r e -p r o o f The principle of the input gate of LSTM is shown in the following forms. (5), where ‫ݐ݅‬ refers to a sigmoid output, and ‫ݐܥ‬ refers to a tanh output. Here, W i denotes weight matrices, and b i represents the input gate bias of LSTM. The LSTM's forget gate then allows the selective passage of information using a sigmoid layer and a dot product. The decision about whether to forget related information from a previous cell with a certain probability is executed using (6), in which W f refers to the weight matrix, b f is the offset, and σ is the sigmoid function. The LSTM's output gate determines the states that are required for continuation by the ℎ‫ݐ‬ −1 and ‫ݐݔ‬ inputs following (7) and (8) . The final output is obtained and multiplied by the state decision vectors that pass new information, C t , through the tanh layer. where ܹo and ܾo are the output gate's weighted matrices and LSTM bias, respectively. In this study, a combined method was developed to automatically detect COVID-19 cases using three types of X-ray images. The structure of this architecture was designed by combining CNN and LSTM networks, where the CNN is used to extract complex features from images, and LSTM is used as a classifier. Fig. 5 illustrates the proposed hybrid network for COVID-19 detection. The network has 20 layers: 12 convolutional layers, five pooling layers, one FC layer, one LSTM layer, and one output layer with the softmax function. Each convolution block is combined with two or three 2D CNNs and one pooling layer, followed by a dropout layer characterized by a 25% dropout rate. The convolutional layer with a size of 3 × 3 kernels is used for feature extraction that is activated by the ReLU function. The max-pooling layer with a size of 2 × 2 kernels is used to reduce the dimensions of an input image. In the last part of the architecture, the function map is transferred to the LSTM layer to extract time information. After the convolutional block, the output shape is found to be (none, 7, 7, 512). Using the reshape method, the input size of the LSTM layer has become (49, 512) . The summary of the proposed architecture is shown in Table 2 . After analyzing the time characteristics, the architecture sorts the X-ray images through a fully connected layer to predict whether they belong under any of the three categories (COVID-19, pneumonia, and normal). The following metrics are used to measure the performance of the proposed system: TP denotes the correctly predicted COVID-19 cases, FP denotes the normal or pneumonia cases that are misclassified as COVID-19 by the proposed system, TN denotes the normal or pneumonia cases that are correctly classified, and FN denotes the COVID-19 cases that are misclassified as normal or pneumonia cases. In the experiment, the dataset was split into 80% and 20% for training and testing, respectively. The results were obtained using 5-fold cross-validation technique. The proposed network consists of 12 convolutional layers, as described in Table 2 , the learning rate is 0.0001, and the maximum epoch number is 125, as determined experimentally. The CNN and CNN-LSTM networks were implemented using Python and the Keras package with TensorFlow2 on an Intel(R) Core(TM) i7-2.2 GHz processor. In addition, the experiments were executed using the graphical processing unit (GPU) NVIDIA GTX 1050 Ti with 4 GB and 16 GB RAM, respectively. The overall accuracy, specificity, sensitivity, and F1-score for each case of CNN architecture are summarized in Table 3 and visually shown in Fig. 9 . The CNN network achieved 98.2% specificity, 99.0% sensitivity, and 97.7% F1-score for the COVID-19 cases. For the pneumonia classification, it recorded 99.7% specificity, 96.4% sensitivity, and 97.8% F1-score. In the normal cases, it obtained 99.8% specificity, 100% sensitivity, and 99.8% F1-score. While the highest specificity, sensitivity, and F1score were obtained in the normal cases, the lower values of sensitivity were found in the pneumonia cases. Furthermore, the ROC curves are added between the true positive rate (TPR) and the false positive rate (FPR) to compare the overall performance in Fig. 11 Finally, Gradient-weighted Class Activation Mapping (Grad-CAM) refers to the heat map used for the visual explanation of our experiment using the gradients of a target concept. A coarse localization map highlights the important regions in the image for prediction after passing into the final layer. In Fig.12 , the heat map of the classified test examples is shown for COVID-19, pneumonia, and normal cases both for CNN and CNN-LSTM architecture. By analyzing the results, it is demonstrated that a combination of CNN and LSTM has significant effects on the detection of COVID-19 based on the automatic extraction of features from X-ray images. The proposed system could distinguish COVID-19 from pneumonia and normal cases with high accuracy. A comparison between existing systems and our proposed system, in terms of accuracy and computational time, is shown in Table 5 . From Table 5 , it is found that some of the proposed systems [53] , [54] , [28] , [31] , [22] , [31] , [55] , and [56] obtained a slightly lower accuracy in range of 80.6% to 92.3%. The moderately highest accuracy of 93.5%, 95.2%, 95.4%, 98.3% and 98.3% are found in [26] , [23] , [29] , [57] , and [25] respectively. The system developed in [58] obtained an overall accuracy of 98.08% considering the multi-classes. Moreover, a comparison between existing systems in terms of computational time depicted that the system developed in [23] Our proposed CNN-LSTM architecture provides good performance and it is also faster than the CNN approach, taking 18372.0s and 113.0s for training and testing which is proportionally faster than other studies. Overall, the result of our proposed system is superior compared to other existing systems. As COVID-19 cases are increasing daily, many countries are facing resource shortages. Hence, it is necessary to identify every single positive case during this health emergency. We introduced a deep CNN-LSTM network for the detection of novel COVID-19 from X-ray images. Here, CNN is used as a feature extractor and the LSTM network as a classifier for the detection of coronavirus. The performance of the proposed system is improved by combining extracted features with LSTM that differentiate COVID-19 cases from others. The developed system obtained an accuracy of 99.4%, AUC of 99.9%, specificity of 99.2%, sensitivity of 99.3%, and F1-score of 98.9%. The proposed CNN-LSTM and competitive CNN architecture are applied both on the same dataset. Our extensive experimental results revealed that our proposed architecture outperforms a competitive CNN network. In this global COVID-19 pandemic, we hope that the proposed system would be able to develop a tool for COVID-19 patients and reduce the workload of the medical diagnosis for COVID-19. The proposed system has some limitations. Firstly, the sample size is relatively small that needs to be increased to test the generalizability of the developed system. This would be overcomed if more images are found in the coming days. Secondly, it only focuses on the posterior-anterior (PA) view of X-rays, hence it cannot differentiate other views of X-rays such as anterior-posterior (AP), lateral, etc. Thirdly, COVID-19 images comprising multiple disease symptoms cannot be efficiently classified. Finally, the performance of our proposed system was not compared with radiologists. Hence, a comparison of the proposed system with radiologists would be part of a future study. The authors declare no conflicts of interest. Clinical features of patients infected with 2019 novel coronavirus in Coronavirus Disease Correlation of Chest CT and RT-PCR Testing in Coronavirus Disease 2019 (COVID-19) in China: A Report of 1014 Cases The Incubation Period of Coronavirus Disease 2019 (COVID-19) From Publicly Reported Confirmed Cases: Estimation and Application Predictive Data Mining Models for Novel Coronavirus (COVID-19) Infected Patients Recovery COVID-19): Role of Chest CT in Diagnosis and Management Sensitivity of Chest CT for COVID-19: Comparison to RT-PCR A Survey on Computer Vision for Assistive Medical Diagnosis from Faces Prediction of breast cancer using support vector machine and K-Nearest neighbors Performance Evaluation of Random Forests and Artificial Neural Networks for the Classification of Liver Disorder Mathematical model development to detect breast cancer using multigene genetic programming Diabetes Prediction: A Deep Learning Approach Coronary Artery Heart Disease Prediction: A Comparative Study of Computational Intelligence Techniques Developing IoT Based Smart Health Monitoring Systems: A Review, Rev. d'Intelligence Artif Development of Smart Healthcare Monitoring System in IoT Environment Feature extraction for image recognition and computer vision Optimal deep learning model for classification of lung cancer on CT images Bone suppression of chest radiographs with cascaded convolutional networks in wavelet domain Deep back propagation-long shortterm memory network based upper-limb sEMG signal classification for automated rehabilitation A New Modified Deep Convolutional Neural Network for Detecting COVID-19 from X-ray Images Covid-2019 Detection Using X-Ray Images And Artificial Intelligence Hybrid Systems Within the Lack of Chest COVID-19 Xray Dataset: A Novel Detection Model Based on GAN and Deep Transfer Learning COVIDiagnosis-Net: Deep Bayes-SqueezeNet based diagnosis of the coronavirus disease 2019 (COVID-19) from X-ray images Covid-19: automatic detection from X-ray images utilizing transfer learning with convolutional neural networks Machine Learning Approach for Confirmation of COVID-19 Cases: Positive, Negative, Death and Release CoroNet: A Deep Neural Network for Detection and Diagnosis of Covid-19 from Chest X-ray Images Detection of coronavirus Disease ( COVID-19 ) based on Deep Features X-Ray Image based COVID-19 Detection using Pre-trained Deep Learning Models COVIDX-Net: A Framework of Deep Learning Classifiers to Diagnose COVID-19 in X-Ray Images Transfer learning based ensemble support vector machine model for automated COVID-19 detection using lung computerized tomography scan data Deep transfer learning -based automated detection of COVID-19 from lung CT scan slices Composite Monte Carlo decision making under high uncertainty of novel coronavirus epidemic using hybridized deep learning and fuzzy rule induction COVID-19 Image Data Collection COVID-19 chest X-ray Welcome to The Cancer Imaging Archive -The Cancer Imaging Archive (TCIA) COVID-19 DATABASE | SIRM Mendeley Data -Augmented COVID-19 X-ray Images Dataset Kaggle chest x-ray repository Object Detection Using Convolutional Neural Networks Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning? Combining Deep and Handcrafted Image Features for MRI Brain Scan Classification Recent advances in convolutional neural networks A Novel Method for Classifying Liver and Brain Tumors Using Convolutional Neural Networks, Discrete Wavelet Transform and Long Short-Term Memory Networks Classification of COVID-19 patients from chest CT images using multi-objective differential evolution-based convolutional neural networks An overview of deep learning in medical imaging focusing on MRI Deeplearning convolutional neural networks accurately classify genetic mutations in gliomas The vanishing gradient problem during learning recurrent neural nets and problem solutions A Gentle Tutorial of Recurrent Neural Network with Error Backpropagation Within the Lack of Chest COVID-19 Xray Dataset: A Novel Detection Model Based on GAN and Deep Transfer Learning, Symmetry 2020 On-Device COVID-19 Screening using Snapshots of Chest X-Ray COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-Ray Images Using X-ray images and deep learning for automated detection of coronavirus disease Can AI help in screening Viral and COVID-19 pneumonia Automated detection of COVID-19 cases using deep neural networks with X-ray images COVID-ResNet: A Deep Learning Framework for Screening of COVID19 from Radiographs