key: cord-1055394-sfhjrv53 authors: Nandi, Ritika; Mulimani, Manjunath title: Detection of COVID-19 from X-rays using hybrid deep learning models date: 2021-09-21 journal: Res DOI: 10.1007/s42600-021-00181-0 sha: 106a9c5b3a3a25df1299a66f14ace39e0333daa5 doc_id: 1055394 cord_uid: sfhjrv53 PURPOSE: To propose a model that can detect the presence of Covid-19 from chest X-rays and can be used with low hardware resource-based personal digital assistants (PDA). METHODS: In this paper, a hybrid deep learning model is proposed for the detection of coronavirus from chest X-ray images. The hybrid deep learning model is a combination of ResNet50 and MobileNet. Both ResNet50 and MobileNet are light deep neural networks (DNNs) and can be used with low hardware resource-based personal digital assistants (PDA) for quick detection of COVID-19 infection. RESULTS: The performance of the proposed hybrid model is evaluated on two publicly available COVID-19 chest X-ray datasets. Both datasets include normal, pneumonia, and coronavirus-infected chest X-rays and we achieve 84.35% and 94.43% accuracy on Dataset 1 and Dataset 2 respectively. CONCLUSION: Results show that the proposed hybrid model is better suited for COVID-19 detection. The unprecedented increase in the number of cases of the new respiratory disease COVID-19 has left the healthcare sector frantic. Due to this sudden increase, many countries across the world are facing a shortage of resources to battle this pandemic. An important step towards mitigating this crisis is the quick and accurate detection of infected patients in order to utilize these limited resources wisely. The most effective tool for successfully identifying the infected patients is polymerase chain reaction (PCR) testing. PCR tests are utilized to directly sight the presence of the COVID-19 antigen, instead of the body's invulnerable reaction, or antibodies. The tests can tell whether or not someone has the virus very early on, as they detect the viral RNA, which will be present in the body before symptoms start to appear or antibodies are formed (Kent 2020) . However, this is a very complicated manual process that is extensively laborious, and time-consuming. Due to the sheer volume of reported cases, this process is in short supply. It was established by early research (Huang et al. 2020; Ng et al. 2020 ) that patients infected with COVID-19 display abnormalities in the radiography scans. As a result, detection of visual indicators associated with COVID-19 infection through chest radiography images (e.g., X-ray and CT scans) could be used as an alternative to PCR testing successfully. One of the biggest challenges in this avenue is the ability of radiologists to successfully identify and interpret the minute indicators of In an attempt to overcome this restriction, we suggest a novel model based on deep learning which can recognize the indicators of these infections in the chest X-ray images with almost perfect accuracy. However, we wish to stress the fact that we are not advocating for the employment of this model as a substitute to standard testing procedures for the detection of COVID-19, but merely as a sorting tool which can aid clinicians in improved screening (Fig. 1 ). Over the last few years, the quantity of medical information in machine-readable form has increased tremendously, notably in medical imaging. New forms of neural networks have excelled at the task of identifying abnormalities from high-dimensional information, where previous generations of algorithms failed. Tasks like segmentation, detection, and classification from X-ray, CT scans, and MRIs have quickly become the "de facto" standard (Lundervold and Lundervold 2019) . As additional proof, models dependent on neural networks command the leaderboard in most clinical imaging difficulties (Li et al. 2017) . Various deep learning models have been deployed to detect the presence of respiratory diseases, particularly pneumonia from chest X-ray scans (Rajpurkar et al. 2017; Ekagra et al. 2018; Weng et al. 2017) in recent years. These models typically range from basic convolutional neural networks (Jadhav and Yadav 2019) to complex network architectures like DenseNet121 (Rajpurkar et al. 2017; Li et al. 2017) . A benchmark model for the detection of pneumonia from chest radiography images is CheXNet (Rajpurkar et al. 2017 ) which is built upon DenseNet121. Keeping pace with the recent pandemic situation, several researchers have worked on machine learning and deep learning models. Machine learning algorithms like support vector machines (SVM) and random forest have been used by Alqudah et al. (2020) , to detect COVID-19 cases with an accuracy of 90.5% and 81%, respectively. Ghoshal and Tucker (2020) suggested a model based on Bayesian CNN to detect COVID-19 from chest X-ray images and obtained an accuracy of 90%. CNN-based architectures have been proposed by various works (Salman et al. 2020; Narin et al. 2020; Wang and Wong 2020; Mangal et al. 2020 ) that can be used effectively to identify the presence of COVID-19 infection from Chest Radiology images. The COVID-Net architecture, suggested by (Wang and Wong 2020) , is one of the first CNN-based models which identifies COVID-19 as well as bacterial and viral pneumonia while demonstrating significant sensitivity for COVID-19 detection. One disadvantage of this model is that it is an extremely resource-intensive model. COVIDAid model (Mangal et al. 2020) , which is built upon the CheXNet (Rajpurkar et al. 2017) , performs great on the task of detecting COVID-19 from images, achieving 90.5% accuracy. Abbas et al. (2020) proposed a DeTraC (Decompose, Transfer, and Compose) deep convolutional neural network for the classification of COVID-19 chest X-ray images and obtained an accuracy of 93.1% using this architecture. The Covidx-net model (Hemdan et al. 2020) , proposes automatic detection of coronavirus infected persons using chest X-ray images using seven different transfer learning models. They obtained an average accuracy of 91% for COVID cases using VGG and DenseNet models. Unsurprisingly, deep learning models (Salman et al. 2020; Narin et al. 2020; Wang and Wong 2020; Mangal et al. 2020 ) perform better than machine learning models (Alqudah et al. 2020; Ghoshal and Tucker 2020) for the classification task. However, these DNN architectures are primarily built on heavy transfer learning models like DenseNet and VGG. They are deeper, slower, more complicated, and may not be suitable for personal digital assistants. In this paper, we utilize two light frameworks -MobileNet (Howard et al. 2017) , and ResNet50 (He et al. 2016 ) and mould it according to our problem statement, for quick identification of COVID-19 infection. We have chosen two models -COVID-Net (Wang and Wong 2020) and COVIDAid (Mangal et al. 2020 ) as our baseline for an accurate comparison, as we have performed our experiments on the same data split as Covid-Net and COVIDAid models. Two versions of the COVIDx data proposed by Wang and Wong (2020) are used in this paper. The sample-wise distribution of both the datasets is mentioned in Table 1 . For both datasets, we have applied 5-fold cross-validation to the training set. 10% of the training data has been used for validation in each fold. We have used the average accuracy obtained from each fold as the final accuracy. The first version of the dataset consists of 5949 posterior chest radiography images divided into 4 classesnormal, bacterial pneumonia, viral pneumonia, and COVID-19. Two publicly available datasets, (1) COVID chest X-ray dataset by Cohen et al. (2020) and (2) Kaggle chest X-ray images (pneumonia) dataset by Kermany et al. (2018) , are combined to create this dataset. The train/test distribution of the classes is depicted in Fig. 2a . The second version contains 15,476 posterior chest radiography images divided into 3 classes-normal, (2018), which used publicly available CXR data from , and (5) COVID-19 radiography database by Chowdhury et al. (2020) . The train/test distribution of the classes are shown in Fig. 2b . The proposed hybrid model is a combination of ResNet50 and MobileNet, which are explained below in brief. Proposed by He et al. (2016) , ResNet-50 is a light architecture of a deep convolutional neural network. It contains 50 layers, which, instead of learning unreferenced functions, reformulate as learning residual functions with reference to the layer inputs (He et al. 2016 ). The ResNet model consists of a stack of comparable or "residual" blocks. This block acts as a stack of convolutional layers. The output of a block is additionally connected with its own input through an identity mapping path. To preserve the time complexity per layer, the feature mapping is repeatedly down-sampled by strided convolution along with the increase in channel depth (He et al. 2016) . In the ResNet50 model, to allow for a reduced computational load while calculating the 3 * 3 convolution, we have a 3-layer bottleneck block that uses 3 * 3 convolutions to lower and consequently reestablish the channel depth. MobileNet Proposed by Howard et al. (2017) , MobileNet is a deep learning model that is lightweight in nature. The base paper (Howard et al. 2017 ) presents comprehensive tests on the resource and accuracy tradeoffs and demonstrating superior performance compared to other popular models. MobileNet uses depthwise separable convolutions, i.e, it performs a single convolution on each color channel rather than combining all three and flattening it as in standard 2D CNN. This has the effect of filtering the input channels. The model performance increases due to these depthwise convolutions, and the input features are divided into two layers. Each layer is partitioned into the following layer by joining it with the output features until the procedure is finished. ReLU activation function is used between layers in MobileNet architecture (Howard et al. 2017 ). Thus, it allows the non-linear outputs from the previous layer to be flattened and fed as input to the subsequent layer. We believe that our architecture works well as it combines identity mapping of ResNet50, diminishing the vanishing gradient problem and boosting the gradient backward flow in the network, and depthwise separable convolutions of MobileNets, which increases model performance. Large volumes of data are required to train a neural network from scratch. As a result, due to the scarcity of openly available Chest X-Ray samples (especially of COVID-19), we conduct our experiments using two deep learning models, ResNet50 and MobileNet, pre-trained on the ImageNet (Deng et al. 2009 ) dataset. These models are used separately and their individual results are concatenated to produce results. This is followed by two fully connected layers. Our final classification layer consists of 3 classes in the case of normal, pneumonia, COVID-19 configuration, and 4 classes where pneumonia is split into bacterial or viral. The architecture of the model is detailed in Fig. 3 . This model is then trained on the COVID-19 dataset. Adam optimizer is used to update the neural network weights. We use early stopping on the lowest validation loss. The hyperparameters for training are: learning rate=1e-3, batch size=32, patience=15. The problem is treated as a multiclass classification problem, where each sample belongs to one of the 3 or 4 classes. We use the categorical crossentropy loss as given in (1). The parameters chosen for training the models are the recommended values given by previous research which ensure an optimum performance. These recommended values have been taken as the baseline and subsequently, fine tuning of the hyperparameters has been performed to yield better results. The architecture was built and evaluated using the Keras, a python library for deep learning tasks which uses TensorFlow as a backend. The complexity of each model depends on various factors like the number of parameters, time taken to train the model, and optimizers used as shown in Table 2 . Performance metrics like accuracy and area under ROC (AUROC) curves are used to carry out an experimental analysis. We have presented the fold-wise accuracy of two baseline models-ResNet50, and MobileNet, and our proposed hybrid model in Table 3 . The overall accuracy in Table 4 is calculated as the average of accuracies obtained from each fold. Table 4 also presents the class-wise AUROC of each model. The class-wise ROC curves are presented in Fig. 5 . Using the pre-trained ResNet50 model, we obtained an average accuracy of 79.50% and 91.13% on Datasets 1 and 2 respectively. The pre-trained MobileNet model classifies the images with an average accuracy of 83.26% and 93.73% on Datasets 1 and 2 respectively. The best result is obtained by concatenating the individual outputs of ResNet50 and MobileNet, before passing them through the fully connected layer. Using this hybrid approach, we obtain an average accuracy of 84.35% and 94.43% on Datasets 1 and 2 respectively. This architecture leads to the detection of COVID-19 from the radiography images with an AUROC (Area under ROC curve) of 1.00 for the 4-class classification and 0.99 for the 3-class classification. The confusion matrix obtained from the hybrid model, for both the versions of the datasets, is shown in Fig. 4 . In Dataset 1, it can be seen that some pneumonia cases have been misclassified as normal cases (Fig. 4a) . We believe that the misclassified images may have a lower viral or bacterial infection load, due to which they are being misclassified as normal cases. However, this is just a hypothesis on our part (Fig. 5) . Table 5 shows the difference in the performance of the ResNet50 and MobileNet models, and the ResNet50 + MobileNet hybrid model proposed in this paper by Ttest. There were statistically significant differences between ResNet50 and MobileNet models, when compared with the hybrid model (p < 0.5). In order to obtain a general picture of the performance of our proposed model on the two case studies, we compare our model with COVIDAid (Mangal et al. 2020 ) and Covid-Net (Wang and Wong 2020) by evaluating on two separate datasets to eliminate intrinsic biases and impart robustness to the findings. Since we have used the same dataset and data split as the COVID-Net model, we use the values obtained from (Wang and Wong 2020) as it is. In the same way, for COVIDAid, we evaluate their approach on the same data split as ours and obtain the accuracy, ROC curve, and Confusion Matrix. As shown in Table 6 , we obtain higher accuracy than the COVID-Net and COVIDAid models on Table 6 also shows that there are statistically significant differences between both baseline models, and the hybrid model (p < 0.5). Hence, the hybrid model is more effective in coronavirus detection. Fig. 6a and b presents the AUROC obtained from the COVIDAid model. Comparing with the ROC with our proposed hybrid approach in Fig. 5e and f, in 4 class classification, our model performs comparable to the COVIDAid model in normal, bacterial pneumonia, and COVID-19 classes, but performs significantly better in the viral pneumonia class. In 3 class classification, our model performs better in the normal and COVID-19 classes than the COVIDAid model. This hybrid approach clearly establishes a higher baseline for detection of pneumonia or COVID-19. In this paper, ResNet50, MobileNet, and a hybrid approach combining ResNet50 and MobileNet are used for the detection of COVID-19 from chest X-rays. The performance of the proposed model is evaluated on two publicly available datasets. Results show that our approach outperforms competitive COVID-Net and COVIDAid. It indicates that the proposed model has a significant contribution to the detection of COVID-19 infected chest X-rays. Proposed models for COVID-19 detection are light DNN as compared to other DNNs and can be used with low resource devices for quick identification of coronavirus. Data availability The data that support the findings of this study are openly available and details regarding the same are mentioned in Section 1. The code that supports the findings of this study is not openly available currently and will be made available after the publication of the work. Classification of covid-19 in chest x-ray images using detrac deep convolutional neural network Covid-2019 detection using X-ray images and artificial intelligence hybrid models Can ai help in screening viral and covid-19 pneumonia Actualmed covid-19 chest x-ray dataset initiative Figure 1 covid-19 chest x-ray dataset initiative Covid-19 image data collection ImageNet: A large-scale hierarchical image database Jointly learning convolutional representations to compress radiological images and classify thoracic diseases in the compressed domain Estimating uncertainty and interpretability in deep learning for coronavirus (covid-19) detection Deep residual learning for image recognition Covidx-net: a framework of deep learning classifiers to diagnose covid-19 in x-ray images Mobilenets: efficient convolutional neural networks for mobile vision applications Clinical features of patients infected with 2019 novel coronavirus in Wuhan Deep convolutional neural network based medical image classification for disease diagnosis Different paths to the same destination: screening for covid-19 Labeled optical coherence tomography (oct) and chest x-ray images for classification Learning to diagnose from scratch by exploiting dependencies among labels An overview of deep learning in medical imaging focusing on mri Covidaid: Covid-19 detection using chest x-ray Automatic detection of coronavirus disease (covid-19) using x-ray images and deep convolutional neural networks Imaging profile of the covid-19 infection radiologic findings and literature review Radiologist-level pneumonia detection on chest x-rays with deep learning Rsna pneumonia detection challenge Covid-19 detection using artificial intelligence Covid-net: a tailored deep convolutional neural network design for detection of covid-19 cases from chest x-ray images Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases Summers RM. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases Chexnet for classification and localization of thoracic diseases Publisher's note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.