key: cord-0786583-5ti1ij9j authors: Das, Debanjan; Ghosal, Sagnik; Mohanty, Saraju P. title: CoviLearn: A Machine Learning Integrated Smart X-Ray Device in Healthcare Cyber-Physical System for Automatic Initial Screening of COVID-19 date: 2022-02-03 journal: SN Comput Sci DOI: 10.1007/s42979-022-01035-x sha: ab3cb2d71139bc0e5a7d552a26ce21b1d2d4a1ab doc_id: 786583 cord_uid: 5ti1ij9j The pandemic of novel Coronavirus Disease 2019 (COVID-19) is widespread all over the world causing serious health problems as well as serious impact on the global economy. Reliable and fast testing of the COVID-19 has been a challenge for researchers and healthcare practitioners. In this work, we present a novel machine learning (ML) integrated X-ray device in Healthcare Cyber-Physical System (H-CPS) or smart healthcare framework (called “CoviLearn”) to allow healthcare practitioners to perform automatic initial screening of COVID-19 patients. We propose convolutional neural network (CNN) models of X-ray images integrated into an X-ray device for automatic COVID-19 detection. The proposed CoviLearn device will be useful in detecting if a person is COVID-19 positive or negative by considering the chest X-ray image of individuals. CoviLearn will be useful tool doctors to detect potential COVID-19 infections instantaneously without taking more intrusive healthcare data samples, such as saliva and blood. COVID-19 attacks the endothelium tissues that support respiratory tract, and X-rays images can be used to analyze the health of a patient’s lungs. As all healthcare centers have X-ray machines, it could be possible to use proposed CoviLearn X-rays to test for COVID-19 without the especial test kits. Our proposed automated analysis system CoviLearn which has 98.98% accuracy will be able to save valuable time of medical professionals as the X-ray machines come with a drawback as it needed a radiology expert. Coronavirus disease (COVID-19) is a respiratory tract infectious disease that has spread across the world [1] . It belongs to a family of viruses whose infection can cause complications that vary from typical cold to shortness of breath [2] . Patients also develop pneumonia termed, Novel Coronavirus Pneumonia (NCP), that results in acute respiratory failure with a very poor prognosis and high mortality [3, 4] . Subsequently, the pandemic nature of the coronavirus and the absence of reliable vaccines make COVID-19 diagnosis an urgent medical crisis. At present, the standard testing method for COVID-19 diagnosis is the real-time Reverse Transcription Polymerase Chain Reaction (rRT-PCR) test. In this test, nasal swab is collected from the patient and kept in a special medium called the "virus transport medium", to protect the RNA. Upon reaching the lab, the swab is further processed to determine whether or not the patient is positive for the coronavirus [5] . The entire process takes several hours and the results generally arrive after a day or two depending on the time taken from the swab to reach the lab. The spread of the COVID-19 virus at this point advocates the requirement of its quick diagnosis and treatment. Studies such as [6, 7] have proved that the COVID-19 virus infects the lungs and creates smooth and thick mucus in the patient's affected lungs that is visible when chest X-rays and CT scans are performed. However, the analysis of X-ray images is a tedious task and require expert radiologists. In this endeavor, several computer algorithms and diagnosis tools such as [8, 9] have been proposed to get detailed insights from the X-ray images. Although these studies have performed efficiently, they lack in terms of higher accuracy, generalization, computational time, and error rate. To mitigate the shortcomings, recent studies such as [10] [11] [12] [13] have incorporated machine learning (ML) and deep learning (DL) tools to investigate the chest X-ray images. The selection of proper DL-based automated analyzer and predictor for coronavirus patients will be very beneficial and helpful for the medical department and society. Additionally, ML-DL approaches can provide test results faster and more economically as compared to the laboratory-based tests. Furthermore, as COVID-19 is spreading rapidly through person-to-person contact, hospitals and healthcare professionals are becoming increasing overburdened, sometimes to the point of complete breakdown. Clearly, an alternative, remote-based, online diagnostic and testing solution is required to fill this urgent and unmet need. The Internet of Medical Things (IoMT) could be extended to achieve this healthcare-specific solution. With this motivation, the present work proposes an AI-based Healthcare Cyber-Physical System (H-CPS) that incorporates convolutional neural networks (CNNs) (see Fig. 1 ). The model allows healthcare practitioners to promptly and automatically screen positive and negative COVID-19 patients by considering their chest X-ray images. The organization of the paper is as follows: "Related prior research works" discusses the working of existing COVID-19 detection models, their shortcomings, and our contributions in the H-CPS framework. "Proposed CoviLearn model for automatic initial screening of COVID-19" explains the proposed solution and its functioning, followed by "Performance evaluation" that validates the model using real-life data. Finally, "Conclusions and future scope" gives a compact conclusion and mentions the area of future study. Over the course of 2 years, many techniques have been proposed for effective COVID-19 detection [14] . However, from the exhaustive list of works, we have selected some of the state-of-the-art methods focusing only on the deep learning based COVID-19 detection. A CNN called COVIDNet was trained in [15] using more than 15000 chest radiography images of COVID-19 positive and negative cases. The deep neural network (DNN) reported accuracy of 92.4% and sensitivity of 80% . A three-dimensional convolutional ResNet-50 network, termed COVNet, was proposed in [16] that utilized volumetric chest CT images consisting of community acquired pneumonia (CAP) and other non-pneumonia cases. The reported AUC metric by the model was 0.96. A similar ResNet-50 model proposed by [17] reported an AUC of 0.996 although tested to a much lesser dataset. In [18] , a location-attention network using ResNet-18 was proposed using disparate CT samples from COVID-19 patients, influenza-A infected, and healthy individuals to classify COVID-19 cases, that reported an accuracy of 86.7% . Samples from 4 classes: healthy, bacterial pneumonia, non-COVID-19 pneumonia, and COVID-19 were used in [19] to train dropweight-based Bayesian CNNs that reported an accuracy of 89.92%. In [20] , a modified inception transfer-learning model that reported an accuracy of 79.3% , specificity of 0.83, and sensitivity of 0.67 was proposed. In [21] , a multilayer perceptron combined with an LSTM neural network was implemented, that was trained using clinical data collected from 133 patients out of which 54 belonged to the critical care domain. The authors in [22] implemented a two-dimensional deep CNN architecture, while the authors in [23] combined threedimensional UNet and ResNet-50 architectures. Both were trained using volumetric CT scanned data of patients categorized as COVID-19 positive and negative. The method in [24] used a pre-trained ResNet-50 network using chest X-ray images from 50 COVID-19 positive and 50 COVID-19 negative patients and reported an accuracy of 98% . In [25] four state-of-the-art DNNs: AlexNet, Resnet-18, DenseNet-201, and SqueezeNet were ensembled. The model also used chest X-ray images of normal, viral pneumonia, and COVID-19 cases. A novel CNN augmented with a pre-trained AlexNet using transfer learning was proposed in [26] . The model was tested on both X-ray and CT scanned images with reported accuracies of 98% and 94.1% , respectively. Although the domain is very new and many studies pertaining to the deep learning-based methodology have been proposed, most of them suffer from shortcomings such as lower accuracy, model generalization, computational cost, and error rate. Even when certain research works achieve higher accuracy, they either suffer from lower sensitivity, specificity or have a small test dataset. Moreover, the prospect of augmenting IoMT frameworks with COVID-19 diagnosis is new and its incorporation can further assist the existing healthcare system to cope in this difficult times. Also, the training dataset for certain methods is limited because of class imbalance, that is, less number of coronavirus images as compared to normal lung images. This problem of dataset imbalance results in lesser model accuracy and less efficient. Table 1 provides a comprehensive comparison of the existing research works. We propose an AI-based H-CPS framework termed "Cov-iLearn" to provide healthcare professionals the leverage to perform automatic screening of COVID-19 patients using their chest X-ray images. With a deep neural network (DNN) in its core, the CoviLearn model is implemented on the server for ubiquitous deployment. The hyperparameters of the DNN have been adjusted to make its functioning reliable, accurate, and specific. By just uploading the X-ray images, the model automatically identifies the symptoms and reports unbiased results. CoviLearn augmented with H-CPS brings patients, doctors, and test lab in a single smart healthcare platform, as illustrated in Fig. 2 . The reported results can be uploaded to the IoMT platform from where it may be transferred to nearby COVID-care hospitals, the Center for Disease Control (CDC), and state and local health bureaus. Hospitals could subsequently offer online health consultations based on the patient's condition and monitor vital equipments and quarantine requirements. Therefore, the proposed H-CPS provides people the leverage to dynamically monitor their disease status, receive proper medical needs, and eventually curb the spread of the virus. The major contributions of the work are: -An architecture of H-CPS framework augmented with a next generation smart X-ray machine architecture at the interface is proposed to combat the spread of COVID-19. -An efficient heuristic search technique is incorporated which automatically finds an optimal feature subset present in the input chest X-ray images. -An end-to-end automatic functioning DNN model that extracts the features from X-ray images is incorporated. -The CNN blocks are reliable, accurate, and very specific that makes the overall model very effective. Furthermore, the model can be easily integrated into embedded and mobile devices, thereby assisting health practitioners to effectively diagnose COVID-19. The CoviLearn Device for Next-Generation X-ray Screening As discussed in the earlier sections, COVID-19 and other related pneumonia diseases can be screened and diagnosed by analyzing chest X-ray images. However, the existing X-ray diagnosis suffers from limited access and lack of experienced personnel. To address this issue, we propose a next-generation X-ray system in the H-CPS perspective. The H-CPS and IoMT together bring all the necessary agents of smart healthcare in a universal communication and connectivity platform. This linking of technologies extends the efficiency services such as telemedicine, teleconsultation, and endorse smart-medical care. Figure 3 shows the system-level block diagram of the next-generation X-ray machine integrated with CoviLearn for automatic screening of infectious diseases. It identifies most of its components, such as X-ray apparatus (tube), flat panel detector, onboard memory, DICOM protocol converter, Image processing, CoviLearn diagnosis, wired/wireless data communication, display, or user interface, along with system controller. In the proposed X-ray machine, X-ray image is captured by an array of sensor in the digital and radiography flat panel detector. The flat panel also includes the devices of communication to next stages. The image is then saved and converted to DICOM X-ray image. Subsequently, the image is processed and based on the quality and requirement the exposure of the X-ray tube is adjusted. The captured image is stored temporarily in the local memory, after which it is displayed on monitor screen with the help of the controller. After acquiring the quality assured image, it is then transferred to the CoviLearn model which automatically classifies the image either as normal or COVID-19 affected. The image classification is performed either locally in the presence of sufficient resources or on cloud by transmitting the images over network. The test results automatically synchronize with the H-CPS platform for necessary medical and administrative actions. The controller unit is responsible for controlling the entire sequence of events. Fig. 3 The proposed next-generation X-ray device of CoviLearn integrated with machine learning models Page 6 of 11 To overcome the problem of class imbalance, we have manually collected chest X-ray of patients having coronavirus. These images are from various resources such as pyimagesearch, radiopedia, sirm, and eurorad. For the normal chest X-ray, we have used the chest X-ray dataset from the National Institute of Health (NIH), USA [27] . The count of images from both the sources was 250. Subsequently, the dataset has been divided into two classes: patient's diagnosed as COVID-19 positive and negative. For training 80% of the dataset ( ∼ 200 images) is used from which 30% is used for validation ( ∼ 60 images). The testing of the model is performed on 20% ( ∼ 50 images) of the dataset. Based on this validation dataset, the loss and validation graphs have been plotted. All the images are processed and mixed to prevent undue biasing as discussed in the following subsections. All the captured images have different sizes, and therefore, data pre-processing was essential before doing further analysis. The pre-processing is performed in three stages: first, the individual data are normalized by subtracting the mean RGB values; second, all the pixels in the input image data are scaled within the range of 0 to 1. Finally, the tensor is reshaped appropriately, so that it fits the model (in this case, the tensor is reshaped into 224 × 224 pixels). Deep learning models are ravenous for data and since our model only has around 250 images for each class; hence, the volume of our data needs to be increased and this can be achieved through data augmentation. Therefore, similar to the process mentioned in [28] , the input images are augmented by random crop, adjust contrast, flip, rotation, adjust brightness, horizontal-vertical shift, aspect ratio, random shear, zoom, and pixel jitter. As a result of this augmentation, the proposed CoviLearn system became more efficient. CoviLearn uses transfer learning to predict the classification results. Transfer learning substitutes for the requirement of large dataset and has been used in different applications, such as healthcare, manufacturing, etc. It uses the knowledge learned in training a large dataset and transfers that same knowledge in some different and smaller dataset. In the present work, four different DNNs: ResNet-50, ResNet-101, DenseNet-121, and DenseNet-169, along with different blocks to train the individual networks. The hyperparameters have been adjusted to report the highest accuracy. Detailed structural organizational of network layers is as illustrated in Fig. 4 where each network is divided into phases, starting from getting an image input, followed by training the model by sequentially passing the set of images into convolutional networks, to finally predicting the results using a classification layer. Following subsection discusses the base classifiers and the difference between them. The CoviLearn model uses four deep neural networks as the base classifiers. Two of these belong to the ResNet family [29] (ResNet-50 and ResNet-101) and remaining two belong to the DenseNet family [30] (DenseNet-121 and DenseNet-169). As the convolutional neural networks become deeper, the back propagated error from any layer is required to traverse the entire depth where repeated weight multiplications occur. As a result of these multiplications, the original error significantly diminishes and the neural network's performance is satisfactorily affected. To combat this, researchers have proposed many architectures, out of which the current state-of-the-art includes the DenseNet and the ResNet models. DenseNet or Dense Convolutional Network solves the problem using shorter connections between the layers. In other words, inside the DenseNet network, the each layer is connected to all its higher layers. Equation (1) This arrangement allows feature reusing without having to travel the entire depth or entire depth of the network. In comparison to a traditional CNN, DenseNet requires fewer parameters, because features learned in one layer are sent to the higher layers, thereby eliminating redundancy. A typical DenseNet architecture involves a convolution layer, followed by a pooling layer. These are followed by 4 dense blocks and 3 transition blocks placed one after the other. Inside the dense block, there are two convolutional layers with filters of different sizes, while the transition layer involves an average pooling layer. The dissimilarity between the DenseNet-121 and DenseNet-169 networks is with respect to the number of hidden layers. For the former, the total number of convolution layers in the four Dense Blocks is 121, while for the latter that is 169. Increasing the layers does not necessarily improve the accuracy and depends upon the particular situation. Residual Networks or ResNet solves the problem of vanishing gradient decent by utilizing a skip connection between the original input and the final convolution layers. By overlooking the in between layers and attaching the given input directly to the output allows the presence of an additional path for the back propagated error to flow and therefore solving the problem of vanishing gradient descent. For a DenseNet, the equation changes to (3) A typical ResNet architecture involves four stages. The first stage is responsible for performing zero-padding operation on the input data. The second stage is made up of convolutional blocks that performs convolution along with batchnormalization and max-pooling. The penultimate layer consists of identity blocks augmented with filters, followed by the final stage that comprises a GAP layer, a fully connected dense layer, and classifier function. All convolution layers use ReLU as the activation function. Similar to DenseNet, the two types of ResNets that is ResNet-50 and ResNet-101 differ in the depth of the network. It has been observed that certain variations of ResNet have redundant layers that barely contribute. The presence of them results in ResNet handling larger parameters and weights. On the other hand, DenseNet are relatively narrow (fewer number of filters) (2) P l = T l [P 0 , P 1 , P 2 , … , P l−1 ]. (3) P l = T l (P l−1 ) + P l−1 . and simply add the new feature maps. Another difference between the DenseNet and ResNet models is that the former does not sum the output feature maps of the preceding layers but rather concatenates them, unlike the latter where summation happens. This is evident from Eqs. (2) and (3). The CoviLearn model takes the input image, swaps the color channels, and resizes it to 224 × 224 pixels. Afterwards, the data and label list are converted into an array, while the pixel intensities are normalized between 0 and 1, by dividing the entire input image by 255. Subsequently, one-hot encoding is performed on the labels, following which various models are loaded one at a time by freezing few upper layers and a base layer is created with dropout. Finally, the input tensor of size 224 × 224 is loaded onto the model and compiled using Adam optimizer and binary cross entropy loss. To compare the performance of different models, three evaluation parameters: accuracy, sensitivity, and specificity have been considered. As the test images are converted into 224 × 224 tensor, the model predicts the above-mentioned three metrics. Table 2 In context of coronavirus detection, True Positive (TP) is when the patient has coronavirus and the model detects coronavirus, True Negative is when the patient does not have coronavirus and the model also predicts the same. False Positive (FP) is when the the patient is not infected with (4). Additional metrics such as sensitivitythe ability to identify coronavirus patients correctly-and selectivity-the ability to identify non-coronavirus patients correctly-are as defined by Eqs. (5) and (6), respectively Figure 5 shows the confusion matrices of COVID-19 and normal test results of the different pre-trained models. The graphs show a well-defined pattern of the training-validation accuracy that increases, and the training-validation loss that decreases, with increasing epochs. Because of the limited computational resources, the comparison between different parameters is done for 25 epochs only. Besides the confusion matrix, receiver-operating characteristic (ROC) curve plots and areas for each model are given in Fig. 6 . DNNs which are trained with DenseNet pre-trained blocks appear to be very higher than DNN trained with ResNet blocks, with DNN III having the highest AUC of 99% . One of the interesting findings is the DNN which when used with the ability of the DenseNet model achieves higher sensitivity and specificity. This ensures the reduction of false positives for both the COVID-19 and the healthy classes. As is evident from the relationship between accuracy and epoch, DNN-III shows the highest accuracy followed by DNN-IV, DNN-II, and DNN-I. The accuracy increases with each subsequent epoch except at few as illustrated in Fig. 7 . A similar trend is shown in loss graphs where the loss decreases with each subsequent epochs and a similar trend is followed, that is, DNN-III shows the lowest loss followed by DNN-IV, DNN-II, and DNN-I (see Fig. 8 ). The results as reported by the proposed CoviLearn model are compared with the existing research works and tabulated in Table 3 . In [18] , detects COVID-19 using classification of CT samples by CNN models with an accuracy of 86.7% , sensitivity of 98.2% , and specificity of 92.2% . CovidNet in [15] reported an accuracy of 93.3% . The CNN-based DarkCovidNet model [31] to detect COVID-19 from chest X-ray also has an accuracy of 98.08% . In comparison, the proposed model has an accuracy of 98.98% , sensitivity of 0.984, and specificity of 0965. CoviLearn has significantly outperformed existing deep learning-based COVID-19 detection techniques such as [15, [18] [19] [20] 23] . Also, the sensitivity of the proposed model has outperformed existing models such as [15, 20, 23] both in terms of sensitivity and specificity. [17, 21, 24] achieved similar accuracy; however, their test dataset size is relatively smaller than the one used in the current work. The deep neural architectures proposed in [25, 26] involved many hyperparameters, estimation of which increased the overall computation cost and resulted in ubiquitous deployment. On the other hand, CoviLearn because of its transfer-learning ability and selected deep neural networks has the advantage of rejecting redundant parameters and thereby reducing the overall computational cost. Finally, all these models lacked a smart healthcare framework, which has been proposed and implemented in CoviLearn in the form of H-CPS. The comparison of existing research works is compactly summarized in Tables 3 and 4 . The initial neural network when trained reported accuracy, sensitivity, and specificity values of 0.5981, 0.6041, and 0.5923, respectively. To improve these substantially, we used the concept of transfer learning. It is done by freezing the layers of the existing models and replacing with the penultimate layer (the layer responsible for performing classification) of state-of-the-art neural networks trained on larger datasets to perform final classification. This step improved the accuracy, sensitivity, and specificity metrics to 0.9225, 0.9319, and 0.9135, respectively. Following this step, finetuning is performed on the model's hyperparameters to further improve the model's performance by ∼ 5% . Therefore, despite a small training dataset of 250 images, embedding the transfer learning helped improve the model's classification performance significantly. Table 5 compares the metrics obtained in each of the stages. The study presents CoviLearn, a DNN-based transferlearning approach in Healthcare Cyber-Physical System framework to perform automatic initial screening of COVID-19 patients using their chest X-ray image data. An architecture of next-generation smart X-ray machine for automatic screening of COVID-19 is proposed at the interface of H-CPS. ResNet-101 are close to 97% . The highest specificity of DNN III is 98%. Therefore, all these results clearly indicate the ability to classify the deadly coronavirus correctly. The present CoviLearn platform will be very useful tool doctors to diagnosis the coronavirus disease at a lower cost despite being economical and automatic. However, additional study and medical trial are required to full proof the extracted features extracted by machine learning as reliable bio-markers for COVID-19. Furthermore, these machine learning models can be extended to diagnose other chest-related diseases including tuberculosis and pneumonia. A limitation of the study is the use of a limited number of COVID-19 X-ray images. Therefore, in the future, a larger dataset and a cloud based system can be ventured to make the model ubiquitous and more robust. In fact, the results can be used to detect the highly prone corona positive patients in a timely application of quarantine measure, until the rRT-PCR test examinations results are obtained. The proposed CoviLearn can be added to our healthcare CPS framework CoviChain for reliable information sharing right from the source to destination end while accommodating various stake holders [34] . Wang et al. [15] 92. 4 15,000 Xu et al. [18] 86.7 618 Ghoshal et al. [19] 89.92 5941 Wang et al. [20] 79.3 1065 Jin et al. [22] 94.98 2355 Narin et al. [24] 98 100 Chowdhury et al. [25] 98.3 2876 Maghdid et al. [26] 98 (X-ray), 94.1 (CT) 531 CoviLearn 98.98 250 A novel coronavirus outbreak of global health concern Clinical features of patients infected with 2019 novel coronavirus in Wuhan Risk of mortality in patients infected with sars-cov-2 variant of concern 202012/1: matched cohort study Review of the clinical characteristics of coronavirus disease 2019 (COVID-19) Detection of coronavirus disease (COVID-19) based on deep features Radiological society of North America expert consensus statement on reporting chest CT findings related to COVID-19. Endorsed by the society of thoracic radiology, the American college of radiology, and RSNA Automatic lung segmentation in chest radiographs using shadow filter and multilevel thresholding Computer-aided detection in chest radiography based on artificial intelligence: a survey Efficient deep network architectures for fast chest X-ray tuberculosis screening and visualization Automatic lung cancer prediction from chest X-ray images using the deep learning approach Supervised and unsupervised language modelling in chest X-ray radiological reports Lung infection quantification of COVID-19 in CT images with deep learning Consumer technology-based solutions for COVID-19 Covid-net: a tailored deep convolutional neural network design for detection of COVID-19 cases from chest x-ray images Artificial intelligence distinguishes COVID-19 from community acquired pneumonia on chest ct Rapid ai development cycle for the coronavirus (COVID-19) pandemic: Initial results for automated detection & patient monitoring using deep learning CT image analysis A deep learning system to screen novel coronavirus disease Estimating uncertainty and interpretability in deep learning for coronavirus A deep learning algorithm using CT images to screen for Corona Virus Disease (COVID-19) Deep learning for predicting COVID-19 malignant progression Development and evaluation of an artificial intelligence system for COVID-19 diagnosis Ai-assisted CT imaging analysis for COVID-19 screening: building and deploying a medical ai system in four weeks Automatic detection of coronavirus disease (COVID-19) using x-ray images and deep convolutional neural networks Can AI help in screening viral and COVID-19 pneumonia? Diagnosing COVID-19 pneumonia from x-ray and CT images using deep learning and transfer learning algorithms Nih clinical center-cxr8 A survey on image data augmentation for deep learning Deep residual learning for image recognition Densely connected convolutional networks Automated detection of COVID-19 cases using deep neural networks with X-ray images Pneumonia identification in chest X-Ray images using EMD A deep feature learning model for pneumonia detection applying a combination of mrmr feature selection and machine learning models CoviChain: a blockchain based framework for nonrepudiable contact tracing in healthcare cyber-physical systems during pandemic outbreaks We would like to acknowledge IIIT-NR undergraduate students Chirag Samal, Deewanshu Ukey, and Gourav Chowdhary for their help on this paper. The authors declare that they have no conflict of interests.