key: cord-1039606-ndmpgzg7
authors: Shankar, K.; Perumal, Eswaran
title: A novel hand-crafted with deep learning features based fusion model for COVID-19 diagnosis and classification using chest X-ray images
date: 2020-11-12
journal: Complex Intell
DOI: 10.1007/s40747-020-00216-6
sha: 6758aadeda8bb22c55a9f2b81a90b735654fd0a3
doc_id: 1039606
cord_uid: ndmpgzg7

COVID-19 pandemic is increasing in an exponential rate, with restricted accessibility of rapid test kits. So, the design and implementation of COVID-19 testing kits remain an open research problem. Several findings attained using radio-imaging approaches recommend that the images comprise important data related to coronaviruses. The application of recently developed artificial intelligence (AI) techniques, integrated with radiological imaging, is helpful in the precise diagnosis and classification of the disease. In this view, the current research paper presents a novel fusion model hand-crafted with deep learning features called FM-HCF-DLF model for diagnosis and classification of COVID-19. The proposed FM-HCF-DLF model comprises three major processes, namely Gaussian filtering-based preprocessing, FM for feature extraction and classification. FM model incorporates the fusion of handcrafted features with the help of local binary patterns (LBP) and deep learning (DL) features and it also utilizes convolutional neural network (CNN)-based Inception v3 technique. To further improve the performance of Inception v3 model, the learning rate scheduler using Adam optimizer is applied. At last, multilayer perceptron (MLP) is employed to carry out the classification process. The proposed FM-HCF-DLF model was experimentally validated using chest X-ray dataset. The experimental outcomes inferred that the proposed model yielded superior performance with maximum sensitivity of 93.61%, specificity of 94.56%, precision of 94.85%, accuracy of 94.08%, F score of 93.2% and kappa value of 93.5%.

Coronavirus belongs to a huge family of viruses, which generally cause mild-to-moderate upper-respiratory tract illness similar to cold, namely Middle East respiratory syndrome (MERS) and Severe Acute Respiratory Syndrome (SARS) [1] . These illnesses generally occur in a wide range of animal species; however, in diverse cases, they tend to mutate and infect human beings quickly and spread to other people in an easier way. By the end of 2019, coronavirus 2019 (COVID-19, acronym of COronaVIrus Disease 19) started infecting human beings. The first case was identified by December 2020 surpassing China. In medical perspective, COVID-19 inflammation generates a bunch of incurable pneumonia with medical issues alike SARS-CoV. Generally, patients experience influenza-like signs, such as difficulty in breathing, dry cough, tiredness and fever. In serious cases where the person has comorbidities, i.e., affected by other diseases like blood pressure, diabetes or heart problems, pneumonia develops rapidly resulting in acute renal failure and finally death in worst cases. But several patients are diagnosed with COVID-19 without symptoms. In Vo' Euganeo, 50 km west of Venice, the total population of the country was made to undergo pharyngeal swab test while 50-75% of the populations were tested positive in swab, yet remained asymptomatic.

At present, the best method to determine COVID-19 is to perform swab test and examine the biotic material collected from patients using real-time reverse transcriptase polymerase chain reaction (RT-PCR). However, it is a challenge that the swab test is taken only for those individuals with COVID-19 symptoms. The existing COVID-19 patients without symptoms could not be recognized, until they approach the hospitals. Though the disease can be diagnosed by polymerase chain reaction, COVID-19 patients who are infected with pneumonia can be diagnosed using chest X-rays and computed tomography (CT) images only. In one of the studies conducted recently, COVID-19 can be slightly identified by human eye too [2] . COVID-19 transmission rate is calculated on the basis of volume of affected patients who are consistently diagnosed with minimum false negatives. Additionally, a less false-positive rate is essential to ensure not to push the medical system to extreme ends, by unreasonably revealing patients to isolation. With suitable contamination controller, it is proved that the earlier discovery of diseases enables the execution of helpful care essentials to COVID-19 patients.

By the end of January 2020, China conducted a research upon COVID-19 in terms of medical and paramedical specifications. The research conveyed that the COVID-19 cases exhibited some abnormal behaviors in chest CT scan images. World Health Organization (WHO) issued some other diagnostic protocols. Diagnosis is performed by real-time reverse transcriptase polymerase chain reaction (rRT-PCR) examination on biotic samples collected from patients. The experiments can be conducted in blood samples and the results are mostly obtained in within limited hours or within a day. As demonstrated earlier, COVID-19 can be probably deducted well by radiological images. Therefore, in this research, the authors estimate the prospects for the deduction of COVID-19 disease directly from medical images and X-ray scans.

Machine learning (ML)-based applications are currently employed for automatic disease diagnosis in healthcare sector [3] . DL is one of the common research domains in AI which allows the creation of end-to-end technique to attain assured outcomes. This is done utilizing intake data with-out any manual feature extraction. DL method has been effectively used in a number of issues like lung segmentation, skin cancer classification, fundus image segmentation, brain disease classification, pneumonia detection from chest X-ray images, breast cancer detection, and arrhythmia detection. Coronavirus pandemic is quickly raising the need for knowledge in this domain. It has improved awareness and emphasized the need for automatic detection technique based on AI. It is a risky process to provide radiologists for all the hospitals because of the scanty skilled manpower. Thus, the modest, precise, and fast AI methods might be useful to overcome these issues and give support to patients in correct time [4] [5] [6] .

This paper introduces an effective fusion model (FM), hand-crafted with deep learning features called FM-HCF-DLF model for diagnosis and classification of COVID-19. The proposed FM-HCF-DLF model comprises three major processes, namely Gaussian filtering (GF)-based preprocessing, FM for feature extraction and classification. FM model incorporates the fusion of handcrafted features (HCF) using local binary patterns (LBP), whereas deep learning features (DLF) utilize convolutional neural network (CNN)-based Inception v3 approach. To further improve the performance of Inception v3 model, a learning rate scheduler using Adam optimizer has been applied in the current study. Finally, multilayer perceptron (MLP)-based classification process was executed to classify COVID-19 into different sets of classes. The proposed FM-HCF-DLF model was experimentally validated using chest X-ray dataset and the experimental outcome defined the superior performance of the presented model.

With the advancements in healthcare image processing methods, there is a drastic increase observed in prediction and diagnostic devices [7] . ML methods are broadly known as projected tools to improve the diagnostic and prediction processes of numerous diseases [8] . Though effective feature extraction methods [9] are required to attain efficient ML techniques, DL is an extensive method which is approved in healthcare image system, thanks to its automated extraction feature like ResNet. Yu et al. [10] utilized Conventional Neural Network for classification of COVID-19-affected patients using chest CT imaging. Nardelli et al. [11] employed 3-DCNN to distinguish the respiratory artery veins from chest CT imaging. Shin et al. [12] utilized deep CNN to categorize the interstitial lung disease from CT imaging.

Xie et al. [13] categorized benign (lesion less than 3 cm) and malignant (lesion more than 3 cm) tumors based on pulmonary nodule classification. The study [14] arranged the melanoma dermoscopy images by DL with outstanding accu-racy. The authors [15] observed the respiratory fissure in CT with the help of supervised discriminative learning platform. Setio et al. [16] implied multi-view convolutional networks to detect the lung nodules in CT imaging. Xia et al. [17] suggested deep adversarial networks to achieve segmentation on stomach CT imaging. Pezeshk et al. [18] utilized 3-D CNN to diagnose the pulmonary nodules in chest CT images. Zreik et al. [19] used a classifier method for recurrent CNN in the classification of Coronary Artery Plaque and Stenosis in Coronary CT.

Bhandary et al. [20] recommended a method to diagnose other respiratory disorder with the help of DL platform. Gao et al. [21] employed 3D block-based residual deep learning framework to detect severe stages of tuberculosis in CT scan and lungs' X-ray imaging. Singh et al. [22] introduced particle swarm optimization related to adaptive neuro-fuzzy inference system (ANFIS) to improve the rate of classification. Zeng et al. [23] applied gated bidirectional CNNs (GCNN) which can be used for classifying COVID-19-affected patients. Based on in-depth analysis, it is determined that DL technique might attain effective outcomes for COVID-19 disease classifier from lung CT imaging. But these outcomes can be enhanced further if effective feature methods like variants of ResNet are used. In addition to this, the DL approaches can be hyper-tuned by transfer learning. Thus, a new deep transfer learning (DTL) development, related to COVID-19-affected patient classifier technique, is the major motivation behind the current research work. 

The execution of 2D Gaussian filter is employed extensively for smoothing and noise elimination. It needs massive processing resources whereas its efficiency in implementing is an inspiring research area. Convolution operators are defined as Gaussian operators and Gaussian smoothing is suggested by convolution. 1-D Gaussian operator is provided herewith

The best smoothing filter for images undergoes localization in spatial and frequency domains, where the uncertainty relation is satisfied as cited in the literature [24] :

2D Gaussian operator is demonstrated as follows:

where σ (Sigma) is the SD of a Gaussian function. When it has the maximum value, the image smoothing would be greater. (x, y) denotes the Cartesian coordinates of the image that showcases the dimensions of window.

FM model incorporates the fusion of HCF using LBP and DLF with the help of Inception v3 technique. To further improve the performance of Inception v3 model, the learning rate scheduler is applied using Adam optimizer.

LBP model is used in various domains and medical image analysis [25] . In LBP, the histograms are integrated as an individual vector where each vector is called as a pattern vector. Alternatively, the integration of LBP texture features and self-organizing map (SOM) is employed to find the effectiveness of the model. LBP is named as operator for texture definition based on differential symptoms over neighbor and central pixels. For all pixel values in the image, a binary code is obtained using thresholding of neighborhood with the help of middle pixel. The binary code is said to be a binary pattern. Therefore, the neighbor pixel is 1 when the pixel value is maximum than the threshold value. It becomes 0 when the pixel value is minimum than the threshold value. Following that, the histogram is deployed to calculate the frequency measured for binary pattern and every pattern denotes the possibility of binary pattern in an image. The basic module of LBP operator utilizes the value of intermediate pixel as a threshold to 3 × 3 neighbour pixels. Threshold task is applicable to deploy a binary pattern 

LBP(u c , v c ) shows the LBP value at middle pixel (u c , v c ). I n and I (u c , v c ) are the measures of neighboring and centre pixels and index n defines the index of neighbour pixels. The function g(u) may be 0 while u < 0 and g(u) 1 if u ≥ 0. The adjacent pixels might be 0, if the scores are lower than the threshold value. On the contrary, it may be 1 if the neighbor pixels are maximum than threshold. LBP value is estimated by scalar multiplication between binary and weight matrices. At last, the multiplication results are utilized to depict the LBP value.

CNNs are enclosed with five layers, namely input, convolutional, pooled, FC, and output. GoogLeNet network is meant to be a CNN and is deployed in Google. It applies inception network method as it limits the number of network attributes and enhances the depth of a system. Therefore, it is extensively employed in image classifications. The instances of a general CNN are viewed as cited earlier [26] and are illustrated in Fig. 2 .

Convolution layer Convolution layer gets varied from a NN in which not all the pixels are linked to upcoming layer with a weight and bias. However, the whole image is divided into tiny regions after which weights and bias are used. Such weights and bias are named as filters or kernels that are convoluted with all small regions in the input image that offers Fig. 2 The structure of CNN a feature map. Such filters are referred to simple 'features' which can be explored from input image in this layer. The count of parameters is essential for this convolution task, which might be lower since a similar filter is traversed across the whole image for a single feature. The count of filters, size of local region, stride, and padding are referred to hyperparameters of convolution layer. According to size and genre of an input image, the hyperparameters undergo tuning to accomplish optimal outcomes.

Pooling layer Pooling layer is applied to reduce the spatial dimensions of an image and the parameter count, and minimize the process. It performs a fixed function for an input without any parameters. Different types of pooling layers are available, such as average pooling, stochastic pooling, and max pooling. Max pooling is a common type and is applied in pooling algorithm, where n × n window is slid across and down the input with a stride of 's'. Every position of maximum value in n × n region is consumed and the input size becomes limited. It offers translational invariance where a small difference in a location would be applicable to analyze the image. Hence, the position is lost at the time of reducing the size.

Here, the flattened result of a last pooling layer is provided as input to FC layer. It acts as a CNN in which all the neurons of existing layer are linked to current layer. Thus, the count of parameters is maximum in the convolution layer. This FC layer is associated with an output layer named as classifier.

Activation function Diverse activation functions are applied over different structures of CNN. Nonlinear activation functions have shown optimal outcome than former sigmoid or tangent functions. Such nonlinear functions are applied to enhance the training speed. Thus, various activation functions are applied and ReLU shows remarkable performance than alternate models.

CNN learning method relies upon vector calculus and chain rule. Assume z to be a scalar (i.e., z ∈ R) and y ∈ R H as a vector, when z is a function of y, the partial derivative of z, in terms of y, is a vector and can be determined as:

In particular, ∂z ∂ y is a vector containing similar size as y, and its ith element is

In addition, assume x ∈ R W is another vector, and y is a function of x. After that, the partial derivative of y in terms of x is determined by:

In the fractional derivative H × W matrix, it is accessed at the juncture of ith row and jth column i.e., ∂ y i ∂ x i . It looks simple to see that z is a function of x in a chain-like argument. Also, a function maps x to y, and another function maps y to z. The chain rule is utilized to compute as given herewith.

The cost or loss function is utilized to measure the difference between the prediction of a CNN x L and the goal t,

The predictive outcome is seen as argmax i x L i . A convolution method is represented as follows:

Filter f has size (H × W × D l ), so that the convolutional layer contains the spatial size of H l − H + 1 ×(W l −W +1) with D slices which implies that y x l+1 in R H l+1 ×W l+1 ×D l+1 , H l+1 H l − H + 1, W l+1 W l − W + 1 and D l+1 D.

The possibility of all labels k ∈ {1, . . . K } is applied to train instance which is calculated by P(k|x)

where z is a non-normalized log possibility. A ground truth shared over labels q(k|x) is normalized such that k q(k|x) 1. In this method, the loss is provided by crossentropy and is defined below:

The cross-entropy loss is a differential value in terms of the logit z k and it is utilized in gradient training of deep methods since the gradient has the easier form ∂l ∂z k p(k) − q(k), bounded between − 1 and 1. Generally, if cross-entropy gets minimized, it implies that the log possibility of accurate label is maximized. Inception V3 is regarded as shared above labels which are independent of training instances u(k) with a smooth parameter , as a training instance, the label shared q(k|x) δ k,y is easily returned by:

Otherwise, these are interpreted as cross-entropy as given below:

So, the label-smoothing regularization is same for executing a single cross-entropy loss H (q, p) and a couple of losses H (q, p) and H (u, p) . Among these, the second loss penalizes the variation of the forecast label shared p from prior u with comparative weight (1− ) .

The major objective of GoogLeNet network is to perform like an Inception network structure due to which the GoogLeNet method is named as Inception network [27] . It contains the maximum number of GoogLeNet versions which are classified into different versions, such as Inception v1, Inception v2, Inception v3, Inception v4, and Inception-ResNet. Thus, Inception generally includes three different sizes of convolution and maximum pooling. The result of network in previous layer is defined as the channel which is collected after the completion of convolution task and after nonlinear fusion is carried out. Similarly, the expression function of this network can be applied to various scales which can be enhanced while at the same time, the over-fitting problem can be eliminated. Figure 3a implies the structure of Inception Fig. 3 a Structure of inception model. b-d Inception V3 based inception module network. Inception v3 refers a network structure deployed by Keras which is pre-trained in Image Net. The input size of the fundamental images is 299*299 with three channels. Also, Inception v3 network structure is applied in this study as shown in Fig. 3b . When compared to Inceptions v1 and v2, Inception v3 network structure employs a convolution kernel splitting model to divide massive volume integrals into minimum convolutions. For instance, a 3*3 convolution is divided into 3*1 and 1*3 convolutions. Using this splitting model, the count of attributes could be limited; thus, the network training speed can be enhanced at the time of extracting spatial feature in an effective manner. Simultaneously, Inception v3 optimizes the Inception network structure with the help of three different sized grids like 35*35, 17*17, and 8*8.

Learning rate scheduler In DL training phase, it is suitable to limit a learning rate (γ t ), when there is a progress development in training phase. The count of weights gets improved while training and this step is referred to step size or 'learning rate'. Specifically, learning rate is an adjustable hyperparameter and is used for NN training using minimum positive values from 0.0 and 1.0. Additionally, learning rate balances the method of resolving the problems. Minimum learning rates require higher training epochs and they offer smaller alterations for weights whereas if the learning rates intends to offer enormous modifications, in such a case, it requires lower training epochs. The performance of tuning a learning rate is highly complex. The maximum learning rate results in divergent training process, while the minimum learning rate leads to slow convergence. An effective result can be accomplished by stimulating various learning rates at the time of training. The method applied for scheduling the leaning rate is named as 'learning rate scheduler'. General learning rate schedules are different types, such as time-based decay, step decay as well as exponential decay.

Adam optimizer is an adaptive moment estimate optimizer which pursues a technique to 1st-order gradient-based optimizer. It depends on the adaptive estimation of lower-order moments. Here, g t represents the gradients, θ t is the parameter at time t, β 1 and β 2 are assigned to be (0, 1), and α is the learning rate. Here, g 2 t denotes the element-wise square of g t g t and the presented default settings are α 0.001, β 1 0.9, β 2 0.999 and ε 10 −8 . Every process on vector is element-wise defined, i.e., β t 1 and β t 2 in which β 1 and β 2 indicate to the power of t. The pseudocode for Adam technique is provided herewith.

Data fusion has been employed in diverse ML and computer vision sectors. The features' fusion is a significant operation that integrates a maximum number of feature vectors. The projected method depends upon features' fusion by entropy. In addition, the obtained features are combined into single vector. There are three vectors computed herewith.

Then, the feature extraction is combined as a single vector.

where f implies a fused vector. The entropy is implemented on features' vector for selected features only on the basis of a value given herewith.

In Eqs. (15) and (16), p denotes features' probability and He defines entropy. Finally, the selected features are offered to classification models so as to distinguish the X-rays from COVID.

MLP network consists of three layers, namely input, hidden, and output layers. MLP network is capable of possessing numerous hidden layers. This is possible through the activation of network to hold processing abilities for the generation of system outputs. MLP is preferred over other classifiers due to the reasons listed herewith. MLP has adaptive learning process, i.e., capable of learning on how to perform tasks depending upon the training data. Besides, MLP does not require any consideration of the underlying probability density function. In addition, it offers the required decision function directly through training process. Figure 4 implies an MLP network with one hidden layer, which has few weights connecting among the layers. The final outcome scores are determined based on the given procedures. Initially, the addition of weights is estimated as following:

where x i denotes an input variable, w ij defines the weight between input variable x i and neuron j, and β i depicts the bias term of the input variable. Then, the final values of the neurons in hidden layers are produced from the obtained values of weighted summation (Eq. 17), by an activation function. A well-known choice of these functions is said to be a sigmoid function as given herewith.

where f j represents the sigmoid function for neuron j and S j refers to sum of weights. As a result, the result of neuron j is determined as following: 

where y j signifies the result of neuron j, w ij denotes the weight from output variable y i and neuron j, f j indicates the activation function for neuron j, and β i depicts the bias term of the final variable.

The 

The proposed FM-HCF-DLF model was assessed for its performance using chest X-ray dataset [28] . The dataset is composed of 27 images under normal class, 220 images under COVID-19, 11 images under SARS and 15 images in Pneumocystis class. A sample set of images from the dataset is shown in Fig. 5 . The authors used fivefold cross-validation. In line with this, under fold 9, the applied FM-HCF-DLF method yielded a better sensitivity and specificity of 94.62% and 95.54%, respectively. Simultaneously, under fold 10, the applied FM-HCF-DLF model produced the maximum sensitivity and specificity values, such as 94.10% and 95.89%, respectively. values of 94.68% and 93.75%, respectively. In line with this, under fold 6, the applied FM-HCF-DLF model attained the optimal F score and kappa values, such as 93.23% and 94.55%, correspondingly. Under fold 7, the implied FM-HCF-DLF model secured optimal F score value, i.e., 94.15% and kappa value i.e., 93.51%. In fold 8, the provided FM-HCF-DLF technique depicted maximum F score and kappa values of 93.57% and 94.37%, respectively. The proposed FM-HCF-DLF approach yielded better F score and kappa values of 94.26% and 95.45% when applied under fold 9. In alignment with this, under fold 10, the deployed FM-HCF-DLF technology implied a high F score and kappa of 94.49% and 93.58%, respectively. XGBoost and LR technologies achieved same F score value of 92%. Simultaneously, the MLP model resulted in a better F score value of 93%. The presented FM-HCF-DLF method yielded an optimal F score value of 93.20%. The above-mentioned tables and figures indicate that the FM-HCF-DLF model is an effective classification model compared to other models. The experimental outcomes indicate that the proposed model demonstrated its effective performance by attaining the maximum average sensitivity of 93.61%, specificity of 94.56%, precision of 94.85%, accuracy of 94.08%, F score of 93.20% and kappa value of 93.50%. The proposed model accomplished better performance due to the inclusion of fusion-based feature extraction model and Adam optimizer. 

The authors developed an effective FM-HCF-DLF model for COVID-19 diagnosis and classification. The FM-HCF-DLF model involved preprocessing stage using GF technique to remove the noise that exists in the image. Then, the FMbased feature extraction process was performed to extract the useful set of features from the preprocessed image. The HCF features used LBP while the DLF used CNN-based Inception v3 model. Besides, Adam optimizer was applied to adjust the learning rate of Inception v3 model. At last, MLP-based classification process was performed to identify and classify the chest X-ray images into different set of classes. The FM-HCF-DLF model was simulated using chest X-ray dataset which attained the maximum outcome. The respective parameters were maximum sensitivity 93.61%, specificity 94.56%, precision 94.85%, accuracy 94.08%, F score 93.2% and kappa value 93.5%. In future, the FM-HCF-DLF model can be improved using other classifiers instead of MLP.

Bat coronaviruses in China

Emerging coronaviruses: genome structure, replication, and pathogenesis

A survey on deep learning in medical image analysis

Optimal deep learning based convolution neural network for digital forensics face sketch synthesis in internet of things (IoT)

Alzheimer detection using Group Grey Wolf Optimization based features with convolutional classifier

Optimal bilateral filter and convolutional neural network based denoising method of medical image measurements

Deep neural architectures for prediction in healthcare

A novel training algorithm for convolutional neural network

Fault detection and classification by unsupervised feature extraction and dimensionality reduction

Deep transfer learning for modality classification of medical images

Pulmonary artery-vein classification in CT images using deep learning

Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning

Knowledge-based collaborative deep learning for benignmalignant lung nodule classification on chest CT

Deep learning and handcrafted method fusion: higher diagnostic accuracy for melanoma dermoscopy images

Pulmonary nodule detection in CT images: false positive reduction using multi-view convolutional networks

Liver semantic segmentation algorithm based on improved deep adversarial networks in combination of weighted loss function on abdominal CT images

3-D convolutional neural networks for automatic detection of pulmonary nodules in chest CT

A recurrent CNN for automatic detection and classification of coronary artery plaque and stenosis in coronary CT angiography

Deep-learning framework to detect lung abnormality -a study with chest x-ray and lung CT scan images

Analysis of tuberculosis severity levels from CT pulmonary images based on enhanced residual deep learning architecture

Improved particle swarm optimization based adaptive neuro-fuzzy inference system for benzene detection

Gated bidirectional CNN for object detection

An error-efficient Gaussian filter for image processing by using the expanded operand decomposition logarithm multiplication

Texture feature extraction by applying local binary pattern (LBP)

Benchmarking on offline handwritten Tamil character recognition using convolutional neural networks

Inception v3 based cervical cell classification combined with artificially extracted features

Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations

The authors declare that they have no conflict of interest. The manuscript was written through contributions of all authors. All authors have given approval to the final version of the manuscript.Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecomm ons.org/licenses/by/4.0/.

Homepage of FM-HCF-DLF model.

Training process of FM-HCF-DLF model.

Classification process of FM-HCF-DLF model.

Classification output of FM-HCF-DLF model.