key: cord-0059997-fqbm4ess
authors: Özkaya, Umut; Öztürk, Şaban; Barstugan, Mucahid
title: Coronavirus (COVID-19) Classification Using Deep Features Fusion and Ranking Technique
date: 2020-07-29
journal: Big Data Analytics and Artificial Intelligence Against COVID-19: Innovation Vision and Approach
DOI: 10.1007/978-3-030-55258-9_17
sha: 2685e38816246a94adc7100ea72a2a0008cf6ae1
doc_id: 59997
cord_uid: fqbm4ess

COVID-19, which appeared towards the end of 2019, has become a huge threat to public health. The solution to this threat, which is defined as a global epidemic by the World Health Organization (WHO), is currently undergoing very intensive studies. There is a consensus that the use of Computed Tomography (CT) techniques for early diagnosis of pandemic disease gives both fast and accurate results. This study provides an automated and highly effective method for detecting COVID-19 at an early stage. CT image features are extracted using the convolutional neural network (CNN) architecture, which is the most successful image processing tool of today, for the detection of COVID-19, where early detection is vital for human life. Representation power is increased by combining features from the output of four CNN architectures with data fusion. Finally, the features combined with the feature ranking method are sorted, and their length is reduced. In this way, the dimensional curse is saved. From 150 CT images, 16 × 16 (Subset-1) and 32 × 32 (Subset-2) patches were obtained to create a subset. Within the scope of the proposed method, 3000 patch images are labeled as “COVID-19 (coronavirus)” or “No finding” for use in training and test stages. The Support Vector Machine (SVM) method then classified the processed data. The proposed method shows high performance in Subset-2 with 98.27% accuracy, 98.93% sensitivity, 97.60% specificity, 97.63% sensitivity, 98.28% F1 score and 96.54% Matthews Correlation Coefficient (MCC) metrics.

COVID-19 has become an epidemic similar to some other pandemic diseases, causes patient deaths in China, according to the World Health Organization (WHO) [1] [2] [3] . Early application of treatment procedures for patients with COVID-19 infection increases the patient's chances of survival. The rapid diagnosis of the disease is one of the most critical facts to prevent the spread of the disease. Pathological tests performed in laboratories take time. A fast and accurate diagnosis is necessary for a productive struggle against COVID-19. For this reason, experts have been started to use radiological imaging methods. These procedures are performed with computed tomography (CT) or X-ray imaging techniques. COVID-19 cases have similar features in CT images in the early and late stages. It shows a circular and inward diffusion from within the image [4] . Therefore, radiological imaging provides early detection of suspicious cases with an accuracy of 90%.

Computer-aided diagnosis (CAD) methods have been used in the diagnosis of various diseases for many years. It is automatically interpreted by removing many human features such as the processing of images taken from the diseased areas by the computer and the interpretation of the audio signals. CAD systems have become the most prominent assistant of specialists by speeding up and facilitating many procedures. In the past, CAD systems were manually used for performing simple tasks. With the development of technology and the spread of computers, the direction of CAD systems has changed. Many processes, such as counting of cells, detection of organs, detection of diseased areas, have become automated. Although this development is due to the strength of the proposed algorithms each period, the CNN algorithm has affected the entire image processing area in recent times. It has mostly resolved the automatic image analysis problem in the medical field.

This study used 150 CT images to classify COVID-19 cases. Two different datasets were generated from 150 CT images. These datasets include 16 × 16 and 32 × 32 patch images. Each dataset contains 3000 number of images labeled with COVID-19 and No findings. Deep features were obtained with pre-trained Convolutional Neural Network (CNN) models. These deep features were fused and ranked to train Support Vector Machine (SVM). The proposed method achieved high performance for early diagnosis of COVID-19 cases. The most important of the main contributions of this study is the use of high-level deep features obtained from different pre-trained CNN networks. Another contribution is to increase the performance of the SVM classifier by giving inputs as the deep features obtained through fusion and ranking. The proposed method has a limitation for image classification. t-test feature ranking can be only used in binary classification. It is not appropriate for multi-class problems. When the proposed method results are examined, it is seen that it shows higher performance than other pre-trained CNN networks.

Various COVID-19 dataset started to be shared on the internet, recently. Some studies [5] [6] [7] [8] classified Chest X-ray images to detect the coronavirus disease. Some [9] [10] [11] [12] studied on forecasting of the coronavirus spread. Some studies [13, 14] used blood test results to detect the coronavirus disease. The literature studies use deep learning methods, mostly. Shan et al. [15] proposed a neural network model called VB-Net to segment the COVID-19 regions in CT images. The proposed method has been tested on 300 new cases. A recommendation system has been used to make it easier for radiologists to mark infected areas within CT images. Xu et al. [16] analyzed CT images to determine healthily, COVID-19, and other viral cases. The dataset included 219 COVID-19, 224 viral diseases, and 175 healthy images. The study achieved an 87.6% general classification accuracy with the deep learning method. Li et al. [17] classified three different coronavirus situations such as mild, common, and severe-critical. The study used CT images of 78 patients, and the number of each class is 24, 46, and 8, respectively. The clinical classification method achieved 82.6% sensitivity, 100% specificity, and 0.918 AUC values. Tang et al. [18] used 176 chest CT images that consisted of two different classes as severe and nonsevere. The study extracted 63 quantitative features on images, and the random forest algorithm classified the features. 3-fold cross-validation method was used during the classification process. The proposed method achieved 87.5% classification accuracy and 0.91 AUC value. Fong et al. [19] proposed a Composite Monte-Carlo (CMC) simulation to predict COVID-19 distribution. An increase in the number of cases in nearby cities was tried to be estimated by using the temporal-spatial data related to the city of Wuhan which is the source of the virus. Elghamrawy et al. [20] represented a method as Artificial Intelligence-inspired Model for COVID-19 Diagnosis and Prediction for Patient Response to Treatment (AIMDP). The proposed method was based on prediction on the spread of COVID-19 cases by using feature selection using the Whale Optimization Algorithm. Fong et al. [21] used data mining technique with polynomial neural network with corrective feedback (PNN + cf). This method had acceptable results to predict COVID-19 outbreak. Rajinikanth et al. [22] proposed a segmentation technique to analysis COVID-19 textures. It consisted of four parts as threshold filter, image enhancement, image segmentation and region-ofinterest (ROI) extraction. The method has ability to extract infected regions from lung background. Our previous study used 126 chest X-ray images that have six different classes such as ARd, Covid-19, neumocystis-pneumonia, Sars, streptococcus, and no finding. The study extracted features on images with different methods and combined the features. Then, the feature set was shrunk by different methods and SVM classified the shrunken feature set. The best classification performance was obtained as 94.23%.

This study consists of six sections. The properties of obtained patch images are visualized in Sect. 2. In Sect. 3, the basics of deep learning methods, feature fusion, and ranking techniques are mentioned. Comparative classification performances are given in Sect. 4. Section 5 and Sect. 6 include discussion and conclusion. 

Totally 53 infected CT images were accessed to the Societa Italiana di Radiologia Medica e Interventistica to generate datasets [23] . Patch images obtained from infected and non-infected regions from CT images. Table 1 presents the properties of two different subsets.

The images in the dataset were acquired by different CT imaging tools. Thus, it caused a different infected grey level region in the CT images. This situation affects the classification process and performance quite negatively. CT images have different grey levels. Image patches of 16 × 16 and 32 × 32 dimensions with different characteristics were obtained from CT images. Figure 1 shows the process of patch creation.

Deep learning, which has become quite popular recently, has been used in many areas. Academic studies have been pioneers for their use in e-mail filtering, search engine matching, smartphones, social media, e-commerce areas, etc. [24] . Deep learning is also used for face recognition, object recognition, object detection, text classification, and speech recognition. In machine learning, Deep Belief Networks (DBN) is a productive graphical model or a class of deep neural networks consisting of multiple layers in hidden nodes. When trained on a series of unsupervised samples, the DBN can learn to reconfigure its entries as probabilistic. The layers then act as feature detectors. After this learning phase, a DBN can be trained with more control to make the classification. DBNs can be seen as a combination of simple, unsupervised networks, such as restricted Boltzmann machines (RBMs) or auto-encoders, which serve as the hidden layer of each subnet, the visible layer of the next layer.

Convolutional neural networks (CNN) are a type of neural network with at least one layer of convolution. It has some layers such as convolution, ReLU, pooling, normalization, fully connected, and a softmax layer. Generally, convolution is a process that takes place on two actual functions. To describe the convolution operation, for example, the location of a space shuttle with a laser is monitored. The laser sensor produces a simple x(t) output, which is the space of the space shuttle at time t. Where X and t are actual values, for example, any t is a different value received at a snapshot time. Also, this sensor has a bit noisy. To carry out a less noisy prediction, the designer can take the average of several measurements together. This can be done with the weighting function w(a), which is a measurement period. If a weighted average operation is applied at all times, a new function is obtained, which allows estimating the position more accurately (see Eq. 1):

The convolution process is represented by a star sign (see Eq. 2):

In CNN terminology, the first argument in X function at Eq. 2 is the input matrix, and the second argument for W function is called the kernel. The output is called a feature map. In the above example, the measurement is made without interruption, but this is not realistic. Time is parsed when working on the computer. To realize realistic measurement, one measurement per second is taken. Where t is the time index and is an integer, so X and W are integers (see Eq. 3).

In machine learning applications, the input function consists of a multidimensional array set, and the kernel function consists of a multidimensional array of several parameters. Multiple axes are convolved at one time. So if the input is a two-dimensional image, the kernel becomes a two-dimensional matrix (see Eq. 4).

The above equation means shifting the kernel according to the input. This increases the invariance of convolution [25] . Many machine learning libraries process the kernel without inversion, which is called as cross-correlation that is related to convolution. Because it looks like a convolution, it is called a convulsive neural network (see Eq. 5):

Convolution provides three essential thoughts to improve a machine learning system: infrequent interactions, parameter sharing, and covariant representations. Furthermore, the convolution process can be worked with variable-sized inputs. Convolution neural network layers use a matrix parameter that includes different kinds of the link between each input unit and each output unit. It means that each output unit connects with each input unit. However, CNN typically has infrequent interactions (also called sparse links or sparse weights).

A pooling function changes the output of the network at a specific location with summary statistics of nearby outputs. The pooling process consists of outputs from specific regions. When several parameters in the next layer depend on the input image or feature map size, any reduction in input size also increases the statistical efficiency and reduces the memory requirements for storing parameters.

The Rectified Linear Unit is an activation function type. The Rectified Linear Unit calculates the function F(x) = max(0, x). In other words, activation is thresholded equal to zero. There are several pros and cons to the use of ReLU. ReLU units can become sensitive during the training phase. For example, a large gradient scale flowing through neuron with a ReLU activation function can cause weights to be updated so that the neuron is not reactivated at any data point. That is, it can kill units irrevocably during training because data replication can be disabled. For example, if the learning rate is too high, 40% of the network may be dead. This is a less frequent occurrence with an appropriate adjustment of the learning rate.

In fully connected layers, a reduction of nodes below a certain threshold increased the performance. So it is observed that forgetting the weak information increases learning. Some properties of dropout value are as follows. The dropout value is generally selected as 0.5. Different uses are also common. It varies according to the problem and data set. The random elimination method can also be used for the dropout. The dropout value is defined as a value in the range [0, 1] when used as the threshold value. It is not necessary to use the same dropout value on all layers; different dilution values can also be used.

The softmax function is a sort of classifier. Logistic regression is a classifier of the classifier, and the softmax function is a multi-class of logistic regression. 1/ j e fj term normalizes the distribution. That is, the sum of the values equals 1. Therefore, it calculates the probability of the class to which the class belongs. When a test input is given x, the activation function in j = 1,…,k is asked to predict the probability of p (y = j | x) for each value. For example, it is desirable to estimate the probability that the class tag will have each of the different possible values. Thus, as a result of the activation function, it produces a k-dimensional vector, which gives us our predictive possibilities. The error value must be calculated for the learning to occur, and the error value for the softmax function is calculated by the softmax loss function.

VGG-16, GoogleNet, and ResNet-50 models were used for feature extraction. The obtained feature vectors with these models were fused to obtain higher dimensional fusion features. In this way, the effect of insufficient features obtained from a single CNN network is minimized. Also, there is a certain level of correlation and excessive information among the features. This also increases consuming time and computational complexity. Therefore, it is necessary to rank the features. t-test technique was used in feature ranking. It calculates the difference between the two features and determines its differences statistically [26] . In this way, it performs the ranking process by taking into account the frequency of the same features in the feature vector and the frequency of finding the average feature.

After the feature fusion and ranking functions were performed, the binary SVM classifier was trained for classification. SVM transfers feature into space where it can better classify features with kernel functions [27] . Linear kernel function was used in SVM. The SVM classifier was trained to minimize the squared hinge loss. The squared hinge loss is given in Eq. 6.

Here, x n represents the fusion and the ranking feature vector. The wrong classification penalty is determined by the C hyperparameter in the loss function.

In the proposed method, pre-trained CNN networks were trained for Subset-1 and Subset-2, separately. VGG-16, GoogleNet, and ResNet-50 models were used as a pre-trained network. Patch images were given as input to pre-trained CNN structures. A discriminative method is used to obtain high-level features from pre-trained CNN feature vectors. This process is performed by Canonical Correlation Analysis (CCA) fusion method [28] . CCA is a widely used method that examines the degree of relationship between variables. If X and Y define vectors as matrices containing n features, S xx and S yy represent the in-class covariance matrix, S xy = S yx inter-class covariance matrix. The vector S is expressed as in Eq. 7.

S xx S xy S yx S yy (7) It is challenging to understand the correlation between feature vectors from this matrix. Therefore, the correlation between the two properties is obtained by Eqs. 8 and 9 in the CCA fusion method.

Combining and vectorizing features in the fusion process is obtained as CCA feature vectors Z. 

The t-test evaluates the statistical difference between the two classes. Mean feature frequency and variance are evaluated for each class by Lindeberg-Levy theorem [29] . t-test equation for feature rankings showed as in Eq. 11.

where t is the frequency term, C k is the number of k classes, tZ k feature frequency of between classes, and tZ i in-class frequency. s is the standard deviation and m is equal to √ 1/N k − 1/N . N k is the number of features in the kth class. N indicates the total number of features. t values for class can be performed as in Eq. 12 t-test

Correlation values between features were taken into consideration in the fusion process. The obtained features were ranked by the t-test method. In the t-test ranking process, features close to each other were eliminated according to feature frequency. In the last stage, fusion and ranking deep features were evaluated with the SVM classifier. The method proposed in Fig. 2 is visualized.

This study presents the classification of COVID-19 texture for two different size data sets. First of all, the features obtained from CNN structures have been passed through fusion and ranking processes and classified. Six different metrics were used to evaluate the proposed method. These metrics are sensitivity (SEN), specificity (SPE), accuracy (ACC), precision (PRE), F-score, and Matthews Correlation Coefficient (MCC).

(13) 

There are 6000 pieces of 16 × 16 CT patches in Subset-1. Data distribution between classes is equal. 75% of these images were used for training and 25% for testing. Table 2 shows comparatively classification performance pre-trained CNN networks and of the proposed method. 

Subset-2 includes 3000 COVID-19 and 3000 No finding 32 × 32 CT patches.

Comparative classification results of Subset-2 are given in Table 3 . Table 2 shows that the best performance in Subset-1 was obtained as 95.60%. The highest performance belongs to ResNet-50 models with 98.93% in the sensitivity metric. In specificity and precision metrics, GoogleNet performed best with 98.93% and 98.75%, respectively. The proposed method in F1-score and MCC metrics is the most successful among pre-trained CNN structures with 95.50% and 91.29%, respectively. Comparative performance metrics for Subset-2 are given in Table 3 . The proposed method stands out with its 98.27% performance in accuracy metric. In the sensitivity metric, VGG-16 and ResNet-50 models show the highest performance with 99.20%. In the precision and F1-score metrics, the GoogleNet model achieved 98.80% and 98.78%, respectively. The proposed method achieved the highest metric performance in F1-score and MCC metrics with 98.28% and 96.54%, respectively. Figures 3, 4 presents the confusion matrices of the proposed method for Subset-1 and Subset-2.

The confusion matrix was obtained for the proposed method using Subset-1 in Fig. 3 . When the confusion matrix was evaluated in class, the COVID-19 class was classified with an accuracy rate of 97.9%. Performance of No findings class was lower than COVID-19. 93.3% accuracy rate was obtained for this class. Classification accuracy of 93.6% was obtained in the analysis of positive class. In the negative class, this rate is higher and had a value of 97.8%.

Subset-2 was used in the training and testing process for the proposed method. In Fig. 4 , a confusion matrix was obtained for test data. In-class analysis, a 97.6% accuracy rate of COVID-19 class was obtained. The performance was increased compared to Subset-1 in the No findings class. The accuracy rate was 98.9% for this class. In the positive and negative class evaluation, a classification accuracy of 98.9% and 97.6% were obtained, respectively. Also, the test time of the proposed method for each image is 0.34 s. 

The first case of COVID-19 was found in the Wuhan region of China. COVID-19 is an epidemic disease and threatens the world's health system and economy. COVID-19 virus behaves similarly to other pandemic viruses. This makes it challenging to detect COVID-19 cases quickly. Therefore, COVID-19 is a candidate for a global epidemic. Radiological imaging techniques are used for a more accurate diagnosis in the detection of COVID-19. Therefore, it is possible to obtain more detailed information about COVID-19 using CT imaging techniques. When CT images are examined, shadows come to the fore in the regions where COVID-19 is located. At the same time, a spread is observed from the outside to the inner parts. The images in this study were acquired with different CT devices. Different characteristics of CT devices cause that the grey-levels of the similar part of the images are different. This complicates the analysis of the images. In the study, deep features were obtained by using pre-trained CNN networks. Then, deep features were fused and ranked. The data set was generated by taking random patches on CT images. Pretrained CNN networks were trained using the transfer learning method in the Subset-1 and Subset-2 datasets. With the proposed method, 95.60% accuracy and 91.29% MCC metric performance were obtained for Subset-1. In Subset-2, the proposed method showed 98.27% accuracy, and 96.54% MCC metric performance. Table 4 presents the literature comparison. Table 4 shows that all results are over 85%. The literature studies show that artificial intelligence can help experts to diagnose the coronavirus disease. The accuracy metric reached 100% in the literature [6] . The proposed method is in the second-best place in state-of-the-art methods. The worst performance belongs to 86.7% accuracy [16] . In the literature, most of the studies on COVID-19 are medical studies. The studies on classification and segmentation of COVID-19 on images will be increased. The need for coronavirus dataset is evident, especially for segmentation studies, because the target segmentation area is needed to be labeled by expert radiologists.

In this study, a method, which applies deep features fusion and ranking procedures features from pre-trained CNNs, was proposed for the early diagnosis of COVID-19 cases. In the proposed method, deep features were fused by CCA method, and ranking was performed with a t-test technique. SVM algorithm, a powerful classifier, was used to classify high-level features as a binary type. Also, the proposed method was evaluated on two different data sets as Subset-1 and Subset-2. It showed the highest performance on the Subset-2 dataset with 98.27% accuracy, 98.93% sensitivity, 97.60% specificity, 97.63% sensitivity, 98.28% F1 score and 96.54% MMC metric values. It is seen that the classification process was performed better and more effectively than other pre-trained CNNs. When comparing performance with other studies in the literature, it showed the second-highest performance. Thanks to this high performance, it has been proven to provide an accurate diagnosis of texture classification within CT images. It is foreseen that the early diagnosis of COVID-19 cases is carried out in time with the proposed method. Thus, it can be provided that the number of cases is under control.

Clinical features of patients infected with 2019 novel coronavirus in Wuhan. China. The Lancet

Added value of computer-aided CT image features for early lung cancer diagnosis with small pulmonary nodules: a matched case-control study

Dermatologistlevel classification of skin cancer with deep neural networks

Mining X-ray Images of SARS Patients. Data Mining

Automatic detection of coronavirus disease (COVID-19) using X-ray images and deep convolutional neural networks

COVID-19 detection using artificial intelligence

Covid-19: automatic detection from x-ray images utilizing transfer learning with convolutional neural networks

COVID-19 screening on chest X-ray images using deep learning based anomaly detection

Critical care utilization for the COVID-19 outbreak in Lombardy, Italy: early experience and forecast during an emergency response

Probabilistic prediction of COVID-19 infections for China and Italy, using an ensemble of stochastically-perturbed logistic curves

Analysis and forecast of COVID-19 spreading in China, Italy and France

A simple iterative map forecast of the COVID-19 pandemic

A machine learningbased model for survival prediction in patients with severe COVID-19 infection

Rapid and accurate identification of COVID-19 infection through machine learning based on clinical available blood test results

Lung infection quantification of COVID-19 in CT images with deep learning

Deep learning system to screen coronavirus disease pneumonia

CT image visual quantitative evaluation and clinical classification of coronavirus disease (COVID-19)

Severity assessment of coronavirus disease 2019 (COVID-19) using quantitative features from chest CT Images

Composite monte carlo decision making under high uncertainty of novel coronavirus epidemic using hybridized deep learning and fuzzy rule induction

Diagnosis and prediction model for COVID19 patients response to treatment based on convolutional neural networks and whale optimization algorithm using CT. medRxiv

Finding an accurate early forecasting model from small dataset: a case of 2019-nCoV novel coronavirus outbreak

Harmonysearch and otsu based system for coronavirus disease (COVID-19) detection using lung CT scan images

Improving neural networks by preventing co-adaptation of feature detectors

Non-native children speech recognition through transfer learning

A modified T-test feature selection method and its application on the HapMap genotype data

Statistical learning theory: a tutorial

A new method of feature fusion and its application in image recognition

A modified t-test feature selection method and its application on the hapmap genotype data

Evaluation of the confusion matrix method in the validation of an automated system for measuring feeding behaviour of cattle