key: cord-0805134-n9funmvv
authors: Bruntha, P. Malin; Pandian, S. Immanuel Alex; Anitha, J.; Abraham, Siril Sam; Kumar, S. Niranjan
title: A Novel Hybridized Feature Extraction Approach for Lung Nodule Classification Based on Transfer Learning Technique
date: 2022-03-31
journal: J Med Phys
DOI: 10.4103/jmp.jmp_61_21
sha: 34f8521c74672a06559f27016a9b28c2c2f9a296
doc_id: 805134
cord_uid: n9funmvv

PURPOSE: In the field of medical diagnosis, deep learning-based computer-aided detection of diseases will reduce the burden of physicians in the diagnosis of diseases especially in the case of lung cancer nodule classification. MATERIALS AND METHODS: A hybridized model which integrates deep features from Residual Neural Network using transfer learning and handcrafted features from the histogram of oriented gradients feature descriptor is proposed to classify the lung nodules as benign or malignant. The intrinsic convolutional neural network (CNN) features have been incorporated and they can resolve the drawbacks of handcrafted features that do not completely reflect the specific characteristics of a nodule. In the meantime, they also reduce the need for a large-scale annotated dataset for CNNs. For classifying malignant nodules and benign nodules, radial basis function support vector machine is used. The proposed hybridized model is evaluated on the LIDC-IDRI dataset. RESULTS: It has achieved an accuracy of 97.53%, sensitivity of 98.62%, specificity of 96.88%, precision of 95.04%, F(1) score of 0.9679, false-positive rate of 3.117%, and false-negative rate of 1.38% and has been compared with other state of the art techniques. CONCLUSIONS: The performance of the proposed hybridized feature-based classification technique is better than the deep features-based classification technique in lung nodule classification.

To overcome these difficulties, computer-aided diagnosis (CAD) has come to the front to analyze the large volume dataset. The application of CAD has improved the 5-year survival rate from 15% to over 70%. [7] Hence, the importance of CAD for the management of lung cancer disease has gained traction in the past decade.

Lung nodules are small masses of tissue appearing in the lung due to various reasons. They appear as opaque white objects on a CT image and the sizes vary from 3 mm to 30 mm. There can be benign lung nodules and malignant (cancerous) lung nodules. In few scans and subsequent analyses, a benign nodule can be classified as a cancerous lung nodule. [7] To avoid this error, an efficient classification scheme is the need of the hour. It is not easy to differentiate benign nodules and malignant nodules because both are having similar visual representations. Many CADs available in the literature are based on image processing and traditional machine learning techniques. Very recently, deep learning-based CADs are being introduced in most of the areas for detecting the abnormalities in the medical images.

Classifying benign nodules from malignant nodules is imperative in the analysis of lung cancer. [8] This can be carried out also by a biopsy or a positron emission tomography scan. Although many researches are available for differentiating pulmonary nodules, they depend on image processing-based segmentation and feature extraction techniques. [9, 10] There are two broad categories in the classification of lung nodules. One category is the traditional classification where the features from the nodule images are calculated using different feature engineering techniques and these features are used to classify the lung nodules into benign and malignant. Another category is entirely independent of feature engineering by domain experts. Such method is based on deep learning where the deep learning algorithm itself learns the features from the given input images and classifies the nodules into benign and malignant. Among deep learning methodologies, CNN has been extensively used for extracting the features without manual intervention which are termed as deep features.

In the conventional CAD system, handcrafted features were computed from malignant and benign nodule images for the lung nodule classification using traditional classifiers such as linear discriminant analysis, artificial neural network, and support vector machine (SVM). [11] [12] [13] [14] [15] Deep learning architectures such as deep belief networks and CNN were able to classify the nodules more efficiently than the traditional classifiers which used handcrafted features. [16] A multi-crop convolutional neural network (MC-CNN) was developed to detect the malignancy of the nodules. [17] For improving the lung nodule classification, evolutionary algorithms were incorporated in CNN architecture. [18] Recently, the deep features were combined with the specific handcrafted features to improve the classification accuracy. [19] [20] [21] [22] [23] [24] [25] Deep learning architecture has given better classification accuracies when compared with traditional handcrafted features. However, one inherent difficulty is in obtaining of the large volume of datasets to have significant results which is difficult in medical applications. It is imperative to note that while deep learning architecture is enough for detecting images in biometric systems, it may not be sufficient to outperform the handcrafted features in all cases where these methodologies are employed. For example, the handcrafted features are found to have better output in the cases of face and iris recognition whereas the deep features have outperformed the handcrafted features in fingerprint recognition systems. In the classification of lung nodules by handcrafted features, the accuracy is not better when compared with deep features-based classification. The classification scheme based on deep features may miss out few salient points if it is used alone as it is evident from the literature review. Hence, this research article presents an automatic system to assist clinicians in diagnosing lung nodules with the hybridized feature set where the experiments are carried out using the LIDC-IDRI dataset.

In this proposed approach, hybridization of deep features and handcrafted features are used to classify the lung nodules into benign and malignant. Figure 1 shows the schematic diagram of the proposed methodology. This system uses the modified ResNet50 model using the transfer learning technique for deep feature extraction and integrates the deep features with the traditional histogram of oriented gradient (HOG) features. Since the fused feature set is very large, training the machine learning classifier will be complex and take much time. To reduce the complexity and computation time, principal component analysis (PCA) has been introduced. PCA not only reduces the dimension of the data but also preserves the important information. For differentiating malignant nodules from benign nodules, radial basis function SVM (RBF-SVM) has been employed.

LIDC/IDRI dataset [26] contains 1018 scans and each scan consists of CT images of the chest and an XML file which has annotations of 4 radiologists. In this XML file, the malignancy level for the nodules is specified. The malignancy rating of each nodule is represented in the range of 1-5. Based on the information given in the XML file, 2625 nodules have been extracted for this work with the help of pylidc library as given in Table 1 . [27] The nodules which got the malignancy rating 1 and 2 are termed as "highly unlikely for cancer" and 1136 nodules 

The lung CT images collected from the LIDC-IDRI dataset and each image has the matrix size of 512 × 512. The thoracic CT scan not only consists of lung parenchyma but also comprises of the image of sternum, rib, ascending aorta, superior vena cava, trachea, descending aorta, vertebra, thecal sac with the spinal cord. These extraneous information are not necessary for the experimentation. Thus, the aim of pre-processing is to extract the nodule region from the lung parenchyma. The centroid information of the nodule is available in the XML file and the region of interest of size 64 × 64 is cropped with respect to the centroid information. A sample of cropped lung nodule images from the dataset is shown in Figure 2 .

The images carry a lot of information and processing the entire information requires a huge necessity of memory and computation time. The aim of feature extraction is to extract the important properties or features from the input image which can differentiate one pattern from the rest of the patterns. During this feature extraction process, the irrelevant information will be eliminated without any significant loss in important information related to the input images. [28] Handcrafted feature extraction using histogram of oriented gradient HOG feature descriptor [29] is employed in the present work to extract the features related to the shape characteristics of lung nodules. The lobulated, spiculated, and ragged nodules are having more probability to be malignant whereas round, tentacular and polygonal-shaped nodules are having more probability to be benign. Wang et al. have reported that the HOG features are suitable for describing the shape and edge characteristics of the malignant and benign nodules. [30] In this work for calculating the HOG features, the orientation bin has been set to 9, the size of a cell is (8, 8) 

Deep CNN (DCNN) is the end-to-end machine learning framework which does not need feature engineering of the input images. In natural image analysis, DCNNs have made a giant stride in tasks such as object recognition and image classification. If they are trained with less number of data, they cannot classify or recognize the given input with high accuracy.

There are two difficulties a researcher faces when he/she uses DCNNs in lung nodule classification. Firstly, the datasets available in the public domain like LIDC-IDRI for lung nodule classification task is very small in number when it is compared with millions of data available with the ImageNet dataset. The second difficulty is the subtlety of the nodule classification task as the differences between a benign and malignant nodules are not self-evident. The initial problem can be overcome by transferring the weights from pretrained CNNs which are trained for different applications to the problem at hand. This technique is referred to as transfer learning. The second difficulty can be taken into account by concatenating deep features and handcrafted features and creating hybridized features.

Residual Network (ResNet) is a DCNN model which is less complex than other models. [31] It is easier to optimize the residual network. When the network becomes deep, the accuracy of the network reaches saturation or it starts decreasing suddenly due to vanishing gradient problem. To eliminate the vanishing gradient problem, skip connections have been introduced in the ResNet architecture. The skip connections make a way to add the output from a previous layer to a later layer. [31] The basic principle of the residual network is utilized in the lung nodule classification problem. The initial convolution layer has been modified to accept 64 x 64 size grayscale image input. Using the transfer learning concept, the pretrained weights are used at the starting point. The fully connected layers have been completely removed from the baseline ResNet50 model.

The modified architecture of ResNet50 is shown in Figure 3 . This architecture consists of 5 stages. The first stage has a convolution layer, a batch normalization layer, a ReLu Layer and a max-pooling layer. In stage 2 to stage 5, convolution block and identity blocks are available. The convolution block has 3 convolutional layers and the identity block (ID block) has 3 convolutional layers. The identity block with the layers and skip connections are shown in Figure 4 . The lung nodule image size used in this work is 64 × 64 and this image is given as input to stage 1. 

The hybridized features (f h ) have been formed by concatenating the deep ResNet50 features (f res ) with the handcrafted HOG features (f hog ). It is proposed to hybridize the two different features because one set of feature extraction methodology may overlook the significant results of the other methodology. [20] The hybridization methodology takes complete advantage of the powerful handcrafted features and the highest level DCNN features. [25] The features extracted by traditional feature descriptor and DCNN are complementary in nature and if concatenated, they give features that are combinations of both. [30] Feature reduction using principal component analysis PCA is one of the feature reduction methods. It will transform a large set of predictor variables into a smaller set of predictor variables but maintains most of the information present. [28] The hybridized feature set has a large number of features (predictor variables). If these features are applied to the classifier directly, it will lead to computational complexity and more computation time. To mitigate these problems, the PCA technique is employed in this work without losing much information.

Support vector classifier (SVC) is a highly admired machine learning algorithm for classification. It provides highly accurate classification. The SVC can handle non-linear data points using kernels. Linear kernel, polynomial kernel, and RBF kernel are common types used in SVC. Among these three types, the RBF kernel has been selected in the proposed methodology because of the attractive properties of RBF. RBF kernel is invariant to translation and it is easy to tune this kernel because it has single parameter. Moreover, this kernel is isotropic.

The mathematical representation of the RBF kernel is given in equation 1.

( )

where σ is variance. It is a hyperparameter.

x i represents the support vector and x j represents data point.

x i − x j represents the Euclidean distance.

If the Euclidean distance between the support vector and data point is less, they are similar and the kernel value will be maximum for that data point. If the distance between the support vector and data point is more, they are dissimilar and the kernel value will be minimum for that data point. The maximum value of the kernel will be 1 and the minimum value of the kernel will be 0. It is important to find the optimal value for the parameter σ. This single parameter is tuned by employing Grid Search Cross-Validation approach. A significant advantage of using RBF SVM is less memory requirement because during training, it will store only the support vectors and not the entire data points.

The performance of the proposed model has been evaluated by generating confusion matrix and receiver operating characteristic (ROC) curve. The structure of the confusion matrix for lung nodule classification is shown in Table 3 . True negative (TN) infers that the benign nodule is correctly identified as benign nodule. True positive (TP) tells that the malignant nodule is correctly identified as malignant nodule. False-positive (FP) represents that the benign nodule is wrongly identified as malignant nodule. False-negative (FN) indicates that the malignant nodule is wrongly identified as benign nodule.

Based on the values generated in the confusion matrix, different performance metrics such as Accuracy, Sensitivity, Specificity, Precision, F 1 score, FP rate (FPR) and FN rate (FNR) are calculated.

For comparing the performance of the proposed model, different feature extraction and classifier combinations have been experimented. The handcrafted features extraction techniques such as Gray Level Co-occurrence Matrix (GLCM), [32] Local Binary Pattern (LBP) [33] and HOG [29] are used for extracting features. These feature sets are applied independently to train four different classifiers such as logistic regression, linear SVM, RBF SVM, and Random Forest. In the handcrafted feature-based experiments, 12 different combinations (models) have been analyzed. These models are listed in Table 4 . To analyze the performance of deep features in lung nodule classification, VGG16, VGG19, and ResNet50 features are considered, and these features are used to train four different classifiers such as logistic regression, linear SVM, RBF SVM, and Random Forest. In the deep feature-based experiments, 12 different combinations (models) have been analyzed. These models are listed in Table 5 . The proposed hybridized feature technique has been tested for 12 different combinations and those combinations (models) are listed in Table 6 . Table 4 . From Table 4 , it is understood that even though many models got 62.24% of accuracy, they do not identify any malignant nodule. They have misclassified all the nodules as benign nodules and this led to maximum FNR (100%). It is noted that GLCM features misclassified benign nodules as malignant nodules. Therefore, for GLCM features, Model 1 and 2 have produced high FPR. Among the three feature descriptors, HOG features are better than GLCM and LBP features because they have produced less FNR. Model 11 (HOG+RBF-SVM) performed better in lung nodule classification which has got 78% of accuracy, 95.45% of sensitivity, 67% of specificity, 64% of precision, 0.77 of F 1 score, 32.6% of FPR and 4.5% of FNR. However, in medical image analysis, both FNR and FPR should be low. The FPR of Model 11 is very high. It is observed that the handcrafted features are not able to classify the lung nodules more accurately.

From the deep features-based experimentations, it is observed that ResNet50 + RBF-SVM (Model 23) has performed better when compared to other models (Model 13-24). The performance metrics of deep features in lung nodule classification is given in Table 5 . Model 23 has achieved the accuracy of 83.06%, sensitivity of 95.06%, specificity of 75.28%, the precision of 70.42%, F 1 score of 0.81, FPR of 24.2%, and FNR of 4.94%. It is noted that the accuracy of deep features is better than handcrafted features but FPR is not reduced much. To reduce FPR, the hybridized features and classification combinations are tested.

The dimension of the hybridized feature set is large. From the HOG feature descriptor, 2916 features are computed. The number of deep features from VGG16, VGG19, and ResNet50 models are 2048, 2048, and 8192, respectively. After feature concatenation, the dimension of the hybrid feature set is very large, and it is shown in Figure 5 . If the classifiers are trained with such large dataset, the computation time would be large. To preserve the important features and reduce the computation time, PCA is employed. The number of hybridized features is reduced with the help of PCA and the reduced number of features is shown in Figure 5 . Due to the reduced feature set, the computation time for classification has been reduced and it is shown in Figure 6 .

After including HOG features with deep features, the performance of the classifiers has been improved in most of the cases as it is evident from Figure 7 . Table 7 gives the comparison of the proposed method with other state-of-the-art methods. It can be observed that the proposed methodology has outperformed in lung nodule classification when compared to the related works. Li [25] The proposed method is less complex than the method given by Shulong Li and gives improvement in accuracy by 8.95%, sensitivity by 16.02%, specificity by 4.8%. The FPR and Their approach has given good results because segmentation of nodules has been done for the input images. [36] However in the proposed methodology without segmentation of nodules, the features have been extracted from the 2D nodule patches and the results are better than the methodology given by Antonio et al. with 4.9% improvement in accuracy, 7.92% improvement in sensitivity, and 3.41% improvement in specificity. The proposed methodology promises an agreeable balance between sensitivity and specificity and it implies that it is well balanced in classifying the malignant and benign nodules.

This research work proposes a deep hybridized model to classify the lung nodules into two different categories: malignant nodule and benign nodule. In clinical routine, there are a few complications in the classification of lung nodules due to visual representation of these nodules may appear similar. The proposed methodology enjoys the benefits of both ResNet50 based deep features and the handcrafted HOG features. Due to the hybridization of the features, the classifier can differentiate the malignant nodules and benign nodules with high accuracy of 97.53%, sensitivity of 98.62%, specificity of 96.88%, precision of 95.04%, F 1 score of 0.97, FPR of 3.12%, and FNR of 1.38%. In addition, our proposed approach has been compared to handcrafted feature-based lung nodule classification and deep feature-based lung nodule classification models. The proposed approach outperforms well when it is compared to different deep learning models used in lung nodule classification. The future scope of this work will be focused on the hybridization of EfficientNet with HOG features.

Nil.

There are no conflicts of interest. [25] 88.58 82.60 91.82 --8. 28 17.4 -Wang et al. [30] 91.75 ------0.970 Nibali et al. [34] 89.9. 91.07 88.64 89.35 ---0.946 da Nóbrega et al. [35] 88.41 85.38 -73.48 0.79 --0.932

Xie et al. [19] 87.74 81.11 89.67 ----0.945 Shen et al. [17] 87.14 77 93 ----0.93 de Carvalho et al. [36] 92.63 90.7 93.47 ----0.934

Kumar et al. [37] 75.01 83.35 ------Han et al. [15] -------0.927 Dhara et al. [7] -82.89 80.73 ----0.882 Hussein et al. [38] 91.26 

Cancer statistics, 2020

World Cancer Report 2020. Lyon, France: International Agency for Research on Cancer, World Health Organization

The rising burden of cancer in the developing world

Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries

Public health messaging and strategies to promote "SWIFT" lung cancer detection: A qualitative study among high-risk individuals

Reduced lung cancer mortality with low-dose computed tomographic screening

A combination of shape and texture features for classification of pulmonary nodules in lung CT images

A pattern classification approach to characterizing solitary pulmonary nodules imaged on high resolution CT: Preliminary results

Proficient lung nodule detection and classification using machine learning techniques

Computer-aided diagnostic scheme for distinction between benign and malignant nodules in thoracic low-dose CT by use of massive training artificial neural network

Automated lung nodule classification following automated nodule detection on CT: A serial approach

Computer-aided diagnosis of pulmonary nodules on CT scans: Segmentation and classification using 3D active contours

Usefulness of an artificial neural network for differentiating benign from malignant pulmonary nodules on high-resolution CT: Evaluation with receiver operating characteristic analysis

Lung cancer classification using neural networks for CT images

Texture feature analysis for computer-aided diagnosis on pulmonary nodules

Computer-aided classification of lung nodules on computed tomography images via deep learning technique

Multi-crop convolutional neural networks for lung nodule malignancy suspiciousness classification

Lung nodules diagnosis based on evolutionary convolutional neural network

Fusing texture, shape and deep model-learned information at decision level for automated classification of lung nodules on chest CT

Handcrafted vs. non-handcrafted features for computer vision classification

Combining deep and handcrafted image features for presentation attack detection in face recognition systems using visible-light camera sensors

Local learning with deep and handcrafted features for facial expression recognition

Integrating handcrafted and deep features for optical coherence tomography based retinal disease classification

Deep feature transfer learning in combination with traditional features predicts survival among patients with lung adenocarcinoma

Predicting lung nodule malignancies by combining deep convolutional neural network and handcrafted features

LIDC-IDRI -The Cancer Imaging Archive (TCIA) Public Access-Cancer Imaging Archive Wiki

Lung nodule malignancy classification using only radiologist-quantified image features as inputs to statistical learning algorithms: Probing the Lung Image Database Consortium dataset with two statistical learning methods

Classification of lung nodule using hybridized deep feature technique

Histograms of Oriented Gradients for Human Detection

A hybrid CNN feature model for pulmonary nodule malignancy risk differentiation

Deep Residual Learning for Image Recognition

Textural features for image classification

A comparative study of texture measures with classification based on featured distributions

Pulmonary nodule classification with deep residual networks

Lung nodule malignancy classification in chest computed tomography images using transfer learning and convolutional neural networks

Classification of patterns of benignity and malignancy based on CT using topology-based phylogenetic diversity index and convolutional neural network

Lung Nodule Classification Using Deep Features in CT Images. 12 th Conference on Computer and Robot Vision

Risk Stratification of Lung Nodules Using 3D CNN-Based Multi-Task Learning