key: cord-0615818-q9z0423a
authors: Saihood, Ahmed; Karshenas, Hossein; Nilchi, AhmadReza Naghsh
title: Deep fusion of gray level co-occurrence matrices for lung nodule classification
date: 2022-05-10
journal: nan
DOI: nan
sha: ec764fcbe861fd6cb148750558ffb571be7503ca
doc_id: 615818
cord_uid: q9z0423a

Lung cancer is a severe menace to human health, due to which millions of people die because of late diagnoses of cancer; thus, it is vital to detect the disease as early as possible. The Computerized chest analysis Tomography of scan is assumed to be one of the efficient solutions for detecting and classifying lung nodules. The necessity of high accuracy of analyzing C.T. scan images of the lung is considered as one of the crucial challenges in detecting and classifying lung cancer. A new long-short-term-memory (LSTM) based deep fusion structure, is introduced, where, the texture features computed from lung nodules through new volumetric grey-level-co-occurrence-matrices (GLCM) computations are applied to classify the nodules into: benign, malignant and ambiguous. An improved Otsu segmentation method combined with the water strider optimization algorithm (WSA) is proposed to detect the lung nodules. Otsu-WSA thresholding can overcome the restrictions present in previous thresholding methods. Extended experiments are run to assess this fusion structure by considering 2D-GLCM computations based 2D-slices fusion, and an approximation of this 3D-GLCM with volumetric 2.5D-GLCM computations-based LSTM fusion structure. The proposed methods are trained and assessed through the LIDC-IDRI dataset, where 94.4%, 91.6%, and 95.8% Accuracy, sensitivity, and specificity are obtained, respectively for 2D-GLCM fusion and 97.33%, 96%, and 98%, accuracy, sensitivity, and specificity, respectively, for 2.5D-GLCM fusion. The yield of the same are 98.7%, 98%, and 99%, for the 3D-GLCM fusion. The obtained results and analysis indicate that the WSA-Otsu method requires less execution time and yields a more accurate thresholding process. It is found that 3D-GLCM based LSTM outperforms its counterparts.

One of the primitive issues for humans has always been and is the prevalence of diseases that end up in death. Today, lung cancer is considered the worst cancer in causing death (1) . The available studies reveal that the count of patients diagnosed with lung cancer between 2005 to 2018 has increased by approximately 25% (2) . Researchers at the global scale have observed that the count of people with lung cancer is gradually rising, an alarm for all the human race. The early discovery of this disease increases the probability of its care (3) . The advancement of this disease aggressively separates cells in the lungs; consequently, the count of damaged textures in the lung increases upon its progress.

Image-based Automatic computer-aided diagnosis (CAD) systems are applied to detect and diagnose lung cancer by analyzing C.T. scan images, specifically their textures automatically; however, C.T. images make automatic cancer detection more accurate, the risk of ionizing rays is inevitable (4) . In CAD systems, the following stages are of concern for image processing: 1) noise removal from the image, 2) feature extraction, 3) feature selection, 4) cancer segmentation and 5) cancer classification (5, 6) .

Most automatic lung cancer detection and classification methods combine image processing techniques with deep learning and machine learning methods. There exist many methods proposing lung cancer detection and classification, including support vector machines (SVM) (7) , artificial neural networks (ANN) (8) , random forests (9) , genetic algorithms (G.A.) (10) , fuzzy C-means clustering (11) , and deep learning-based features fusion (12) .

Simple traditional segmentation methods like the similarity-based segmentation techniques (13) , multishape graph-cut approach (13, 14) , region aided geometric snakes approach (15) , and feature-based atlas approach (16) act fast but do not meet the segmentation accuracy on the lung C.T. image with complicated patterns of grey levels. Regional deconvolutional neural networks for images segmentation (17) seek to overcome this drawback at the expense of extended run-time. With the continuous expansion of CT scanning techniques and their application in automatic CAD systems, segmentation methods with high accuracy and low computation complexity have become necessary. One of the wellestablished and efficient image segmentation approaches applied in detecting cancerous tissues is the Otsu thresholding (18) (19) (20) . This optimal thresholding increas between-class variance through an exhaustive search. This process can be time-consuming depending on the count of thresholds (i.e., the segments) sought. The proposed methods that enhance the original Otsu method by preprocessing the input image before segmentation (18) or applying the Otsu algorithm iteratively on regions of interest in the image (21) are many. The intelligent meta-heuristic search has been and is being applied in finding the proper thresholding values among multi-thresholds of Otsu segmentation by applying the particle swarm optimization (PSO) algorithm (22) . The problem with PSO is its low convergence rate which cannot find proper solutions in a limited time. To segment a lung CT scans more accurately, and fast, the WSA is applied to search for thresholding values to overcome these drawbacks during the nonsmall cell lung cancer (NSCLC) detection phase, which avoids outliers. The WSA algorithm has a good convergence rate, obtains reasonable solutions in a limited time when combined with the Otsu segmentation method to extract the lung nodules more accurately. After detection, the pixel intensity in CT, slices would be scaled following a proper range, followed by extracting features from the regions of interest.

Lung cancer classification has not met its accuracy level yet due to its inherent complexities. The convolutional neural networks (CNN) are applied in lung image processing and classification (23) . CNN requires fixed-size input images; consequently, raw images must be resized to the size into the smallest image when different inputs are available in the problem. To confront the fixed-size problem, the 3D-CNN is applied by dividing the input image into a stack of patches (24, 25) . The traditional CNNs based fusion algorithms focus on a one-batch learning mode that produces the best parameters by training the whole data instantly, where, the redundant or irrelevant information highly affects the training process. CNNs are not able to encode the location and familiarization of an object. Although the existing model-based late fusion provides higher representation performances, their impact on learning high-level features is low. This training strategy severely restricts the processing conditions, especially when managing the vast input data volume. The limited dataset restrains CNN models and generates the compelling learning features immediately from the small dataset of CT raw images challenging. To feed textures available in features into GLCMs to 2D-CNN-based architecture, (26) has considered the spatial information in all directions but ignores the correlation between sequenced slices.

Contrast loss and image blurring make the medical images unclear, thus, making their adjustment difficult when conducting in-depth features extraction. The Gray Level Co-occurrence Matrix (GLCM) proposed by (27) is applied, providing a good description of texture features. Given a scaled C.T. scan grey level image, GLCM is computed by enumerating the frequency of pair of pixels values. To fuse high-level texture features by considering the volumetric shape of lung nodules, a new 3D-GLCM computation is proposed in this article, where, instead of applying the 3D-GLCM computations on the whole CT scan, the CT scan is divided into several volume spaces (VSs). Hence, these VSs are fused through the LSTM fusion-based architecture. We divided the CT scans into many VSs to (i) extract more spatially related features through 3D-GLCM calculations (ii) reduce the number of RNN units and time required.

Lung nodules consist of volumetric shapes captured in the sequence of C.T. scan slices. The adjacent slices in the CT scan are arranged in a way to allow information in the first slice to become related to the next slice in their spatial sense. Feed-forward CNN architectures can face problems in extracting discriminative features from this type of data, especially with many slices in a CT scan (i.e., decreasing the chance of long-term dependencies in the texture information extracted from the slices). Compelling features fusion in the deep networks, when applying CNN-based structures is inevitable, reveal a high impact on the lung cancer classification performance (28, 29) , while the immediate consideration of the sequential organization of C.T. slices in feature fusion allows modeling the properties of volumetric nodules, which are influential in classification.

Recurrent neural networks (RNN) can process represented information using feedback connections between several units sequentially; therefore, applying RNNs is proposed to effectively combine GLCM features extracted from C.T. slices in nodule classification. An experimental extension is run based on this proposed LSTM fusion structure to classify lung nodules into the three: benign, malignant, and ambiguous classes. An approximation of the 3D-GLCM with 2.5D-GLCM based LSTM fusion structure is made. The 2D-GLCM based LSTM fusion is approximated to the 2D-slices-based CNN classification. In 2.5D-GLCM, the co-occurrence counts are extracted from every slice in eight directions moving through slices in the CT scan . In 3D-GLCM, the CT scan slices are first divided into many sub-volumes (VS)s, and then the co-occurrence frequencies are calculated for adjacent slices of the 3D VSs in 13 directions.

The main challenges addressed in this proposed method are (i) accurate and fast detection of nodule regions within slices, (ii) extraction of high-level GLCM-based features for nodule representation, and (iii) designing of an RNN-based deep neural architecture for the effective fusion of the extracted features and nodule classification.

Most of the methods applied in lung cancer detection resort to CT scan images. The subject of this study is to detect and classify to determine lung nodules accurately and determine whether they are cancerous or benign. The sequential structure of C.T. scans provides high motivation for volumetric processing of nodules in lung images for classification. This proposed method consis of 1) CT scans preprocessing 2) nodules detection 3) GLCM computations 4) LSTM fusion structure and 5) nodule classification.

The main contributions of this article: 

The lung is a vital organ in the body, and the changes in its texture due to cancer can lead to death. Medical imaging can assist lung tumor detection or tracking. CT scan is one of the best solutions in detecting lung cancer disease and identifying parts of the lung texture exposed to cancer. Screening for malignant lung tissue growth detection through C.T. scan apparatus can speed up detection leading to early treatment and saving lives. The CT scan structure, GLCM computations, RNNs architecture, and Otsu thresholding method are reviewed in this subsection.

CT scan images A computed tomography (CT) image applies X-ray and computer to produce cross-section images of internal organs of the human body. It scans a specific part of the body generating a tiny slice, by considering radio density measured by relative quantitative called Hounsfield unit (HU). One of the widely-applied methods to detect lung cancer is based on CT scan images (30) . Development stages of lung cancer reveal that cancer cells can have continuous growth inside the lung. Lung cancer cells can propagate to other parts of the body and boost the severity of the disease. Cancer stages can be classified and divided into four stages based on the nodules' size and the growth of the damaged area in the lung, Figure 1 ). Only a part of the lung, less than 3 cm, constitutes the cancer mass in the first stage. The bronchi are engaged in the second stage, and in the third stage, the trachea and sections on the lung's upper lope are affected by cancer cells, and in the last stage, both lung lobes become involved, and the probability of death will increase (31). Grey Level Co-Occurrence Matrix A method was proposed by [28] to extract texture features from grey-level images or two-dimensional intensity. In their method, an image texture can be described by counting particular values occurring in a pair of pixels in the image instead of providing information about the light intensity distribution. By establishing a new grey level of pixel neighborhoods matrix in different angles and distances, the features and characteristics of the image are determined through this method. The GLCM matrix is square, with the number of columns equal to the number of intensity levels in the image.

This matrix is defined and formulated through the two radius and angle parameters. Radius is the distance from the reference pixel, usually at 1 or 2, towards the considered neighbors. The degrees of eight neighbors when the radius is 1; are 0°, 45°, 135°, 90°, 180°, 225°, 270°, and 315°. In 2D-GLCM, the Co-occurrence matrix for the spatial relations between pixels is calculated through Eq. (1) [34] :

where I is a selected C.T. scan, p, a slice is the (x,y) coordinate of a pixel in the slice domain, I(p) is the intensity value at the point and , which the pairs of intensity values, d is a distance between pixel at point p and another point along angle . The 2D-GLCM can be expanded to 3D space (32) , where the co-occurrences are calculated from different slices to collect information from volumetric images. Here p in Eq. (1) refers to multiple pieces and has a coordinate of (x,y,z).

The RNN RNNs can process sequentially represented information; some RNN types have a memory to keep the calculation results. In contrast to the deep neural networks, which apply many parameters in their layers, the same parameters are applied in RNNs at each layer, reducing the count of parameters that need to be learned significantly. RNN Training is the same as training traditional neural networks. The backpropagation algorithm is applied because all parameters are applied similarly in any layer in the RNN; thus, the gradient at any output depends on the current and previous steps, named the Backpropagation Through Time (BPTT).

The drawback of this algorithm is the learning dependencies in the far-away context due to the vanishing/exploding gradient problem (33) . A Long-Short-Term Memory tackles a long-term dependency (LSTM) network [36] and remembers information for an extended period.

LSTM cells apply the gating technique to read, write and store information by opening and closing the three forget input and output gates in the LSTM unit. Processing features in the LSTM cell at step t-1 will pass the cell state −1 and the output ℎ −1 to the next LSTM cell in step t; in this step, the forget gate can decide the importance of information passed from LSTM in step t-1 if required, or it can be overlooked by passing the input features in LSTM at step t and ℎ −1 to sigmoid function. Sigmoid will yield value within [0,1] range, Eq.(2). If the value is closer to 1, the ℎ −1 will be applied in LSTM at step t, otherwise neglected.

(2) The same information, the output of the old LSTM unit output ℎ −1 and the current unit's input Are applied to generate a regulation vector through the tanh function, Eq. (3) and an input vector Eq. (4) (34) .

RNN has been and is being applied in classifying CT scans (35, 36) . In this study, an LSTM-based deep neural structure is proposed for fusing the features extracted in different modes of GLCM computation to classify lung nodules into benign, malignant, and ambiguous.

Otsu thresholding This method is one of the widely-applied methods in medical images segmentation. In this method, the image pixels are classified based on their light intensities' thresholding. One of the main challenges in this method is that, upon an increase in thresholds count, the time for thresholding will increase. The WSA algorithm is applied to increase the search speed in locating the classification thresholds to overcome this drawback. This method is applied in automatic image thresholding (37) . This method initially returns a single intensity threshold that separates pixels into the foreground and background classes. This threshold is determined by minimizing the intra-class intensity variance or maximizing the inter-class variance equivalently. This thresholding method is expanded into multi-thresholding. Increasing the thresholding bins increases the time required by an exhaustive search to find the best solutions.

Deep learning and machine learning algorithms have been and are being applied in different studies to detect and classify lung nodules. The noise in the image and nodules' orientation make the detection process complex and overwhelming. Pan et al. (38) introduce a double-path convolutional neural network (DPCNN) to reduce such complexities. It consists of a depletion and detection method in an end-to-end approach, where, first, a learning model is adopted for noise reduction in the preprocessing step; next, the DPCNN is applied as a feature extractor into double paths as the input for lung cancer detection. One path applies MobileNetV2, and the other a more massive input image but short wellproportioned layers. This method needs much learning time and does not provide a mechanism to improve segmentation. Lorz et al. [41] applied C.T. scan images to extract and preprocess histogram features based on local energy to detect lung cancer in different studies. Their proposed method is to improve the detection by applying preprocessing and local energy histograms to enhance the resolution and quality of the obtained images. Deep learning techniques are applied to find fused outputs from the training process and incorporate them in the analysis. Dominic et al. (39) applied the primary preprocessing method, where the Local Energy-Based Shape Histogram technique is applied to obtain better results. In this proposed method, the medium filters are applied to smoothen the images, improve their quality, and segment the images through segmentation techniques.

Priyadharshini et al. (40) developed a lung cancer detection method by applying a CNN combined with Bat Algorithm. They sought to detect cancer nodules in the lungs by applying the input image of the lungs and the structure of lung cancer formation. Features were extracted by applying the discrete wavelet transform. A fuzzy C-means clustering method is applied to determine lung cancer nodules. A modified CNN algorithm based on the bat algorithm is introduced to classify lung cancer effectively. Ray (41) , through analysis, revealed the essence of machine learning algorithms in predicting and detecting different cancers types. Recently, Riquelme et al. (42) assessed the advanced learning algorithms and architectures for automated detection systems applied in lung cancer detection. The existing lung C.T. scan datasets are compared with different techniques in detail. Kriegsmann et al. (43) applied deep learning to classify small and non-small cell lung cancers. A group of images of lung cancer is collected and applied for processing and assessment. An optimized architecture of InceptionV3 CNN, with the highest classification accuracy, is applied for classifying the test set. The Accuracy of CNN's classification results for detecting cancer patients on the test set is 95%. The potential and limitations of CNN-based image classification models in discriminating lung tumors are highlighted.

Bhandary et al. (44) reviewed a deep learning framework for detecting lung abnormality and revealed that the apropos detection of lung abnormalities reduces the risk through fast and efficient treatment. A deep learning framework is applied to analyze pneumonia and lung cancer. Two different deep learning solutions are of concern in assessing the problem: 1) the modified AlexNet for classifying C.T. scan images of the chest, where classification is made through an SVM, and 2) applying a mixture of handmade and learned features from deep learning to improve classification accuracy during lung cancer assessment. The feature extraction based on principal component analysis is applied to enhance the feature vector.

Many studies are run to propose better methods to classify lung cancer through 3D-CNN based on C.T. scan images. Pradhan et al. (45) applied some morphological processes to remove the lung nodules mask and present detected objects to the 3D-CNN model. Khumancha et al. (46) applied cubic masks to specify the region of interest (ROI) in C.T. scans and then presented the predicted regions to 3D-CNN for lung cancers detection. 3D-CNN has a prolonged training time. Most CNNs models fail to recognize noisy images, which leads to irrelevant feature extraction. Wang and Chakraborty (47) enhanced the lung cancer classification training efficiency by applying an average pooling on input nodule volumes to be presented to RNN. The spatial correlation in multiple slices representing nodules is ignored in their work. For lung cancer classification, Nurtiyasari and Abdurakhman (48) applied a wavelet RNN model to remove noise in the input raw images. RNN classifies the images into cancerous and non-cancerous.

Some researchers considered lung nodules classification based on GLCM textures' features. Khan and Alotaibi (49) proposed GLCM based neural network model. This method includes a preprocessing phase to extract the regions of interest from slices through GLCM computations. Then the extracted features are provided to radial bias function neural networks for this classification. Thohir et al. (50) provided texture features extracted from medical images by applying GLCMs to an SVM classifier. They concluded that applying a polynomial kernel yields better results. The count of studies runs on improving image segmentation for lung cancer detection is rising. Yu et al. (54) applied the K-mean method to inspect the similarity of pixels by dividing them into subgroups. These data were classified through deep-learning with instantaneously trained neural networks. Chithra and Roy (19) adopted the Otsu thresholding method for detecting lung nodules. Vas and Dessai (55) performed morphological operations on C.T. scans to remove the foreground edges. The closing procedure is served with a disk structural element to reveal the lung mask, and then a median filter is applied to process the internal features of the lung. Nizami et al. (56) applied the K-mean clustering to isolate the lung tissue, then applied wavelet packets containing five sub-bands with the most considerable energy and entropy to cluster coefficients in this method. Sha et al. (6) applied median and average filters to enhance the input images for thresholding through Otsu. Helen et al. (18) sought optimal thresholds by combining PSO with Otsu thresholding for CT scans segmentation by seeking to avoid the exhaustive search that may be taken through the Otsu search.

This method consists of five steps: CT scans preprocessing, nodules detection, GLCM computations, LSTM fusion structure, and nodules classification, Figure ) .

In the preprocessing process, the CT scans from the Hounsfield unit to greyscale are scaled, and the noise is removed from the images; the Otsu-WSA segmentation method is applied in lung nodules detection; the volumetric GLCM computations are made from the divided CT images into many VSs: the LSTM architecture-based deep fusion combines the computed volumetric GLCMs for the adjacent VSs. and the lung nodules are classified into benign, malignant, and ambiguous. 3D-GLCM computations The nodule volume is divided into many sub-volumes (i.e., VSs) for this proposed 3D-GLCM computations. To adjust the LSTM based fusion structure, the 3D-GLCMs from the sequenced subvolumes are computed to allow the volumetric shape of nodules structure to be of concern. In computing the 3D-GLCM, three-dimensional voxels are of concern instead of pixels in 13 directions to obtain voxels co-occurrences. Considering more directions may add more information, leading to unnecessary complexity and possible redundancy in the texture data. The fusion of adjacent VSs allows the model to consider more spatial information between slices in C.T. scans. The GLCMs are computed for each VS in the sense that the adjacent 3D-GLCMs are combined through this proposed LSTM fusion architecture.

Deep recurrent structure for fusion The structure of the proposed 3D-GLCMs fusion model is shown in Fig. (3) . The 3D calculations in each VS consider more spatial information between slices within the same VS. The extracted features from one VS is fused with the adjacent VSs in the LSTM layer. The slices of CT scan X are divided into the p count of the VSs., and computations are run for each VS the 3D-GLCM. The obtained 3D-GLCMs are provided to p count of LSTM units, consequently, the representation of the spatial information from the features extracted from the adjacent VSs are adjusted internally. The contextual information, the features of one VS, processed in one LSTM unit is transferred to the following LSTM. 

The WSA is combined with the Otsu thresholding technique to enhance image segmentation. In this proposed method in phase one, many random threshold vectors as female and some male insects are established initially as the members of the water strider (WS) population. In phase two, a male solution can mate with a probability of p, Eq. Error! Reference source not found. to be used for updating the thresholds using the mating process:

where +1 is the position of WS or an Otsu thresholding vector at iteration t+1, is the location of the ℎ WS at step t, r is a random count within [0,1] range, and R is the distance between a male WS and a female WS is computed through on Eq.(6):

When the foraging behavior of water striders is of concern, the Otsu thresholds can be updated through Eq. (7):

A WS searches the space between itself and the best solution to find a better solution. If a solution is not improved after foraging , it is moved to another territory for more foraging (getting closer to the best solution). If the quality of the solution is still low, then it is replaced by a new random WS larva inside the source territory, according to Eq.(7) By applying this optimization approach to the Otsu thresholding algorithm, the best thresholds at each iteration of the WSA are extracted. The last iteration contains the best thresholds applicable in image segmentation.

To define the maximized objective function, let be the proportion of pixels with an (i) intensity obtained through Eq.(8):

where is the frequency of pixels in the image with an intensity of i, N, and L are the total pixel count and the grey level count in the image, respectively. For thresholding images into multiple classes { 1 , 2 ,.... }, m is the thresholds count sought and WSA is the population of threshold vectors. In Otsu thresholding, the summation of intensity frequencies in each class (mean proportion of grey levels in a class) is obtained through Eq.(9):

The weighted average of intensities based on the considered thresholds (cumulative probability) is obtained through Eq.(10). In the next step, the average intensity in all of the classes for the whole image is obtained through Eq.(11): 

in the above equation, is the intensity weighted average in the ℎ class and is the image intensities average. The objective function will be defined through Eq. (12) to obtain the maximum value (57):

The Pseudo-code of the hybridization process is shown in Algo1. if the objective function is more valuable than the earlier case, then 9.

recover the new position 10.

Else 11.

The new position of the WS is updated for it to move to a new territory containing more food through Eq . Error! Reference source not found.

The WS is evaluated in the new territory by the Otsu objective function. 13. endif 14.

if The objective function value is still low, then 15.

the WS will be replaced by a ripened larva through Eq. (1) 

The experiments are run to assess the available methods' potential in lung cancer detection and classification through an LSTM-based deep fusion structure with different modes of GLCM computations. TensorFlow version 2.4.0 is applied for all implementations, and the experiments are run in a system with Windows server O.S and a GEFORCE RTX 2080 GPU.

Noise removal is considered one of the more accurate segmentation and features extraction preprocessing measures. In this context, the average filter or median filter methods, can be adopted. As to the median filter, no new noise intensity is introduced, consequently, it is of concern in this study. The output of the median noise removal method is expressed through Eq. (13):

where K is the pixels count adjacent to the central pixel, and is the input image, with a ̌ value after noise removal.

Experimental setup The following three experiments are run to express the superiority of this LSTM based fusion structure in this study:

In the first, the fusion structure considering GLCMs computations for 2D-slices is established. Each slice in a CT scan is scaled into (0-255) greyscale levels, and after nodules detection, the 2D-GLCMs are calculated for each of the eight directions. The interaction between spatial information calculated from each direction is learned by fusing these GLCMs in an LSTM-based architecture. In this experiment, each slice is labeled benign, malignant, and ambiguous. The features extracted from ordered directions beginning from 0° to 315° represent the spatial context information fused internally in the network LSTM layer. The information is obtained from the first LSTM unit, where the features from direction (0°) are processed, and passed to the next LSTM unit, where the features from direction (45°) are processed, and this trend continues; thus, the information passing from every unit is considered in the subsequent units. Such a mechanism captures the dynamic variations in a contextual state over input features history while fusing these features to determine the correlation between each reference pixel and its neighbors in all directions, which ignores the reality of volumetric nodules shapes.

In the second, the 2.5D-GLCM computations are of concern, where the features are extracted from multiple slices. The CM coordination is (x,y,z) for each slice, where z is the stacked feature at every direction Figure 3 showing the fusion structure proposed for 2.5D-GLCMs. The volumetric shape of nodules in CTs motivates researchers to construct this model. The LSTM network fuses the features extracted from different slices by an internal representation of the contextual spatial information among adjacent slices. In this experiment, a richer context of information is obtained not merely based on the information of each slice but all in a CT scan. The processed features at every LSTM unit in the network pass the contextual state information to the next unit. When fusing these GLCMs through this proposed LSTM-based deep neural architecture, the correlation of adjacent slices is considered to give it an advantage over other fusion methods where CNNs is applied [34] , which effectively ignores the spatial information among the slices. Each LSTM unit should wait until the previous LSTM unit completes its processing, and this waiting will elongate time because every CT scan consists of long series of slices. In the third, this newly proposed LSTM fusion structure is adopted to fuse the 3D-GLCM computations, where the nodule volumes are divided into many VSs and then computed through the 3D-GLCM for the adjacent VSs are of concern.

In the 2D-GLCM method, the CM is calculated for each slice in eight directions, the shape of the extracted features (8×255×255). After vectorizing the features, the shape will become (8×65025), where 65025 of the features will be provided for deep neural LSTM architecture. Eight LSTM units are considered in the input layer of the model. This model contains three LSTM and one dense layer. The vectorized features are fed into eight parallel sequence-to-sequence LSTMs with 128 units; the third LSTM layer is a sequence-to-one that outputs a vector of 128 tensors. This vector is provided for three units of dense layer for classification. In this model, the Relu activation function is involved in the LSTM layers and the softmax in the last layer.

In the 2.5D-GLCM mode, the CM is calculated for p slices in a CT scan. The features are padded because lung nodules can extend to a count of slices within the CT scan. The shape of features extracted from each slice is (8×255×255). After stacking these features in p adjusted slices, the shape of features will become (p×8×255×255). Next, vectorizing the features converts the shape of the input feature into (p×520200); where p is the LSTM units count, and 520200 features are provided for every unit. This model consists of three LSTM and two dense layers. Parallel sequence-to-sequence LSTM units fuse the vectorized elements with 128 units. The last LSTM layer is a sequence-to-one layer with a vector output; this vector is fed to a fully connected layer with 32 units. The last layer is a dense layer of three units. The Relu activation function is applied in all layers and softmax in the last layer for classification.

In the LSTM architecture to fuse 3D-GLCMs, the 3D CM is calculated for every VS in 13 directions. The shape of features extracted from each VS is (13×255×255), and after stacking the features extracted from adjusted VSs, it becomes (p×13×255×255). After padding and vectorizing these features, the input shape to the model will be (p×845325), where p is the LSTM units count, each provided with 845325 features. This model is similar to 2.5D-GLCM consisting of three LSTM layers and two dense layers. The last layer is the output layer containing three units of a dense layer with a softmax activation function for classification. The Relu activation function is applied in the remaining layers, Table 1 . Table 1 the essential parameters count applied in the designing this proposed method are reported for the three experiments.

Dataset The LIDC-IDRI dataset proposed in (58) is applied to assess these methods. This dataset embodies the analytical CT scans obtained from lung, designing CAD system to detect and classify lung carcinoma. The cases in this dataset are 1018, collected and compiled through contributions made by multiple medical imaging institutes and academic hubs. Each subject's case includes CT scan images and their associated XML file to allow four radiologists to comment on. The following three nodule sizes are considered as the classification criteria for the entire lesions:

The primary task of a radiologist is discovering as many lung nodules in the entire CT scan images as possible. The C.T. scan images applied in this study are arranged in a sequence of slices for every patient; each image is labeled malignant, benign, or ambiguous.

According to the radiologists' expert views, 796 nodules diameter ≥3mm within the selected CT scans. The initial 300 cases are selected for the experiments. In this study, some cases do not have nodules, or if have they are less than 3mm in diameter, indicating a low probability of being malignant. After removing the CT scans with inconsistent slice space and clean CT scans (i.e., CT scans with no nodules), 779 nodules were obtained for our experiments. Likewise, we adopted the method in(5) to categorize nodules classes.

In this study, one slice from each nodule (779) is selected randomly to assess the 2D-GLCM model and WSA-Otsu segmentation. The data applied for training is 80% and for testing the models is 20%. All the slices are first converted into greyscale intensities (0, 255).

6.1. Evaluation criteria Lung cancer detection is a classification problem where the CT scan images are classified as malignant, benign, and ambiguous. In most of the available studies, the focus is on lung cancer detection, where indicates like accuracy, sensitivity, and specificity are of great concern. In this study, these indicators are applied to determine the efficiency of the proposed approaches. A multiclass one-vs-rest (OvR) algorithm is applied to compute the ROC AUC curves average for the three classes. The macro and micro averaging methods are applied for experimental comparison. The 10-fold cross-validation in training and testing is applied here.

The experimental performance of this proposed WSA-Otsu segmentation is compared with other comparative thresholding methods. For this comparison, 779 slice images are selected to cover different types, shapes, and sizes of the nodules. The detection accuracy is computed according to the reference masks provided by the dataset. The comparison made between the WSA-Otsu and the latest proposed thresholding methods is tabulated in Table 2 , where, as observed, WSA-Otsu outperforms other methods in nodules detection. The F-score indicates the highest value in the proposed methods. 96.8% The thresholding time of the WSA-Otsu method and the latest proposed methods, where 2, 3, 4, and 5 thresholds are applied in nodule detection of the C.T. scan images of the lung, are shown in Fig. (5) . The algorithm's execution time increases, and the segmentation becomes slow as the count of thresholds increases is the primary challenge in the Otsu thresholding method. Exploiting WSA to find the proper thresholds in this proposed method reduces the execution time of thresholding. By analyzing the diagram in Figure 4) , it is evident that as the count of the thresholds increase, the thresholding time in the competitive thresholding methods becomes considerably higher than that of this proposed method. Assessing the acceleration and speed of the segmentation process reveals that when the count of thresholds is 5, this proposed method provides a 9.1 times faster rate than that of the Otsu method in finding the thresholds and a 2.3 times faster rate than that of the DPSO-Otsu method. 

The three modes of GLCM are of concern, to extract the texture features and provide them to LSTM neural networks for fusion and classification. The fusion network for 2D-GLCMs is trained with data from 779 slices selected from the dataset for different patients, where 155 are booked for the testing phase. Every slice is labeled into one of the benign, malignant, and ambiguous classes, depending on the nodule they are taken from. ROC AUC curves are first plotted by applying the macro-average. In this setup, the metric is computed independently for each class, followed by computing their average. The ROC AUC plot computed based on macro-average is shown in Fig. (6) . Because of the high rate of ambiguous classes within the subject dataset, allocating an equal average to all classes may prevail the other classes; the ROC AUC is computed based on the micro-average method. In this case, the impact of each class is aggregated for computing the average metric. The ROC AUC plot for averagemicro computations is shown in Fig. (7) . The final test results on the accuracy, sensitivity and specificity for 2D-GLCM fusion are 94.4%, 91.6%, and 95.8%, respectively. For 2.5D-GLCM, 300 CT scans are selected from this dataset to be analyzed by expert radiologists. There exist 779 nodules with diameter ≥ 3mm in the CT scans, among which 155 are tested. The accuracy, sensitivity, and specificity of 2.5-GLCM fusion are 97.33%, 96%, and 98%, respectively.

The LSTM-fusion for 3D-GLCM is assessed on 155 nodules, and the obtained accuracy, sensitivity, and specificity are 98.7%, 98%, and 99%, respectively. The objective of this study is to reveal how features extracted from a series of CT scan slices are represented to alleviate the hurdle of classifying lung nodules by CNNs. The different CNN schemes are implemented to confirm the potential of this proposed method in the run experiments. MobileNetV2 (59), EfficientNet-B5 (60) and 3D-CNN (46) are trained through the raw CT images. In this study, the last layers of the networks are considered as the output of the three-class nodules, Table 3 . As observed, the 2D-GLCM based LSTM network outperforms MobileNetV2 and EfficientNet-B5, which are trained through the raw 2D-slices. However, it is of a lower performance than 3D-CNN trained with multiple raw 2D-slices. The 2.5D-GLCM, extracted from multiple 2D-slices of a nodule and provided to the proposed LSTM-based fusion layer, outperforms the 3D-CNN trained through multiple raw 2D-slices. Acknowledging the volumetric appearance of lung nodules can raise the effectiveness of model's classification. The 3D-GLCM calculation is based on the VSs within the CT scan, while in [34] , it is based on the whole CT scan. The results obtained through VSs are provided to the proposed LSTM-based deep neural architecture for a fusion where robust performance is observed compared to the same presented in (32) . It is observed that applying and fusing compelling texture features for training classification models leads to better efficiency than end-to-end learning.

The performance of the methods proposed here is compared with the latest alternative approaches Table  4 . The CAD proposed in (65) , which applies linear discriminant analysis, achieved 70% sensitivity. The CAD system in (66) , which applied an artificial neural network, achieved only 80.3% sensitivity. Moreover, the CAD system (67) , which used cylindrical filters and SVM to detect and classify NSCLC, showed just 80% sensitivity.

The results indicate that 3D-GLCM based LSTM classification model has higher accuracy in classifying the lung nodules. The LSTM fusion approach can learn higher level and more effective representations from the valuable features extracted from the 3D-GLCM compared with the raw CT scan image data applied in (68) . Note that the advantage of applying 2.5D-GLCM fusion over raw images for lung cancer classification is of concern. Ablation study Two experiments are run on the separated nodules dataset to exhibit the contribution of the volumetric GLCM computations in this study. In the first, 16 texture feature descriptors are calculated from the 3D-GLCMs, to be fed to the features through this proposed LSTM fusion architecture; in the second, the 3D-GLCMs are combined for all VSs for 13 directions to allow this LSTM fusion architecture to combine the texture features extracted from each direction, Table 5 . 88.3% As observed, when the feature descriptors are calculated from the 3D-GLCMs, the accuracy, sensitivity, and specificity decrease. Combining the 3D-GLCMs for 13 directions based on this fusion architecture decreases the model classification performance. The first experiment results can be assigned to valuable information lost when the texture feature descriptors are estimated; the second experiment results can be attributed to the uncorrelated GLCMs' directions.

Discrimination of malignant and benign tumors is one of the most challenging tasks of physicians in analyzing CT scan information. In this study, a 3D-GLCM based VSs fused in an LSTM-based deep neural is proposed to learn more valuable representation features, a task difficult for CNNs based models. In this proposed method, the 3D-GLCM from VSs within one C.T. scan is computed to represent more valuable tissue texture data. This LSTM-based deep network fusion for adjacent VSs features is applied for classifying lung nodules on behalf of the spatially correlated information between sequenced VSs.

Resorting to machine learning models to classify and detect lung cancer based on C.T. scan images was a common approach among the researchers; nowadays, deep learning methods like the CNNs prevail in the same context. In practice, applying 3D-GLCM features fused in an LSTM based architecture is more accurate than providing 3D-GLCM to CNN-based model presented in (32) .

In its experimental sense, LSTM is proved to be better for modeling the irregularity of sequential data patterns in the series of slices in CT scans. GLCM is calculated in different modes, where, first, each slice is considered separately and classified into begin, malignant, and ambiguous, and next, the LSTM based architecture fused features from different directions (2D-GLCM). LSTM manifests higher accuracy than other deep transfer CNN methods, like the MobileNetV2, EfficientNet-B5, and ResNet50, which apply a single slice as the input.

The features are extracted from the sequenced slices and fed to the LSTM (2.5D-GLCM), the accuracy of which is more than that of the 2D-GLCM mode.It is higher accuracy than providing GLCM to CNN is presented in (32) . Fusing these features into LSTM based neural network achieves the best accuracy. The significance of calculating 3D-GLCM from adjacent VSs is experimentally proven.

The advantage of the improved Otsu method is that it requires less execution time than the conventional Otsu, especially at a high threshold count, and increases the neural network's learning speed. Based on the experiments' results, these proposed WSA-Otsu method is highly contributive in lung nodules detection. The thresholding time is reduced by 9.1-fold compared to the standard Otsu and 2.3-fold compared to the Otsu-based DPSO heuristic algorithm. The results of applying this proposed WSA-Otsu and other thresholding methods on medical images are tabulated in Table 2 , where, as observed, WSA-Otsu has the highest score.

Lung nodules texture features can be extracted in different methods. It is possible to increase the efficiency of features to be highly valuable. Fusing features like the local binary patterns, shape index histograms, local energy patterns, and grey level length matrix can accomplish the set objectives.

Combining the similar elements in a feature fusion framework to learn the adopted features close to one another can be a theme/topic for future studies. As to the newly developed approach named the attention mechanism focusing on high-level features, the researchers here plan to run a study on how to avoid irrelevant or redundant features when GLCMs are computed from different directions by applying the attention mechanism. The, non-local operations can be designed to extract the global and local highlevel features of the lung nodules.

Early Neutrophil Responses to Chemical Carcinogenesis Shape Long-Term Lung Cancer Susceptibility. iScience

Lung Cancer 2020: Epidemiology, Etiology, and Prevention

Segmentation of Lung Lobes and Nodules in CT Images

18F-FDG PET/CT and whole-body MRI diagnostic performance in M staging for non-small cell lung cancer: a systematic review and meta-analysis

Prediction of Malignancy in Lung Nodules Using Combination of Deep, Fractal, and Gray-Level Co-Occurrence Matrix Features. Big data

Measuring continuous landscape patterns with Gray-Level Co-Occurrence Matrix (GLCM) indices: An alternative to patch metrics? Ecol Indic

A Survey on Convolutional Neural Network (Deep-Learning Technique) -Based Lung Cancer Detection

Effect of Radiation Doses to the Heart on Survival for Stereotactic Ablative Radiotherapy for Early-stage Non-Small-cell Lung Cancer: An Artificial Neural Network Approach

An effective tumor classification with deep forest and self-training

A k-NN method for lung cancer prognosis with the use of a genetic algorithm for feature selection

Validating deep learning inference during chest X-ray classification for COVID-19 screening

Deep learning combined with radiomics may optimize the prediction in differentiating high-grade lung adenocarcinomas in ground glass opacity lesions on CT scans

A Fast 2-D Otsu lung tissue image segmentation algorithm based on improved PSO

Multi-shape graph cuts with neighbor prior constraints and its application to lung segmentation from a chest CT volume

Design Computer-Aided Diagnosis System Based on Chest CT Evaluation of Pulmonary Nodules

Robust Initialization of Active Shape Models for Lung Segmentation in CT Scans: A Feature-Based Atlas Approach. Wei G, editor

Unsupervised-Restricted Deconvolutional Neural Network for Very High Resolution Remote-Sensing Image Classification

Segmentation of pulmonary parenchyma in CT lung images based on 2D Otsu optimized by PSO

Otsu's Adaptive Thresholding Based Segmentation for Detection of Lung Nodules in CT Image

A Threshold Selection Method from Gray-Level Histograms

A New Iterative Triclass Thresholding Technique in Image Segmentation

Modified and Optimized Method for Segmenting Pulmonary Parenchyma in CT Lung Images, Based on Fractional Calculus and Natural Selection

A Tutorial on Support Vector Machines for Pattern Recognition

Multi-modal trained artificial intelligence solution to triage chest X-Ray for COVID-19 using pristine ground-truth, versus radiologists

Fusing texture, shape and deep model-learned information at decision level for automated classification of lung nodules on chest CT

3D Texture Feature Extraction and Classification Using GLCM and LBP-Based Descriptors

Textural Features for Image Classification

Importance of Preprocessing in Histopathology Image Classification Using Deep Convolutional Neural Network

UniFormer: Unifying Convolution and Self-attention for Visual Recognition

Content-based image retrieval with a Convolutional Siamese Neural Network: Distinguishing lung cancer and tuberculosis in CT images

Efficient pre-processing and segmentation for lung cancer detection using fused CT images

A 3-Dimensional Gray-Level Co-Occurrence Matrix-Based CNN Model for Polyp Classification via CT Colonography

On the difficulty of training recurrent neural networks

Application of Deep Learning in Lung Cancer Imaging Diagnosis

Automatic detection of abnormalities in retinal blood vessels using dtcwt, glcm feature extractor and cnn-rnn classifier

Detection and Classification of Pulmonary Nodules Using Convolutional Neural Networks : A Survey

Texture feature extraction methods: A survey

Dual-Resolution Dual-Path Convolutional Neural Networks for Fast Object Detection. Sensors (Basel)

Lung Cancer Detection using Local Energy-Based Shape Histogram ( LESH ) Feature Extraction Using Adaboost Machine Learning Techniques

Bat-inspired Metaheuristic Convolutional Neural Network Algorithms for CAD-based Lung Cancer Prediction

A Survey on Application of Machine Learning Algorithms in Cancer Prediction and Prognosis

Deep Learning for Lung Cancer Nodules Detection and Classification in CT Scans

Deep Learning for the Classification of Small-Cell and Non-Small-Cell Lung Cancer. Cancers (Basel)

Deeplearning framework to detect lung abnormality -A study with chest X-Ray and lung CT scan images

Lung Cancer Detection using 3D Convolutional Neural Networks

Lung Cancer Detection from Computed Tomography (CT) Scans using Convolutional Neural Network

Evaluation of Malignancy of Lung Nodules from CT Image Using Recurrent Neural Network

The application of Wavelet Recurrent Neural Network for lung cancer classification

Computerised Segmentation of Medical Images using Neural Networks and GLCM

Classification of Colposcopy Data Using GLCM-SVM on Cervical Cancer

A Least-Squares Framework for Component Analysis

Multimodel Feature Reinforcement Framework Using Moore-Penrose Inverse for Big Data Analysis

Deep Learning Algorithm for Classification and Prediction of Lung Cancer using CT Scan Images

Deep Learning Assisted Predict of Lung Cancer on Computed Tomography Images Using the Adaptive Hierarchical Heuristic Mathematical Model

Lung cancer detection system using lung CT image processing

A wavelet frames + K-means based automatic method for lung area segmentation in multiple slices of CT scan

Optimal multi-level thresholding using a two-stage Otsu optimization approach

Available from: wiki.cancerimagingarchive.net,LIDC-IDRI

MobileNetV2: Inverted Residuals and Linear Bottlenecks

Deep convolutional neural network-based classification of cancer cells on cytological pleural effusion images

Ensembling EfficientNets for the Classification and Interpretation of Histopathology Images

The History Began from AlexNet : A Comprehensive Survey on Deep Learning Approaches

Xception: Deep Learning with Depthwise Separable Convolutions

Learning Transferable Architectures for Scalable Image Recognition

Computerized Detection of Pulmonary Nodules on CT Scans 1

Massive training artificial neural network (MTANN) for reduction of false positives in computerized detection of lung nodules in lowdose computed tomography

Fast lung nodule detection in chest CT images using cylindrical noduleenhancement filter

Medical image analysis based on deep learning approach

Computerized detection of pulmonary nodules on CT scans

A new computationally efficient CAD system for pulmonary nodule detection in CT imagery

A novel computer-aided lung nodule detection system for CT images

Fast and Adaptive Detection of Pulmonary Nodules in Thoracic CT Images Using a Hierarchical Vector Quantization Scheme

Shape and texture based novel features for automated juxtapleural nodule detection in lung CTs

Feature extraction and LDA based classification of lung nodules in chest CT scan images

Lung tumour detection and classification using EK-Mean clustering

Classification of lung image and nodule detection using fuzzy inference system

Content-based image retrieval for Lung Nodule Classification Using Texture Features and Learned Distance Metric

Small lung nodules detection based on local variance analysis and probabilistic neural network

A novel approach to CAD for the detection of small cell and non-small cell lung cancers

Nodule Generation of Lung CT Images Using a 3D Convolutional LSTM Network BT -Advances in Visual Computing