key: cord-0875840-pdzlmij7
authors: Zhang, Xiaofang; Liu, Xiaomin; Zhang, Bin; Dong, Jie; Zhang, Bin; Zhao, Shujun; Li, Suxiao
title: Accurate segmentation for different types of lung nodules on CT images using improved U-Net convolutional network
date: 2021-10-08
journal: Medicine (Baltimore)
DOI: 10.1097/md.0000000000027491
sha: adc62ad533c105e89a144d9854425d326c845187
doc_id: 875840
cord_uid: pdzlmij7

Since lung nodules on computed tomography images can have different shapes, contours, textures or locations and may be attached to neighboring blood vessels or pleural surfaces, accurate segmentation is still challenging. In this study, we propose an accurate segmentation method based on an improved U-Net convolutional network for different types of lung nodules on computed tomography images. The first phase is to segment lung parenchyma and correct the lung contour by applying α-hull algorithm. The second phase is to extract image pairs of patches containing lung nodules in the center and the corresponding ground truth and build an improved U-Net network with introduction of batch normalization. A large number of experiments manifest that segmentation performance of Dice loss has superior results than mean square error and Binary_crossentropy loss. The α-hull algorithm and batch normalization can improve the segmentation performance effectively. Our best result for Dice similar coefficient (0.8623) is also more competitive than other state-of-the-art segmentation algorithms. In order to segment different types of lung nodules accurately, we propose an improved U-Net network, which can improve the segmentation accuracy effectively. Moreover, this work also has practical value in helping radiologists segment lung nodules and diagnose lung cancer.

Pulmonary hypertension, [1] COVID-19 [2] and lung cancer are all severe lung diseases. Cancer is a major health problem worldwide. [3] With the highest morbidity and mortality rate, lung cancer continues to be the most commonly diagnosed cancer. It is the leading cause of cancer-related deaths among men and the second leading cause of cancer-related deaths among women in China. [4] Therefore, early detection, diagnosis and treatment of lung cancer is essential to improve the five-year survival rate. [5] Lung nodule is the early manifestation of lung cancer, so accurate detection and segmentation of lung nodule is crucial to diagnosis and treatment of lung cancer. In recent years, different medical imaging modalities such as computed tomography (CT), magnetic resonance imaging, and positron emission tomography have played an important role in disease diagnosis. Due to the advantages of highest sensitivity, rapid acquisition, and low application cost, [6] CT imaging analysis is a versatile tool in lung nodule detection and diagnosis system.

Since the appearance of a lung nodule is a small, rounded or irregular opacity, it is a time-consuming and laborious task for radiologists to detect and segment all the suspicious diseased regions manually from CT imaging slices. Based on this, computer-aided detection (CAD) systems have been developed to help radiologists diagnose lung cancer at an early stage by detecting and segmenting lung nodules. [7] At present, accurate lung nodule segmentation is still a prerequisite step in CAD system. Lung nodules can have different shapes, contours, textures, and locations. Besides, some of them may be attached to surrounding tissues, such as blood vessels or pleural surfaces. From this point of view, they can be divided into 4 categories named well-circumscribed, juxta-vascular, juxta-pleural, and pleural tail nodules. [8] Until now, accurate segmentation for different types of lung nodules is still a complex and challenging problem.

Lung nodule segmentation mainly solves the problem of identifying the contour of nodule and separating nodule from lung region. Up to now, there are many approaches to accomplish this task. In general, these approaches can be classified into traditional segmentation methods and machine learning algorithms. As for the former, thresholding, region growing, connected component analysis (CCA), morphological filter, clustering algorithm, active contour model, and graph-cuts have been widely used. [9] [10] [11] [12] [13] [14] [15] For example, Setio et al [16] detected juxtapleural pulmonary nodules by using thresholding, region growing, CCA and morphological filter. Farhangi et al [7] applied sparse linear combination of training shapes based level set to detect nodules of all types. Cha et al [15] proposed graph-cuts method incorporating shape prior and motion to detect wellcircumscribed, vascularized, and juxta-pleural pulmonary nodules. In the machine learning algorithms, researchers have proposed a variety of networks for the task of lung nodule segmentation. For example, Wang et al [17] developed central focused convolutional neural networks (CF-CNN) for lung nodule segmentation. Pang et al [18] proposed a novel unified and end-to-end adversarial learning framework CTumorGAN for automatic tumors segmentation from CT scans. Deep deconvolutional residual network was proposed by Singadkar et al [19] for the lung nodule segmentation from the CT images. Even though the above approaches were used to segment lung nodules, some of them could not segment different types of lung nodules and some did not describe the types of lung nodules. Besides, for juxta-pleural and pleural tail nodules that usually have a similar density to neighboring pleural surfaces, some approaches used morphological or rolling-ball method to correct lung contours so that these nodules could be included into the lung regions. However, these 2 contour correction methods are difficult to obtain optimal parameters for different types of lung nodules automatically and often have under-repair or over-repair problems. [20] As a special kind of convolutional neural network, U-Net was first introduced by Ronneberger et al in 2015 [21] for biomedical image segmentation. This network can be trained end-to-end with very few images and yield more precise segmentations. Since then, many researches based on U-Net have appeared. Shaziya et al [22] implemented U-Net ConvNet on lungs dataset to perform lungs segmentation. Tong et al [23] introduced residual network and batch normalization (BN) to improve the U-NET network for lung nodule segmentation. Cao et al [24] designed a U-Net segmentation network based on ResDense structure to perform initial detection of lung nodules. Zhang et al [25] proposed an architecture that combined the inception module with the densely connected convolutions based on U-Net architecture for blood vessel and lung segmentation. Although U-Net networks with different architectures were applied in literature, some were proposed to segment lung regions or blood vessels. Besides, some segmented lung nodules for classification task and did not provide segmentation performance. Furthermore, introduction of BN was used only in a few previous U-Net networks.

In this study, we propose an accurate lung nodule segmentation method that is based on an improved U-Net convolutional network. The main contributions of this study are in the following aspects: our method can segment different types of lung nodules accurately, including well-circumscribed, juxta-vascular, juxta-pleural and pleural tail nodules; in order to segment juxtapleural and pleural tail nodules, we apply a-hull algorithm to correct lung contours so that these nodules can be included into the lung regions. The a-hull algorithm can adaptively obtain optimal a values for different types of lung nodules and so it can avoid under-repair and over-repair problems effectively; and BN is introduced into the U-Net network to improve the segmentation performance.

The flowchart of system architecture is illustrated in Figure 1 . First, we segment the lung parenchyma, and then the contour of lung parenchyma has been corrected. After that, image pairs of patches containing lung nodules in the center and the corresponding ground truth are extracted to train the U-Net network into which BN is introduced for improving the segmentation performance. 

On CT slices containing lung nodules, we first convert the pixel values into Hounsfield unit and use threshold -480 Hounsfield unit to get binary images, then morphological opening and holefilling operations are applied to obtain thorax mask. Afterwards, Otsu thresholding method [26] is adopted on thorax region, morphological opening operation is carried out to remove blood vessels and CCA is used to label connected regions. The region with the area between 1500 and 50,000 pixels is selected as the lung parenchyma. By applying hole-filling operation once again, left and right lung masks are obtained.

Among the 4 lung nodule categories, juxta-pleural and pleural tail nodules are at the border of lung parenchyma and connected with pleural surface. In the lung parenchyma segmentation phase, these nodules may not be included into the lung regions. For this reason, the contour of lung parenchyma should be corrected. Some researchers have been devoted to solving this problem. Wang et al [27] presented a border correction scheme based on concaveconvex estimation to correct boundary. Gong et al [28] applied chain code algorithm to repair lung volume. Shaukat et al [29] used rolling ball algorithm to repair lung volume.

In this paper, we use the a-hull algorithm [30] to correct the lung contour. This algorithm aims to capture the "crude shape" of point sets. [31] The a values change from 10 units to 60 units with an interval of 5 units. The optimal a values for left and right lung are selected by comparing lung areas and Hausdorff distance between lung contours before and after applying the a-hull algorithm, respectively. Because for different types of lung nodules, the a-hull algorithm can obtain optimal a values adaptively, it can avoid under-repair and over-repair problems effectively.

In order to segment different types of lung nodules, we propose an improved U-Net convolutional network. It is based on the architecture of U-Net for lung segmentation in literature 22. By introducing BN, the improved network is depicted in Figure 2 . BN was first proposed by Ioffe and Szegedy in 2015. [32] It can adjust the distribution of activation values to an appropriate breadth for each layer and speed up the learning process. In addition, the model is less dependent on the initial value and overfitting can be suppressed.

As the input to the U-Net network, an image patch of 32 Â 32 pixels with lung nodule in the center and the corresponding ground truth with the same pixels are extracted. In the image patch, we use an intensity window range of [À1150, 350] to convert the 16-bit image into 8-bit image, and then normalize it to a range of 0 to 1. Adam optimizer was used to update the parameters of the model. The learning rate, batch size, and training epochs are 0.001, 32, and 10, respectively. For discussing the segmentation performance of different loss functions, mean square error (MSE), Binary_crossentropy loss, and Dice loss are applied in the model. These loss functions are defined as follows:

Binary crossentropy ¼ À y GT is the target value, y seg is the predicted value, A GT is the size of the ground truth, A seg is the size of the segmentation result by using the method, j.j is the size operator, ∩is the intersection operator, smooth is a nonzero parameter which can avoid the denominator in Eq. (3) to be zero. In the experiment, smooth is set to 1.0.

In this paper, experiments are executed by using Python 3.5.4 (Guido van Rossum, Google) and MATLAB R2015a (Math-Works) on a computer with 2.5 GHz CPU, 6GB RAM, and NVIDIA GeForce GT 640 M GPU.

The image dataset in this work has been obtained from the Lung Image Database Consortium and Image Database Resource Initiative (LIDC/IDRI) database (https://wiki.cancerimagingarch ive.net/display/Public/LIDC-IDRI), [33] which is the largest open resource of lung nodules in the world. This database is a webaccessible international resource for development, training and evaluation of CAD methods for lung cancer detection and diagnosis. Users can download relevant data for free for research purposes. Our study is based on this open source database, so ethical approval is not necessary. This LIDC/IDRI database includes 1018 exams. Each exam is composed of hundreds of thoracic images in DICOM format and corresponding diagnostic annotations in XML format that includes analyses made by up to four experienced radiologists. Concerning diagnostic annotations, there are 3 categories: ≥3 mm nodules, <3 mm nodules, and ≥3 mm non-nodules. In the first category, different radiologists have marked X and Y coordinates of each nodule's contour points independently. In the other 2 categories, only X and Y coordinates of each nodule's center points have been marked. In order to better evaluate the segmentation performance, we select the first type of nodules.

In the scope of this study, we include 358 lung CT slices and each image is 512 Â 512 pixels. In-plane pixel size varies from 0.5859 to 0.8984 mm/pixel and the average pixel size is 0.7609 mm/pixel. Nodule diameter ranges from 4 mm to 27 mm.

For better evaluating the segmentation performance of this method, we define the ground truth as GT 3 and the specific definition process is as follows:

(1) The XML file is parsed to obtain the X and Y coordinates of each nodule's contour points marked by different radiologists. Only the nodules with 4 radiologists' annotations are selected. 

Given its accuracy and simplicity, Jaccard index (JI) is a measure of spatial overlap and used for comparing the agreement analysis. JI is defined by:

∪ is the union operator. Moreover, Dice similar coefficient (DSC), Precision, Recall, and F1-score are also used as evaluation metrics and defined as follows:

Recall

TP denotes true positives, FN denotes false negatives and FP denotes false positives. These 3 indexes are calculated from the overlapping area of GT 3 and segmentation result.

There are different kinds of data augmentation methods such as rotation, flipping, shift, zoom and cropping to expand training data. In this work, rotation and flipping are applied to the patch of 32 Â 32 pixels. The patch is rotated counterclockwise with rotation angle 45°, 90°, 135°, 180°, 225°, 270°, 315°, respectively and flipped horizontally and vertically. Figure 3 illustrates the different images of data augmentation. After rotation and flipping, we obtain 3580 image pairs including image patches and the corresponding ground truth. As the inputs of U-Net network, these image pairs are split into training dataset and test dataset with ratio of 4:1 randomly.

To analyze the segmentation agreement, mean JI between 4 radiologists (lung nodule segmentation of each radiologist is compared to the other 3 and the mean JI can be obtained by averaging the inter-radiologist JI) is used as the reference in this study. Table 1 gives the mean JI between different radiologists for different types of lung nodules. It can be found that the overall JI is 0.7581 ± 0.0874, demonstrating that there is a significant variability even between experienced radiologists. The highest JI is 0.7778 for well-circumscribed nodules, the worst JI is 0.7399 for juxta-pleural and pleural tail nodules and the standard deviation is approximately 8%. This indicates a great disparity between radiologists for different types of nodules.

It illustrates performance comparison among different loss functions in Table 2 . From the experimental results, it can be seen that Dice loss has better results than MSE and Binary_ crossentropy. It is more than 6% higher in DSC than MSE As for JI, Dice loss has the best result 0.7637, which is higher than other 2 loss functions (0.7541, 0.7548).

In this paper, we apply a-hull algorithm to correct the lung contour and introduce BN to improve the U-Net network. To validate the effectiveness of these 2 components, ablation experiments are designed. In Table 3 , the first row (except the header) verifies the validity of applying a-hull algorithm, the second row verifies the validity of introducing BN and the third row is the results of our proposed method with both a-hull algorithm and BN.

Lung nodules can have a variety of shapes and different degrees of connection to surrounding tissues, so it is challenging to determine exactly the contour of nodules. For juxta-vascular, juxta-pleural and pleural tail nodules, because intensities in these cases are very similar to this in vascular and pleural space, segmentation is particularly difficult. As visible in Table 1 , there is disparity in JI for different types of nodules, illustrating the level of segmentation complexity especially for the last three categories. At present, multiple loss functions for training a network model are used for biomedical image segmentation tasks, including MSE, Binary_crossentropy loss and Dice loss. In this study, we have compared the performance of different loss functions for lung nodule segmentation. As it can be seen in Table 2 , Dice loss can achieve superior performance under most evaluation metrics than other functions. Furthermore, comparing the JI corresponding to different loss functions with JI between different radiologists, a significant finding we can observe is that results of MSE and Binary_crossentropy loss (0.7541, 0.7548) are lower than the latter (0.7581), but JI corresponding to Dice loss (0.7637) is higher than that. This verifies the effectiveness of applying Dice loss to train our proposed U-Net network.

To verify the validity of a-hull algorithm and BN, we design ablation experiments. Comparing the first row and third row in Table 3 , it can be found that the proposed method by applying a-hull algorithm to correct lung contour and introducing BN to improve U-Net network is an average of nearly 3.1% higher than the method only with a-hull. Besides, comparing the last 2 rows of the table, our proposed method (with a-hull and BN) is also acceptable. Although the result in Recall is lower than method with BN, the JI is improved by 2.6% and DSC is improved by 1.76%. Overall, by applying a-hull algorithm and BN in our method, the segmentation performance has been improved effectively.

Segmentation performance comparison of our proposed method with several state-of-the-art algorithms is summarized in Table 4 . Wang et al [17] proposed a data-driven model CF-CNN to segment lung nodules and showed strong performance for segmenting juxta-pleural nodules. This method extracted 3D patch 3 Â 35 Â 35 and 2D multi-scale patch 2 Â 35 Â 35 as the inputs to the CNN model. It was evaluated on LIDC dataset with average DSC of 0.8215. Mukherjee et al [34] used graph cut with a deep learned prior to segment lung nodules. The segmentation performance was evaluated on a separate validation dataset consisting of 93 solid-nodules and 35 part-solid nodules from LIDC database and the input image size was 80 Â 80 for the deep neural network. This algorithm achieved an average DSC of 0.69 for solid nodules and 0.65 for the part-solid nodules. Tong et al [23] presented an improved U-Net network for lung nodule segmentation. The input image size of network is 64 Â 64 and the final evaluation result was 0.736 for DSC. Pang et al [18] proposed a novel framework consisting of a generator network and a discriminator network for automatic segmentation of any kinds of tumors. This algorithm used the nonsmall cell lung cancerradiomics data from TCIA(https://wiki.cancerimagingarchive. net/display/Public/NSCLC-Radiomics). The input image size of the CTumorGAN network is 256 Â 256 and the final DSC is 0.7108. As for our proposed method, a significant result in Table 4 is that the DSC is 0.8623, which achieves superior performance than other algorithms for the task of lung nodule segmentation.

Lung nodule segmentation is an indispensable step for diagnosis and treatment of lung cancer. Because of different shapes, contours, textures and attachment of nodules, accurate segmentation is still a complex task in CAD systems. In this study, we propose an improved U-Net network with introduction of BN to segment different types of lung nodules accurately on CT images. Otsu thresholding is applied to segment lung parenchyma. After that, a-hull algorithm is used to correct lung contours so that juxta-pleural and pleural tail nodules can be included into the lung regions. The a-hull algorithm can obtain optimal a values adaptively for different types of lung nodules and so it can avoid under-repair and over-repair problems effectively. In order to train the improved U-Net network, patches of nodules and corresponding ground truth are extracted from CT slice images. We validate the method by using 358 lung nodules from the LIDC/IDRI database including well-circumscribed, juxta-vascular, juxta-pleural, and pleural tail nodules. After data augmentation, 3580 image pairs are obtained as the inputs of the U-Net network. Compared with MSE and Binary_crossentropy loss, Dice loss is more suitable for segmentation tasks. Contour correction by applying a-hull algorithm and introduction of BN into the U-Net network have played a good role in improving the segmentation performance. Our best result for DSC is 0.8623, which is also more competitive than other state-of-the-art segmentation algorithms. Our study can have practical value in helping radiologists accurately segment lung nodules and diagnose lung cancer.

However, this study also has some limitations and we plan to further improve it from several aspects in the future. Firstly, because lung contours are not convex for diaphragm, the a-hull algorithm in this study cannot correct juxta-pleural and pleural tail nodules attached to diaphragm accurately, so our next work will concentrate on improvement of contour correction algorithm. Secondly, because the nodule diameter ranges from 4 mm to 27 mm, we set the patch size as 32 Â 32 pixels. As for nodules with smaller diameter or larger diameter, this patch size will not be appropriate. Therefore, selecting proper patch size for nodules Table 4 Segmentation comparison of our proposed method with other state-of-the-art algorithms (overstriking represents the best segmentation performance in DSC).

Wang et al [17] 0.8215 Mukherjee et al [34] 0.69/0.65 Tong et al [23] 0.7360 Pang et al [18] 0.7108 Our proposed method (dice loss) 0.8623 DSC = Dice similar coefficient. with different diameters is also our next work. Last but not the least, we will further increase the dataset and extend the twodimensional network to the three-dimensional network so that more three-dimensional image information can be mined to improve the segmentation performance.

Beyond thrombosis: the role of platelets in pulmonary hypertension

Lung physiological variations in COVID-19 patients and inhalation therapy development for remodeled lungs

Cancer statistics

Retrospect and prospect for lung cancer in china: clinical advances of immune checkpoint inhibitors

Early lung cancer action project: overall design and findings from baseline screening

Automatic nodule detection for lung cancer in CT images: a review

3-D active contour segmentation based on sparse linear combination of training shapes (SCoTS)

Three-dimensional segmentation and growth-rate estimation of small pulmonary nodules in helical CT images

Multistage segmentation model and SVM-ensemble for precise lung nodule detection

Lung cancer detection using fuzzy auto-seed cluster means morphological segmentation and SVM classifier

A segmentation framework of pulmonary nodules in lung CT images

A novel approach to CAD system for the detection of lung nodules in CT images

An assisted diagnosis system for detection of early pulmonary nodule in computed tomography images

Pulmonary nodule detection in CT images based on shape constraint CV model

Segmentation and tracking of lung nodules via graph-cuts incorporating shape prior and motion from 4D CT

Automatic detection of large pulmonary solid nodules in thoracic CT images

Central focused convolutional neural networks: developing a data-driven model for lung nodule segmentation

CTumorGAN: a unified framework for automatic computed tomography tumor segmentation

Deep deconvolutional residual network based automatic lung nodule segmentation

An automated segmentation method for lung parenchyma image sequences based on fractal geometry and convex hull algorithm

Convolutional Networks for Biomedical Image Segmentation

Automatic Lung Segmentation on Thoracic CT Scans using U-Net Convolutional Network

Improved U-NET network for pulmonary nodules segmentation

A two-stage convolutional neural networks for lung nodule detection

DENSE-INception U-net for medical image segmentation

A threshold selection method from gray-level histograms

Pulmonary Nodule Segmentation with Modified Variable N-Quoit Filter Combining Border Smoothing and Correction

Automatic detection of pulmonary nodules in CT images by incorporating 3D tensor filtering with local image feature analysis

Fully automatic detection of lung nodules in CT images using a hybrid featureset

Automatic detection and segmentation of lung nodules in different locations from CT images based on adaptive a-hull algorithm and DenseNet convolutional network

On the shape of a set of points in the plane

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): a completed reference database of lung nodules on CT scans

Lung Nodule Segmentation Using Deep Learned Prior Based Graph Cut

Editor: Neeraj Lalwani.The authors have no conflicts of interest to disclose.The datasets generated during and/or analyzed during the current study are publicly available.