key: cord-0101247-bzkmhn6s
authors: Zhang, Lipei; Liu, Aozhi; Xiao, Jing; Taylor, Paul
title: Dual Encoder Fusion U-Net (DEFU-Net) for Cross-manufacturer Chest X-ray Segmentation
date: 2020-09-11
journal: nan
DOI: nan
sha: eeb7b153d2cd9b4aa3ce620bb2f1e538596a9309
doc_id: 101247
cord_uid: bzkmhn6s

A number of methods based on the deep learning have been applied to medical image segmentation and have achieved state-of-the-art performance. Due to the importance of chest x-ray data in studying COVID-19, there is a demand for state-of-the-art models capable of precisely segmenting chest x-rays before obtaining mask annotations about this sort of dataset. The dataset for exploring best pre-trained model is from Montgomery and Shenzhen hospital which had opened in 2014. The most famous technique is U-Net which has been used to many medical datasets including the Chest X-ray. However, most of variant U-Net mainly focus on extraction of contextual information and dense skip connection. There is still a large space for improving extraction of spatial feature. In this paper, we propose a dual encoder fusion U-Net framework for Chest X-rays based on Inception Convolutional Neural Network with dilation, Densely Connected Recurrent Convolutional Neural Network, which is named DEFU-Net. The densely connected recurrent path extends the network deeper for facilitating context feature extraction. In order to increase the width of network and enrich representation of features, the inception blocks with dilation have been used. The inception blocks can capture globally and locally spatial information by various receptive fields. Meanwhile, the features fusion of two path by summation preserve the context and the spatial information for decoding part. This multi-learning-scale model are benefiting in Chest X-ray dataset from two different manufacturers (Montgomery and Shenzhen hospital). The DEFU-Net achieves the better performance than basic U-Net, residual U-Net, BCDU-Net, modified R2U-Net and modified attention R2U-Net. This model is proved the feasibility for mixed dataset. The open source code for this proposed framework will be public soon.

In the era of COVID-19 pandemic, the medical image automatic segmentation, especially in chest X-rays, becomes a key step for automatic abnormality recognition and analysis. The segmentation models with accurate and high performance can accelerate clinic work flow and assist doctors to make more reasonable decision for patients. Deep learning is promoting the development of medical image segmentation. Compared to traditional models of computer vision, the deep learning method transcend the limitations of scope of application [15] . This near-radiologist level achievement of deep learning can also be attributed to the rise to convolutional neural network (CNN). Filter transformation and efficient representation learning are crucial characteristic. Ever since AlexNet [14] has gained huge improvement on the classification on the ImageNet dataset [6] , various convolutional structures have been proposed such as residual block [8] , densely connected block [9] and inception block [22] . The networks are able to reach deeper and wider, which helps extraction of lowdimensional and high-dimensional features. Moreover, some useful activation functions help network simulate the human brain to output results such as ReLU, LeakyReLU, Sigmoid and Softmax. Some efficient optimization algorithms update parameters and accelerate the convergence. For example, stochastic gradient descent (SGD) and Adam optimizer are used in the most of training.

Back to the medical image segmentation, many networks based on the CNN make performance approaching the judgement of radiologist. The ground-breaking segmenting network is Fully convolutional network (FCN) [17] . After that, more researchers proposed more complex frameworks for improving efficiency of encoder and decoder. For most medical image datasets, the images have high similarity and do not like images in ImageNet with obvious differences on the edge or shape. For example, the chest X-ray have blurry edge and similar region between normal and abnormal scans. Occasionally, a dataset may be from different X-ray machines or include a small number of images with low quality caused by low contrast, lack of costophrenic angle and biased annotations [24] . Thus, the limited receptive fields and insufficient extraction of contextual information from FCN may result in poor performance in some practical datasets. Many more complicated networks have been proposed such as PSPNet [27] , U-Net [20] and DeepLab [4] . They have more sufficient receptive fields and larger capacity of capturing richer contextual information to achieve better performance. U-Net is the most classic network in medical image segmentation [21] which is applied widely due to its ability to concatenate contextual information by skipping connections between encoder and decoder. In order to improve efficiency and accuracy of the network, a number of extensions on U-Net have been proposed. Deep Residual U-Net [26] , which employs residual block into each layer of encoder and decoder, made the network deeper and improved the performance metrics. Recurrent Convolutional layers (RCNN) and Recurrent Residual Convolutional layers (R2CNN) were proposed by Alom et al. [1] , which utilize feature accumulation with the recurrent mechanism. BCDU-Net used bi-directional ConvLSTM instead of skip connection and block of dense convolutions was applied into the bottom encoding layer [2] . Attention mechanism was introduced in skip connections of U-Net [18] . In addition to modified network structures, the kernel size is discussed and explored recently [19] . These models focus on modifying context feature extraction in one path and connection between encoder and decoder. They perform the-state-of-art in some tasks of medical image segmentation. However, they do not focus on the extraction of spatial and contextual information simultaneously, which may cause mis-classification on the pixel of nearby-border. Moreover, the different source datasets may cause uncertainty of segmentation and this influence had been explored on the OCT scans [7] . The single 3 × 3 convolution kernel, which previous most networks have used, cannot adjust variety of object and domain-shift in cross-manufacturer. Therefore, the model, that can adjust the different device source, deserve to be studied.

In this paper, we proposed a novel extension of U-Net called DEFU-Net to address these problems. We build a dual path encoder for improving the performance of the model. The dual path encoder comprises a densely connected recurrent encoding blocks as well as an inception encoding blocks with dilation. Some researchers used an inception block to replace the convolution block on each layer in some segmentation task [3] . We employ inception blocks with dilation as the second path to scale up the width of the network. The input features of the second densely connected recurrent block and the first inception block with dilation are shared. The inception block with dilation can adjust both global and local distributions and extract multiple spatial features [22] . It avoids spatial information loss resulted by max-pooling as well. The densely connected recurrent convolution block (DCRC) can facilitate network to extract high-level information and avoid gradient vanishing problem as well [1] . Before concatenation, the extracted information from current DCRC block and inception block with dilation will be fused by summation at each layer. Meanwhile, the fused information will be transmitted to the next inception block with dilation to extract spatial features. Thus, this process helps the network to obtain more accurate result than other U-Net variations on the cross-manufacturers dataset from Montgomery and Shenzhen hospital since our model is able to reduce the influences of the variance among datasets.

Inspired by the advantages of U-Net [21] , inception block [22] , DenseNet [9] and recurrent structure [1] , the dual encoder fusion U-Net is proposed which is shown in Fig. 1 . This new framework follows the classical encoding and decoding structure of U-Net. The blue boxes represent the set of feature maps in each layer and the grey boxed are the set of feature map from inception extracting path. On each layer of the encoder, we apply two recurrent blocks with densely connections and max-pooling (grey arrow). After feature maps extracted by first DCRC blocks (blue arrow), they will be copied to inception extractor with dilation (black arrow). The information extracted by DCRC block and inception block will be fused by pixel-wisely summation (adding (+)) in the rest part of the encoder. Each set of fused features will be taken as an input to the next inception block with dilation. These operations can enrich spatial and context features. The fused feature maps are prepared for concatenating to decoder layers accordingly. The concatenation encourages the information reused in decoder. In order to avoid increasing parameters, we utilize up-sampling in decoding part. The green arrow represents DCRC block + Upsampling. The upsampling is beneficial to recover boundaries location from lowdimensional features. Moreover, we modified the number of filters in the bottom layer from 1024 to 512, as same as 4 th layer. This modification can reduce computational budget and avoid yielding useless feature maps.

The densely connected recurrent convolution block (DCRC block) in our network is inspired by R2 block proposed by Altom et al. [1] and DenseNet [9] . The unfolded structure is shown in Fig. 3 . The recurrent unit can help feature accumulation and extract useful information precisely. The multiple kernels will extract information from accumulated feature maps. The structure has been illustrated in Fig.3 . The batch normalization is employed into the blocks for increasing the stability of training [10] . Meanwhile, the block includes the densely connected mechanism. After each recurrent unit, the number of channels will be increased with densely connection mechanism because the output and all of previous features are concatenated. With limited computational resource, we choose Conv(1 × 1)-LeakyReLU for recover the number of channels, which is similar to the bottleneck layers used in DenseNet [9] to reduce the number of channels. The multi-connections can enforce individual layer obtain deep supervision additionally from loss function [16] . The network can become deeper and the convergence is faster in training process.

In our network, we take the classical Inception V2 and V3 [23] as a reference. The height of lungs in the X-ray scans is generally larger than width. Inspired by the success of atrous/dilated convolution [5] in medical image segmentation [28] , Conv(3 × 3) with dilation rate (3, 1) and (1, 2) are introduced in this block for expand receptive fields on height and width respectively. The detail of structure is shown in Fig. 4 . This structure use three ways to decrease dimension, including 1×1 convolution, 3×3 convolution with stride 2 and 3×3 average-pooling. In order to keep convolutional continuity and achieve down-sampling, the 3 × 3 kernel with stride 2 can help retain spatial characteristics and avoid information loss directly caused by max-pooling. The 1 × 1 kernel with stride 2 can enhance the non-linear capacity [23] . Especially, two branches include dilated convolutions. Based on the equations 1) + KernelSize, the new kernel size will be 7 × 3 and 3 × 5 respectively. Thus, dilated receptive fields on height is larger than dilated receptive field on width for adjusting difference of learning on shape. The high-level semantic features on the height or width will be learned. By combining with other kernels and average-pooling, this modified inception can aggregate multi-scale contextual information for dense prediction architectures increasing the performance [25] . From the second layer to the bottom of encoder, the feature extracted by the last inception block and the feature extracted by DCRC block will be fused by summation and they will be concatenated to the decoder. The summation on the element-wise feature has been proved to have a great performance outside U-Net [12] . The rich spatial and context features are integrated to the decoding 

We mainly evaluated the DEFU-Net on chest X-ray from Montgomery and Shenzhen hospital which opened in 2014 [11] . This dataset contains many diagnoses such as infiltrate, fibrosis, pneumonia and tuberculosis and so forth. These diseases have similar radiogram and we mainly focus on segmenting lung tissue. Therefore, we use this dataset to investigate a state-of-the-art pre-trained model. This dataset includes chest X-ray scanned from two manufacturers. 138 patient's images and 566 patient's images are applied in this segmentation task respectively. Totally, the amount of normal lung have 359 and while the number of abnormal lung is 345 images. The size of X-ray from Montgomery Country is either with 4020 × 4892 or 4892 × 4020 pixels. The size of Shenzhen chest X-ray is 3K × 3K. The pixel-wise lung mask annotations are offered in the two datasets. Specially, X-ray scans from Montgomery are annotated in left and right lung respectively. Thus, we combined left and right lung segmentation masks from Montgomery and resized all the Xray scans from two dataset to 512 × 512 pixels. All scans were transformed to single channel as grey-scale. All masks were dilated to gain more information on the edge of lungs for training. After data pre-processing, the dataset had been split that 528 images for training set and 76 images for validation set while 100 for testing set. In the training process, the training data were augmented by rotation, shifting, shearing, zooming and flipping in order to avoid overfitting [14] .

The training environment was based on Keras 2.2.4 with Tensorflow 1.13 backend. GPU was 1080Ti. The batch size was set to 2. All the variant U-Nets were 4 times downsampling and up-sampling. Residual U-Net (Res-U-net) had made some differences with structure proposed by Zhang et al. [26] . The input and output were 512 × 512 × 1. We used classic dice loss Eq. 1 for semantic segmentation and Adam optimizer [13] with default parameters.

Where g i ∈ 0, 1 represents the ground truth on the pixel level. p i ∈ [0, 1] refers to the probability value of the prediction on each pixel. N is the total number of the pixel. Adding 1 is for preventing divided by zero.

The reduce learning rate on plateau" with initial learning rate 1 × 10 −5 , factor = 0.2 and patience = 5) and Early Stopping (patience = 5) were applied in training. The total training epochs was set to 175. Meanwhile, In order to understand the capacity of inception extractor with dilation, we combine it with Res-U-Net. Moreover, the recurrent blocks of R2U-Net and Attention R2U-Net have been modified. The structures of recurrent block are same with Fig.3 because more convolutional operations can facilitate convergence and accuracy. The U-Net, Incept-Res-U-Net, BCDU-Net, R2U-Net and R2-Att-U-Net, DEFU-Net had all been trained on our dataset as well.

We used 7 evaluation metrics for our training and testing predictions, including binary accuracy (AC), dice coefficient (Dice Coef = -dice loss), intersection over union (IOU), precision, recall, F1-score and area under curve (AUC). The Dice Coef and IOU can be expressed in the following equations, where GT represent the ground truth and PR refers to the prediction result.

In order to calculate AC, precision, recall, F1-score, we need to employ True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN). The metrics can be obtain by following equations. The AUC can be calculated by receiver operating characteristic curve.

P recision = T P T P + F P (5) 

IV. RESULTS Evaluation metrics of training, validation and testing are shown in the Table 1, Table 2 and Table 3 for comparison purposes. DEFU-Net generates the highest Dice, AC, IOU, precision, recall, F1-score and AUC in training, validation dataset. In testing dataset, the DEFU-Net still outperforms most of the other metrics. Meanwhile, under the "Early Stopping' mechanism, U-Net, Res-U-Net, BCDU-Net and Incep-Res-U-Net stop training at 70 th epoch, 50 th epoch, 72 th and 60 t h respectively. The R2U-Net and Att-R2U-Net stop at 100 th . Our model can train 145 epochs with higher metrics. The AC of our model can reach 0.9776 after two epochs. They demonstrate that our model has fast convergence and fits our data better in training. Moreover, we can see that the inception path with dilation can boost performance beyond U-Net, Res-U-Net and BCDU-Net. It is shown that this path are actually effective for our cross-manufacturer segmentation task. After combining the inception blocks with dilation and densely connection recurrent blocks as dual path encoder, the dataset can be fit with the best performance. We visualise the difference between prediction and ground truth of Montgomery and Shenzhen dataset in Fig. 5 and Fig. 6 . From left to right, the predictions fit the ground truth gradually. Because of the X-ray scanns from two manufacturers, there are differences in the distributions of lung size, shape and position. In order to adapt to this diversity, the model needs to learn these differences and be generally adaptable. From the comparison diagrams, they show the advantage of inception with dilation clearly. This extractor path improves classification on the pixel of nearby-border and helps reduce the rate of True Negative and the False Positive. The DEFU-Net achieve superior performance. The combination of DCRC block and inception block with dilation accelerate the convergence and improve the accuracy of the prediction on the small areas.

Against the backdrop of COVID-19, the state-of-the-art pretrained model about segmentation is significant for the future COVID-19 chest X-ray to diagnose pneumonia. In this paper, we proposed an innovative network structure called DEFU-Net to segment the opened cross-manufacturer chest X-ray dataset with great performance. We applied a dual path framework to enrich the extracted features in the encoder. The inception path with dilation can help the model to capture spatial information with different size and increase the width of the network. The densely connected recurrent block increases the depth of the network. The low level to high level information can be captured. The pixel-wise summations of features from two paths preserve more optimal information during decoding. Meanwhile, we change the number of feature maps in bottom layer to 512 while the traditional U-Net and R2U-Net use 1024 filters. The computational space is half that of R2U-Net and Att-R2U-Net. On this combination of two public datasets, the DEFU-Net have better fitting ability on the segmentation of edges and small areas. We observe our model performing state-of-the-art compared to the aforementioned model. In the future, our model may be experimented for the segmentation tasks under cross-manufacturer COVID-19 X-rays and we may explore feasibility of extending to 3D images for more complicated segmentation.

Recurrent residual convolutional neural network based on u-net (r2u-net) for medical image segmentation

Bi-directional convlstm u-net with densley connected convolutions

Inception modules enhance brain tumor segmentation

Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs

Encoderdecoder with atrous separable convolution for semantic image segmentation

Imagenet: A large-scale hierarchical image database

Clinically applicable deep learning for diagnosis and referral in retinal disease

Deep residual learning for image recognition

Densely connected convolutional networks

Batch normalization: Accelerating deep network training by reducing internal covariate shift

Two public chest x-ray datasets for computer-aided screening of pulmonary diseases

Cnn-based segmentation of medical imaging data

Adam: A method for stochastic optimization

Imagenet classification with deep convolutional neural networks

Deep learning

Deeply-supervised nets

Fully convolutional networks for semantic segmentation

Large kernel mattersimprove semantic segmentation by global convolutional network

U-net: Convolutional networks for biomedical image segmentation

Lung ct image segmentation using deep neural networks

Going deeper with convolutions

Rethinking the inception architecture for computer vision

Xlsor: A robust and accurate lung segmentor on chest x-rays using criss-cross attention and customized radiorealistic abnormalities generation

Multi-scale context aggregation by dilated convolutions

Road extraction by deep residual u-net

Pyramid scene parsing network

Acnn: a full resolution dcnn for medical image segmentation