key: cord-0456345-p5xa609n
authors: Vidal, Pl'acido L; Moura, Joaquim de; Novo, Jorge; Ortega, Marcos
title: Multi-stage transfer learning for lung segmentation using portable X-ray devices for patients with COVID-19
date: 2020-10-30
journal: nan
DOI: nan
sha: 267520e218e329334150500932630b13ec455208
doc_id: 456345
cord_uid: p5xa609n

In 2020, the SARS-CoV-2 virus causes a global pandemic of the new human coronavirus disease COVID-19. This pathogen primarily infects the respiratory system of the afflicted, usually resulting in pneumonia and in a severe case of acute respiratory distress syndrome. These disease developments result in the formation of different pathological structures in the lungs, similar to those observed in other viral pneumonias that can be detected by the use of chest X-rays. For this reason, the detection and analysis of the pulmonary regions, the main focus of affection of COVID-19, becomes a crucial part of both clinical and automatic diagnosis processes. Due to the overload of the health services, portable X-ray devices are widely used, representing an alternative to fixed devices to reduce the risk of cross-contamination. However, these devices entail different complications as the image quality that, together with the subjectivity of the clinician, make the diagnostic process more difficult. In this work, we developed a novel fully automatic methodology specially designed for the identification of these lung regions in X-ray images of low quality as those from portable devices. To do so, we took advantage of a large dataset from magnetic resonance imaging of a similar pathology and performed two stages of transfer learning to obtain a robust methodology with a low number of images from portable X-ray devices. This way, our methodology obtained a satisfactory accuracy of $0.9761 pm 0.0100$ for patients with COVID-19, $0.9801 pm 0.0104$ for normal patients and $0.9769 pm 0.0111$ for patients with pulmonary diseases with similar characteristics as COVID-19 (such as pneumonia) but not genuine COVID-19.

Due to the overload of the health services, portable X-ray devices are widely used, representing an alternative to fixed devices to reduce the risk of crosscontamination. However, these devices entail different complications as the image quality that, together with the subjectivity of the clinician, make the diagnostic process more difficult.

In this work, we developed a novel fully automatic methodology specially designed for the identification of these lung regions in X-ray images of low

The World Health Organization (WHO) declared a global health emergency on January 30th, 2020, due to the spread of SARS-CoV-2 and its disease COVID- 19 beyond the People's Republic of China. Thus, the pandemic surpassed the million of deaths as well as tens of millions of people infected worldwide [1] .

One of the first and most prominent symptoms is the development of viral pneumonia, highlighting fever, cough, nasal congestion, fatigue, and other respiratory tract related affections [2] . In more serious cases, the patients may present acute respiratory distress syndrome or even systemic symptomatic manifestations [3, 4, 5] .

Because of this characteristics of primarily affecting the respiratory tract, real-time polymerase chain reaction (RT-PCR) detection in mucosal and sputum samples of known virus DNA strands became the standard method of diagnosis at the beginning of the pandemic [6, 7] . However, this test has demonstrated improvable sensitivity and specificity. Furthermore, these tests do not account for the emergence of new variations in the DNA of the virus or are not updated quickly enough to successfully control the pandemic [7] .

Due to the saturation of the health services worldwide and the necessity of a quick diagnosis and analysis of severe cases, among others, the use of computerized tomography (CT) scans and chest X-ray image analysis has been prominent, even complementary to the PCR tests. These devices are especially useful after initial triage and referral to emergency services to further assess known lung regions of virus infection (despite the expertise needed to correctly study them [8] ).

For these reasons, at the dawn of the pandemic, proven computational methodologies of medical image analysis have been tested, as well as developing new ones with the aim of facilitating, accelerating and reducing the subjectivity factor of diagnostics at a critical moment for humanity [9, 10] . Most of these methodologies are based on deep learning strategies, except for some particular proposals that use classic machine learning approaches [11] or others that actually use these techniques as support for deep learning methods [12, 13] .

Regarding methodologies that aimed to help with the diagnostic of COVID-19 based on deep learning, one of the first trends is to use these strategies to perform a medical screening. These methodologies return a label or severity of a COVID-19 candidate patient [14, 15, 16, 17] .

Other trend with deep learning automatic approaches is to aid in the segmentation of the pulmonary region of interest. This region, as mentioned, is hard to correctly assess due to the difficulties of analyzing a radiography [8] but critical, as the COVID-19 clinical picture mainly manifests its effects in the lung parenchyma (even after the patient has been discharged [18] ). These works are usually integrated as input of other methodologies to improve their results by reducing the search space to only the region of interest or as a mean to greatly improve a posterior visualization of these results [19] .

The third trend consists in, instead of trying to segment these lung regions, as they tend to be obfuscated by other tissues in the chest region, try to directly obtain the pathological structures of COVID-19 [20, 21] .

And, finally, works that try to palliate or complement their approaches by merging some (or all) of the mentioned trends into a single methodology [22, 23, 24] .

Our work aims at following the second paradigm, extracting the lung regions, but specifically for images that are captured by portable X-ray devices, with lower details and, therefore, higher complexity. This is specially difficult and, to the best of our knowledge, there are no other systems specially designed to work with chest radiographs obtained from these portable machines. This is specially relevant as these devices are recommended by the American College of Radiology (ACR), as in cases as the COVID-19, they help to minimize the risk of cross-infection and allow for a comfortable and flexible imaging of the patients [25] . These systems are ideal for emergency and saturation of the healthcare services situations, as they do not require strict structuring of the established circuit and protocol [26] .

However, performing a diagnostic with these portable devices is particularly challenging, as the generated images are of lesser quality due to the capture conditions, more difficult to diagnose (as they usually only allow for a supine image instead of the usual multiple perspectives) and, due to the fact that they are obtained in emergencies, less available to researchers. Nonetheless, the use of these radiological studies is critical, since they not only serve for the monitoring and assessment of patients already referred by triage, but they also permit to compensate and mitigate the false negative rates of PCR studies in that same previous triage step. Additionally, their use facilitates the work in crowded wards [27] . As an example, Figure 1 shows three random images from clinical practice with these portable devices for three different cases: patients with diagnosed COVID-19, patients with pathologies unrelated to COVID-19 but with similar impact in the lungs, and normal lungs. These images show how the images that are taken with these portable devices tend to blur the tissues of the lung region, as well as the pathological artifacts (specially in the images from afflicted lungs).

For this reason, in this work we designed a segmentation methodology especially for images of low quality from portable devices. To the best of our knowledge, there is no other methodology specifically designed to analyze a set of images including COVID-19, also being taken in these particular challenging capture conditions. To solve this issue, we developed a training methodology based on two stages of transfer learning between designed subsequent domains.

Pathological Normal Firstly, we took advantage of the knowledge learnt by a segmentation network from another medical imaging domain trained with a larger number of images and adapted it to be able to segment general lung chest images of high quality, including COVID-19 patients. Then, using a limited dataset composed by images from portable devices, we adapted the trained model from general lung chest X-ray segmentations to work specifically with images from these portable devices.

The proposal would allow to delimit the pulmonary region of interest, critical for the location of the pathological structures caused by COVID-19, independently from the subjectivity of the clinician (a subject particularly sensitive in situations of high stress and psychological wear) and under adverse capture conditions. Moreover, this system can be used as input to another methodology to reduce the search space to the lung region of interest or facilitate the subsequent visualization of the results.

In summary, the main contributions of this article are:

• Fully automatic methodology to segment the pulmonary region in X-ray images.

• Tested with images from COVID-19, pulmonary pathologies with similar characteristics to COVID-19 and normal lungs.

• Datasets obtained from real clinical practice with portable devices (recommended when risk o cross-contamination and crowded hospital wings).

• Multiple stages of transfer learning between designed subsequent image domains to work with a limited number of portable X-ray samples.

• To the best of our knowledge, our proposal represents the only fully automatic study specifically designed to work with portable capture devices.

• Robust and accurate even with poor quality images from these portable devices.

This document is divided into three parts. Section 2 Materials and Methods presents the resources that are used in this work, as well as the followed methodology in detail. Section 3 Results presents the realization of all the steps presented in the previous section. Finally, in Section 4 Discussion, we analyze the relevance of each obtained result as well as the final conclusions (Subsection 4.1) drawn from the research.

This first dataset was formed from public available datasets [28, 29] . The dataset contains images with varying resolutions, ranging from 5600 × 4700 pixels to 156 × 156 pixels including chest, lateral X-rays and CT images. For our purpose we discarded the latter two types. This was done because the portable devices of the consulted healthcare services were used only for chest 

The second dataset was provided by the Radiology Service of the Complexo Hospitalario Universitario de A Coruña (CHUAC) obtained from three portable X-ray devices in a real diagnostic scenario: an Agfa dr100E GE, an Optima Rx200 and a Siemens Ysio Max. All the images were obtained after triage in medical wings specially dedicated for the treatment and monitoring of patients and 200 patients with, in principle, no pulmonary afflictions but that may be affected by other diseases, for a total of 600 images. We want to remark the complexity of the presented pathological scenario (in addition to the complexity of the portable X-ray images).

Regarding the software resources, we have used Python 3.7.9 with Pytorch 1.6.0 [31] and Scikit-Learn 0.23.2 [32] . Additionally, we used a pre-trained model from the work of Buda et al. [33, 34] , trained with images from 110 patients for a total of 7858 images [35] . This network is an implementation of an U-Net [36] dedicated to the identification of brain structures in magnetic resonance imaging or MRI [33] . Specifically, the original network has been trained to detect gliomas, a type of brain tumor diagnosed mainly by this imaging modality [37, 33] , problematic that share similar characteristics to our case, which is herein exploited. Thanks to the use of this model trained with a large number of images from a similar domain, we are able to compensate for the limited number of chest radiographs obtained from portable devices available to the method.

To successfully develop a system able to work with radiographs from portable devices with a limited amount available from the saturated health services, we followed a workflow that allowed us to progressively adapt information from a different medical imaging domain and pathology to ours. The main workflow followed in this work can be seen in the Figure 2 .

As can be seen in this figure, the proposed fully automatic methodology was divided into two main stages of transfer learning. A first transfer learning stage to adapt the filters developed in the network for the MRI domain to chest radiography and a second one to further refine these weights specifically into the sensibly harder radiographs from portable devices. In the following sections each stage will be explained in more detail.

For this first step, we started from a model previously trained in a medical imaging domain with a large and heterogeneous set of images that presents similar characteristics to those we would find in the target domain (from which we have available a limited number of samples). In our case, we used the U-Net trained with MRI images for glioma segmentation. As can be seen in Figure 3 , these tumors look very similar to normal brain tissues. The same way, the pulmonary regions of interest can be found superimposed with the rest of the tissues that the X-rays have had to pass through (due to the way the capture method itself works) or even mixed with pathological structures resulting from different pulmonary lesions caused by the considered pathologies.

For this reason, the knowledge transfer was direct. This was not only because of the similarity of characteristics of both image domains, but also because of the similar complications present in both image domains and pathologies. Also, it has successfully proven its worth and robustness on its medical imaging domain.

These factors made it an ideal candidate network to be the "knowledge donor" for our purpose. In this stage, we also divided the dataset of 600 chest radiographs from portable X-ray devices obtained during clinical practice in the CHUAC into two datasets of 300 samples. This was done to use only one of the dataset halves to perform the knowledge transfer, and the other to evaluate the performance and improvement of the system before and after this stage.

In order to maintain consistency and allow for proper transfer learning, we have employed the same loss function used in the model trained with the brain MRI images for the subsequent transfer learning stages. Therefore, both models have been trained using the Smooth Dice Loss (Equation 1).

Where Op represents the predicted system output and Ot the expected output (target). λ is the smoothing factor, which has been defined as 1 in this work.

As optimizer, we have used adaptive moment estimation (ADAM) [38] , with a learning rate of 0.001 that is adjusted dynamically according to the necessities and training progression of the model. Finally, for the calculation of the number of training epochs we have used an early stopping strategy. That is, the algorithm will automatically stop when the model is not able to improve its performance. Specifically, the system evaluated the validation loss and had a patience of 20 epochs without obtaining any improvement. As for the distribution of the dataset, 60% of the samples have been used for the training of the model, 20%

for the validation, and the remaining 20% for the unbiased testing of the model. Specifically, the images were randomly rotated random degrees between -10º and +10º with a probability of 75% to simulate feasible postural variations of the patient.

To evaluate the performance of our proposal in each stage, we analyzed the results in a wide range of metrics that allowed us to study the performance of 

P REC 1 = T P T P + F P (5)

SEN S = T P T P + F N (8)

Finally, AUC returns the probability that the analyzed model has of assigning a higher value to a positive sample over a negative sample [39] .

In this section we will proceed to present the results that were obtained during the development of this work, product of the previously presented methodology.

Now, we will proceed to present the results for the inter domain learning stage, where we took advantage of a model that was trained with a large number of images from a similar image domain (that allowed us to generate a robust methodology despite the scarcity of images available from portable devices). On this first stage, we adapted from this domain to common lung radiographs. The average of all the repetitions for this training process can be seen in Figure 4 , and the mean test results of each of the chosen models in Table 1 . In this Figure   4 , we see that (on average) it does not need too many cycles to learn the patterns of the new domain, thanks to the already mentioned transfer of knowledge from similar modalities instead of starting the training from scratch. As can be seen, thanks to the knowledge transfer stage we obtain a system capable of successfully segmenting the pulmonary region of interest. The only weak measurement is the one referring to the sensitivity of the model, with a considerably high standard deviation as well. However, the specificity obtained is considerably high, and with a very low deviation (which indicates consistency throughout the repetitions). These two factors indicate that the model is overadjusting some of the detections. This is possibly due to the images showing foreign bodies such as pacemakers or other such objects, as the dataset masks (targets) have been corrected to try to estimate the complete lung surface even if it is obscured by these objects.

After this first inter domain transfer learning, we now present the results of the inter device type transfer learning step. In this step, we used the model adapted for a general chest X-ray and continued the training to adapt this model to the final objective of this work: obtaining a robust system able to successfully segment lung regions in images taken in adverse conditions with portable devices.

In Figure 5 we Finally, as can be seen in the test results of the chosen model in Table 2 , the system appears to return more balanced results across all the metrics. We can see how the sensitivity of the system has sensibly improved and the system is now more balanced. Now, we will proceed to evaluate both systems under an unbiased dataset to better assess their differences and improvements.

For this final test we used the 300 independent images from the portable CHUAC dataset that we separated for further analysis. The results of these tests can be seen detailed in Tables 3 and 4 ; where we present the results for the test of the model before and after the second stage of transfer learning (where the model is adapted to portable X-ray devices), respectively.

Complementarily, this improvement is better observed in the comparison plots of Figures 6, 7 and 8 . These graphs show that where the most noticeable change has been in images that have some kind of pathology with effects similar to COVID-19, improving by almost 0.02 points in Jaccard and DICE coefficients.

On the other hand, we also noticed a remarkable increase in the sensitivity of the models, being this measurement critical in systems oriented to the medical sciences and clinical practice and also highly increased after the inter device type transfer learning stage into the portable X-ray image domain. 

In Figure 9 we can see more closely examples of outputs generated by the proposed method. There is a clear distinction between the behavior of the two models. As we saw in the statistics, the model that was trained with a large number of MRI images and then its knowledge exploited to improve the training with the common lung radiographs tends to generate more adjusted and limited segmentations. This is particularly noticeable in those images that present less 

In this work, we have proposed a system with the purpose of segmenting lung regions in thoracic radiographs, especially aimed at those obtained by portable X-ray devices in adverse conditions and with a limited supply of images. These devices, which represent an alternative to fixed devices to reduce the risk of crosscontamination in the diagnosis of diseases such as COVID-19, are critical for of transfer learning allows not only to better estimate the pulmonary region, but also to eliminate various artifacts resulting from the lower sample quality present in the images from portable devices.

Thus, as a final result of this work, we have successfully obtained a fully automatic methodology based on deep methodologies, using a limited number of images from portable devices and capable of working with these images in a robust and consistent way, regardless of the image quality and capture conditions.

In this way, as future work, we are considering its application as an intermediate 

Coronavirus 2019-ncov global cases by johns hopkins csse

The COVID-19 epidemic

Organ-specific manifestations of COVID-19 infection

Venous and arterial thromboembolic complications in covid-19 patients admitted to an academic hospital in milan, italy

COVID-19 and multiorgan response

Detection of 2019 novel coronavirus (2019-nCoV) by real-time RT-PCR

Positive RT-PCR test results in patients recovered from COVID-19

Chest X-ray in clinical practice

Review of artificial intelligence techniques in imaging data acquisition, segmentation and diagnosis for COVID-19

Automated detection and forecasting of covid-19 using deep learning techniques: A review

Automatic x-ray COVID-19 lung image classification system based on multi-level thresholding and support vector machine

Artificial intelligence-enabled rapid diagnosis of patients with COVID-19

Detection of coronavirus disease (COVID-19) based on deep features

A combined deep cnn-lstm network for the detection of novel coronavirus (covid-19) using x-ray images

Fully automatic deep convolutional approaches for the analysis of covid-19 using chest x-ray images

Automated detection of COVID-19 cases using deep neural networks with x-ray images

Deep convolutional approaches for the analysis of covid-19 using chest x-ray images from portable devices

Abnormal pulmonary function in covid-19 patients at time of hospital discharge

Covid-19 chest ct image segmentation-a deep convolutional neural network solution

Inf-net: Automatic COVID-19 lung infection segmentation from CT images

Residual attention u-net for automated multiclass segmentation of covid-19 chest ct images

Covid mtnet: Covid-19 detection with multi-task deep learning approaches (2020)

Viral pneumonia screening on chest x-ray images using confidence-aware anomaly detection

Lung infection quantification of covid-19 in ct images with deep learning

American college of radiology, acr recommendations for the use of chest radiography and computed tomography (ct) for suspected covid-19 infection

Statements/Recommendations-for-Chest-Radiography-and-CT-for-Suspected-COVID19-Infection

Portable chest x-ray in coronavirus disease-19 (COVID-19): A pictorial review

Frequency and distribution of chest radiographic findings in patients positive for COVID-19

Covid-19 image data collection: Prospective predictions are the future

Labeled optical coherence tomography (oct) and chest x-ray images for classification

Pytorch: An imperative style, high-performance deep learning library

Scikit-learn: Machine learning in Python

Association of genomic subtypes of lower-grade gliomas with shape features automatically extracted by a deep learning algorithm

Brain MRI images together with manual FLAIR abnormality segmentation masks obtained from The Cancer Imaging Archive (TCIA)

Convolutional networks for biomedical image segmentation

Low-grade gliomas

Adam: A method for stochastic optimization

The use of the area under the ROC curve in the evaluation of machine learning algorithms