key: cord-0722182-i0dtb0ar authors: Vidal, Plácido L.; Moura, Joaquim; Novo, Jorge; Ortega, Marcos title: Multi-stage transfer learning for lung segmentation using portable X-ray devices for patients with COVID-19 date: 2021-02-12 journal: Expert Syst Appl DOI: 10.1016/j.eswa.2021.114677 sha: bb8feffbfd9a5ff396629a80dc938e21b9252d28 doc_id: 722182 cord_uid: i0dtb0ar One of the main challenges in times of sanitary emergency is to quickly develop computer aided diagnosis systems with a limited number of available samples due to the novelty, complexity of the case and the urgency of its implementation. This is the case during the current pandemic of COVID-19. This pathogen primarily infects the respiratory system of the afflicted, resulting in pneumonia and in a severe case of acute respiratory distress syndrome. This results in the formation of different pathological structures in the lungs that can be detected by the use of chest X-rays. Due to the overload of the health services, portable X-ray devices are recommended during the pandemic, preventing the spread of the disease. However, these devices entail different complications (such as capture quality) that, together with the subjectivity of the clinician, make the diagnostic process more difficult and suggest the necessity for computer-aided diagnosis methodologies despite the scarcity of samples available to do so. To solve this problem, we propose a methodology that allows to adapt the knowledge from a well-known domain with a high number of samples to a new domain with a significantly reduced number and greater complexity. We took advantage of a pre-trained segmentation model from brain magnetic resonance imaging of a unrelated pathology and performed two stages of knowledge transfer to obtain a robust system able to segment lung regions from portable X-ray devices despite the scarcity of samples and lesser quality. This way, our methodology obtained a satisfactory accuracy of 0.9761 ± 0.0100 for patients with COVID-19, 0.9801 ± 0.0104 for normal patients and 0.9769 ± 0.0111 for patients with pulmonary diseases with similar characteristics as COVID-19 (such as pneumonia) but not genuine COVID-19. The World Health Organization (WHO) declared a global health emergency on January 30th, 2020, due to the spread of SARS-CoV-2 and its disease COVID-19 beyond the People's Republic of China. Thus, the pandemic surpassed the million of deaths as well as tens of millions of people infected worldwide 5 (Coronavirus Resource Center, Johns Hopkins, 2020). For these reasons, at the dawn of the pandemic, proven computational methodologies of medical image analysis have been tested, as well as developing new ones with the aim of facilitating, accelerating and reducing the subjectivity factor of diagnostics at a critical moment for humanity Shoeibi 10 et al., 2020) . Most of these methodologies are based on deep learning strategies, except for some particular proposals that use classic machine learning approaches (Hassanien et al., 2020) or others that actually use these techniques as support for deep learning methods (Mei et al., 2020; Sethy & Behera, 2020) . Regarding methodologies that aimed to help with the diagnostic of COVID-19 15 based on deep learning and convolutional neural networks (CNN), one of the first trends is to use these strategies to perform a medical screening. These methodologies return a label or severity of a COVID-19 candidate patient (Islam et al., 2020; Ozturk et al., 2020; de Moura et al., 2020a,b; . Other trend with deep learning automatic approaches is to aid in the segmen-20 tation of the pulmonary region of interest. This region, as mentioned, is hard to correctly assess due to the difficulties of analyzing a radiography (Joarder & Crundwell, 2009 ) but critical, as the COVID-19 clinical picture mainly manifests its effects in the lung parenchyma (even after the patient has been discharged (Mo et al., 2020) ). These works are usually integrated as input of other method-25 ologies to improve their results by reducing the search space to only the region of interest or as a mean to greatly improve a posterior visualization of these results (Yan et al., 2020) . The third trend consists in, instead of trying to segment these lung regions, as they tend to be obfuscated by other tissues in the chest region, try to directly 30 obtain the pathological structures of COVID-19 (Fan et al., 2020) . And, finally, works that try to palliate or complement their approaches by merging some (or all) of the mentioned trends into a single methodology (Alom et al., 2020; Chen et al., 2020) . Our work aims at following the second paradigm, extracting the lung regions, 35 but specifically for images that are captured by portable X-ray devices. These devices present lower capture quality and, therefore, higher complexity. To the best of our knowledge, there are no other systems specially designed to work with chest radiographs obtained from these portable machines. This is specially relevant as these devices are recommended by the American College of Radiology 40 (ACR) during emergency situations because they help to minimize the risk of cross-infection and allow for a comfortable and flexible imaging of the patients (American College of Radiology, 2020). In addition, these systems are ideal for emergency and saturation prevention of the healthcare services, as they do not require strict structuring of the established circuit and protocol (Jacobi et al., 45 2020; Wong et al., 2020) . A comparative summary of all the aforementioned proposals against ours can be seen in Table 1 . As an example, Figure 1 shows three representative images from clinical practice with these portable devices for three different cases: patients with diagnosed COVID-19, patients with pathologies unrelated to COVID-19 but 50 with similar impact in the lungs, and normal lungs. These images show how the images that are taken with these portable devices tend to blur the tissues of the lung region, as well as the pathological artifacts (specially in the images from afflicted lungs). One of the first and most prominent symptoms of COVID-19 is the develop-55 ment of viral pneumonia, highlighting fever, cough, nasal congestion, fatigue, and other respiratory tract related affections (Velavan & Meyer, 2020) . These symptoms manifest themselves in the lungs as ground glass abnormalities, patchy consolidations, alveolar exudates and interlobular involvement (Garg et al., 2019; Brunese et al., 2020) . On the one hand, the ground glass abnormalities in chest radiographs are reach an homogeneous texture if the disease is quite advanced. These structures appear when regions of the lungs are filled with foreign fluids instead of normal air that alter the density. An example of these two cases can be seen in Figure 2 . In more serious cases, the patients may present acute respiratory distress syndrome or even systemic symptomatic manifestations (Gavriatopoulou et al., 70 2020; Lodigiani et al., 2020; Zaim et al., 2020) . Performing a diagnostic with these portable devices is particularly challenging, as the generated images are of lesser quality due to the capture conditions, more difficult to inspect visually (as they usually only allow for a supine image instead of the usual multiple perspectives) and, due to the fact that they are obtained 75 in emergencies, less available to researchers. For this reason, in this work we designed a segmentation methodology especially for images of low quality from portable devices and that is able to work with a limited number of samples. To the best of our knowledge, there is no other methodology specifically designed to analyze a set of images including COVID-19, also being taken in these particular 80 challenging capture conditions and scarcity of samples. To solve this issue, we developed a training methodology based on two stages of transfer learning between designed subsequent domains. Firstly, we took advantage of the knowledge learnt by a segmentation network from another medical imaging domain trained with a larger number of images and adapted 85 it to be able to segment general lung chest images of high quality, including COVID-19 patients. Then, using a limited dataset composed by images from portable devices, we adapted the trained model from general lung chest X-ray segmentations to work specifically with images from these portable devices. The proposal would allow to delimit the pulmonary region of interest, critical In summary, the main contributions of this article are: • Fully automatic proposal to segment the pulmonary region in low quality chest radiographs. • Multiple stages of transfer learning between designed subsequent image 100 domains to work with a limited number of portable X-ray samples. • Datasets obtained from real clinical practice with portable devices (recommended when risk o cross-contamination and crowded hospital wings). • To the best of our knowledge, our proposal represents the only fully automatic study specifically designed to work with portable capture devices. • Robust and accurate even with poor quality images from these portable devices. • Tested with images from COVID-19, pulmonary pathologies with similar characteristics to COVID-19 and normal lungs. The present document is divided into six main sections. Section 2: "Mate-110 rials", presents all the resources needed to fully reproduce our work in detail. Section 3: "Methodology" includes a complete explanation of the algorithm and strategy followed in this work and the particular parameters for each experiment. Section 4: "Results" presents the outcomes of the experiments described in Section 3 employing different metrics to evaluate different and complementary 115 points of view. All these results are analyzed in Section 5: "Discussion", where we comment on different strengths, weaknesses and highlights of the methodology. Finally, Section 6: "Conclusions", which includes a final notes drawn for this research and a commentary on future lines of work. Below, we will proceed to describe in detail the required materials and resources for the implementation and full reproduction of our work. In this section, the reader can see information and references of the datasets (Subsection 2.1), different software resources and their precise versions (Subsection 2.2) and hardware information as well as particular configuration of the equipment where the present project was conducted (Subsection 2.3). In this work, as we perform a double knowledge transfer, we need two different chest radiography datasets: the first one illustrating the general image domain and from which a larger number of samples are available (which we will call 130 "General COVID lung dataset") and another dataset containing explicit samples from the target domain. This second dataset will contain images obtained in live clinical practice from a local hospital during the COVID-19 pandemic. Specifically, from the Universitary Hospital Complex of A Coruña (CHUAC, by its acronym in Spanish). For this reason, we will address this second dataset 135 as the "CHUAC dataset". We will now proceed to explain in more detail the specifications and construction of each of the two datasets mentioned above. This first dataset was formed from public available datasets (Cohen et al., 2020; Kermany, 2018) . The dataset contains images with varying resolutions, 140 ranging from 5600 × 4700 pixels to 156 × 156 pixels including chest, lateral X-rays and CT images. For our purpose we discarded the latter two types. This was done because the portable devices of the consulted healthcare services were used only for chest X-rays. The dataset was labeled online in collaboration with different experts through the Darwin platform (V7 Labs, 2020) and is composed 145 of 6,302 chest radiographs, from which 438 correspond to patients diagnosed with COVID-19, 4,262 with lung pathologies similar to COVID-19 and 1,602 belonging to patients who (in principle) do not suffer from any of the previously mentioned conditions (albeit they can be affected by other pathologies). The second dataset was provided by the radiology service of the CHUAC from A Coruña, Galicia (Spain) obtained from two portable X-ray devices: an Agfa dr100E GE, and an Optima Rx200. For the acquisition procedure, the patient lies in a supine position and a single anterior-posterior projection is recorded. For this purpose, the X-ray tube is connected to a flexible arm that is extended over 155 the patient to be exposed to a small dose of ionizing radiation, while an X-ray film holder or an image recording plate is placed under the patient to capture images of the interior of the chest. All the images were obtained after triage in live medical wings specially dedicated for the treatment and monitoring of patients (Buda, 2020a) . This network is an implementation of an U-Net (Ronneberger et al., 2015) dedicated to the identification of brain structures in magnetic resonance imaging or MRI (Buda et al., 2019) . Specifically, the original network Figure 3 : Architecture of the pretrained convolutional neural network. Notice the encoderdecoder strategy with skip-connections, ideal for medical imaging segmentaton. has been trained to detect gliomas, a type of brain tumor diagnosed mainly by this imaging modality (Forst et al., 2014; Buda et al., 2019) , problematic that 185 share similar characteristics to our case, which is herein exploited. The precise architecture of this network is presented in Figure 3 . As can be seen in the figure, the network used is based on an encoder-decoder architecture. While the encoder learns the relevant filters to abstract the important information and process it in the bottleneck, the decoder will gradually generate the target segmentation. Since tests were conducted to evaluate the performance of the methodology as well as its use of resources (and to allow full reproducibility of results), we include 200 in Table 2 , the full disclosure of the components, drivers and software that have been used throughout the realization of this work and may have influenced its performance. To successfully develop a system able to work with radiographs from portable 205 devices with a limited amount available from the saturated health services, we followed a workflow that allowed us to progressively adapt information from a different medical imaging domain and pathology to ours. The main workflow followed in this work is detailed in Figure 4 , where we can see that each of the training stages performed in our project is repeated domain to chest radiography and a second one to further refine these weights specifically into the sensibly harder radiographs from portable devices. In the following two subsections, each stage will be explained in more detail. As both transfer learning stages share the same training algorithm, we will explain them 220 together in Subsection 3.3: "Training details". For this first step, we started from a model previously trained in a medical imaging domain with a large and heterogeneous set of images that presents similar characteristics to those we would find in the target domain (from which 225 we have available a limited number of samples). In our case, we used the U-Net trained with MRI images for glioma segmentation as shown in Section 2.2. As can be seen in Figure 5 , both image modalities present bright-to-dark perform the knowledge transfer, and the other to evaluate the performance and improvement of the system before and after this stage. In order to maintain consistency and allow for proper transfer learning, we have employed the same loss function used in the model trained with the brain 265 MRI images for the subsequent transfer learning stages. Therefore, both models have been trained using the Smooth Dice Loss (Equation 1). Where Op represents the predicted system output and Ot the expected output (target). λ is the smoothing factor, which has been defined as 1 in this work. As Specifically, the images were randomly rotated random degrees between -10º and +10º with a probability of 75% to simulate feasible postural variations of 285 the patient. The detailed strategy followed for each training cycle is depicted in Figure 6 . To evaluate the performance of our proposal in each stage, we analyzed the results in a wide range of metrics that allowed us to study the performance of Finally, AUC returns the probability that the analyzed model has of assigning 305 a higher value to a positive sample over a negative sample (Bradley, 1997) . In this section we will proceed to present the results that were obtained during the development of this work, product of the previously presented methodology. this first stage, we adapted from this domain to common lung radiographs. The average of all the repetitions for this training process can be seen in Figure 7 , and the mean test results of each of the chosen models in Table 3 . In this Figure 7 , we see that ( For this final test we used the 300 independent images from the CHUAC dataset that we separated for further analysis. The results of these tests can be seen detailed in Tables 5 and 6 ; where we present the results for the test of the model before and after the second stage of transfer learning (where the model is adapted to portable X-ray devices), respectively. Complementarily, this improvement is better observed in the comparison plots of Figures 9, 10 and 11 . These graphs show that where the most noticeable change has been in images that have some kind of pathology with effects similar to COVID-19, improving by almost 0.02 points in Jaccard and DICE coefficients. On the other hand, we also noticed a remarkable increase in the sensitivity of 370 the models, being this measurement critical in systems oriented to the medical sciences and clinical practice and also highly increased after the inter device type transfer learning stage into the portable X-ray image domain. Next, we present the results of an independent performance test with the Inter domain transfer learning Inter device transfer learning Finally, and exclusively measuring the network processing time of an image without taking into account any previous preprocessing and data transactions, the time required by the network to process each of the 300 images of the test set takes an average of 3.831 milliseconds with a standard deviation of 0.286. In Figure 12 we can see more closely examples of outputs generated by the proposed method. There is a clear distinction between the behavior of the two that our methodology is based on an early stopping when no improvement is achieved for a given number of epochs) is also significantly lower. This indicates that not only this amount of time is the result of a lower dataset, but also that the system converges earlier than in the first stage. In this work, we have proposed a system with the purpose of segmenting lung regions in thoracic radiographs, especially aimed at those obtained by portable X-ray devices in adverse conditions and with a limited supply of images. These devices, which represent an alternative to fixed devices to reduce the risk of crosscontamination in the diagnosis of diseases such as COVID-19, are critical for 460 medical emergency situations, so methodologies aimed to help in the diagnostic process that are functional with these images are critical. To solve the problem of poor image quality due to the capture conditions and devices themselves, we propose a fully automatic methodology based on two stages of transfer learning. A first stage based on knowledge transfer from a domain similar to radiographs 465 and trained with a large number of images (ensuring its robustness) to common chest radiographs obtained from different public sources and a second stage, in which knowledge is refined to adapt it to specific radiographs of a dataset obtained in adverse conditions in the clinical practice during the pandemic. As we have shown in the metrics of the results and in the discussion, while 470 the first stage of transfer learning allows the system to acquire the knowledge bases of the domain to generate an initial segmentation, the second stage of knowledge transfer to the particular domain manages to refine satisfactorily the obtained segmentations even with a limited set of samples. This second stage of transfer learning allows not only to better estimate the pulmonary region, 475 but also to eliminate various artifacts resulting from the lower sample quality present in the images from portable devices. Thus, as a final result of this work, we have successfully obtained a fully automatic methodology based on deep methodologies, using a limited number of images from portable devices and capable of working with these images in a 480 robust and consistent way, regardless of the image quality and capture conditions. As future work, it would be interesting to study mechanisms to adapt the network input resolution so that it could support variable input sizes (in addition to study the performance difference between both proposals) to solve the border degradation in the segmentation product of the rescaling of the images. Another 485 aspect that would be desirable to improve is the network that was used as a basis for knowledge transfer. This network is receiving as input an image of resolution 256 x 256 x 3. However, the pulmonary radiography images we use only have useful information in grayscale (ergo the information is replicated along the three input channels). It would be interesting to explore other works as 490 foundation that, like our images, employ a network with single-channel input to make the training more efficient and possibly improve its generalization capacity (by reducing the overall size of the network). Another known problem of transfer learning with the technique we use is the abrupt changes of the gradient during the training that can cause the degradation 495 of features already learnt by the network during the pretraining. An alternative technique for knowledge transfer is the addition of new layers at the end of a pre-trained network and freezing the weights of the original layers. By doing so, the network would be extended with a sort of "domain translator module". Thus, the feature extraction mechanism of the original network would be kept 500 static (its weights would not be altered during training) and, consequently, the features learned during the basic training would be fully preserved. On the other hand, given the positive results obtained in the application of this methodology, we see that, in fact, the features present in MRI image of cerebral glioma are reusable in the field of lung region segmentation in portable 505 chest X-rays. Another interesting future work would consist in the so-called "deep feature analysis", which would allow to study the common features learned by the network in both domains and thus help to better understand and improve the present and future clinical diagnostic support systems. Additionally, given that all the images analyzed in the portable dataset come almost no patients with 510 implants or foreign objects that could leave artifacts in the chest radiographs, it would be interesting to study the impact of these devices on the capabilities of the system to correctly infer the lung region, as well as (connecting to the previous topic) the effect on these artifacts on the features the networks deems relevant to detect them. CRediT author statement Covid mtnet: Covid-19 detection with multi-task deep learning ap-540 proaches ACR Recommendations for the use of Chest Radiography and Computed Tomography (CT) for Suspected COVID19-Infection accessed Covid-19: automatic detection from x-ray images utilizing transfer learning with convolutional neural networks The use of the area under the ROC curve in the evaluation of machine learning algorithms Explainable deep learning for pulmonary disease and coronavirus COVID-19 detection from x-rays Brain mri images together with manual flair abnormality segmentation masks obtained from the cancer imaging archive (tcia) Available at: https: 565 //pytorch.org/hub/mateuszbuda brain-segmentation-pytorch unet Association of genomic subtypes of lower-grade gliomas with shape features automatically extracted by a deep learning algorithm Residual attention u-net for automated multi-class segmentation of covid-19 chest ct images Covid-19 image data collection: Prospective predictions are the future Coronavirus 2019-ncov global cases by johns hopkins csse Inf-net: Automatic COVID-19 lung infection segmentation from CT images Low-grade gliomas Spectrum of imaging findings in pulmonary infections Organspecific manifestations of COVID-19 infection Automatic x-ray COVID-19 lung image classification system based on multi-level thresholding and support vector machine. medRxiv A combined deep cnn-lstm network for the detection of novel coronavirus (covid-19) using x-ray images Portable chest x-ray in coronavirus disease-19 (COVID-19): A pictorial review Chest X-ray in clinical practice Labeled optical coherence tomography (oct) and chest x-ray images for classification Adam: A method for stochastic optimization Venous and arterial thromboembolic complications in covid-19 patients admitted to an academic hospital in milan, italy Within the lack of chest 625 COVID-19 x-ray dataset: A novel detection model based on GAN and deep transfer learning. Symmetry, 12 , 651 Artificial intelligence-enabled rapid diagnosis of patients with COVID-19 Abnormal pulmonary function in covid-19 patients at time of hospital discharge Fully automatic deep convolutional approaches for the analysis of covid-19 using chest x-ray images. medRxiv Deep convolutional approaches for the analysis of COVID-19 using chest x-ray images from portable devices Automated detection of COVID-19 cases using deep neural networks with x-ray images Pytorch: An imperative style, high-performance deep learning library Advances in Neural Information Processing Systems Scikitlearn: Machine learning in Python A modified deep convolutional neural 670 network for detecting COVID-19 and pneumonia from chest x-ray images based on the concatenation of xception and ResNet50v2 U-net: Convolutional networks 675 for biomedical image segmentation Detection of coronavirus disease (COVID-19) based on deep features. Preprints: The Multidisciplinary Preprint Platform, 680 Review of artificial intelligence techniques in imaging data acquisition, segmentation and diagnosis for COVID-19 Automated detection and forecasting of covid-19 using deep learning techniques: A review COVIDiagnosis-net: Deep bayes-SqueezeNet based diagnosis of the coronavirus disease 2019 (COVID-19) from x-ray images Medical Hypotheses Labeled COVID-19 Chest X-Ray Dataset, Darwin, V7Labs The COVID-19 epidemic Frequency and distribution of chest radiographic findings 705 in patients positive for COVID-19 Covid-19 chest ct image segmentation-a deep convolutional neural network solution COVID-19 and multiorgan response Viral pneumonia screening on chest x-ray images using confidence-aware anomaly detection