key: cord-0762040-cnd4dhkc
authors: Li, Chun; Yang, Yunyun; Liang, Hui; Wu, Boying
title: Transfer learning for establishment of recognition of COVID-19 on CT imaging using small-sized training datasets [Image: see text]
date: 2021-02-06
journal: Knowl Based Syst
DOI: 10.1016/j.knosys.2021.106849
sha: 1d3637c12b2b4c3f086fcf70cfa8e41ec1f9a2d6
doc_id: 762040
cord_uid: cnd4dhkc

The coronavirus disease, called COVID-19, which is spreading fast worldwide since the end of 2019, and has become a global challenging pandemic. Until 27th May 2020, it caused more than 5.6 million individuals infected throughout the world and resulted in greater than 348,145 deaths. CT images-based classification technique has been tried to use the identification of COVID-19 with CT imaging by hospitals, which aims to minimize the possibility of virus transmission and alleviate the burden of clinicians and radiologists. Early diagnosis of COVID-19, which not only prevents the disease from spreading further but allows more reasonable allocation of limited medical resources. Therefore, CT images play an essential role in identifying cases of COVID-19 that are in great need of intensive clinical care. Unfortunately, the current public health emergency, which has caused great difficulties in collecting a large set of precise data for training neural networks. To tackle this challenge, our first thought is transfer learning, which is a technique that aims to transfer the knowledge from one or more source tasks to a target task when the latter has fewer training data. Since the training data is relatively limited, so a transfer learning-based DensNet-121 approach for the identification of COVID-19 is established. The proposed method is inspired by the precious work of predecessors such as CheXNet for identifying common Pneumonia, which was trained using the large Chest X-ray14 dataset, and the dataset contains 112,120 frontal chest X-rays of 14 different chest diseases (including Pneumonia) that are individually labeled and achieved good performance. Therefore, CheXNet as the pre-trained network was used for the target task (COVID-19 classification) by fine-tuning the network weights on the small-sized dataset in the target task. Finally, we evaluated our proposed method on the COVID-19-CT dataset. Experimentally, our method achieves state-of-the-art performance for the accuracy (ACC) and F1-score. The quantitative indicators show that the proposed method only uses a GPU can reach the best performance, up to 0.87 and 0.86, respectively, compared with some widely used and recent deep learning methods, which are helpful for COVID-19 diagnosis and patient triage. The codes used in this manuscript are publicly available on GitHub at (https://github.com/lichun0503/CT-Classification).

The coronavirus disease 2019 (COVID-19) is spreading fast worldwide since the end of 2019, and has become a global challenging pandemic. As shown in has declared the COVID-19 as a global health emergency on January 30, 2020, [1] , which poses a great threat to international human health. Because it is highly infectious and vaccine development is still in clinical trials, early diag- The chest computed tomography (CT) is useful for the clinical assistant diagnosis of COVID-19 [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] , however, since the terrible increase in the number of COVID-19 infections, there is a huge shortage of clinicians and radiologists, 20 moreover, the routine use of CT puts a huge burden on the radiology department, meanwhile, there is a potential infection in CT equipment. As a result, many infected people cannot get a timely diagnosis, so they continue to infect others unconsciously. Therefore, it is favorably hunger for to researchers explore automated computer-aided diagnostic system classification methods for and the health of people all over the world, if the diagnostic performance of CT images is improved, even if CT images cannot completely replace the nucleic 30 acid detection method of COVID-19, CT images information can contribute to treatment decisions based on more accurate and reproducible data. Other benefits include reduced need for CT scans, which reduces delays, radiation levels, and costs. Consequently, in the field of medical image analysis, more and more imaging-based artificial intelligence diagnostic systems were developed to 35 combat COVID-19. Those systems can be roughly divided into several categories: segmentation-based [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] , classification-based [22] [23] [24] [25] [26] , and follow-up and prognosis [27] . Those methods so that categorize the population through a computer-aided diagnosis system based on CT classification, which can reduce the medical burden of the hospital, make effective use of limited medical 40 resources, and enable effective treatment of critically ill patients, thereby reducing the spread of disease. In terms of image-based disease diagnosis, some valuable works should be introduced, for example, the related works for multimodal disease diagnosis are as follows [28, 29] . Additionally, Zhou et al. [30] proposed a novel hybrid-fusion network (Hi-Net) for multi-modal MR image assisted-diagnosis. Unfortunately, the current public health emergency, which 60 has caused great difficulties in collecting a large set of precise data for training neural networks. Under the current grim circumstances, COVID-19 patients occupy a large degree of medical resources (clinicians and radiologists), therefore, they are unlikely to have time to collect and annotate a large number of COVID-19 CT scans. Consequently, one of the main concentration of this pa- 65 per is to develop a deep learning-based method such that can obtain the desired diagnostic effect for COVID-19 from CT scans, in the case of limited training samples.

To this end, in this paper, we propose a novel transfer learning-based method with a small-size of medical image data for the classification between COVID-19 70 with Non-COVID-19. Transfer Learning (TL) is not a strange thing for human beings, which is the ability to learn by analogy. For example, If we have learned to ride a bicycle, learning to ride a motorcycle becomes very simple for us. If we have learned to play badminton, learning to play tennis is not that difficult.

However, for computers, the TL is a technology that allows the existing model 75 algorithms to be slightly adjusted to be applied to a new field or function. Regarding TL, the following researchers have done several valuable works, such as Long et al. [40] developed a conditional adversarial domain adaptation focused on discriminative information conveyed in the classifier predictions. Also, Wang et al. [41] presented visual domain adaptation with manifold embed-80 ded distribution alignment. Furthermore, Liu [42] developed an unsupervised approach for heterogeneous domain adaptation. According to the scenarios of TL, which can be split into three main categories: inductive (the source and target domains have different learning tasks), transductive (the source domain and target domain are different and the learning tasks are the same), and un-role in shortening the difference in the data distribution between domains when the data are transmitted cross-domain [43, 44] . Currently, several references address this problem. In contrast, in terms of heterogeneous TL, the feature spaces from the source and target domain are nonequivalent, moreover, they even are usually non-overlapping, also, the dimensions of the feature spaces 95 may be different. Therefore, the key role of this approach is not only to deal with the gaps in knowledge transfer but to handle the differences in data distribution of cross-domain. Hence, this situation is more challenging, since few same modal or features can be shared. From the perspective of transforming the feature spaces, heterogeneous TL can be roughly divided into two categories: 100 symmetric and asymmetric transformation [43, 44] . More currently, with the de- the deep neural network model is a data-hungry method. When training on a small-sized dataset, it is easy to produce the risk of overfitting. In typical transfer learning, researchers usually divide the data for transfer learning into 115 two categories, one is source data and the other is target data. Source data refers to additional data and is not directly related to the task to be solved, while target data is data directly related to the task. Source data is often huge, and target data is often small. How to make good use of source data to help and even improve the performance of the model on target data is the issue to be 120 considered for transfer learning. Furthermore, the purpose of transfer learning is The rest of the paper is organized as follows, in Section 2, we first present 7 J o u r n a l P r e -p r o o f some related works for the diagnosis of COVID-19. Furthermore, we introduce the materials utilized in this study and the proposed transfer learning for recognition of COVID-19 in Section 3. After that, in Section 4, we describe the experimental settings, experimental results, the limitations of the current study, 145 and present several future research directions. Finally, we summarize this paper in Section 5.

In this section, we briefly review the most relevant studies from the following three aspects: 1) prediction-based learning method for identification of COVID- in 3D CT images. Selvan et al. [48] used lung segmentation from chest X-rays using the variational data imputation method aimed at automated risk scoring of COVID-19 from CXRs. Oh, et al. [15] proposed a patch-based convolutional neural network approach for COVID-19 diagnosis by learning COVID-19 180 features on CXR using limited training data sets. Moreover, U-net [49] , as a classic successful model for medical image segmentation, the related works of using U-net to predict COVID-19 are presented in [50] [51] [52] , and several variants of U-Net have been applied to the diagnosis or severity assessment of COVID-19 [53] [54] [55] [56] . from CT scans and claimed that they built the largest of publicly-available CT-dataset for COVID-19.

First of all, in this section, we will introduce the dataset used in our inves- and Xie [57] . They claimed that it is the largest COVID-19-CT dataset that is publicly available to date. They collected 760 preprints about COVID-19 from bioRxiv3 and medRxiv2 from Jan 19th to Mar 25th. According to the original description of the database established by them, numerous of these preprints contain the cases of COVID-19 and the CT images in their report.

The authors used the python package (PyMuPDF4) to extract the CT images from these preprints, after that, they manually selected all CT images, then determine whether the CT is positive. For each CT image, the authors collected the meta information extracted from the paper, such as patient age, gender, location, medical history, scan time, the severity of COVID-19, and 210 radiology report [57] . Through various selections, they got 349 CT images la- For detailed information about the database, please refer to [57] .

Since the CT images have different sizes, therefore, they were resized to 224 × 224, we adopted the method of dataset splitting by the authors, namely, a training, validation, and a test set by patient IDs with a ratio of 0.6: 0.15: 0.25. Since 

Recently, numerous works show that if a network performs satisfactory results, image-based classification also will have high performance on the COVID-19 task. Numerous network structures [58] [59] [60] [61] [62] [63] [64] [65] have been proposed for visual object recognition, and achieved gratifying performance. Both ResNets [58] 250 and DensNets [59] architectures are successfully applied to in various image recognition tasks, and achieved outstanding performance.

Supervised and semi-supervised learning as the traditional machine learning algorithms aim to make predictions on the future data employing some models that are trained on previously collected labeled or unlabeled training data 255 and assume that the labeled and distributions of unlabeled data are the same domains. Transfer learning, in contrast, allows the domains, tasks, and distributions used in training and testing to be different [66] . It is a technique that aims to transfer the knowledge from one or more source tasks to a target task when the latter has fewer training data. Transfer learning is only interested in 260 the target task instead of source tasks. Consequently, a typical approach is to J o u r n a l P r e -p r o o f Journal Pre-proof pre-train a deep learning algorithm, which is used for feature extraction on large datasets in the source tasks by fitting the human-annotated labels therein, then fine-tune this pre-trained network on the target task [67] .

Testing stage 

Our pre-defined models are as follows, ResNet34 [58] , ResNet50 [58] , resNet101 [58] , DensNet121 [59] , and the latest Network named ResNest [68] . We did a lot 

Since the Pneumonia detection can be seen as a binary classification problem, therefore, the binary cross-entropy loss function L (Θ) was exploited to compute the loss between predicted-labels and the ground-truth labels, where the input is a lung CT images and the output t ∈ {0, 1} is a binary label indicating the presence or absence of Pneumonia. The formulation of the loss function L (Θ)

is as follow,

here, p (Y = i|X) is the probability that the network assigned to label i. the dataset consists of three parts, namely, the training set (X train , Y train ), validate set (X val , Y val ), and test set (X test , Y test ), the batch-size of the training set is 8, then by iteratively optimizing (fine-tuning) the networks to reduce the objective loss function:

where L(.), M (x i , Θ) refer to the deep model that predicts class y i for input

x i and the given weight Θ. Let layer l be a convolutional layer, then we can compute the input layer l − 1 as follows:

where B l i , Θ i refer to a bias matrix, weight matrix, respectively. After that, the activation function as follows can be applied to the convolutional layer:

The common activation functions are sigmoid, tanh, rectified linear units (ReLu), and so on. In our method, we use ReLu function as the activation function, namely,

During the procedure of testing, data augmentation strategies were not used, and we test the model one time after training 1 epoch. After that, the predicted probabilities of all COVID-19 cases and the corresponding true labels were accumulated for statistical analysis. The loss function training history, ACC, and F1-score are shown in Figure 6 . Densenet-121 with a single output unit, and then they applied the nonlinear sigmoid function to output an image containing the probability of pneumonia. As COVID-19 is a two-class problem, we change the neurons in the output layer to two neurons, and the weights of the neurons are initialized by Kaiming's initialization [72] . 295 

In this section, we will describe more details about experimental setup and results while testing the performance of our method.

In our experiments, five criteria are employed to evaluate the COVID-19 300 identification performance, and the definitions are shown as follows. We use five types of commonly used metrics to evaluate the classification performance achieved by different methods in the classification task. Table 1 . About the data split information for the patient ID of Non-COVID-19, the number of train-id, test-id, and validate-id is 105, 42, and 24, respectively. Also, the data split information for the patient ID of COVID-19, the number of train-id, test-id, and validate-id is 191, 98, and 60, respectively. For more detail please refer to [57] 330 and https://github.com/UCSD-AI4H/COVID-CT/blob/master/Data-split/.

We trained the deep neural network models on our equipped with a GeForce RTX 1080 super GPU. The deep network software was developed based on the Pytorh framework. In the procedures of training, the binary cross-entropy loss function and SGD optimization function with momentum were used, re-335 spectively. Batch normalization is used through all models, and the training BathSize is applied through all models.

We evaluate the COVID-19 vs. Non-COVID-19 classifications on the COVID-CT-Dataset. The detection results of COVID-19 were presented and analyzed 340 using ACC and F1-score, and AUC. Because Precision and Recall are already reflected in the computing formula of F1-score, it is omitted here. Specifically, ACC and F1-score, and AUC are some critical indicators that are used to estimate the deep learning algorithm, for medical analysis. Table 2 shows the quantitative results (i.e. ACC, F1-score, and AUC) achieved by different 345 methods. From Table 2 , we also can observe that our deep classification model archives the best performance in terms of ACC, F1-score. However, it is worth noting that our AUC value is lower than SEDLCD model, it is also possible to 18 J o u r n a l P r e -p r o o f be due to the number of GPUs and computer configuration, which caused our AUC to be lower than the SEDLCD (4 GPUs required).

For quantitative evaluation, as shown in Table 2 , both conventional learning models and the proposed transfer learning methods achieve considerable improvement with the learned latent representation in terms of all two metrics. though the data used in this study came from the large of publicly-available CT-dataset for COVID-19, because deep learning belongs to the data-hungry method, so a few hundred images are not enough for it, to be further crossvalidated in the more publicly-available dataset. Thirdly, it is worth noting that our AUC value is lower than SEDLCD model, it is also possible be due to 370 the number of GPUs and computer configuration, which caused our AUC to be lower than the SEDLCD (4 GPUs required). Moreover, we still face difficulties when trying to capture the features of COVID-19 and Non-COVID-19, since data dependence is one of the most serious problems in deep learning, insufficient training data is an inevitable problem in some special fields, especially in 375 the early stage of the COVID-19 outbreak. Although TL relaxes the assumption that training data must be independent and identically distributed with test data, however, in the training stage, since insufficient data such that over-fitting is easy to occur, resulting in high accuracy in the training stage, and there is a certain gap in the accuracy of the test and the training phase. Data distribution 380 is another problem, in future work, we will combine TL and model-driven methods to reduce over-fitting, thereby reducing the gap between training accuracy and test refinement, and improving the performance of the model. Also, more features (patient age, gender, location, medical history, scan time, the severity of COVID-19) will be considered in our proposed method aiming to improve 385 further performance. Furthermore, such as the Laplacian smoothing optimizer

[76] also will be leveraged to speed up network optimization, shorten training time, and improve classification accuracy.

In this paper, we propose a transfer learning-based deep method that aims to 390 identify COVID-19 from Non-COVID-19 using relatively small-sized CT images.

We confront one of the challenging issues that have great difficulties in collecting a large set of precise data for training neural networks. Our proposed method 20 J o u r n a l P r e -p r o o f employs the characteristic of transfer learning that transfers the knowledge from one or more source tasks to a target task when the latter has fewer training data.

CheXNet as the pre-trained network was used for the target task (COVID-19 classification) by fine-tuning the network weights on the small-sized dataset in the target task. Experimental results on a real COVID-19 CT image dataset demonstrate that our method achieves good performance in severity assessment of COVID-19 patients, compared with several state-of-the-art methods, except 400 AUC.

J o u r n a l P r e -p r o o f Journal Pre-proof

World Health Organization declares global emergency: A review of the 2019 novel coronavirus (COVID-19)

Chest CT for Typical 2019-nCoV Pneumonia: Relationship to Negative RT-PCR Testing

CT Imaging Features of 2019 Novel Coronavirus (2019-nCoV)

Coronavirus disease (COVID-19): spectrum of CT findings and temporal progression of the disease

Diagnosis of the Coronavirus disease (COVID-19): rRT-PCR or CT?

Chest computed tomography in children with COVID-19 respiratory infection

Clinical evaluation of an immunochromatographic IgM/IgG antibody assay and chest computed tomography for the diagnosis of COVID-19

Chest computed tomography images of early coronavirus disease (COVID-19)

Chest Computed Tomography Manifestation of Coronavirus Disease 2019 (COVID-19) in Patients With Cardiothoracic Conditions

Can computed tomography be a primary tool for COVID-19 de-435 tection? Evidence appraisal through meta-analysis

Coronavirus Disease 2019-related dyspnea cases difficult to interpret using chest computed tomography

AI-assisted CT imaging analysis for COVID-19 screening: Building and deploying a medical AI system in four weeks

Machine learning-based CT radiomics model for 445 predicting hospital stay in patients with pneumonia associated with SARS-CoV-2 infection: A multicenter study

Synergistic Learning of Lung Lobe Segmentation and Hier-22

archical Multi-Instance Classification for Automated Severity Assessment 450 of COVID-19 in CT Images

Deep learning COVID-19 features on CXR using limited training data sets

Review of artificial intelligence techniques in imaging data acquisition, seg-455 mentation and diagnosis for COVID-19

Inf-Net: Automatic COVID-19 Lung Infection Segmentation from CT Scans

JCS: An explainable COVID-19 diagnosis system by joint classification and segmentation

Modelling COVID-19 outbreak: segmented regression to assess lockdown effectiveness

Multi-task Deep Learning Based CT Imaging Analysis For COVID-19: Classification and Segmentation, medRxiv

COVID-19 Detection with Multi-Task Deep Learning Ap-470 proaches

Adaptive Feature Selection Guided Deep Forest for COVID-19 Classification with Chest CT

Sample-Efficient Deep Learning for COVID-19 Diagnosis Based on CT Scans

Comorbidities and multi-organ injuries in the treatment of COVID-19

CT image visual quantitative evaluation and clinical classification of coronavirus disease (COVID-19)

Large-scale screening of COVID-19 from community acquired pneumonia using infection size-aware classification

Review of artificial intelligence techniques in imaging data acquisition, seg-490 mentation and diagnosis for COVID-19

Effective feature learning and fusion of multimodality data using stage-wise deep neural network for dementia diagnosis

Latent representation learning for Alzheimer's disease diagnosis with incomplete multi-modality neuroimaging and genetic data

Hi-net: hybrid-fusion net-500 work for multi-modal MR image synthesis

Rapid AI development cycle for the coronavirus (COVID-19) pandemic: Initial results for automated detection & 505 patient monitoring using deep learning CT image analysis

A tailored deep convolutional neural network design for detection of COVID-19 cases from chest radiography images

Automatic detection of coronavirus disease (COVID-19) using X-Ray images and deep convolutional neural networks

Covidx-Net: A framework of deep learning classifiers to diagnose COVID-19 in X-Ray images

Extracting possibly representative COVID-19 Biomarkers from X-Ray images with Deep Learning approach and image data related to Pulmonary Diseases

A deep learning framework for screening of COVID-19 from radiographs

COVID-CAPS: A capsule network-based framework 525 for identification of COVID-19 cases from X-Ray images

Synergistic Learning of Lung Lobe Segmentation and Hierarchical Multi-Instance Classification for Automated Severity Assessment of COVID-19 in CT Images

Conditional Adversarial Domain 535 Adaptation

Visual Domain Adaptation with Manifold Embedded Distribution Alignment

Heterogeneous Domain Adaptation: An Unsu-540 pervised Approach

A survey on heterogeneous transfer learning

A survey of transfer learning

A survey on deep transfer learning

Automated deep 550 transfer learning-based approach for detection of COVID-19 infection in chest X-Rays

Fuzzy multiple-source transfer learning

Lung 555 Segmentation from Chest X-Rays using Variational Data Imputation

U-net: Convolutional networks for biomedical image segmentation

Machine learning-based CT radiomics model for predicting hospital stay in patients with pneumonia associated with SARS-CoV-2 infection: A multicenter study

Deep learning-based detection for COVID-19 from chest CT using weak label, medRxiv

Serial quantitative chest CT assessment of COVID-19: deep-learning approach

Lung infection quantification of COVID-19 in CT images with deep learning

AI-assisted CT imaging analysis for COVID-19 screening: Building and deploying a medical AI system in four weeks

Unet++: A nested u-net architecture for medical image segmentation

V-net: Fully convolutional neural networks for volumetric medical image segmentation

CT scan dataset 585 about COVID-19

Deep residual learning for image recognition

Densely con-590 nected convolutional networks

Photo-realistic single image super-resolution using a generative adversarial network

Very deep convolutional networks for largescale image recognition

Highway networks

Contextual point wise convolution for object recognition

Deep sensorimotor learning for RGB-D object recognition

Assistance via IoT networking cameras and evidence theory for 3D object instance recognition: 610 Application for the NAO humanoid robot

A survey on transfer learning

Rich feature hierarchies for 615 accurate object detection and semantic segmentation

ResNeSt: Split-Attention Networks

Chexnet: Radiologist-level pneumonia detection on chest X-Rays with deep learning

Chestx-ray8: Hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases

On the momentum term in gradient descent learning algorithms

Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

Support vector machines for classification and regression

Classification and regression by random Forest

Manuscript title: Transfer learning for establishment of recognition of COVID-19 on CT imaging using small-sized training datasetsThe authors whose names are listed immediately below certify that they have NO affiliations with or involvement in any organization or entity with any financial interest (such as honoraria; educational grants; participation in speakers' bureaus; membership, employment, consultancies, stock ownership, or other equity interest; and expert testimony or patentlicensing arrangements), or non-financial interest (such as personal or professional relationships, affiliations, knowledge or beliefs) in the subject matter or materials discussed in this manuscript.Author names: Chun Li, Yunyun Yang, Hui Liang, Boying Wu This statement is signed by all the authors to indicate agreement that the above information is true and correct (a photocopy of this form may be used if there are more than 10 authors):