key: cord-0955208-bq6f7q2k authors: Ali, Abbas Raza; Budka, Marcin title: An Automated Approach for Timely Diagnosis and Prognosis of Coronavirus Disease date: 2021-04-29 journal: International Joint Conference on Neural Networks (IJCNN) DOI: 10.1109/ijcnn52387.2021.9533786 sha: 71f023e111af5250ff0ddcb69c72cd101f29d6f3 doc_id: 955208 cord_uid: bq6f7q2k Since the outbreak of Coronavirus Disease 2019 (COVID-19), most of the impacted patients have been diagnosed with high fever, dry cough, and soar throat leading to severe pneumonia. Hence, to date, the diagnosis of COVID-19 from lung imaging is proved to be a major evidence for early diagnosis of the disease. Although nucleic acid detection using real-time reverse-transcriptase polymerase chain reaction (rRT-PCR) remains a gold standard for the detection of COVID-19, the proposed approach focuses on the automated diagnosis and prognosis of the disease from a non-contrast chest computed tomography (CT)scan for timely diagnosis and triage of the patient. The prognosis covers the quantification and assessment of the disease to help hospitals with the management and planning of crucial resources, such as medical staff, ventilators and intensive care units (ICUs) capacity. The approach utilises deep learning techniques for automated quantification of the severity of COVID-19 disease via measuring the area of multiple rounded ground-glass opacities (GGO) and consolidations in the periphery (CP) of the lungs and accumulating them to form a severity score. The severity of the disease can be correlated with the medicines prescribed during the triage to assess the effectiveness of the treatment. The proposed approach shows promising results where the classification model achieved 93% accuracy on hold-out data. obstruction, runny nose, dry cough, sore throat, fatigue, respiratory symptoms, dyspnea, myalgia, and diarrhoea [1] . Moreover, even asymptomatic patients hospitalised for observation showed lung abnormalities on a CT scan, ranging from GGO to CP [16] . In order to address the discrepancies of rRT-PCR, effective and efficient diagnosis and triage of COVID-19 are urgently needed. Therefore, an automated medical imaging-based approach to identify COVID-19 from the chest CT scan has been proposed in this paper. We have achieved automated diagnosis of the disease using deep learning [17] techniques, able to instantly identify radiographic changes from the CT scans, allowing to make the diagnosis much earlier than the pathogenic testing like rRT-PCR, which can, in turn, save critical time for minimising the spread of the virus [18] . In the proposed approach, once a patient is diagnosed as positive at an early stage, the same diagnostic process repeats periodically during the course of the treatment. Along with the diagnosis of the disease, an important aspect is the quantification of its impact. One way of achieving it using CT is by computing the diameter of multiple patchy areas of GGO and shadows of the CP of the lungs [19, 20] . This quantification of the disease serves two purposes, 1) monitoring the progression of the disease over time, and 2) in the absence of a vaccine, assessment of the treatment by correlating the effectiveness of the general medication prescribed to a patient. The medication includes cough syrup, ibuprofen [21, 22] , acetaminophen [22] , non-steroidal anti-inflammatory drugs (NSAIDs) [23] , anti-inflammatory drug dexamethasone [24] and anti-malaria drugs, namely hydroxychloroquine (HCQ) and chloroquine (CQ) along with anti-viral medication remdesivir. However, due to heartbeat irregularity side-effects associated with HCQ, it is no longer prescribed to COVID-19 patients in some countries [14] . The proposed approach revolves around an automated classification and measurement of COVID-19. The highprecision has been achieved by training chest CT segmentation and classification models using deep learning techniques. These techniques usually require large datasets to train (in excess of 1000 examples per class for a relatively simple classification task) that are not available particularly for COVID-19. Therefore, transfer learning has been used to adapt a pre-trained network, which was originally trained on a large dataset -ImageNet [25] , on a new domain having a comparatively small number of chest CT images. Here, transfer learning is used in combination with Metalevel Learning (MLL) to choose an optimal architecture, the approach suggested by [26] , and its Hyper-parameters Optimization (HPO) [27] which could effectively adapt to a new domain. The deep learning techniques have been shown to work better if only critical, noise-free areas of the scan are fed for diagnosis [28] . Hence, before passing the image to the classification model, an existing deep learning-based image segmentation model of [29] has been applied to the raw images as a pre-processing step to extract the interesting features [30, 31, 32] . The COVID-19 positive classified cases are further quantified to measure the severity of the disease. To this end, the area of all the GGO and CP identified during classification are accumulated to compute an overall severity level. The classification and quantification processes repeat periodically until the patient is fully recovered and the severity score approaches zero. The severity is plotted on a timeline to help medical staff analyse the history of the patient along with the progression of the disease over time (see Figure 5 for an example). For the recovered patients, the timeline is correlated with the prescribed medications to get an indication of their effectiveness. This analysis also explains the patterns of patients' clinical data, such as age, gender, location, initial symptoms, and prior medical history versus the medications which leads to recovery. The novelty of this work is the development of an end-to-end pipeline for real-time diagnosis, quantification, progression analysis, and assessment of the medication to improve the overall treatment process of the COVID-19. The remainder of this paper is organised as follows. Section 2 further elaborates the proposed approach. This section covers the datasets used to train and validate the chest CT classification model. The disease quantification and assessment of treatment based on the severity of the diagnosis are also covered in this section. The results of various experiments have been presented in Section 3. The paper is concluded in Sections 4 and 5. The proposed approach is capable of providing a timely diagnosis of COVID-19 along with quantification, progression, and assessment during the course of the treatment. These four phases are interlinked with each other where the process initiates with automated diagnosis of the suspected case. The non-contrast chest CT is found to be the most effective clinical examination for the diagnosis of the disease since its outbreak [13] . In order to make the diagnosis efficient, a medical imaging-based model for the chest CT was trained using deep learning techniques to accurately discriminate COVID-19 positive from negative cases. The classification results of each consecutive CT scan have been recorded in the Electronic Health Record (EHR) system along with the clinical information of a patient. The results include positive or negative classification of the disease and for positive cases, the measurement of the infected segments [7] . The initial diagnosis has been used for early detection of the disease which can be later cross-checked with the results of rRT-PCR. Moreover, the early clinical diagnosis has proved effective for the patients showing at least mild symptoms which are mentioned earlier [33] . Once a patient is confirmed with COVID-19 (using either rRT-PCR or the approach proposed in this paper) and admitted for treatment, the monitoring process is initiated which periodically repeats the same diagnosis and quantification procedure to analyse the progression of the disease over the course of the treatment. The progression of the disease has been quantified by estimating the size of the abnormalities (GGO and CP) found during the diagnosis. The disease progression can be visualised over time since the beginning of the treatment to help medical staff assess the effectiveness of medication ( Figure 5 ). In the absence of the vaccine, this system can also correlate the progression of the disease with the prescribed medicines which are primarily not developed for COVID-19. However, due to lack of access to relevant data, the correlation analysis has not been performed in this study. The workflow of the four-phase process is shown in Figure 1 . The following sections elaborate on the approach used to pre-process the training data, chest CT modelling, quantification of the disease from the positive cases, and their assessment by correlating the severity level with the prescribed medicines. The chest CT features generally observed in COVID-19 patients are bilateral multiple rounded GGO, CP, calcification, and lesion distribution [34, 35, 36, 37] . In order to train a model that can discriminate scans pertaining these features from the normal ones, a dataset of COVID-19 patients' chest CT scans were acquired from multiple sources which are listed in Table 1 . The merged dataset consists of 9,493 scans of 6,673 patients out of which 4,050 are COVID-19 positive, 2,274 have community-acquired pneumonia (CAP), and remaining 3,169 are healthy. A total of 2,522 scans are collected from the Third Hospital of Jilin University, Ruijin Hospital of Shanghai Jiao Tong University, Tongji Hospital of Huazhong University of Science and Technology, Shanghai Public Health Clinical Center of Fudan University, and Hangzhou First Peoples Hospital of Zhejiang University [38] . The scans are further divided into 1,495 COVID-19 positive cases which are confirmed by rRT-PCR and 1,027 scans belong to CAP patients. The dataset used by [11] consists of only 112 COVID-19 cases as the study revolves around key findings of COVID-19. [39] proposed classification of CAP cases versus the healthy ones, hence, only healthy patients' scans of 2,265 have been collected from the Lung Image Database Consortium (LIDC) [42] . The scans and their labels from five different data sources were pre-processed, initially, by segregating the thorax region of both lungs from the background. These regions of a scan were segregated by applying a gray-level distribution of the Wiener-filtered image where different spikes of the distribution correspond to the lung, fat and muscle of the thorax region [43] . The lung CT scan is a combination of X-ray photons taken from different angles to produce cross-sectional slices of the lungs, therefore, arbitrarily two images per scan were selected. Further, automated semantic segmentation was applied to identify the infected region of interests (ROIs), such as, different organs and lesions, from the CT slices which are crucial information for both diagnosis and quantification of the disease [44] . For this purpose, a commonly used medical image segmentation model, UNet++, trained on LIDC by [29] , was used to extract ROIs from the image. The segmentation of GGO at an early stage of the disease can be seen on the top whereas bottom scans show the segmentation of both GGO and CT at the critical stage of the disease as shown in Figure 2 . The data augmentation mitigates the issue of limited training data by adding synthesized image-label pairs into the training set. The pre-processed data was augmented with affine transformation which consists of 15 and 20 degrees rotation and translation respectively, random crop and horizontal flip [40] . The scans gathered from multiple sources having different sizes were rescaled to 224x224 pixels and normalized using the per channel mean and standard deviation of the ImageNet dataset. An end-to-end data pre-processing, model training, and diagnosis pipeline are demonstrated in Figure 3 . The training of deep learning models on a comparatively small dataset may lead to over-fitting where the model performs well on the training dataset, however, it doesn't generalise. In order to mitigate this limitation, transfer learning techniques have been adopted which acquire knowledge on a specific problem and reduce it to a different but related task [45] . Although, transfer learning is usually performed to fine-tune only the final fully-connected (FC) layer(s) of a pre-trained network to a new domain [46, 47] , it does not guarantee state-of-the-art accuracy, particularly on relatively different tasks. Hence, in this work, a meta-transfer learning approach has been used for model training which fine-tunes an increasing number of layers of the network based on the complexity and relevance of both tasks as proposed by [26] . For the disease classification modelling, a 50-layers Residual Network (ResNet) [48] was used with weights pre-trained on the ImageNet dataset. ResNet-50 is composed of a set of residual blocks stacked on top of the input layer and followed by an FC layer. A block consists of a sequence of two convolution layers with filter sizes 1x1 and 3x3, respectively, where a stride of 2 is used by the first convolution layer to reduce feature map size. ResNet-50 provides a good trade-off between training time and performance. Generally, the first few layers of the network capture low-level features of the image like edges, curves, etc., where subsequent layers learn shapes and more abstract features. The final layers have learned more specific features corresponding to a particular problem in-hand. The pre-trained ResNet-50 network was fine-tuned on ROIs extracted from the CT scans. The network was fine-tuned in an iteratively increasing number of blocks starting from the FC layer, while the lower blocks of the network act as a fixed feature extractor. However, a block with its successive blocks was fine-tuned at a time, as elaborated in [26] . A schematic view of ResNet-50 architecture, labelled with the block numbers, can be seen in Figure 4 . The weights of the network were trained using Adam optimiser [49] with a variable learning rate which decays by a factor of 10 on every alternative epoch with an initial value of 0.01. The training process automatically ends in case of no improvement found in the cross-validation loss for consecutive 3 epochs. The weights were initialised with Xavier-initialisation [50] and a training batch-size of 16 was chosen. To classify a scan as COVID-19 positive or negative, the ratio of positive classified segments was compared against negative segments whereas the scan was declared positive when the ratio exceeds 0.5 threshold. The dataset was randomly divided into train, test, and validation sets with a ratio of 70%, 15%, and 15% respectively. The CAP category was excluded in the final training experiment. In order to thoroughly evaluate the proposed approach, a number of experiments have been carried-out by fine-tuning one layer/block per training cycle. Table 2 summarises the number of positive and negative images in each set. The network's hyper-parameters recommended by the Meta-RL [27] are listed in Table 3 . The next step of the proposed pipeline is the quantification of the disease. Not all patients diagnosed with positive COVID-19 will need intensive care. Accordingly, [51] classified COVID-19 into four severity levels including minimal, common, severe and critical. The minimal and common severity patients both show clinical symptoms however only in the common severity level the chest CT scan shows opacity in the lung. The severe cases have signs of either respiratory distress, low blood oxygen saturation, or partial pressure of arterial blood oxygen [52] . The critical patients are reported with organs and respiratory failure needing ventilation and intensive care. The CT diagnosis and quantification of the disease could be very useful for patients falling under common or severe severity levels. The severity score is the sum of the area of all the infectious segments that are found in the positively classified scans. Therefore, the gradient-weighed class activation maps (CAMs) is employed for detecting characteristic features from the image [53] . The feature maps of the last convolution layer retain high enough spatial information which can be used for coarse localization of the pathological regions, whose size can, in turn, be related to the disease severity. Algorithm 1 Disease severity quantification algorithm 1: A k ∈ R h×w ∵ k = 1, ..., N A is activation maps of the last convolutional layer, h and w are height and width of a feature map, and N represents total number of feature maps 2: dy c dAij weights of the class c scores with respect to feature maps of A ReLU is applied to features of positive influence on c The disease severity quantification algorithm is outlined in Algorithm 1 which has been computed for each segment of a scan for which the model has classified infected. The quantification value Q for all the segments of a scan has been summed up which is elaborated in Figure 5 . The assessment of the treatment is the post-processing phase of the approach which comes after diagnosis and quantification of disease. The quantification score is further used for patient-specific monitoring of the disease progression during the course of treatment. The progression is monitored by the severity score S which normalizes the quantification score of a particular day by the initial score, see Equation 1 . The severity score is extrapolated to forecast the condition of the critical patients which allows the hospitals to plan ahead for the crucial resources like medical staff, ventilators and intensive care units (ICUs). The disease progression is plotted over time since the beginning of treatment along with prior medical history and medication prescribed during the treatment. Since there is no vaccine available for COVID-19, this approach correlates the progression of the disease with prescribed medicines. Such analysis can help physicians to assess the effectiveness of medication on patients and come up with a pattern of the patient's severity level and medication. fully-connected (FDA) granted permission to use some general medicines, mentioned earlier, for the treatment of severely affected COVID-19 patients. Those medicines are correlated with the progression to evaluate their effectiveness along with the clinical data and prior medical history of the patient to predict the most suitable set of medicines for the new patients [54] . The COVID-19 prognostics has been demonstrated in Figure 5 . A comprehensive set of experiments has been conducted to evaluate the effectiveness of the proposed approach. These experiments evaluate various models which are fine-tuned on a different combination of the layers of a network. The experiments were performed on Nvidia Tesla v100 GPU. The disease classification is modelled using ResNet-50 architecture. The architecture takes less time to train a model as compared to the other architectures such as Inception-v3 [55] and Inception-ResNet-v2. The network was fine-tuned in an iteratively increasing number of blocks starting from the final block, only one block was fine-tuned in a training cycle at a time. Table 4 summarises the performance of the block-wise network fine-tuning on hold-out test-set. The performance metrics consist of accuracy, precision, recall, and F1 score. The fine-tuning of the final FC layer gives the best classification with 93% accuracy, 92% precision, 90% recall, and 91% F1 score. These results were extracted from a training cycle consuming pre-processed segments of images. However, the fine-tuning of the rest of the blocks could outperform the current best accuracy in case of the availability of more training data. The precision and recall show promising results with a low false-negative rate. This paper presents an effective and efficient pipeline for diagnosis and triage of a suspected patient which is crucial to control COVID-19 and to provide better treatment of the infected patients. The proposed approach uses deep learning-based CT image classification techniques for the diagnosis of the disease. These techniques usually require large training datasets, however, the transfer learning technique was used to adapt a pre-trained model on the available CT scan data. In fact, deep learning techniques are highly sensitive which requires focusing on critical areas of the CT scan for precise diagnosis. Hence, a deep learning-based image segmentation technique was applied to the raw images as a pre-processing step to extract useful features from the image. The diagnosis of the disease from a CT scan takes significantly less time than the rRT-PCR test. Once a patient is admitted to the hospital for treatment based on the results of the diagnosis, the same process repeats periodically during the course of the treatment. Further, the deep learning classification model also quantifies the diseases by computing the area of GGO and CP which are identified by the model as infectious segments. This process helps physicians to monitor the progression of the disease over time from an early stage of the treatment. Moreover, in the absence of anti-viral medicine for COVID-19, a post-processing analysis correlates the general medication prescribed to a patient with the quantification of the disease severity score for better assessment of the treatment. The novelty of this work is the development of an end-to-end pipeline for timely and accurate diagnosis, quantification, progression analysis, and assessment of treatment of COVID-19. The effectiveness of the approach relies on the performance of the diagnosis both in terms of speed and accuracy. Hence, the Automatic Machine learning approach, such as MLL has been used to find the optimal number of network blocks that need to be fine-tuned. The final model gave an accuracy of 93% on the hold-out set with high precision and sensitivity. The disease diagnosis process is composed of two deep learning models including segmentation and classification. These models have millions of parameters across multiple layers/blocks which makes the modelling process hard to explain and less transparent. Perhaps, only technical experts can interpret these models in the context of the transformations and operations that the data goes through. However, for the end-users, mostly high-level performance measures are the only parameter to trust in the working of the models. It would become critical when the end-user of these models belongs to the domain of medical science who needs to make a decision based on very abstract insights. Therefore, the prognosis of the disease can be further enhanced by interpreting the following areas of diagnosis: 1. Data Audit: What if a model is overfitted due to class biases and gives high performance? A detailed data audit that covers the class distribution along with other information would be useful for the end-user. (a) Along with the data audit, explainability of the black box deep learning models by demonstrating how a model processed a scan by interpreting each layer/block of the network? (b) What kind of examples does the model perform poorly on? (c) Can a prediction be attributed to adversarial behavior, or to undesirable priors in the training set? 3. Quantification of the disease: For a COVID-19 positive case, how critical is the condition of the patient. One possible solution could be precise quantification of the disease by volumetric measurement of multiple patchy areas of GGO and shadows of the CP of the lungs. 4. Localization of Disease: The model's outcome of COVID-19 is based on the patches which actually belong to COVID-19 or CAP. This can be interpreted by visualizing the final feature maps of the network to discriminative regions of the scan. This can be achieved by localization of disease using Partial Dependence Plots (PDPs) [56] , Gradient-weighted Class Activation Mapping (Grad-CAM) [57] and saliency mapping [58] . 5. Relating insights of scan and clinical data: What value patient's clinical data, demographics, oxygen saturation level, etc. can add along with the diagnosis of the scan towards explainability. A possibility is through the clustering of the clinical data and diagnosis to uncover insightful patterns. Coronavirus disease 2019 (covid-19) situation report -76 Correlation of chest ct and rt-pcr testing in coronavirus disease 2019 (covid-19) in china: A report of 1014 cases The spectrum of severe acute respiratory syndrome-associated coronavirus infection Sensitivity of chest ct for covid-19: Comparison to rt-pcr Review of artificial intelligence techniques in imaging data acquisition, segmentation and diagnosis for covid-19 Real-time rt-pcr in covid-19 detection: issues affecting the results Rapid ai development cycle for the coronavirus (covid-19) pandemic: Initial results for automated detection and patient monitoring using deep learning ct image analysis Diagnosis of the coronavirus disease (covid-19): rrt-pcr or ct? Deep learning for screening covid-19 using chest x-ray images Combination of rt-qpcr testing and clinical features for diagnosis of covid-19 facilitates management of sars-cov-2 outbreak Chest ct findings of covid-19 pneumonia by duration of symptoms Sample-efficient deep learning for covid-19 diagnosis based on ct scans Handbook of covid-19 prevention and treatment Fda fact sheet: Serological testing for antibodies to sars-cov-2 infection. U.S. Food and Drug Administration Coronavirus disease 2019 (covid-19) situation report -41 Clinical and immunological assessment of asymptomatic sars-cov-2 infections Deep learning A deep learning algorithm using ct images to screen for corona virus disease (covid-19) A novel coronavirus from patients with pneumonia in china Epidemiological and clinical characteristics of 99 cases of 2019 novel coronavirus pneumonia in wuhan, china: a descriptive study An assessment of the safety of pediatric ibuprofen: A practitioner-based randomized clinical trial Efficacy and safety of ibuprofen and acetaminophen in children and adults: A meta-analysis and qualitative review Acute respiratory infection and use of nonsteroidal anti-inflammatory drugs on risk of acute myocardial infarction: A nationwide case-crossover study Dexamethasone in the management of covid-19 Imagenet large scale visual recognition challenge Towards meta-learning of deep architectures for efficient domain adaptation A meta-reinforcement learning approach to optimize parameters and hyper-parameters simultaneously Application of cad systems for the automatic detection of lung nodules Unet++: A nested u-net architecture for medical image segmentation. DLMIA, ML-CDS Improving the segmentation of anatomical structures in chest radiographs using u-net with an imagenet pre-trained encoder Unsupervised deep learning for bayesian brain mri segmentation Deep learning-based model for detecting 2019 novel coronavirus pneumonia on high-resolution computed tomography: a prospective study Evolution of ct findings in patients with mild covid-19 pneumonia Coronavirus disease 2019 (covid-19): Role of chest ct in diagnosis and management Covid-19 infection presenting with ct halo sign Spectrum of chest ct findings in a familial cluster of covid-19 infection Development and evaluation of an ai system for covid-19 diagnosis Adaptive feature selection guided deep forest for covid-19 classification with chest ct Using deep learning for classification of lung nodules on computed tomography images Covid-ct-dataset: a ct scan dataset about covid-19 A large dataset of real patients ct scans for covid-19 identification. Harvard Dataverse V1 Lung image database consortium Automatic classification of normal and cancer lung ct images using multiscale am-fm features Inf-net: Automatic covid-19 lung infection segmentation from ct images How transferable are features in deep neural networks? Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases Learning to read chest x-rays: Recurrent neural cascade model for automated image annotation Deep residual learning for image recognition Adam: A method for stochastic optimization Understanding the difficulty of training deep feed-forward neural networks National Health Commission NHC. The diagnosis and treatment of pneumonia infected by novel coronavirus Chest ct severity score: An imaging tool for assessing severe covid-19 Learning deep features for discriminative localization Early triage of critically ill covid-19 patients using deep learning Rethinking the inception architecture for computer vision Analysis of explainers of black box deep neural networks for computer vision: A survey Grad-cam: Visual explanations from deep networks via gradient-based localization Efficient saliency maps for explainable ai The authors wish to thank Dr. Muhammad Naeem medical specialist at Services Hospital for taking part in the subject discussion and review of this work.