key: cord-0273549-cpkjhlln authors: Nanivadekar, A.; Zirpe, K.; Dwivedi, A.; Patel, R.; Pant, R.; Gupte, T.; Lokwani, R.; Shende, D.; Kulkarni, V.; Kharat, A. title: An artificial intelligence system for predicting mortality in COVID-19 patients using chest X-rays: a retrospective study date: 2021-09-23 journal: nan DOI: 10.1101/2021.09.22.21263956 sha: 90d998e13758ebfca200ba3ae6416df715da8580 doc_id: 273549 cord_uid: cpkjhlln Background: Early prediction of disease severity in COVID-19 patients is essential. Chest X-ray (CXR) is a faster, widely available, and less expensive imaging modality that may be useful in predicting mortality in COVID-19 patients. Artificial Intelligence (AI) may help expedite CXR reading times, and improve mortality prediction. We sought to develop and assess an artificial intelligence system that used chest X-rays and clinical parameters to predict mortality in COVID-19 patients. Methods: A retrospective study was conducted in Ruby Hall Clinic, Pune, India. The study included patients who had a positive real-time reverse transcriptase-polymerase chain reaction (RT-PCR) test for COVID-19 and at least one available chest X-ray at the time of their initial presentation or admission. Features from CXR images and clinical parameters were used to train the Random Forest model. Results: Clinical data from a total of 201 patients was assessed retrospectively. The average age of the cohort was 51.4 {+/-} 14.8 years, with 29.4% of the patients being over the age of 60. The model, which used CXRs and clinical parameters as inputs, had a sensitivity of 0.83 [95% CI: 0.7, 0.95] and a specificity of 0.7 [95% CI: 0.64, 0.77]. The area under the curve (AUC) on receiver operating characteristics (ROC) was increased from 0.74 [95% CI: 0.67, 0.8] to 0.79 [95% CI: 0.72, 0.85] when the model included features of CXRs in addition to clinical parameters. Conclusion: An Artificial Intelligence (AI) model based on CXRs and clinical parameters demonstrated high sensitivity and can be used as a rapid and reliable tool for COVID-19 mortality prediction. During the current COVID-19 pandemic, there was a sudden surge in demand for diagnostic tools, exposing the lacunae in diagnostic infrastructure and reporting methods. There have also been instances of discrepancies in radiologists' interpretations and diagnostic errors (12) . Deep learning, a popular research area of Artificial Intelligence (AI), may help accelerate CXR reading times, making it an excellent candidate in the clinical delivery of patient care. Utilizing artificial intelligence, Chest X-rays can strengthen clinical data in predicting the severity of COVID-19 in patients (13) (14) . A recent study by Murphy et al. reported that an AI system could detect COVID-19 pneumonia on CXRs with the same accuracy as six independent radiologists (15) . However, the role of AI as a prognostic tool using CXRs in COVID- 19 patients has yet to be explored in detail. Only a few studies have integrated artificial intelligence into chest X-ray analysis to predict progression risk, demonstrating the advantage of chest X-rays over the clinical dataonly prediction models (16) (17) . Herein, we aimed to evaluate the association between CXR findings and clinical outcomes, as well as the use of an AI system as an early prognostic tool in detecting COVID-19. The study intends to introduce an AI based prognosis model that takes an X-Ray image and other clinical data as input and predicts the outcome to categorize the patient as high risk (death) or low-risk (discharge). This is significant because the outcome of COVID-19 is heavily influenced by patient demographics, as well as some key clinical factors. The study was conducted at Ruby Hall clinic, a tertiary care centre in Pune, Maharashtra, India. The data used for AI modeling included radiological findings on Chest X-rays (CXRs), and clinical findings and observations such as symptoms and comorbidities. The study was conducted at Ruby Hall Clinic, Pune, and no filters were used based on the demographic/geospatial or contact tracing information of the patient. In this retrospective investigation, only patients with a positive RT-PCR report and at least one X-ray imaging post-admission were included. The study did not include the patients' past and current hospital treatment procedures. Random Forest model was trained on the selected clinical and demographic parameters. Random forests are learning methods for classification tasks that operate by constructing a group of decision trees at training time, which helps in decision making. The end-to-end process involves pre-processing the data that involves converting raw data to data suitable for the machine learning model. For this study, the data was cleaned (details in the data preprocessing section) and the clinical parameters were reduced to the significant ones. Hyperparameter grid search was used for optimization to find the right parameters to get the best model performance. Hyper-parameters are the parameters that are not directly learnt by the algorithm and are tuned manually. If an optimal combination of hyper-parameters is set for the model, it optimizes better for specific metrics. Grid search optimization essentially provides a range of hyper-parameters to the model, which then attempts to find the optimal set by cross-validating on the entire dataset by attempting all possible combinations in different iterations of the algorithm. The threshold of 0.4 was used to classify the patient as being at high risk of mortality using the optimization algorithm and cross-validation on the entire dataset. This involves two steps: data cleaning and dimensionality reduction. Data was anonymized for any of the patient's personal or demographic details, apart from the patient's age and gender, as well as the date of the scan that were used for modeling. Entries for patients with missing or null values were removed. Due to skewness and the sensitivity of medical data, statistical techniques were not applied to fill up the missing entries. After removing the entries, separate features for symptoms were created, and manually pre-processed for human errors. Every symptom was treated as a separate parameter. All the binary entries for presence/absence of symptoms were replaced with boolean values (0/1). The co-morbidity column was also converted to boolean values, which represent their presence or absence. The All rights reserved. No reuse allowed without permission. perpetuity. preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in The copyright holder for this this version posted September 23, 2021. ; https://doi.org/10.1101/2021.09.22.21263956 doi: medRxiv preprint absolute difference between the date of onset of symptoms and the scan date was calculated and added as a feature. Clinically, if the patient has COVID-19 pneumonia in the lungs, it is likely to worsen if not treated promptly. Hence, the number of days between the X-ray and the onset of symptoms would presumably be useful in determining the outcome. We further added two features from the X-ray COVID-19 detection model: probability of prediction and area of the predicted mask with respect to the image size. The probability of prediction indicates the confidence with which the model has predicted COVID-19 in the Xrays. The area of the mask showed the severity of the COVID-19 pneumonia in the lungs. Both of these features were normalized to values in the range [0,100], where 0 = no lung involvement, and 100 = severe lung involvement. To avoid any discrepancies, we removed the patients with "admitted" as their outcome and kept a boolean output, where discharge was assigned 0 and death was assigned 1. The final data used for model building was that of 201 patients, with the prevalence of death being 8.9%. The number of parameters was reduced from 66 to 32 features using the above filtering and pre-processing criteria. The parameters were then verified for accuracy before training the model. We trained the Random Forest model on the dataset. Hyper-parameter grid search optimization was used on the whole set to finalize the algorithm's hyper-parameters. The algorithm was trained with Gini as the entropy, with a total of 100 estimators and a maximum depth of 2. Class weights were used to balance and penalize the classes based on the data skewness. We performed four-fold cross-validation on the entire set before calculating the aggregated results. The flowchart depicting the prediction of COVID-19 associated mortality by the model is represented in Figure 1 . All rights reserved. No reuse allowed without permission. perpetuity. preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in McNemar's test was used to compute the p-value for each demographic feature between patient subgroups. McNemar's test was also used to calculate the p value between survivors and non-survivors for each radiographic finding on the CXRs. p value < 0.05 was considered statistically significant. The 95% Confidence Interval (CI) was calculated using the Empirical Bootstrapping method. To assess the performance of the AI model, a receiver operating characteristic (ROC) curve was generated. The area under the ROC curve with 95% CI was calculated. Clinical data from 201 patients was analyzed retrospectively. There was a male preponderance (74.1% males vs. 25.9% females) among the patients, but there was no significant difference in the mortality rate between the two genders. The average age of the cohort was observed to be 51.4 ± 14.8 years, with 29.4 % of the patients being over the age of 60. There was a highly significant difference in mortality proportion (78.8% vs. 22.2%) between age groups > 60 and ≤60 years, respectively (p<0.0001). The vast majority of patients (60.2%) had associated comorbidities, and the proportion of cases with comorbidities in the mortality group was significantly higher (88.9% vs. 11.1%) than their non-comorbid All rights reserved. No reuse allowed without permission. perpetuity. preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in Table 2 . All rights reserved. No reuse allowed without permission. perpetuity. preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in The copyright holder for this this version posted September 23, 2021. ; https://doi.org/10.1101/2021.09.22.21263956 doi: medRxiv preprint and area of infection predicted by the X-ray AI model. The performance of the AI model is depicted in Table 3 . perpetuity. preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in The Random forest model was trained as described in materials and methods. Each variable was given an importance rank according to the tree statistics of the training set. The age of the patient was ranked among the top predictors of mortality in the cohort. This was followed by the AI prediction of the CXRs and co-morbidities amongst the patients. Different presenting symptoms within the cohort, such as dyspnea, vomiting, cough, and diarrhoea, were given boolean values, i.e., 0 for the absence and 1 for the presence of a particular symptom. The top 10 independent clinical variables associated with mortality are listed in All rights reserved. No reuse allowed without permission. perpetuity. preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in Table 4 . These clinical parameters describe the feature importance or dominance necessary to understand the decision-making process. A graphical representation of these features along with their percentage of importance is shown in Supplementary Figure 1 . (24) . AI-enabled telehealth contributes to quality improvement and enhancement of existing processes, as well as the implementation of new models of care (25) (26) , which are desperately needed in the overburdened Indian healthcare system. This study established a substantial role of AI in diagnostic radiology, and further implementation of AI in clinical imaging is poised to transform the practice of radiology. A relatively high sensitivity of our model makes it a rapid and reliable tool for COVID-19 mortality prediction, particularly in resource-constrained environments. Such AI models have the potential to be used as an initial screening tool for triaging COVID-19 patients, as well as to increase vigilance for severe cases. The usefulness of the model was successfully demonstrated in a tertiary care centre in India for prediction of COVID-19 mortality and can be easily adopted by imaging experts and clinicians. However, since the study is retrospective, it may result in an observer's bias. Because this was a single-centre study conducted in one of the referral tertiary care centres in India, it's likely that only moderate to severe cases were included. Further studies with multicentric data may reduce the selection bias and improve the sensitivity of the model with greater precision. All rights reserved. No reuse allowed without permission. perpetuity. preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in The copyright holder for this this version posted September 23, 2021. perpetuity. preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in COVID-19: The first documented coronavirus pandemic in history Weekly epidemiological update on COVID-19 -14 India-situation-report-84.pdf Learning from Rapid Response, Innovation, and Adaptation to the COVID-19 Crisis: Proceedings of a Workshop-in Brief Correlation of Chest CT and RT-PCR Testing for Coronavirus Disease 2019 (COVID-19) in China: A Report of 1014 Cases Performance of Radiologists in Differentiating COVID-19 from Non-COVID-19 Viral Pneumonia at Chest CT Artificial Intelligence Augmentation of Radiologist Performance in Distinguishing COVID-19 from Pneumonia of Other Origin at Chest CT | Radiology The Role of Chest Imaging in Patient Management During the COVID-19 Pandemic: A Multinational Consensus Statement From the Fleischner Society COVID-19) infection: findings and correlation with clinical outcome The role of initial chest X-ray in triaging patients with suspected COVID-19 during the pandemic Errors, discrepancies and underlying bias in radiology with case examples: a pictorial review. Insights Imaging Prognostication of patients with COVID-19 using artificial intelligence based on chest x-rays and clinical data: a retrospective study Artificial intelligence in radiology COVID-19 on Chest Radiographs: A Multireader Evaluation of an Artificial Intelligence System Automated assessment of COVID-19 pulmonary disease severity on chest radiographs using convolutional Siamese neural networks COVID-Classifier: An automated machine learning model to assist in the diagnosis of COVID-19 infection in chest x-ray images Differences between First wave and Second wave of COVID-19 in India Lower mortality of COVID-19 by early recognition and intervention: experience from Jiangsu Province. Ann Intensive Care Frequency and Distribution of Chest Radiographic Findings in Patients Positive for COVID-19 Imaging Profile of the COVID-19 Infection: Radiologic Findings and Literature Review Initial chest radiographs and artificial intelligence (AI) predict clinical outcomes in COVID-19 patients: analysis of 697 Italian patients Telemedicine market in India to reach USD 5.5 bn by 2025: EY-IPA study Role of Artificial Intelligence within the Telehealth Domain The future of radiology augmented with Artificial Intelligence: A strategy for success All rights reserved. No reuse allowed without permission. perpetuity preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in The copyright holder for this this version posted