key: cord-0851464-4lgqq01g authors: Tezza, Fabiana; Lorenzoni, Giulia; Azzolina, Danila; Barbar, Sofia; Leone, Lucia Anna Carmela; Gregori, Dario title: Predicting in-Hospital Mortality of Patients with COVID-19 Using Machine Learning Techniques date: 2021-04-24 journal: J Pers Med DOI: 10.3390/jpm11050343 sha: 0366579a06f45d37b6931e4cbd50370c1ebd188c doc_id: 851464 cord_uid: 4lgqq01g The present work aims to identify the predictors of COVID-19 in-hospital mortality testing a set of Machine Learning Techniques (MLTs), comparing their ability to predict the outcome of interest. The model with the best performance will be used to identify in-hospital mortality predictors and to build an in-hospital mortality prediction tool. The study involved patients with COVID-19, proved by PCR test, admitted to the “Ospedali Riuniti Padova Sud” COVID-19 referral center in the Veneto region, Italy. The algorithms considered were the Recursive Partition Tree (RPART), the Support Vector Machine (SVM), the Gradient Boosting Machine (GBM), and Random Forest. The resampled performances were reported for each MLT, considering the sensitivity, specificity, and the Receiving Operative Characteristic (ROC) curve measures. The study enrolled 341 patients. The median age was 74 years, and the male gender was the most prevalent. The Random Forest algorithm outperformed the other MLTs in predicting in-hospital mortality, with a ROC of 0.84 (95% C.I. 0.78–0.9). Age, together with vital signs (oxygen saturation and the quick SOFA) and lab parameters (creatinine, AST, lymphocytes, platelets, and hemoglobin), were found to be the strongest predictors of in-hospital mortality. The present work provides insights for the prediction of in-hospital mortality of COVID-19 patients using a machine-learning algorithm. The prognosis prediction of patients infected with SARS-CoV-2 is one of the most relevant topics in COVID-19 research [1] . The pandemic poses a severe burden to the healthcare system of countries worldwide given the long symptoms' duration [2] , the higher risk of severe complications requiring hospitalization/intensive care compared to seasonal influenza [3] , and the documented excess of mortality associated with the virus spread [4] . Such conditions have led to a dramatic imbalance between healthcare resources and hospital/Intensive Care Unit (ICU) bed demands [5] . Italy, the first and most affected European country during the first COVID-19 wave [6] , adopted emergency solutions to face such imbalance [5] . However, the COVID-19 wave is still ongoing in almost all European countries, including Italy. For these reasons, there is an urgent need for strategies to prevent such an imbalance occurs again. With the adoption of public health strategies to contain the virus spread [7] and the improvement of the health care resources available, the development of tools to predict patients' prognosis would be part of the solution to prevent resource imbalance [8] . Such tools would be helpful to assist physicians' decision-making. Risk stratification is essential in this context to identify an adequate referral pathway for each COVID-19 patient, allowing for an appropriate resource allocation. Furthermore, such tools help the identification of the parameters that contribute the most to the outcome, allowing physicians to monitor/treat such parameters. Several tools have been proposed from the beginning of the pandemic, especially for in-hospital mortality prediction [9] . A systematic review published at the beginning of the pandemic, i.e., April 2020, identified several mortality risk prediction tools for COVID-19 patients [9] , and several others have been proposed after that. Several methods have been employed to develop such tools, from traditional statistical approaches, e.g., multivariable analysis [10] , to more advanced machine learning techniques (MLTs) [11] [12] [13] [14] . MLTs are increasingly used for outcome prediction in the clinical setting, since they present several advantages over traditional methods [15] . They help disentangle complex relationships between covariates and outcome of interest, even though a low number of events have occurred in front of many variables to be tested. Not least, the MLT predictive ability may improve as new data is provided. Our study aims to identify predictors of COVID-19 in-hospital mortality testing a set of MLTs, comparing their ability to predict the outcome of interest. The model estimation is based on patients treated in a COVID-19 referral center in the Veneto region, Italy, during the COVID-19 first wave. Together with Lombardia, Veneto was the first Italian region affected by the COVID-19 outbreak at the beginning of the year [16] . The study involved patients with COVID-19, proved by PCR test, admitted to the "Ospedali Riuniti Padova Sud" COVID-19 referral center in the Veneto region, Italy, in March and April 2020. Information on sociodemographic characteristics; medical history (comorbidities, smoking habits, and weight status); clinical and instrumental parameters; and laboratory data were collected at hospital admission. Furthermore, drug therapy was registered. The outcome was represented by in-hospital mortality. Descriptive statistics were reported as I quartile/median/III quartile for continuous variables and percentages (absolute numbers) for categorical variables. Wilcoxon-type tests were performed for the continuous variables and the Pearson chi-square test, or Fisher-exact test, whichever is appropriate, for the categorical variables. A set of MLTs was tuned to identify the predictors of in-hospital mortality. The predictive tools were tuned using a bootstrapping algorithm (50 runs). The algorithms considered for the classification task were the Recursive Partition Tree (RPART), the Support Vector Machine (SVM), the Gradient Boosting Machine (GBM), and Random Forest. The RPARTs are classification models based on a top-down methodology in which, starting from a root node, binary splits of data are generated until a certain criterion (i.e., the minimization of the node impurity) is encountered [17] . This method is prone to overfitting on training data. The cross-validation or bootstrapping procedure is a useful method to limit the overfitting, leading to defining the proper tuning of the Decision Tree (DT) parameters and optimizing the model accuracy [18] . • Random Forest is a tree-based algorithm, which involves the computation of hundreds to thousands of RPART trees; the method merges the DT output to increase the model's generalizability [19] . The GBM is based on a sequential boosting improvement of weak RPART classifiers (high bias and low variance). The GBM idea is to add a classifier sequentially so that the next classifier is trained to improve the already trained RPART. A random forest algorithm, instead, trains each classifier independently from the others [19] . The SVM algorithm's main objective is to find an optimal hyperplane of the feature's N-dimensional space (N-the number of variables) that distinctly classify the data points into a binary partition [20] . Several hyperplanes may separate the classes of data points. The SVM algorithm considers the hyperplanes, which maximize the margin (the distance between data points of classes). The resampled performances were reported for each MLT, considering the sensitivity, specificity, and the Receiving Operative Characteristic curve (ROC) measures, with 95% confidence intervals (C.I.s). Among the proposed MLTs, the tool with the best performance was chosen for the classification task. c, pairwise differences were computed and tested to assess if the difference was equal to zero. The Bonferroni correction was used to calculate the p-values and adjust the confidence interval limits for the reported differences. For the most promising MLT, the variable importance plot was reported together with the ROC curve and the median balanced accuracy measure within the resampling. The ROC and accuracy measures were also reported to evaluate the MLT tool's performance according to the increasing data availability during the study period. The database was ordered according to the hospitalization date. The MLT was built on the first 40, 60, 80,100, 120, 140, 160, 180, 200, 220, 240, 260, 280, 300, 320, and 340 hospitalized patients. The ROC and accuracy measures were reported according to the different sample sizes. The performance was computed, together with the variable importance measures, by tuning and developing the tool for cumulative fractions of the sample. A Shiny web application was developed based on the algorithm with the best performance. The tool calculates the in-hospital death probability, according to the patients' characteristics. Analyses were performed using R 3.4.2 [21] with the rms [22] and caret [23] packages. The study enrolled 341 patients admitted to the "Ospedali Riuniti Padova Sud" COVID-19 referral center between March and April 2020. Table 1 presents the subjects' baseline characteristics according to the admission date (before-after 21 March 2020, the day on which the national lockdown was declared in Italy). Seventy-five patients out of 341 died while hospitalized. Overall, the prevalence of the male gender was found to be higher than the female one (57% in the sample overall), even though it was found to be higher in the first period of admissions compared to the second one (65% vs. 51%; p-value 0.007). The median age of the subjects enrolled was 74 years, and it was lower in the first period compared to the second period (median of 70 vs. 78; p-value < 0.001). Subjects admitted to the hospital in the first period were significantly more likely to suffer from diabetes (p-value 0.031) and less likely to suffer from dementia (p-value < 0.001) and from cerebrovascular diseases (p-value 0.019). No significant differences were detected in the distribution of the other comorbidities. For what concerns the drug therapy, subjects admitted after March 21 were found to be significantly more likely to be treated with low molecular weight heparin (LMWH) (p-value 0.001) but less likely to be administered with antiviral drugs. The Random Forest algorithm outperformed the other MLTs in predicting the inhospital mortality (Figure 1) , with a ROC of 0.84 (95% C.I. 0.78-0.9). In addition to that, looking at the values of sensitivity and specificity, Random Forest provided a better balance than the other techniques between two such measures. Such results were confirmed when the pairwise comparisons were made (Table S1 Supplementary Materials). Furthermore, no class imbalance issues were detected ( Figure S1 , Supplementary Materials). Since Random Forest was found to be the MLT with the best performance, it was chosen to identify the predictors of in-hospital mortality. The algorithm was tuned to achieve the optimal performance in the correspondence of 500 trees and 6 mtry (number of variables available for splitting each tree node). Figure 2 reports the plots of the variable importance measures according to the mean decrease accuracy. Age, together with vital signs (oxygen saturation and the quick SOFA) and lab parameters (creatinine, AST, lymphocytes, platelets, and hemoglobin), were found to be the strongest predictors of in-hospital mortality (Figure 2) . Conversely, comorbidities were found to provide only a small contribution in predicting the in-hospital mortality in such patients. Furthermore, the algorithm stability investigation as the availability of data increases was performed to inform clinicians about the minimum number of cases required to obtain a reliable algorithm. Interestingly, the ROC parameter underwent stabilization after the first 100 patients (Figure 3 ). The variable importance plots calculated on cumulative fractions of the sample are provided in Table S2 (Supplementary Materials). The web application based on the algorithm developed is available at https://r-ubesp. dctv.unipd.it/shiny/Schiavonia/. The present study showed that the Random Forest is a feasible machine-learning algorithm to stratify the mortality risk in a sample of patients admitted to a COVID-19 referral center in the Veneto region, Italy, during the first COVID-19 wave. The sample characteristics, in terms of gender and comorbidities distribution, are consistent with those reported by previous studies on subjects admitted to the Italian hospitals during the first wave of the pandemic [24] , even though the median age of the patients enrolled was about ten years higher than that previously reported [24] . For what concerns drug therapy, the increasing use of LMWH documented between the first and second study periods reflects the publication of the preliminary evidence about the beneficial effects of LMWH in patients with COVID-19 [25] . Risk stratification is a hot topic in COVID-19 research, given the shortage of healthcare resources for the management of COVID-19 patients. For this reason, an impressive number of risk stratification tools for in-hospital mortality in COVID-19 patients has been developed. Both traditional, i.e., logistic regression, and nontraditional, i.e., MLTs, statistical techniques have been employed to create such tools [9] , but the traditional ones have been most frequently used, probably for the ease of use and interpretability in the clinical setting. However, MLTs have been shown to be feasible to develop risk stratification tools in the COVID-19 research setting [11, 13] , with promising results The present study showed that the Random Forest is a feasible machine-learning algorithm to stratify the mortality risk in a sample of patients admitted to a COVID-19 referral center in the Veneto region, Italy, during the first COVID-19 wave. The sample characteristics, in terms of gender and comorbidities distribution, are consistent with those reported by previous studies on subjects admitted to the Italian hospitals during the first wave of the pandemic [24] , even though the median age of the patients enrolled was about ten years higher than that previously reported [24] . For what concerns drug therapy, the increasing use of LMWH documented between the first and second study periods reflects the publication of the preliminary evidence about the beneficial effects of LMWH in patients with COVID-19 [25] . Risk stratification is a hot topic in COVID-19 research, given the shortage of healthcare resources for the management of COVID-19 patients. For this reason, an impressive number of risk stratification tools for in-hospital mortality in COVID-19 patients has been developed. Both traditional, i.e., logistic regression, and nontraditional, i.e., MLTs, statistical techniques have been employed to create such tools [9] , but the traditional ones have been most frequently used, probably for the ease of use and interpretability in the clinical setting. However, MLTs have been shown to be feasible to develop risk stratification tools in the COVID-19 research setting [11, 13] , with promising results underlying such algorithms' potentials to assist clinicians' decision-making in everyday clinical practice. The present work showed an outperformance of Random Forest compared to the other algorithms tested. Interestingly, the variables found to contribute to in-hospital mortality were age and those related to hospital presentation (vital signs and lab parameters). At the same time, the comorbidities played a minor role in the mortality risk prediction. Such findings are only partially in-line with those of international studies, showing that, together with clinical presentation, comorbidities also play a relevant role in risk prediction [11] . However, it is worth pointing out that it is difficult to compare results from the literature, since each tool has been developed on different patient populations, e.g., hospitalized patients, mechanically ventilated patients, elderly patients hospitalized, and the general population. Furthermore, each tool has been developed using a different set of baseline variables, including sociodemographic ones, laboratory data, instrumental parameters, and clinical measures. The present work provides a useful tool able to assist physicians' decision-making in facing the COVID-19 emergency. Furthermore, the feasibility of MLTs in this research context has been shown, which represents an added value of the present work, since MLTs are able to overcome the limitations of more traditional statistical techniques. Supplementary Materials: The following are available online at https://www.mdpi.com/article/10 .3390/jpm11050343/s1: Table S1 . Differences in ROC (Panel A) and F statistic (Panel B) among the MLTs employed to predict in-hospital death. Table S2 : Variable importance plots for each Random Forest developed. Figure S1 . ROC, sensitivity, and specificity measures of the null model and the random forest. Institutional Review Board Statement: Review and approval were waived for this study, due to the retrospective nature of the study based on anonymized data. Informed Consent Statement: Patient consent was waived, since only anonymized data was collected. The data presented in this study are available upon request from the corresponding author. The authors declare no conflict of interest. Prediction models for covid-19 outcomes Symptom Duration and Risk Factors for Delayed Return to Usual Health Among Outpatients with COVID-19 in a Multistate Health Care Systems Network-United States Risk for In-Hospital Complications Associated with COVID-19 and Influenza-Veterans Health Administration Temporal dynamics in total excess mortality and COVID-19 deaths in Italian cities Critical Care Utilization for the COVID-19 Outbreak in Lombardy, Italy: Early Experience and Forecast During an Emergency Response COVID-19 and Italy: What next? A first estimation of the impact of public health actions against COVID-19 in Veneto (Italy) Laboratory Findings Associated With Severe Illness and Mortality Among Hospitalized Individuals With Coronavirus Disease 2019 in Eastern Massachusetts Prediction models for diagnosis and prognosis of covid-19: Systematic review and critical appraisal Development and external validation of a prediction risk model for short-term mortality among hospitalized U.S. COVID-19 patients: A proposal for the COVID-AID risk tool Machine learning prediction for mortality of patients diagnosed with COVID-19: A nationwide Korean cohort study Clinical features of COVID-19 mortality: Development and validation of a clinical prediction model Machine learning based early warning system enables accurate mortality risk prediction for COVID-19 Machine Learning to Predict Mortality and Critical Events in a Cohort of Patients With COVID-19 in New York City: Model Development and Validation Big Data and Machine Learning Algorithms for Health-Care Delivery Is a More Aggressive COVID-19 Case Detection Approach Mitigating the Burden on ICUs? Some Reflections from Italy Classification and Regression Trees The Elements of Statistical Learning: Data Mining, Inference, and Prediction Support-vector networks R: A Language and Environment for Statistical Computing; R Foundation for Statistical Computing Regression Modeling Strategies Building Predictive Models in R Using the caretPackage Baseline Characteristics and Outcomes of 1591 Patients Infected with SARS-CoV-2 Admitted to ICUs of the Lombardy Region The Versatile Heparin in COVID-19