key: cord-0918452-0pbzz1nv authors: Fernandes, Fernando Timoteo; de Oliveira, Tiago Almeida; Teixeira, Cristiane Esteves; Batista, Andre Filipe de Moraes; Dalla Costa, Gabriel; Chiavegatto Filho, Alexandre Dias Porto title: A multipurpose machine learning approach to predict COVID-19 negative prognosis in São Paulo, Brazil date: 2021-02-08 journal: Sci Rep DOI: 10.1038/s41598-021-82885-y sha: 1b415a4626ad7b250598e517876ea417e51b66d7 doc_id: 918452 cord_uid: 0pbzz1nv The new coronavirus disease (COVID-19) is a challenge for clinical decision-making and the effective allocation of healthcare resources. An accurate prognostic assessment is necessary to improve survival of patients, especially in developing countries. This study proposes to predict the risk of developing critical conditions in COVID-19 patients by training multipurpose algorithms. We followed a total of 1040 patients with a positive RT-PCR diagnosis for COVID-19 from a large hospital from São Paulo, Brazil, from March to June 2020, of which 288 (28%) presented a severe prognosis, i.e. Intensive Care Unit (ICU) admission, use of mechanical ventilation or death. We used routinely-collected laboratory, clinical and demographic data to train five machine learning algorithms (artificial neural networks, extra trees, random forests, catboost, and extreme gradient boosting). We used a random sample of 70% of patients to train the algorithms and 30% were left for performance assessment, simulating new unseen data. In order to assess if the algorithms could capture general severe prognostic patterns, each model was trained by combining two out of three outcomes to predict the other. All algorithms presented very high predictive performance (average AUROC of 0.92, sensitivity of 0.92, and specificity of 0.82). The three most important variables for the multipurpose algorithms were ratio of lymphocyte per C-reactive protein, C-reactive protein and Braden Scale. The results highlight the possibility that machine learning algorithms are able to predict unspecific negative COVID-19 outcomes from routinely-collected data. Algorithms performance. We analyzed the predictive performance of the algorithms for three negative prognostic outcomes: ICU admission (n = 263, 25.5%), mechanical ventilation (MV) intubation (n = 106, 10.2%) and death (n = 92, 9.4% ). First, we tested the predictive performance of the machine learning algorithms for a specific individual outcome (e.g. death) to get a baseline for comparison. Then, we used observations from patients who had the other two outcomes (in this specific example, mechanical ventilation and ICU admission) to train an aggregated model. In the aggregated model, we tested the performance when predicting the severe outcome not included in training (e.g. death). Finally, we compared the performance of the two strategies (e.g. individual against aggregated models) using the 95% confidence interval of the area under the receiver operating characteristic curve (AUROC). Table 2 shows the results of the models trained with the aggregated outcomes and the models with a single outcome. Every model, even the ones trained with different outcomes, presented high predictive performance, always with an AUROC over 0.91 in the test set. The individual models presented better AUC compared to the aggregated models when predicting ICU, MV or death with AUROC over 0.959, 0.945 and 0.972 respectively. Despite the individual models being overall better, the difference between the aggregated and individual models were all within the 95% confidence intervals. Supplementary Fig. 1 shows the AUROC for each model. The sensitivity and specificity of the machine learning algorithms were also very high, in most cases over 0.8, with an average sensitivity of 0.92 and specificity of 0.82. The positive predictive values (PPV) for the aggregated models were higher than the individual models when predicting mechanical ventilation and ICU, reaching 0.398 and 0.729 respectively, while for death there was a decrease to 0.290. This means that two out of three of the aggregated models had higher PPV when predicting which patients would develop severe illness and require hospital resources than the individual models. In Supplementary Table 2 we present the final hyperparameters for each model. Figure 1 presents the prediction density for each individual outcome according to the different training strategies. The results point to a low overlap between negative and positive cases, indicating a good discriminative ability of the algorithms irrespective of the training strategy. Figure 2 presents the top five variables that most contributed to predict a severe outcome in the aggregated models, according to the Shapley values. The variables are ranked according to the contribution for each specific algorithm. The Braden score played an important role in the aggregated outcome algorithms, ranking as the most important predictor in two of the three models. Also, the C-reactive protein and ratio of lymphocytes per C-reactive protein were found to be good predictors, appearing in the top five in all three models. Urea, age, creatinine, and arterial lactate were important for only one of the aggregated models. Previous studies have used machine learning to develop early COVID-19 prognostic models for a specific severe outcome with overall good performance 21,23 , frequently reaching over 0.90 AUROC 26 . We used a different approach, by combining severe outcomes to train algorithms to predict another outcome, in order to test its potential for predicting multiple untrained outcomes. We found that machine learning algorithms were able to predict negative prognostic outcomes with high overall performance for COVID-19, even when the specific outcome was not included in the training of the algorithms. All models presented an AUROC higher than 0.91 (average of 0.92) in the test set, with high sensitivity www.nature.com/scientificreports/ and specificity (average of 0.92 and 0.82, respectively). The results highlight the possibility that high-performance machine learning algorithms are able to predict unspecific negative COVID-19 outcomes using routinely-collected data. The development of multipurpose prognostic algorithms, i.e. algorithms that identify nonspecific outcomes and overall future clinical deterioration, can be used in a large number of situations, especially in the case of complex and unknown diseases that lead to the development of several different negative outcomes. Instead of having to develop a different algorithm for each of the specific outcomes, multipurpose models can provide more comprehensive and clinically relevant information about the risks of future health problems of patients. The algorithms can be embedded in an app for smartphones or in electronic medical records to be used with routinely-collected data to perform simple predictions for each incoming patient, thus supporting screening procedures and decision-making. In the case of developing countries, while the issue of current availability of electronic medical records in poorer areas is still a challenge, in Brazil there have been promising recent advances regarding the use of electronic medical records 27 . Brazil is currently the third country in the world in total number of cases and second in deaths from COVID-19 28 . There is a growing demand in Brazil, and in many other developing countries, for decision support in the allocation of scarce hospital resources, especially in relation to the availability of ICU beds and mechanical ventilators 29, 30 . From a clinical standard, knowledge about immediate risks of negative prognosis can also contribute to the early start of preventive measures and new interventions, and thereby increase patient survival 5, 6 . For every outcome, variable importance analysis identified that age, C-reactive protein (CRP), creatinine, urea and the Braden Scale were usually among the most important. While the age of the patient is widely found to be an important predictor for most negative health outcomes, CRP has been increasingly included among the main inflammatory biomarkers for the prognosis of cardiovascular 31 and respiratory diseases 32 . High levels of CRP have been also previously associated with individual severity of SARS-CoV-2 33, 34 . Interestingly, previous studies have also identified that chronic kidney disease is associated with developing severe conditions in COVID-19 patients [35] [36] [37] , where it has been observed that patients with higher levels of creatinine and urea are more at risk 38 . The Braden Scale is often used as a predictor for pressure ulcers, a common clinical classification scale for predicting pneumonia 39 during clinical reception, and in this study, it was an important predictor for negative prognosis in COVID-19 patients. The scale has a score between 1 (worst score) and 4 (best score) where the factors included are sensory perception, skin moisture, activity, mobility, nutritional status and friction 40 . The percentage of lymphocytes in the blood has been described as a strong predictor of prognosis for the severity of the new coronavirus. A randomized study by Tan et al. 41 suggested that, in most confirmed cases, the percentage of lymphocytes was reduced to 5% in 2 weeks after the onset of COVID-19, in line with other studies findings 42 . The study has a few limitations that need to be mentioned. First, some of the outcomes overlap, which may have helped the performance of the aggregated models, even though in the majority of cases the outcomes were independent. In the case of ICU admission, 55% of the patients did not die or used MV, while in the case of MV and death, 63% and 70% of their respective aggregated model was trained on other outcomes. Ideally, the outcomes would never overlap, but this is clinically unfeasible given the interlaced nature of negative prognostic outcomes. Another limitation is that we analyzed data from an urban COVID-19 hotspot in Brazil, in a period where clinical protocols for the disease were still being established, so this could affect the incidence of prognostic outcomes and may not directly generalize to other periods. In conclusion, we found that machine learning algorithms can predict severe outcomes in COVID-19 patients with high performance, including previously unobserved outcomes, using only routinely-collected laboratory, clinical and demographic data. The use of multipurpose algorithms for the prediction of overall negative prognosis is a promising new area that can support doctors with clinical and administrative decisions, especially regarding priorities for hospital admission and monitoring. 43 . Individual patient data was collected from electronic medical records. We included as predictors only variables collected in early hospital admission, i.e. within 24 h before and 24 h after the RT-PCR exam. From a total of 82 routinely-collected variables from the hospital, 57 variables were selected for the development of the predictive models, after removing variables with 90% or higher missing values, highly-correlated variables (above 0.9) and identifying variables such as patient number and hospital identification variables. The flowchart for feature selection is described in Supplementary Fig. 2 and the complete variable list, including demographic data, laboratory tests and vital signs is described in Supplementary Table 1. Figure 3 illustrates the overall process. Five of the most popular machine learning models for structured data (artificial neural networks 44 , extra trees 45 , random forests 46 , catboost 47 , and extreme gradient boosting 48 ) were trained with 70% of the data, and tested in the other 30%, simulating new unknown data. All the results reported in this study are from the test set. K-fold cross-validation with 10 folds was used to adjust the hyperparameters with Bayesian optimization (HyperOpt). Due to the unbalanced nature of the outcomes, random undersampling was performed in the training set, by randomly selecting examples from the majority class for exclusion. This technique was implemented using the RandomUnderSampler imbalanced-learn class 49 . Variables with more than two categories were represented by a set of dummy variables, with one variable for each category. Continuous variables were standardized using the z-score. Variables with a correlation greater than 0.90 (mean arterial pressure, total bilirubin, and creatine kinase) were discarded, and missing values were imputed by the median. To assess the performance of the models, measures such as accuracy, sensitivity (also known as recall), specificity, positive predictive value (PPV) (also known as precision), negative predictive value (NPV), and F1 score were analyzed. The value of the AUROC was used to select the best model. To understand the individual contribution of each variable to the predictive models, we calculated their respective Shapley values. All the analyzes were performed using the Python programming language with the scikit-learn library. The data comes from electronic medical records from BP-A Beneficência Portuguesa de São Paulo Hospital in Brazil and it is not publicly available as it contains sensitive information of patients. All the code written to process and analyze the data can be made available upon request to the corresponding author. Received: 19 August 2020; Accepted: 14 January 2021 The algorithm was trained and tested using a combination of two outcomes. The same algorithm was then used to predict the remaining outcome. The COVID-19 pandemic: Effects on low-and middle-income countries Evidence synthesis communities in low-income and middle-income countries and the COVID-19 response The impact of COVID-19 and strategies for mitigation and suppression in low-and middle-income countries COVID-19: Intensive care units, mechanical ventilators, and latent mortality profiles associated with case-fatality in Brazil Using machine learning to predict ICU transfer in hospitalized COVID-19 patients COVID-19: Immunopathology and its implications for therapy Evolution and epidemic spread of SARS-CoV-2 in Brazil The COVID-19 pandemic in Brazil: Analysis of supply and demand of hospital and ICU beds and mechanical ventilators under different scenarios COVID-19-The availability of ICU beds in Brazil during the onset of pandemic Demand for hospitalization services for COVID-19 patients in Brazil Epidemiological and clinical characteristics of the COVID-19 epidemic in Brazil Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: A retrospective cohort study The characteristics and clinical course of patients with COVID-19 who received invasive mechanical ventilation in Osaka Interim clinical guidance for management of patients with confirmed coronavirus disease (COVID-19) Clinical course and outcomes of critically ill patients with SARS-CoV-2 pneumonia in Wuhan, China: A singlecentered, retrospective, observational study Clinical course and outcomes of critically ill patients with COVID-19 infection: A systematic review D-dimer levels on admission to predict in-hospital mortality in patients with COVID-19 A predictive model and scoring system combining clinical and CT characteristics for the diagnosis of COVID-19 A fully automatic deep learning system for COVID-19 diagnostic and prognostic analysis Building a COVID-19 vulnerability index An interpretable mortality prediction model for COVID-19 patients COVID-19 diagnosis prediction in emergency care patients: A machine learning approach Early risk assessment for COVID-19 patients from emergency department data using machine learning Hematological findings and complications of COVID-19 Prediction models for diagnosis and prognosis of COVID-19 infection: Systematic review and critical appraisal Machine learning based early warning system enables accurate mortality risk prediction for COVID-19 Evaluation of the use of electronic medical record systems in Brazilian intensive care units COVID-19) weekly epidemiological update and weekly operational update Fair allocation of scarce medical resources during COVID-19 pandemic: ethical considerations. Einstein Respiratory support in COVID-19 patients, with a focus on resource-limited settings Impaired cardiac function is associated with mortality in patients with acute COVID-19 infection Plasma C-reactive protein levels are associated with improved outcome in ARDS Plasma CRP level is positively associated with the severity of COVID-19 C-Reactive protein level may predict the risk of COVID-19 aggravation The role of biomarkers in diagnosis of COVID-19-A systematic review Chronic kidney disease is associated with severe coronavirus disease 2019 (COVID-19) infection Kidney disease is associated with in-hospital death of patients with COVID-19 Potential biochemical markers to identify severe cases among COVID-19 patients Braden scale for assessing pneumonia after acute ischaemic stroke Predicting factors of pressure ulcers in older Thai stroke patients living in urban communities Lymphopenia predicts disease severity of COVID-19: A descriptive and predictive study Lymphopenia in severe coronavirus disease-2019 (COVID-19): Systematic review and meta-analysis Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): Explanation and elaboration Neural Networks for Pattern Recognition Extremely randomized trees Random forests Gradient boosting with categorical features support XGBoost: A scalable tree boosting system Imbalanced Learning: Foundations, Algorithms, and Applications We would like to thank the BP-A Beneficência Portuguesa de São Paulo Hospital for its willingness to contribute to the research. This work was supported by National Council for Scientific and Technological Development (CNPq) under Grant Number 402626/2020-6 and Paraíba Research Foundation FAPESQPB with Grant Number 206/2020. Initial study concept and design: The authors declare no competing interests. The online version contains supplementary material available at https ://doi. org/10.1038/s4159 8-021-82885 -y.Correspondence and requests for materials should be addressed to F.T.F. Publisher's note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creat iveco mmons .org/licen ses/by/4.0/.