key: cord-0721398-i45s1v0s authors: Endo, Hideki; Ohbe, Hiroyuki; Kumasawa, Junji; Uchino, Shigehiko; Hashimoto, Satoru; Aoki, Yoshitaka; Asaga, Takehiko; Hashiba, Eiji; Hatakeyama, Junji; Hayakawa, Katsura; Ichihara, Nao; Irie, Hiromasa; Kawasaki, Tatsuya; Kurosawa, Hiroshi; Nakamura, Tomoyuki; Okamoto, Hiroshi; Shigemitsu, Hidenobu; Takaki, Shunsuke; Takimoto, Kohei; Uchida, Masatoshi; Uchimido, Ryo; Miyata, Hiroaki title: Conventional risk prediction models fail to accurately predict mortality risk among patients with coronavirus disease 2019 in intensive care units: a difficult time to assess clinical severity and quality of care date: 2021-06-01 journal: J Intensive Care DOI: 10.1186/s40560-021-00557-5 sha: e22755f735745dea26c8eb93d595ea32e159bebb doc_id: 721398 cord_uid: i45s1v0s Since the start of the coronavirus disease 2019 (COVID-19) pandemic, it has remained unknown whether conventional risk prediction tools used in intensive care units are applicable to patients with COVID-19. Therefore, we assessed the performance of established risk prediction models using the Japanese Intensive Care database. Discrimination and calibration of the models were poor. Revised risk prediction models are needed to assess the clinical severity of COVID-19 patients and monitor healthcare quality in ICUs overwhelmed by patients with COVID-19. Since the start of the coronavirus disease 2019 (COVID-19) pandemic, intensive care units (ICUs) worldwide have struggled to treat affected patients who require a completely different approach to treatment than other patients [1] . Although many severe cases are admitted to ICUs, it is unknown whether the conventional risk scoring systems that were developed for ICU patients can be applied to patients with COVID-19. With unknown predictive performance, healthcare professionals have faced difficulties in assessing the clinical severity of patients with COVID-19 and monitoring the quality of care in ICUs. New risk prediction models for COVID-19 patients have been developed [2] , but most of these were not developed specifically for ICU patients, and it is unknown whether they perform as well in clinical practice as they did in the model development studies. It is also likely that overwhelmed ICUs lack the time to derive and validate novel risk scores. In such circumstances, ICUs must use conventional scoring systems, such as the Acute Physiology and Chronic Health Evaluation (APACHE) and Simplified Acute Physiology Score (SAPS). Several recent studies have used APACHE and SAPS to provide information on the clinical severity of COVID-19 [3] [4] [5] . However, very few reports have examined their validity of applying them to patients with COVID-19. One letter from the UK reported that APACHE II underestimated the risk of death, concluding that the risk scoring systems that were widely used before the pandemic were inappropriate for evaluating the clinical severity of COVID-19 [6] . In Japan, a research group recently developed the Japan Risk of Death (JROD), a prediction model that recalibrated the APACHE III-j model [7] . However, this model may show limited validity in patients with COVID-19 because it was developed using the data collected before the pandemic and it was designed for general use in ICUs. Therefore, we investigated whether conventional risk prediction models, such as APACHE II, SAPS II, APACHE III-j, and JROD, can be applied to patients with COVID-19 and determined their predictive performance. We obtained data for confirmed cases of COVID-19 admitted between January 2020 and February 2021 from the Japanese Intensive Care Patient Database (JIPAD) [8] . We used JROD to predict mortality in the same way as in the previous study [7] , but with a development period of January 2019 to December 2019. This was then applied to predict mortality in the study cohort and defined as JROD 2019 predicted mortality. The predictive performances of APACHE II, SAPS II, APACHE III-j, and JROD 2019 were assessed using the area under the receiver operating characteristic curves, Brier scores, Hosmer-Lemeshow tests, calibration plots, and standardized mortality ratios. A total of 444 patients admitted to 40 ICUs in Japan were extracted from the JIPAD for analysis. The clinical characteristics of patients are shown in Table 1 . The model performance statistics are presented in Table 2 and Fig. 1 . Death at hospital discharge was recorded in 69 patients (15.5%), which was less than half the mortality reported by Stephens et al., although the APACHE II scores were comparable [6] . Using JIPAD data, the APACHE II, SAPS II, and APACHE III-j models overestimated the risk of death, whereas JROD 2019 underestimated the risk. The discrimination and calibration of APACHE III-j and JROD were poor compared with those reported in the JROD development study [7] . Although the results are dissimilar to a previous report [6] in terms of the direction of estimated risk (i.e., overestimation/underestimation), we make the same conclusion that the risk models used before the pandemic are not suitable for patients with COVID-19. Of note, even JROD 2019 , a model that was developed to improve the predictive ability of APACHE III-j, displayed suboptimal predictive performance. Owing to the poor predictive performance, it is difficult to incorporate the predicted mortality calculated using these risk models in quality APACHE Acute Physiology and Chronic Health Evaluation, AUROC area under the receiver operating characteristic curve, CI confidence interval, JROD Japan Risk of Death, SAPS Simplified Acute Physiology Score assessment tools, such as funnel plots and exponentially weighted moving average charts, with high reliability. Consequently, it will be difficult to implement quality assessment and improvement in ICUs, particularly those where patients with COVID-19 occupy a high proportion of ICU beds. Calibration can be improved with simple update methods, like that done in the JROD study, but discrimination can only be improved by updating the coefficients of each predictor and/or adding other relevant predictors [9] . Thus, a revised risk prediction model designed specifically for COVID-19 patients together with logistical support for its implementation in ICUs are urgently needed. This paper was written as a part of the JIPAD project, which was funded by the Japanese Society of Intensive Care Medicine. The authors' agreement with the JIPAD project does not allow us to publish the data used for this manuscript or to share it with others. Ethics approval and consent to participate This study was approved by the Research Ethics Committee of the University of Tokyo (Approval number: 2020242NI-(1)). Consent to participate was waived because of the anonymous nature of the data. Not applicable. Fig. 1 Calibration plots. APACHE, Acute Physiology and Chronic Health Evaluation; JROD, Japan Risk of Death; SAPS, Simplified Acute Physiology Score. Note: Observed mortality is plotted against predicted mortality. The study population was divided according to the predicted mortality into 10 equally sized groups, which are presented as a rug plot along the horizontal axis. A natural spline was drawn for the plots. The shaded area indicates the 95% confidence interval. If the calibration is perfect, the plot aligns with the diagonal line 10 Department of Emergency and Critical Care Medicine, National Hospital Organization Tokyo Medical Center, 2-5-1, Higashigaoka, Meguro-ku, Tokyo 152-8902, Japan. 11 Department of Emergency and Critical Care Medicine Department of Critical Care Medicine, St. Luke's International Hospital Tokyo 104-8560, Japan. 17 Department of Intensive Care Medicine, Graduate School of Medicine 20 Department of Emergency and Critical Care Medicine Critical care capacity during the COVID-19 pandemic: global availability of intensive care beds Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal Clinical characteristics and outcomes of critically ill patients with novel coronavirus infectious disease (COVID-19) in China: a retrospective multicenter study Patient characteristics, clinical course and factors associated to ICU mortality in critically ill patients infected with SARS-CoV-2 in Spain: a prospective, cohort, multicentre study Clinical characteristics and day-90 outcomes of 4244 critically ill adults with COVID-19: a prospective cohort study Analysis of critical care severity of illness scoring systems in patients with coronavirus disease 2019: a retrospective analysis of three U.K. ICUs Development and validation of the predictive risk of death model for adult patients admitted to intensive care units in Japan: an approach to improve the accuracy of healthcare quality measures The Japanese Intensive care PAtient Database (JIPAD): a national intensive care unit registry in Japan Clinical prediction models Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations