key: cord-0961814-yb6o6xjp authors: Gamberini, Lorenzo; Tartivita, Chiara Natalia; Guarnera, Martina; Allegri, Davide; Baroncini, Simone; Scquizzato, Tommaso; Tartaglione, Marco; Alberto Mazzoli, Carlo; Chiarini, Valentina; Picoco, Cosimo; Coniglio, Carlo; Semeraro, Federico; Gordini, Giovanni title: External validation and insights about the calibration of the return of spontaneous circulation after cardiac arrest (RACA) score date: 2022-04-01 journal: Resusc Plus DOI: 10.1016/j.resplu.2022.100225 sha: e3e497a68c37993b57c6b7e8db6f8d0cd6000917 doc_id: 961814 cord_uid: yb6o6xjp BACKGROUND: The return of spontaneous circulation (ROSC) after cardiac arrest (RACA) score was developed as a tool to predict ROSC probability (pROSC) based on easily available information and it could be useful to compare the performances of different EMS agencies or the effects of eventual interventions. We performed an external validation of the RACA score in a cohort of out of hospital cardiac arrest (OHCA) patients managed by the EMS of the metropolitan city of Bologna, Italy. METHODS: We analyzed data from 2,310 OHCA events prospectively collected between January 2009 and June 2021. Discrimination was assessed with the area under the ROC curve (AUROC), while the calibration belts were used for the comparison of observed versus expected ROSC rates. The AUROCs from our cohort and other validation cohorts were compared using a studentized range test. RESULTS: The AUROC for the study population was 0.691, comparable to that described by previous validation studies. Despite an acceptable overall calibration, we found a poor calibration for asystole and low pROSC ranges in PEA and shockable rhythms. The model showed a good calibration for patients aged over 80, while no differences in performance were found when evaluating events before and after the implementation of 2015 ERC guidelines. CONCLUSIONS: Despite AUROC values being similar in different validation studies for RACA score, we suggest separating the different rhythms when assessing ROSC probability with the RACA score, especially for asystole. Out of hospital cardiac arrest (OHCA) is still a global health burden, with an estimated annual incidence between 67 and 170 cases per 100,000 inhabitants in Europe. Resuscitation of victims suffering OHCA is attempted or continued by the emergency medical services (EMS) personnel in about 50-60% of cases. 1 OHCA management is complex and involves multiple specialities from the initial resuscitation to hospital discharge, and the final patients' outcome could be influenced by multiple aspects in this chain. The local EMS are firstly involved, and play a key role in determining patient survival to the next phases of assistance. However, there is considerable variability in terms of prehospital assistance among Europe and reported prehospital ROSC range between 8% and 42%. 3, 4 Clearly, there is a need to build useful instruments to benchmark the performance of different EMS systems and to evaluate the effects of eventual interventions within a system. In 2011, Grä sner et al. developed and validated the return of spontaneous circulation after cardiac arrest (RACA) score, 5 a predictive tool aiming at estimating the probability of ROSC on the field for patients undergoing OHCA. The RACA score is based on eight variables (sex, age, aetiology, witnessed OHCA, location, initial ECG, bystander CPR and EMS arrival time) easily accessible from cardiac arrest registries and represents an effective solution for predicting ROSC probability. The external validations of the RACA score performed in recent years in Europe 6, 7 and Asia, 8 gave similar results concerning the discrimination capacity of the model. On the other hand, the Finnish external validation study reported a better calibration of the model for patients with shockable versus non-shockable rhythms, 6 and it also resulted poorly calibrated when applied to the Pan-Asian Resuscitation Outcomes Study (PAROS) registry, 8 which required an adjustment of the original constant coefficient. Additional validation studies, especially focusing on the calibration of the model in different contexts, could help to understand these differences. The primary objective of the present study was to externally validate the RACA score in a cohort of OHCA patients extracted from the local Utstein-style registry of the urban EMS of Bologna, Italy. We particularly focused on the calibration of the model for the different initial rhythms. The secondary objective of the study was to assess the calibration of the model based on patients' age and year of the OHCA event, to account for the recent revision of the European Resuscitation Council guidelines on resuscitation in 2015, and the different potential implications of age on resuscitation-related decisions. For this prospective observational study, we considered the historical cohort of EMS interventions for patients suffering OHCA in the metropolitan area of Bologna. Data were prospectively collected from January 2009 to June 2021 as part of the System Saving Lives study (clinicaltrials.gov identifier: NCT04510480). The metropolitan area of Bologna consists of over 450,000 inhabitants, and it is covered by a physician-staffed EMS composed of two advanced life support units (ALS -medical cars), eight ambulances equipped with Basic Life Support skilled rescuers (BLS vehicles), and four ambulances equipped with ILS skilled nurses (ILS vehicles). A detailed description of the EMS dispatch criteria has been published previously. 9 ILCOR systematic review recently recommended the use of termination of resuscitation (TOR) rules to assist clinicians in deciding whether to discontinue resuscitation efforts out of hospital. 10 The ERC guidelines 2021, recently updated evidence about ethics based on scoping reviews of 22 research questions, and expert opinions from different countries. 11 The variation of ethics approach worldwide is very wide and the culture and legislation of the single country affect deeply the behaviour of healthcare professionals in the management of cardiac arrest. Concerning OHCA, in Italy, only physicians are allowed to withhold, withdraw, or terminate resuscitation, therefore, only at ALS team arrival, basic life support can be interrupted if ROSC hasn't been already achieved. Termination of resuscitation (ToR) follows the contemporary ERC guidelines. 12, 13 The study was approved by the local Ethics committee. We included all the patients suffering OHCA assisted by the metropolitan EMS in whom a full resuscitation was attempted by the ALS healthcare personnel attending the scene. The exclusion criteria were incomplete data and immediate resuscitation withhold or withdrawal at the arrival of an ALS unit. The primary outcome measure was ROSC, defined as a palpable pulse at any point during resuscitation for over 30 seconds, patients transported to the emergency department (ED) with ongoing CPR were classified as no-ROSC. The RACA score was calculated as originally described by Grä sner et al., 5 please see supplement Table 1 for details. The expected probability of ROSC for every patient was calculated based on the RACA score as follows: pROSC = 1/(1 + e Àx ), where x is the obtained value for the RACA score. Data were collected in an electronic case report form (FileMaker Pro 16; FileMaker, Inc, Santa Clara, California, USA) and analyzed using Stata/CI 16 (College Station, Texas, USA) and Microsoft Excel 2019 (Microsoft Corporation, Redmond, Washington, USA). Continuous variables were expressed as a median and interquartile range [IQR], while categorical variables were expressed as numbers and percentages. Comparisons between continuous variables were performed with the Mann-Whitney U test, while the Chisquare test or, if appropriate, the Fisher's exact test were used for categorical variables. Discrimination was examined by calculation of the area under the receiver operating characteristic curve (AUROC). The observed AUROC for the study population, together with those reported in the other external validation studies available from the literature, [6] [7] [8] and the internal validation cohort of the original study 5 were compared using a studentized range test. 14 Calibration was assessed using the calibration belts, a graphical approach designed to evaluate the goodness of fit of binary outcome models examining the relationship between estimated probabilities and observed outcome rates. 15 This approach allows the creation of confidence belts for the calibration curve by fitting the observed data with a general calibration function and plotting the corresponding curve. The resulting plot contains a bisector, indicating the expected probabilities from the model, while the calibration belts represent the confidence intervals for observed probabilities, therefore allowing to finely discriminate the overall calibration and the ranges in which the model miscalibrates, in addition to indicating the direction of this phenomenon. The graphical approach is paired to a statistical test, also available for external validation, thus synthesizing the calibration assess- ment in a standard hypothesis testing framework. 15 This method thus offers a more analytical view in the assessment of calibration of dichotomous models, compared to other approaches such as the Hosmer and Lemeshow goodness of fit method, 16 and has already been used in evaluating the calibration for the predictive scores. 7, 17 Results During the study period, 12,919 OHCA cases were managed by the metropolitan EMS of Bologna, and resuscitation was attempted or continued by the ALS units in 2,322 of them (18.0%) that were considered for inclusion in this study, supplement Fig. 1 shows the yearly distribution of the OHCA events and resuscitation attempts. Twelve cases had missing data about the first observed rhythm and were subsequently excluded from the analyses, therefore, the final study population in all the analyses included 2,310 out of 2,322 patients (99.48%), supplement Fig. 2 shows the flow of patients throughout the study. Globally, patients were mostly men (n = 1,563, 67.7%), with a median age of 71 years. The most frequent first observed rhythm was PEA (n = 905, 39.2%), followed by VF/VT (n = 730, 31.6%) and asystole (n = 675, 29.2%), and the most frequently suspected etiology, following the cathegorization used in the RACA score, was medical (n = 1,942, 84.1%), followed by hypoxia (n = 223, 9.7%). In the majority of cases, the OHCA event was witnessed (n = 1,084, 87.6%), and resuscitation was started by the bystanders in 1,437 out of 2,310 patients (62.2%), median EMS arrival time was 9 minutes. The median expected ROSC rate following the RACA score for the whole population was 44% (IQR 34.6-58.9%), and the observed ROSC rate was 42.3%. Table 1 shows the main characteristics of the population concerning RACA score variables and the first observed rhythm, while Fig. 1 shows the distribution of the different rhythms within the annual number of OHCA and the deciles of ROSC probability (pROSC) calculated from the RACA score. As expected, based on the RACA scoring system, shockable rhythms mostly fell in the upper range of pROSC, PEA was mostly present in the fifth decile (pROSC 0.4-0.5), while asystole was mostly expressed in the lower deciles (pROSC 0.1-0.4). Fig. 2 shows the ROC curve of the model, the calculated AUROC was 0.691 (95% CI: 0.669-0.713), while Fig. 3 displays the multiple AUROC comparisons between the study population and the different populations from available literature, expressed in the graph with their 95% confidence intervals. The studentized range test showed no significant differences among the AUROCs (p = 1.000). Fig. 4 shows the calibration belts for the model considering the whole population and the different subpopulations based on the first observed rhythm, age and year of enrollment. Concerning the overall population, despite the test statistic being significant, the 95% CI belt ran over the bisector, demonstrating an observed ROSC rate higher than expected, only for pROSC 0.01-0.02. The calibration belts performed on the different subgroups based on the first observed rhythm were also significant and showed larger incongruences between the predicted and observed ROSC rates, especially for lower values of pROSC in PEA (<0.2) and shockable rhythms (<0.28), where observed ROSC rates were higher than predicted. Finally, for asystole, the observed 95% CI belt ran under the bisector, showing observed ROSC rates lower than expected, for most of the probability ranges (0.15-0.71). Up to a quarter of the urban EMS missions are related to OHCA, 18 and the achievement of ROSC represents the first step in the care process for these patients. Moreover, ROSC could be considered a quality indicator for the EMS taking into account both organizational and technical aspects of the delivered resuscitation. 19, 20 Therefore, the development and validation of a predictive score for ROSC such as the RACA score could be useful in benchmarking different Emergency services or evaluating the effects of interventions. In this external validation study of the RACA score applied to an Italian metropolitan EMS, we found moderate discrimination (AUROC: 0.691), that was comparable to those reported in other external validation studies. [6] [7] [8] Calibration was deemed acceptable for the overall population despite a significant statistic test for the calibration belt due to the curve running over the bisector for the lowest 1% of expected ROSC probability (Fig. 4) . Since the number of patients in this decile was very low, this aspect should be considered less relevant (see Fig. 1 ). However, when the different first observed rhythms were separately analysed, suboptimal calibration was observed for the shockable rhythms and PEA, showing higher than expected ROSC rates, especially for lower pROSC. Moreover, the observed ROSC rates for asystole were almost constantly under the bisector, therefore constantly lower than expected. Differences between expected and observed ROSC rates for non-shockable rhythms were already reported in previous validation studies, 6, 8 and they were attributed to the different protocols in regards to the cessation of resuscitation attempts. In particular, Kupari et al., 6 showed similar results with a good overall calibration for shockable rhythms but lower than expected ROSC rates for low pROSC, in particular for PEA and Asystole. On the other hand, Caputo et al. 7 showed a good calibration of the model for all the pROSC ranges explored. The populations evaluated in these calibration studies were notably different, in particular for age, with a proportion of patients aged over 80 ranging from 15% (Kupari et al.) to 39% (Caputo et al.) . Also, the rate of bystander CPR and the first observed rhythms showed notable differences. In fact, bystander CPR prevalence ranged from 14.6% (Grä sner et al.) to 62.2% (reported in our population), moreover, our population and Finnish population had a lower relative frequency asystole (29% and 22%, respectively) compared to both the Caputo et al. population (45%) and the score development study (46.4%). Table 2 summarises the population differences among the evaluated studies. OHCA events encompass a wide range of underlying mechanisms and situations, moreover, policies about do-not-attempt-cardio pulmonary-resuscitation (DNACPR) records and orders, 21 withdrawal and termination of resuscitation (ToR) may vary greatly across the different EMS systems even within the same nation. Abbreviations: NA -not assessable either because directly reported in the ROSC/no ROSC groups without information about the overall population or not evaluated. In Italy, DNACPR records are not in place by law, and only physicians can withhold, withdraw or terminate the resuscitation attempts, therefore, basic life support may be started by non-medical health personnel and subsequently stopped once ALS vehicles arrive. In particular, once that resuscitation is started from the first arrived personnel, situational and emotional aspects could have an impact on the decision to provisionally pursue resuscitative attempts even in cases with perceived futility and eventually early stop the manoeuvres. [22] [23] [24] This could partially explain the notable reduction in the observed versus expected ROSC rate for asystole. Moreover, in support of this hypothesis, the age-based subanalysis of calibration belts showed a better calibration of the model when assessing patients aged over 80, where asystole was less represented among the first observed rhythms (24.6% vs 30.9%), probably reflecting the higher attitude of the local EMS personnel to withdraw resuscitation in elderly people with rhythms associated to a low survival probability. It could be hypothesized that the calibration of the RACA score in different EMS contexts could be influenced by the attitude of the EMS personnel to initiate and terminate resuscitation efforts, especially for the low pROSC intervals, involving mostly asystole as first observed rhythm. Probably, the influence of the propensity of EMS personnel to initiate or terminate resuscitation should be furtherly investigated in this field. Criteria for withholding, withdrawing and terminating resuscitation (ToR) are still being debated, especially in the prehospital setting, 10,11 therefore, different socio-cultural, personal and situational aspects could introduce variability in these decisions, 25 finally influencing the access of patients to advanced resuscitation. Moreover, in recent years, the resources available for advanced resuscitation became more complex with the advent of extracorporeal CPR, the advances in trauma-related resuscitative algorithms 26, 27 and the growing collective consciousness of the need for organ donors harvesting, 28 that could have furtherly influenced the decisions based on the different capabilities of the local health system. These aspects could partially explain the reduced calibration of the model in the cohort of patients experiencing OHCA after 2015. Finally, the described limitation of the RACA score calibration for low pROSC and its associated rhythms could be theoretically applicable to other scores evaluating the probability of ROSC since the inclusion of patients with low pROSC could be different in different populations. The progressive definition of guidelines addressing the clinical decisions concerning the beginning and continuation of resuscitative efforts should reduce this phenomenon. The main limitation of this study relies upon its monocentric nature, which could have emphasized the differences in terms of resuscitation decisions due to the local clinicians' behaviour, however, this aspect could also have underlined a potential role of the RACA score in fostering local audit and discussion about DNACPR, withdrawal and ToR policies. In this external validation of the RACA score encompassing eleven years of activity of a single Italian EMS system, we found similar discrimination (AUROC: 0.691) to that reported in other contexts. Calibration was overall acceptable in the global population for the most represented deciles of expected ROSC, however, when the single first observed rhythms were analysed separately, we observed sig-nificant discrepancies between observed and expected ROSC rates at low baseline pROSC, in particular for asystole. Therefore, we suggest that, when interpreting the observed versus expected ROSC with the RACA score in different contexts, separate analyses based on the first observed rhythms should be performed to avoid inclusion biases due to the different behaviours of the EMS in conditions of low pROSC, particularly when asystole is the first observed rhythm. The study was approved by the Institutional Review Board (IRB) of Maggiore Hospital, Bologna, Italy. No relationship exists between any of the authors and any commercial entity or product mentioned in this manuscript that might represent a conflict of interest. No inducements have been made by any commercial entity to submit the manuscript for publication. All LG and FS concepted and designed the work. CNT, MG, SB and CP acquired and interpreted the data. DA and LG performed the statistical analysis. LG, MT and VC drafted the article. TS, CC, FS, and GG substantively revised the article. All the Authors have approved the submitted version and have agreed both to be personally accountable for the author's own contributions and to ensure that questions related to the accuracy or integrity of any part of the work, even ones in which the author was not personally involved. European Resuscitation Council Guidelines 2021: Epidemiology of cardiac arrest in Europe Determinants of cognitive outcome in survivors of out-of-hospital cardiac arrest Description of Emergency Medical Services, treatment of cardiac arrest patients and cardiac arrest registries in Europe Survival after out-of-hospital cardiac arrest in Europe -Results of the EuReCa TWO study ROSC after cardiac arrest-the RACA score to predict outcome after out-of-hospital cardiac arrest External validation of the ROSC after cardiac arrest (RACA) score in a physician staffed emergency medical service system Validation of the return of spontaneous circulation after cardiac arrest (RACA) score in two different national territories Validation of the ROSC after cardiac arrest (RACA) score in Pan-Asian out-of-hospital cardiac arrest patients Improving the Appropriateness of Advanced Life Support Teams' Dispatch: A Before-After Study Out-of-hospital cardiac arrest termination of resuscitation (TOR) rules (EIT #642 revised): Systematic Review European Resuscitation Council Guidelines 2021: Ethics of resuscitation and end of life decisions Section 11. The ethics of resuscitation and end-of-life decisions Resuscitation Council Guidelines for Resuscitation 2010 Section 10. The ethics of resuscitation and end-of-life decisions Comparing the areas under more than two independent ROC curves Calibration belt for quality-of-care assessment based on dichotomous outcomes Assessing the calibration of dichotomous outcome models with the calibration belt Comparison between SAPS II and SAPS 3 in predicting hospital mortality in a cohort of 103 Italian ICUs. Is new always better? Characteristics of Emergency Medical Service Missions in Out-of-Hospital Cardiac Arrest and Death Cases in the Periods of Before and After the COVID-19 Pandemic The association between AHA CPR quality guideline compliance and clinical outcomes from out-of-hospital cardiac arrest Cardiopulmonary resuscitation quality: Improving cardiac resuscitation outcomes both inside and outside the hospital: A consensus statement from the American heart association Do-not-attemptcardiopulmonary-resuscitation decisions: an evidence synthesis Factors influencing physician decision making to attempt advanced resuscitation in asystolic out-of-hospital cardiac arrest Variability in the initiation of resuscitation attempts by emergency medical services personnel during out-of-hospital cardiac arrest Beyond prognostication: Ambulance personnel's lived experiences of cardiac arrest decision-making Time of on-scene resuscitation in out of-hospital cardiac arrest patients transported without return of spontaneous circulation European Resuscitation Council Guidelines 2021: Cardiac arrest in special circumstances Resuscitative endovascular balloon occlusion of the aorta and traumatic out-of-hospital cardiac arrest: A nationwide study European Resuscitation Council and European Society of Intensive Care Medicine guidelines 2021: post-resuscitation care Supplementary data to this article can be found online at https://doi. org/10.1016/j.resplu.2022.100225.