key: cord-0209938-qh2yga3t
authors: Xie, Feng; Zhou, Jun; Lee, Jin Wee; Tan, Mingrui; Li, Siqi; Rajnthern, Logasan SO; Chee, Marcel Lucas; Chakraborty, Bibhas; Wong, An-Kwok Ian; Dagan, Alon; Ong, Marcus Eng Hock; Gao, Fei; Liu, Nan
title: Benchmarking emergency department triage prediction models with machine learning and large public electronic health records
date: 2021-11-22
journal: nan
DOI: nan
sha: 0c03d78b6428e7c6922dfb4929d2c605de8733a1
doc_id: 209938
cord_uid: qh2yga3t

The demand for emergency department (ED) services is increasing across the globe, particularly during the current COVID-19 pandemic. Clinical triage and risk assessment have become increasingly challenging due to the shortage of medical resources and the strain on hospital infrastructure caused by the pandemic. As a result of the widespread use of electronic health records (EHRs), we now have access to a vast amount of clinical data, which allows us to develop predictive models and decision support systems to address these challenges. To date, however, there are no widely accepted benchmark ED triage prediction models based on large-scale public EHR data. An open-source benchmarking platform would streamline research workflows by eliminating cumbersome data preprocessing, and facilitate comparisons among different studies and methodologies. In this paper, based on the Medical Information Mart for Intensive Care IV Emergency Department (MIMIC-IV-ED) database, we developed a publicly available benchmark suite for ED triage predictive models and created a benchmark dataset that contains over 400,000 ED visits from 2011 to 2019. We introduced three ED-based outcomes (hospitalization, critical outcomes, and 72-hour ED reattendance) and implemented a variety of popular methodologies, ranging from machine learning methods to clinical scoring systems. We evaluated and compared the performance of these methods against benchmark tasks. Our codes are open-source, allowing anyone with MIMIC-IV-ED data access to perform the same steps in data processing, benchmark model building, and experiments. This study provides future researchers with insights, suggestions, and protocols for managing raw data and developing risk triaging tools for emergency care.

Emergency Departments (ED) experience large volumes of patient flows and growing resource demands, particularly during the current COVID-19 pandemic 1 . This growth has caused ED crowding 2 and delays in care delivery 3 , resulting in increased morbidity and mortality 4 . The ED triage models [5] [6] [7] [8] [9] provide opportunities for identifying high-risk patients and prioritizing limited medical resources. ED triage centers on risk stratification, which is a complex clinical judgment based on factors such as the patient's likely acute course, availability of medical resources, and local practices. 10 The widespread use of Electronic Health Records (EHR) has led to the accumulation of large amounts of data, which can be used to develop predictive models to improve emergency care 11, 12 . Based on a few large-scale EHR databases, such as Medical Information Mart for Intensive Care III (MIMIC-III) 13 , eICU Collaborative Research Database 14 , and Amsterdam University Medical Centers Database (AmsterdamUMCdb) 15 , several benchmarks have been established [16] [17] [18] . These benchmarks standardized the process of transforming raw EHR data into readily usable data to construct predictive models. They have provided clinicians and methodologists with easily accessible and high-quality medical data, accelerating research and validation efforts 19, 20 . These non-proprietary databases and open-source pipelines make it possible to reproduce and improve clinical studies in ways that would otherwise not be possible 16 . While there are some publicly available benchmarks, most pertain to intensive care settings, and there are no widely accepted benchmarks related to the ED. An ED-based public benchmark would lower the entry barrier for new researchers, allowing them to focus their efforts on novel research.

Machine learning has seen tremendous advances in recent years, and it has gained increasing popularity in the realm of ED triage prediction models [21] [22] [23] [24] [25] [26] [27] [28] . These prediction models involve machine learning, deep learning, interpretable machine learning, and others. However, we have found that researchers often develop an adhoc model for one clinical prediction task at a time, using only one dataset [21] [22] [23] [24] [25] [26] . There is a lack of comparative studies among different methods and models to predict the same ED outcome, undermining the generalizability of any single model. Generally, existing prediction models are developed on retrospective data without prospective validation in real-world clinical settings. Hence, there remains a need for prospective, comparative studies on the accuracy, interpretability, and utility of risk models in the ED. Using an extensive public EHR database, we aimed to standardize data preprocessing and establish a comprehensive ED benchmark dataset alongside comparable risk triaging models for three ED-based tasks. It is expected to facilitate reproducibility and model comparison, and accelerate progress toward utilizing machine learning in future ED-based studies.

In this paper, we proposed a public benchmark suite for the ED using a large EHR dataset and introduced three ED-based outcomes: hospitalization, critical outcomes, and 72-hour ED reattendance. We implemented and compared several popular methods for these clinical prediction tasks. We used data from the publicly available MIMIC IV Emergency Department (MIMIC-IV-ED) database 29, 30 , which contains over 400,000 ED visit episodes from 2011 to 2019. Our code is open-source (https://github.com/nliulab/mimic4ed-benchmark) so that anyone with access to MIMIC-IV-ED can follow our data processing steps, create benchmarks, and reproduce our experiments. This study provides future researchers with insights, suggestions, and protocols to process the raw data and develop models for emergency care in an efficient and timely manner.

This section consists of three parts. First, we describe raw data processing, benchmark data generation, and cohort formation. Next, we introduce baseline models for benchmark tasks. Finally, we elaborate on the experimental setup and model performance evaluation.

We standardized terminologies as follows. Patients are referred to by their subjects_id. Each patient has one or more ED visits, identified by stay_id in edstays.csv. If there is an inpatient stay following an ED visit, this stay_id could be linked with an inpatient admission, identified by hadm_id in edstays.csv. subjects_id and hadm_id can also be traced back to the MIMIC-IV 31 database to follow the patient throughout inpatient or ICU stay and patients' future or past medical utilization, if needed. In the context of our tasks, we used edstays.csv as the root table and stay_id as the primary identifier. As a general rule, we have one stay_id for each prediction in our benchmark tasks. All raw tables were linked through extract_master_dataset.ipynb, illustrated in Figure 1 . The linkage was based on the root table, and merged through different identifiers, including stay_id (ED), subjects_id, hadm_id, or stay_id (ICU). We extracted all high-level information and consolidated them into a master dataset (master_dataset.csv).

To construct the master dataset, we reviewed a number of existing literature 5, 7, [32] [33] [34] to identify relevant variables and outcomes. Moreover, we consulted clinicians and informaticians familiar with the raw data and ED operation to identify and confirm all ED-relevant variables. We excluded variables that were irrelevant, repeated, or largely absent. A list of high-level constructed variables is presented in Table 1 , including patient history, variables collected at triage and before ED disposition, and primary ED-relevant outcomes. The final master dataset includes 448,972 ED visits by 216,877 unique patients. 

The data processing workflow (data_general_processing.ipynb), illustrated in Figure  2 , begins with the master dataset generated from Section 2.1 to generate the benchmark dataset. In the first step, we filtered out all ED visits with patients under 18 years old and those without primary emergency triage class assignments. A total of 441,437 episodes remained after the filtering process.

The raw EHR data cannot be used directly for model building due to missing values, outliers, duplicates, or incorrect records caused by system errors or clerical mistakes. We addressed these issues with several procedures. For vital signs and lab tests, a value would be considered an outlier and marked as missing if it was outside the plausible physiological range as determined by domain knowledge, such as a value below zero or a SpO2 level greater than 100%. We followed the outlier detection procedure used in MIMIC-EXTRACT 18 , a well-known data processing pipeline for MIMIC-III. We utilized the thresholds available in the source code repository of Harutyunyan at el. 35 , where one set of upper and lower thresholds was used for filtering outliers. Any value that falls outside of this range was marked as missing. Another set of thresholds was introduced to indicate the physiologically valid range, and any value that falls beyond this range was replaced by its nearest valid value. These thresholds were suggested by clinical experts based on domain knowledge. 

In MMIC-IV, each hospital admission is associated with a group of ICD diagnosis codes (in diagnoses_icd.csv), indicating the patients' comorbidities. We embedded the ICD codes within a time range (e.g., five years) from each ED visit into Charlson Comorbidity Index (CCI) 36 and Elixhauser Comorbidity Index (ECI) 37 according to the mapping proposed by Quan H et al. 38 We adopted the codebase from Cates et al. 39 and developed the neural network-based embedding with similar network structures to Med2Vec 40 .

Following are three ED-relevant clinical outcomes. They are all of utmost importance to clinicians and hospitals due to their immense implications on costs, resource prioritization, and patients' quality of life. Accurate prediction of these outcomes with the aid of big data and artificial intelligence has the potential to transform health services.

• The hospitalization outcome is met with an inpatient care site admission immediately following an ED visit [41] [42] [43] . Patients who transitioned to ED observation were not considered hospitalized unless they were eventually admitted to the hospital. As hospital beds are limited, this outcome indicates resource utilization and may facilitate resource allocation efforts. The hospitalization outcome also suggests patient acuity, albeit in a limited way, since hospitalized patients represent a broad spectrum of disease severity. • The critical outcome 33 is compositely defined as either inpatient mortality 44 or transfer to an ICU within 12 hours. This outcome represents the critically ill patients who require ED resources urgently and may suffer from poorer health outcomes if care is delayed. Predicting the critical outcome at ED triage may enable physicians to allocate ED resources efficiently and intervene on high-risk patients promptly. • The ED reattendance outcome refers to a patient's return visit to ED within 72 hours after their previous discharge from the ED. It is a widely used indicator of the quality of care and patient safety and is believed to represent patients who may not have been adequately triaged during their first emergency visit 45 .

Various triage systems, including clinical judgment, scoring systems, regression, machine learning, and deep learning, were applied to the benchmark dataset and evaluated on each benchmark task, as detailed in Table 2 . A five-level triage system, Emergency Severity Index (ESI) 46 , was assigned by a registered nurse based on clinical judgments. Level 1 is the highest priority, and level 5 is the lowest. Several scoring systems were also calculated, including the Modified Early Warning Score (MEWS) 47 , National Early Warning Score (NEWS, versions 1 and 2) 48 , Rapid Emergency Medicine Score (REMS) 49 , and Cardiac Arrest Risk Triage (CART) 50 . It is important to note that there are no neurological features (i.e., Glasgow Coma Scale) in the MIMIC-IV-ED dataset, which may lead to incomplete scores. Three machine learning methodslogistic regression (LR), random forest (RF), and gradient boosting (GB)were benchmarked as well as deep learning methods multilayer perceptron (MLP) 51 , Med2Vec 40 , and long short-term memory (LSTM) [52] [53] [54] . These neural network structures are illustrated in eFigure 1. We used the scikit-learn package 55 with the default parameters for machine learning methods and Keras 56 for deep learning methods. In addition, the interpretable machine learning method, AutoScore 57,58 , was implemented with its R software package 59 . 

We conducted all experiments on a server equipped with an Intel Xeon W-2275 processor, 128GB of memory, and an Nvidia RTX 3090 GPU, and the running time at model training was recorded. Deep learning models were trained using the Adam optimizer and binary cross-entropy loss. The AutoScore method optimized the number of variables through a parsimony plot. As the implementation was only for demonstration purposes, Module 5 of the clinical fine-tuning process in AutoScore was not implemented. We conducted the receiver operating characteristic (ROC) and precision-recall curve (PRC) analysis to evaluate the performance of all triage prediction models. The area under the ROC curve (AUROC) and the area under the PRC (AUPRC) values were reported as an overall measurement of predictive performance. Model performance was reported on the test set, and 100 bootstrapped samples were applied to calculate 95% confidence intervals (CI). Furthermore, we computed the sensitivity and specificity measures under the optimal cutoffs, defined as the points nearest to the upper-left corner of the ROC curves.

We compiled a master dataset comprising 448,972 ED visits of 216,877 unique patients. After excluding incomplete or pediatric visits, a total of 441,437 adult ED visits were finally included in the benchmark dataset. They were randomly split into 80% (353,150) training data and 20% (88,287) test data. Table 3 summarizes the baseline characteristics of the entire cohort, stratified by outcomes. The average age of the patients was 52.8 years old, and 54.1% (n=242,844) of them were females. Compared with other patients, those with critical outcomes displayed a higher body temperature and heart rate and were prescribed a greater amount of medication. Additionally, they were more likely to have fluid and electrolyte disorders, coagulopathy, cancer, cardiac arrhythmias, valvular disease, and pulmonary circulation disorders. 

With a descending order of variable importance extracted from RF, the top 10 variables selected for each benchmark task are presented in Table 5 . Vital signs show significant predictive value in all three tasks. Age is also among the top predictive variables for all tasks, underscoring the impact of aging on emergency care utilization. While the triage level (i.e., ESI) is highly related to the hospitalization and critical outcome, it is not relevant to 72-hour ED reattendance. Conversely, despite its lower importance for hospitalization and critical outcomes, ED length of stay becomes the top variable for 72-hour ED reattendance prediction. The previous health utilization variable seems to be a less important feature for the ED-based tasks. 

Machine learning exhibited a higher degree of discrimination in predicting all three outcomes. Gradient boosting achieved an AUC of 0.881 (95% CI: 0.877-0.886) for the critical outcome and an AUC of 0.820 (95% CI: 0.818-0.823) for the hospitalization outcome. However, the corresponding performance for 72-hour ED reattendance was considerably lower. Compared with gradient boosting, deep learning could not achieve even higher performance. While traditional scoring systems did not show good discriminatory performance, interpretable machine learning-based AutoScore achieved an AUC of 0.846 (95% CI: 0.842-0.851) for critical outcomes with seven variables, and 0.793 (95% CI: 0.791-0.797) for hospitalization outcomes with 10 variables. Supplementary eTable 1 presents the performance of critical outcome prediction at ED disposition. Moreover, as shown in Table 6 and Figure 3 , the performance of a variety of widely used machine learning and scoring systems is assessed by various metrics on the test set. Table 6 : Comparison of the performance of different models based on three different outcomes. 

This paper proposed standardized benchmarks for future researchers interested in analyzing large-scale ED clinical data. Our study provides a pipeline to process raw data from the newly published MIMIC-IV-ED database, and generates a benchmark dataset, the first of its kind in the ED context. The benchmark dataset contains approximately half a million ED visits, and is conveniently accessible by researchers who plan to replicate our experiments or further build upon our work. Additionally, we demonstrated several triage prediction models (e.g., machine learning and clinical scoring systems) on routinely available information using this benchmark dataset for three ED-relevant outcomes: hospitalization, critical outcome, and ED reattendance. Our benchmark dataset also supports linkage to the main MIMIC-IV database, allowing researchers to analyze a patient's clinical course from the time of ED presentation through the hospital stay.

Our study showed that machine learning models demonstrated higher predictive accuracy, consistent with the previous studies 9,17,60 . Complex deep learning 61 models such as Med2Vec and LSTM did not perform better than simpler models. These results suggest that overly complex models do not necessarily improve performance with relatively simple and low-dimensional data in the ED. Furthermore, predictions made by black-box machine learning have critical limitations in clinical practice 62, 63 , particularly for decision-making in emergency care. Although machine learning models outperform in terms of predictive accuracy, the lack of explanation makes it challenging for frontline physicians to understand how and why the model reaches a particular conclusion. In contrast, scoring systems combine just a few variables using simple arithmetic and have a more explicit clinical representation 57 . This transparency allows doctors to understand and trust model outputs more easily and contributes to the validity and acceptance of clinical scores in real-world settings 64, 65 . In our experiments, predefined scoring systems were unable to achieve satisfactory accuracy. However, AutoScore-based data-driven scoring systems complemented them with much higher accuracy while maintaining the advantages of the point-based scores 7 .

The primary goals of ED triage prediction models are to identify high-risk patients accurately and to allocate limited resources efficiently. While physicians can generally determine the severity of a patient's acute condition, their decisions are often subjective and depend on an individual's knowledge and experience. This study explored data-driven methods to provide an objective assessment for three EDrelevant risk triaging tasks based on large-scale public EHRs. The open nature of the models makes them suitable for reproducibility and improvement. The scientific research community can make full use of the data and the triage prediction models to improve emergency care. In addition, the three ED triaging tasks are interrelated, yet represent distinct groups of predictors. Hospitalization and critical outcomes share a similar set of predictors, whereas the prediction of ED reattendances depend on various other variables.

This study contributes to the scientific community by standardizing research workflows and reducing barriers of entry for both clinicians and data scientists engaged in ED research. In the future, researchers may use this data pipeline to process raw MIMIC-IV-ED data. They may also develop new models and evaluate them against our ED-based benchmark tasks and prediction models. Additionally, our pipeline does not focus exclusively on ED data; we also provide linkages to the MIMIC-IV main database with all ICU and inpatient episodes. Data scientists interested in extracting ED data as additional variables and linking them to the other settings of the MIMIC-IV database can exploit our framework to streamline their research without consulting different ED physicians. With the help of this first largescale public ED benchmark dataset and data processing pipeline, researchers can conduct high-quality ED research without needing a high level of technical proficiency.

This study has several limitations. First, although the study is based on an extensive database, it is still a single-center study. The performance of different methods used in this study may differ in other healthcare settings. Despite this, the proposed benchmarking pipeline could still be used as a reference for future big data research in the ED. Furthermore, examining whether models trained on the benchmark data generalize to other clinical datasets would be interesting. Second, the benchmark dataset established in this study is based on EHR data with routinely collected variables, where certain potential risk factors, such as socioeconomic status and neurological features, were not recorded. In addition, the dataset lacks sufficient information to detect out-of-hospital deaths, which may bias our models. Despite these limitations, the data processing pipeline can be leveraged widely when new researchers wish to conduct ED research using the MIMIC-IV-ED database.

Comparison of performance of different models applied to critical outcomes at ED disposition. 

Trends in Emergency Department Visits and Hospital Admissions in Health Care Systems in 5 States in the First Months of the COVID-19 Pandemic in the US

Emergency department crowding: A systematic review of causes, consequences and solutions

The impact of delays to admission from the emergency department on inpatient outcomes

Effect of emergency department crowding on outcomes of admitted patients

Emergency department triage prediction of clinical outcomes using machine learning models

A simple clinical assessment is superior to systematic triage in prediction of mortality in the emergency department

Development and Assessment of an Interpretable Machine Learning Triage Tool for Estimating Mortality After Emergency Admissions

Heart rate n-variability (HRnV) and its application to risk stratification of chest pain patients in the emergency department

Developing machine learning models to personalize care levels among emergency room patients for hospital admission

Problems for clinical judgement: 3. Thinking clearly in an emergency

The effect of electronic health record implementation on community emergency department operational measures of performance

Emergency medicine electronic health record usability: where to from here?

MIMIC-III, a freely accessible critical care database

The eICU Collaborative Research Database, a freely available multi-center database for critical care research

Sharing ICU Patient Data Responsibly Under the Society of Critical Care Medicine/European Society of Intensive Care Medicine Joint Data Science Collaboration: The Amsterdam University Medical Centers Database (AmsterdamUMCdb) Example

Multitask learning and benchmarking with clinical time series data

Benchmarking deep learning models on large healthcare datasets

MIMIC-Extract: a data extraction, preprocessing, and representation pipeline for MIMIC-III

Multitask prediction of organ dysfunction in the intensive care unit using sequential subnetwork routing

A novel model to label delirium in an intensive care unit from clinician actions

Predicting progression to septic shock in the emergency department using an externally generalizable machine learning algorithm. medRxiv

Predicting 30-day mortality of patients with pneumonia in an emergency department setting using machine-learning models

Application of Machine Learning Techniques to Analyze Patient Returns to the Emergency Department

Using Machine Learning to Predict Bacteremia in Febrile Children Presented to the Emergency Department

An Integrated Approach of Machine Learning and Systems Thinking for Waiting Time Prediction in an Emergency Department

Predicting emergency department orders with multilabel machine learning techniques and simulating effects on length of stay

Artificial Intelligence Applications for COVID-19 in Intensive Care and Emergency Settings: A Systematic Review

Predicting hospital admission at the emergency department triage: A novel prediction model

PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals

MIMIC-IV (Version 1.0)(PhysioNet, 2021)

Agreement and validity of electronic patient self-triage (eTriage) with nurse triage in two UK emergency departments: a retrospective study

Machine-Learning-Based Electronic Triage More Accurately Differentiates Patients With Respect to Clinical Outcomes Compared With the Emergency Severity Index

An Electronic Emergency Triage System to Improve Patient Distribution by Critical Outcomes

Python suite to construct benchmark machine learning datasets from the MIMIC-III clinical database

A new method of classifying prognostic comorbidity in longitudinal studies: development and validation

Comorbidity measures for use with administrative data

Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data

A Python package for standardizing medical data

Multi-layer representation learning for medical concepts

A simple tool to predict admission at the time of triage

Development and validation of an admission prediction tool for emergency departments in the Netherlands

Predicting hospital admission for older emergency department patients: Insights from machine learning

Novel model for predicting inpatient mortality after emergency admission to hospital in Singapore: retrospective observational study

Characteristics of patients who made a return visit within 72 hours to the emergency department of a Singapore tertiary hospital

The emergency severity index triage algorithm version 2 is reliable and valid

Validation of a modified early warning score in medical admissions

Royal College of P. National early warning score (NEWS) 2. Standardising the assessment of acute-illness severity in the NHS

Rapid Emergency Medicine score: a new prognostic tool for in-hospital mortality in nonsurgical emergency department patients

Derivation of a cardiac arrest prediction model using ward vital signs

Connectionist learning procedures

Patient Subtyping via Time-Aware LSTM Networks

LSTM Model for Prediction of Heart Failure in Big Data

A Clinical Prediction Model in Health Time Series Data Based on Long Short-Term Memory Network Optimized by Fruit Fly Optimization Algorithm

Scikit-learn: Machine learning in Python. the Journal of machine Learning research

Deep learning with Keras

AutoScore: A Machine Learning-Based Automatic Clinical Score Generator and Its Application to Mortality Prediction Using Electronic Health Records

AutoScore-Survival: Developing interpretable machine learning-based time-to-event scores with right-censored survival data

AutoScore': An Interpretable Machine Learning-Based Automatic Clinical Score Generator

Early hospital mortality prediction using vital signals

Deep learning for temporal data representation in electronic health records: A systematic review of challenges and methodologies

Shining Light Into the Black Box of Machine Learning

Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead

The impact of the use of the Early Warning Score (EWS) on patient outcomes: A systematic review

Early warning scores for detecting deterioration in adult hospital patients: systematic review and critical appraisal of methodology

GB: Gradient Boosting LR: Logistic Regression LSTM: Long short-term memory MEWS: Modified Early Warning Score MLP: Multilayer Perceptron NEWS: National Early Warning Score REMS: Rapid Emergency Medicine Score RF: Random Forest * The unit of the running time in seconds. eFigure1: Neural network structure of MLP, LSTM and Med2Vec. LSTM: Long short-term memory MLP