key: cord-1033043-qn6kswe9
authors: Castro, Victor M.; Hart, Kamber L.; Sacks, Chana A.; Murphy, Shawn N.; Perlis, Roy H.; McCoy, Thomas H.
title: Longitudinal validation of an electronic health record delirium prediction model applied at admission in COVID-19 patients
date: 2021-11-02
journal: Gen Hosp Psychiatry
DOI: 10.1016/j.genhosppsych.2021.10.005
sha: f4f25cca7f797b1b19c01833999a6f9ae7776296
doc_id: 1033043
cord_uid: qn6kswe9

OBJECTIVE: To validate a previously published machine learning model of delirium risk in hospitalized patients with coronavirus disease 2019 (COVID-19). METHOD: Using data from six hospitals across two academic medical networks covering care occurring after initial model development, we calculated the predicted risk of delirium using a previously developed risk model applied to diagnostic, medication, laboratory, and other clinical features available in the electronic health record (EHR) at time of hospital admission. We evaluated the accuracy of these predictions against subsequent delirium diagnoses during that admission. RESULTS: Of the 5102 patients in this cohort, 716 (14%) developed delirium. The model's risk predictions produced a c-index of 0.75 (95% CI, 0.73–0.77) with 27.7% of cases occurring in the top decile of predicted risk scores. Model calibration was diminished compared to the initial COVID-19 wave. CONCLUSION: This EHR delirium risk prediction model, developed during the initial surge of COVID-19 patients, produced consistent discrimination over subsequent larger waves; however, with changing cohort composition and delirium occurrence rates, model calibration decreased. These results underscore the importance of calibration, and the challenge of developing risk models for clinical contexts where standard of care and clinical populations may shift.

interventions [43] [44] [45] [46] [47] ; however, optimal allocation of these multicomponent prevention resources, as with any other scarce resource, requires a way of identifying individuals at greatest risk for experiencing delirium [48] .

The COVID-19 pandemic, with its associated increased demands on caregivers and hospital clinical resources, has increased the potential consequences of a case of delirium, as critical care services and individual caregivers are already overburdened. It has also increased the difficulty of delivering multicomponent interventions that could reduce delirium risk, by necessitating additional infection control precautions and protective equipment conservation efforts [49] [50] [51] . This combination of circumstances, which developed during care of the various surges of COVID-19 patients, increased the importance of zero-contact stratification of delirium risk.

Although delirium is often underdiagnosed and under coded, electronic health records (EHR) data can be used to study both the epidemiology and biology of delirium [52] [53] [54] [55] [56] [57] [58] [59] .

Building on this potential, EHR data produced during routine care have been repurposed for secondary use in developing clinical prediction models of delirium risk [60] [61] [62] [63] [64] [65] [66] [67] . We previously reported the development and contemporaneous external validation of a machine learning model of delirium risk among patients with COVID-19 presenting during the spring of 2020 [68] . As the characteristics, treatments, and outcomes of COVID-19 have changed over the course of the pandemic, and clinical model predictive accuracy can vary over time, we sought to replicate this model using the later waves of pandemic patients. In other words, we aimed for a replication which is longitudinal with respect to the patients included in the study but not with respect to an individual patient [69] [70] [71] [72] [73] [74] . This work builds on a readily adopted and freely available model, J o u r n a l P r e -p r o o f applicable during emergency department care, based on facts already present in the EHR prior to inpatient hospitalization. It seeks to evaluate the extent to which the predictive quality of the model remained stable over successive waves of pandemic care despite evolving patient characteristics and treatment approaches.

We have previously shown that EHR data can be used at the time of hospital admission to predict delirium diagnosis over the subsequent hospital stay in patients with COVID-19. This prediction is possible using a statistical model developed through machine learning which makes a prediction based on diagnostic history, lab test results, medication orders, and other patient characteristics. This development occurred during the first wave of COVID-19 cases; however, over the intervening waves of COVID-19 cases, the composition of patients and treatment approaches changed a great deal [74, 75] . The aim of this study was to re-validate the model in these subsequent waves of patients to establish whether the quality of statistical predictions had decreased. The overall methodological approach below outlines who was studied, how the data in their health records were made presentable to the statistical model, and finally the multiple approaches taken to establishing the quality of the predictions made by the statistical model.

This study included all adults hospitalized with PCR-confirmed SARS-CoV-2 between June 1, 2020, and April 30, 2021, across six hospitals, including two academic medical centers J o u r n a l P r e -p r o o f and four community hospitals. The original development of the previously published model occurred during the first surge of patients and included those hospitalized prior to May 31, 2020. This validation is longitudinal in that it occurs in a cohort who were cared for afterin calendar timethose in whom the model was original developed and validated. As such, it provides an estimate of how the model would have performed had it been deployed clinically during this time. While this study is longitudinal with respect to cohort and evolution of the pandemic, it is only minimally longitudinal with respect to the individual patients included as the span of individual time considered is merely that between emergency room presentation and hospital discharge.

All data for this study-including the demographic, laboratory, diagnostic, medication, and other clinical data-were extracted from the health system EHR [68] . These included date of birth for calculation of age at admission, home zip code for area deprivation index calculation, body mass index (BMI), and lifetime smoking status [76] . The study protocol was approved by the Mass General Brigham Human Research Committee. No participant contact was required in this study which relied on secondary use of data produced by routine clinical care, allowing waiver of informed consent.

To make statistical predictions from the data in the EHR, those chart-like clinical facts must be systematically reshaped into spreadsheet-like data which are suitable for predictive modeling. This reshaping of EHR data into spreadsheet-like data used the same steps as those of the predictive model's initial development and validation [68] . and low values were extracted as required by the model from the first occurrence within the episode of care (i.e., the first emergency room collection of a given test for those who had multiple instances of the test prior to admission). For count encoded features, medications or diagnostic codes which did not occur were counted zero times. Missing continuous values (e.g., laboratory values) were replaced through medium imputation from the original training sample and thus a prediction was possible on every admission over the study period.

To facilitate cohort description and evaluation of model performance within clinical and demographic substrata, we extracted additional clinical and demographic data from the EHR, including data which occurred after prediction time. These data were not made available to the original model training and played no part in prediction; thus, these evaluative features were unable to contaminate the antecedent predictions (sometimes referred to as 'peeking' forward in time [80] ). Instead, these features were merely used in secondary analysis of predictive accuracy.

Given the importance and frequency of delirium in the critical care setting and delirium superimposed on dementia, dementia diagnosis and subsequent critical care were selected [81] [82] [83] [84] [85] ; however, it is notable that the former could conceptually be excluded at the time of prediction based on prevalent dementia diagnosis whereas the latter eventual incident cases of critical illness could not. To evaluate model performance over demographic strata, we extracted race and gender from structured EHR demographics tables. Recognizing the clinical importance of critical illness and cognitive impairment in delirium, we extracted both subsequent need for critical care and a previously described cognitive impairment feature from the EHR [86] . (3) information which played no part in the prediction or outcome but was merely used for post hoc secondary description of predictive accuracy in specific subgroups (e.g. need for critical care).

We characterized the COVID-19 patient cohort and contexts in which patients received care using descriptive statistics (i.e., counts and percentages for discrete variables, means and standard deviations for continuous measures) stratified by eventual diagnosis of delirium.

Thereafter, we evaluated the discriminationthe extent to which those who went on to develop delirium were consistently predicted as higher risk than those who did notand calibrationthe extent to which all those predicted to have a given risk of delirium did develop delirium at the predicted rateof delirium risk predictions made at the time of admission by reference to the J o u r n a l P r e -p r o o f presence or absence of an eventual coded diagnosis of delirium over the course of the subsequent hospitalization. We evaluated model discrimination, both in the pooled cohort and by clinical and demographic strata, using area under the receiver operating characteristic (ROC) curve [87] .

We evaluated model calibration using the quantile-by-quintile comparison of predicted and observed outcome rate, Spiegelhalter's z statistic, and inspection of calibration plots including both logistic and flexible loess curves [88] [89] [90] [91] . We also calculated the familiar measures of diagnostic testing accuracy (e.g. sensitivity, specificity, negative predictive value, and positive predictive value) by dichotomizing risk predictions at the high-risk threshold identified in the initial model validation cohort [92] . To quantify the expected benefits from intervention in any given case we completed a decision curve analysis of this cohort [93] [94] [95] . Parallel sensitivity check analysis using the NLP augmented delirium outcome identified only a small number of additional cases and the prediction results were consistent with the primary analysis done by reference to the coded diagnosis definition and thus the duplicative NLP results are not shown.

All analysis was conducted using R version 4 [96] .

The cohort included 5,102 patients of whom 716 were diagnosed with delirium (14% Spiegelhalter's Z was significant (Z = -2.29, p = 0.02) and calibration plot showed lower expected outcome rates in those with high predicted risk (Figure 1, right) . As such, in this model those who did develop delirium likely had higher risk scores whereas those patients at a specific risk score were less likely to go on to develop delirium at exactly that predicted rate. Model those with a predicted risk greater than 0.15 as "positive" cases and those less than the cut off as 

This replication of a previously developed, externally validated delirium prediction model based on EHR data available at the time of hospital admission showed comparable discrimination and diminished calibration in this cohort of more than 5,000 COVID-19 patients. [97] ; however, the most directly comparable application of these methods to outcomes other than delirium in patients with COVID-19 produced superior discrimination to that seen in J o u r n a l P r e -p r o o f this replication cohort of delirium as an outcome [98] . This stability over subsequent waves of patients with COVID-19 is reassuring evidence of model robustness given the rapidly evolving patient population and treatment norms over the course of consecutive surges in patient volume [74, [99] [100] [101] [102] . Model calibrationhow closely the observed probability of the outcome matches the predicted probability of the outcome [103, 104] declined in this longitudinal validation cohort relative to that previously reported in the initial cross sectional external validation cohort. Nevertheless, patients with high predicted risk were highly enriched for actual observed cases with 27.7% of cases occurring in the highest decile of predicted risk scores. In light of emerging concern for risk of bias in machine learning models, it is notable that the overall sample AUC fell within or below the confidence interval for AUCs stratified by both race and gender [105] ; however, clinical characteristics such as older age, dementia history, need for critical care, and treatment at a community hospital were associated with reduced discriminative accuracy.

The longitudinal reduction in calibration we observe is consistent with the change in observed delirium rate from 18.9% of patients in whom this model was developed to 14.0% in this subsequent replication cohort. Although the initial cohort from the spring of 2020 surge in whom the model was developed and this subsequent longitudinal replication cohort had similar average ages (62.9 vs 62.7 years respectively) the prevalence of preexisting dementia diagnosis dropped from 11.3% in the initial surge of patients to 5.6% in this replication cohort as did the proportion of patients requiring critical care, which dropped from 22.5% in the initial surge to 15 .6% in this longitudinal cohort. This reduction in critical care is consistent with the declining rates of mechanical ventilation and fluctuating mortality reported elsewhere [73, 99] . It is possible that the reduction in dementia diagnostic history represents a shift from nursing home residents J o u r n a l P r e -p r o o f to community dwelling older adults over subsequent waves [106, 107] . These underlying changes in patient characteristics are an instance of concept drift and demonstrate the associated compromise of internal validity consistent with a rapidly evolving pandemic [108] [109] [110] [111] . That model discrimination held despite this drift is reassuring; however, the impact on calibration highlights the importance of automated methods for continuous model validation and potential approaches to recalibration [112, 113] . In sum, this model could be used to support population risk stratification to direct a scarce resource (e.g., delirium prevention programs) toward those patients who are most likely to be diagnosed with delirium. Whether risk predictions made by this model are adequately calibrated for counseling of individual patients about their specific predicted risk would be context specific but is of secondary relevance to the question of optimal allocation of a limited resource in the setting of pandemic associated scarcity.

This result must be viewed in the context of its strengths and weaknesses. The reliance on EHR data is a strength in that it facilitates deployment to clinical practice without expectation of any change in routine operations or real time clinical data generation; however, the use of an

EHR covering an open system means that care rendered elsewhere, as in another health system, cannot be considered. The absence of relevant prior datafor example, dementia diagnosis made elsewherelikely bias prediction results toward the null, as relevant clinical detail that would indicate increased risk may not be available. Similarly, as delirium is underdiagnosed in routine practice [114] [115] [116] [117] , the use of a clinical diagnosis instead of active bedside screening for model evaluation likely biases results toward more conservative estimates of discrimination. The impact of this discordance between delirium diagnosis and the syndrome of delirium may not be uniformly distributed over delirium motoric subtypes in this COVID-19 population [118, 119] .

Outside of pandemic illness where contact and resource scarcity present unique constraints, it is J o u r n a l P r e -p r o o f possible that predictive approaches of this kind might be used to direct delirium assessments to improve on the current problem of under recognition [120] . While granting that delirium is underdiagnosed in claims data, potentially by an order of magnitude relative to formal assessment studies, secular trends in delirium diagnosis suggest increasing rates of recognition in routine practice. The 14% occurrence rate here is within the range of rates observed in studies of delirium in COVID [7, 8, 58, 114, 121] . Whereas previous research using free text identified more evidence of delirium than reliance on coded diagnosis alone that was not the case in this analysis in which sensitivity checks of our primary coded diagnosis outcome using a previously described outcome which included discharge summary text was not substantially changed and thus is not shown [57] . as dementia, like delirium, is often underdiagnosed [122] [123] [124] [125] and when present delirium superimposed on dementia presents an ambiguous clinical picture [81, 126] .

A key strength of this work is that it is a longitudinal replication using an unmodified previously published predictive model and thus it is protected against overfitting and associated threats to replication [127] [128] [129] . This replication provides important information on the likely robustness of predictive models developed in the early phases of a pandemic for application in later phases of a pandemic illness as would be required in any attempt to do algorithmic allocation [130] ; however, it does so in the setting of a single geographic region, thus this result J o u r n a l P r e -p r o o f represents a replication over time but not a replication over place. Nevertheless, complete replications of this sort are rare and an important assessment of model robustness [131] .

Additionally, this model makes a single risk prediction at the moment of inpatient admission about risk of delirium diagnosis over the course of the subsequent hospitalizationthis is both a strength and a limitation. The strength of this setup is that the prediction is based on facts which were already available in the EHR at the time of admission and thus the reported predictions could be made in the operational clinical EHR and doing so would be well timed to stratify delirium preventative resources; on the other hand, the limitation of this prediction setup is that it is incapable of targeting the specific momentover a potentially lengthy hospitalizationat which the fluctuation course of delirium would produce the greatest symptom burden. Delirium has been formulated in a diathesis stress model within that frame this pre-admission risk prediction is attending primarily to the diathesis [132] . Finally, it is important to note that this model of all patients with COVID-19 did not achieve equivalent discrimination to that produced by models developed for targeted clinical context like surgical and critical care [133, 134] . This the initial phases of a pandemic and applied thereafter; however, close monitoring would be required to assess model performance over subsequent waves in the setting of changing treatment patterns or patient cohorts. Nevertheless, it is reassuring that the performance of this model was stable despite changes in the patient population and treatment approaches over the months following initial model development. Beyond the pandemic setting, this longitudinal replication provides evidence that predictive models applied to EHR data on admission could support optimal allocation of scarce resources like prevention efforts or formal diagnostic efforts to improve recognition.

The IRB approval under which this individual health information was used does not allow redistribution of the clinical data. Table 1 : Logistic regression predictive model coefficients of previously developed and validated model which were used without modificationrefitting, calibrating, or otherwisein this longitudinal replication. Table 2 : Area under the ROC curve (AUC) and 95% confidence intervals by clinical and demographic strata of the cohort for comparison to the overall full cohort AUC as shown in Figure 2 .

J o u r n a l P r e -p r o o f

Neurological manifestations of COVID-19: a systematic review

Neurological manifestations and complications of COVID-19: A literature review

Central nervous system manifestations of COVID-19: A systematic review

Neurological Manifestations of COVID-19: A systematic review and current update

Neurological manifestations of COVID-19: a systematic review and meta-analysis of proportions

A complication of coronavirus disease 2019: delirium

Delirium in COVID-19: epidemiology and clinical correlations in a large group of patients admitted to an academic hospital

Prevalence and risk factors for delirium in critically ill patients with COVID-19 (COVID-D): a multicentre cohort study

The emerging spectrum of COVID-19 neurology: clinical, radiological and laboratory findings

Acute Brain Failure: Pathophysiology, Diagnosis, Management, and Sequelae of Delirium

Mapping the Delirium Literature Through Probabilistic Topic Modeling and Network Analysis: A Computational Scoping Review

Readings on Psychosomatic Medicine: Survey of Resources for Trainees

Prognosis of delirium in elderly hospital patients

Long-Term Effects of Postoperative Delirium in Patients Undergoing Cardiac Operation: A Systematic Review

Clinical phenotypes of delirium during critical illness and severity of subsequent long-term cognitive impairment: a prospective cohort study

Effect of Delirium and Other Major Complications on Outcomes After Elective Surgery in Older Adults

Association of Delirium With Long-term Cognitive Decline: A Meta-analysis

Changes in health-related quality of life and factors predicting long-term outcomes in older adults admitted to intensive care units

Persistent Delirium Predicts Increased Mortality

The long-term cognitive and functional outcomes of postoperative delirium after cardiac surgery

The importance of delirium: economic and societal costs

One-year health care costs associated with delirium in the elderly population

Delirium predicts 12-month mortality

Delirium in older medical inpatients and subsequent cognitive and functional status: a prospective study

Long-term cognitive impairment after critical illness

Delirium is a robust predictor of morbidity and mortality among critically ill patients treated in the cardiac intensive care unit

Outcome of delirium in critically ill patients: systematic review and meta-analysis

A hospital-wide evaluation of delirium prevalence and outcomes in acute care patients -a cohort study

Poorer outcomes and greater healthcare costs for hospitalised older people with dementia and delirium: a retrospective cohort study

The Cost of ICU Delirium and Coma in the Intensive Care Unit Patient

What does delirium cost?

Longterm outcome of delirium during intensive care unit stay in survivors of critical illness: a prospective cohort study

Impact of delirium on clinical outcome in critically ill patients: a meta-analysis

Delirium in critically ill patients: impact on long-term health-related quality of life and cognitive functioning

The delirium experience: delirium recall and deliriumrelated distress in hospitalized patients with cancer, their spouses/caregivers, and their nurses

Impact of delirium and recall on the level of distress in patients with advanced cancer and their family caregivers

The Caregiver Burden of Delirium in Older Adults With Alzheimer Disease and Related Disorders

Preliminary investigation of predictors of distress in informal caregivers of patients with delirium superimposed on dementia

Delirium superimposed on dementia: A quantitative and qualitative evaluation of informal caregivers and health care staff experience

Terminal delirium: recommendations from bereaved families' experiences

Family and healthcare staff's perception of delirium

Experiences, understandings and support needs of family carers of older patients with delirium: a descriptive mixed methods study in a hospital delirium unit

Hospital Elder Life Program: Systematic Review and Meta-analysis of Effectiveness

Effectiveness of multicomponent nonpharmacological delirium interventions: a meta-analysis

Preventing Delirium Takes a Village: Systematic Review and Meta-Analysis of Delirium Preventive Models of Care

Evaluating the effects of the pharmacological and nonpharmacological interventions to manage delirium symptoms in palliative care patients: systematic review. Current Opinion in Supportive & Palliative Care

Effect of the Tailored, Family-Involved Hospital Elder Life Program on Postoperative Delirium and Function in Older Adults: A Randomized Clinical Trial

Risk stratification of patients admitted to hospital with covid-19 using the ISARIC WHO Clinical Characterisation Protocol: development and validation of the 4C Mortality Score

COVID-19: ICU delirium management during SARS-CoV-2 pandemic

Delirium: a missing piece in the COVID-19 pandemic puzzle

COVID-19 inpatients with psychiatric disorders: Real-world clinical recommendations from an expert team in consultation-liaison psychiatry

Documentation of delirium in the VA electronic health record

A novel model to label delirium in an intensive care unit from clinician actions

Effect of delirium motoric subtypes on administrative documentation of delirium in the surgical intensive care unit

A Chart-Based Method for Identification of Delirium: Validation Compared with Interviewer Ratings Using the Confusion Assessment Method

Evaluation of algorithms to identify delirium in administrative claims and drug utilization database: Delirium Identification in Claims Data

Enhancing Delirium Case Definitions in Electronic Health Records Using Clinical Free Text

Characterizing and predicting rates of delirium across general hospital settings

Genome-wide association identifies a novel locus for delirium risk

Stratified delirium risk using prescription medication data in a state-wide cohort

Delirium misdiagnosis risk in psychiatry: a machine learninglogistic regression predictive algorithm

Risk prediction models for delirium in the intensive care unit after cardiac surgery: a systematic review and independent external validation

Evaluation of emergency department derived delirium prediction models using a hospital-wide cohort

Systematic review of prediction models for delirium in the older adult inpatient

Estimating patients' risk for postoperative delirium from preoperative routine data -Trial design of the PRe-Operative prediction of postoperative DElirium by appropriate SCreening (PROPDESC) study -A monocentre prospective observational trial

Delirium prediction in the intensive care unit: comparison of two delirium prediction models

Multinational development and validation of an early prediction model for delirium in ICU patients

Development and External Validation of a Delirium Prediction Model for Hospitalized Patients With Coronavirus Disease

Trends in Patient Characteristics and COVID-19 In-Hospital Mortality in the United States During the COVID-19 Pandemic

Classification of the drifting data streams using heterogeneous diversified dynamic class-weighted ensemble

Assessment of Time-Series Machine Learning Methods for Forecasting Hospital Discharge Volume

Improved concept drift handling in surgery prediction and other applications

Estimating the infection-fatality risk of SARS-CoV-2 in New York City during the spring 2020 pandemic wave: a model-based analysis

Dexamethasone in Hospitalized Patients with Covid-19

The Effect of Vaccination Rates on the Infection of COVID-19 under the Vaccination Rate below the Herd Immunity Threshold

Introduction of an Area Deprivation Index Measuring Patient Socio-economic Status in an Integrated Health System: Implications for Population Health

Healthcare Cost and Utilization Project (HCUP). Clinical Classifications Software (CCS) for ICD-10-PCS. Agency for Healthcare Research and Quality

Utilizing RxNorm to support practical computing applications: Capturing medication history in live electronic health records

Validation of a risk stratification tool for fall-related injury in a state-wide cohort

The "peeking" effect in supervised feature selection on diffusion tensor imaging data

Delirium Superimposed on Dementia: A Systematic Review

Consequences of Not Recognizing Delirium Superimposed on Dementia in Hospitalized Elderly Individuals

Recognizing Delirium Superimposed on Dementia: Assessing Nurses' Knowledge Using Case Vignettes

Prevalence and Symptoms of Delirium Superimposed on Dementia

Long-Term Outcomes in ICU Patients with Delirium: A Population-based Cohort Study

Stratifying risk for dementia onset using large-scale electronic health record data: A retrospective cohort study

An introduction to ROC analysis

Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors

A calibration hierarchy for risk models was defined: from utopia to empirical data

Probabilistic prediction in patient management and clinical trials

Applied logistic regression

Prediction Modeling Methodology

Decision curve analysis: a novel method for evaluating prediction models

Extensions to decision curve analysis, a novel method for evaluating diagnostic tests, prediction models and molecular markers

Using relative utility curves to evaluate risk prediction

R: A Language and Environment for Statistical Computing

Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal

Laboratory Findings Associated With Severe Illness and Mortality Among Hospitalized Individuals With Coronavirus Disease 2019 in Eastern Massachusetts

Improved outcomes over time for adult COVID-19 patients with acute respiratory distress syndrome or acute respiratory failure

Decreasing median age of COVID-19 cases in the United States-Changing epidemiology or changing surveillance?

Trends in COVID-19 Risk-Adjusted Mortality Rates

Improving Survival of Critical Care Patients With Coronavirus Disease 2019 in England: A National Cohort Study

Beyond discrimination: A comparison of calibration methods and clinical usefulness of predictive models of readmission risk

The Importance of Calibration in Clinical Psychology

Dissecting racial bias in an algorithm used to manage the health of populations

Epidemiology of Covid-19 in a Long-Term Care Facility in King County, Washington

Estimates of COVID-19 Cases and Deaths Among Nursing Home Residents Not Reported in Federal Data

Big data. The parable of Google Flu: traps in big data analysis

Implications of non-stationarity on predictive modeling using EHRs

A Review of Challenges and Opportunities in Machine Learning for Health

Calibration drift in regression and machine learning models for acute kidney injury

A nonparametric updating method to correct clinical prediction model drift

Driftage: a multi-agent system framework for concept drift detection

Underreporting of Delirium in Statewide Claims Data: Implications for Clinical Care and Predictive Modeling

Documentation of delirium in elderly patients with hip fracture

Incidence, prevalence, and under-recognition of delirium in urology patients

Delirium documentation in hospitalized pediatric patients with cancer

Incidence and Prevalence of Delirium Subtypes in an Adult ICU: A Systematic Review and Meta-Analysis*

Motoric subtypes of delirium in mechanically ventilated surgical and trauma intensive care unit patients

Dealing with missing delirium assessments in prospective clinical studies of the critically ill: a simulation study and reanalysis of two delirium studies

Delirium in an adult acute hospital population: predictors, prevalence and detection

Underdiagnosis of Dementia: an Observational Study of Patterns in Diagnosis and Awareness in US Older Adults

Prevalence and determinants of undetected dementia in the community: a systematic literature review and a meta-analysis

Detection of probable dementia cases in undiagnosed patients using structured and unstructured electronic health records

Underdiagnosis of dementia in primary care: Variations in the observed prevalence and comparisons to the expected prevalence

Delirium Diagnostic and Classification Challenges in Palliative Care: Subsyndromal Delirium, Comorbid Delirium-Dementia, and Psychomotor Subtypes

Replication is fundamental, but is it common? A call for scientific selfreflection and contemporary research practices in gambling-related research

Increasing the Reproducibility of Science through Close Cooperation and Forking Path Analysis

Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking

Existing Crisis Standards of Care Triage Protocols May Not Significantly Differentiate Between Patients With Coronavirus Disease 2019 Who Require Intensive Care

External validation of new risk prediction models is infrequent and reveals worse prognostic discrimination

Delirium in elderly people

DELirium Prediction Based on Hospital Information (Delphi) in General Surgery Patients

Development and validation of PRE-DELIRIC (PREdiction of DELIRium in ICu patients) delirium prediction model for intensive care patients: observational multicentre study

Funding: This study was funded by the National Institute of Mental Health [1R01MH120991,