key: cord-0739418-rrab9hj7
authors: Fijačko, Nino; Masterson Creber, Ruth; Gosak, Lucija; Kocbek, Primož; Cilar, Leona; Creber, Peter; Štiglic, Gregor
title: A Review of Mortality Risk Prediction Models in Smartphone Applications
date: 2021-11-04
journal: J Med Syst
DOI: 10.1007/s10916-021-01776-x
sha: 591d662116362decbd2580aee1eed8e1e48f7e5b
doc_id: 739418
cord_uid: rrab9hj7

Healthcare professionals in healthcare systems need access to freely available, real-time, evidence-based mortality risk prediction smartphone applications to facilitate resource allocation. The objective of this study is to evaluate the quality of smartphone mobile health applications that include mortality prediction models, and corresponding information quality. We conducted a systematic review of commercially available smartphone applications in Google Play for Android, and iTunes for iOS smartphone applications. We performed initial screening, data extraction, and rated smartphone application quality using the Mobile Application Rating Scale: user version (uMARS). The information quality of smartphone applications was evaluated using two patient vignettes, representing low and high risk of mortality, based on critical care data from the Medical Information Mart for Intensive Care (MIMIC) III database. Out of 3051 evaluated smartphone applications, 33 met our final inclusion criteria. We identified 21 discrete mortality risk prediction models in smartphone applications. The most common mortality predicting models were Sequential Organ Failure Assessment (SOFA) (n = 15) and Acute Physiology and Clinical Health Assessment II (n = 13). The smartphone applications with the highest quality uMARS scores were Observation—NEWS 2 (4.64) for iOS smartphones, and MDCalc Medical Calculator (4.75) for Android smartphones. All SOFA-based smartphone applications provided consistent information quality with the original SOFA model for both the low and high-risk patient vignettes. We identified freely available, high-quality mortality risk prediction smartphone applications that can be used by healthcare professionals to make evidence-based decisions in critical care environments.

Critical care is a complex and multidisciplinary specialty designed to care for patients with critical illnesses [1] . In intensive care units (ICU), healthcare professionals use mortality prediction models (MPMs) to triage patients [2] [3] [4] , quantify the risk of sepsis and death [5, 6] , and to estimate the cost of medical treatment [7] [8] [9] . MPMs are also used to prognosticate weaning from ventilators, length of ICU stay, mortality, and rate of recovery [10] [11] [12] [13] [14] [15] . The MPM algorithms use physiologic measures [16] within 24 h of admission into the ICU [17] to calculate a risk score [18, 19] . In combination with other patient-level variables, MPMs help healthcare professionals identify patients who will likely need additional intensive care support [20, 21] . The three most common MPMs are: Acute Physiology and Clinical Health Assessment (APACHE), Sequential Organ Failure Assessment (SOFA), and Simplified Acute Physiology Assessment (SAPS) [5, 6] . The choice of MPM depends on the ease of use, effectiveness and reliability in the critical care environment [17] .

Advances in point of care technologies, including smartphones [22] have played a key role in advancing access to healthcare information at the bedside [13] , with critical care medicine at the forefront of these advances [23] . Healthcare professionals have been increasingly using smartphone applications (apps) in practice to provide users easier and faster real-time access to different models, to enhance decision making [24, 25] , and assist in patient monitoring, counseling, data collection, and documentation [26] . In many countries that do not have access to electronic medical records (EMR) that automatically calculate MPM scores, healthcare professionals are using their smartphones to calculate these risk scores using apps [27] [28] [29] .

The rapid global spread of COVID-19 has made smartphone-based MPM models increasingly relevant, especially as hospitals around the world converted operating rooms and medical units to intensive care units to handle patient volume [30] [31] [32] . Using stand-alone apps for risk prediction can support healthcare professionals who are providing inpatient care for patients [30] [31] [32] [33] [34] , especially in the ICUs. Given the shortage of resources and increased risk of sepsis and death, the use of MPMs by healthcare professionals can facilitate clinical decision making [5, 6] . In this systematic review of commercially available apps, we evaluated both overall quality and information quality of MPMs.

The Population Intervention Comparison Output (PICO) [35] framework was used to develop the research question. Population was ICU professionals, the intervention was MPMs in apps for critically ill patients in ICUs, and the output was information quality of MPMs in appsthere was no comparison in this study. Reporting for this systematic review followed Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) recommendations [36] . We used the following search terms: "ICU mortality", "mortality scoring system", "APACHE", "SOFA", "SAPS", "NEWS", "MODS", "LODS" and "medical calculator" for identifying MPMs in apps. Inclusion criteria included being freely available and in English. Apps were excluded if they could not be identified by the name, icon, or description, and require in-app purchases for the MPM.

We conducted the first screening of apps in January 2020, and the secondary screening of apps in June 2020. Each keyword was used separately in Google Play and iTunes. We used iPadium [37] as a simulator for apps in the iTunes stores to be able to conduct the evaluations on a desktop computer. Duplicate apps from each search term in the smartphone app store and simulator were removed after they were copied into Excel spreadsheets independently by two authors (NF, LG). A third author was available for a discussion to help resolve disagreements in scores (GS). Apps which met inclusion criteria were downloaded and evaluated on a Samsung Galaxy S8 phone (Android 9.0) and iPhone 7 (iOS 12.3.1). The apps were sorted into two groups based on whether they included single or multiple MPM medical calculators.

The quality of apps was evaluated with the Mobile Application Rating Scale: user version (uMARS) [38] by two ICU nurses with over five years of critical care experience. The uMARS contains four objective quality scales: engagement, functionality, aesthetics, information quality, and one subjective quality assessment, all of which are graded on a five-point scale. The subjective quality and perceived impact of uMARS was not calculated. Interrater reliability was computed using R version 3.6.0 [39] .

We used the freely available Medical Information Mart for Intensive Care III (MIMIC III), version 1.3 [40] , which contains over 10 years (2001-2012) of de-identified critical care data from 46,520 ICU patients at Beth Israel Deaconess in Boston. Using the MIMIC III database, we developed two patient vignettes representing low and high risk of mortality based on SOFA scores. The SOFA MPM has six different scores, ranging from 0 to 4 for each organ system (respiratory, cardiovascular, hepatic, coagulation, renal, and neurological) [41] . The low-risk patient vignette had a SOFA score for each organ system from 0 to 2, and the high-risk vignette who had a SOFA score of 2 to 4 for Page 3 of 13 107 each organ system. The data were analysed using R, version 3.6.0 [39] .

As reported in our PRISMA flow diagram, we identified 3051 apps. After removing duplicates between keywords, 2758 apps remained. Based on pre-specified exclusion criteria (e.g., inappropriate name, icon, imagery, and images), we excluded 2690 apps. We added 5 apps after secondary app screening. After downloading and testing apps, a total of 33 apps were included in the final analysis ( Fig. 1 and Table 1 ). Inter-rater reliability between raters for the uMARS was acceptable (reviewer one vs. reviewer two; Kalpha = 0.89).

The quality of apps was evaluated using a standardized methodology, including the uMARS tool, and the overall uMARS app quality score was 3.66 (SD = 0.65), which is considered as a moderate overall quality in comparison to the other studies [42] [43] [44] [45] [46] . Overall, 33.3% (n = 11) of the apps were developed by small or medium-sized enterprises, 6.1% (n = 2) by healthcare-related agencies, and 3% (n = 1) by an educational organization (Table 1) . Apps developed by individuals had lower overall quality, compared to apps developed by enterprises, educational or healthcare institutions (M = 3.40; SD = 0.52 vs. M = 3.88; SD = 0.68; p = 0.001). The top-rated app was MDCalc Medical Calculator (4.75), which also had high ratings across all four domains.

We identified 21 different MPMs in apps. The most common MPMs in apps were SOFA (n = 15) and APACHE II (n = 13) (Fig. 2 ).

Less than a half of the apps (n = 13) included multiple MPM calculators (e.g., Nursing calculator with SOFA and MEWS) and the others (n = 20), included a single MPM calculator. Two apps, MDCalc Medical Calculator and Doctor Calci included a total of 10 different MPMs (Table 2) . Single MPM medical calculators had a lower mean app quality score (M = 3.37; SD = 0.57; p = 0.002) compared to multiple MPM medical calculators (M = 4.03; SD = 0.52). Table 3 represents a list of 23 clinical variables in SOFAbased apps (n = 15). Variables were divided into six organ systems, as described by Vincent and colleagues in the SOFA validation study [41] . The lowest number of included variables in apps was 6 (e.g., app 3: SOFA), and the highest was 15 (e.g., app 24: Medical Calculators). Clinical data were most commonly inserted into the SOFA-based app using either a drop-down menu, or they were selected from a pre-populated list.

We evaluated the information quality of each of the SOFA-based apps against the low and high-risk vignettes, where low-risk vignette had a count of 6 points on the SOFA score and the high-risk vignette had a count of 18 points on the SOFA score (Table 4 ). There was greater variation (from < 10% to < 33%) in the risk of mortality in the lower risk vignette (Table 5 ).

Healthcare professionals need accurate, real-time, high quality information to make medical decisions for the most vulnerable patients in critical care environments. Many hospitals worldwide do not have EMRs, which calculate mortality risk prediction; therefore, smartphonebased MPMs are commonly used in clinical practice to predict hospital mortality [27] [28] [29] . This study systematically reviewed both the overall app quality and information quality of MPMs in apps.

Based on the overall uMARS quality assessment, the MPM apps provided moderate information quality. The most commonly downloaded app, MDcalc medical calculator, also had the highest quality rating and the most comprehensive, evidence-based MPM information. The highest rated apps had better visual information and incorporated high-quality scientific evidence [67, 68] . Most apps for mortality prediction are designed to optimize speed and minimize manual data entry (e.g., numeric text input) by using drop-down menus or choosing from a pre-populated list. The limitation of pre-populated values is that they may not include some value ranges, or they may not enable the functionality to toggle between metric and imperials units [69] . For example, MDCalc Medical Calculator resolved this problem by including a choice between units and providing additional alerts for healthcare professionals to check the input values. Relevant to protecting patient privacy, none of the apps included personally identifiable information.

To evaluate information quality, we used the SOFA score, because, in this review, it was the most widely used MPM across all of the apps. The apps generate a SOFA score and percentage for ICU mortality risk, which healthcare professionals interpret and use for medical decision making. When the quality of the MPM apps was evaluated against the two vignettes, the consistency of the app generated a high risk of mortality for the sicker patient and consistent scores for the lower risk patient, but with variability in the risk of mortality. We speculated that the discrepancies were due to differences in mortality algorithms and potential differences in predictions for in-hospital versus 30-day mortality. In addition, there was a wide variety of clinical variables that were used as predictors of mortality in SOFA-based apps, which was particularly relevant to the respiratory and cardiovascular organ systems. For example, when classifying respiratory function, PaO 2 /FiO 2 can be classified individually as the Carrico index or separately. From the perspective of the healthcare professional end user, this can be confusing and a barrier to clinical utility [70] .

Smartphone app stores, like Google Play Store and App Store, should consider adding additional review criteria to include a rating for the scientific information quality of apps. The United Kingdom National Health Service [71] uses a publicly available app review service, Organization for the Review of Care and Health Applications [72], where users can find a list of healthcare apps that have been evaluated by healthcare professionals.

Future research should focus specifically on which apps are most applicable for patients with COVID-19. Paradoxically, among patients with COVID-19 in-hospital deaths are associated with low SOFA scores [17, 73, 74] . As such, MPM apps should include relevant laboratory values such as D-dimer and neutrophil to lymphocyte ratio [75] [76] [77] [78] [79] [80] , to better predict ICU mortality risk for patients with COVID-19. These findings are consistent with recent COVID-19 clinical trials, which also used SOFA and APACHE II most frequently [17, 74, 75, [81] [82] [83] . Some pandemic triage plans and protocols [84] [85] [86] [87] . A few important limitations are recognized in the study. Firstly, our two calculated vignettes based on the MIMIC III database may not represent patients who may or may not be at high or low-risk of mortality. For example, the high-risk patient vignette had a SOFA score of 18 but was not on mechanical ventilation, which most of the higher risk of ICU mortality patients are on. On the other hand, the mean SOFA calculated for lower-risk patient vignettes was similar to hospitalized COVID-19 patients [17, 74, 75, [81] [82] [83] , who do have a high risk of mortality. A better solution for developing vignettes to evaluate the quality of the information provided by apps can be found in published papers where vignettes are based on the mean values of clinical parameters. A second limitation is that there are regional adaptations in the smartphone app stores, and, in this case, the search Fig. 2 Distribution of mortality prediction model in evaluated apps. The "Other" category includes MPMs that were included only once (i.e., APACHE III, APACHE IV, ICD mortality risk score, mSOFA, REMS, SAPS III, SIRS, and Sepsis Assessment) was conducted using a European IP address so that it may have influenced the final set of apps obtained from the search engine. A potential bias of the review was the inclusion of freely available apps; however, this was a deliberate decision to represent available apps that do not pose a financial burden on the end-user, and are accessible to a wide audience of healthcare professionals, inclusive of low-and middle-income countries.

An important application of this work is for the education of healthcare professionals. Combining themes with vignettes based on simulation learning can increase student knowledge, critical thinking, and psychomotor skills for performing a better clinical evaluation of future patients [25] . For the next reviews, researchers should include specific medical calculator's apps because they provide relevant information. 

There is pressing urgency in ICU environments globally for accurate mortality risk prediction. Results from this systematic review support the overall quality and information quality of the MDCalc Medical Calculator for in-hospital mortality risk prediction. The benefits of MDCalc Medical Calculator are that it was developed to be used by healthcare professionals for critically ill adult ICU patients, it is available on both Android and iOS platforms, free, uses validated mortality prediction models, includes high-quality information MPMs, has less time-consuming methods for data entry, includes metric and imperial units, and is regularly updated. The MDCalc and Calculate by QxMD webpages also provide separate COVID-19 smartphone-based MPM calculators, which can be used when atypical physical spaces in healthcare systems are being used as make-shift ICUs. Smartphone MPMs can also be used for non-ICU patients to estimate time to potential clinical deterioration [88] , or for triaging an ICU patient for palliative care services [86, 89] .

Authors' contributions The evaluation presented here was carried out in collaboration with all authors. NF developed a study design and supervised the study. NF, LC, and RMC drafted the manuscript. NF,

LG, GS, and PK conducted data collection and analysis. PC interpreted results from a medical point of view. RMC conducted a comprehensive content review. All authors read, revised, and approved the final manuscript.

Funding This work was supported by the Slovenian Research Agency grant numbers N2-0101, P2-0057, and from the European Commissionfunded Erasmus + project Digital Toolbox for Innovation In Nursing Education (I-BOX), grant number 2019-1-ES01-KA203-065836. We also acknowledge funding for RMC through NIH/NINR (grant number R00NR016275).

Conflicts Of interest/Competing interests None declared.

Critical care -where have we been and where are we going?

Sequential organ failure assessment scoring and prediction of patient's outcome in Intensive Care Unit of a tertiary care hospital

A modified sequential organ failure assessment score for critical care triage

Scoring systems in the intensive care unit: A compendium

Comparison of Different Scoring Systems Used in the Intensive Care Unit

Nurse staffing and patient outcomes in critical care: A concise review

Patient Outcomes and Cost-Effectiveness of a Sepsis Care Quality Improvement Program in a Health System

Implications of ICU triage decisions on patient mortality: A cost-effectiveness analysis

Intensive care unit prognostic scoring systems to predict death: a cost-effectiveness analysis

An Overview of the Predictor Standard Tools for Patient Weaning from Mechanical Ventilation

The SOFA score -Development, utility and challenges of accurate assessment in clinical trials

Sequential organ failure assessment scoring and prediction of patient's outcome in Intensive Care Unit of a tertiary care hospital

The Sequential Organ Failure Assessment score for predicting outcome in patients with severe sepsis and evidence of hypoperfusion at the time of emergency department presentation

Can Serial qSOFA Measurement Aid in Sepsis Identification and Triage Decisions?

Analysis of mortality prognostic factors using model for end-stage liver disease with incorporation of serum-sodium classification for liver cirrhosis complications: A retrospective cohort study

Scoring Systems in Assessing Survival of Critically Ill ICU Patients

Clinical course and outcomes of critically ill patients with SARS-CoV-2 pneumonia in Wuhan, China: a singlecentered, retrospective, observational study

Relationship between nursing documentation and patients' mortality

Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review

Prognosis and prognostic research: what, why, and how ?

Commentary: Prognostic models: clinically useful or quickly forgotten?

Virtually Perfect? Telemedicine for Covid-19

Evaluation of the Anticipated Burden of COVID-19 on Hospital-Based Healthcare Services Across the United States

Mobile Clinical Decision Support Systems and Applications : A Literature and Commercial Review

TBI prognosis calculator: A mobile application to estimate mortality and morbidity following traumatic brain injury

Mobile medical applications in neurology

Development of mobile electronic health records application in a secondary general hospital in Korea

How mobile health technology and electronic health records will change care of patients with Parkinson's disease

Early experiences with mobile electronic health records application in a tertiary hospital in Korea

Transforming ORs into ICUs

Transforming operating rooms into intensive care units and the versatility of the physician anesthesiologist during the COVID-19 crisis

Intensive care during the coronavirus epidemic

How to face the novel coronavirus infection during the 2019 -2020 epidemic : the experience of Sichuan Provincial People ' s Hospital

Novel coronavirus infection during the 2019 -2020 epidemic : preparing intensive care units -the experience in Sichuan Province

Essentials of Nursing Research: Appraising Evidence for Nursing Practice

Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement

iPadian Premium -The Best iOS and iPad simulator

Development and Validation of the User Version of the Mobile Application Rating Scale (uMARS)

R version 3.6. 0. R: A language and environment for statistical computing. R Foundation for Statistical Computing

MIMIC-III , a freely accessible critical care database

The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure

Prostate Cancer Risk Calculator'mobile applications (Apps): a systematic review and scoring using the validated user version of the Mobile Application Rating Scale (uMARS)

Assessing the quality of mobile phone apps for weight management: Usercentered study with employees from a Lebanese university

Nutrition-related mobile apps in the China App Store: Assessment of functionality and quality

The Effects of Gamification and Oral Self-Care on Oral Hygiene in Children: Systematic Search in App Stores and Evaluation of Apps

Assessing the quality of mobile apps used by occupational therapists: Evaluation using the user version of the mobile application rating scale

Sequential assessment of multiple organ dysfunction as a predictor of outcome

Assessment of clinical criteria for sepsis for the third international consensus definitions for sepsis and septic shock (sepsis-3)

APACHE II: a severity of disease classification system

The APACHE III prognostic system: Risk prediction of hospital mortality for critically III hospitalized adults

APACHE-acute physiology and chronic health evaluation: a physiologically based classification system

Acute Physiology and Chronic Health Evaluation (APACHE) IV: Hospital mortality assessment for today's critically ill patients

The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death

Raoyal College of Physicians (2017) National Early Warning Score

Validation of a modified early warning score in medical admissions

Multiple organ dysfunction score: a reliable descriptor of a Page

A simplified acute physiology score for ICU patients

SAPS 3 -From evaluation of the patient to evaluation of the intensive care unit. Part 2: Development of a prognostic model for hospital mortality at ICU admission

A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study

Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis

Systemic inflammatory response syndrome (SIRS): Where did it come from and is it still relevant today?

The third international consensus definitions for sepsis and septic shock (sepsis-3)

The Logistic Organ Dysfunction system: a new way to assess organ dysfunction in the intensive care unit

Mortality Probability Models (MPM II) based on an international cohort of intensive care unit patients

Rapid Emergency Medicine Score can predict long-term mortality in nonsurgical emergency department patients

Prediction of nonarrhythmic mortality in primary prevention implantable cardioverter-defibrillator patients with ischemic and nonischemic cardiomyopathy

Empirical Studies on Usability of mHealth Apps: A Systematic Literature Review

Smartphone applications for cancer patients; what we know about them?

Mobile Applications for Type 2 Diabetes Risk Estimation: a Systematic Review

How do patients evaluate and make use of online health information?

Clinical course and mortality risk of severe COVID-19

Clinical features and short-term outcomes of 221 patients with COVID-19 in Wuhan

Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study

A machine learning-based model for survival prediction in patients with severe COVID-19 infection

ACP risk grade: a simple mortality index for patients with confirmed or suspected severe acute respiratory syndrome coronavirus 2 disease (COVID-19) during the early stage of outbreak in Wuhan

Dysregulation of immune response in patients with COVID-19 in Wuhan, China

Neutrophil-to-lymphocyte ratio and lymphocyte-to-C-reactive protein ratio in patients with severe coronavirus disease 2019 (COVID-19): A meta-analysis

Neutrophil-to-lymphocyte ratio as an independent risk factor for mortality in hospitalized patients with COVID-19

A comparison of existing risk prediction models in patients undergoing venoarterial extracorporeal membrane oxygenation

Coronavirus Disease 2019 (COVID-19): A critical care perspective beyond China

Core Outcome Set for Clinical Trials on Coronavirus Disease 2019 (COS-COVID). Engineering

Outcome reporting from protocols of clinical trials of Coronavirus Disease 2019 (COVID-19): a review

Diagnosis and Management of COVID-19 Disease

The toughest triageallocating ventilators in a pandemic

Managing ICU surge during the COVID-19 crisis: rapid guidelines

Comparison of risk prediction scoring systems for ward patients: A retrospective nested case-control study

Covid-19: Can France's ethical support units help doctors make challenging decisions