key: cord-0078849-8oz96vet
authors: Edelson, Maxim; Kuo, Tsung-Ting
title: Generalizable Prediction of COVID-19 Mortality on Worldwide Patient Data
date: 2022-05-20
journal: JAMIA Open
DOI: 10.1093/jamiaopen/ooac036
sha: c34401059c2b3986a2bcbf09ee412461af6c9302
doc_id: 78849
cord_uid: 8oz96vet

OBJECTIVE: Predicting COVID-19 mortality for patients is critical for early-stage care and intervention. Existing studies mainly built models on datasets with limited geographical range or size. In this study, we developed COVID-19 mortality prediction models on worldwide, large-scale “sparse” data and on a “dense” subset of the data. MATERIALS AND METHODS: We evaluated six classifiers, including Logistic Regression (LR), Support Vector Machine (SVM), Random Forest (RF), Multi-Layer Perceptron (MLP), AdaBoost (AB), and Naive Bayes (NB). We also conducted temporal analysis and calibrated our models using Isotonic Regression. RESULTS: The results showed that AB outperformed the other classifiers for the sparse dataset, while LR provided the highest-performing results for the dense dataset (with Area Under the receiver operating characteristic Curve, or AUC ≈ 0.7 for the sparse dataset and AUC = 0.963 for the dense one). We also identified impactful features such as symptoms, countries, age, and the date of death/discharge. All our models are well-calibrated (p > 0.1). DISCUSSION: Our results highlight the tradeoff of using sparse training data to increase generalizability versus training on denser data, which produces higher discrimination results. We found that covariates such as patient information on symptoms, countries (where the case was reported), age, and the date of discharge from the hospital or death were the most important for mortality prediction. CONCLUSION: This study is a stepping-stone towards improving healthcare quality during the COVID-19 era and potentially other pandemics. Our code is publicly available at: https://doi.org/10.5281/zenodo.6336231. LAY SUMMARY: Our study aims to develop a globally generalizable COVID-19 death prediction tool. To achieve this, we used a large quantity of publicly available COVID-19 patient data collected from across the globe. We also examined the effects of data quality on our results by forming a high missing-value dataset and a low missing-value one, and then comparing their respective results. We used a variety of classification models, and found that patient information on symptoms, countries (where the case was reported), age, and the date of discharge from the hospital or death were the most important for deciding patients’ COVID-19 mortality outcomes. Our models provide a reference for improving the healthcare quality that patients receive during the COVID-19 pandemic era.

COVID-19 has resulted in more than 5.2 million confirmed deaths [1] and spans across almost every country in the world. The World Health Organization (WHO) has declared that the infection fatality ratio (aka the mortality rate) among all infected individuals of COVID-19 converges at 0.5 -1.0%. [2] Thousands of people worldwide continue to be deceased due to COVID-19 [3] and this trend is likely to continue for the foreseeable future as cases continue to spike sporadically, vaccine mandates are fiercely resisted, and new mutations emerge. It is therefore imperative to identify patients with higher risk of fatality, so that healthcare institutions can provide adequate early-stage care and interventions to reduce the risk of COVID-19 mortality.

The Centers for Disease Control and Prevention (CDC) has recognized older age, kidney disease, lung disease, and certain neurological and developmental conditions as factors that can increase a patient's risk for COVID-19 mortality. [4] Based on these factors, several existing studies [5] [6] [7] [8] [9] [10] [11] have proposed pipelines aiming to leverage Artificial Intelligence/Machine Learning (AI/ML) into predicting mortality using patients' data. Most of these studies were performed on smaller datasets collected from one city [5] [6] or on a moderate cohort size (< 5000 patients). [6] [7] [8] [9] [10] These datasets contain detailed/curated clinical information on each patient, contain a low missing value ratio, and are specific to one geographic location. However, in a real, clinical COVID-19 setting, overwhelmed hospitals or intensive care units may not have the resources or time to contact the patients' primary care providers to complete the missing medical history information, and thus the missing data ratio tends to be high. [12] [13] Also, a model built on data from a particular hospital or city may be less pertinent to patients outside of that region. In addition, there are some studies that use a relatively large dataset with the assumption that the dataset is balanced. For example, a recent study [11] used a dataset containing > ~110,000 patients and adopted a pre-processing step to balance the dataset between deceased and discharged patients; this effectively creates a mortality rate of 50%, which may limit the application for real clinical use. While these studies showed the effectiveness of adopting AI/ML methods to predict patient fatality, the data assumptions of (1) a low missing value ratio, (2) a single region, and (3) a balanced mortality rate may hinder the generalizability for real-world clinical applications.

Our goal is to create a model that is generalizable to the world in retrospectively predicting COVID-19 patient mortality using real-world data (1) with a medium to high missing value ratio, (2) with multiple regions encompassed, and (3) without manual balancing of the discharged and deceased patients' relative ratios.

To address the overall goal of model generalizability, we utilized an open-source COVID-19 dataset [14] [15] collected from government sources, scientific papers, and news websites, which contained 2,676,403 COVID-19-confirmed patients from around the world as of 2021/03/31. The Institutional Review Board (IRB) at University of California San Diego (UCSD) approved this study (#190385) . While the earlier versions of this dataset were also used by previous predictive modeling studies, [9] [10] [11] we used a more recent version, which is therefore more complete.

We kept 2,567,823 patients with a known COVID-19 confirmation date ( Figure 1A ) and discarded those without it. The average age was 45 (standard deviation = 20) and the gender was 47.6% female. The countries who are represented among  1% of the total number of patients are the following: India = 11.3% (positive = 5.1%), USA = 4.5% (positive = 87.5%), France = 4.1% (positive = 42.9%), and China = 1.6% (positive = 20.0%). Demographic statistics were obtained from nonunknown data.

We used two subsets of the same dataset: a sparse dataset and a dense dataset. For the sparse dataset, our inclusion criteria for the dataset included (a) patients with a known a COVID-19 confirmation date (i.e., COVID-19+ patients), and (b) patients with known outcomes (i.e., either "deceased" or "discharged"). We manually reviewed the outcome values to combine semantically equivalent ones (e.g., "death" was considered the same as "deceased"); this process was executed by extracting all unique outcomes, and then manually separating them into "deceased", "discharged", or "ambiguous". All observations with "ambiguous" outcomes (e.g., "under treatment") were then discarded. We did not exclude patient data using any other criteria. Based on this inclusion criteria, the sparse dataset contains 104,047 patients, with a deceased (or positive) rate of 5.73% (5,958 positive patients) and a discharged (or negative) rate of 94.27% (98,089 negative patients), as shown in Figure 1B .

To examine the effects of data sparsity (i.e., various levels of missing data) and to cross-examine the results with the sparse dataset, we created the dense dataset ( Figure 1C) , which is a subset of the sparse one. The major difference in the dense dataset is that the fraction of deceased patients is 21.1% or 1,452 patients out of 5,441; just like with the sparse dataset, we left the positive ratio as-is without balancing them. Each observation in the dense dataset was extracted from the sparse one and the basis of inclusion was whether they reported demographic data for age, sex, symptoms, chronic diseases, or optional dates (optional dates are all date features excluding the confirmation date), as shown in Figure 1D . That is, an observation only needed to include one of those fields to merit being placed in the dense dataset. The sparse dataset is a superset of the dense set, meaning that every patient in the dense dataset is also in the sparse dataset.

A high-level overview of our methodology is illustrated in Figure 2 . The following section will introduce our data pre-processing steps in Section 3.3. The six machine learning classifiers that we used will be explored in Section 3.4. Lastly, our validation, calibration, and evaluation framework will be described in Section 3.5.

Both the sparse and the dense datasets contained 33 fields. After manual review, we kept 12 relevant fields ( Table 1 ). The original dataset contained 33 fields, and after manual review, we kept 12 relevant fields. The manual review process included removing the following fields that are potentially irrelevant, redundant, or too specific: ID, City, Province, Latitude, Longitude, Geographic Resolution, Lives in Wuhan, Travel History Location, Reported Market Exposure, Additional Information, Source, Sequence Available, Notes for Discussion, Location, Admin 3, Admin 2, Admin 1, New Country, Admin ID, Data Moderator Initials, and Travel History Binary.

The non-discarded fields and their statistics are summarized in Table 1 . The missing value ratio for most of the included fields is high to reflect the fact that a real-world COVID-19 dataset must be flexible in its assumptions to be generalizable for countries around the entire globe. We preprocessed the 12 data fields, to extract 1 binary outcome label (i.e., whether the patient was deceased/positive or discharged/negative, #1 in Table 1 ) and 55 features. Specifically, we extracted the features from the following fields: 1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  60  Age (#2 in Table 1 ). We split this field into Age Lower and Age Upper because certain ages were given as ranges. For ages given as a single value, we assign both Age Lower and Age Upper to the same age value.

 Sex and chronic disease flag (#3 and #4 in Table 1 ). We converted the sex field into two features, the first to indicate male and the second to indicate female (if both are zero, then the sex was considered unreported). The chronic disease flag field was made into a single binary feature (one if a patient has chronic diseases, otherwise zero). The chronic disease binary flag does not necessarily align with the chronic disease field; that is, even if the chronic disease flag is one (meaning a patient suffers from chronic diseases), the chronic disease field may still contain no data.

 Chronic diseases, symptoms, and country (#5, #6, and #7 in Table 1 , respectively). We manually reviewed the values of these fields to combine equivalent values.

 Date confirmation (#8 in Table 1 ). To enable comparison between dates, we converted this date into an "absolute" day with a reference to the earliest confirmation date available in our entire dataset (i.e., 2020/01/06) inclusive of the last day. For example, if the current patient's COVID-19 confirmation date is 2020/06/03, the absolute days for date confirmation for this patient would be 150. In addition, we also use this field as the "base date" for other types of dates to compute "relative" days (details in the next bullet).

 Date of onset symptoms, date of admission hospital, and date of death or discharge (#9, #10, and #11 in Table 1 , respectively). For these types of dates, we convert each of them into both "absolute" and "relative" days. The process of absolute date conversion is the same as the one for date confirmation (i.e., computing the difference between a specific date and the earliest value for that type of date, inclusive of that specific day). On the other hand, each relative date value is the difference between the patient's date confirmation and the date in question. For example, if the patient's date confirmation was 2020/03/21 and their date of death or discharge was 2020/05/04, their relative days for date of death or discharge would be 45. Note that the date of death or discharge only contains a date without revealing outcome information.

 Travel history dates (#12 in Table 1 ). Many of this specific type of date were given as ranges.

Therefore, we first split this field into Travel History Dates Begin and Travel History Dates End, (similarly to age, we give these two dates the same value if the original data field contains only one value). Then, for each of the "begin" and "end" fields, we further extracted both absolute day and relative day features, resulting in 4 features in total.

We created dummy variable features for categorical fields. Then, we normalized numerical features to [0, 1] using the equation of (current value -minimum value) / (maximum valueminimum value). For fields with missing values, we added a missing indicator feature. For the dense dataset, we further removed all features that were not represented among  5 unique observations, to ensure that one feature would not be unrealistically predictive due to only one observation having that feature. As many of the fields have high missing value ratios, this allowed us to use a much denser subset of the sparse dataset to examine the effects of sparsity. 1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58 59 60

We adopted six classifiers for our COVID-19 mortality prediction (binary classification) task:

Logistic Regression (LR), Support Vector Machine (SVM), Random Forest (RF), Multi-Layer Perceptron (MLP), AdaBoost (AB), and Naive Bayes (NB). All hyper-parameter combinations are shown in Table A1 in Appendix A. For SVM, we used a linear version. [17] [18] 26] For MLP, we set the learning rate to 0.1, number of hidden layers = 1, the number of hidden neurons = 110, the learning rate decay = false, and the threshold for consecutive errors = 20. Appropriate hyperparameter options were discovered through previous studies [16] [17] [18] [19] [20] [21] [22] using similar implementations for the classifiers. While many of the previous studies differed in their datasets and application, we adopted a grid search hyper-parameter tuning approach. We selected the initial values of the grid search based on the previous studies' explored hyper-parameter combinations to optimize the performance of our models. We expanded our grid search hyperparameter values as necessary; we determined necessity based on whether the highestperforming hyper-parameter combination was an edge case in the grid search. We implemented the classifiers using the WEKA library. [23] [24] SVM was implemented using the LibLINEAR API [17] [18] 25] (also WEKA). 

Our validation, calibration and evaluation processes are shown above in Figure 2 . The data was split into three parts: 90% for training/validation (Figure 2B) , the 1 st 5% for calibration ( Figure   2D ), and the 2 nd 5% for evaluation ( Figure 2E) . We used the full Area Under the receiver operating characteristic Curve (AUC) as our evaluation metric for the classifiers. We built and tested our models on an Amazon Web Services virtual machine with 2 vCPUs, 8GB RAM, and 100GB SSD.

(1) For training/validation, we performed 10-fold cross validation for each classifier on the 90% training data to tune the hyper-parameters, averaged the validation results in AUC over 10 folds, and calculated the 95% Confidence Intervals (CIs) of AUC using the best-performing hyperparameter combinations.

(2) For calibration, the best hyper-parameter combination for each classifier was trained on the validation data, 1 st 5%, and then tested on the testing data, 2 nd 5%, which then provided the input for Isotonic Regression. [26] [27] (3) For evaluation, we tested the AUC and computed the Hosmer-Lemeshow (H-Statistic [28] )

Test on the evaluation data. Given the change in the COVID-19 viral variants, it is imperative to further show our model's ability to predict mortality in different epochs of time. Thus, following the CDC's COVID-19 timeline, [29] we split the evaluation data into two parts separated by 2020/05/02 (i.e., when the WHO declared that COVID-19 was a global health crisis). The first part contains all the data before 2020/05/02 and the second part contains all other data (inclusive of 2020/05/02). We split the evaluation data for both the sparse and dense datasets in this manner.

The discrimination results of each classifier on the full evaluation data for both datasets are demonstrated in Figure 3 . Table 2 . We also analyzed the top 10 most important features derived from LR for both the sparse and the dense datasets ( Table 3) and ordered them by their decreasing absolute value of their trained weights. Symptoms, countries, ages, and dates of death or discharge were found to be among the most predictive factors.

The temporal and calibration results are shown in Table 4 . RF outperformed the other classifiers for the "before 2020/05/02" time period, while MLP performed best for the "on-and-after 2020/05/02" epoch for both datasets. All models (including the full evaluation data, evaluation data from before 2020/05/02, and evaluation data from on-and-after 2020/05/02 for both the sparse and the dense datasets) are well calibrated (p > 0.1). The training time measurements on the full 90% training/validation data for each classifier are shown in Figure 4 , and MLP took by far the longest time to train with both datasets (sparse and dense). 1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57 58 59 1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57 58 59 1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57 58 59 For temporal analysis, all our models performed even better for the patients whose confirmation dates were before 2020/05/02 (with RF's AUC being 0.912 as the best result for the sparse dataset, and 0.997 for the dense one). This is significantly higher than the evaluation on the full evaluation dataset (of which the best models only reached AUC ≈ 0.7 for the sparse dataset and 0.963 for the dense one). On the other hand, for the patients whose confirmation dates were onand-after 2020/05/02, all the AUC values are less than that of the results for the full evaluation data on both sparse and dense datasets. This may be a result of the first part of the dataset (data 

There are several limitations for this study:

(1) Using a longitudinal dataset would allow us to showcase the relative dangers that each COVID-19 variants (e.g., alpha, delta, and omicron) and their mixes (e.g., percent delta/alpha or delta/omicron) poses. While exploring such a dataset may have the potential to reveal certain symptoms or risk factors that are correlated with specific variants, we are yet to extend our study to model such type of data.

(2) Calculating the "optimal" decision threshold is often desirable because the threshold is usually problem-specific, and the real threshold value may be more biased towards one outcome over the other. [30] Moreover, this "optimal" decision threshold can potentially affect our precision and recall results, especially for the dense dataset. Additionally, performing calibration near the estimated "optimal" decision threshold can be important because even a minute change in prediction scores near the decision threshold can flip the predicted class.

We are yet to consult with clinical experts to estimate such "optimal" decision threshold, as well as performing subsequent calibration around the estimated threshold and recompute the precision and recall results. reported only five observations, and all five patients died due to COVID-19, then this feature may be receiving a higher absolute weight (importance) in determining patient outcome than it should be. Additionally, the dataset is lacking data from certain countries, which may result in a model that does not directly represent a global sample. Therefore, further investigation of the potential geographical biases in the dataset may be required.

(4) Our hyper-parameter exploration process involved iterating through a grid search algorithm, which is a computationally intensive process. Therefore, alternative hyper-parameter tuning techniques (e.g., random search [31] ) may allow us to search hyper-parameter combinations more efficiently, and thus warrant further studies.

(5) The information on what treatments patients received was not present in our dataset, and therefore the effects of certain treatments on mortality were not compared. We are yet to include such additional information to examine if certain treatments might directly affect the probability of survival. Moreover, we are yet to consult with clinical experts to perform "blind assessment" of the features and mortality used in our data, as well as to create risk groups for stratifying which specific groups of patients are more susceptible to high risk of death.

(6) We adopted more traditional classification methods, prioritizing the simplicity and explainability of the models. On the other hand, advanced techniques such as Deep Learning could also be considered. For example, our tabular datasets can potentially be converted to sequential representations [32] for Recurrent Neural Network (RNNs), [33] or to two-dimensional representations [34] for Convolutional Neural Network (CNNs). [35] These advanced Deep

Learning methodologies for predicting patient mortality warrant further exploration.

(7) There have been several COVID-19 clinical prediction instruments developed since the start of the pandemic including the AIFELL [36] and the 4C [37] scores. The AIFELL score was designed to differentiate between severe and less severe COVID-19 cases in emergency room environments. The 4C score was developed to directly inform clinicians in their decision-making process, as well as to separate COVID-19 hospital admittees into different risk management groups.

We have yet to compare our prediction results with those of these existing tools, or to combine various models to introduce a mortality prediction tool with potentially better predictive capability.

In this study, we demonstrated the feasibility to build generalizable COVID-19 mortality predictive models. To do this, we used a worldwide dataset that contained high missing value ratios for the most of our included fields. We evaluated six classifiers on a COVID-19 dataset featuring patients from around the world and reached an AUC ≈ 0.7 for the sparse dataset and AUC = 0.963 for the dense dataset. This study is a stepping-stone to creating highly generalizable models that can predict mortality for COVID-19 patients with the goal of improving healthcare quality during the COVID-19 era and other future pandemics. 1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58 59 60

The data underlying this article are available in Zenodo, at https://doi.org/10.5281/zenodo. 6336231. The datasets were derived from sources in the public domain: https://github.com/ beoutbreakprepared/nCoV2019.

None declared. 1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  60 https://mc.manuscriptcentral.com/jamiao Manuscripts submitted to JAMIA Open   1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  60 https://mc.manuscriptcentral.com/jamiao Manuscripts submitted to JAMIA Open   1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58 59 60 We then split the dataset to obtain 90% training data. (C) Next, we performed 10-fold cross validation with the training data by feeding our data to our six classifiers.

(D) We calibrated our models using the 1 st 5% of the holdout data. (E) Finally, we evaluated our calibrated models using the 2 nd 5% of the holdout data. 1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58 1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45 46 47 1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  Table A1 . Hyper-parameters explored for the six classifiers adopted in our study. Notation: m is the number of features. 1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57 58 59 1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  60   Table A3 . The 55 feature weights and intercept collected from performing Logistic Regression on the dense dataset using a ridge parameter of 10 2 . The features are ordered by descending absolute weight (except for the intercept). 1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  60 

Centers for Disease Control and Prevention

Estimating mortality from covid-19

World Health Organization. World Health Organization

COVID-19) deaths [Internet]. Our World in Data

People with certain medical conditions

Symptom prediction and mortality risk calculation for COVID-19 using machine learning. medRxiv

Comparing of data mining techniques for predicting in-hospital mortality among patients with covid-19

COVID-19 in adults with congenital heart disease

Common cardiovascular risk factors and inhospital mortality in 3,894 patients with COVID-19: survival analysis and machine learning-based findings from the multicentre Italian CORIST Study. Nutrition, Metabolism and Cardiovascular Diseases

Risk factors for mortality among COVID-19 patients. Diabetes research and clinical practice

Risk factors associated with mortality among patients with novel coronavirus disease

Journal of racial and ethnic health disparities

Predicting mortality risk in patients with COVID-19 using artificial intelligence to help medical decision-making

Patient outcomes following transfer between intensive care units during the COVID-19 pandemic

Key considerations for transferring patients to relief healthcare facilities when responding to community transmission of COVID-19 in the United States

Epidemiological data from the COVID-19 outbreak, realtime case information. Scientific data

Open COVID-19 Data Working Group. Detailed epidemiological data from the COVID-19 outbreak

Ridge estimators in logistic regression

To tune or not to tune: recommending when to adjust SVM hyper-parameters via meta-learning

An empirical study of software metrics selection using Support Vector Machine

Location extraction from tweets. Information Processing & Management

Hyperparameters and tuning strategies for random forest

Out-of-bag estimation of the optimal sample size in bagging. Pattern Recognition

Enhanced version of AdaBoostM1 with J48 Tree learning method

The WEKA workbench. Online appendix for "Data Mining: Practical machine learning tools and techniques

The WEKA data mining software: an update

LIBLINEAR: A library for large linear classification. The Journal of machine Learning research

Isotone optimization in R: pool-adjacent-violators algorithm (PAVA) and active set methods

The WEKA data mining software: an update

A tutorial on calibration measurements and calibration models for clinical prediction models

Centers for Disease Control and Prevention

Beyond discrimination: a comparison of calibration methods and clinical usefulness of predictive models of readmission risk

Random search for hyper-parameter optimization

Recurrent neural networks with nonsequential data to predict hospital readmission of diabetic patients

Recurrent neural networks: design and applications

Converting tabular data into images for deep learning with convolutional neural networks. Scientific reports

Understanding of a convolutional neural network

In2017 international conference on engineering and technology (ICET)

The AIFELL Score as a Predictor of Coronavirus Disease 2019 (COVID-19) Severity and Progression in Hospitalized Patients

Risk stratification of patients admitted to hospital with covid-19 using the ISARIC WHO Clinical Characterisation Protocol: development and validation of the 4C Mortality Score

*89*1#$:'2*

8';".#$D-'89 I'8'+#-*

&(='&.'8$I1