key: cord-0306838-w587b059 authors: Sonthalia, S.; Muharrom, M. A.; Sani, L. L.; Herlinda, O.; Bella, A.; Swashtika, D.; Hadisoemarto, P.; Saminarsih, D.; Lutungan, N.; Irwanto, A.; Taher, A.; Greenstein, J. L. title: COVID-19 Likelihood Meter: a machine learning approach to COVID-19 screening for Indonesian health workers date: 2021-10-18 journal: nan DOI: 10.1101/2021.10.15.21265021 sha: 7aee640485c3947626e4d1f5b1bb961e52b3fa66 doc_id: 306838 cord_uid: w587b059 The COVID-19 pandemic poses a heightened risk to health workers, especially in low- and middle-income countries such as Indonesia. Due to the limitations to implementing mass RT-PCR testing for health workers, high-performing and cost-effective methodologies must be developed to help identify COVID-19 positive health workers and protect the spearhead of the battle against the pandemic. This study aimed to investigate the application of machine learning classifiers to predict the risk of COVID-19 positivity (by RT-PCR) using data obtained from a survey specific to health workers. Machine learning tools can enhance COVID-19 screening capacity in high-risk populations such as health workers in environments where cost is a barrier to accessibility of adequate testing and screening supplies. We built two sets of COVID-19 Likelihood Meter (CLM) models: one trained on data from a broad population of health workers in Jakarta and Semarang (full model) and tested on the same, and one trained on health workers from Jakarta only (Jakarta model) and tested on an independent population of Semarang health workers. The area under the receiver-operating-characteristic curve (AUC), average precision (AP), and the Brier score (BS) were used to assess model performance. Shapley additive explanations (SHAP) were used to analyze feature importance. The final dataset for the study included 3979 health workers. For the full model, the random forest was selected as the algorithm of choice. It achieved cross-validation mean AUC of 0.818 {+/-} 0.022 and AP of 0.449 {+/-} 0.028 and was high performing during testing with AUC and AP of 0.831 and 0.428 respectively. The random forest model was well-calibrated with a low mean brier score of 0.122 {+/-} 0.004. A random forest classifier was the best performing model during cross-validation for the Jakarta dataset, with AUC of 0.824 {+/-} 0.008, AP of 0.397 {+/-} 0.019, and BS of 0.102 {+/-} 0.007, but the extra trees classifier was selected as the model of choice due to better generalizability to the test set. The performance of the extra trees model, when tested on the independent set of Semarang health workers, was AUC of 0.672 and AP of 0.508. Our models yielded high predictive performance and may have the potential to be utilized as both a COVID-19 screening tool and a method to identify health workers at greatest risk of COVID-19 positivity, and therefore most in need of testing. Since the first confirmed case of COVID-19 in Indonesia in March 2020, there have been 1.8 million confirmed cases and more than 50 thousand deaths resulting from COVID-19 infection [1] . Health workers in Indonesia are at a high risk of COVID-19 exposure and infection due to the nature of the profession, with 654 COVID-19 deaths recorded by January 2021 [2, 3] . The COVID-19 pandemic is placing an enormous burden on Indonesia's public health and economy, especially due to Indonesia having the highest fatality rate for health workers in Asia [4] . Despite implementation of large-scale social restrictions at the national and regional level, vaccination of high-risk population groups, and advocation for the usage of personal protective equipment (PPE) [5] , Indonesia still had the highest daily and cumulative COVID-19 cases in The Association of Southeast Asian Nations (ASEAN) in September 2021 [6] . Furthermore, due to the often-asymptomatic nature of COVID-19 infection [7] , efforts to prevent COVID-19 transmission were constrained by the ability to immediately detect and isolate the infected people [8] [9] [10] . Mass testing by reverse transcription-polymerase chain reaction (RT-PCR), the gold standard of COVID-19 diagnostic testing, remains one of the key measures to reduce transmission of COVID-19 [11] . However, the implementation of mass RT-PCR testing is limited in developing countries such as Indonesia due to financial, capital, and logistical constraints [12, 13] . Despite the rapidly increasing COVID-19 cases since the Eid al-Fitr holiday in May 2021, some regions of Indonesia have been facing limited reagent supply and inadequate laboratory capacity to provide sufficient testing. In June 2021, Indonesia had the second-lowest testing rate in Southeast Asia with only 7.5 tests per confirmed case, far below the World Health Organization (WHO) recommendation of 10-30 tests per confirmed case [14] [15] [16] . The healthcare system of Indonesia has further been weakened by surges in case numbers and patients requiring hospitalization, leading to depleted medical supplies [17] . Furthermore, in the event of a surge, one study assessed that several provinces in Indonesia would likely have suboptimal diagnostic capabilities even if using rapid diagnostic technologies in referral hospitals [18] . The limitations of mass RT-PCR implementation in Indonesia underscore the development of a COVID-19 detection method that is accurate, affordable, and accessible to users with minimum equipment and personnel required to prioritize PCR testing for health workers. Rapid diagnostic antigen testing is another common testing modality that is less accurate than RT-PCR [17] . While antigen testing provides rapid results, implementation of mass testing may still be limited by the coordination of stakeholders who may have to balance resources. A free testing or screening modality would encourage hospitals and health systems to test their workers more frequently and can be implemented in resource-strapped communities. Machine learning tools can achieve these goals and have already shown promise in several countries such as the United States, China, Israel, and Slovenia [19] [20] [21] [22] . Several machine learning-based models have already been developed for COVID-19 screening using data from sources such as Computed Tomography (CT) scans [19, 23] , clinical symptoms [20] [21] , and laboratory tests [24] [25] [26] . These tools' sensitivity and specificity values are high, ranging from 0.86-0.93 and 0.56-0.98 respectively. Currently, most machine learning COVID-19 screening tools have been deployed in technologically advanced countries [19-21, 24, 26] . Since most of the available models used data from hospitalized patients, the tools may not be effective for COVID-19 screening for health workers, due to differences in available features between hospitalized patients and health workers. Hospitalized patients are highly likely to have different symptomatology, behavioral tendencies, and PPE usage requirements than health workers who are not hospitalized, the target population of this study. We aim to apply machine learning algorithms to develop software tools to augment COVID-19 screening for Indonesian health workers. This method is expected to ease the burden in Indonesia's healthcare system caused by COVID-19 through the implementation of a fast, accessible, and widespread screening methodology that can allow for accurate triage and systematic allocation of RT-PCR testing for health workers. The study was designed as a cross-sectional observational study with a total of 3979 health workers, including medical professionals and non-medical professions working at healthcare facilities. The data was collected between January 20 and September 15, 2021, at forty-four healthcare provider locations: 15 healthcare providers in Greater Jakarta Area, 1 hospital in Bandung city, and 28 healthcare providers in Semarang 1 . The hospitals and healthcare providers were selected through online recruitment, recommendations from medical associations, and a partnership agreement with the following inclusion criteria: 1) the ability to conduct swab collection for RT-PCR testing by trained health workers, 2) the presence of healthcare or nonhealthcare staff with COVID-19 symptoms or close contact with COVID-19 patients, and 3) the support from the health facility management to participate in the research. The proportion of respondents from Jakarta was 3477 (87.4%) and 502 from Semarang (12.6%). The hospitals had the authority to recommend that a member of their staff to be included in the study by following this criterion: either (1) had close contact with at least one COVID-19 patient within the last fourteen days or (2) developed COVID-19 related symptoms within the last fourteen days. For each respondent, we collected oropharyngeal or nasopharyngeal swab specimens for RT-PCR as well as data for COVID-19 symptoms, comorbidities, COVID-19 protective behaviors, working conditions, and COVID-19 vaccination status through a selfadministered questionnaire. All oropharyngeal/nasopharyngeal swab specimens from respondents in participating healthcare providers in Greater Jakarta Area and Bandung were tested at Primaya Hospital, East Bekasi, while specimens from respondents in participating healthcare providers in Semarang were tested at Diponegoro National Hospital. a t a f r o m 2 8 h e a l t h c a r e p r o v i d e r s i n S e m a r a n g c a m e f r o m s e v e r a l s u r r o u n d i n g h e a l t h c a r e p r o v i d e r s i n C e n t r a l J a v a , i n c l u d i n g : 5 2 . The survey questions included behavioral (protective, social, and travel) tendencies, COVID-19 vaccination status, working conditions, symptoms, comorbidities, and level of COVID-19 exposure and interaction with infected patients at health facilities. The dependent variable in this study was the result of the COVID-19 RT-PCR test taken within three days of filling the survey. Respondents with inconclusive RT-PCR results were not included in the processed dataset. Behavioral questions were chosen based on general and medical workerspecific risk factors identified in the current literature and encompassed handwashing, maskwearing, PPE adherence, social distancing, and domestic and foreign travel tendencies. Handwashing behaviors were assessed by the level of adherence to the six-step handwashing protocol [27, 28] . Mask-wearing and social distancing behaviors were assessed according to current WHO guidelines [29, 30] . To predict COVID-19 diagnosis in our cohort, we trained and evaluated several machine learning classification algorithms, including random forest [31] , extra trees classifier [32] , and model ensembles. These were implemented using the scikit-learn Python library [33] , while XGBoost [34] was implemented using scikit-learn compatible packages in Python. These models were chosen after experiments with various algorithms, such as optimization and deep learning methods, during preliminary modeling. Preprocessed respondent features were used as inputs for each model, generating an output prediction risk score with a value between 0 and 1. The output was then converted to a class label by a thresholding function. Hyperparameters were tuned and chosen using the random search optimization method in scikit learn [35] . Feature selection was implemented using the sequential feature selection method in scikit learn. Model performance was analyzed and interpreted using the area under the receiving-operating characteristic curve (AUC) and area under the precision-recall (PR) curve, average precision (AP), while model calibration was assessed using the Brier Score (BS) for each model. Feature importance and model interpretability was assessed using Shapley additive explanations (SHAP) from the SHAP package [36] in Python. Two sets of models were developed: one using respondents from health facilities in both Jakarta and Semarang (full model), and one on respondents from Jakarta health facilities only (Jakarta model). The full model was trained with stratified 5-fold cross-validation on 80% of the dataset and tested on the remaining 20%. The Jakarta model was tested on inputs from the Semarang health workers and was developed to assess the predictive capability of this approach in an independent population of health workers within Indonesia. Due to class imbalance, adaptive synthetic oversampling (ADASYN) [37] was used during training for the positive class before validation for both modeling approaches. Data from 3,979 health workers were included in the final dataset for model building. A summary of cohort demographics and survey responses is provided in Table 1 . Approximately 74% of respondents were female and the average age was 30 years old. As health workers are currently the priority group for receiving the COVID-19 vaccine in Indonesia, 86% of respondents had received at least the first dose of the COVID-19 vaccine. Regarding protective behaviors in the prior month, most respondents indicated they had been trained in PPE standards (98%) and six-step hand washing techniques (97%). Around 18% of respondents were currently self-isolating after having close contact with COVID-19 patients and 30% were involved in aerosol-generating procedures on COVID-19 patients. Furthermore, approximately half of positive cases were currently self-isolating. Additionally, around 73% and 59% of respondents reported having activity in a closed room and having outdoor activities at least 1-3 times a month, respectively. This study also found that 76% of respondents always used a mask outside the home, 53% always avoided shaking hands, and 53% always maintained physical distance. In terms of the use of public transportation, 83% of respondents never used mass public transportation and 76% never used door to door transportation. Approximately 80% of positive cases were symptomatic. Cough (48.52%), headache (44.43%), runny nose (39.34%), chills (38.36%), and fever (37.54%) were the five most reported symptoms among those who had COVID-19 positive test results. 90% of respondents stated that they did not have any comorbidities. Lung disease (3.11%), hypertension (2.46%), and pregnancy (2.30%) were the most reported comorbidities among those who were COVID-19 positive. The Jakarta model was developed to investigate the model performance on a geographically independent test set, which was the data from the Semarang health workers. The full model contains data from both cities and is thus a model built on a wider population and would be generalizable to a more heterogeneous testing population. The Jakarta model was trained and cross-validated on health workers from Jakarta only and was tested on respondents from Semarang. The test set was composed of 12.6% of the entire dataset. Figure 1 displays the test set receiver operating characteristic (ROC) curvd, precisionrecall (PR) curve, the cross-validation calibration curve as well as both cross validation and test sensitivity, specificity, positive-predictive value (PPV), negative-predictive value (NPV), and the F1 score of several algorithms on the Semarang data. The prevalence of COVID-19 positive health workers in the validation and testing datasets was 29.5%. The random forest classifier had the best predictive performance during cross-validation, followed by the voting classifier, composed of an ensemble of a XGBoost classifier and a random forest classifier. The mean AUC for random forest on the cross-validation sets was 0.824 ± 0.008, followed by that of the voting classifier (0.822 ± 0.007) and was greater than those of XGBoost (0.786 ± 0.014) and extra trees classifier (0.815 ± 0.008) (Fig. S1A) . The random forest produced the best AP (0.397 ± 0.019), followed by voting classifier (0.395 ± 0.022), extra trees (0.371 ± 0.022), and XGBoost (0.335 ± 0.030) (Fig. S1B) . The calibration curves showed all models were well-calibrated, but calibration might be greatly improved if the predicted risk is scaled down (Fig. 1C) . The voting classifier produced the lowest brier score of 0.096 ± 0.008. When testing the models on the Semarang dataset, the voting classifier had the best AUC of 0.674 followed by random forest, extra trees, and XGBoost with AUCs of 0.673, 0.668, and 0.643 respectively (Fig. 1A) . XGBoost had the best AP of 0.533, followed by that of the voting classifier (0.514), random forest (0.513) and extra trees classifier (0.510) (Fig. 1B) . The voting classifier model was selected as the best classifier due to producing the best test AUC and second-best F1 score. Figure 2 displays a SHAP summary plot for the voting classifier Jakarta model. Being asymptomatic was the most important feature ranked by the SHAP analysis and were strongly associated with lower predicted risk of COVID-19 infection. Of COVID-19 symptoms, cough, fever, and chills ranked among the most important features. Of the behavioral questions, frequent handwashing in different scenarios, such as after touching or disposing trash, after shaking hands with people or after touching animals was associated with lower risk of COVID-19 infection. Health workers that wore shoe covers, hazmat suits, face shields and medical gloves regularly were also at lower risk of COVID-19 diagnosis and these behaviors ranked among the most important features. Furthermore, health workers that wore PPE such as medical gloves when handling COVID-19 specimens were also at lower risk of COVID-19. PPE such as masks and gloves have been shown to reduce risk for COVID-19 infection [36] . Furthermore, the higher average density of people in the room most frequented by the health worker was highly associated with COVID-19 positivity by the Jakarta model. Health workers that frequently work in crowded areas may be at increased risk for COVID-19. Figure 3 displays test set ROC and PR curves as well as cross validation calibration curves for all the algorithms applied to the full patient cohort training set. The prevalence of positive classes in the validation and testing datasets was 15.3%. The best performing model during 5-fold stratified cross-validation was the random forest with the highest AUC (0.818 ± 0.022), followed by voting classifier, which is an ensemble of a random forest and extra trees classifier (0.817 ± 0.024), extra trees (0.813 ± 0.026), and the XGBoost classifier (0.795 ± 0.027) (Fig. S2A) . The random forest also had the best AP (0.449 ± 0.028) followed by that of the voting classifier (0.446 ± 0.034), XGBoost (0.440 ± 0.017) and extra trees (0.438 ± 0.039) (Fig. S2B) . The calibration curves, derived from validation folds of 5-fold cross validation, showed all models were relatively well-calibrated, with the voting classifier and extra trees classifier producing the lowest mean brier score of 0.116 ± 0.005 (Fig. 3C) . On the held-out test set, random forest produced the highest AUC of 0.831 as compared to voting classifier (0.828), extra trees (0.822), and XGBoost (0.776) (Fig. 3A ). The random forest had the highest AP (0.428) compared to voting classifier (0.426), XGBoost (0.410) and extra trees (0.402) (Fig. 3B) . The random forest was overall the best performing algorithm for the full model, due to high operating point predictive performance on the held-out test set with sensitivity, specificity and F1 score of 0.787, 0.770 and 0.515 respectively. Other performance metrics such as the positive and negative predictive values are displayed in Figure 3D . Feature importance and model explainability were assessed through SHAP values from the training set predictions. Figure 4 displays the SHAP summary plot for the top 20 most important features in the full model. Like the Jakarta model, being asymptomatic and not handling COVID-19 specimens were ranked among the most important features for reduced COVID-19 risk. Surprisingly, health workers that performed aerosol-generating procedures on COVID-19 patients, such as tracheal intubation, non-invasive ventilation, tracheostomy, and swab collection among others, were also at lower risk of infection. This observation is likely attributable to more stringent PPE requirements and behavioral measures for health workers working closely with COVID-19 patients. Fever was highly predictive of COVID-19 infection, as well as other common COVID-19 symptoms such as chills and cough. Health workers that wore surgical hoods and medical gloves during work were at lower risk of COVID-19 infection. Inadequate access to PPE has already been reported in Indonesia [38, 39] . Employees that regularly wore a hazmat suit at the health facility were at greatly reduced risk of being COVID-19 positive. Health facilities that reported a shortage of hazmat suits in Indonesia instructed health workers to wear thin plastic raincoats while transporting COVID-19 patients [40]. These limitations highlight the importance of a sufficient supply of PPE for the health workforce during the pandemic. This study investigated the capability of information collected via questionnaire including protective behavioral tendencies, COVID-19 vaccine status and information, symptoms, comorbidities, and working conditions among others to predict COVID-19 diagnosis for Indonesian health workers. We demonstrated that machine learning methods, specifically the random forest, XGBoost, extra trees and ensemble algorithms of these models were able to predict COVID-19 diagnosis with high performance. Models built on symptom data only have been found to be insufficient for application to clinical practice [41], perhaps due to the lack of behavioral, comorbidity, or other critical risk factors for COVID-19 infection. The models developed here incorporate many of these factors, and specifically include inputs on behavioral tendencies that are important in COVID-19 transmission and infection. The importance of behavioral tendencies is demonstrated by their high ranking in the SHAP summary plots for both models (Figs. 2 and 4) , where behavioral tendencies ranked amongst the most important features for the models. Associations between features and COVID-19 positivity revealed by the models do not necessarily indicate causal relationships and must be interpreted in this context. The full model performed well during training and testing with the random forest and was therefore chosen as the classifier of choice for the full model. The random forest performed better in terms of both AUC and AP than the extra trees, XGBoost and voting classifier models during 5-fold cross validation and on the test set. The best performing Jakarta model had poorer predictive performance on the test set as compared to that of the full model. The poorer performance of the Jakarta model on the test set of Semarang health workers is likely due to potential drifts and biases in the testing data relative to the training data. The purpose of this model was to assess performance on an independent population of health workers relative to the training data for the model. Although this model displays high test set sensitivity, reduced specificity compared to the full model may reduce adoption and applicability of models built on city-specific data to data from health workers in other cities. The full model displays generalizable performance to unseen data and may have higher potential to generalize to wider populations of health workers within Indonesia. Improved protective behavioral tendencies, such as handwashing and sanitizing [42] , were also associated with lower COVID-19 risk as reasoned by the models. The models we built are inclusive of many behavioral risk factors of COVID-19 infection among a myriad of other inputs and perform well on unseen data. Mass health worker testing, adequate PPE supply, selfisolation and quarantine, and education are the main recommendations previously issued for saving the frontline health workers during the pandemic [43] . CLM and our results may have the potential to assist in most of these guidelines, through assessing COVID-19 risk, prioritizing testing and thus isolation measures, and demonstrating the importance of utilizing PPE. Furthermore, CLM can be used to reduce costs, as the survey can be provided to health facilities at no cost. This study has several limitations. The first limitation is the self-reported nature of the survey, which poses risks of over or underreporting. Methods of fraud detection and error handling must be applied if using the model in real-time. The authority for hospitals to recommend their staff to be included in the study may introduce selection bias in the data collection process. Additionally, recall bias may be introduced in answers to retrospective questions in the survey, such as symptoms within the previous 14 days. Future work for the study includes collecting more data for these models, as well as investigating models for using CLM survey and other data to predict additional outcomes, such as hospitalization and mortality. Recruitment of health workers for the study also will expand to several other provinces within Indonesia. Notwithstanding the limitations, our results demonstrate predictive capability for COVID-19 in health workers using machine learning. Our preliminary models showed high predictive performance, especially when trained and tested on similar population groups. When used in practice, CLM can be tuned by training on local populations that resemble target populations in which it will be used. The models can potentially be used to prioritize RT-PCR testing in regions where diagnostic resources are scarce. Allocating testing using the model predictions may lead to reductions in the challenges health workers in Indonesia are facing due to the pandemic. Our study may also have the potential to inform policy decisions regarding health worker PPE requirements, such as the promotion of the use of face shields and protective goggles. All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. 1 -3 t i m e s a w e e k 3 7 7 9 . 4 7 3 2 1 9 . 5 2 5 6 9 . 1 8 4 -6 t i m e s a w e e k 1 , 3 8 5 3 4 . 7 5 1 , 1 3 9 3 3 . 7 4 2 4 6 4 0 . 3 3 M o r e t h a n 6 t i m e s a w e e k 7 6 7 1 9 . 2 6 6 4 2 1 9 . 0 3 1 2 5 2 0 . 4 9 H a d i n -p e r s o n m e e t i n g s N e v e r 2 , 3 8 7 5 9 . 9 3 1 , 9 All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. . Resilient and people-centred health systems: Progress, challenges and future directions in Asia. World Health Organization. Regional Office for South-East Asia. [41] Callahan, A., Steinberg, E., Fries, J.A. et al. Estimating the efficacy of symptom-based screening for COVID-19. npj Digit. Med. 3, 95 (2020) . https://doi.org/10.1038/s41746-020-0300-0 All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. World Health Organization. Prevention, Identification and Management of Health Worker Infection in the Context of COVID-19 654 Tenaga Kesehatan Gugur Lawan Pandemi Covid-19 di Indonesia Coronavirus kills 647 health workers in Indonesia Setahun Covid-19: Upaya Indonesia Akhiri Pandemi, dari PSBB hingga Vaksinasi CSIS. Southeast Asia Covid-19 Tracker Covid-19: identifying and isolating asymptomatic people helped eliminate virus in Italian village Detection of COVID-19 Infection from Routine Blood Exams with Machine Learning: A Feasibility Study Critical care crisis and some recommendations during the COVID-19 epidemic in China Critical Care Utilization for the COVID-19 Outbreak in Lombardy, Italy WHO provides one million antigen-detecting rapid diagnostic test kits to accelerate COVID-19 testing in Indonesia Global partnership to make available 120 million affordable, quality COVID-19 rapid tests for low-and middle-income countries Covid-19 di Indonesia diprediksi sampai awal Juli, daerah lain bisa menyusul Kudus Our World in Data. Coronavirus (COVID-19) Testing -Statistics and Research Ridwan Kamil Kritik Pelacakan COVID-19 RI Jauh dari Standar WHO The Capacity of the Indonesian Healthcare System to Respond to COVID-19 Assessing the COVID-19 diagnostic laboratory capacity in Indonesia in the early phase of the pandemic Low performance of rapid antigen detection test as frontline testing for COVID-19 diagnosis A deep learning algorithm using CT images to screen for Corona virus disease (COVID-19) A Prediction Model to Prioritize Individuals for a SARS-CoV-2 Test Built from National Symptom Surveys Strong associations and moderate predictive value of early symptoms for SARS-CoV-2 test positivity among health workers, the Netherlands Machine learning-based prediction of COVID-19 diagnosis based on symptoms Using machine learning of clinical data to diagnose COVID-19: a systematic review and meta-analysis A Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) Prediction Model From Standard Laboratory Tests COVID-19 diagnosis by routine blood tests using machine learning A Novel Triage Tool of Artificial Intelligence-Assisted Diagnosis Aid System for Suspected COVID-19 Pneumonia in Fever Risk Factors of health workers With Coronavirus Disease 2019: A Retrospective Cohort Study in a Designated Hospital of Wuhan in China Assessment of hand hygiene techniques using the World Health Organization's six steps Coronavirus disease (COVID-19) advice for the public: When and how to use masks Random Forests Extremely randomized trees API design for machine learning software: experiences from the scikit-learn project XGBoost: A scalable tree boosting system Random search for hyper-parameter optimization A unified approach to interpreting model predictions Adaptive synthetic sampling approach for imbalanced learning How to Protect Yourself & Others Letter from Indonesia COVID-19 outbreak and health worker behavioural change toward hand hygiene practices Saving the frontline health workforce amidst the COVID-19 crisis: Challenges and recommendations