key: cord-0750752-ryibszjm
authors: Shoer, S.; Karady, T.; Keshet, A.; Shilo, S.; Rossman, H.; Gavrieli, A.; Meir, T.; Lavon, A.; Kolobkov, D.; Kalka, I.; Godneva, A.; Cohen, O.; Kariv, A.; Hoch, O.; Zer-Aviv, M.; Castel, N.; Ekka Zohar, A.; Irony, A.; Geiger, B.; Hizi, D.; Shalev, V.; Balicer, R.; Segal, E.
title: Who should we test for COVID-19?A triage model built from national symptom surveys
date: 2020-05-21
journal: nan
DOI: 10.1101/2020.05.18.20105569
sha: f3eca202febb9a227ae68405dca3a108356b8cd3
doc_id: 750752
cord_uid: ryibszjm

The gold standard for COVID-19 diagnosis is detection of viral RNA in a reverse transcription PCR test. Due to global limitations in testing capacity, effective prioritization of individuals for testing is essential. Here, we devised a model that estimates the probability of an individual to test positive for COVID-19 based on answers to 9 simple questions regarding age, gender, presence of prior medical conditions, general feeling, and the symptoms fever, cough, shortness of breath, sore throat and loss of taste or smell, all of which have been associated with COVID-19 infection. Our model was devised from a subsample of a national symptom survey that was answered over 2 million times in Israel over the past 2 months and a targeted survey distributed to all residents of several cities in Israel. Overall, 43,752 adults were included, from which 498 self-reported as being COVID-19 positive. The model provides statistically significant predictions on held-out individuals and achieves a positive predictive value (PPV) of 46.3% at a 10% sensitivity. As our tool can be used online and without the need of exposure to suspected patients, it may have worldwide utility in combating COVID-19 by better directing the limited testing resources through prioritization of individuals for testing, thereby increasing the rate at which positive individuals can be identified and isolated.

The rapid and global spread of COVID-19 led the World Health Organization (WHO) to declare it a pandemic on March 11, 2020. One major factor that contributes to the spread of the virus is the apparently large number of undiagnosed infected individuals. This knowledge gap facilitates the silent propagation of the virus, delays the response of public health officials and results in an explosion in the number of cases 1,2 .

One reason for this knowledge gap is insufficient testing. While the current gold standard for COVID-19 diagnosis is detection of viral RNA in a reverse transcription PCR test, the number of tests is limited by financial and logistic constraints. In a time when almost all countries are faced with the same health challenge, resources are scarce. This creates the need for a prioritization mechanism to allocate tests and resources more efficiently towards individuals who are more likely to test positive, leading to earlier identification of COVID-19 patients and reduced spread of the virus. Despite this need, most countries still employ a simplistic testing strategy based on display of symptoms associated with the disease and either close epidemiological contact with a confirmed COVID-19 case or belonging to a high risk group 3 . In practice, these strategies lead to a relatively small fraction of positive tests among those tested and thus to inefficient use of the precious testing resources.

Here, we present a model that provides statistically significant estimates of the probability of an individual to test positive for SARS-CoV-2 infection in a PCR test, based on a national symptom survey that we distributed in Israel. Notably, while most studies describing the clinical characteristics of COVID-19 cases were based on symptoms of hospitalized patients 4 , our survey data allowed us to also study symptoms of milder cases and reveal which symptoms hold the highest predictive power for COVID-19 diagnosis. Using our model, the risk for a positive COVID-19 test can be evaluated in less than a minute and without added costs, or risk of exposure to a suspected patient. Our approach can thus be used globally to make more efficient use of available testing capacities, by significantly increasing the fraction of positive tests obtained, and by rapidly identifying individuals that should be isolated until definitive test results are obtained.

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted May 21, 2020. . https://doi.org/10.1101/2020.05.18.20105569 doi: medRxiv preprint

In this study, we utilized data that originates from two versions of a one-minute survey that was developed and deployed by our research group in the early stages of the COVID-19 spread in Israel 5 . The online version of the survey includes questions relating to age, gender, prior medical conditions, smoking habits, self-reported symptoms and geographical location (see section 1a in the supplementary appendix). Questions regarding prior medical conditions and symptoms included in the survey were carefully chosen by medical professionals. Each participant is asked to fill the survey once a day for himself and for family members that are unable to fill it for themselves (e.g. children and the elderly). The survey is anonymous to maintain the privacy of the participants, and has been collected since March 14th, 2020. As the number of COVID-19 diagnosed individuals in Israel rose, in some cities more than others, a shortened version of the survey was deployed using an Interactive Voice Response (IVR) platform. This version of the survey included information on respondents' age group, gender, presence of prior medical conditions, general feeling and a partial list of symptoms, including fever, cough, shortness of breath, sore throat and loss of taste or smell (see Section 1b in the supplementary appendix). Cities were targeted to participate in the IVR version of the survey according to the number of diagnosed patients and an increased concern for COVID-19 outbreaks (Table S1) . Starting April 5, 2020, citizens in the targeted cities were contacted and responses were collected anonymously. This study followed the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guideline 6 .

Overall, 695,586 and 66,447 responses were collected up until April 26th, 2020, from the online and IVR versions of the survey, respectively. Since children express different clinical manifestations of COVID-19 infection 7,8 , we decided to focus our analysis only on adults (age above 20 years old). To avoid translation discrepancies, we included only responses in Hebrew. We also excluded responses that did not meet quality control criteria, such as a reasonable age (<120 years old) and body temperature (35-43℃) or responses suspected as spam (see Section 3 in the supplementary appendix) (Figure 1 ). A total of 33,737 responses were . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted May 21, 2020. . https://doi.org/10.1101/2020.05.18.20105569 doi: medRxiv preprint eventually included from the IVR version of the survey. Since surveyed cities were at relatively high risk (Table   S1 ), the prevalence of COVID-19 diagnosed responders in the IVR data was 1.14%, 6 times higher than the national prevalence of 0.18% 3 . These cities also had very high response rates, between 6% to 16% of the cities' population (Table S1 ). From the online version of the survey, we randomly sampled a single response for each individual that was recognized by a unique identifier we created (see Section 4 in the supplementary appendix), and when an individual reported a COVID-19 diagnosis, we randomly sampled one response only from those which included a positive diagnosis answer. A total of 131,166 responses were identified in the online version, of which 0.09% reported a positive COVID-19 diagnosis, which is closer to the national prevalence of 0.18% 3 .

The characteristics of these unique responders are described in Table S2 of the supplementary appendix.

For the integration of the two survey versions, all 33,737 IVR responses were combined together with all 114 uniquely identified responders in the online survey that self-reported COVID-19 diagnosis and a random sample of 9,901 undiagnosed responders in the online version, to maintain the same diagnosis prevalence as in the IVR version ( Figure 1 ). Overall, 43,752 responses were eventually included in the study, of which 498 self-reported as being COVID-19 diagnosed. The characteristics of these responders are described in Table 1 .

We defined survey self-reporting of a COVID-19 laboratory confirmed diagnosis as our outcome. We constructed two models. The first, which we term the primary model, was constructed from the integrated responses from both the IVR and online versions of the survey and included the reduced set of questions that were surveyed in the IVR version. The second model, which we term the extended features model, was constructed using only responses from the online version, and included additional symptoms and questions that were not part of the IVR survey. We trained both the primary and extended features models using Logistic

Regression. In addition, in order to capture nonlinear interactions and interactions amongst features, in both cases we also constructed models using a Gradient Boosting Decision Trees algorithm 9 . To test the validity of our models, all model constructions were done using the framework of cross-validation, in which model performance is evaluated on a subset of the data that was not used in the model's construction. For more . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted May 21, 2020. . https://doi.org/10.1101/2020.05.18.20105569 doi: medRxiv preprint information on handling of missing values and the models construction process, see Section 6 in the supplementary appendix.

The primary model was constructed using responses to both the IVR and the online version of the survey.

Features included in this model were determined by the IVR version, since it included a subset of the online version questions. These consisted of age group, gender, presence of prior medical conditions, general feeling, and the following symptoms: fever, cough, shortness of breath, sore throat and loss of taste or smell.

The extended features model was constructed using only responses from the online version of the survey, as it had 14 additional features that were not available in the IVR version. This extended list added dry cough and moist cough (instead of general cough in the primary model), fatigue, muscle pain, rhinorrhea, diarrhea, nausea or vomiting, chills, confusion and reporting on presence of specific prior medical conditions separately (as opposed to the presence of any of the prior medical conditions in the primary model).

To assess the contribution of reported symptoms and prior medical conditions to both the primary model and the extended features model (in both the Logistic Regression and the Gradient Boosting Decision Trees versions), we constructed baseline models using only age group and gender information to predict our outcome.

To gain insight into the features that contribute most to the predicted probability of being diagnosed with COVID-19 of our models, we analyzed feature contribution in the Gradient Boosting Decision Trees models using SHAP (SHapley Additive exPlanation) 10 . SHAP aims to interpret the output of a machine learning model by estimating the Shapley value of each feature, which represents the average change in the output of the model, by conditioning on that feature while introducing other features one at a time, over all possible features ordering.

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted May 21, 2020. . https://doi.org/10.1101/2020.05.18.20105569 doi: medRxiv preprint and to identify which features dominated in each prediction.

We further analyzed SHAP interaction values, which uses the 'Shapely interaction index' to capture local interaction effects between features 10 . Interaction values are calculated for each pair of the model's features, and for each individual prediction of the model, allowing us to uncover interaction patterns between pairs of features.

We placed particular emphasis on the contribution of participants' age and the interaction of age with all other features.

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted May 21, 2020. (Figure 2A-B) . Aside from discrimination performance measures, we also tested whether the model was calibrated. In a perfectly calibrated model, the distribution of the predicted probabilities is equal to the distribution of outcomes observed in the training data. We found that our primary model is well calibrated across the relevant prediction range ( Figure 2C) . The model has a positive predictive value (PPV) of 46.3%, at 10% sensitivity, while the baseline model only has 0.014%, at 10% sensitivity.

As an additional validation for the risk scores obtained, we compared the model predictions on the online survey . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. features the algorithm found to be most contributing ( Figure 4B ).

As age was reported as a dominant factor in COVID-19 infection and its clinical manifestation 11 , we examined the interaction of age with every symptom using SHAP interaction values (see Methods). The contribution of age to the probability of being diagnosed with COVID-19 is the highest in the oldest age group (>70 years old) ( Figure 5B ). Presence of cough and loss of taste or smell exhibits a sharp transition-type (sigmoid-like) interaction with age, such that above the age of 40 years old, presence of each of these symptoms sharply increases the model's predicted probability of COVID-19 infection ( Figure 5G-H) . In contrast, shortness of breath and sore throat show a more gradual (parabolic-like) interaction with age with presence of these symptoms increasing the model's prediction more gradually as the age of the subject being predicted increases ( Figure 5I -J). Negative answers to all these features show no interaction with age. Other examined features, such as fever and general feeling, do not show such interactions with age.

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted May 21, 2020. . https://doi.org/10.1101/2020.05.18.20105569 doi: medRxiv preprint

In this study we constructed a model that predicts the probability of individuals to test positive for COVID-19.

Our model is based on 9 simple questions that every person can easily answer in less than a minute from the comfort of their home. Our model can assist the worldwide fight against COVID-19 by better prioritizing the limited tests available without additional costs or risk of exposure to suspected patients, thereby increasing the rate at which positive individuals can be identified and isolated.

In Israel, as well as in many other countries, due to limited testing resources, suspected patients are only tested if they were exposed to a COVID-19 confirmed patient as well as exhibited acute respiratory symptoms 3 . By taking an unbiased approach to predicting COVID-19 diagnosis from symptoms data, our analysis highlights the importance of additional symptoms. Of note, anosmia and ageusia that were less described in patients in the early stages of the COVID-19 pandemic 4,12 were the most impactful features in both models for predicting COVID-19 diagnosis as well as in an odds-ratio analysis. This is in line with recent literature demonstrating the importance of these symptoms in early detection and identification of the disease 13, 14, 15 . Our model also successfully recapitulated patterns of the disease that are described in the literature, such as its complex relationship with age 11 . In addition, our model unraveled several patterns that are not described in the literature, such as the different patterns of interactions that particular symptoms have with age, suggesting variation of the clinical manifestation in different age groups. Although our analysis is purely predictive and not causal, these new patterns may be used to devise better testing policies, and pave the way for future studies that can uncover new aspects of the disease that were not studied to date.

Analysis of an extended features model that included 23 features compared to 9 in the primary model, validated our choice of questions in the shortened version of the survey and suggested that fatigue should also be considered. In addition, the extended features model suggested that while dry cough has an essential role in predicting COVID-19 diagnosis, moist cough does not and thus may help distinguish between cases of COVID-19 and other infections. Some of the most contributing symptoms to the prediction of a COVID-19 diagnosis are . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted May 21, 2020. . https://doi.org/10.1101/2020.05.18.20105569 doi: medRxiv preprint currently not included in the Israeli testing policy 3 , such as loss of taste or smell and sore throat. Our analysis suggests that adding these symptoms to the testing policy may help discriminate which individuals should be tested, and improve testing prioritization.

While informative, our feature contribution analysis has several limitations. First, we did not include children in our datasets and thus, symptoms such as nausea or vomiting and diarrhea that were mostly described in children 7, 8 , may have a more significant part in models designed for younger age groups. Second, although we included a large set of prior medical conditions that may have a role in COVID-19 susceptibility, some of these conditions are not highly prevalent in our dataset and their contribution may thus be underestimated in our model. Finally, body temperature was the only non-mandatory question in our survey, and may thus have higher predictive power than portrayed within our model.

Several studies attempted to simulate and predict different aspects of COVID-19, such as hospital admissions, diagnosis, prognosis and mortality risk, using mostly age, body temperature, medical tests and symptoms 16 .

Most diagnostic models published to date were based on datasets from China and included complex features that had to be extracted through blood tests and imaging scans 16 . In this work, we devised a prediction model which was based solely on self-reported information, and as such it could be easily deployed and used instantly in other countries.

Our study has several additional limitations. First, our data is biased by Israel's MOH ever changing testing policy, such that at some point all of the COVID-19 positively diagnosed participants in our study had to be eligible for a test under that policy. An ideal dataset for purposes of devising a classifier should include a large random sampling of the population, but such data coupled with symptom surveys is currently unavailable at large-scale. Accordingly, all the diagnosed responders in our study are not in the first stage of showing symptoms, but are in some time-lag after diagnosis. In addition, our study is based on self reports of willing participants and is therefore bound to suffer from some selection bias. The bias is significantly reduced in the data collected via the IVR platform, since all residents in the IVR-surveyed cities were actively contacted only . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted May 21, 2020. . https://doi.org/10.1101/2020.05.18.20105569 doi: medRxiv preprint once, on the same day and in the same manner. In the online version of the survey we made attempts to reduce this bias by promoting it in several media outlets and by engaging leaders of underrepresented communities.

Finally, we acknowledge that the ideal choice of a baseline model would be the Israeli MOH testing policy, but since it constantly changed during the pandemic and included questions that were not part of our survey we were unable to compare to it. Nevertheless, at an unknown sensitivity (as we do not know the actual number of cases), the overall predictive value (PPV) of all tests in Israel was 4.6% 3 , and although the measures are not directly comparable, we believe that our model substantially outperforms this PPV.

In conclusion, our constructed model predicts COVID-19 PCR test results with high discrimination (positive predictive value (PPV) of 46.3% at a 10% sensitivity) and calibration. It also suggests that several symptoms that are currently not included in the Israeli testing policy exhibit intriguing interactions with age and should probably be integrated into revised testing policies. Overall, our approach can be utilized worldwide to direct the limited resources towards individuals who are more likely to test positive for COVID-19, leading to faster isolation of infected patients and therefore to reduced rates of virus spread.

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted May 21, 2020. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted May 21, 2020. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted May 21, 2020. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted May 21, 2020. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted May 21, 2020. . https://doi.org/10.1101/2020.05.18.20105569 doi: medRxiv preprint . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted May 21, 2020. . https://doi.org/10.1101/2020.05.18.20105569 doi: medRxiv preprint . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted May 21, 2020. . https://doi.org/10.1101/2020.05.18.20105569 doi: medRxiv preprint

Models and tables of de-identified, aggregated data are available at https://github.com/hrossman/Covid19-Survey.

Source code for the survey is available at https://github.com/hasadna/avid-covider as an open source project, and can be readily adapted to use in other countries.

The study protocol was approved by the Weizmann Institute of Science review board (IRB). Informed consent was waived by the IRB, as all identifying details of the participants were removed before the computational analysis. Participants were made fully aware of the way in which the data will be stored, handled and shared, which was provided to them and is in accord with the privacy and data-protection policy of the Weizmann Institute of Science (https://weizmann.ac.il/pages/privacy-policy).

Critical care crisis and some recommendations during the COVID-19 epidemic in China

Critical Care Utilization for the COVID-19 Outbreak in Lombardy, Italy: Early Experience and Forecast During an Emergency Response

Incidence, clinical characteristics and prognostic factor of patients with COVID-19: a systematic review and meta-analysis. medRxiv

A framework for identifying regional outbreak and spread of COVID-19 from one-minute population-wide surveys

Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement

Epidemiological characteristics of 2143 pediatric patients with 2019 coronavirus disease in China. Pediatrics

Clinical Manifestations of Children with COVID-19: a Systematic Review. medRxiv

A Scalable Tree Boosting System

From local explanations to global understanding with explainable AI for trees. Nat Mach Intell

Coronavirus disease 2019 (COVID-19): A literature review

Spread of SARS-CoV-2 in the Icelandic Population

Loss of smell and taste in combination with other symptoms is a strong predictor of COVID-19 infection. medRxiv

Association of chemosensory dysfunction and Covid-19 in patients presenting with influenza-like symptoms. Int Forum Allergy Rhinol

Olfactory and gustatory dysfunctions as a clinical presentation of mild-to-moderate forms of the coronavirus disease (COVID-19): a multicenter European study

Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal

We thank the following for their contributions to our efforts: 

The authors declare no competing interests.