key: cord-0950838-r1p7xn3a
authors: Ng, Ming-Yen; Wan, Eric Yuk Fai; Wong, Ho Yuen Frank; Leung, Siu Ting; Lee, Jonan Chun Yin; Chin, Thomas Wing-Yan; Lo, Christine Shing Yen; Lui, Macy Mei-Sze; Chan, Edward Hung Tat; Fong, Ambrose Ho-Tung; Yung, Fung Sau; Ching, On Hang; Chiu, Keith Wan-Hang; Chung, Tom Wai Hin; Vardhanbhuti, Varut; Lam, Hiu Yin Sonia; To, Kelvin Kai Wang; Chiu, Jeffrey Long Fung; Lam, Tina Poy Wing; Khong, Pek Lan; Liu, Raymond Wai To; Man Chan, Johnny Wai; Ka Lun Alan, Wu; Lung, Kwok-Cheung; Hung, Ivan Fan Ngai; Lau, Chak Sing; Kuo, Michael D.; Ip, Mary Sau-Man
title: Development and Validation of Risk Prediction Models for COVID-19 Positivity in a Hospital Setting
date: 2020-09-15
journal: Int J Infect Dis
DOI: 10.1016/j.ijid.2020.09.022
sha: 67719a88f30147756a1e056d716b87d217547237
doc_id: 950838
cord_uid: r1p7xn3a

OBJECTIVES: To develop:(1) two validated risk prediction models for COVID-19 positivity using readily available parameters in a general hospital setting; (2) nomograms and probabilities to allow clinical utilisation. METHODS: Patients with and without COVID-19 were included from 4 Hong Kong hospitals. Database was randomly split 2:1 for model development database (n = 895) and validation database (n = 435). Multivariable logistic regression was utilised for model creation and validated with the Hosmer-Lemeshow (H-L) test and calibration plot. Nomograms and probabilities set at 0.1, 0.2, 0.4, 0.6 were calculated to determine sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV). RESULTS: 1330 patients (mean age 58.2 ± 24.5 years; 50.7% males; 296 COVID-19 positive) were recruited. First prediction model developed had age, total white blood cell count, chest x-ray appearances and contact history as significant predictors (AUC = 0.911 [CI = 0.880-0.941]). Second model developed has same variables except contact history (AUC = 0.880 [CI = 0.844-0.916]). Both were externally validated on H-L test (p = 0.781 and 0.155 respectively) and calibration plot. Models were converted to nomograms. Lower probabilities give higher sensitivity and NPV; higher probabilities give higher specificity and PPV. CONCLUSION: Two simple-to-use validated nomograms were developed with excellent AUCs based on readily available parameters and can be considered for clinical utilisation.

 Developed two simple-to use nomograms for identifying COVID-19 positive patients  Probabilities are provided to allow healthcare leaders to decide suitable cut-offs  Variables are age, white cell count, chest x-ray appearances and contact history  Model variables are easily available in the general hospital setting.

Objectives:

To develop: (1) two validated risk prediction models for COVID-19 positivity using readily available parameters in a general hospital setting; (2) nomograms and probabilities to allow clinical utilisation.

Patients with and without COVID-19 were included from 4 Hong Kong hospitals. Database was randomly split 2:1 for model development database (n=895) and validation database J o u r n a l P r e -p r o o f calibration plot. Models were converted to nomograms. Lower probabilities give higher sensitivity and NPV; higher probabilities give higher specificity and PPV.

Two simple-to-use validated nomograms were developed with excellent AUCs based on readily available parameters and can be considered for clinical utilisation. Coronavirus disease 2019 has spread rapidly worldwide and as of 6 th September 2020, there are now ~27 million cases worldwide and ~900,000 deaths 1 .

Respiratory and non-respiratory complications of COVID-19 are also becoming increasingly apparent 2, 3 . Reverse transcription polymerase chain reaction (RT-PCR) is regarded as a vital tool in identifying the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) and quarantining COVID-19 patients to prevent further spread of the disease 4 . Furthermore, it is the definitive test in confirming the diagnosis of COVID-19. However, availability of RT-PCR kits maybe difficult in various countries 4 and from specimen collection to report generation, the tests could take 48-72 hours to confirm a positive or negative result 5 . Therefore, clinical assessment, blood tests and imaging have been recommended to help identify potential COVID-19 positive patients 6 .

Various strategies have been proposed including widespread computed tomography (CT) scanning 7-9 , greater use of chest x-rays (CXR) 10, 11 , identification of low lymphocyte counts 8, 11 to determine patients more likely to have COVID-19 12, 13 and thus more suitable for testing.

As yet, the data which supports these strategies are predominantly based on data of COVID-19 patients 12, 13 but without comparisons to patients with other conditions and symptoms overlapping with COVID-19 (eg. fever, shortness of breath, cough).

Several issues have arisen in trying to determine the likelihood of a COVID-19 diagnosis.

Firstly, in the early stages of the pandemic when the disease was limited to a few countries, travel and contact history may have been helpful to increase suspicion of a COVID-19 J o u r n a l P r e -p r o o f diagnosis, but in some countries where there is established community transmission, this has resulted in patients being COVID-19 positive but with no knowledge of possible contact.

Secondly, different countries have adopted different strategies due to socioeconomic factors and healthcare resources. Thus, a COVID-19 prediction model based on clinical, laboratory and radiological findings which presents the sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) would allow public healthcare systems to decide a suitable strategy on prioritizing tests when such RT-PCR availability is constrained.

In this study, we aimed to construct a prediction model utilising patient characteristics, commonly available hematological and biochemical blood tests and CXR findings which can identify COVID-19 patients within a cohort of patients who presented to hospitals for various disease conditions and underwent testing for COVID-19. In addition, we aimed to create a separate model in the event that contact history is not available in order to determine the presence of COVID-19.

Research ethics approval was obtained from the Hong Kong West Cluster CXR images were searched via the electronic patient record system. Baseline CXR images were reviewed and interpreted by radiologists blinded to the patient's COVID-19 status.

Assessment was based on identifying the common findings of COVID-19 on CXR which were (i) consolidation or ground glass opacity and (ii) absence of pleural effusion 14, 17 . This was done in a binary format (present or absent) to make this more reproducible in the clinical environment for front-line clinicians.

Image quality was assessed in 702 randomly chosen CXRS (53% of entire cohort of CXRs) by 8 radiologists separately. We ensured that the CXRs were taken from each of the 4 hospitals. Image quality was assessed on a scale from 1 to 3. See supplementary table 2 for examples of CXRs graded as 1, 2 and 3. Briefly, CXRs which could not be interpreted with any confidence were graded 1. CXRs with suboptimal image quality but lung changes and pleural effusion could be interpreted with some confidence were graded 2. CXRs with good quality such that lung changes and pleural effusions can be diagnosed with high confidence were graded 3.

Patients positive for COVID-19 were compared to those negative for COVID-19 patients.

Continuous variables were compared using student t-tests. Categorical variables were compared using chi-squared tests. The database was randomly split on a 2:1 basis for the The selection was finished until the difference in BIC of all remaining risk factors <10. 18 To test the nonlinear effect of selected clinical parameters, quadratic term of significant continuous predictors were considered. Given that patients can present without knowledge of contact history with an infected person, a further model was developed with one having contact history removed, to represent an event in which contact history is unknown.

In order to validate the model, the discrimination and calibration power of models were examined. The area under the receiver operating characteristic curve (AUC) were conducted to evaluate the discrimination power, where 0.7 to 0.8 of AUC is considered acceptable, 0.8 J o u r n a l P r e -p r o o f to 0.9 is considered excellent, and more than 0.9 is considered outstanding discrimination power. Meanwhile, Hosmer-Lemeshow (H-L) test and calibration plot was used to test how well the percentage of observed COVID-19 positive matches the percentage of predicted COVID-19 positive over deciles of predicted risk. A p-value >0.05 is needed to conclude that there are insignificant differences between the observed and expected outcomes and therefore the model has good overall calibration. Different probabilities were used to evaluate the model performance based on the sensitivity, specificity, PPV and NPV.

Sensitivity analysis was conducted to examine the robustness of the model. Multiple imputation was applied to handle missing data. The chained equation method was used to impute each missing value twenty times, adjusted for all baseline covariates and outcomes.

Moreover, 10-fold cross validation was applied to evaluate the discrimination and calibration power.

To facilitate the risk prediction models used for screening in routine busy clinical practice, simple nomograms were developed. The effect of each predictor in the model was converted to a score and summation of all predictors that can be mapped to an estimated risk of COVID-19 positive. The nomograms were plotted using nomolog package in Stata 19 .

Sensitivity, specificity, PPV, NPV were determined for the following probabilities which were: Figure 3 and Figure 4 were developed based on the derived risk prediction models. Using the overall cohort model nomogram (Figure 3 ) as an example, if a patient suspected to have COVID-19 is aged 50, has no contact history, WCC of 2x10 9 cells/L and a CXR with no consolidation/GGO and absent pleural effusion (PEff) the scoring will be as follows: Age has two steps, so for age 50 at step 1, allocate 2 points; for step 2: allocate 7.5 points. For no contact history which is step 3, allocate 0 points. For a total white cell count (WCC) of 2x10 9 cells/L at step 4: allocate 7 points. For a CXR with no consolidation/GGO and absent PEff at step 5, allocate 0 points. Therefore, they would be allocated a total score of 16.5 points which equates to 0.6-0.7 probability (ie. 60-70% probability) of being COVID-19 positive.

In our study, we have developed two risk prediction models for determining COVID-19 positive patients which have been validated with a separate dataset. Both models have an excellent AUC with good matching with the validation dataset. The models are based on parameters (ie. total WCC, CXR consolidation/GGO with absent pleural effusions) which are available in general hospitals as well as clinical data (ie. age with or without contact history).

We have also provided nomograms to determine probability of COVID-19 with several different probabilities illustrated to show the sensitivity, specificity, PPV and NPV so that clinicians or healthcare systems can decide which probabilities would make the best cut-offs for RT-PCR testing. The development of these nomograms will hopefully improve frontline clinicians' diagnostic accuracy in identifying patients with COVID-19 where RT-PCR may not be available or rapid results cannot be provided. Commission. Thus our data provides evidence that these initial observations of COVID-19 were indeed accurate.

CXR consolidation/GGO with absent pleural effusions is the typical appearance of COVID-19 radiologically 14 . This model confirms that using CXR in addition to other parameters is J o u r n a l P r e -p r o o f helpful in identifying COVID-19 patients. This has already been incorporated into societal recommendations and our models provide evidence to support this approach despite the lower sensitivity of CXR compared to CT 10, 11 . Our model did not incorporate CT as CT was not easily available for our COVID-19 positive patients and indeed the negative patients.

This would likely be the scenario globally during this pandemic. CT with its higher sensitivity 21 will likely improve diagnostic accuracy but this is dependent on the facilities in each health service. Not all health services can dedicate CT scanners for COVID-19 diagnosis due to either a lack of scanner availability and/ or the extensive cleaning required after each COVID-19 scan which reduces the radiology department's productivity 22 . In our study, we wanted to focus on parameters which would be easily accessible to all patients seen in the general hospitals, as some health systems even struggle to make chest x-rays and WCC available 23 .

In our cohort, age is a significant predictor for COVID-19. In this, study, the COVID-19 patients were significantly younger than the negative patients. This can be partly explained by younger patients being more mobile and thus being more susceptible to develop COVID-19 compared to the older population who may travel less. Review of previous publications have indicated that patients with COVID-19 are usually younger. In Korea, one paper indicated that >60% of patients 24 were <49 years old whilst in China, 51.2%-55.1% 12, 13 of patients were <49 years old. The two nomograms in this study allocated higher scoring to the younger patients including children. This is possibly due to children having less symptoms and even less radiological changes 25, 26 making the identification of COVID-19 more difficult. Indeed, this possibly explains the noticeably less children confirmed to have COVID-19 and possibly explains the statistical significance of age in the models for determining patients who are positive for SARS-CoV2. However, age as a predictor is very much representative of this cohort. In a different healthcare system where more elderly patients present, age as a predictor will likely need to be further investigated.

The models we have established can set different probabilities in order to allow medical systems to self-determine the pre-test probability required for RT-PCR testing. Moreover, the nomograms have been developed to visualize the sophisticated mathematical equation so that it can be adopted in the routine busy clinical practice. However, it should be emphasised that RT-PCR remains the gold standard for diagnosing COVID-19 and that focus should be made on making RT-PCR easily available for testing patients as well as increasing the time taken for results to be made available.

Our study has several limitations. Firstly, the COVID-19 cases are reflective of practice in Hong Kong which has been active in screening for COVID-19 which has included asymptomatic patients (5.6% in this cohort) with contact history and patients with mild symptoms. This may not be representative in other health systems worldwide so this model needs to be validated in those health systems. Secondly, the chest x-rays were assessed by radiologists, so whether these results will be similar with frontline clinicians is uncertain.

However, the assessment was simplified in order that frontline clinicians can focus their search on CXR to consolidation/GGO and absence of pleural effusions. Furthermore, some health systems have access to radiology support to review CXRs and this model possibly justifies this practice if logistically feasible. Thirdly, inflammatory markers like C-reactive protein, creatnine kinase, lactacte deyhydrogenase were not included in the model as a significant proportion of patients did not have these markers measured at time of admission.

Whether these markers prove useful will require further study. Lastly, asymptomatic patients made up a very small proportion of patients and thus further validation with an asymptomatic cohort would be required to validate this model.

In conclusion, we present two models which have 3 or 4 readily available parameters to improve the accuracy of identifying COVID-19 amongst patients suspected of having COVID-19 with or without known contact history. This will help identify patients most likely to benefit from RT-PCR testing and thus help better allocate RT-PCR testing where this resource is limited. Table 2 . A total score is calculated from the addition of the scores for the variables chest x-ray (CXR) consolidation/ ground glass opacity (GGO), contact history, white cell count and age. Note that age has two steps whilst other variables only have 1 step. The total score can then be marked on the bottom row and compared with the probability scale above. For example, a patient suspected to have COVID-19 aged 50 (step 1: allocate 2 points; step 2: allocate 7.5 points), has no contact history (step 3: allocate 0 points), total white cell count (WCC) of 2x10 9 cells/L (step 4: allocate 7 points) and a CXR with no consolidation/GGO and absent pleural effusion (PE) (step 5: allocated 0 points), would receive a total score of 16.5 points which equates to a probability of between 0.6 and 0.7. A clinician then refers to the probability table (table 3) and decides what degree of sensitivity, specificity, positive predictive value or negative predictive value is adequate for their setting. Table 4 . A total score is calculated from the addition of the scores for the variables pleural effusion, chest x-ray (CXR) consolidation/ ground glass opacity (GGO), white cell count, age and vomiting symptom. Note that age has two steps whilst other variables only have 1 step. The total score can then be marked on the bottom row and compared with the probability scale above. For example, a patient suspected to have COVID-19 aged 40 (step 1: allocate 2 points; step 2: allocate 8.5 points), total white cell count (WCC) of 8x10 9 cells/L (step 3: allocate 3 points) and a CXR with consolidation/GGO and absent pleural effusion (PE) (step 4: allocated 1.5 points), would receive a total score of 15 points which equates to a probability of between 0.6 and 0.7. A clinician then refers to the probability table (table 3) and decides what degree of sensitivity, specificity, positive predictive value or negative predictive value is adequate for their setting.

World Health Organization. Coronavirus disease (COVID-19) Weekly Epidemiological Update

Recovered COVID-19 Patients Show Ongoing Subclinical Myocarditis as Revealed by Cardiac Magnetic Resonance Imaging

World Health Organization. Laboratory testing strategy recommendations for COVID-19

Guidance and standard operating procedure COVID-19 virus testing in NHS laboratories

The Role of Chest Imaging in Patient Management during the COVID-19 Pandemic: A Multinational Consensus Statement from the Fleischner Society

COVID-19): A Perspective from China

Therapeutic and triage strategies for 2019 novel coronavirus disease in fever clinics

COVID-19 pneumonia: what has CT taught us? The Lancet Infectious Diseases

ACR Recommendations for the use of Chest Radiography and Computed Tomography (CT) for Suspected COVID-19 Infection

British Society of Thoracic Imaging. BSTI NHSE COVID-19 Radiology Decision Support Tool

Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China. The Lancet

2020;0:201160. 15. World Health Organization. Global surveillance for COVID-19 caused by human infection with COVID-19 virus: Interim guidance

Improved molecular diagnosis of COVID-19 by the novel, highly sensitive and specific COVID-19-RdRp/Hel real-time reverse transcription-polymerase chain reaction assay validated <em>in vitro</em> and with clinical specimens

Imaging Profile of the COVID-19 Infection: Radiologic Findings and Literature Review

Bayesian Model Selection in Social Research

Design and evaluation of analytical tools for emergency department management based on machine learning techniques

Index for rating diagnostic tests

Sensitivity of Chest CT for COVID-19: Comparison to RT-PCR

A role for CT in COVID-19? What data really tell us so far

Adoption of COVID-19 triage strategies for low-income settings

Transmission potential and severity of COVID-19 in South Korea

Clinical and epidemiological features of 36 children with coronavirus disease 2019 (COVID-19) in Zhejiang, China: an observational cohort study. The Lancet Infectious Diseases

Differences in Clinical and Imaging Presentation of Pediatric Patients with COVID-19 in Comparison with Adults

Sensitivity, specificity, positive predictive value and negative predictive value are stated in percentage with 95% confidence intervals in brackets