key: cord-0805526-eaxqejk7
authors: Nafilyan, V.; Humberstone, B.; Metha, N.; Diamond, I.; Lorenzi, L.; Pawelek, P.; Schofield, R.; Morgan, J.; Brown, P.; Lyons, R.; Sheikh, A.; Hippisley-Cox, J.
title: An external validation of the QCovid risk prediction algorithm for risk of mortality from COVID-19 in adults: national validation cohort study in England
date: 2021-01-25
journal: nan
DOI: 10.1101/2021.01.22.21249968
sha: 185fdcba4b6a1f50c55ebb484c565b5cd56355a8
doc_id: 805526
cord_uid: eaxqejk7

Background: To externally validate a risk prediction algorithm (QCovid) to estimate mortality outcomes from COVID-19 in adults in England. Methods: Population-based cohort study using the ONS Public Health Linked Data Asset, a cohort based on the 2011 Census linked to Hospital Episode Statistics, the General Practice Extraction Service Data for pandemic planning and research, radiotherapy and systemic chemotherapy records. The primary outcome was time to COVID-19 death, defined as confirmed or suspected COVID-19 death as per death certification. Two time periods were used: (a) 24thJanuary to 30thApril 2020; and (b) 1st May to 28th July 2020. We evaluated the performance of the QCovid algorithms using measures of discrimination and calibration for each validation time period. Findings: The study comprises 34,897,648 adults aged 19-100 years resident in England There were 26,985 COVID-19 deaths during the first time-period and 13,177 during the second. The algorithms had good calibration in the validation cohort in both time periods with close correspondence of observed and predicted risks. They explained 77.1% (95% CI: 76.9% to 77.4%) of the variation in time to death in men in the first time-period (R2); the D statistic was 3.76 (95% CI: 3.73 to 3.79); Harrell's C was 0.935 (0.933 to 0.937) Similar results were obtained for women, and in the second time-period In the top 5% of patients with the highest predicted risks of death, the sensitivity for identifying deaths in the first time period was 65.9% for men and 71.7% for women. People in the top 20% of predicted risks of death accounted for 90.8% of all COVID-19 deaths for men and 93.0% for women. Interpretation: The QCovid population-based risk algorithm performed well, showing very high levels of discrimination for COVID-19 deaths in men and women for both time periods. It has the potential to be dynamically updated as the pandemic evolves and therefore, has potential use in guiding national policy. Funding: National Institute of Health Research

The first cases of SARS-CoV-2 infection were reported in the United Kingdom (UK) on the 24 th January 2020, with the first COVID-19 death on the 28 th February 2020. As of 19 January 2021, there have been over 90,000 deaths from COVID-19 in the UK, and over 2 million deaths globally 1 .

Emerging evidence throughout the course of the pandemic, initially from case series, and then cohorts of individuals with confirmed SARS-CoV-2 infection, has demonstrated associations of age, sex, certain co-morbidities, ethnicity and obesity with adverse COVID-19 outcomes such as hospitalisation and death [2] [3] [4] [5] [6] [7] [8] [9] . There is now a growing knowledge base regarding risk factors for severe . As many countries are re-introducing 'lockdown' measures and vaccination programmes have started being rolled out, there is an opportunity to develop more nuanced guidance 10 based on predictive algorithms to inform risk management decisions. Better knowledge of individuals' risks could also help guide decisions on managing occupational risk and in targeting of vaccines to those most at risk. Whilst several risk prediction models have been developed, a recent systematic review found that they all have high risk of bias and that their reported performance is optimistic 11 .

The use of primary care datasets such as QResearch with linkage to registries such as death records and hospital admissions data represents an innovative approach to clinical risk prediction modelling for COVID-19 which has successfully been developed, validated, and implemented in the NHS over the last 10 years 12-14 . It provides accurately coded, individual-level data for very large numbers of people representative of the national population. This approach was used to develop the QCovid prediction models 15 drawing on the rich phenotyping of individuals with demographic, medical and pharmacological predictors to allow robust statistical modelling and evaluation. Such linked datasets have a track record for the development, and evaluation of established clinical risk models including for cardiovascular disease 12 , diabetes 14 and mortality 13 . Whilst QCovid predicts both COVID-19 hospital related admission and COVID-19 death, the aim of this analysis is to validate the mortality outcome which estimate the risks of becoming infected and subsequent death due to COVID-19 in an extremely large national cohort. A companion study currently underway will externally validate these, using datasets from Wales using 16 SAIL 17 and Scotland using EAVE-II 18 and will be reported separately.

Advisory Group (NERVTAG), to develop and validate a clinical risk prediction model for COVID -19 in line with the emerging evidence. The resulting QCovid model was developed and validated using the QResearch database and reported in accordance with TRIPOD 19 and RECORD 20 guidelines and with input from a patient advisory group. The original protocol is published 21 is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint the paper reporting the original derivation and validation of the model 15 . This paper reports the validation of the model on an independent data source.

We undertook a validation cohort study of individuals aged 19- QCovid algorithm, but also includes patients registered with practices using IT systems other than EMIS, such as TPP (used by 35% of GP practices).

We identified a cohort of all individuals aged 19-100 years who were enumerated at the 2011 Census and registered alive and resident in England on 24 th January 2020. Patients entered the cohort on 24 th January 2020 (date of first confirmed COVID-19 case in UK) and were followed up until they had the outcome of interest or 28 th July 2020, which is the date up to which linked data were available at the time of the analysis. This also extends the period of observation beyond the original QCovid study. We divided the study period into two time periods: 24 th January 2020 to 30 th April 2020 and 01

May 2020 to 28 th July 2020.

The outcome was time to COVID-19 death (either in hospital or out of hospital), defined as confirmed or suspected COVID-19 death as identified by two ICD10 codes (U07.1 or U07.2) recorded on the death certification.

We derived pre-existing conditions and demographic characteristics using the same definitions as used to develop the QCovid algorithm. The primary care records used in the ONS Public Health Linked Data Asset were based on an existing GPES dataset which included many but not all of the relevant clinical codes used to develop the QCovid algorithm. Nonetheless, we derived most of the preexisting conditions, although we could not identify patients who had a solid organ or bone marrow transplant in the past six months; those on kidney dialysis; those with sickle cell disease or severe combined immune deficiency syndrome; Similarly, we could not distinguish between patients suffering from type 1 or type 2 diabetes. Variables used to validate the QCovid algorithm are listed in Box 1.

We fitted an imputation model to replace missing values for body mass index (BMI), using predicted values from linear regression models stratified by sex. Predictors included all predictor variables in the QCovid algorithm, interacted with age.

We applied the QCovid risk equations (version 1) to men and women in the validation dataset and evaluated R 2 values 22 , Brier scores and measures of discrimination and calibration 23 24 with corresponding 95% confidence intervals (CIs) over the two time periods. R 2 values refer to the proportion of variation in survival time explained by the model. Brier scores measure predictive accuracy where lower values indicate better accuracy 25 . The D statistic is a discrimination measure which quantifies the separation in survival between patients with different levels of predicted risks and the Harrell's C-statistics is a discrimination metric which quantifies the extent to which people with higher risk scores have earlier events. Model calibration was assessed in the two time periods by comparing mean predicted risks with observed risks by twentieths of predicted risk. Observed risks were derived in each of the 20 groups using non-parametric estimates of the cumulative incidences.

The performance metrics were calculated in the whole cohort and in the following pre-specified subgroups: 5-year age-sex bands, nine ethnic groups, and within each quintile of the Townsend Index, a measure of deprivation. We also estimated the performance metrics on a sample restricted to patients registered with practices using the TPP system, and therefore not used at all to derive the algorithm.

The ethics approval for the development and validation of QCovid was granted by the East Midlands-Derby Research Ethics Committee [reference 18/EM/0400].

This study is funded by a grant from the National Institute for Health Research following a commission by the Chief Medical Officer for England whose office contributed to the development of the study question and facilitated access to relevant national datasets, contributed to interpretation of data, drafting of the report.

Overall, 34,897,648 people in England aged 19-100 years met our inclusion criteria. Out of the 40,136,597 people aged 19-100 who were enumerated at the 2011 Census and were alive on 24 th January 2020, 5,238,949 (13.1%) people were excluded because they did not have a valid NHS number or had an NHS number that could not be linked to the GPES data. This could be because they migrated out of England, and therefore were no longer registered with the NHS in England. Our data cover 80.0% of is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint care home. Those who had a COVID-19 related death in period 2 had a similar profile to those in period 1 but were on average older (88.4% aged 70 and over) and more likely to live in a care home (31.4%). Table 2 shows the performance of the risk equations in the validation cohort for women and men in the two time periods. Overall, the values for the R 2 , D and C statistics were high and similar in women and men in both periods. In the first period for women, the equation explained 76.3% of the variation in time to COVID-19 death; Harrell's C statistic was 0.945 and the D statistic 3.67. The corresponding values in men were 77.1%, 0.935 and 3.76 respectively. All these discrimination metrics are higher than in the original QResearch cohort used to validate the algorithm. The results were similar for the second validation period. In period 2 for women, the R 2 was 75.4%, Harrell's C statistic 0.956, the D statistic 3.58. The corresponding values for men were 77.4%, 0.944 and 3.78 respectively. Similar results were obtained when restricting the sample to 14,104,452 patients registered with practices using the TPP system (Supplementary Table 2 ). is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

The copyright holder for this this version posted January 25, 2021. ; Figure 1 displays Harrell's C statistic by age-band for men and women in period 1 (Panel A) and period 2 (Panel B). The Harrell's C statistics are over 0.7 for all age bands indicating that even within each age band the model discriminates well. The C statistics are lower for patients aged 90 or over than for younger patients. The C statistics, R 2 , and D by age-band, deprivation quintile and ethnic group in men and women for both periods are reported in Supplementary Tables 3, 4 and 5. Performance was generally similar to the overall results except for age where the performance was lower within individual age-bands. Figure 2 displays the calibration plots for the COVID-19 mortality equation for men and women and in both periods. Overall, both sets of equations were well calibrated, as the predicted and observed risks were similar. However, as in the original QResearch validation cohort, the model underestimates the risk of COVID-19 death for those in the top 5% of the predicted risk score. Figure 3 shows the sensitivity values for the mortality equation in period 1 (24 January 2020 -30 April 2020) and period 2 (1 May 2020 -28 July 2020) evaluated at different thresholds based on the centiles of the predicted absolute risk in the validation cohort. Full results are reported in Supplementary   Table 6 . The sensitivity was higher in women than in men, and in period 2 than period 1. In period 1, 65.94% of deaths in men occurred in those in the top 5% for predicted absolute risk of death from Covid-19 (90 day predicted absolute risks above 0.289%) and 71.67% of deaths in women occurred in the top 5% (predicted absolute risks above 0.188%). In period 2, 71.10% of deaths occurred in men in the top 5% for predicted absolute risk of death from Covid-19 (predicted absolute risks above 0.278%) and 77.16% of deaths occurred in women in the top 5% (predicted absolute risks above 0.181%).

Supplementary Figure 1 shows the sensitivity for the two time periods based on relative risks (defined as the ratio of the individual's predicted absolute risk to the predicted absolute risk for a person of the same age and sex with a white ethnicity, body mass index of 25, and mean deprivation score with no other risk factors). In period 1, 40.56% of deaths occurred in men in the top 5% for predicted relative risk of death from Covid-19, and 42.63% for women. In period 2, 42.62% of deaths occurred in men in the top 5% for predicted relative risk of death from Covid-19, and 43.57% for women.

We have validated the QCovid clinical risk prediction model for mortality due to COVID-19 using a national external linked dataset. We have used national linked datasets from the 2011 Census, general practice, death registry data for a population-representative sample of nearly 35 million adults.

The risk models have excellent discrimination (Harrell's C statistics>0.9), are well calibrated and have a high sensitivity. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

The copyright holder for this this version posted January 25, 2021. Our study had a number of important strengths. First, we used a unique linked dataset based on the 2011 Census for nearly 35 million people living in England. Second, we used a wide range of metrics, over two time periods to validate the QCovid predictive model, extending the period of observation beyond the original study. All the metrics in the two time periods for both men and women indicate that the algorithm performs well, and the metrics are comparable with the original validation of QCovid in the QResearch database 15 . Finally, we showed that the model's performance was similar when restricting the sample to patients that were registered with practices using a different clinical computer system provider (TPP), and therefore not used to derive the QCovid model.

This study also has several limitations. First, because of data limitations, we could not derive all predictors in the same way as in the derivation cohort. Despite these inconsistencies, the model had excellent discrimination and calibration. Second, we only focused on COVID-19 related deaths, but not hospital admissions, because of the lack of data. Finally, because the Public Health Data Asset is based on the 2011 Census, our sample was restricted to patients who were enumerated in 2011, that is about 94% of the population living in England in 2011. Recent migrants were excluded from this study, but they tend to be younger than the native population and therefore at lower risk of COVID-19 death.

QCovid represents a new approach for population risk-stratification for adverse outcomes from COVID-19, and our validation indicates that the risk algorithm performs well on external data not used for its derivation. Whilst it has been specifically designed to inform UK health policy and interventions to manage COVID-19 related risks, it also has international potential, subject to local validation. It could also be deployed in a number of health and care applications, either during the current phase of the pandemic, or in subsequent 'waves' of infection. These could include supporting targeted recruitment for clinical trials, vaccine prioritisation, and discussions between patients and clinicians in relation to work and health risks, for example through weight reduction since obesity is the single most important modifiable risk factor for serious COVID-19 complications 8 .

In conclusion, this study presents a robust validation of a new prediction model that could be used to support population risk stratification in relation to public health interventions, for example vaccine utilisation. We anticipate that the algorithms will be updated regularly as understanding of COVID-19 increases, as more data become available, as new variants emerge, effective treatments for COVID become available, the vaccination program rolls-out, immunity levels change or as behaviour in the population changes and hence we anticipate that this validation will need to be repeated on a regular basis. It is important for patients/carers, and clinicians that there is a common appropriately developed evidence-based model that is consistently implemented and is supported by the academic, clinical and patient communities. This will then help ensure consistent policy and clear national communication between policy makers, professionals, employers and the public. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint Rahman from the UK Department of Health and Social Care, Joy Preece and Dan Ayoubkhani from the Office for National Statistics. This project involves data derived from patient-level information collected by the NHS, as part of the care and support of cancer patients. Access to the data was facilitated by the PHE Office for Data Release. The Hospital Episode Statistics data used in this analysis are re-used by permission from NHS Digital who retain the copyright for these data.

The ONS Public Health Linked Data Asset will be made available on the ONS Secure Research Service for Accredited researchers. Researchers can apply for accreditation through the Research Accreditation Service . The data will include all variables used in this analysis, except predictors based on radiotherapy and systemic chemotherapy records, which cannot be shared.

Committee [reference 03/4/021]. VN had full access to all data in the study and takes responsibility of the integrity of the data and the accuracy of the data analysis. The lead author (VN) affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned have been explained.

This study is funded by a grant from the National Institute for Health Research following a commission by the Chief Medical Officer for England whose office contributed to the development of is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

The copyright holder for this this version posted January 25, 2021. ; https://doi.org/10.1101/2021.01.22.21249968 doi: medRxiv preprint the study question and facilitated access to relevant national datasets, contributed to interpretation of data, drafting of the report. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

The copyright holder for this this version posted January 25, 2021. ;

. CC-BY-NC-ND 4.0 International license It is made available under a perpetuity.

is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

The copyright holder for this this version posted January 25, 2021. ; is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

The copyright holder for this this version posted January 25, 2021. ; is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

The copyright holder for this this version posted January 25, 2021. ; https://doi.org/10.1101/2021.01.22.21249968 doi: medRxiv preprint Centiles based on predicted absolute risks in men and women in each period. Sensitivity (cumulative % deaths) is percentage of total deaths in the period that occurred within the group of patients above the predicted risk threshold. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

The copyright holder for this this version posted January 25, 2021. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

The copyright holder for this this version posted January 25, 2021. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

The copyright holder for this this version posted January 25, 2021. Risk threshold is the centile value giving the cut-off of predicted risk over 97 days for defining each group Sensitivity is percentage of total deaths over 97 days that occurred within the group of patients above the predicted risk threshold

Supplementary Figure 1 Centiles based on predicted relative risks compared with someone of the same age/sex with no risk factors. Sensitivity is percentage of total deaths in the period that occurred within the group of patients above the predicted risk threshold.

. CC-BY-NC-ND 4.0 International license It is made available under a perpetuity.

is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

The copyright holder for this this version posted January 25, 2021. ; https://doi.org/10.1101/2021.01.22.21249968 doi: medRxiv preprint

COVID-19 Map -Johns Hopkins Coronavirus Resource Center

Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study

COVID-19 and African Americans

Clinical characteristics of 113 deceased patients with coronavirus disease 2019: Retrospective study

Clinical course and mortality risk of severe COVID-19

Variation in COVID-19 Hospitalizations and Deaths Across New York City Boroughs

Obesity and COVID-19: a call for action from people living with obesity. The Lancet Diabetes & Endocrinology

Obesity Is a Risk Factor for Severe

Prevalence of co-morbidities and their association with mortality in patients with COVID-19: A systematic review and meta-analysis

Shielding from covid-19 should be stratified by risk

Development and validation of QRISK3 risk prediction algorithms to estimate future risk of cardiovascular disease: prospective cohort study

Development and validation of QMortality risk prediction algorithm to estimate short term risk of death and assess frailty: cohort study

Living risk prediction algorithm (QCOVID) for risk of hospital admission and mortality from coronavirus 19 in adults: national derivation and validation cohort study

OpenSAFELY: factors associated with COVID-19 death in 17 million patients

The Impact of COVID-19 on Adjusted Mortality Risk in Care Homes for Older Adults in Wales, United Kingdom: A retrospective population-based cohort study for mortality in

Early Pandemic Evaluation and Enhanced Surveillance of COVID-19 (EAVE II): protocol for an observational study using linked Scottish national data

Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): The TRIPOD StatementThe TRIPOD Statement

The REporting of studies Conducted using Observational Routinely-collected health Data (RECORD) Statement

Protocol for the development and evaluation of a tool for predicting risk of short-term adverse outcomes due to COVID-19 in the general UK population

Explained variation for survival models

A new measure of prognostic separation in survival data

Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors

We acknowledge the contribution Jenny Harries, Nazmus Haq, Joanna Moody and Shamim