key: cord-0666204-s3fmqo61
authors: Covello, Len; Gelman, Andrew; Si, Yajuan; Wang, Siquan
title: Routine Hospital-based SARS-CoV-2 Testing Outperforms State-based Data in Predicting Clinical Burden
date: 2021-04-09
journal: nan
DOI: nan
sha: 9131f1c2c980b4df917c8c3fb792d240424395f3
doc_id: 666204
cord_uid: s3fmqo61

Throughout the COVID-19 pandemic, government policy and healthcare implementation responses have been guided by reported positivity rates and counts of positive cases in the community. The selection bias of these data calls into question their validity as measures of the actual viral incidence in the community and as predictors of clinical burden. In the absence of any successful public or academic campaign for comprehensive or random testing, we have developed a proxy method for synthetic random sampling, based on viral RNA testing of patients who present for elective procedures within a hospital system. We present here an approach under multilevel regression and poststratification (MRP) to collecting and analyzing data on viral exposure among patients in a hospital system and performing statistical adjustment that has been made publicly available to estimate true viral incidence and trends in the community. We apply our MRP method to track viral behavior in a mixed urban-suburban-rural setting in Indiana. This method can be easily implemented in a wide variety of hospital settings. Finally, we provide evidence that this model predicts the clinical burden of SARS-CoV-2 earlier and more accurately than currently accepted metrics.

Early knowledge of incidence and trends of SARS-Cov-2 transmission in communities is crucial, but in the absence of universal screening or random testing, interested parties have been left to extrapolate impressions of community viral behavior from nonrepresentative data. Public health professionals have relied on state-sourced positivity rates and raw numbers of positive tests in any given jurisdiction as proxies for the true SARS-Cov-2 burden. Unfortunately, these presumed proxies are subject to significant selection bias, as most testing protocols understandably target symptomatic and presumed-exposed populations. Further, tests have been applied with different criteria over time and geography according to test availability, perceived community SARS-Cov-2 burden, and disparate clinical or political testing norms. The uncontrolled nature of the data raises questions or criticism about their validity as determinants of policy that delimits clinical or economic behavior. Alleva et al. provide a review of strategies and experiences currently in progress to estimate the SARS-CoV-2 incidence in the community (1) . Briefly, the existing approaches include massive test campaigns without a formal sampling design, diagnostic tests through a probabilistic sample, volunteer massive surveys and supplements of existing sample surveys. Absent randomized testing of the population or community, we need a means of normalizing currently available data to better track trends in the true underlying incidence, either as a more reliable metric or as a reassurance of the validity of our current ones in predicting clinical behavior of SARS-CoV-2.

In the present article, we apply multilevel regression and poststratification (MRP), a standard adjustment method used in survey research that is particularly effective when sample sizes are small in some demographic or geographic slices of the data (2, 3) . MRP has increasingly been shown to be useful in public health surveys, and even be successful in highly unrepresentative 4 probability or non-probability samples (4) (5) (6) . We work with data from the Community Hospital group in Indiana, which serves an urban-suburban-rural mix of patients. COVID testing was already being performed for patients in this hospital system, and it was relatively costless to augment this data collection with the statistical analysis presented here. For this reason, we believe that this method can be easily implemented in a wide variety of hospital settings.

Upon reopening without restriction to elective medical and surgical procedures after the early spring COVID-19 outbreak, clinical professionals in our hospital system were sufficiently concerned about asymptomatic viral shedding to test all patients for acute SARS-Cov-2 infection before performing any such procedure. All elective patients for invasive procedures are presumptively asymptomatic, as any potential surgical patient acknowledging symptoms or presenting a recent history of known viral exposure would have the procedure canceled or deferred. All prospective surgical (and other invasive procedure) patients were subjected to a preoperative evaluation of these issues and excluded if they showed evidence of symptoms or exposure.

This population presented a potentially valuable resource. All patients used the same test administered within a health system by similar health care professionals. There is a broad age, racial/ethnic, and economic diversity to this group, and its only overt correlation to disease status is that it is specifically selected for a lack of symptoms and a negative exposure history. By way of contrast, cumulative state-wide data include testing data from multiple sources (private clinics, large hospitals, pop-up clinics, large employers, universities, etc.), with different types of tests, different levels of training for testers, different test settings (clinic office or drive through clinic),   where the test results are valuable but likely much more variable than those under consideration. More importantly, the criteria used in the cumulative state data to determine whether to test individuals in the first place are subject to varying prior assumptions. As an example, an outpatient exposed to a family member suspected of active COVID has a substantially different prior than a dyspneic patient admitted to the emergency department, yet the implication of the state data trends is that these positive cases should be handled similarly.

Though not ideal, our test group is therefore a promising proxy for the general community.

SARS-CoV-2 has clearly shown the ability to spread throughout the population via both asymptomatic and symptomatic infection. If we were to assume the as yet unverified but reasonable hypothesis that, for any uniform demographic, the ratio of asymptomatic-tosymptomatic viral infection is constant, then the asymptomatic population in a community would vary in a strict ratio with overall prevalence, and could therefore serve as an excellent proxy for true viral incidence. The trending of asymptomatic infection would be expected to be strictly proportional to clinical infection. To the degree to which external comorbidities or other factors might contribute to variation in this ratio across the sample, we anticipate that much of this variation would be captured by our demographic adjustments.

Of crucial importance, our sample group varies from a true random sample in predictable ways.

It is selected rigorously for asymptomatic/non-exposed status, and age, racial/ethnic, and geographic demographics are well documented in the hospital electronic health records (EHR). It remains only to normalize our sample to the demographics of the larger community to represent the general population.

We subjected all patients to polymerase chain reaction (PCR) testing for viral RNA, 4 days before their intended procedure. Samples were submitted to LabCorp for analysis using the Roche cobas system. This testing regime was used throughout the study interval and continues to be employed without change to the present day. A 70% clinical sensitivity is presumed for this test, based on near 100% internal agreement with positive controls on in vitro analytics (7) and broadly observed clinical performance of PCR testing throughout the pandemic (8); however, asymptomatic and pre-symptomatic patients may be harder to detect than predicted by these analytic data, as dates of infection as well as symptom status/onset are known to have a large effect on sensitivity (9) . These effects would need to be acknowledged and, to the degree possible, accounted for in the model. Specificity is near 100%, with false positives likely generated only by cross-contamination or switched samples. These false positives become important when underlying prevalence is near zero (10), as was the case for our community this summer, and we have applied a Bayesian procedure to account for the false positivity. We evaluate whether the estimated trends and magnitudes are robust against the sensitivity and specificity parameters.

We are interested in rates of SARS-CoV-2 infection in two populations: 1) Individuals undergoing care within the hospital system as patients, and 2) the community from which the hospital draws as a whole. In addition to adjusting for measurement error associated with PCR testing for SARS-CoV-2 infection, we need to generate standardized estimates that reflect prevalence in the populations of interest rather than merely our sample of elective surgery patients we are drawing on. We anticipate that this sample of asymptomatic patients is a fairly representative group with minor discrepancy selected from the community-at-large, but also expect that poststratification to the target population with matching sociodemographics would help enhance the accuracy of our conclusions. We acknowledge that those who seek even elective hospital-based procedures may further vary from the overall community with respect to their comorbidities (hence, their susceptibility to symptomatic COVID-19), but believe it is reasonable to infer that MRP normalization of their age, race/ethnicity, gender, and geography will account for much of that discrepancy.

We use a Bayesian approach to account for unknown sensitivity and specificity and apply MRP to testing records for population representation, here using the following adjustment variables: We particularly care about changes in SARS-Cov-2 incidence over time. Indeed, even if our demographic and geographic adjustment is suspect (given systematic differences between sample and populations), the greatest clinical utility lies in being able to predict how much the clinical burden present today is likely to change in the future. Here, the adjustment may be particularly important, as the mix of patients has changed somewhat during the study interval. The statistical 8 details are included in the Supplement. We perform all computations in R (11); data and code are publicly available at https://github.com/yajuansi-sophie/covid19-mrp.

We began the data collection with a few hypotheses or speculations. First, we expected that the ratio between asymptomatic and symptomatic patients would be relatively constant, for a uniform demographic distribution specific to age, gender, and race/ethnicity. Second, we anticipated that changes in PCR positivity among asymptomatic individuals would precede changes in symptomatic PCR-detected infections by several days, because of the known temporal relationship of viral shedding to the onset of clinical disease (12) . Third, these hypotheses would imply that trends in our asymptomatic SARS-COV-2 infections would predict the behavior of the virus within the community as a whole. To this end, we aim to determine whether our model mirrors or predicts hospitalization rates as a proxy for clinical viral burden.

In summary, we anticipated that appropriate modeling of the PCR dataset would allow us to measure changes in acute infection incidence as an early warning metric to grasp the developing trend of the disease, or at least in concert with any changes. The procedure provides accurate assessment of trends, rather than incidence, and offers more temporally relevant information than the current use of percent-testing-positive. Further, we aimed to evaluate the validity of positivity and counts of positive cases as metrics to predict clinical burden. We present the weekly number of asymptomatic patients seeking elective surgeries in the hospital system, which shows stable sample sizes, and examine the observed sociodemographic distributions of asymptomatic and symptomatic patients receiving PCR tests over time and find that the asymptomatic patients' profiling is stable, while the sample decomposition of the symptomatic patients changes over time. Details are presented in eFigures 1-2 of the Supplement. This discrepancy provides supporting evidence for our pre-study hypothesis that we should treat the asymptomatic samples as a substantially better proxy sample of the target hospital or community population than the corresponding symptomatic data.

The variation of prevalence could be due to various sample decompositions across time, but variation in thresholds for testing symptomatic people over time and demographic is certainly a likely factor. Overall, our analysis here calls into question relying upon symptomatic data trends-as is currently the norm-in understanding the underlying true viral trends in the community and argues that asymptomatic testing is likely to be a superior proxy.

To correct for discrepancies between the sample demographics and those of the community at large, as necessitated by the above observations, we next apply MRP to model the incidence and poststratify to the hospital and community population for representative prevalence estimates.

The outputs are given in Figure 2 . 

We are interested in evaluating whether MRP-adjusted asymptomatic patients could track SARS-CoV-2 related hospitalization rates-as measured by counts of hospitalizations and emergency department (ED) visits-better than the currently applied metrics within our counties: positivity rate and counts of positive cases. Our expectation was that hospitalization census would lag viral 14 incidence by a week or more and that ED COVID-19 visits would track actual viral incidence, perhaps with a few days' lag. These inferences follow from known lag times from exposure to symptoms to serious illness (12) . To test this conjecture, we focus on the September 2020 through February 2021 interval as that timeframe encompasses all of the observed growth in viral burden after very low levels throughout late spring and the summer.

Our side-by-side analysis is illustrated in Figure 3 . Our analysis indicates that applying our MRP normalization to data on the prevalence of asymptomatic SARS-CoV-2 infection produces a valuable leading indicator of hospital and community risk. When we set out to create a model for tracking viral incidence, we recognized substantial shortcomings in the available testing and its interpretation. While state data has become much richer and testing protocols more uniform since we started applying our model, selection bias is still a substantial concern. Our goal for this study was to develop an easily implemented testing strategy-available to any hospital system-that, after demographic and geographic adjustment, could reasonably approximate a representative sample. In so doing, we hoped to assess the reliability of currently accepted metrics in their prediction of virus trends and, if possible, to improve our ability to anticipate those trends.

The asymptomatic preoperative patients we identify with our protocol are a favorable group to build upon. All sizable hospital systems have a ready-made group of such patients who can produce a large number of data points quite rapidly. As patients continue to seek medical procedures, the population continues to naturally expand over time, and lends itself to trending data. In our nearly 900-bed hospital system, we have thus far generated over 30,000 data points over 43 weeks, representing a community of approximately 700,000 residents. The weekly number of data points has been fairly stable over time, and that observation is likely similar to many hospital systems. We have demonstrated that this sample population is fairly representative of the community demographics as a whole and that there has been minimal sample decomposition over time. That this population stability is not matched by similar demographic stability in the symptomatic population and that we are able to employ MRP to account for any demographic skew and instability in our own protocol both strongly argue that our model is far more representative of random sampling than the currently employed positive case and positivity data. We argue that hospital-based asymptomatic testing with MRP is a more reliably random metric than any currently available and is easily generated from the routine testing of patients prior to their scheduled procedures.

Having established a reasonable statistical validity for our model, we wished to use it to measure the reliability of current state-based metrics. Our analysis finds that in our community, all of the metrics trend similarly during viral surges. We would support the current view that numbers of positive cases and positivity both remain relatively stable during periods that our pseudorandom proxy method predicts to be stable and have increased during periods that our proxy predicts Finally, we wanted to test each of these metrics as predictors of clinical burden. During the entire study period, we have used our model to predict clinical needs: staffing, bed and ventilator availability, personal protection equipment supplies, and so forth. Our general observation was that this proxy provided us some useful lead time to prepare for the virus. When we compare our model's behavior to that of the standard metrics, we find it to be generally a better predictor of clinical burden. The effect is best seen in our November data. During the week of November 3-10, we were able to predict that viral transmission was decreasing and that our hospitalization was likely to be at or near its peak. Comparison of our model with ED COVID presentations in 18 our area demonstrates a precise correlation. It is clear that new acute presentations correlate quite precisely with our metric, and that these changes occur about a week before positive cases and hospitalization census data change. Further, we see positivity rates continue to rise in our area well past the time that our metric and numbers of positive cases have declined. Given that ED visits and hospitalization census rates have also declined in that interval, we find that our model and the number of positive cases appear to be much better and more current predictors of the true viral clinical burden than positivity rates are.

In dealing with a case surge, the extra time of preparedness has been useful and nontrivial. Great benefit may also accrue from recognizing decreased transmission earlier, as we feel the model is able to do, as it may allow for opening up of needed clinical services and socioeconomic commerce in a community earlier than might otherwise be contemplated. In this sense, adherence to positivity rates may be particularly damaging.

We believe our model to be easily generalizable to many hospital systems. As discussed, the sample population and testing regime are readily available, likely to be reasonably representative and stable demographically over time, and easily normalized to true community demographics using the MRP code that we have made available. This approach represents a simple proxy for random sampling for any community that chooses to employ it. Further benefits might be gained by combining information from different hospital systems. The best way forward might be for individual hospitals and medical groups to gather and analyze their data as we propose in this article, with all the (de-identified) data shared in a common public repository, so that it would be possible for researchers to learn more by analyzing trends as they develop in the pooled dataset.

This could be similar to other national data pooling efforts such as in the U.S. and Israel (14) (15) .

In addition, though, we demonstrate the clinical utility of less rigorous approaches as well.

Should a system choose to track its patients according to our testing protocol, but not incorporate the MRP adjustments, the relative stability of the population demographics suggests that the trends remain quite valid. Our regression models have shown potentially strong effects of age and racial/ethnic status on our metric so that one would need to ensure at least reasonable stability of those particular traits to trust observed raw trends without formal MRP adjustment.

We also find that while results depend strongly on the sensitivity of the test being employed, the trends in the results do not (details in the Supplement).

This finding is encouraging for longer-term monitoring. Very inexpensive antigen testing is now becoming broadly available. These tests may be less sensitive or more time-specific than the PCR-based RNA testing we have been using and therefore less able to verify the true magnitude of viral spread. Nonetheless, our data show that they will likely function perfectly well to follow viral transmission and clinical burden trends, especially if normalized by MRP. Practically speaking, these trends are the prime concern of most healthcare entities.

Here we present the supplemental materials on the summary of the paper structure, modeling details, and data showing demographic stability, comparison with publicly released records, and posterior predictive checks, for the paper, "Routine Hospital-based SARS-CoV-2 Testing

Outperforms State-based Data in Predicting Clinical Burden."

Our primary goal is to track SARS-CoV-2 infection prevalence over time, and to do so in a reliable way. We are assuming that the ratio of the number of asymptomatic patients to that of the symptomatic patients is fixed. We would then need to normalize each group from the sample demographics to true demographics via MRP. Once that is accomplished, we posit that our metric represents an appropriate approximation of a true random sample. In particular, it either is superior to what is being done now (tracking positivity and overall positive case numbers) or it provides a respectable metric to verify that those currently used data are reasonable approximations of fact. All of our data representations should either support one of those contentions, or provide evidence supporting or refuting critiques of our approach. To that end, we need to follow prevalence changes over time, but verify (through MRP and statistical analysis) that these trends are real changes in the community and not changes in the sample demographics. We then compare the MRP normalized trends in our model with currently employed metrics of viral prevalence trends: namely, positivity rates and numbers of positive tests in the community. Finally, we compare these metrics with hospitalization rates to determine 23 how predictive our model and the current metrics may be of the community clinical burden of the virus.

We denote the test result for individual i as y # , where y # = 1 indicating a positive result and y # = 0 indicating negative. Let p # = Pr ( -= 1) be the probability that this person tests positive. The analytic incidence p # is a function of the sensitivity δ, specificity , and the true viral incidence π # for individual i: p # = (1 − γ)(1 − π # ) + δπ # . We fit a logistic regression for π # with covariates including sex, age, race, county, time, and the two-way interaction between sex and age. Using the model-predicted incidence π 6 # , we apply the sociodemographic distributions in the hospital system and the community to generate the population level prevalence estimates, as the poststratification step in MRP.

We use the following logistic regression in (1) According to the test protocol, the sensitivity is around 70%, and the specificity is around 100%.

We solicit prior information from previous testing results (2 The positions of the months on the x-axis correspond to the week of data containing the first of that month.

Comparison with publicly released data eFigure 3 compares the MRP estimates of asymptomatic patients with the publicly released positivity rate data in District 1, where our hospital system is located, and the state hospital bed occupancy rate. The district-level test positivity rate is higher than that of MRP estimates, as expected, since the state data largely involves the testing of symptomatic patients. The two positivity rates both have an increasing trend since September, but deviate in November, where MRP estimates start decreasing. The occupancy rate of hospital beds in the data generally follows a similar trend but presents a lower increasing rate in November than the positivity rate in District 1. We observe that MRP asymptomatic data are able to predict clinical behavior a week or two earlier than the District 1 positivity data.

To evaluate the model fitting, we apply a posterior predictive check by generating replicated data from the posterior model distributions with the same sample size as the raw data. We use the collected sample decomposition records every week and estimated prevalence rates of poststratification cells, defined by the cross-tabulation of age, gender, race/ethnicity, and county information, to generate replicated test results. We compare the weekly prevalence rates between the replicated data and observed data. eFigure 4 shows that the model of asymptomatic patients can capture the raw data structure, implying that this aspect of the data is captured well by the fitted model. We account for the uncertainty of sensitivity and specificity in a Bayesian framework and use the meta-analysis study findings as the prior specification (1). The presented results above are based on the prior information concentrated on the sensitivity value of 70% and specificity 100%.

When we set the prior sensitivity data as 70/100, the MRP estimates are similar under the current prior setting. We also compare the PCR results when the prior sensitivity value is set at 65%, 60% and 55%; the results are approximately inflated by the reciprocal of the value of sensitivity,

A sampling approach for the estimation of the critical parameters of the SARS-CoV-2 epidemic: an operational design

Poststratification into Many Categories Using Hierarchical Logistic Regression

Bayesian Hierarchical Weighting Adjustment and Survey Inference

Multilevel Regression and Poststratification: A Modeling Approach to Estimating Population Quantities From Highly Selected Survey Samples

Multilevel regression and poststratification for small-area estimation of population health outcomes: a case study of chronic obstructive pulmonary disease prevalence using the behavioral risk factor surveillance system

Forecasting Elections with Non-Representative Polls

Test performance in individual samples

False Negative Tests for SARS-CoV-2 Infection-Challenges and Implications

Variation in False-Negative Rate of Reverse Transcriptase Polymerase Chain Reaction-Based SARS-CoV-2 Tests by Time Since Exposure

Bayesian Analysis of Tests with Unknown Specificity and Sensitivity

R: A language and environment for statistical computing. R Foundation for Statistical Computing

The Incubation Period of Coronavirus Disease 2019 (COVID-19) from Publicly Reported Confirmed Cases: Estimation and Application

COVID-19 Region-Wide Test, Case, and Death Trends

A framework for identifying regional outbreak and spread of COVID-19 from one-minute population-wide surveys

Weekly sample sizes available across time in different groups

Week Total Male Female

Bayesian Analysis of Tests with Unknown Specificity and Sensitivity

COVID-19 Antibody Seroprevalence

suggesting that the magnitude of our estimates is sensitive to the quality of PCR tests.Nevertheless, the trends within a given sensitivity remain stable.