key: cord-0821169-e6leuz6y
authors: Leiner, Johannes; Pellissier, Vincent; Nitsche, Anne; König, Sebastian; Hohenstein, Sven; Nachtigall, Irit; Hindricks, Gerhard; Kutschker, Christoph; Rolinski, Boris; Gebauer, Julian; Prantz, Anja; Schubert, Joerg; Patzschke, Joerg; Bollmann, Andreas; Wolz, Martin
title: SARS-CoV-2 rapid antigen testing in the healthcare sector: A clinical prediction model for identifying false negative results
date: 2021-09-10
journal: Int J Infect Dis
DOI: 10.1016/j.ijid.2021.09.008
sha: ca192d011f3a2446e1b37b5eb4c15f382a56c7aa
doc_id: 821169
cord_uid: e6leuz6y

Objectives SARS-CoV-2 rapid antigen tests (RAT) provide fast identification of infectious patients when RT-PCR results are not immediately available. We aimed to develop a prediction model for identification of false negative (FN) RAT results. Methods In this multicenter trial, patients with documented paired results of RAT and RT-PCR between October 1st 2020 and January 31st 2021 were retrospectively analyzed regarding clinical findings. Variables included demographics, laboratory values and specific symptoms. Three different models were evaluated using Bayesian logistic regression. Results The initial dataset contained 4,076 patients. Overall sensitivity and specificity of RAT was 62.3% and 97.6%. 2,997 cases with negative RAT results (FN: 120; true negative: 2,877; reference: RT-PCR) underwent further evaluation after removal of cases with missing data. The best-performing model for predicting FN RAT results containing 10 variables yielded an area under the curve of 0.971. Sensitivity, specificity, PPV and NPV for 0.09 as cut-off value (probability for FN RAT) were 0.85, 0.99, 0.7 and 0.99. Conclusion FN RAT results can be accurately identified through ten routinely available variables. Implementation of a prediction model in addition to RAT testing in clinical care can provide a decision guidance for initiating appropriate hygiene measures and therefore helps avoiding nosocomial infections.

Since its onset , the coronavirus disease 2019 (COVID-19) pandemic, caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), emerged to a major burden for the population in general but especially brought great challenges for the healthcare sector (Miller et al., 2020) . Recent statistics presented by the German federal government agency Robert-Koch-Institute (RKI) show that cough, fever, nasal congestion, sore throat and loss of smell or taste are the most common symptoms caused by (RKI, 2021a). High rates (approximately 50%) of loss of smell or taste among COVID patients were reported in several trials (Tong et al., 2020) but those numbers cannot be confirmed in a real-world setting (RKI, 2021a). Other symptoms like dyspnea, fatigue, myalgia and gastrointestinal symptoms (e.g., diarrhea, nausea) were frequently reported among studies in the early phase of the pandemic (Manoharan et al., 2021) .

The gold standard for the detection of SARS-CoV-2 is the nucleic acid amplification Authors of a recent meta-analysis report pooled sensitivity and specificity estimates of 73.1% 6 and 99.7% respectively for different widely used RATs including FIA-based tests (Brümmer et al., 2021) . Sensitivity decreases markedly when asymptomatic persons are tested (Dinnes et al., 2021) . PoC-testing is an important component of the national strategy for pandemic response in Germany and therefore widely implemented (RKI, 2021b). In the healthcare sector, PoC-testing can provide a huge benefit for the fast identification of infectious patients and initiation of appropriate measures like consecutive isolation. Confirmation of positive RAT test results with RT-PCR is required since false positive tests are possible (Seifried J et al., 2021 , WHO, 2021 . Every inpatient should be tested by RT-PCR at admission in addition to PoC-testing to prevent nosocomial SARS-CoV-2 infections according to German RAT results can have a considerably great impact on the healthcare sector because isolation and appropriate hygiene measures would be possibly not initiated when the patient is initially tested negative. Consecutively, this may also influence the course of the pandemic. The aim of the present study was therefore to identify different clinical predictors for FN results of RATs, determined by positive RT-PCR as a reference standard, in a real-world multicenter patient cohort and to develop a corresponding prediction model.

We conducted a multicenter trial in three centers of the German Elblandkliniken GmbH group. The local ethics committee has not raised any objections to the collection and further 7 analyses of data as part of the study (EK-BR-30/21-1). Informed consent has not been obtained due to the retrospective study design.  musculoskeletal symptoms (myalgia, arthralgia, headache, fatigue).

Collected laboratory values included: Standard C-reactive protein (CRP), leucocyte count, platelet count, lactate dehydrogenase (LDH), aspartate aminotransferase (AST). Furthermore, imaging data (X-ray and chest CT), if present, were reviewed for COVID-specific findings.

First, a confusion matrix was created for the whole dataset displaying the RAT and RT-PCR results (RT-PCR results considered as the ground truth). Discrimination metrics (sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV)) were computed. As our primary goal was to create a model predicting the probability of a RAT negative patient being SARS-CoV-2 positive (therefore predicting a FN RAT, as determined by positive RT-PCR), we focused on the proportion of patients with negative RAT results (true negative, TN and false negative, FN) for further analyses.

In a first step, we identified missing values in the dataset of TN and FN patients. Due to the high number of missing values, the medical history variables "known COVID-19 contact" and "loss of smell and taste" were not taken into account for model development as were LDH and AST for laboratory values and all imaging data. The dataset was split in 75%/25% portions for model testing and model training. The sampling was stratified for RT-PCR results (RT-PCR positive/RT-PCR negative ratio was identical in each subset).

Remaining laboratory values (CRP, leucocyte and platelet count) were transformed into categorical variables, using the median as a cut-off value (CRP = 7.4 mg/l, leucocytes = 9.4/nl, thrombocytes = 237/nl). We evaluated three different models for prediction of FN RAT results as determined by RT-PCR consisting of two demographic variables, three laboratory values and five symptoms ( Figure 1 ).

With regard to statistical methods, we utilized a Bayesian implementation of logistic regression, using weak priors (i.e. normally distributed prior, with a mean of 0 and a large variance) (Gelman et al., 2008) . Models were fitted using four Markov chains (Monte Carlo-9 algorithm), each with 10,000 warm-up iterations followed by 10,000 sampling iterations.

Comparisons between models and for individual parameters were carried out using the Bayes Factor (BF) with appropriate interpretation (Supplementary Table S2 ) (Jeffreys, 1961) . Model performance was further evaluated by computing receiver operating characteristic (ROC) area under the curve (AUC). Pearson's Chi-squared Test and Mann-Whitney-U-Test were used for intergroup comparisons where appropriate. All statistical analyses were carried out using the rstanarm package in the R environment for statistical computing (version 4.0.2).

Overall sensitivity and specificity for the Standard F RAT in the whole dataset containing Figure S1 ). Most important variables for model performance with respect to BF were ( Table 2) : leucocyte count, fever, breathing frequency, dyspnea and musculoskeletal symptoms. The variables "high leucocyte count" and "high platelet count" show a negative estimate which means that COVID-19 in our study was rather associated with leukopenia and thrombocytopenia.

The probability for a RT-PCR positive result at the time point of the patient's hospital admission can be calculated by using a specific formula derived from the models. As a first step, all the variables used in the model have to be collected and each coefficient (estimates, see Table 2 ) is then multiplied by the value of the variable. For "Age", this is the age of the patient given in years and for categorical variables the binary numbers 0 and 1 (no/yes) have to be used. The cut-off for laboratory values has to be applied like defined previously: "High CRP" is defined by values above 7.4 mg/l, "High leucocyte count" is defined by values above 9.4/nl and "High platelet count" is defined by values above 237/nl. These multiplications are summed, and -5.08 is added, corresponding to the estimate of the intercept. Finally, this sum can be transformed into a probability using the following equation:

Y is the sum defined above, P is the probability, and e the Euler's number.

In order to compute classical discrimination metrics (sensitivity, specificity, PPV, NPV), a cut-off threshold was defined (Youden's index) (Youden, 1950) . For model 3, we defined the cut-off threshold as 0.09 (9% probability for positive RT-PCR result) because this index yields a high sensitivity of 85% together with a PPV of 70% (specificity = 0.99; NPV = 0.99; accuracy = 0.98; see Figure 4 ). Transferred to the clinical perspective and performance of RATs, this threshold, applied on the test dataset, would result in 85% of the patients being classified correctly as FN with subsequent isolation while 30% (1-PPV) are unnecessarily isolated.

In a multicenter retrospective study of 4,076 patients, we evaluated clinical predictors for probable SARS-CoV-2 infection and compared different prediction models for positive RT-PCR results in the setting of an initially negative RAT. As a result, we propose a model (model 3) utilizing ten easily collectable and fast available variables, consisting of demographics, symptoms and laboratory parameters, with high accuracy and an AUC of 0.971. As the pandemic is ongoing, application of our model in clinical routine could provide a great benefit for the fast assessment of both inpatients and outpatients regarding their COVID-19 status in addition to PoC-testing by RATs as it is crucial to differentiate between FN and TN results in the healthcare sector in order to initiate appropriate hygiene measures and avoid nosocomial infections. For practical use, the development of, for example, a computerized tool which uses the above mentioned formula to calculate individual probabilities is conceivable. In a hospital setting, the patient-specific variables can be obtained directly at the patient's admission (after performance of RAT and RT-PCR respectively) and entered manually into the calculator. The resulting probability value for present COVID-19 (in patients with negative RAT results) can guide hospital staff on whether an initial isolation of the patient while awaiting RT-PCR results is necessary despite negative RAT. This process could easily be carried out and implemented as a routine measure.

Automatic calculators utilizing the HIS as a data source represent another option for realworld application of our prediction models. 

We acknowledge several limitations in connection with this study. First, data collection was performed retrospectively which could have influenced data quality, especially regarding symptom assessment and also resulted in a significant amount of missing values. 

We evaluated clinical predictors of FN results of RATs and propose a prediction model consisting of widely available and easily collectable variables. The model showed an excellent discriminatory performance. Its routine implementation in healthcare both in an inpatient and outpatient setting could help identifying patients at high risk for COVID-19 despite negative RAT result and could therefore influence the course of the pandemic with respect to avoiding nosocomial infections. 

Correlation of Chest CT and RT-PCR Testing for Coronavirus Disease

AWMF): Empfehlungen zur stationären Therapie von Patienten mit COVID-19 -Living Guideline

A Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) Prediction Model From Standard Laboratory Tests

Antigen-Tests zum direkten Erregernachweis des Coronavirus SARS-CoV-2; 2021

Bio-Rad/Seegene: CFX96™ Dx Real-time PCR detection system

Diagnostic Performance of an Antigen Test with RT-PCR for the Detection of SARS-CoV-2 in a Hospital Setting

The accuracy of novel antigen rapid diagnostics for SARS-CoV-2: a living systematic review and meta-analysis

Comparative diagnostic performance of rapid antigen detection tests for COVID-19 in a hospital setting

Comparison of seven commercial SARS-CoV-2 rapid point-of-care antigen tests: a single-centre laboratory evaluation study. The Lancet Microbe

Rapid, point-of-care antigen and molecular-based tests for diagnosis of SARS-CoV-2 infection

European Commission Directorate-General For Health And Food Safety (EC-DG SANTE): A common list of COVID-19 rapid antigen tests, including those of which their test results are mutually recognised, and a common standardised set of data to be included in COVID-19 test result certificates

The Allplex 2019-nCoV (Seegene) assay: which performances are for SARS-CoV-2 infection diagnosis?

Validation of a rapid antigen test as a screening tool for SARS-CoV-2 infection in asymptomatic populations. Sensitivity, specificity and predictive values

Epidemiologic characteristics associated with SARS-CoV-2 antigen-based test results, rRT-PCR 19 cycle threshold values, subgenomic RNA, and viral culture results from university testing

A weakly informative default prior distribution for logistic and other regression models

Clinical Characteristics of Covid-19 in

Clinical Characteristics of Coronavirus Disease 2019 in China

Prospective study of 1308 nasopharyngeal swabs from 1033 patients using the LUMIPULSE SARS-CoV-2 antigen test: Comparison with RT-qPCR

Five Antigen Tests for SARS-CoV-2: Virus Viability Matters

Clinical features of patients infected with 2019 novel coronavirus in Wuhan

Theory of Probability

Development of machine learning models to predict RT-PCR results for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) in patients with influenza-like symptoms using only basic clinical data

Evaluating the clinical utility and sensitivity of SARS-CoV-2 antigen testing in relation to RT-PCR Ct values

Predictors of negative first SARS-CoV-2 RT-PCR despite final diagnosis of COVID-19 and association with outcome

Performance of a novel diagnostic assay for rapid SARS-CoV-2 antigen detection in nasopharynx samples

Serum biomarkers for prediction of mortality in patients with COVID-19

Evaluation of rapid antigen test for detection of SARS-CoV-2 virus

Evaluating clinical characteristics studies produced early in the Covid-19 pandemic: A systematic review

Disease and healthcare burden of COVID-19 in the United States

SARS-CoV-2 antigen rapid immunoassay for diagnosis of COVID-19 in the emergency department

Development and validation of risk prediction models for COVID-19 positivity in a hospital setting. 21 International journal of infectious diseases : IJID : official publication of the

National Institutes of Health (NIH): COVID-19 Treatment Guidelines

Klinische Performance eines neuen SARS-CoV-2-AntigenTests in der Notaufnahme eines Maximalversorgers

Evaluation of two fluorescence immunoassays for the rapid detection of SARS-CoV-2 antigen-new tool to detect infective COVID-19 patients

Most frequent symptoms caused by the coronavirus (COVID-19) in Germany in 2021

Nationale Teststrategie -wer wird in Deutschland auf das Vorliegen einer SARS-CoV-2 Infektion getestet?

SD Biosensor/Bestbion dx: Standard F COVID-19 Ag FIA; 2021a

Bestbion dx: Standard F COVID-19 Ag FIA Instruction Manual

Allplex™ SARS-CoV-2/FluA/FluB/RSV Assay

Antigentests als ergänzendes Instrument in der Pandemiebekämpfung

Epidemiological and Clinical Predictors of COVID-19

The Prevalence of Olfactory and Gustatory Dysfunction in COVID-19 Patients: A Systematic Review and Metaanalysis

Machine Learning Prediction of SARS-CoV-2 Polymerase Chain Reaction Results with Routine Blood Tests

Comparison of seven commercial RT-PCR diagnostic kits for COVID-19

Antigen-detection in the diagnosis of SARS-CoV-2 infection using rapid immunoassays -interim guidance 2021

Index for rating diagnostic tests

Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study

A Novel Coronavirus from Patients with Pneumonia in China

Machine learning-based prediction of COVID-19 diagnosis based on symptoms

Thrombocytopenia Is Associated with COVID-19 Severity and Outcome: An Updated Meta-Analysis of 5637 Patients with Multiple Outcomes

Age (years), mean (SD)

Gender: female, n (%) 1,480 (51%)

Gender: male, n (%) 1,397 (49%) 70 (58%)

Laboratory values CRP (mg/l), mean (SD)

Leucocyte count (/nl), mean (SD)

Platelet count mean (/nl), mean (SD)

Symptoms Fever (Temperature ≥ 38 °C), n (%)

Abdominal symptoms, n (%) 10 (0.3%) 18 (15%)

Musculoskeletal symptoms, n (%) 7 (0.2%) 20 (17%)

FN = false negative RAT result

SD = standard deviation; TN = true negative RAT result

We declare no conflicts of interest associated with this publication.Funding: This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.