key: cord-0905153-lt69p04w
authors: Hayer, Johannes; Kasapic, Dusanka; Zemmrich, Claudia
title: Real-world clinical performance of commercial SARS-CoV-2 rapid antigen tests in suspected COVID-19: A systematic meta-analysis of available data as per November 20, 2020
date: 2021-05-17
journal: Int J Infect Dis
DOI: 10.1016/j.ijid.2021.05.029
sha: 1458a995486ce72b77b7bf25e07adc70be3b3a13
doc_id: 905153
cord_uid: lt69p04w

OBJECTIVES: Rapid antigen tests, or RATs, are a type of lateral flow chromatographic immunoassay that have been utilized to aid diagnosis of SARS-CoV-2 infection. We performed a systematic meta-analysis to compare the real-world performance of commercially available RATs. METHODS: We searched several databases and websites for manufacturer-independent prospective clinical performance studies comparing SARS-CoV-2 RATs and RT-PCR. Only studies on RATs that did not need a separate reader for result retrieval and that reported data on viral load, patients’ symptom status, sample type, and PCR assay used were included. RESULTS: 19 studies utilizing 11,109 samples with 2,509 RT-PCR-positives were included. RAT sensitivity varied between 28.9% (95% CI 16.4–44.3) and 98.3% (95% CI 91.1–99.7), likely dependent upon population characteristics, viral load, and symptom status. RAT specificity varied between 92.4% (95% CI 87.4–95.9) and 100% (95% CI 99.7–100) with one outlier. The RATs by Roche Diagnostics/SD Biosensor and Abbott had the highest pooled sensitivity (82.4% [95% CI 74.2–88.4] and 76.9% [95% CI 72.1–81.2], respectively). Sensitivity in high-viral-load samples (cycle threshold ≤25) showed heterogeneity among the different RATs. CONCLUSION: The RATs offered by Roche Diagnostics/SD Biosensor and Abbott provide sufficient manufacturer-independent, real-world performance data to support their use for detection of current SARS-CoV-2 infection, particularly in high-viral-load populations.

European Centre for Disease Prevention and Control, 2020) . RT-PCR requires a professionally run laboratory with molecular-biological competence, as well as transport infrastructure between the place of sample collection and the laboratory. Rapid antigen tests, or RATs, are a type of lateral flow chromatographic immunoassay used to support the rapid diagnosis of individuals suspected of SARS-CoV-2 infection, either in those presenting symptoms or in those who have had contact with positive cases. These point-of-care tests are less clinically sensitive than RT-PCR assays but offer a comparable specificity (Centers for Disease Control and Prevention, 2020) . Several RATs have been authorized for use under EUA and/or the CE mark (US Food and Drug Administration, 2020; World Health Organization, 2020b) , presenting manufacturer-generated clinical performance data across heterogeneous patient populations.

Numerous variables contribute to the sensitivity and specificity of RATs and therefore the applicability in different testing scenarios. Notably, there are large differences in sensitivity according to viral load (Dinnes et al., 2020) . As such, most RATs are intended for use in patients up to 5-7 days after symptom onset in order to increase the probability of having sufficiently high viral load for detection. If RATs are used to assess asymptomatic contacts of index cases, time from symptom onset is not available and the date of infection can only be assumed as the date of contact. The use of a RAT for screening within a low-prevalence population may not be J o u r n a l P r e -p r o o f appropriate, as fewer cases with a detectable high viral load are expected within this group, decreasing the positive predictive value of the test accordingly (Centers for Disease Control and Prevention, 2020) .

Whilst studies that conduct direct head-to-head comparisons benefit from reduced experimental heterogeneity (e.g. PCR assay performance differences), repeat sample extraction is invasive, and the ability to conduct these studies is potentially hampered by the high number of screened persons required to detect a sufficient number of positive cases.

In order to provide clarity on the real-world clinical performance of RATs, we compiled all available manufacturer-independent, prospectively collected clinical data using RATs. Data were from RATs commercially available as of November 20, 2020 and intended for the qualitative detection of SARS-CoV-2 present in human nasopharynx in individuals suspected of SARS-CoV-2 infection. We aimed to harmonize the data regarding the aforementioned performanceimpacting factors, using mathematical methods to ensure that the data are comparable despite varying methods of presentation in the publications considered for this analysis.

We searched MEDLINE®, EMBASE®, BIOSIS™ (ProQuest®), and Derwent Drug File (ProQuest®) for any clinical performance studies using a commercial SARS-CoV-2 RAT for the following search terms: "COVID-19" OR "SARS-CoV-2" OR "2019-nCoV" OR "coronavirus disease 2019" OR "novel corona virus" OR MESH Entries for Coronaviridae (incl. narrow terms) OR EMTREE Entries for Coronaviridae (incl. narrow terms) OR MESH/EMTREE Entries for "severe acute respiratory syndrome" (incl. narrow terms) AND "rapid antigen test*" OR "rapid antigen assay*" OR "standard Q covid-19 ag" AND "sensitivity" OR "specificity" OR "clinical performance" OR J o u r n a l P r e -p r o o f "positive agreement" OR "negative agreement" OR "concordance" OR "validation" OR "evaluation" OR "accuracy". 

Eligible studies were those: i) reporting clinical performance data of standalone RATs (i.e. tests that did not require a separate reading device); ii) that measured the performance of RATs against any RT-PCR assay (reference standard; commercial RT-PCR assay or in-house); iii) performed independent of funding by the manufacturer or distributor; iv) that utilized only nasopharyngeal or combined oro-/nasopharyngeal sample types; v) where tests were performed at the point-of-care or at a laboratory after sample transport in viral transport media; vi) that provided information on cycle threshold (Ct) values or symptom status. We excluded retrospective laboratory studies.

The data were extracted to an electronic database and stratified according to RAT. Information on the test utilized, number of participants, percentage of symptomatic patients, specimens (number of PCR positives), and clinical performance (stratified by Ct if available) were recorded for each study.

Performance results of the RATs were reported as sensitivity and specificity measured against the reference standard of RT-PCR and summarized in tables. As confidence intervals reported in the publications used different methods, for better comparability, all confidence intervals were recalculated using the exact Clopper-Pearson method. Due to the heterogeneity in sub-groups and the small number of studies for some RATs, we report the differences between tests descriptively rather than statistically.

The meta-analysis of the performance results of the RATs against the RT-PCR methods was performed using the statistical software R (R Foundation for Statistical Computing, 2020). The metaprop function from the "meta" package (Schwarzer, 2020) was used to calculate the effect size for each individual test and pooled overall. The results of the AAZ-LMB and RapiGEN RATs were included, despite only one study being available for each of the tests. The results are shown as a forest plot summarizing the sensitivities found in the different studies.

The bivariate model (Reitsma et al., 2005) was fitted as a linear mixed model and variance components were estimated by restricted maximum likelihood, using the reitsma function from the "mada" package (Doebler, 2017) for each system investigated in more than one study. The results are presented as a summary receiver operating characteristic (SROC) curve plot (Rutter et al., 2001) showing the results of all systems (including those investigated in only one study).

The single studies, summary estimates, SROC curves, and confidence regions are depicted.

The relationship between sensitivity and viral load represented by Ct value is visualized in a scatterplot. The single study results for sensitivity below a certain Ct threshold are plotted against these Ct values and categorized by the different RATs. If in a single study sensitivity estimates for more than one Ct value were available, those results are connected by a line.

According to our search criteria, a total of 59 publications were originally selected, of which 19 studies (ten peer-reviewed and nine pre-prints) were included ( Figure 1 ).

Test from Roche Diagnostics, equivalent to the STANDARD Q COVID-19 Ag Test by SD Biosensor (henceforth called "Roche/SDB"); ii) the Panbio TM COVID-19 Ag Test by Abbott (henceforth called "Abbott"); iii) the COVID-19 Ag Respi-Strip® by Coris BioConcept (henceforth called "Coris"); iv) the COVID-Viro® by AAZ-LMB (henceforth called "AAZ-LMB"); v) the BIOCREDIT COVID-19 Ag by RapiGEN (henceforth called "RapiGEN").

The 19 clinical studies provided data on 11,109 samples, including 2,509 samples with confirmed SARS-CoV-2 by RT-PCR; see Tables 1-4. Sensitivity of the investigated RATs ranged between 28.9% (95% CI 16.4-44.3) and 98.3% (95% CI 91.1-99.7). Specificity ranged between 92.4% and 100% with one outlier (45%) (Khairat et al., 2020) . The two RATs with the most comprehensive available database of more than eight studies, the Roche/SDB and Abbott, reported a specificity of ≥97% in the majority of the trials.

The Coris RAT ranged between 95.8% and 100% for specificity but this was combined with unacceptable low sensitivity. The AAZ-LMB RAT showed very good results, with specificity of 100% and sensitivity of 84.1%, but was only evaluated in a single study (Schwob et al., 2020) .

RapiGEN showed unacceptable low specificity of 45% combined with low sensitivity in the only published study (Khairat et al., 2020) .

We undertook a statistical pooling of estimates across the 19 studies. Pooled sensitivities for each test with more than one study ranged between 37.7% (95% CI 27.8-48.7) and 82.4% (95% J o u r n a l P r e -p r o o f 8 CI 74.2-88.4) ( Figure 2 ). There was substantial heterogeneity across estimates of all the RATs, with I 2 values ranging from 59% to 88%.

SROC curves, summary estimates and confidence intervals were generated for RATs with multiple data points. Of these, the RATs by Abbott and Roche/SDB had overlapping confidence intervals, showing comparable performance; summary estimates were in the region of 80% ( Figure 3 ).

As expected, all RATs performed better in samples with high viral loads, but sensitivity dropped more rapidly at Ct >20 for the Coris test and less rapidly for the Roche/SDB and Abbott tests ( Figure 4) . All tests showed a lower sensitivity at Ct >30-32 and variable accuracy at Ct 25-30.

Summaries of the studies are provided in Tables 1-4. All studies provided descriptions of the study populations, regarding mean age and gender distribution (data not shown), as well as symptoms, prevalence rates, and Ct of the RT-PCR-positive samples.

Local definitions of "patients suspected of SARS-CoV-2 infection" either included i) only patients presenting clinical symptoms, or ii) asymptomatic persons with recent direct contact with suspected or confirmed cases. Some asymptomatic contact case groups were limited to persons with high-risk contact with the severely ill (Krüger et al., 2020) , contact with a case or high-risk exposure in a cluster (Berger et al., 2020) , but others included travelers returning from high risk areas (Cerutti et al., 2020; Krüger et al., 2020) , which meant that this population varied in terms of pretest probability. The percentage of symptomatic samples varied widely, ranging between J o u r n a l P r e -p r o o f 14.0% (Nalumansi et al., 2020) and 97.8% (Berger et al., 2020) . Not every report clearly stated the ratio between symptomatic RT-PCR-positive and symptomatic RT-PCR-negative samples.

This ratio seemed to differ considerably; some studies reported >90% symptomatic persons gaining approximately 15% RT-PCR-positive samples, and the study which investigated only 14% symptomatic persons tested 34.4% RT-PCR-positive samples (Nalumansi et al., 2020) . No study provided a direct comparison of RAT results between asymptomatic and symptomatic patients.

Prevalence rate -here meaning the number of RT-PCR-positive samples within the study population -varied between 1.9-100%. The prevalence of SARS-CoV-2 in some of these studies did not reflect the prevalence in the local populations, as additional pre-specified testing criteria qualified patients for study entry, thus creating a preselection bias.

The mean Ct value was reported in six studies ranging between 20.3 (Bulilete et al., 2020) and 31.4 (Scohy et al., 2020) . The median Ct value was reported by seven studies, and ranged between 18.57 (Khairat et al., 2020) and 33 (Scohy et al., 2020) . The definition of high viral load varied considerably within the trials, from Ct <18.57 in an Egyptian study (Khairat et al., 2020) to ≤37 in a Ugandan trial (Nalumansi et al., 2020) . One study did not report Ct values but reported RNA copies/mL (Schwob et al., 2020) . Notably the threshold for negativity (Ct >38 or >40) varied between the studies.

The Ct values, which were summarized and analyzed by group, also differed considerably. A majority, but not all, reported data at Ct thresholds close to Ct ≤20, ≤25, ≤30, and/or ≤35.

The cobas® SARS-CoV-2 Assay by Roche and the Allplex™ 2019-nCoV Assay by Seegene were the most frequently used RT-PCR assays (Supplemental Table 1 ), but even within a single study up to five different assays were used (Krüger et al., 2020) . The most commonly reported target was the envelope gene (pan-Sarbecovirus-and SARS-CoV-2-specific); other targets included the nucleocapsid, RNA-dependent RNA polymerase, spike, and ORF1a/b genes. Where dual target or multiplex assays were used, the target gene used to report the Ct value was frequently not stated (Supplemental Table 1 ; Tables 1-4).

This review presents an overview of manufacturer-independent commercial SARS-CoV-2 RATs not requiring a reading instrument. Altogether, 19 studies investigating five different RATs presented detailed population characteristics and Ct values. Only three commercial SARS-CoV-2

RATs have been assessed in multiple independent real-world studies, and of these, only the Roche/SDB and Abbott tests had adequate levels of performance; their summary estimates were in the region of 80%, exceeding or approaching health authorities' requirements for sensitivity ≥80% (European Centre for Disease Prevention and Control, 2020; World Health Organization, 2020a). The two RATs with the most comprehensive available database of more than eight studies, Roche/SDB and Abbott, reported a specificity of ≥97% in the majority of the trials, meeting specificity requirements (European Centre for Disease Prevention and Control, 2020; World Health Organization, 2020a).

There is one major concern to be highlighted when comparing performance data of RATs originating from different performance studies which is that data presented in different trials It must be noted that symptom classifications considerably differed between the studies. One study even investigated different populations for self-reported versus physician-defined symptoms (Lindner et al., 2020) . A uniform definition of "clinical symptoms suggestive for SARS-CoV-2 infection" would be desirable but is currently not available.

Aside from viral load, none of the 19 included studies reported sufficient detail to allow high confidence formal analysis of the effect of these variables on the performance of the RATs evaluated. Additionally, the lack of standardization (e.g. variable cut-offs and study designs), and the low number of positive samples in the individual studies, precluded such analysis.

Moreover, the included studies exhibited high heterogeneity. This heterogeneity was likely to be attributable to differences in the patient population between studies and other influencing factors, as mentioned above.

Due to methodological reasons, the detection limit for SARS-CoV-2 RNA from clinical samples tested by RT-PCR is always lower than the detection limit for SARS-CoV-2 antigen. The detectability of even the best performing RAT deteriorates with decreasing viral load; however, RATs still have utility in this context. Cell culture studies have shown that the probability for positive viral cell culture (a surrogate of viral transmission and infectivity) is lower at low viral load/high Ct (Bullard et al., 2020; van Beek et al., 2020) . This translates into very limited to no infectiousness of the infected patients, even if RT-PCR may still show positive signals for up to three more weeks after peak Ct value (Magleby et al., 2020; Wölfel et al., 2020) .

Additionally, a high positive predictive value requires testing at a high pre-test probability setting; a high negative predictive value in a low pre-test probability setting can help to safely rule out infectious or high-viral-load individuals (Peeling et al., 2020) . Use of RATs requires careful preselection and confirmation of recent contact to confirmed cases and/or knowledge about the underlying local population prevalence. If RATs are used for screening of asymptomatic cases in low-prevalence scenarios, a lower positive predictive value of the according result has to be considered.

Based on a systematic review and meta-analysis of RAT data published until November 2020, only three RATs had been assessed in multiple real-world manufacturer-independent studies. Of these, the Roche/SDB and Abbott RATs had adequate levels of performance and provide the strongest evidence base to recommend their use for the detection of current SARS-CoV-2 infection, particularly in high-viral-load patient populations.

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Editorial assistance was provided by Elements Communications Ltd, Westerham, UK and funded by Roche Diagnostics. COBAS is a trademark of Roche. All other product names and trademarks are the property of their respective owners.

This study did not require ethical approval because the meta-analysis was based on published research.

Johannes Hayer and Dusanka Kasapic are employees of Roche Diagnostics. Claudia Zemmrich works as a freelance contractor for Roche Diagnostics.

The data supporting this meta-analysis are from previously reported studies and datasets which have been cited. The processed data are available from the corresponding author upon request.

This work was supported by Roche Diagnostics. 

Field evaluation of a rapid antigen test (Panbio™ COVID-19 Ag Rapid Test Device) for COVID-19 diagnosis in primary healthcare centres

Diagnostic accuracy of two commercial SARS-CoV-2 antigen-detecting rapid tests at the point of care in community-based testing centers

Evaluation of the Panbio™ rapid antigen test for SARS-CoV-2 in primary health care centers and test sites. medRxiv

Predicting infectious severe acute respiratory syndrome coronavirus 2 from diagnostic samples

Centers for Disease Control and Prevention. Interim guidance for antigen testing for SARS-CoV

Urgent need of rapid tests for SARS CoV-2 antigen detection: evaluation of the SD-Biosensor antigen test for SARS-CoV-2

Rapid SARS-CoV-2 antigen detection assay in comparison with real-time RT-PCR assay for laboratory diagnosis of COVID-19 in Thailand

Diagnostics Global Health. Rapid antigen tests for the diagnosis of a SARS-CoV-2 infection

point-of-care antigen and molecular-based tests for diagnosis of SARS-CoV-2 infection

The sensitivity of SARS-CoV-2 antigen tests in the view of large-scale testing

European Centre for Disease Prevention and Control. Options for the use of rapid antigen tests for COVID-19 in the EU/EEA and the UK

European Commission. COVID-19 In Vitro Diagnostic Devices and Test Methods Database

Evaluation of the Panbio Covid-19 rapid antigen detection test device for the screening of patients with Covid-19

Real-life validation of the Panbio COVID-19 Antigen Rapid Test (Abbott) in community-dwelling subjects with symptoms of potential SARS-CoV-2 infection. medRxiv

Clinical evaluation of the

Biosensor rapid antigen test with symptomatic, non-hospitalized patients in a municipal health service drive-through testing site

Evaluation of two rapid antigen tests for detection of SARS-CoV-2 virus

Evaluation of the accuracy, ease of use and limit of detection of novel, rapid, antigen-detecting point-of-care diagnostics for SARS-CoV-2

Comparison of the SARS-CoV-2 rapid antigen test to the real star SARS-CoV-2 RT PCR kit

Evaluation of a rapid diagnostic assay for detection of SARS-CoV-2 antigen in nasopharyngeal swabs

Panbio antigen rapid test is reliable to diagnose SARS-CoV-2 infection in the first 7 days after the onset of symptoms

Head-to-head comparison of SARS-CoV-2 antigen-detecting rapid test with self-collected anterior nasal swab versus professional-collected nasopharyngeal swab

Impact of SARS-CoV-2 viral load on risk of intubation and mortality among hospitalized patients with coronavirus disease 2019

Field evaluation of the performance of a SARS-CoV-2 antigen rapid diagnostic test in Uganda using nasopharyngeal samples

Scaling up COVID-19 rapid antigen tests: promises and challenges

R Foundation for Statistical Computing. R: A language and environment for statistical computing

Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews

A hierarchical regression approach to meta-analysis of diagnostic test accuracy evaluations

Antigen rapid tests, nasopharyngeal PCR and saliva PCR to detect SARS-CoV-2: a prospective comparative clinical trial. medRxiv

Low performance of rapid antigen detection test as frontline testing for COVID-19 diagnosis

The Foundation for Innovative New Diagnostics. SARS-CoV-2 diagnostic pipeline

US Food and Drug Administration

From more testing to smart testing: data-guided SARS-CoV-2 testing choices

Diagnosis value of SARS-CoV-2 antigen/antibody combined testing using rapid diagnostic tests at hospital admission. medRxiv. 2020; 09.19

Virological assessment of hospitalized patients with COVID-2019

COVID-19 target product profiles for priority diagnostics to

Regulation and prequalification

J o u r n a l P r e -p r o o f