key: cord-0325145-zwygvmwv
authors: Baro, B.; Rodo, P.; Ouchi, D.; Bordoy, A. E.; Saya Amaro, E. N.; Salsench, S. V.; Molinos, S.; Alemany, A.; Ubals, M.; Corbacho-Monné, M.; Millat-Martinez, P.; Marks, M.; Clotet, B.; Prat, N.; Ara, J.; Vall-Mayans, M.; G-Beiras, C.; Bassat, Q.; Blanco, I.; Mitjà , O.
title: Performance characteristics of five antigen-detecting rapid diagnostic test (Ag-RDT) for SARS-CoV-2 asymptomatic infection: a head-to-head benchmark comparison
date: 2021-02-12
journal: nan
DOI: 10.1101/2021.02.11.21251553
sha: 4d2b886e427d791db725c501743cf1e42212621e
doc_id: 325145
cord_uid: zwygvmwv

Background Mass testing for early identification and isolation of infectious COVID-19 individuals, irrespective of concurrent symptoms, is an efficacious strategy to reduce disease transmission. Antigen-detecting rapid diagnostic tests (Ag-RDT) appear as a potentially suitable tool for mass testing on account of their ease-of-use, fast turnaround time, and low cost. However, benchmark comparisons are scarce, particularly in the context of unexposed asymptomatic individuals. Methods We used nasopharyngeal specimens from unexposed asymptomatic individuals to assess five Ag-RDTs: PanBio™ COVID-19 Ag Rapid test (Abbott), CLINITEST® Rapid COVID-19 Antigen Test (Siemens), SARS-CoV-2 Rapid Antigen Test (Roche Diagnostics), SARS-CoV-2 Antigen Rapid Test Kit (Lepu Medical), and COVID-19 Coronavirus Rapid Antigen Test Cassette (Surescreen). Samples were collected between December 2020-January 2021 during the third wave of the epidemic in Spain. Findings The analysis included 101 specimens with confirmed positive PCR results and 185 with negative PCR. For the overall sample, the performance parameters of Ag-RDTs were as follows: Abbott assay, sensitivity 38·6% (95% CI 29·1–48·8) and specificity 99·5% (97–100%); Siemens, sensitivity 51·5% (41·3–61·6) and specificity 98·4% (95·3–99·6); Roche, sensitivity 43·6% (33·7–53·8) and specificity 96·2% (92·4–98·5); Lepu, sensitivity 45·5% (35·6–55·8) and specificity 89·2% (83·8–93·3%); Surescreen, sensitivity 28·8% (20·2–38·6) and specificity 97·8% (94·5–99·4%). For specimens with cycle threshold (Ct) <30 in RT-qPCR, all Ag-RDT achieved a sensitivity of at least 70%, with Siemens, Roche, and Lepu assays showing sensitivities higher than 80%. In models according to population prevalence, all Ag-RDTs will have a NPV >99% and a PPV<50% at 1% prevalence. Interpretation Two commercial, widely available assays can be used for SARS-CoV-2 antigen testing to achieve sensitivity in specimens with a Ct<30 and specificity of at least 80% and 96%, respectively. Estimated negative and positive predictive values suggests the suitability of Ag-RDTs for mass screenings of SARS-CoV-2 infection in the general population. Funding Blueberry diagnostics, Fundació Institut d’Investigació en Ciències de la Salut Germans Trias i Pujol, and #YoMeCorono.org crowdfunding campaign. Evidence before this study In December 2020, we searched on PubMed for articles containing the terms “antigen”, “test” (or Ag-RDT), and “SARS-CoV-2” or “COVID-19” either in the title or the abstract. Our search yielded 79 entries corresponding to articles written in English. Of them, 33 were articles presenting the diagnostic performance of qualitative lateral-flow antigen-detecting rapid diagnostic tests (Ag-RDT). Four of these articles reported the results of head-to-head comparisons of various Ag-RDTs; in all cases, the number of tests was lower than the recommended for retrospective assessments of diagnostic performance (i.e., minimum of 100 PCR positive and 100 PCR negative). Furthermore, all head-to-head comparisons found in the literature included specimens obtained among individuals with varying disease status (none of which asymptomatic), thus limiting the adequacy of the estimates for an asymptomatic screening strategy. Added value of this study We compared for the first time head-to-head five Ag-RDT using a powered set of fresh respiratory specimens PCR-confirmed positive or negative, collected from unexposed asymptomatic individuals during screening campaigns for early detection of SARS-CoV-2 infection. The sample size was large enough to draw robust conclusions. Our analysis identified four Ag-RDTs (i.e., assays marketed by Abbott, Siemens, Roche, and Surescreen) with specificity higher than 96%. Despite the low sensitivity for the overall sample (range 29% to 51%), the corresponding values for the subset of samples with Ct <30 were higher than 80% for Siemens, Roche, and Lepu assays. The estimated NPV for a screening performed in an area with 1% prevalence would be >99% for all tests, while the PPV would be <50%. Implications of all the available evidence Current data on the diagnostic performance of Ag-RDTs is heterogeneous and precludes benchmark assessments. Furthermore, the screening of asymptomatic populations is currently not considered among the intended uses of Ag-RDT, mostly because of lack of evidence on test performance in samples from unexposed asymptomatic individuals. Our findings add to the current evidence in two ways: first, we provide benchmarking data on Ag-RDTs, assessed head-to-head in a single set of respiratory specimens; second, we provide data on the diagnostic performance of Ag-RDTs in unexposed asymptomatic individuals. Our findings support the idea that Ag-RDTs can be used for mass screening in low prevalence settings and accurately rule out a highly infectious case in such setting.

Antigen Test Cassette (Surescreen). Samples were collected between December 2020-January 2021 during the third wave of the epidemic in Spain.

The analysis included 101 specimens with confirmed positive PCR results and 185 with negative PCR. For the overall sample, the performance parameters of Ag-RDTs were as follows: Abbott assay, sensitivity 38·6% (95% CI 29·1-48·8) and specificity 99·5% (97-100%);

Siemens, sensitivity 51·5% (41·3-61·6) and specificity 98·4% (95·3-99·6); Roche, sensitivity 43·6% (33·7-53·8) and specificity 96·2% (92·4-98·5); Lepu, sensitivity 45·5% (35·6-55·8) and specificity 89·2% (83·8-93·3%); Surescreen, sensitivity 28·8% (20·2-38·6) and specificity 97·8% (94·5-99·4%). For specimens with cycle threshold (Ct) <30 in RT-qPCR, all Ag-RDT achieved a sensitivity of at least 70%, with Siemens, Roche, and Lepu assays showing sensitivities higher than 80%. In models according to population prevalence, all Ag-RDTs will have a NPV >99% and a PPV<50% at 1% prevalence.

Interpretation: Two commercial, widely available assays can be used for SARS-CoV-2 antigen testing to achieve sensitivity in specimens with a Ct<30 and specificity of at least 80% and 96%, respectively. Estimated negative and positive predictive values suggests the suitability of Ag-RDTs for mass screenings of SARS-CoV-2 infection in the general population.

Germans Trias i Pujol, and #YoMeCorono.org crowdfunding campaign.

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted February 12, 2021. ; https://doi.org/10.1101/2021.02.11.21251553 doi: medRxiv preprint

Evidence before this study

In December 2020, we searched on PubMed for articles containing the terms "antigen", "test" (or Ag-RDT), and "SARS-CoV-2" or "COVID-19" either in the title or the abstract. Our search yielded 79 entries corresponding to articles written in English. Of them, 33 were articles presenting the diagnostic performance of qualitative lateral-flow antigen-detecting rapid diagnostic tests (Ag-RDT). Four of these articles reported the results of head-to-head comparisons of various Ag-RDTs; in all cases, the number of tests was lower than the recommended for retrospective assessments of diagnostic performance (i.e., minimum of 100 PCR positive and 100 PCR negative). Furthermore, all head-to-head comparisons found in the literature included specimens obtained among individuals with varying disease status (none of which asymptomatic), thus limiting the adequacy of the estimates for an asymptomatic screening strategy.

We compared for the first time head-to-head five Ag-RDT using a powered set of fresh respiratory specimens PCR-confirmed positive or negative, collected from unexposed asymptomatic individuals during screening campaigns for early detection of SARS-CoV-2 infection. The sample size was large enough to draw robust conclusions. Our analysis identified four Ag-RDTs (i.e., assays marketed by Abbott, Siemens, Roche, and Surescreen) with specificity higher than 96%. Despite the low sensitivity for the overall sample (range 29% to 51%), the corresponding values for the subset of samples with Ct <30 were higher than 80% for Siemens, Roche, and Lepu assays. The estimated NPV for a screening performed in an area with 1% prevalence would be >99% for all tests, while the PPV would be <50%.

Current data on the diagnostic performance of Ag-RDTs is heterogeneous and precludes benchmark assessments. Furthermore, the screening of asymptomatic populations is currently not considered among the intended uses of Ag-RDT, mostly because of lack of evidence on test performance in samples from unexposed asymptomatic individuals. Our findings add to the current evidence in two ways: first, we provide benchmarking data on Ag-RDTs, assessed headto-head in a single set of respiratory specimens; second, we provide data on the diagnostic performance of Ag-RDTs in unexposed asymptomatic individuals. Our findings support the idea that Ag-RDTs can be used for mass screening in low prevalence settings and accurately rule out a highly infectious case in such setting.

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted February 12, 2021.

Mass testing for early identification and isolation of individuals infected with the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), irrespective of symptoms, is potentially an efficacious strategy to reduce disease transmission. 1 Recent advances on the validation of Antigen-detecting Rapid Diagnostic Tests (Ag-RDTs) show promise to replace central laboratory techniques for epidemiological control of the SARS-CoV-2 through mass testing.

Reverse transcription-polymerase chain reaction (RT-qPCR) is the current gold standard for identifying the presence of the SARS-CoV-2 in respiratory specimens. 2 More recently, transcription-mediated amplification (TMA) of the SARS-CoV-2 genome has been added to the repertoire of nucleic acid amplification tests (NAAT) for SARS-CoV-2 detection. 3 Despite their high sensitivity, NAATs are associated with drawbacks that limit their use for community-based testing strategies, including the need for laboratory-processing, high cost, and long turnaround from sampling to results release.

Ag-RDTs, commonly used in diagnosing other infectious diseases, have emerged as an alternative tool that meets the requirements for frequent testing at the point-of-care: rapid turnaround time, low cost, and ease-of-use. 4 Overall, Ag-RDTs have lower sensitivity than NAATs; however, clinical validation studies have consistently reported increasing sensitivities in specimens with higher viral loads. These findings, along with the growing body of evidence on the lack of infectivity of cases with low viral load, [5] [6] [7] [8] and the potential long tail of positivity when using highly sensitive methods such as PCR, suggest that frequent testing with Ag-RDTs-even those with low sensitivity-may be more effective than less frequent testing with RT-qPCR or TMA for mass screening campaigns to improve SARS-CoV-2 control. 8, 9 The performance parameters of Ag-RDTs are mostly based on testing respiratory specimens from clinically suspected cases [10] [11] [12] [13] and contacts after exposure to a positive case. 14-17 However, the sensitivity bias associated with the viral load leads to high heterogeneity in the reported performance parameters, which strongly depend on the disease status and potential exposure (e.g., symptomatic vs. asymptomatic, contact vs. unexposed) of tested individuals. This heterogeneity precludes comparative analyses between tests assessed in different studies and challenges benchmarking of Ag-RDTs. Furthermore, head-to-head comparisons are scarce, particularly in samples from asymptomatic individuals, the target population of communitybased screening strategies. 18, 19 In this study, we used fresh nasopharyngeal samples collected in routine mass screening campaigns of unexposed asymptomatic individuals to perform a head-tohead comparison of five Ag-RDTs.

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted February 12, 2021. ; https://doi.org/10.1101/2021.02.11.21251553 doi: medRxiv preprint 6

As part of the surveillance program for pandemic control in Catalonia ( 

We calculated that a sample size of at least 73 positive specimens and 165 negative specimens would give 80% power to estimate overall sensitivity and specificity of Ag-RDT assays in our study. We based our calculation on the expected sensitivity and specificity in asymptomatic population of 65% and 96%, 16 The primary analysis of the head-to-head comparison was the sensitivity and specificity of each Ag-RDT. Sensitivity and specificity were calculated as defined by Altman et al., 23 and reported as a percentage and the exact binomial 95% confidence interval (CI). Sensitivity was also analysed in a subset of samples with Ct<30, considered at high risk of transmission.

Secondary analyses were done assessing discordance between results obtained in each Ag-RDTs. Positive and negative-predictive values for each Ag-RDT at population prevalence between 1% and 15% for SARS-CoV-2 infection were modelled 24 and plotted with the exact binomial 95% CI. 25 All analyses and plots were performed using R version 3·6. 26 

The funders of the study had no role in the study conception, design, conduct, data analysis, or writing of the report. All authors had full access to all the data in the study and had final responsibility for the decision to submit for publication.

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted February 12, 2021. ; https://doi.org/10.1101/2021.02.11.21251553 doi: medRxiv preprint

Our sample collection included 316 fresh nasopharyngeal swabs from unexposed asymptomatic individuals who had a RT-qPCR result available. Of these, 30 Ag-RDTs ranged from 28·7% to 51·5% and 89·2% to 99·5%, respectively (Table 1) Table S2 summarizes the cycle threshold distribution across discordances.

To provide an estimate of misidentified cases-either false-positive or false-negative cases that can be used for making decisions in the public health setting, we modelled the positive and negative predictive value for a prevalence range consistent with a mass screening of unexposed asymptomatic individuals ( Figure 4A ). For the overall study sample, the estimated positive predictive value (PPV) at a 1% prevalence ranged from 4·1% to 41·9%, with the Lepu assay and the Abbott assay, respectively (Table S3 ). The estimated PPVs notably increased for the <30 Ct subgroup of samples ( Figure 4B ), and when prevalence in the population was higher.

The estimated negative predictive value (NPV) at 1% prevalence ranged from 99·3% to 99·5%, with the Surescreen assay and the Siemens assay, respectively.

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted February 12, 2021. ; https://doi.org/10.1101/2021.02.11.21251553 doi: medRxiv preprint

In this study, we compared head-to-head the sensitivity and specificity of five Ag-RDTs to screen SARS-CoV-2 infected individuals with unknown exposure and no clinical suspicion of COVID-19. Four of the tested Ag-RDTs (i.e., Abbott, Siemens, Roche, and Surescreen assays) showed a specificity higher than 96%. Regarding sensitivity, despite it was low for the overall sample (range 29% to 51%), the corresponding values for the subset of samples with a RT-qPCR value Ct <30 were higher than 80% for the Siemens, Roche, and Lepu assays. This finding is of particular interest for the proposed use of Ag-RDT as a reliable alternative to RT-qPCR for the rapid detection of individuals with higher risk of infectivity in mass screening of asymptomatic individuals. Pre-clinical studies have persistently reported a very low infectious capacity of respiratory specimens with viral loads below 10 6 genome copies/mL, which usually correspond to a Ct of approximately 29 -31. 4, 7, 28 These findings align with the significant increase of the secondary attack rate for values of Ct <30, 29 indicating higher infectiousness among individuals with viral loads below this Ct threshold.

Although sensitivity and specificity are important intrinsic characteristics of a test, the number of expected errors when using the test for screening purposes strongly depends on the prevalence of the infection in the screened sample. Hence, positive and negative predictive values are a mainstay for making public health decisions regarding the use of a test. The reported prevalence of SARS-CoV-2 infection in PCR-based untargeted screenings of the general population typically ranges between 1% and 3%, depending on the virus transmission context. 22, 30 In low prevalence settings, Ag-RDTs will have a high NPV but a low PPV.

According to our estimate, the NPV for SARS-CoV-2 infections at 1% prevalence was higher than 99% for all test, suggesting that a negative test may not require confirmation. In contrast, the PPV at 1% prevalence was lower than 50% in all tests, suggesting that a positive result will need immediate confirmation by RT-qPCR, even for highly specific assays.

Our study has several strengths and limitations. We used the same fresh set of samples for assessing five different Ag-RDTs and the sample size met the FIND recommendation for retrospective assessments of the clinical performance of these tests. Furthermore, to our knowledge, this is the first head-to-head comparison of Ag-RDT in asymptomatic screenings, an intended use proposed by various authors. 4, 9, 16, 22 On the other hand, our study was limited by the small number of specimens with Ct <30, a threshold deemed of interest for the use of Ag-RDT in screenings of the general population. In our sample, specimens below this threshold accounted for 30%; however, other authors have reported proportions of nearly 60% in random screenings of the general population. 22 Of note, we used specimens in transport medium. This approach is convenient for mass screening strategies in which individuals with positive Ag-. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted February 12, 2021. ; https://doi.org/10.1101/2021.02.11.21251553 doi: medRxiv preprint RDT results may need further diagnostic confirmation by PCR. However, only one manufacturer (i.e., the Roche assay) provided instructions on how to process samples collected in virus transport medium. The consistency of our results across assays, particularly regarding negative results, suggests that the use of this media had a little or negligible impact on test performance. Finally, it is worth mentioning that all nasopharyngeal swabs in our analysis were collected by trained healthcare professionals. According to a recent report of lateral flow viral antigen detection devices, the positivity rate might be lower in screenings performed by nontrained people. 8 Our results provide policymakers with evidence on the use of Ag-RDT for mass screening of unexposed, asymptomatic individuals. Two commercial, widely available assays can be used for SARS-CoV-2 antigen testing to achieve sensitivity in specimens with a Ct<30 and specificity of at least 80% and 96%, respectively. While these tests may overlook SARS-CoV-2 infection with low viral loads, they accurately detect individuals with high viral loads and, therefore, at higher risk of transmission. Our findings also support the idea that Ag-RDTs can be used for mass screening in low prevalence settings and accurately rule out a highly infectious case in such setting. In models according to population prevalence, all Ag-RDTs will have a NPV >99% and a PPV<50% at 1% prevalence. Together with the ease of use, low cost, and short turnaround time, this feature makes them an excellent tool for frequent mass screenings of asymptomatic people. In low-income countries with limited laboratory resources, the trade-off between targeted PCR analyses and massive screenings with Ag-RDTs should be carefully considered. 

We declare no conflicts of interest.

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) All samples were nasopharyngeal swabs collected from unexposed asymptomatic individuals during mass screening campaigns. Sensitivity and specificity results are presented with the 95% confidence interval.

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

The copyright holder for this preprint this version posted February 12, 2021. ; https://doi.org/10.1101/2021.02.11.21251553 doi: medRxiv preprint Bars show the number of samples for each discordance pattern. Black dots and grey dots indicate the assays showing positive and negative results in each discordance pattern. Table S2 summarizes the cycle threshold distribution across discordances. Table   S3 provides detailed values and confidence intervals for predicted false negative and false positives in the investigated prevalence.

. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. 

The effectiveness of population-wide, rapid antigen test based screening in reducing SARS-CoV-2 infection prevalence in Slovakia

Diagnostic testing for SARS-CoV-2

High-Throughput Transcription-mediated amplification on the Hologic Panther is a highly sensitive method of detection for SARS-CoV-2

Rethinking COVID-19 test sensitivity-A strategy for containment

Virological assessment of hospitalized patients with COVID-2019

Viral RNA load as determined by cell culture as a management tool for discharge of SARS-CoV-2 patients from infectious disease wards

Longitudinal Surveillance for SARS-CoV-2 RNA Among Asymptomatic Staff in Five Colorado Skilled Nursing Facilities

Virologic and Sequence Analysis

Preliminary report from the Joint PHE Porton Down & University of Oxford SARS-CoV-2 test development and validation cell: Rapid evaluation of Lateral Flow Viral Antigen detection devices (LFDs) for mass community testing

Test sensitivity is secondary to frequency and turnaround time for COVID-19 screening

Evaluation of novel antigen-based rapid detection test for the diagnosis of SARS-CoV-2 in respiratory samples

Evaluation of a rapid diagnostic assay for . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) detection of sars-cov-2 antigen in nasopharyngeal swabs

Head-to-head comparison of SARS-CoV-2 antigen-detecting rapid test with self-collected anterior nasal swab versus professionalcollected nasopharyngeal swab

Field evaluation of a rapid antigen test (Panbio TM COVID-19 Ag Rapid Test Device) for COVID-19 diagnosis in primary healthcare centres

Panbio antigen rapid test is reliable to diagnose SARS-CoV-2 infection in the first 7 days after the onset of symptoms

Urgent need of rapid tests for SARS CoV-2 antigen detection: Evaluation of the SD-Biosensor antigen test for SARS-CoV-2

Analytical and clinical performance of the panbio COVID-19 antigen-detecting rapid diagnostic test

Evaluation of a rapid antigen test (Panbio TM COVID-19 Ag rapid test device) for SARS-CoV-2 detection in asymptomatic close contacts of COVID-19 patients

Comparative evaluation of four rapid SARS-CoV-2 antigen detection tests using universal transport medium

Foundation for Innovative New Diagnostics. Comparative evaluation of lateral flow assay tests that directly detect antigens of SARS-CoV-2

Seegene Inc. Allplex TM 2019-nCoV Assay

Performance characteristics of a rapid SARS-CoV-2 antigen detection assay at a public plaza testing site in San Francisco. medRxiv . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity

Statistics Notes: Diagnostic tests 1: Sensitivity and specificity

Statistics Notes: Diagnostic tests 2: Predictive values

Modelling Binary Data

R: A language and environment for statistical com puting

Antigen-detection in the diagnosis of SARS-CoV-2 infection using rapid immunoassays

Duration of infectiousness and correlation with RT-PCR cycle threshold values in cases of COVID-19

Transmission of Covid-19 in 282 clusters in Catalonia, Spain: a cohort study

Prevalence of Asymptomatic SARS-CoV-2 Infection : A Narrative

The authors would like to thank Gerard Carot-Sans (PhD) for providing professional medical writing support during the preparation of the manuscript. We thank Laia Comí, Josefa Gómez, 

Review. Ann Intern Med 2020; 173: 362-7.. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

The copyright holder for this preprint this version posted February 12, 2021. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity (which was not certified by peer review)