key: cord-0744978-n42uekc6
authors: Bruzzone, Bianca; De Pace, Vanessa; Caligiuri, Patrizia; Ricucci, Valentina; Guarona, Giulia; Pennati, Beatrice M.; Boccotti, Simona; Orsi, Andrea; Domnich, Alexander; Da Rin, Giorgio; Icardi, Giancarlo
title: Comparative diagnostic performance of different rapid antigen detection tests for COVID-19 in the real-world hospital setting
date: 2021-04-27
journal: Int J Infect Dis
DOI: 10.1016/j.ijid.2021.04.072
sha: f1116f2c9254f3b886b07a8d8de578e71b1e1728
doc_id: 744978
cord_uid: n42uekc6

Background The availability of accurate and rapid diagnostic tools for COVID-19 is essential for tackling the ongoing pandemic. The aim of this study was to quantify the performance of different available types of antigen-detecting rapid diagnostic tests (Ag-RDTs) in the real-world hospital setting. Methods In this retrospective analysis the diagnostic performance of seven different Ag-RDTs was compared with the real-time reverse transcription quantitative polymerase chain reaction (RT-qPCR) in terms of sensitivity, specificity and expected predictive values. Results A total of 321 matched Ag-RDT–RT-qPCR samples were analyzed retrospectively. The overall sensitivity and specificity of Ag-RDTs were 78.7% and 100%, respectively. However, a wide range of sensitivity estimates by brand (66.0–93.8%) and cycle threshold (Ct) cut-off values (Ct <25: 96.2%; Ct 30–35: 31.1%) was observed. The optimal Ct cut-off value that maximized sensitivity was 29. Conclusions The routine use of Ag-RDTs may be convenient in moderate-to-high intensity settings when high volumes of specimens are tested every day. However, diagnostic performance of the commercially available tests may differ substantially.

The ongoing pandemic spread of the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has been associated with a significant burden and unprecedented pressure on healthcare systems (McArthur et al., 2020; Greene et al., 2020; Rahimi et al., 2020) . The availability of accurate and rapid diagnostic tools for the coronavirus disease 2019 (COVID-19) is therefore essential for both active monitoring of cases and contact tracing strategies in order to reduce circulation of the COVID-19 causative agent (Greene et al., 2020; Rahimi et al., 2020; Venter et al., 2020; Hu et al., 2021) .

The etiological diagnosis of SARS-CoV-2 infection may be performed through both direct identification of the viral ribonucleic acid or antigens and indirect identification of specific antibodies. The direct methods include, for example, real-time reverse transcription quantitative J o u r n a l P r e -p r o o f polymerase chain reaction (RT-qPCR), transcription-loop-mediated isothermal amplification and different antigen-based tests. The indirect diagnostics, which is useful for establishing a previous exposure to the virus and/or vaccine, may be done through enzyme-linked immunoassay, chemiluminescence immunoassay and some other serological tests (Venter et al., 2020; Russo et al., 2020) .

Currently and owing to its high diagnostic performance, RT-qPCR may be dubbed as a "gold standard" assay for the laboratory diagnosis of both symptomatic and asymptomatic COVID-19 cases (Russo et al., 2020) . To the best of our knowledge, RT-qPCR technique is included in all principal international diagnostic protocols, including that issued by the World Health Organization (WHO) (2020a). However, from the point of view of public health, RT-qPCR has some intrinsic limits, including relatively high costs, availability of qualified personnel and sometimes suboptimal turnaround time (Russo et al., 2020) .

In order to address these shortcomings, antigen-detecting rapid diagnostic tests (Ag-RDTs) have been promptly developed and become increasingly common in the clinical setting with many brands available up to date (WHO, 2020a; ECDC, 2020) . If Ag-RDTs are highly accurate, they may exercise a greater public health impact than RT-qPCR for the following reasons: (i) no need for high-level technical expertise and laboratory capacity; (ii) may be performed locally in a decentralized modality with associated logistic advantages; (iii) may facilitate timely decisions regarding quarantine and/or treatment regimens and epidemiological investigations of novel clusters (Dinnes et al., 2021; ECDC, 2020 ).

On the other hand, Ag-RDTs are deemed to be less sensitive than RT-qPCR (ECDC, 2020) . A recent Cochrane review (Dinnes et al., 2021) has highlighted a considerable between-study variability in terms of sensitivity and specificity estimates; these also varied by Ag-RDT brand and viral load. For instance, a subgroup analysis by viral load defined by the cycle threshold (Ct) has quantified a 53.8% J o u r n a l P r e -p r o o f absolute difference in sensitivity between samples with Ct ≤25 and >25, respectively (Dinnes et al., 2021) . In this regard, the WHO (2020b) recommends that SARS-CoV-2 Ag-RDTs had at least 80% sensitivity and 97% specificity. The European Centre for Disease Control and Prevention (ECDC) (2020) has recently proposed a more conservative threshold of ≥90% for the sensitivity parameter, especially in low incidence settings.

The rationale behind this retrospective analysis is determined by the paucity of data on the comparative performance of Ag-RDTs in the hospital setting. Moreover, ECDC (2020) has also recommended that single European Member States perform independent and setting-specific evaluation of Ag-RDTs before their wide implementation. Thus, our primary goal was to assess the diagnostic accuracy of Ag-RDTs in the real-world setting as compared with RT-qPCR. Moreover, considering a significant evolution of the available Ag-RDTs, we also aimed to compare different available Ag-RDTs assays from the point of view of diagnostic accuracy.

This retrospective analysis was conducted at the regional reference laboratory for COVID-19 diagnostic located in San Martino Policlinico Hospital, Hygiene Unit (Genoa, Italy). To be included in the study, samples had to be tested by both an Ag-RDTs (in urgent routine modality) and RT-qPCR (following confirmation of the Ag-RDT output). All available matched samples collected between July and December 2020 were eligible. Swabs were performed by using a flocked probe and were eluted in the universal transport medium (UTM™, Copan Diagnostics Inc, US). All tests were performed within 8 hours from the arrival of samples at the laboratory.

No formal ethical approval for this retrospective study was needed since it was conducted as part of the routine SARS-CoV-2 testing.

Each sample underwent the extraction-free RT-qPCR on Nimbus IVD, (Seegene Inc., Republic of 

Ag-RDTs used were classified into two categories. The first was composed of lateral flow immunochromatographic tests (LFTs) and included the following kits: STANDARD™ Q COVID-19 Ag All the Ag-RDTs were performed according to the manufacturers' instructions.

RT-qPCR was considered as a reference standard against Ag-RDTs (FIND, 2020). On the basis of Ct values and therefore viral load, RT-qPCR positive samples were divided into three categories: <25 (high load), 25-29.9 (medium load) and 30-34.9 (low load). Indeed, the infectiousness is likely associated with high viral loads with Ct values <25/30 and Ag-RDTs are expected to perform better in these cases (WHO, 2020a; ECDC, 2020; Dinnes et al., 2021) .

Diagnostic performance of Ag-RDTs was assessed through calculation of sensitivity and specificity overall and stratified by assay type and Ct value categories. A receiving operation curve (ROC) was then constructed and the area under the curve (AUC) was quantified. An optimal cut-off of the Ct value was estimated using Youden's J statistic. The expected positive (PPV) and negative (NPV) predictive values were calculated from the overall sensitivity and specificity and hypothetical positivity prevalence of 0.5%, 1%, 5%, 10% and 20% (ECDC, 2020; Dinnes et al., 2021) .

All analyses were performed using R stats packages v. 4.0.3 (R Core Team, 2020). Of Ag-RDTs used 37.1% (n = 119) and 62.9% (n = 202) belonged to LFT and FIA types, respectively.

Overall diagnostic performance of Ag-RDTs used Table 1 reports raw data on the performance of any Ag-RDT analysed: the proportion of false negative results was 21.3% (n = 54), while no false positive specimes were revealed. The overall sensitivity and specificity were therefore 78.7% (95% CI: 73.2-83.3%) and 100% (95% CI: 94.7-100%), respectively. The expected NPVs at the positivity rate of 0.5%, 1%, 5%, 10% and 20% would be 99.9%, 99.8%, 98.9%, 97.7% and 94.9%, respectively, while the expected PPV would be constantly 100%. The ROC curve constructed using the average Ct value versus positive Ag-RDT test showed an AUC of 0.88 (95% CI: 0.82-0.93) with an optimal Ct cut-off value of 29.

There was a significant (as shown by non-overlapping 95% CIs) trend between sensitivity and Ct values: low Ct value specimens were associated with a sensitivity of 96.2%, while those with high Ct values had a sensitivity of only 31.1%. Use of FIA test types was associated with a lower number of false negatives, independently from Ct. However, LFTs performed reasonably well at low-tomedium Ct values (Table 2) .

On the other hand, there was a high between-brand heterogeneity ( Table 3 ). The best performing assays were STANDARD™ Q COVID-19 Ag and FREND™ COVID-19 Ag tests.

In the present study we have evaluated the diagnostic performance of different Ag-RDTs in the realworld hospital setting. The overall sensitivity of Ag-RDTs was about 79%, while the specificity was 100% with no false positive results. On the other hand, the sensitivity of Ag-RDTs increased up to 96.2% for high viral load (Ct <25) samples. This means that the overall sensitivity was driven by a relatively high frequency of false negative results among specimens with Ct values of 30-35. On average FIA tests performed better than LFTs, although a substantial between-brand heterogeneity was observed.

To the best of our knowledge, there is still an ongoing debate on which RT-qPCR cut-off value should be used to dub a specimen positive, weakly positive or negative. In this regard, a recent systematic review (Jefferson et al., 2020) has established that Ct values were significantly lower in samples producing live virus culture. Bullard et al. (2020) have estimated that the infectiousness (defined in that study by growth in the Vero cell culture) has been significantly reduced at Ct >24; this may mean that RT-qPCR positivity persists beyond infectiousness. In our study, however, in order to be in line with the previous body of primary research identified by the available systematic reviews (Van Walle et al., 2020; Dinnes et al., 2021) we used a more conservative Ct threshold of 35. Indeed, we found that the optimal Ct-value cut-off that maximized sensitivity was 29; the same Ct cut-off with a sensitivity of 92% has been recently documented by Nalumansi et al. (2020) .

A systematic review on the diagnostic accuracy of Ag-RDTs against RT-qPCR performed by ECDC researchers (Van Walle et al., 2020) has documented a high variability of sensitivity estimates (range J o u r n a l P r e -p r o o f of 29-93.9%), while the specificity was constantly high (range of 98.8-100%). The more recent Cochrane review (Dinnes et al., 2021) has reported a wider range for both sensitivity (0-94%) and specificity (90-100%) parameters. The pooled results obtained by Dinnes et al. (2021) are consistent with the sensitivity estimates obtained in our study. For instance, for samples with Ct values <25 we calculated the sensitivity of 96.2% (95% CI: 90.5-98.5%) that is in line with the Cochrane review estimate of 94.5% (95% CI: 91.0-96.7%). By contrast, we found a higher overall sensitivity for specimens with Ct values >25 [66.4% (95% CI: 58.5-73.5%) vs 40.7% (95% CI: 31.8-50.3%)]. This may be partially explained by the fact that in our study more than one thirds of Ag-RDTs belonged to highly performing FIA (STANDARD™ Q COVID-19 Ag and FREND™ COVID-19 Ag tests), while the above-mentioned review (Dinnes et al., 2021) has mainly dealt with LFTs. Apart from the higher sensitivity of some FIA tests, these latter have an advantage of a shorter readout time. On the other hand, these rapid tests may be associated with higher purchase costs; a critical assessment of the local epidemiological situation with a clear cost-benefit reasoning may aid the decision-making process.

More generally, our results are consistent with the recent suggestions proposed by the ECDC (2020).

Thus, in case of a low prevalence, Ag-RDTs would be associated with low PPVs and high NPVs. Ag-RDTs may be useful to screen positive patients with high viral loads; these, however, should be subsequently confirmed by RT-qPCR. Negative tests instead may not require a subsequent RT-qPCR.

By contrast, in settings with high incidence the situation changes: positive results will most probably identify true positives with no need for subsequent RT-qPCR, while for negative tests the ECDC suggests an immediate RT-qPCR (ECDC, 2020) . Again, a cost-benefit reasoning should be applied in this circumstance.

We have to acknowledge that our study may suffer from some limitations. First, although the overall sample size was comparatively [the medium sample size of 77 studies included in the Cochrane J o u r n a l P r e -p r o o f review by Dinnes et al. (2021) was 182] large, it was skewed to RT-qPCR positive samples. It has been suggested (FIND, 2020) that in the retrospective validation studies of SARS-CoV-2 Ag-RDTs a minimum number of both RT-qPCR positive and negative samples should be 100. In this study we had only 68 negative samples. On the other hand, shifting a positivity cut-off to Ct <30 (Dinnes et al., 2021; ECDC, 2020) would produce a total of 105 RT-qPCR negative samples. This is the reason why we believe our study results are sufficiently powered. Second, during the study period the availability of single Ag-RDTs differed substantially and therefore the usage of some tests was underrepresented. In particular, a post-hoc power analysis for the observed sensitivity revealed that the estimates reported for COVID-19 Antigen Rapid Test Prima Professional (Prima Lab, Switzerland), BIOCREDIT COVID-19 Ag (RapiGEN Inc., Republic of Korea) and LumiraDx SARS-CoV-2 Ag Test (LumiraDx UK ltd.) were likely underpowered (α<0.8). Another potential shortcoming is ascribable to the fact that the last specimen analyzed was collected in December 2020 and therefore the SARS-CoV-2 population at the time of study may be not representative of the current one. The rapid diffusion of novel SARS-CoV-2 variants (Kypferschmidt, 2021; Mahase, 2021) may interfere with the accuracy of Ag-RDTs available; therefore, a continuous monitoring of the performance of Ag-RDTs is warranted. Finally, for several privacy issues, we were not able to link laboratory test results to the sociodemographic and clinical features of patients.

To conclude, this study demonstrated that some Ag-RDTs used fulfilled the required diagnostic performance criteria proposed by the WHO (2020a; 2020b) and ECDC (2020), especially in patients with high viral loads. Rapid point-of-care antigen tests are therefore useful in the everyday clinical practice for their ease of use with a minimum training time, availability of results in few minutes and very high specificity. We believe that the use of Ag-RDTs may be particularly convenient in moderate-to-high intensity settings when high volumes of specimens must be tested every day. In fact, our results suggest that at the positivity rate of 10-20% the expected PPV and NPV are close J o u r n a l P r e -p r o o f J o u r n a l P r e -p r o o f

Predicting infectious severe acute respiratory syndrome coronavirus 2 from diagnostic samples

point-of-care antigen and molecular-based tests for diagnosis of SARS-CoV-2 infection

European Centre for Disease Prevention and Control. Options for the use of rapid antigen tests for COVID-19 in the EU/EEA and the UK

Comparative evaluation of lateral flow assay tests that directly detect antigens of SARS-CoV-2

Coronavirus disease 2019: International public health considerations

Characteristics of SARS-CoV-2 and COVID-19

Viral cultures for COVID-19 infectious potential assessment -a systematic review

Review of Burden, Clinical Definitions, and Management of COVID-19 Cases

Field evaluation of the performance of a SARS-CoV-2 antigen rapid diagnostic test in Uganda using nasopharyngeal samples

R: A language and environment for statistical computing

Tackling the COVID-19 Pandemic

Current status of laboratory diagnosis for COVID-19: A narrative review

Meta-analysis of the clinical performance of commercial SARS-CoV-2 nucleic acid, antigen and antibody tests up to 22

Towards effective diagnostic assays for COVID-19: a review

The authors would like to thank De Rosa Gaetano, Falletta Alessandro, Ferraris Monica, Galano Barbara, Lioce Michela and Nigro Nicola for performing laboratory tests.J o u r n a l P r e -p r o o f