key: cord-0909347-cufmlmc4
authors: Chan, Hiu Tat; Keung, Marco H.T.; Nguyen, Ivy; Ip, Ellen L.O.; Chew, Su M; Siler, Danielle; Saville, Marion; Hawkes, David
title: Exploring beyond the limit: how comparative stochastic performance affects retesting outcomes in six commercial SARS CoV-2 nucleic acid amplification tests
date: 2022-04-30
journal: J Clin Virol Plus
DOI: 10.1016/j.jcvp.2022.100079
sha: 36060e1d531ed6fa6cee717d7ccb8a2f8d4f0463
doc_id: 909347
cord_uid: cufmlmc4

OBJECTIVES: : To examine the comparative stochasticity profile of six commercial SARS-CoV-2 nucleic acid amplification tests (NAATs) and how this may affect retesting paradigms. METHODS: : Commercial quality control (QC) material was serially diluted in viral transport media to create a panel covering 10 – 10,000 copies/ml. The panel was tested across six commercial NAATs. A subset of high cycle threshold results was retested on a rapid PCR assay to simulate retesting protocols commonly used to discriminate false positives. RESULTS: : Performance beyond the LOD differed among assays, with three types of stochasticity profiles observed. The ability of the rapid PCR assay to reproduce a true weak positive specimen was restricted to its own stochastic performance at the corresponding viral concentration. CONCLUSION: : Stochastic performance of various NAATs overlap across low viral concentrations and affect retesting outcomes. Relying on retesting alone to discriminate false positives risk missing true positives even when a more sensitive assay is deployed for confirmatory testing.

Objectives: To examine the comparative stochasticity profile of six commercial SARS-CoV-2 nucleic acid amplification tests (NAATs) and how this may affect retesting paradigms.

Methods: Commercial quality control (QC) material was serially diluted in viral transport media to create a panel covering 10 -10,000 copies/ml. The panel was tested across six commercial NAATs. A subset of high cycle threshold results was retested on a rapid PCR assay to simulate retesting protocols commonly used to discriminate false positives. Results:

Performance beyond the LOD differed among assays, with three types of stochasticity profiles observed. The ability of the rapid PCR assay to reproduce a true weak positive specimen was restricted to its own stochastic performance at the corresponding viral concentration. Conclusion: Stochastic performance of various NAATs overlap across low viral concentrations and affect retesting outcomes. Relying on retesting alone to discriminate false positives risk missing true positives even when a more sensitive assay is deployed for confirmatory testing.

Keywords: SARS-CoV-2, nucleic acid amplification tests, comparative sensitivity, stochastic performance, limit of detection, false positive, false negative

Interpreting weak positive results from SARS-CoV-2 nucleic acid amplification tests (NAATs) can be challenging as there are concerns that high cycle threshold (ct) value ('weak') positives could be potentially false positives (e.g. due to non-specific amplification) [1] . At different stages of the pandemic, some jurisdictions retest to "confirm" such results, with initial positives being over-ridden by subsequent negatives from retesting [1, 2] . However, weak positive results may mean the samples contain small quantities of nucleic acids, beyond the limit of detection (LOD) of the NAATs. At these low levels, NAATs' performance is probabilistic, depending on capture of viral material through sampling and the assay chemistry. An appreciation of the comparative stochastic performance of various NAATs is therefore important to understand the benefits and pitfalls of confirmatory retesting.

Research comparing the stochasticity of different NAATs and its implication in confirmatory retesting is lacking [3] [4] [5] . We therefore examined the stochastic performance of six commercial SARS-CoV-2 NAATs using serial dilutions of a commercial QC material and examined its relationship to retesting outcomes.

Abbott Alinity m SARS-CoV-2 (Alinity m system), Abbott RealTime SARS-CoV-2 (m2000 system), Cepheid Xpert Xpress SARS-CoV-2 (GeneXpert IV system), Hologic Aptima SARS-CoV-2 (Panther instrument), Roche cobas SARS-CoV-2 (cobas 6800 system), and Seegene Allplex SARS-CoV-2 (Seegene STARlet/BioRad CFX instruments) assays were evaluated in this study. All assays and materials were used as per the manufacturer's instructions for use.

Briefly, commercial lyophilised QC material registered with Therapeutic Goods Administration (Australia) for use as an in vitro diagnostic device (Microbiologics, Minnesota, USA, catalogue number: HE0065N) was rehydrated and serial-diluted with viral transport media (KangJian, Jiangsu, China, catalogue number: KJ502-19) to the described concentrations (table 1) for testing across six NAATs between January and May 2021. The six NAATs were run in parallel six times, each time with freshly prepared serially-diluted QC material, (except the cobas assay which was run twice due to reagent availability issue). To better demonstrate probabilistic detection beyond LOD with limited assay availability during pandemic times, more replicates were ran at close intervals of lower viral copies concentration (rather than higher concentrations). Fewer samples were also run on the Xpert due to test cartridge scarcity.

Some laboratory services utilised the Xpert for a rapid retest of specimens with high ct values to "confirm" initial positive results. To examine such a retesting protocol, a subset of the low concentration samples, tested positive with high ct on PCR-based NAATs, were retested on the Xpert assay. The TMA-based Hologic Aptima assay produces RLU values that are not linear against viral concentrations so there is no equivalent of a high ct value and was therefore excluded from the retesting component of the study [6] .

We observed three different stochasticity profiles in NAATs beyond the concentrations where they detect 100%: gradual decline in detection probability (Seegene, Aptima, Realtime), quick drop off (Xpert), or maintenance of high detection probability until very low viral concentrations (cobas, Alinity m). For example, the Seegene assay detected 100% at 500 copies/mL and took a five-fold reduction in specimen analyte concentration (100 copies/mL) until detection probability halves. In contrast, the Xpert detected 100% at 100 copies/mL but detection probability halves as soon as analyte concentration halves (50 copies/mL), while the Alinity m detected with high probabilities until 10 copies/mL.

In the retesting experiments, a panel of 35 specimens at 10 to 100 copies/mL, initially tested positive with high ct values, were retested. The retesting assay, Xpert, gave a positive result at a probability similar to that at the corresponding concentrations as in its initial assessment ( Table 1 & 2) .

The current study provides the first analysis of the comparative stochastic performance across six commercial SARS-CoV-2 NAATs. Retesting of positive specimens was performed in a similar manner to previously described testing protocols [2, 7, 8] . The focus of this study was not to verify the LODs, but to examine a range of commonly utilised commercial assays' differing stochastic characteristics beyond the LODs. The study also demonstrated that the assays' probabilistic detection overlaps significantly across low analyte levels.

There have been concerns regarding results with high ct values to be false positives [1, 2] .

Many services therefore retest, using either the same assay and/or another assay with better sensitivity, to discriminate these false positives [2, 7] . Retesting protocols assume positive results that cannot be reproduced ("confirmed") to be false [2, 7] . Thus, some NAATs are essentially deployed as "confirmatory". However, none of the NAATs evaluated in this study were validated by the manufacturers as confirmatory assays, and there is a lack of guidance on the process of confirming SARS-CoV-2 molecular diagnosis. Further validation of the NAATs as laboratory developed confirmatory tests will be required.

The current study demonstrates that even with a more sensitive retesting assay, a true positive from a less sensitive initial NAAT need not be "confirmed" ( Table 2 ). The retesting assay can only "confirm" positive results at a probability consistent with its intrinsic sensitivity at the corresponding concentrations. Moreover, the stochastic performances of many assays overlap over lower concentrations (Table 1 ). In our case, while Xpert has a 5fold lower LOD (100 copies/mL vs 500 copies/mL) than Seegene, Seegene still detect samples with viral concentrations beyond the LOD of Xpert at high enough frequencies. At 50 copies/mL, Xpert only detects about twice as much as Seegene, and may miss a true positive detected by the initial Seegene assay at high enough frequency.

It is also worth considering how the Xpert assay is applied for retesting when the initial weak positives came from an assay having a higher sensitivity/better stochastic performance, e.g., at 50 copies/mL both the Alinity m and the cobas were more likely to produce a positive result than the Xpert (Table 1) .

High ct values results may represent, in addition to false positives, various clinically significant scenarios, including early infection or ongoing viral shedding. Using retesting results to over-ride the initial positive results on the assumption that discrepant results are due to non-specific NAAT reactions, may erroneously call some true positives as false positives. Fundamentally, the relative stochastic performances of the two assays will determine the probability of reproducing a positive result on any assay combinations. Given vaccinated individuals can have lower viral loads when infected [10] , high ct value results may be increasingly encountered with increasing vaccination coverage. Laboratories should be aware of how the stochastic performance of their assays affect their retesting protocol s to avoid erroneously over-riding an otherwise valid positive result.

Our study has several strengths. Using commercially available control material, we addressed reproducibility issues raised by some researchers [3, 11] . Our study design also allowed easy comparison of the evaluated NAATs, and resolved conflicting findings regarding relative sensitivity in previous studies [4] [5] [6] [12] [13] [14] . Serial dilutions at close intervals of low viral concentration levels were tested, generating consistent data to demonstrate stochasticity. There are several limitations, however. Only control materials utilizing inactivated virus were tested. Therefore, the matrix effect was not investigated, and the relative sensitivity of the various NAATs may not reflect real life conditions. However, the matrix effect will not invalidate the concept of stochasticity as demonstrated in this study. Another limitation of this study is that only a relatively small number of replicates were tested, and retesting was limited due to availability of assay kits.

In summary, our study illustrated that, even with a more sensitive "confirmatory" assay, true positives need not be reproducible on retesting. The more sensitive "confirmatory" assay only reproduced a 'true' result based on its intrinsic stochastic performance. Caution must be exercised if relying solely on retesting to identify false positives, and additional means e.g., testing additional patient samples should be considered. Percentage positive (number positive/number tested). Colour intensity represents detection probability (0-19%, 20-39%, 40-59%, 60-79%, 80-95%, 96-100%). Grey/NA -Not Assessed.

Diagnostic testing for SARS-CoV-2 Interim Guidance

Specificity and positive predictive value of SARS-CoV-2 nucleic acid amplification testing in a low-prevalence setting

The Limit of Detection Matters: The Case for Benchmarking Severe Acute Respiratory Syndrome Coronavirus 2 Testing

Direct Comparison of SARS-CoV-2 Analytical Limits of Detection across Seven Molecular Assays

Comparison of the analytical sensitivity of seven commonly used commercial SARS-CoV-2 automated molecular assays

Performance Characteristics of a High-Throughput Automated Transcription-Mediated Amplification Test for SARS-CoV-2 Detection

Accuracy amidst ambiguity: false positive SARS-CoV-2 nucleic acid tests when COVID-19 prevalence is low

Streamlining SARS-CoV-2 confirmatory testing to reduce false positive results

Viral Dynamics and Real-Time RT-PCR Ct Values Correlation with Disease Severity in COVID-19

Prevention and Attenuation of Covid-19 with the BNT162b2 and mRNA-1273 Vaccines

Guidance for the design and reporting of studies evaluating the clinical performance of tests for present or past SARS-CoV-2 infection

High-Throughput Transcription-mediated amplification on the Hologic Panther is a highly sensitive method of detection for SARS-CoV-2

Verification and Validation of SARS-CoV-2 Assay Performance on the Abbott m2000 and Alinity m Systems

Comparative analysis of point-of-care, high-throughput and laboratory-developed SARS-CoV-2 nucleic acid amplification tests (NATs)

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests:Roche, Abbott, and Seegene had in the past provided one of the authors (DH) with conference support. Roche, Abbott, Seegene, Cepheid and Hologic had in the past provided VCS pathology with assay kits targeting other pathogens (not SARS-CoV-2) for research studies.