key: cord-0956955-peysss2r
authors: Price, Travis K.; Bowland, Brian C.; Chandrasekaran, Sukantha; Garner, Omai B.; Yang, Shangxin
title: Performance Characteristics of SARS-CoV-2 PCR Tests in A Single Health System: Analysis of over 10,000 Results from Three Different Assays
date: 2020-12-05
journal: J Mol Diagn
DOI: 10.1016/j.jmoldx.2020.11.008
sha: 756a8fe37d8456ff26147d4dab7b191e6e5f1d54
doc_id: 956955
cord_uid: peysss2r

The current pandemic of SARS-CoV-2 has resulted in the approval of numerous molecular diagnostic assays with various performance and technical capacities. There are limited data comparing performance among assays. We conducted a retrospective analysis of >10,000 test results among three widely used RT-PCR assays for COVID-19 (CDC, Simplexa Direct, and TaqPath) to assess performance characteristics. We also re-tested remnant weakly positive specimens to assess analytical sensitivity. All assays had strong linear correlation and little bias among cycle threshold (Ct) values for PCR targets. In patients with first-test negative results (n=811), most (795, 98.0%) remained negative for all subsequent testing. Re-testing of weakly positive specimens (Ct>30) showed sensitivities as TaqPath (97.8%), CDC (91%), Simplexa (75.3%). Our analysis showed no performance difference among PCR targets within the same assay, suggesting a single target is sufficient for SARS-CoV-2 detection. Lower respiratory tract (LRT) specimens had higher negative predictive value (NPV) (100%) than upper respiratory tract specimens (98%), highlighting the utility of testing LRT specimens when clinically indicated. NPV did not increase upon further repeat testing, providing strong evidence for discouraging unnecessary repeat testing for SARS-CoV-2.

SARS-CoV-2, which emerged in December 2019 in Wuhan, China, has resulted in the currently ongoing pandemic of Coronavirus Disease 2019 (COVID-19)(1) (2) (3) (4). The first molecular diagnostic assays were developed (5) shortly after the characterization of the viral genome. In February 2020, the Centers for Disease Control and Prevention (CDC) received an Emergency Use Authorization (EUA) for the COVID-19 Real-Time Reverse Transcriptase PCR (RT-PCR) assay. Following that, numerous commercial molecular diagnostic assays were approved with EUA for COVID-19, with various performance, throughput, speed and technical complexity. Since then, numerous reports comparing analytical performance between and among assays have been published(6) (7) (8) (9) (10) (11) (12) (13) . However, these data were obtained based on small sample sizes, making robust conclusions difficult.

Here, we report retrospective data collected from March 9 th to April 29 th on over 10,000 clinical specimens from 8948 unique patients tested using three assays: CDC COVID-19 RT-PCR, Simplexa COVID-19 Direct Real-Time RT-PCR (DiaSorin Molecular, Cypress, CA), and TaqPath COVID-19 RT-PCR (Thermo Fisher Scientific, Waltham, MA). We measured negative predictive values (NPVs) among assays, and bias among gene targets within each assay. We also report data for over 900 patients with multiple test results to assess the NPV of the first result. Finally, we re-tested >100 previously weakly positive specimens by all three assays to compare analytical sensitivities.

J o u r n a l P r e -p r o o f

Three commercially available assays approved under the EUA were used for the qualitative detection of SARS-CoV-2 as part of standard of care testing at the University of California Los Angeles (UCLA) Health System. We performed a retrospective analysis of PCR results obtained from these assays between 3/9/2020 and 4/29/2020. During this period, testing was mostly performed on patients with symptoms or exposure history. This study is exempt from Institutional Review Board (IRB) review.

Tests were performed on various specimen types: Nasopharyngeal (NP) swab (n=10215), Bronchoalveolar Lavage (BAL) (n=121), Expectorated sputum (n=22), and miscellaneous sample types (n= 35). Of the 630 positive tests, 613 were NP swabs, 11 were BAL, 2 were expectorated sputum, and 4 were other sample types.

The CDC COVID-19 RT-PCR assay targets two regions of the SARS-CoV-2 N gene (i.e., N1 and N2). Nucleic acid extraction was performed using EZ1 Advanced XL (Qiagen, Hilden, Germany) or NUCLISENS easyMAG (bioMérieux, Hazelwood, MO). RT-PCR was performed on the Applied Biosystems 7500 Fast Real-Time PCR instrument (Thermo Fisher Scientific, Waltham, MA). Detection of both gene targets was considered positive; detection of 1 of 2 targets was deemed "inconclusive". This assay was mainly performed on bronchioalveolar lavage (BAL) and sputum since it was the only assay approved for the lower respiratory tract (LRT) sample types in the early phase of the COVID-19 outbreak. Other respiratory specimens, such as tracheal aspirate, were tested and resulted with a disclaimer stating a non-validated specimen source was used. deemed "inconclusive". This assay was performed on NP swabs, NP aspirates, and BAL specimens that are not viscous. This test was primarily used on ambulatory patients and lowrisk healthcare workers due to its highest throughput.

A total of 107 weakly positive [defined as any target regardless of assay, with a cycle threshold (Ct) value of 30 or more] NP swab specimens were selected. These frozen specimens (stored at -80°C for 3-8 weeks) were thawed and re-tested by all three assays within 8 hours.

Categorical variables were reported as frequencies and percentages; continuous variables were reported as means with standard deviations (SD). Pearson Chi-square or Fisher's Exact Tests and T-tests or Analysis of Variance (ANOVA) were used to compare categorical and continuous results, respectively.

Negative Predictive Value (NPV) was calculated for the retrospective analysis of withinpatient data. NPV was defined as the true negatives over the true negatives plus the false J o u r n a l P r e -p r o o f negatives. True negatives were defined as patients with repeatedly negative test results. False negatives were defined as an initial negative test result followed by a subsequent positive test result. 

Of We also performed NPV calculation based on results from the 3 different assays and found that NPVs did not differ significantly among assays ( Table 2 ). In the 16 cases (2.0%, 16 /811) that met our "false-negative" definition (i.e., initial test negative and then became positive subsequently), 15 of them were tested positive by the same assay subsequently. Only in one case, an initial negative result by Simplexa was followed by a positive result using CDC.

In this case, the negative result was an NP swab and the positive was a Tracheal suction; the specimens were collected on the same day. These data show that false-negative results are unrelated to the specific assay performed in our institution.

NPVs differed slightly by initial specimen type (i.e., URT or LRT) ( Table 2) . Patients with initially negative URT specimens (n=785) had an NPV of 98.0%, while patients with initially negative LRT specimens (n=26) had no false-negatives, giving an NPV of 100%. Twenty-nine patients had an URT and LRT specimen collected and tested on the same day. The same day specimens were usually both negative (86.2%, 25/29); one pair (3.4%) were both positive, and three pairs (10.4%) had conflicting results. Among those with conflicting results, all were URT-/LRT+.

A total of 107 weakly positive (any target Ct ≥30) NP specimens were selected for retesting on all three assays. These samples consisted of 17 (15.9%), 64 (59.8%), and 26 (24.3%) originally tested by CDC, Simplexa and TaqPath, respectively. Re-testing was performed on the frozen specimen and all assays were performed within 8 hours to ensure the same sample stability to allow an unbiased comparison among the 3 assays. We found that 61.7% (66/107) of specimens tested positive on all three assays, while 3.7% (4/107) were negative on all three.

The remaining 34.6% (37/107) tested positive on only one (9/37) or two assays (28/37).

TaqPath was shown to be most sensitive (92.5%, 99/107), followed by CDC (90.7%, 97/107) and Simplexa (62.6%, 67/107). The freeze-thaw cycle negatively affected the performance of Simplexa the most, with only 49 of the 64 (76.6%) specimens re-tested positive on the same assay. However, the freeze-thaw cycle seemed to have less impact on the retested positivity rate on the CDC assay (82.4%, 14/17) and the TaqPath assay (92.3%, 24/26).

Because freeze-thaw cycles were shown to affect the assays differently, we excluded specimens that were not reproducibly positive by the same assay (n=18) to remove this confounding factor. The re-calculated sensitivities of the three assays were: 97.8% (TaqPath), 91.0% (CDC), and 75.3% (Simplexa) ( Table 3 ). The 15 specimens that were not repeat positive on Simplexa had a significantly higher (p<0.01) average Ct value (33.6±2.3) than the 49 that J o u r n a l P r e -p r o o f were repeat positive (32.0±1.9). These data show that Simplexa had lower sensitivity for the very weakly positive samples. Similar comparisons for the TaqPath and CDC assays were not statistically significant.

We retrospectively evaluated over 10,000 test results for the detection of SARS-CoV-2 in patient specimens from three RT-PCR assays: CDC, Simplexa Direct, and TaqPath. We plotted Ct values for every pair of targets for each assay and used linear regression to show that values were strongly correlative; this was consistent among assays and specimen types. We also used Bland-Altman plots to show that the mean bias of Ct values between targets was near 0 for all assays. Considering the large sample size used to generate these data, we believe that these findings strongly support the claim that testing of a single target is appropriate for SARS-CoV-2 detection. Multiple targets per assay creates the opportunity for inconclusive results which could lead to increased complexity in result interpretation. The majority of the inconclusive results obtained here tested negative upon repeat testing (15/21).

We calculated the NPV for patients with initial negative test results. The overall NPV was high (98%) and did not differ significantly by number of repeat testing or assay used in our institution. These data show that re-testing negative patients is unlikely to yield a subsequent positive result. These findings are consistent with other reports (14) and serve as important considerations for clinical and epidemiological decision-making regarding suspected COVID-19 patients. We found that the NPV of LRT specimens (100%) was significantly higher than that of URT specimens (98%). Considering the pathogenesis of the virus, these data likely relate to the stage of disease. Several studies have shown that URT specimens are positive earlier in infection, while LRT specimens have prolonged positivity for SARS-CoV-2(15) (16) (17) . By assessing the 29 patients with both URT and LRT specimens collected on the same day, we found that most test results (26/29) matched between paired specimens. However, in three J o u r n a l P r e -p r o o f cases the URT specimens were negative while the LRT samples were positive, highlighting the importance of testing the LRT samples in a setting of negative URT sample but still clinically suspected for COVID-19.

We performed a comparison study using weakly positive samples to assess the sensitivity of the 3 SARS-CoV-2 RT-PCR assays and the results showed Simplexa had a lower sensitivity than TaqPath or CDC. Notably, the sensitivity of the Simplexa assay is greatly reduced for extremely weak positive specimens with Ct values above 33.6. However, this is not a significant concern because only 2.1% (7/338) of the Simplexa-positive specimens had both Ct values >33.6. In addition, its ease of use, fast speed, and our findings that its NPV was not statistically different from the other two assays, justify the use of Simplexa for clinical diagnostic testing of symptomatic patients. Ultimately, we believe pre-analytical parameters including specimen types, sample collection quality and timing are more important factors for the clinical performance of the PCR assays for SARS-CoV-2 detection.

Limitations of our study included the lack of clinical chart review. This would be particularly important to determine the stage of disease of positive patients and could be used to determine clinical false negative results. Variation in specimen collection techniques and collection time relative to symptoms was not considered in this analysis; however, the large sample size likely overcomes this limitation. Additionally, our definition of false negative results is not perfect due to lack of clinical information or further data such as serological test results.

Finally, the weakly positive specimens used for the comparison study were previously frozen and may have degradation which reduced the reproducibility. Our data showed that the freezethaw cycle likely affected PCR assay sensitivity, especially on Simplexa.

In summary, this is one of the first studies that assessed over 10,000 patient test results to compare SARS-CoV-2 PCR assay performance. Our analysis showed no performance difference among different PCR targets within the same assay, suggesting only one target is "True Negative" was defined as all subsequent patient tests (regardless of specimen type) were Negative/Not Detected. "False Negative" was defined as at least one subsequent patient test (regardless of specimen type) was Positive/Detected. J o u r n a l P r e -p r o o f J o u r n a l P r e -p r o o f 

A Novel Coronavirus from Patients with Pneumonia in China

A familial cluster of pneumonia associated with the 2019 novel coronavirus indicating person-toperson transmission: a study of a family cluster

Coronaviridae Study Group of the International Committee on Taxonomy of Viruses. The species Severe acute respiratory syndrome-related coronavirus: classifying 2019-nCoV and naming it SARS-CoV-2. Nat Microbiol

Novel Coronavirus in the United States

Drosten CDetection of 2019 novel coronavirus (2019-nCoV) by real-time RT-PCR

Comparison of Four Molecular In Vitro Diagnostic Assays for the Detection of SARS-CoV-2 in Nasopharyngeal Specimens

The Detection of SARS-CoV-2 using the Cepheid Xpert Xpress SARS-CoV-2 and Roche cobas SARS-CoV-2 Assays

Comparison of Abbott ID Now, Diasorin Simplexa, and CDC FDA EUA methods for the detection of SARS-CoV-2 from nasopharyngeal and nasal swabs from individuals diagnosed with COVID-19

Comparison of Abbott ID Now and Abbott m2000 methods for the detection of SARS-CoV-2 from nasopharyngeal and nasal swabs from symptomatic patients

Comparison of Commercially Available and Laboratory Developed Assays for in vitro Detection of SARS-CoV-2 in Clinical Laboratories

Evaluation of the QIAstat-Dx Respiratory SARS-CoV-2 Panel, the first rapid multiplex PCR commercial assay for SARS-CoV-2 detection

Clinical evaluation of the cobas SARS-CoV-2 test and a diagnostic platform switch during 48 hours in the midst of the COVID-19 pandemic

Comparison of Cepheid Xpert Xpress and Abbott ID Now to Roche cobas for the Rapid Detection of SARS-CoV-2. bioRxiv

Testing for SARS-CoV-2: Can We Stop at Two? Clin Infect Dis

Detection of SARS-CoV-2 in Different Types of Clinical Specimens

Viral load dynamics and disease severity in patients infected with SARS-CoV-2 in Zhejiang province

Virological assessment of hospitalized patients with COVID-2019