key: cord-0836861-lrymf423 authors: Yi, Jie; Han, Xiao; Wang, Ziyi; Chen, Yu; Xu, Yingchun; Wu, Jie title: Analytical Performance Evaluation of Three Commercial Rapid Nucleic Acid Assays for SARS-CoV-2 date: 2021-08-14 journal: Infect Drug Resist DOI: 10.2147/idr.s321227 sha: 14b4d79d06ba2e26566fb303f1dfbb9ca7c46c9f doc_id: 836861 cord_uid: lrymf423 PURPOSE: To cope with the SARS-CoV-2 epidemic, several rapid nucleic acid assays have been approved for use, but the analytical performance has not been well evaluated. In this report, two key performance parameters, analytical sensitivity (limit of detection) and reproducibility, of three approved rapid nucleic acid assays were assessed using heat-inactivated SARS-CoV-2 culture supernatants quantified by digital PCR. METHODS: The LOD (limit of detection) and reproducibility of three approved rapid nucleic acid assays using their own instruments were assessed, while the LOD and reproducibility of two assays on a 7500 Real-Time instrument were assessed at the same time. RESULTS: Using their own instruments, 100% of samples with 1150 copies/mL viral RNA could be detected by the Da An and Coyote assays, while 90% of samples could be detected by the Ustar assay; yet, for 525 copies/mL and 287.5 copies/mL viral RNA, the detection rate of the Ustar assay was higher than that of either the Da An or Coyote assays. However, the three assays did not produce statistically significant results with the three different concentrations of viral RNA (P=0.46, 0.46 and 0.46). Using a 7500 Real-Time instrument, Da An and Coyote assays did not produce statistically significant results with the 1150, 525 and 287.5 copies/mL viral RNA (P>0.99, >0.99 and >0.99). The positive and negative detection rates of the three assays in the intra- and inter-assay stages were 100% on both their own instruments and the 7500 real-time PCR instrument. CONCLUSION: Positive or strongly positive samples can be detected by the rapid nucleic acid assay, but the analytical performance should be optimized, and comprehensive evaluations are also required. The novel SARS coronavirus 2 (SARS-CoV-2), which can cause lethal coronavirus disease, poses a great threat to human health and life 1,2 worldwide. The early detection of new infections affects patient management and the control of transmission; thus, rapid and easy-to-use diagnostic assays are urgently needed. 3 The Real-time PCRbased diagnostic testing of respiratory specimens represents the diagnostic standard of SARS-CoV-2 infection; 4, 5 however, this technique is time-consuming (even up to 24 h for the result) and not always available, and the technical requirements can usually only be met by special training and in special PCR laboratories. Currently, SARS-CoV-2 antigen-detecting rapid diagnostic tests are being extensively researched and applied; these require no special training and generate results quickly, in contrast to molecular-based testing. [6] [7] [8] However, they are less methodologically sensitive than PCR. [9] [10] [11] Thus, molecular-based tests are expected to be an accurate and reliable method for SARS-CoV-2 detection and convenient or conducive to widespread use. To cope with the SARS-CoV-2 epidemic, several rapid, molecular-based testing kits for SARS-CoV-2 have been produced. In this study, we provide a direct comparison of the key performance characteristics of three commercial rapid, molecular-based assays using a quantitative RNA reference panel for the assessment of their analytical sensitivity (limit of detection, LOD) and reproducibility. Reference standard material, heat-inactivated SARS-CoV -2 culture supernatants (BetaCoV/Wuhan/IPBCAMS-WH -01/2019, GenBank: MT019529.1) grown in Vero cells (ATCC CCL-81), was provided by Division II of In Vitro Diagnostics for Infectious Diseases, Institute for In Vitro Diagnostics Control, National Institutes for Food and Drug Control. The initial concentration of viral RNA was 460,000 copies/mL, quantified by digital PCR, and was diluted in viral transport media for the calculation of LOD and reproducibility. For verification of the LOD, the reference material containing SARS-CoV-2 RNA was diluted to three concentrations, which were above, equal to and below the LOD of the three assays. Then, 10 replicates of each concentration of viral RNA were tested. The positive detection rates at each viral RNA concentration were calculated to compare the LODs of the three assays. Both intra-and interassay reproducibilities were assessed at concentrations higher than the LOD with 20 replicates as well as 20 negative clinical samples confirmed by real-time PCR. Five replicates of each assay were applied in the intra-assay stage, and four replicates of the intra-assay stage were included in the inter-assay stage. The positive and negative detection rates were calculated to compare the reproducibility of the three assays. In all tests, the negative and positive control material was consistently detected. A total of 100 μL of diluted, heat-inactivated SARS-CoV-2 culture supernatants was added to 25 μL of nucleic acid extraction and purification reagent. Real-time reverse transcription PCR (rRT-PCR) for SARS-CoV-2 detection was performed using a Detection Kit for 2019-nCoV (PCR-Fluorescence) (Da An Gene Co., Ltd., Guangzhou, China). A total of 15 μL of diluted, heat-inactivated SARS-CoV-2 culture supernatants was added to 15 μL of respiratory tract specimen treatment reagent. rRT-PCR for SARS-CoV-2 detection was performed using a 2019-nCoV nucleic acid detection kit (PCR-Fluorescence) (Coyote Bioscience Co., Ltd., Beijing, China). Each rRT-PCR had a final volume of 52 µL on its own instrument, including 15 µL of purified nucleic acid product and 37 µL of rRT-PCR mixture, while each rRT-PCR had a final volume of 50 µL on its own instrument, including 15 µL of purified nucleic acid product and 35 µL of rRT-PCR mixture. The Coyote kit detects the ORF1ab and N genes. The results could be concluded as positive when the Ct values of both the ORF1ab and N genes were ≤27; otherwise, when the Ct values of both the ORF1ab and N genes were >27, a negative result was considered. Results with ORF1ab or N gene Ct values ≤27 would be concluded as presumptive positive, and retesting would be required; if results with ORF1ab or N gene Ct values ≤27 were detected a second time, a positive result would be considered. The whole detection time was approximately 40 min. diluted, heat-inactivated SARS-CoV-2 culture supernatant was added to the detection tube. Isothermal amplification real-time fluorescence assay was conducted on its supporting testing instrument. The Ustar kit detects the ORF1ab and N genes. The results could be concluded as positive when the Ct values of either the ORF1ab or N gene or both genes were ≤40; otherwise, when the Ct values of both the ORF1ab and N genes were N/A, a negative result was considered. When the result was "Invalid" or "No results", retesting was required. The whole detection time was approximately 60 min. The principle of both the Da An Gene and Coyote 2019-nCoV nucleic acid detection assays was real-time reverse transcription PCR (rRT-PCR). To compare the LOD and reproducibility of these two assays on the same platform, amplifications with the same samples using two reagents were carried out according to the manufacturers' instructions on a 7500 Real-Time PCR instrument (Applied Biosystems, San Francisco, CA, USA). Comparisons between two groups were made using the Mann-Whitney U-test, and comparisons between three groups were made using analysis of variance (ANOVA). A p value less than 0.05 (two sided) was considered statistically significant. The analyses were performed using Prism 7.0 (GraphPad, La Jolla, CA, USA) software. According to the manufacturers' instructions, the LODs of the Da An and Ustar assays were both 500 copies/mL, and the LOD of the Coyote assay was 400 copies/mL; thus, the initial concentration of viral RNA (460,000 copies/mL) was diluted to 1150 copies/mL, 525 copies/mL and 287.5 copies/mL. Samples of the three concentrations were detected ten times using the three rapid nucleic acid assays separately. The results showed that 100% of samples with 1150 copies/mL viral RNA could be detected by the Da An and Coyote assays, while 90% of samples could be detected by the Ustar assay. The result of one sample showed a positive amplification curve without a Ct value. For samples with 525 copies/mL viral RNA, the results were different: 90% samples could be detected by both the Da An and Coyote assays, while 100% of samples could be detected by the Ustar assay. The ORF1ab gene of one sample could not be detected using the Da An assay; neither the N gene nor the ORF1ab gene of another sample could be detected using the Coyote assay, while all ten samples could be detected using the Ustar assay. For samples with 287.5 copies/mL viral RNA, 60%, 70% and 80% of those were detected by the Da An, Coyote and Ustar assays, respectively. Specifically, four samples and the N gene of three samples could not be detected using the Da An assay; three samples and the ORF1ab gene of two samples were not detected using the Coyote assay; and both N and ORF1ab gene of one sample, the N gene of three samples and the ORF1ab gene of one sample were not detected using the Ustar assay, besides one. Overall, the three assays were not significantly different when testing 1150 copies/mL, 525 copies/mL and 287.5 copies/mL of viral RNA (P=0.46, 0.46 and 0.46) ( Table 1) . The reproducibilities of the three rapid nucleic acid assays were assessed by calculating the positive and negative detection rates of both the N and ORF1ab genes in the intra-and inter-assay stages. Samples with 1150 copies/ mL viral RNA were used to detect the positive detection rates. The results showed that both the N and ORF1ab genes of samples with viral RNA could be detected by the three assays conducted on their own instruments. In the negative samples, no genes could be detected by the three assays. Thus, the positive and negative detection rates of the three assays in the intra-and inter-assay stages were 100%. The Da An and Coyote assays were performed on a 7500 Real-Time PCR Instrument, following the manufacturer's instructions. The results showed that 90% and 100% of samples with 1150 copies/mL viral RNA could be detected by the Da An and Coyote assays, respectively, 100% and 90% of samples with 525 copies/mL viral RNA could be detected by the Da An and Coyote assays, respectively, DovePress and 80% and 90% of samples with 287.5 copies/mL viral RNA could be detected by the Da An and Coyote assays, respectively. Specifically, one sample with 1150 copies/mL viral RNA could not be detected by the Da An assay, but all samples with the same concentration of viral RNA could be detected by the Coyote assay. However, one sample with 525 copies/mL viral RNA could not be detected by the Coyote assay, but the N gene of one sample with the same concentration of viral RNA could be detected by the Da An assay. For samples with 287.5 copies/mL viral RNA, two samples and the N gene of two samples could not be detected using the Da An assay, and one sample and the N gene of one sample were not detected using the Coyote assay. Overall, two assays were not significantly different in the samples with 1150 copies/mL, 525 copies/mL and 287.5 copies/mL viral RNA (P>0.99, >0.99 and >0.99) ( Table 2) . With regard to reproducibility, both the N and ORF1ab genes of samples with 1150 copies/mL viral RNA could be stably detected, and in the negative samples, no gene could be detected by either assay. Thus, the positive and negative detection rates of the two assays in the intra-and interassay stages were 100%. The rapid nucleic acid assay has high application value in the field during quarantine amid public health emergencies. The Ebola virus Congo epidemic demonstrated the need for POCT in the prevention and control of new outbreaks. 12 To date, the COVID-19 epidemic has already spread to more than 240 countries worldwide. To prevent and control the epidemic, several rapid nucleic acid assays have been approved for use, which has greatly promoted the development and application of rapid molecular diagnosis worldwide. The superiority of the rapid nucleic acid assay was its excellent timeliness, as the time spent on the entire detection protocol was no more than one hour for three assays in this study. There was no difference between the Da An and Coyote assays in the verification of LOD and reproducibility when the assays were conducted on either their own instruments (P=0.46) or the 7500 Real-Time PCR Instrument (P>0.99) ( Table 2) . Additionally, the three assays exhibited good reproducibility, as the positive and negative detection rates were 100% in the intra-and inter-assay stages. However, compared to conventional rRT-PCR, the rapid nucleic acid assay is less sensitive, of which the LODs are mostly 400~1000 copies/ mL. 13, 14 In this study, 90% and above samples with 525 copies/mL viral RNA, close to the LODs, could be detected by the three assays, but when the viral RNA was diluted to 287.5 copies/mL, all three assays showed a decreased positive detection rate. Therefore, for positive samples with low viral load, false-negative detection may be possible using the rapid nucleic acid assay clinically. Notably, the sensitivity of the Ustar assay was higher than that of the other two assays, and perhaps the sample volume was an important influencing factor. The Ustar assay is a POCT molecular diagnostic assay and needs 500 μL of the sample for detection, while the other two assays are rapid nucleic acid assays but not POCT, requiring 100 μL and 15 μL for the Da An and Coyote assays, respectively. Moreover, the rapid nucleic acid assay can only detect two to six samples at the same time, which makes it not applicable for large population screening. In the LOD evaluation, the results of the two samples showed a positive curve without a Ct value, which may be caused by data collection error within the instruments; therefore, the instrument performance should be improved. Although the rRT-PCR test has become the standard method for SARS-CoV-2 infection diagnosis, in clinical practice, it is important to take into account the moment of the suspected infection, the patient's medical history, the symptoms and the general clinical picture for a successful outcome of the diagnostic test. 15, 16 Molecular tests will have a greater chance of being positive at the onset of symptoms, while the search for IgM and IgG antibodies begins to yield positivity at approximately one month after the presumed infection, and the levels of these immunoglobulins remain high for long periods of time. 17 Therefore, IgM and IgG antibody tests exhibited an effective complement to the false-negative results from a nucleic acid test for SARS-CoV-2 infection diagnosis after onset. [18] [19] [20] In summary, this study was a simple comparison of the LODs and reproducibilities of three commercial rapid nucleic acid assays. The initial sample was preserved in a −80 °C freezer for a few days, so the results may be affected by the freshness of the samples, the sample dilution ratio, the operation process, and other factors. Therefore, our study does not represent the overall detection performance of these assays. Rapid nucleic acid assays can be a powerful complement to conventional real-time PCR, but their performance still needs to be optimized, and comprehensive evaluations are also required. In general, the most appropriate assay Gene N Gene Gene N Gene Gene N Gene Gene N Gene ORF1ab Gene should be selected according to the local epidemic trend of disease, the actual situation of the clinical samples, the purpose of detection, and the laboratory conditions. -Notes: -*, Could not be detected. No value # , Could be detected Note: -*, Could not be detected Early transmission dynamics in Wuhan, China, of novel coronavirus-infected pneumonia A novel coronavirus emerging in China -key questions for impact assessment Laboratory diagnosis of emerging human coronavirus infections -the state of the art Laboratory diagnosis of SARS-CoV-2 -A review of current methods Positive rate of RT-PCR detection of SARS-CoV-2 infection in 4880 cases from one hospital in Diagnostic performance of a SARS-CoV-2 rapid antigen test in a large, Norwegian cohort Accuracy of a nucleocapsid protein antigen rapid test in the diagnosis of SARS-CoV-2 infection Evaluation of the rapid antigen test CerTest SARS-CoV-2 as an alternative COVID-19 diagnosis technique Clinical evaluation of Roche SD biosensor rapid antigen test for SARS-CoV-2 in municipal health service testing site, the Netherlands The evaluation of a newly developed antigen test (QuickNavi™-COVID19 Ag) for SARS-CoV-2: a prospective observational study in Japan Clinical performance of the Abbott Panbio with nasopharyngeal, throat, and saliva swabs among symptomatic individuals with COVID-19 Molecular and point-of-care diagnostics for Ebola and new threats: national POCT policy and guidelines will stop epidemics Analytic sensitivity of 3 nucleic acid detection assays in diagnosis of SARS-CoV-2 infection Comparison of three TaqMan real-time reverse transcription-PCR assays in detecting SARS-CoV-2 Sensitivity, specificity and predictive values of molecular and serological tests for COVID-19: a longitudinal study in emergency room Temporal dynamics of viral load and false negative rate influence the levels of testing necessary to combat COVID-19 spread Update on serologic testing in COVID-19 Analysis of adjunctive serological detection to nucleic acid test for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection diagnosis Evaluation of nucleocapsid and spike protein-based enzyme-linked immunosorbent assays for detecting antibodies against SARS-CoV-2 Drug Resistance is an international, peer-reviewed openaccess journal that focuses on the optimal treatment of infection (bacterial, fungal and viral) and the development and institution of preventive strategies to minimize the development and spread of resistance. The journal is specifically concerned with the epidemiology of antibiotic resistance and the mechanisms of resistance development and diffusion in both hospitals and the community. The manuscript management system is completely online and includes a very quick and fair peerreview system The authors would like to thank Professor Haiwei Zhou for supplying SARS-CoV-2 culture supernatants. We also thank the Elsevier workshop for the language editing services. This work is supported by the Beijing Key Clinical Specialty for Laboratory Medicine Excellent Project (No. ZK201000). The authors report no conflicts of interest in this work.