key: cord-0733375-uc838r4w
authors: Wang, Di; Wang, Zhidong; Gao, Ying; Wu, Xiao; Dong, Lianhua; Dai, Xinhua; Gao, Yunhua
title: Validation of the analytical performance of nine commercial RT-qPCR kits for SARS-CoV-2 detection using certified reference material
date: 2021-09-11
journal: J Virol Methods
DOI: 10.1016/j.jviromet.2021.114285
sha: 4f994435f591449449a8535a5fdf05e6804b78fa
doc_id: 733375
cord_uid: uc838r4w

The ongoing coronavirus disease 2019 (COVID-19) pandemic has become a public health emergency. Although many reverse-transcription PCR (RT-PCR) assays have been developed, their performance, especially sensitivity assessment, has been insufficiently tested. In this study, a preliminary comparison of the analytical sensitivity of nine RT-qPCR kits from different manufacturers was first conducted using a certified reference material derived from the genomic RNA of SARS-CoV-2 as the template. Subsequently, three of the nine kits, comprising two highly sensitive kits (DAAN, Huirui) and one less sensitive kit (Geneodx), were selected for further sensitivity and specificity validation. The results revealed variations in the performance between kits of the two groups. For the two highly sensitive kits, the limits of detection at 95% probability (LOD(95%)) were 5.6 copies of the N gene and 3.5 copies of the ORF 1ab per reaction (DAAN), and 6.4 (N) and 4.6 (ORF 1ab) copies per reaction (Huirui). These LOD(95%) values were approximately 3 to 4-fold better than those of the Geneodx Kit. However, none of these three Kits showed cross-reactivity against 6 other types of human coronaviruses or respiratory viruses. Because most of these commercial kits are approved as in vitro diagnostics (testing specimens without direct human contact), it would be beneficial for their manufacturers to improve the diagnostic capability of these kits and thus reduce the clinical risks associated with false-negative results.

Since its outbreak in December 2019 (Li et al., 2020) , the ongoing pandemic of coronavirus disease 2019 caused by the severe acute respiratory syndromerelated coronavirus 2 (SARS-CoV-2) has placed an enormous burden on society, economy and healthcare systems worldwide (Holshue et al., 2020; Rothe et al., 2020) . This enveloped, positive-strand RNA virus is a member of the subgenus Sarbecovirus (Han et al., 2020; Lu et al., 2020) and its genome sequence is closely related to that of the severe acute respiratory syndrome-related coronavirus (SARS-CoV) which caused the 2003 outbreak of SARS disease in humans (Guan et al., 2020) . Both SARS-CoV and SARS-CoV-2 are now classified within the virus species Severe acute respiratory syndromerelated coronavirus (CSG and ICTV, 2020) .

Reliable diagnosis is among the foremost priorities in public health interventions in order to confirm suspected cases, contact testing and monitor disease spread. Real-time RT-qPCR detection of SARS-CoV-2, regarded as the gold standard in COVID-19 diagnostics, are being applied widely to test for the virus (WHO, 2020a). Among the various PCR targets selected by diagnostic laboratories, the open reading frame 1ab (ORF 1ab), nucleocapsid gene (N) and envelope gene (E) of the SARS-CoV-2 genome are the most frequently chosen targets for RT-qPCR assays (Chu et al., 2020; Corman et al., 2020; Pfefferle et al., 2020) .

Managing this once-in-a-century COVID-19 pandemic is an enormous challenge for molecular diagnostics. Many laboratory and commercial assays have been developed to detect SARS-CoV-2, resulting in a flood of new tests into the diagnostic market. The performance characteristics of RT-qPCR assays can vary with the reagents supplied, the thermocycling conditions, instrumentation and sampling protocols, leading to different J o u r n a l P r e -p r o o f diagnostic results. For example, a series of false-negatives were diagnosed in cases for which clinical symptoms and computed tomography imaging strongly implicated COVID-19 infection (Winichakoon et al., 2020; Wu et al., 2020; Xie et al., 2020) . Wang et al. (2020) also showed that of 68 patients with confirmed COVID-19, 20.6% had initial and follow-up nasopharyngeal swabs that had tested negative for SARS-CoV-2, but the corresponding sputum specimens all tested positive in RT-qPCR assays. In a systematic review of RT-qPCR assays, false-negative rates ranging from 2% to 33% were found in repeated tests of samples (Arevalo et al., 2020) . Apart from sub-optimal specimen collection or sub-standard diagnostic operation by personnel, false-negatives could also arise from inadequate sensitivity and reliability of the RT-qPCR assays (Bustin et al., 2009) . Supply issues can also be a problem. In February, 2020, some laboratories reported delays in regional testing roll-out in various countries caused by reagent contamination (Mögling et al., 2020) . Although several vaccines are available and effective in conferring resistance to SARS-CoV-2 (WHO, 2021b), the vaccination rates in many countries are despairingly low. Vaccination will also take time to immunize entire populations.

Therefore, more and better tests are still required for the large numbers of cases arising daily and in subsequent waves of infections as well as for contact tracing.

Molecular diagnostic tests for SARS-CoV-2, such as RT-qPCR, are in vitro diagnostics (IVD) whose used is regulated by local health agencies. In the USA, they are approved and labeled as research use only (RUO) (FDA, 2013) or under emergency use authorisations (EUA) for use in clinical tests (FDA, 2021a; . EUAs are issued for each individual test with certain conditions of authorization required of the manufacturer and authorized laboratories. Additionally, unlicensed IVDs can be placed under the WHO Emergency Use Listing (EUL) with the ultimate aim of expediting the availability of these products for emergency use (WHO, 2020b) . With numerous molecular tests available on the market in China, it is essential that these tests are fully evaluated before being employed by diagnostic clinics. Therefore, the aim of this study was to compare the analytical performance of nine RT-qPCR kits for SARS-CoV-2 diagnosis from different manufacturers in China, including 6 kits approved for in-vitro diagnostics use (IVD EUA), and 3 research use only (RUO) kits which could potentially be approved for IVD use. To standardise the assay, a certified reference material (CRM) genomic RNA of SARS-CoV-2 was used as the standard template control. To account for any loss of quality during transportation and storage, the quality and concentration of the CRM was checked using a RT-digital droplet PCR (RT-ddPCR) assay (Dong et al., 2020) . The diagnostic kits were J o u r n a l P r e -p r o o f 4 of 26 first evaluated for analytical sensitivity. Then three kits with different levels of sensitivity were selected for further analysis of their RT-qPCR amplification efficiency, limit of quantification (LOQ) (Pavšic et al., 2016; .Kralik et al., 2017) , 95% limit of detection (LOD95%) (Corman et al., 2020; Pfefferle et al., 2020) and cross-reactivity with other human coronaviruses and respiratory viruses.

The nine commercial SARS-CoV-2 RT-qPCR assay kits were purchased from each manufacturer. Information on the target genes and regulatory status of these kits is listed in Authorization.

The SARS-CoV-2 certified reference material (CRM) was purchased from National

Reference Material (CNRM, Product GBW(E)091099). The product is approved by the China NMPA and contained SARS-CoV-2 genomic RNA isolated from infected patients.

Specifications of the CRM provided by the manufacturer are listed in Table 2 . The specified copy number concentrations of the ORF1ab, N and E genes, respectively, were double-checked by RT-ddPCR. After confirming the copy number concentrations, the CRM stock was diluted 10-fold serially four times (S1-S4) with the RNA storage solution provided (4.5 copies to 4.5  10 3 copies per reaction (c/r) for ORF 1ab, and 8.7 copies to 8.7  10 3 c/r for the N gene). Yeast carrier RNA (1 mg/mL) was added to the dilutions to avoid degradation of the CRM. Each dilution was divided into equal aliquots and stored at -80 °C. J o u r n a l P r e -p r o o f

The RT-ddPCR assay employed to confirm the copy number concentrations of the ORF1ab, and N and E genes in the CRM were conducted using the One-Step RT-ddPCR included in the experiment. After thermal cycling, the plates were transferred to a droplet reader (Bio-Rad, Laboratories, USA) to read the droplets. The final copy number of the CRM determined by RT-ddPCR was calculated using Equation (1):

Where C is the copy number per microliter, D is the dilution factor used to further dilute the RNA with the PCR master mixture, Vp is the droplet volume, P is the number of positive droplets, N is the total number of accepted droplets. Copy numbers were estimated from four replicates. 

The nine RT-qPCR kits for SARS-CoV-2 detection were tested in parallel using freshly thawed samples of each serially diluted CRM S1 to S4 as templates to determine their dose response and sensitivity for detecting ORF1ab and N gene. All the RT-qPCR assays were performed by following the manufacturers' instructions and reaction conditions (Table S1) Distilled water was used as the template for negative controls. Each assay was repeated three times. The results of the assay dose response were used to design the concentration of CRM for use in subsequent experiments. Table S1 . Reaction conditions of nine commercial SARS-CoV-2 RT-qPCR kits.

J o u r n a l P r e -p r o o f

Three of the nine kits, comprising two highly sensitive kits (DAAN, Huirui) and one less sensitive kit (Geneodx), were selected for further sensitivity and specificity validation as follows.

The CRM was diluted 3-fold serially in storage buffer as described above (ranging from 18.5 copies to 4.5  10 3 c/r for ORF 1ab, 11.9copies to 8.7  10 3 c/r for N gene) and

used as templates for RT-qPCR assays with each of the three selected commercial RT-qPCR kits. The Cq values were used to generate a RT-qPCR standard curve for each target gene by plotting the Cq values against the corresponding logarithm (base 10) of the copy number concentrations of the CRM dilutions. The standard curve of each assay is described by the equation (2):

Where x is the logarithm of the copy number concentrations and y is Cq values; k is the slope and b is the intercept of the standard curve. The PCR efficiency (E) was then calculated using equation (3):

probability of detection (LOD95%)

The limit of quantification (LOQ) is the lowest template concentration that an assay could accurately quantify based on the linear portion of the standard curve. The relative standard deviation (RSD) or coefficient of variance of the LOQ value determined should be ≤ 25% (Pavšic et al., 2016; Kralik et al., 2017) . To determine the LOQ of the RT-qPCR assays of the ORF 1ab and N gene for each of the three kits, a 3-fold serial dilution of the CRM, ranging from 16.7 to 4.5  10 2 c/r, were tested using eight replicates of each assay.

The limit of detection (LOD95%) of an RT-qPCR assay is the lowest concentration at which there is a 95% probability of detecting the target (Corman et al., 2020; Pfefferle et al., 2020) . To determine the LOD95% of the RT-qPCR assay for each target gene for each kit, a series of two-fold dilution of the S2 CRM sample, ranging from 1.8 to 4.5  10 2 c/r, were used as templates in each assay. The LOD95% was then determined by probit analysis of the assay results from eight replicates. The probit analysis was conducted using the SPSS 17.0 software (SPSS, Chicago, IL, USA).

The RT-qPCR ORF 1ab and N gene assays of each of the three kits were assessed for crossreactivity against a panel of viral nucleic acids of human viruses other than SARS-CoV-2.

These viral nucleic acids used as templates in the assays. were extracted from confirmed clinical samples, included human coronaviruses (SARS-CoV, Middle East respiratory syndrome coronavirus, HCoV-NL63, HCoV-OC43, HCoV-229E and HCoV-HKU1) and

other respiratory viruses (Table S2) . CRM template was the positive control and healthy J o u r n a l P r e -p r o o f human DNA was used as the template in the negative control. The amount of RNA of each virus was ranged from 1 to 10 pg/μL, 5 μL for each assay. .

The amplification products of the RT-qPCR assays were purified using the QIAquick PCR Purification Kit (Qiagen, Germany) and sent out for sequencing (Sangon Biotech, China). The resulting sequences were aligned using the NCBI Blast tool to confirm the sequence identity of the regions of the ORF1ab and N gene amplified by the assays. The reference full-length sequences of SARS-CoV-2 (GenBank accession number NC_045512.2) and SARS-CoV (AY394997.13) used for sequence alignment were retrieved from the NCBI (http://www.ncbi.nlm.nih.gov/genbank/) database.

The dot plots results of RT-ddPCR assays of the CRM to validate the specified concentrations of ORF1ab (Fig. 1a) , N (Fig. 1b) and E (Fig. 1c) showed that all the positive droplets (upper dots) were separated clearly from the negative droplets (lower dots), and only a few scattered dots were observed between them. The copy number concentrations of the three target sequences determined by the RT-ddPCR assays were 921 copies/μL for ORF1ab, 1689 copies/μL for N and 1098 copies/μL for E, while NRT and NTC controls were both negative. These results were statistically the same as the corresponding concentrations specified for the CRM by the manufacturer (Fig. 1d) . Thus, the CRM was used as the standard template in subsequent sensitivity analysis of the commercial RT-qPCR kits produced for SARS-CoV-2 detection. 

The results of ORF1ab and N gene RT-qPCR assays on the dilution series of CRM S1-S4 using the nine commercial RT-qPCR kits revealed that all these test kits were capable of amplifying the ORF1ab and N gene fragments at dilutions S1 (4.5  10 3 c/r for ORF 1ab and 8.7  10 3 c/r for the N gene) to S3 (4.5  10 1 c/r for ORF 1ab and 8.7  10 1 c/r for the N gene), with Cq values ranging from 25.6 to 37.7. At the S4 dilution level (4.5 c/r for ORF 1ab and 8.7 c/r for the N gene), the nine kits differed in their ability to detect these J o u r n a l P r e -p r o o f genes (Table 4) . At this level, five kits (Kit-1 to Kit-5) detected both target sequences in all three replicate tests, while three kits (Kit-6, Kit-8 and Kit-9) were unable to detect ORF1ab at all, but detected the N gene in some of the replicates only. In contrast, Kit-7 could not detect either ORF1ab or N in all three replicates (Table 4 ). Based on the above result, two of the highly sensitive kits (Kit-1 and Kit-2) and the least sensitive Kit-7 were selected as the representatives for further analysis. The RT-qPCR amplification curves generated for

ORF1ab and the N gene by these three kits using various template concentrations are shown in Fig. S1 . The template dose response results were used to determine the template concentrations used in subsequent experiments. qPCR assays using S1 to S4 dilutions of the CRM RNA and three replicates of each dilution.

From the standard curves, the RT-qPCR efficiencies (E) of the three kits for detecting ORF 1ab and N gene ranged from 91.43 to 103.10% (R 2 ≥ 0.98) (Fig. 2) . The LOQ of the three kits for detecting ORF 1ab (Fig. 2 , left column panels) was 23 (Kit-1), 23

(Kit-2) and 61 (Kit-7) c/r, respectively. The corresponding LOQ for the N gene (Fig. 2, right panels), was 26, 26 and 78 c/r, respectively. Compared with the other two kits, the standard curves of Kit-7 also exhibited a poor linearity range for quantifying the target genes. pointed to the LOQ of each assay. Each RT-qPCR assay was the mean of eight replicates; each error bar represented the standard deviation of the mean. For standard curves of targeting ORF 1ab, the template concentrations of Kit-1, Kit-2, Kit-7 were ranged from 23 copies to 4.5  10 3 c/r, 23 copies to 4.5  10 3 c/r and 61 copies to 4.5  10 3 c/r, respectively.

For standard curves of targeting N gene, those were ranged from 26 copies to 8.7  10 3 c/r, 26 copies to 8.7  10 3 c/r and 78 copies to 8.7 10 3 c/r.

The results of the probit analysis (Fig. 3) of each RT-qPCR assay revealed a LOD95% of 3.5 c/r for ORF1ab and 5.6 copies for the N gene using Kit-1 (Fig. 3a) . For Kit-2, the LOD95% was 4.6 copies for ORF1ab and 6.4 copies for N gene (Fig. 3b) . In contrast, the resulting LOD95% for Kit-7, the least sensitive kit, was much higher, being 14.3 copies for ORF1ab and 20.4 for the N gene (Fig. 3c ).

J o u r n a l P r e -p r o o f 

Specificity evaluation of the three RT-qPCR kits showed that all of them were specific for SARS-CoV-2 only, with no cross-reactivity with the other human viruses tested (Table S2) or with the healthy human DNA negative control. 

Sequence analysis of the amplicons obtained from the RT-qPCR assays of the obtained from the kits showed that their sequences matched completely with the corresponding sequences from the SARS-CoV-2 genome but some mismatches with the SARS-CoV genome were found (Fig. 4b, c) . 

This comparative analysis of the performances of nine RT-qPCR kits indicated that their sensitivities (LOQ and LOD95%) and efficiencies (E) for detecting ORF1ab and N gene differed substantially between them. Comparison of the three selected kits showed that the standard curves of Kit-7 (low sensitivity) exhibited a poor linearity range for quantifying the target genes compared to those of the highly sensitive Kit-1 and Kit-2. However, given that all these kits were designed to detect at least two targets simultaneously, the chances of identifying specimens with low viral loads could be enhanced (LeBlanc et al., 2020) .

Comparison of the regulatory status of the three kits showed that both Kit-1 and Kit-7 are EUA approved by China NMPA, IVD approved by EU and approved under WHO EUL for expediting availability, while Kit-2 is approved for RUO. The lower E, LOQ and LOD95% observed for Kit-7 compared to the other two kits suggest that caution is still required in the use of approved IVD products.

To check the accuracy of the LOD95% values obtained for the RT-qPCR assays in this study (3.5 c/r for ORF 1ab), we repeated Corman's PCR assay (Corman et al., 2020) using the reported protocol and reagents, while using the CRM as template. The resulting LOD95% of the RT-qPCR assay for the RdRp gene was 3.2 copies/reaction (95% confidence intervals: 2.4 -8.1) (data not shown) which agreed with the reported value by Corman et al. (2020) . Furthermore, the LOD95% of SARS-CoV-2 detection by six RT-qPCR kits, which included DAAN (Kit-1) and Geneodx (Kit-7), was consistent with our finding that the LOD95% differed significantly between Kit-1 and Kit-7 (Wang et al., 2020) . The LOD95% difference between them was found to increase 16-fold when they were tested on clinical samples. For Kit-1 and Kit-2, their LOD95% values were roughly in line with the theoretical LOD95% of the RT-qPCR assay of three molecules per reaction according to the Poisson distribution (Burns et al., 2007; Forootan et al., 2017; Kralik et al., 2017) .

The PCR products of Kit-2 and Kit-7 were identical to those amplified using the primer/probe sets recommended by the Chinese Center for Disease Control and Prevention (CCDC) for SARS-CoV-2 detection. This indicated that the differences in the performance of these kits was influenced by other factors, such as reaction volume (Kit-7 used 50 μL), thermocycling conditions and differences in reagents.

The clinical implication of the performance results observed for these kits may be limited by the lack of clinical sample testing, due to the strict control policy in China. To mimic the features of clinical specimens, the CRM contained total RNA extracted from SARS-CoV-2-infected patients, instead of in vitro-transcribed RNA standards or virion RNA. However, it is still quite possible that these results may be different from those J o u r n a l P r e -p r o o f obtained using clinical samples where differences in RNA quality and RT-qPCR inhibitors are likely to exist.

Interestingly, the number of RNA copies present in the CRM differed between the three targets tested according to both the manufacturer's specification and the RT-ddPCR results of this study. This is consistent with what was found by Chu et al. (2020) . Several factors may be responsible, but the presence of subgenomic RNA (sgRNA) in the CRM might be a major contributor (Kim et al., 2020; Alexandersen et al., 2020) . Because the CRM was derived from clinical specimens containing SARS-CoV-2 infected cells that expressed sgRNAs, it is not unexpected that more copies of the N and E genes were detected in the CRM (Chu et al., 2020) . In addition, differences in the efficiencies of the various steps of the RT-qPCR, such as reverse-transcription, primer annealing, amplicon size, etc., could be other reasons (Schwaber et al., 2019; Niu et al., 2021) .

High sensitivity is merely one factor to be considered for SARS-CoV-2 detection.

In certain situations, diagnostic laboratories need to meet the unprecedented demand for tests. Accordingly, rapid diagnostic test (RDT) systems may offer the possibility of rapid, simple and portable detection of COVID-19 cases under these situations, despite their relatively lower sensitivity (Patriquin et al., 2021) . Some studies indicated that a regime of repeated RDTs for target populations (e. g., hospital patients) might increase the probability of identifying SARS-CoV-2 infectious individuals, as well as overcoming the limitation of the poor sensitivity of these tests (Mina et al., 2020; Larremore et al., 2020) . However, conditions, such as a minimal acceptable sensitivity and an optimal testing frequency, still need to be well-defined for RDTs to be successful (Patriquin et al., 2021) .

The results of the validation tests on three commercial RT-qPCR kits with differing levels of sensitivities for the ORF1ab and N gene, when tested using CRM as templates, showed that none of them showed cross-reactivity against other tested human respiratory viruses. However, the differences in performance provided by the different kits indicated that the manufacturers need to analyze and further improve their products to increase their diagnostic capability for clinical samples with low viral load. Critically, testing laboratories should conduct appropriate in-house validations before using any new RT-qPCR kits.

J o u r n a l P r e -p r o o f

SARS-CoV-2 genomic and subgenomic RNAs in dignostic samples are not an indicator of active replication

False-negative results of initial RT-PCR assays for COVID-19: A systematic review

Modelling the limit of detection in real-time quantitative PCR

The MIQE guidelines: minimum information for publication of quantitative real-time PCR experiments

Molecular diagnosis of a Novel Coronavirus (2019-nCo-V) causing an outbreak of pneumonia

Certified Reference Material of 2019 Novel Corona Virus (2019-nCoV) Ribonucleic Acid Genome

Detection of 2019 novel coronavirus (2019-nCoV) by realtime RT-PCR

The species severe acute respiratory syndrome-related coronavirus: classifying 2019-nCoV and naming it SARS-CoV-2

Tolerance of droplet-digital PCR vs real-time quantitative PCR to inhibitory substances

Highly accurate and sensitive diagnostic detection of SARS-CoV-2 by digital PCR

Sensitivity assessment of droplet digital PCR for SARS-CoV-2 detection

Distribution of in vitro diagnostic products labeled for research use only or investigational use only. Guidance for Industry and FDA Staff

19-emergency-use-authorizations-medical-J o u r n a l P r e -p r o o f devices/in-vitro-diagnostics-euas

In vitro diagnostics EUAs -Molecular diagnostic tests for SARS-CoV-2

Methods to determine limit of detection and limit of quantification in quantitative real-time PCR (qPCR)

Isolation and characterization of viruses related to the SARScoron-avirus from animals in southern China

Coronavirus 2019-nCoV: A brief perspective from the front line

First case of 2019 Novel Coronavirus in the United States

The digital MIQE guidelines: minimum information for publication of quantitative digital PCR experiments

The architecture of SARS-CoV-2 transcriptome

A basic guide to real time PCR in microbial diagnostics: definitions, parameters, and everything. Front Microbiol

Test sensitivity is secondary to frequency and turnaround time for COVID-19 surveillance

Real-time PCR-based SARS-CoV-2 detection in Canadian laboratories

Early transmission dynamics in Wuhan, China, of novel coronavirus infected pneumonia

Genomic characterisation and epidemiology of 2019 novel coronavirus: implications for virus origins and receptor binding

Rethinking Covid-19 test sensitivity-a strategy for containment

Digital PCR hits its stride

Delayed laboratory response to COVID-19 caused by molecular diagnostic contamination

Quantitative analysis of RNA by HPLC and evaluation of RT-dPCR for coronavirus RNA quantification

SARS-CoV-2 sensitivity limbo -How low can we go?

Assessment of the real-time PCR and different digital PCR platforms for DNA quantification

Evaluation of a quantitative RT-PCR assay for the detection of the emerging coronavirus SARS-CoV-2 using a high throughput system

Transmission of 2019-nCoV infection from an asymptomatic contact in Germany

Shedding light: The importance of reverse transcription efficienc-y standards in data interpretation

ddPCR: a more accurate tool for SARS-CoV-2 detection in low viral load specimens

Differences of SARS-CoV-2 shedding duration in sputum and naso-pharyngeal swab specimens among adult inpatients with COVID-19

Limits of detection of 6 approved RT-PCR kits for the Novel SAR-S-Coronavirus-2 (SARS-CoV-2)

Coronavirus disease 2019 (COVID-19) Situation Report -39

Emergency Use Listing Procedure for in vitro diagnostics

Laboratory testing for 2019 novel coronavirus (2019-nCoV) in suspected human cases

COVID-19 Clinical management: living guidance

Status of COVID VAX

Basics of estimating measurement uncertainty

Negative Nasopharyngeal and oropharyngeal swab does not rule out COVID-19

Clinical characteristics of imported cases of COVID-19 in Jiangsu province: A multicenter descriptive study

Chest CT for typical 2019-nCoV pneumonia: relationship to negative RT-PCR testing

We thank Liqing Wu and Weifei Zhang for technical assistance.

The authors declare they have no conflict of interest.