key: cord-0950990-6201nhwx
authors: Etievant, Sibyle; Bal, Antonin; Escuret, Vanessa; Brengel-Pesce, Karen; Bouscambert, Maude; Cheynet, Valérie; Generenaz, Laurence; Oriol, Guy; Destras, Gregory; Billaud, Geneviève; Josset, Laurence; Frobert, Emilie; Morfin, Florence; Gaymard, Alexandre
title: Performance Assessment of SARS-CoV-2 PCR Assays Developed by WHO Referral Laboratories
date: 2020-06-16
journal: J Clin Med
DOI: 10.3390/jcm9061871
sha: aa48eacf61485281f7d2adde3d2f04eba117ad92
doc_id: 950990
cord_uid: 6201nhwx

A reliable diagnostic assay is crucial to early detect new COVID-19 cases and limit severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) transmission. Since the onset of the COVID-19 pandemic, the World Health Organization has published several diagnostic molecular approaches developed by referral laboratories, including Charité (Germany), HKU (Hong Kong), China CDC (China), US CDC (United States), and Institut Pasteur, Paris (France). We aimed to compare the sensitivity and specificity of these different RT-PCR assays using SARS-CoV-2 cell culture supernatants and clinical respiratory samples. Overall, the different RT-PCR assays performed well for SARS-CoV-2 detection and were all specific except the N Charité (Germany), and N2 US CDC (United States) assays. RdRp Institut Pasteur (IP2, IP4), N China CDC, and N1 US CDC were found to be the most sensitive assays. The data presented herein are of prime importance to facilitate the equipment choice of diagnostic laboratories, as well as for the development of marketed tests.

A new human coronavirus called severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) emerged in China in December 2019 [1] . SARS-CoV-2 is responsible for coronavirus disease 2019 (COVID- 19) , which was declared a pandemic on 12 March 2020 by the World Health Organization (WHO) [2] . As of 26 May 2020, 5,404,512 cases have been reported, including 343,514 deaths [3] . A reliable diagnostic assay is crucial to limit the spread of SARS-CoV-2 as early detection of new cases leads to patient isolation and contact tracing. The first SARS-CoV-2 genome was published on 10 January 2020 [4] , enabling the rapid design of a real-time reverse-transcriptase polymerase chain reaction (RT-PCR) assay by Charité (Germany) [5, 6] . This test was the first to be dispatched by WHO [7] and was widely implemented in clinical virology laboratories worldwide [8] . Since then, WHO has published [9] other approaches developed by referral laboratories, including HKU (Hong Kong) [10, 11] , China CDC (China) [12] , US CDC (United States) [13] , and Institut Pasteur, Paris (France) [14] . These assays included several RT-PCRs targeting two or three different SARS-CoV-2 gene regions, including RdRp (RNA-dependent RNA polymerase), N (nucleocapsid protein), E (envelope protein), ORF1ab nsp10 (non-structural protein 10) , and ORF1b nsp14 (non-structural protein 14) . In the present study, we aimed to compare the sensitivity and specificity of these different RT-PCR assays. Overall, the different RT-PCR assays performed well for SARS-CoV-2 detection and were all specific except N Charité, (Germany) and N2 US CDC (United States) assays. RdRp Institut Pasteur (IP2, IP4), N China CDC, and N1 US CDC were found to be the most sensitive assays.

To assess sensitivity, different RNA concentrations of SARS-CoV-2 cell culture supernatants, as well as clinical samples with different viral loads (n = 4), were tested using all RT-PCR assays. For the three most sensitive assays, limit of detection (LoD) was assessed, and additional clinical samples (n = 16) with low viral loads were also tested. To assess specificity, clinical samples negative for SARS-CoV-2 (n = 50) were tested using all RT-PCR assays. All clinical samples (nasopharyngeal aspirates) were provided by the Hospices Civils de Lyon-University Hospital, France, and frozen at 80 • C before extraction.

The sensitivity for each RT-PCR assay was first assessed using ten-fold serial dilutions from 10 −3 to 10 −9 of SARS-CoV-2 cell culture supernatants (one replicate for 10 −3 and 10 −4 , three replicates for 10 −5 and 10 −6 , and five replicates for 10 −7 to 10 −9 ). Four positive clinical samples were then tested using all RT-PCR assays to confirm these results.

For the three most sensitive assays, we estimated the LoD using a probit analysis by including five additional replicates of each dilution of the cell culture supernatants. Probit analysis consists of describing the relationship between the probability of detection and concentration using a cumulative probability curve. For each dilution, the ratio (the hit rate) is computed as the number of replicates with a detected outcome per the total number of replicates tested. These hit rates are converted mathematically into cumulative normal probability units (probits) and fitted using a regression model vs. their respective concentrations. The LoD is defined as the lowest amount of viral genome that can be detected with a 95% hit rate. For these three most sensitive assays, the results obtained with probit analysis were confirmed by additional testing of sixteen clinical samples with low viral concentration.

The specificity for each RT-PCR assay was assessed using clinical samples (n = 50) tested negative for SARS-CoV-2, including clinical samples (n = 30) tested positive for other respiratory viruses: human coronaviruses 229E, OC43, HKU1, and NL63, human influenza A and B viruses, rhinovirus, respiratory syncytial virus, parainfluenzavirus, adenovirus, metapneumovirus, and picornavirus.

Exploration of false-positive results was performed with additional negative clinical samples, water, and one additional clinical sample tested positive for each target. Amplicon size was analyzed using Agilent DNA 1000 kit (Agilent Technologies, Santa Clara, CA, USA).

Cell culture supernatants were obtained from a positive clinical sample cultivated in a biosafety level 3 laboratory on buffalo green monkey cells (cell line provided by the Université Louis Pasteur, Strasbourg, France) [15] . The SARS-CoV-2 culture had an infectious titer of 8.27 log 10 TCID 50 /mL as assessed by the Reed and Muench statistical method [16] .

RNA extraction was performed using the EMAG ® platform (bioMérieux, Marcy-l'Étoile, France), according to manufacturer's instructions.

RT-PCR was performed following published instructions [5, 6, [10] [11] [12] [13] [14] , which are summarized in Table 1 ; Table 2 . Since the China CDC protocol does not specify polymerase, thermocycler, volume of nucleic acid extract, and amplification cycles, the same instructions as those for the HKU assay were applied. RdRp IP2 and IP4 assays from Institut Pasteur, Paris (France) can be multiplexed or used in simplex [14] . Preliminary comparison on SARS-CoV-2 cell culture supernatants found that RdRp IP4 performed better when used in multiplex, whereas IP2 was not significantly impacted (Supplementary Table S1 ). The CFX 96 Touch™ Real-Time PCR (Bio-Rad, Hercules, CA, USA) was used for all RT-PCR assays. 

The E Charité and N2 US CDC assays were positive for all specimens, including negative samples and negative controls (water). These false-positive results were explored (details below), but the sensitivity of these assays was not further assessed.

Sensitivity was first assessed using SARS-CoV-2 cell culture supernatants. Using both specific SARS-CoV-2 (S) and non-specific (NS; detecting SARS-CoV-2, SARS-CoV, and bat-SARS-related CoVs) RdRp Charité assays, all replicates of the 10 −5 dilution (and inferior); 1/3 (S RdRp) and 3/3 (NS RdRp) of the 10 −6 dilution replicates; and none of the 10 −7 , 10 −8 , and 10 −9 dilutions (0/5) were detected. ORF1b and N HKU, and ORF1ab China CDC assays detected all replicates of dilutions inferior or equal to 10 −6 and detected 4/5, 3/5, and 2/5 for 10 −7 dilutions, respectively. None of these assays detected replicates of 10 −8 (0/5) and 10 −9 (0/5) dilutions. In contrast, N Charité, N China CDC, N1 and N3 US CDC, and duplex RdRp IP2/IP4 were positive for most replicates of the 10 −7 (5/5, 5/5, 5/5, 4/5, 5/5 and 5/5, respectively) and 10 −8 dilutions (3/5, 2/5, 4/5, 5/5, 3/5, 3/5, respectively; Figure 1 ). At 10 −9 dilution, N China, N1 and N3 US CDC, and duplex RdRp IP2/IP4 assays were able to detect replicates (1/5, 1/5, 2/5, 3/5, 1/5, respectively). 

Due to a limited quantity of clinical sample available, only one target from each referral laboratory providing the most sensitive assays was tested: N China CDC, N1 US CDC, and RdRp IP2. The LoD of N3 US CDC was not determined as this assay is not specific for SARS-CoV-2 detection and was removed from the new version of the US CDC assay [13] . We chose to determine the LoD for RdRp IP2 and not IP4 as IP2 detected more replicates at the 10 −9 dilution on cell culture supernatants.

The (Figure 2 ), but no significant differences (Ct difference < 2) were observed. The mean cycle threshold (Ct) values obtained for each assay were then compared for dilutions 10 −5 to 10 −8 (Figure 1, Supplementary Table S2 ). Since the accepted technical variability of RT-PCR is below 0.5 log 10 , we considered a difference of 2 Ct as significant [17] . At Clinical samples (n = 4) were then tested using all RT-PCR assays to confirm the results obtained on SARS-CoV-2 cell culture supernatants (Supplementary Table S3 ). ORF1b and N HKU, ORF1ab and N China CDC, N1 and N3 US CDC, and RdRp IP2 and RdRp IP4 assays detected all four positive samples. S and NS RdRp, and N Charité assays did not detect the positive sample with the lowest viral concentration.

Taken together, N China CDC, N1 and N3 US CDC, as well as RdRp IP2 and IP4 were the most sensitive assays.

Due to a limited quantity of clinical sample available, only one target from each referral laboratory providing the most sensitive assays was tested: N China CDC, N1 US CDC, and RdRp IP2. The LoD of N3 US CDC was not determined as this assay is not specific for SARS-CoV-2 detection and was removed from the new version of the US CDC assay [13] . We chose to determine the LoD for RdRp IP2 and not IP4 as IP2 detected more replicates at the 10 −9 dilution on cell culture supernatants.

The 95% hit rate obtained was 1.36 log 10 TCID 50 /mL [0.8; 3.09] for N China CDC, 0.44 log 10 TCID 50 /mL [0.05; 1.83] for N1 US CDC, and 0.63 log 10 TCID 50 /mL [0.25; 1.9] for RdRp IP2. The differences observed were not statistically significant. For these three assays, the results were confirmed by additional testing of clinical samples (Figure 2, Supplementary Table S4 ). N1 US CDC and RdRp IP2 had lower Ct values than N China CDC (Figure 2 ), but no significant differences (Ct difference < 2) were observed. 

No false-positive results were obtained on clinical samples that tested negative for SARS-CoV-2 and/or positive for other viruses than SARS-CoV-2, except for E Charité and N2 US CDC, which were positive for all specimens (Supplementary Table S5 ). 

No false-positive results were obtained on clinical samples that tested negative for SARS-CoV-2 and/or positive for other viruses than SARS-CoV-2, except for E Charité and N2 US CDC, which were positive for all specimens (Supplementary Table S5 ).

Since E Charité and N2 US CDC assays were positive for all specimens and replicates, including negative samples and controls, false-positive results were further explored (Supplementary Figure S1) .

For E Charité, negative samples showed two amplicons, one at 84 base pairs (bp) and one at 121 bp, whereas the positive sample only had one amplicon at 121 bp, which is close to the expected size of a specific amplification (Table 1) . Thus, the false-positive amplification obtained using E Charité might be derived from a contamination (amplicon size at 121 bp) but could also be associated with an aspecific amplification (amplicon size at 84 bp). Using the N2 US CDC assay, negative samples showed one amplicon at 73 bp, which is close to the expected size of a specific amplification (Table 1) . Thus, the false-positive amplification obtained using N2 US CDC might be due to a contamination. Sequencing of these amplicon products should be performed for further investigation.

The present study compared the performance of five RT-PCR-based methods developed by referral laboratories. N China CDC, N1 US CDC, and RdRp IP2 and IP4 were found to be the most sensitive assays on SARS-CoV-2 cell culture supernatants and clinical respiratory samples. Vogels et al. compared the performance of SARS-CoV-2 PCR assays developed by the same referral laboratories, except those from Institut Pasteur. Using RNA-spiked mock samples, they found that ORF HKU was one of the most sensitive assays [18] . Herein, ORF HKU was more sensitive than RdRp Charité but slightly less sensitive than other assays, such as N1 US CDC or N China. Although RdRp Charité performed well for the lowest dilutions, it was nevertheless found to be less sensitive than others, a result in line with those of Vogels et al. [18] . It is worth noting that the Charité assay was the first to be published at the early stage of the pandemic [9] and has been widely used worldwide [8] . This assay was initially designed for the diagnosis of SARS-related CoVs and then optimized for SARS-CoV-2 detection [5] . Thanks to this assay, an important number of COVID-19 diagnoses were made, which contributed to limiting the spread of the outbreak. In line with the present results, it was reported that RdRp IP2 and RdRp IP4 sensitivity was similar when used in multiplex [14] , suggesting that the Institut Pasteur assay should preferentially be used in multiplex. Of note, we did not apply the Ct cut-off values above, in which a sample would be considered negative, since such values were not provided in the protocols made available by the referral laboratories.

As previously reported [19] , we identified probable primer contamination using N2 US CDC and E Charité, which prevented us from further evaluating their sensitivity and specificity. Except for these two assays, no false-positive results were observed when testing a wide range of respiratory viruses, a result consistent with previous studies [5, 6, 10, 11, 13, 14] . Although not observed herein, the amplification of non-specific products for ORF1 and N China CDC, and N2 and N3 US CDC has also been reported [18] .

The performance of other RT-PCR tests recently developed [7] should be explored in further studies. In addition, the quantification of SARS-CoV-2 could be performed to assess the effectiveness of potential treatments. The data presented herein are of prime importance to facilitate the equipment choice of all diagnostic laboratories, as well as for the development of marketed tests. Sensitive tests should be widely implemented to limit the spread of the current outbreak and prepare for the post-epidemic phase and future seasonal epidemics.

The following are available online at http://www.mdpi.com/2077-0383/9/6/1871/s1. Figure S1 : Electropherograms of amplicon sizes obtained using Agilent DNA 1000 kit (Agilent Technologies) for one positive sample and one negative sample for E Charité (Germany) and N2 US CDC (United States), respectively. Table S1 : Cycle threshold values and mean ± standard deviation for serial dilutions of SARS-CoV-2 cell culture supernatants obtained using IP2 and IP4 Institut Pasteur, Paris (France) when used in simplex and multiplex. Table S2 : Cycle threshold values and mean ± standard deviation for serial dilutions of SARS-CoV-2 cell culture supernatants obtained using five RT-PCR assays for SARS-CoV-2 detection made available by WHO, Table S3 : Cycle thresholds values obtained on clinical samples (n = 4) using five RT-PCR assays for SARS-CoV-2 detection made available by WHO, Table S4 : Cycle threshold values obtained on clinical samples (n = 16) using the three most sensitive methods: N China CDC, N1 US CDC, and IP2 Institut Pasteur, Paris (France). Table S5 : Specificity assessment of five RT-PCR assays made available by WHO. 

Coronaviridae Study Group of the International Committee on Taxonomy of Viruses The species Severe acute respiratory syndrome related coronavirus: Classifying 2019-nCoV and naming it SARS-CoV-2

World Health Organization WHO Announces COVID-19 Outbreak a Pandemic

COVID-19)-Situation Report-127

Diagnostic Detection of 2019-nCoV by Real-Time RT-PCR

Coronavirus and the Race to Distribute Reliable Diagnostics

Laboratory readiness and response for novel coronavirus (2019-nCoV) in expert laboratories in 30 EU/EEA countries

Coronavirus Disease (COVID-19) Technical Guidance: Laboratory Testing for 2019-nCoV in Humans

Molecular diagnosis of a novel Coronavirus (2019-nCoV) causing an outbreak of pneumonia

Detection of 2019 Novel Coronavirus (2019-nCoV) in Suspected Human Cases by RT-PCR

China CDC Primers and Probes for Detection 2019-nCoV

Centers for Disease Control and Prevention. A CDC 2019-Novel Coronavirus (2019-nCoV) Real-Time RT-PCR Diagnostic Panel

Protocol: Real-Time RT-PCR Assays for the Detection of SARS-CoV-2

Characteristics of the BGM line of cells from African green monkey kidney

A Simple Method of Estimating Fifty Per Cent Endpoints

SFM comparaison de methodes

Analytical sensitivity and efficiency comparisons of SARS-COV-2 qRT-PCR assays

Early release-Delayed laboratory response to COVID-19 caused by molecular diagnostic contamination

We would like to thank all the patients, clinicians, laboratory technicians, and informatics department who contributed to this investigation. We are also grateful to Véréna Landel and Philip Robinson (DRCI, Hospices Civils de Lyon) for help in the manuscript preparation. We also gratefully acknowledge the authors and laboratories that originated and submitted the sequence from GISAID's used to calculate amplicon sizes.