key: cord-1018015-esuftqmx
authors: Mannonen, L.; Kallio-Kokko, H.; Loginov, R.; Jaaskelainen, A.; Jokela, P.; Antikainen, J.; Vare, P.; Kekalainen, E.; Kurkela, S.; Jarva, H.; Lappalainen, M.
title: Comparison of two commercial platforms and a laboratory developed test for detection of SARS-CoV-2 RNA
date: 2020-07-06
journal: nan
DOI: 10.1101/2020.07.03.20144758
sha: 804b3849f49f81c261bd5d738a47d2766ee87fd9
doc_id: 1018015
cord_uid: esuftqmx

Mitigation of the ongoing COVID-19 pandemic requires reliable and accessible laboratory diagnostic services. We evaluated the performance of one LDT and two commercial tests, cobas(R) SARS-CoV-2 (Roche) and Amplidiag(R) COVID-19 (Mobidiag), for the detection of SARS-CoV-2 RNA in respiratory specimens. 183 specimens collected from suspected COVID-19 patients were studied with all three methods to compare their performance. In relation to the reference standard, which was established as the result obtained by two of the three studied methods, the positive percent agreement (PPA) was highest for cobas(R) test (100%), followed by Amplidiag(R) test and the LDT (98.9%). The negative percent agreement (NPA) was lowest for cobas(R) test (89.4%), followed by Amplidiag(R) test (98.8%) and the highest value was obtained for LDT (100%). The dilution series conducted for specimens, however, suggests significantly higher sensitivity for the cobas(R) assay in comparison with the other two assays and the low NPA value may be due to the same reason. In general, all tested assays performed adequately. Both the time from sample to result and hands-on time per sample were shortest for cobas(R) test. Clinical laboratories need to be prepared for uninterrupted high-throughput testing during the coming months in mitigation of the pandemic. To secure that, it is of critical importance for clinical laboratories to maintain several simultaneous platforms in their SARS-CoV-2 nucleic acid testing.

Mitigation of the ongoing COVID-19 pandemic requires reliable and accessible laboratory diagnostic services. The specific diagnosis of SARS-CoV-2 infection relies on molecular methods, especially on RT-PCR, although other technologies including serologic immunoassays are emerging 1,2 . The first methods for SARS-CoV-2 detection were laboratory developed RT-PCR tests (LDTs), one of the first methods published was described by Corman et al. 3 . This method was later endorsed by WHO and widely implemented in clinical laboratories. Roche Molecular Systems (Branchburg, NJ, USA) cobas® SARS-Cov-2 test was the first commercial test to get EUA from FDA on March 12 2020. Since then (as of June 25, 2020) more than 100 commercial molecular IVD-tests have been granted the FDA or other national authorities' EUA and/or CE-mark 4 . Both LDTs and commercial tests have been set up in a high time pressure. Therefore, it is of great importance to evaluate the tests in clinical laboratory settings. In this study we evaluated the performance of one LDT and two commercial tests, namely cobas® SARS-CoV-2 (Roche) and recently CE/IVD marked Amplidiag® COVID-19 (Mobidiag, Espoo, Finland), for the detection of SARS-CoV-2 RNA in respiratory specimens.

Altogether 237 respiratory tract specimens referred to Helsinki University Hospital Laboratory (HUS Diagnostic Center, HUSLAB), Department of Virology and Immunology, Finland were included in this study. 54 specimens collected from patients with respiratory symptoms in 2018-2020 were used to verify analytical specificity of the tests. The specimens from 2020 were negative for SARS-CoV-2 RNA by the LDT. 183 specimens collected from suspected COVID-19 patients in 2020 were studied with all three methods to compare the performance of the tests in detecting SARS-CoV-2 RNA. Although most of the specimens tested were nasopharyngeal swabs, also oropharyngeal and nasal swabs were included due to the global shortage of the swab sticks needed for nasopharyngeal sampling. The specimens were collected either in Copan UTM (Copan, Brescia, Italy) or in tubes containing 0.9% saline due to the global shortage of Copan UTM tubes. The suitability of the 0.9% saline tubes as alternative to viral transport media for SARS-CoV-2 testing has been shown before 5 .

The specimens (n=183) deployed to compare the performance of the three studied RT-PCR methods comprised of two sets of specimens collected within two time frames. The first set of specimens (n=37) was part of the material used for the initial verification of cobas® SARS-CoV-2/Amplidiag® COVID-19 tests. These specimens were collected between March 5 and March 18, 2020, and the SARS-CoV-2 positive specimens (n=18) represent virus strains from the early epidemic. The second set of specimens (n=146) were collected between May 4 and May 8 2020, and the positive specimens (n=90) thus represent strains from a declining phase of the epidemic in Finland. Specimens were stored at -20 °C/-70 °C after initial analysis and were thawn upon analysis. Specimens were not thawn more than twice before RT-PCR.

In addition, QCMD 2020 Coronavirus Outbreak Preparedness EQA Pilot Study proficiency samples (Glasgow, Scotland, UK) were used to evaluate the performance of the methods.

Inactivation and lysing of the specimens All specimens were inactivated in a biosafety cabinet in a biosafety level 2 laboratory that has a negative pressure. At the beginning of the epidemic FFP3 mask, protective glasses and protective clothing were worn during working in the laboratory. Later in the epidemic visor and surgical mask replaced the FFP3 masks and protective glasses.

The possible SARS-CoV-2 in the specimen was inactivated by adding 250 µl of MagNA Pure Lysis/Binding Buffer (Roche Diagnostics GmbH, Mannheim, Germany) to 250 µl of patient specimen. Lysates were incubated for a minimum of 10 minutes before processed further.

If needed, the specimens were first equilibrated to room temperature, after which the possible SARS-CoV-2 in the specimen was inactivated by adding 350 µl of MagNA Pure Lysis/Binding Buffer (Roche Diagnostics GmbH, Mannheim, Germany) to 350 µl of patient specimen. Lysates were incubated for a minimum of 10 minutes before processed further.

The possible SARS-CoV-2 in the specimen was inactivated either by adding 600 µl of the specimen to an eNAT-tube (Copan) or 360 µl to an mNAT-tube (Mobidiag). eNAT tubes were incubated for a minimum of 30 minutes and mNAT-tubes for a minimum of 5 minutes before processed further.

The real-time LDT SARS-CoV-2 RT-PCR used in this study is a modification of the method published by Corman et al. 3 . The test is suitable for detection of SARS-CoV-2 RNA from sputum, nasopharyngeal/tracheal aspirates, nasal, nasopharyngeal and oropharyngeal swab specimens and faeces. Initially, all target genes (E, RdRP and N) were included in the diagnostic assay. In addition, a PCR for beta-globin gene 6 

The cobas® SARS-CoV-2 test (Roche Molecular Systems) is a qualitative test for fully automated cobas® 6800/8800 platforms. The test is validated to detect SARS-CoV-2 RNA from nasal, nasopharyngeal and oropharyngeal swab specimens. The test amplifies two SARS-CoV-2 targets: orf1ab (Target 1), which is specific for SARS-CoV-2 and a conserved region of the E-gene (Target 2), which is pan-Sarbeco specific and detects also SARS-CoV and other Sarbecoviruses currently unknown to infect humans. In addition, the test includes an internal RNA-control (IC), which is added to the specimens before extraction. The test includes also positive and negative controls, which are processed the same way as the samples.

8 600 µl of specimen lysate was subjected to cobas® 6800 system. Testing was performed according to the manufacturer's instructions apart from the inactivation step.

The Amplidiag® COVID-19 test (Mobidiag) is a qualitative test for the detection of SARS-CoV-2 RNA in nasopharyngeal specimens. The test amplifies two SARS-CoV-2 targets: orf1ab and N-gene, which are both specific for SARS-CoV-2. In addition, the test amplifies the human RNase P gene (RP), which serves as a sampling control. The test includes positive and negative control.

eNAT or mNAT tubes were subjected to Amplidiag® Easy system, which extracts nucleic acids and does the PCR setup. The PCR plate was then transferred to a real time PCR-machine (CFX-96, Bio-Rad, Hercules, CA, USA), which contains the Amplidiag® Analyzer software. The Amplidiag® Analyzer software transfers directly data from Amplidiag® Easy system and automates the result interpretation. Testing was performed according to the manufacturer's instructions, apart from the specimen type (validated only for nasopharyngeal specimens) and inactivation step with eNAT-tubes. Verification of the performance of all three studied methods on specimens collected to Copan UTM versus 0.9 % saline and using eNAT versus mNAT in the lysing/inactivation step with Amplidiag COVID-19 test When the epidemic started, the specimens were collected to Copan UTM tubes (3 ml), but later to tubes containing 1.5 ml of 0.9% saline, because of the global shortage in supply of the Copan tubes. In order to verify that the used methods were performing at the same level two independent dilution series were constructed: (i). A patient specimen known to contain SARS-CoV-2 RNA originally collected to Copan tube was diluted to pooled Copan-collected negative specimen. (ii). A patient specimen known to contain SARS-CoV-2 RNA originally collected to 0.9% saline was diluted to pooled 0.9% saline-collected negative specimen. The ct-value for SARS-CoV-2 detection by RT-PCR was around 24 for both the Copan-collected and saline-collected original positive specimens. A dilution series of 1:10 -1:100000 for both specimens (in Copan UTM or saline) in respective specimen pools were constructed and analyzed with all three studied methods.

Due to the shortage of the eNAT tubes a change to mNAT tubes was necessary.

This change was evaluated with a small number of specimens. Eleven specimens defined positive by the cobas® test were inactivated/lysed in parallel by pipetting the specimen to an mNAT and eNAT tube and analyzed in the same PCR run.

The analytical specificity of the tests was studied by analyzing patient specimens containing other respiratory viruses. Altogether 54 specimens were included, but not all samples were run with all three methods. Some of the specimens contained several viruses (n=11). In four specimens, the presence of a respiratory virus nucleic acid was defined by xTAG RVP Fast (Luminex Diagnostics, Toronto, Canada) assay, and in 50 specimens by the Allplex Respiratory Panel 1/2/3 (Seegene, Seoul, Republic of Korea). The specimens were chosen to contain moderate to high concentrations of viral nucleic acids (ct-value <30 for specimens by Seegene test or high mean fluorescence intensity for Luminex test). The specimens were collected during 2018-2020 representing recent virus strains in Finland. The specimens are described in Table 1 .

Statistical methods sided 95% confidence intervals (CI) were calculated with an on-line MEDCALC® tool (https://www.medcalc.org/calc/diagnostic_test.php). The overall agreement of the evaluated assays was evaluated by the kappa value, which was calculated (including two-sided 95% CI) with an on-line QuickCalcs tool (https://www.graphpad.com/quickcalcs/kappa1/). Kappa values were interpreted as follows: No agreement (< 0), slight agreement (0-0.20), fair agreement (0.21-0.40), moderate agreement (0.41-0.60), substantial agreement (0.61-0.80), and almost perfect agreement (0.81-1). All values were calculated relative to the reference standard, which was established as the result obtained by two of the three studied methods.

The significance of the difference of the ct values obtained by Amplidiag® test from eNAT vs mNAT tubes was estimated by two tailed students T-test. P-value < 0.05 was considered significant.

Clinical performance of the evaluated assays Altogether, 183 specimens were analyzed by all three evaluated methods. One specimen gave an invalid result by cobas® SARS-Cov-2 test (0.5%), two specimens had an incorrect amount of specimen lysate in LDT and were therefore considered invalid (1.1%), and 10 specimens gave failed results by Amplidiag® COVID-19 test (5.5%). The failed/invalid specimens were excluded from the agreement analysis. In relation to the reference standard the positive percent agreement (PPA) was highest for cobas® test (100%), followed by Amplidiag® test (98.9%) and the LDT (98.9%). Negative percent agreement (NPA) was lowest for cobas® test (89.4%), followed by Amplidiag® test (98.8%) and the best specificity value was obtained for LDT (100%). The overall agreement as defined by kappa-value was excellent for all studied tests ( (Table 3) .

Performance comparison of the evaluated methods on specimens collected to Copan UTM versus 0.9 % saline Two independent dilution series of two positive specimens were constructed to 0.9% saline and copan UTM in respiratory specimen matrix to verify that specimens collected to Copan and saline perform at the same level with the evaluated methods. The results indicate similar performance of all three methods independent of the collection media used (Table 4 and Figure 1 ). However, the data suggest higher overall sensitivity for the cobas® test as the (Table   5 ). However, the test identified correctly another sample containing SARS-CoV-2 RNA 2.3 log10 copies/ml. Amplidiag® test reported four samples as failed because the EQA samples lack human RNase P target needed for valid negative results with the test.

None of the studied SARS-CoV-2 tests gave positive result from patient specimens containing other respiratory viruses (Table 1) .

Altogether, there were 12 discrepant results. In one specimen, which only gave a positive result with Amplidiag® the positive signal was obtained for the orf1ab target with ct value of 40.8. However, in the amplification curve provided by Amplidiag® analyzer software no specific amplification could be seen. Instead of the expected logarithmic rise in the fluorescence signal, only a slow rise in background signal was seen (data not shown). For 1 specimen a positive result was obtained with cobas® and Amplidiag® tests, but a negative result with LDT. In one specimen positive results were obtained for cobas® test and LDT, while the Amplidiag® test gave a negative result. In that case the specimen had gone through 2 freeze-thaw cycles before it was analyzed by Amplidiag® test, but only one or none when analyzed by cobas® or LTD tests, respectively. In 9 specimens a positive result was obtained only by cobas® test. However, positive result was obtained for two specimens, when rerun with the new Amplidiag® COVID-19 software version. In all 9 specimens positive signal was obtained for both amplification targets of the test, but the ct values were high, >30 for 8/9 specimens. (Table 6A) . (Table 6B )

In general, all tested assays performed adequately. Both the time from sample to result and hands-on time per sample were shortest for cobas® test. The cobas® and Amplidiag® assays allow for an automatic transfer of the results to the laboratory information system. On the other hand, both LDT and Amplidag® allow for the evaluation of the actual amplification curves, which is very important for the quality assurance of the results, especially when the tests have been set up in a straining timetable.

Of the three RT-PCR methods evaluated in this study, cobas® SARS-CoV-2 test showed the best overall performance for the detection of SARS-CoV-2 RNA in clinical respiratory specimens. In the absence of a gold standard, the reference value was created as a consensus of two methods. The agreement between the different methods tested was excellent. However, the positive percent agreement was highest for cobas® SARS-CoV-2 test (100%), while its negative percent agreement was the lowest (89.4%). All the specimens positive only with cobas® SARS-Cov-2 test had high ct-values (>30), with one exception, where ct-values of 28.6 (orf1) and 29.1 (E-gene) were obtained. The dilution series conducted for specimens collected in Copan UTM vs. 0.9% saline, however, suggests significantly higher sensitivity for the cobas® assay in comparison with the other two assays and the low NPA value may well be due to the same reason. Previous studies also suggest high sensitivity for the cobas® SARS-CoV-2 test 8, 9 . The second commercial test evaluated, the Amplidiag® COVID-19 test obtained same PPA (98.9%) as LDT, but lower NPA (98.8% vs. 100% respectively). There was one specimen that was positive by Amplidiag® COVID-19 test only, which seemed to be a false positive by interpretation of the amplification curve of the positive target (orf1ab). This specimen was negative by the new Amplidiag® software version, which did not give any false interpretation of negative results in this study. However, it seems that what is gained in specificity is lost in sensitivity: With the new software version three specimens positive by the old version gave negative results, even though specific amplification from the orf1ab target could be seen. All things considered, Amplidiag® test seems to perform on the same par with LDT, apart from the relatively high failure rate with Amplidiag® test (5.5%). The comparison of results from eNAT tubes versus mNAT tubes suggests that the Amplidiag® test is for some reason less sensitive (later ct value) for the sampling control (RNase P gene) from mNAT tubes. The difference in obtained ct values was greater for RNase P gene (~2 ct cycles) than for the specific virus targets (<1 ct cycle). This was also observed in the rising failure percent (from 3.8 % to 8.4%, data not shown) when the change to mNAT tubes was implemented.

Study with all the evaluated methods available in our laboratory. The cobas® test and LDT identified all samples correctly. Amplidiag® missed repeatedly one specimen with dPCR reference value of 3.3 log10 copies/ml, although it correctly identified another sample with dPCR reference value of 2.3 log10 copies/ml. In addition, Amplidiag® test yielded a failed test result for 4 samples, as these samples lacked human RNase P target needed for a valid negative result with the test. The lack of correct matrix in EQA samples is a major drawback in general. The performance of the test identifying the target virus may be very different in the correct matrix in comparison to virus alone. Human nucleic acids, if not yielding a false positive result in a PCR, may take force from the amplification reaction (e.g. if primers and/or probe have binding sites in human genome or transcriptome). For example, the LDT amplifies virus positive specimens without human matrix more efficiently than specimens with human matrix included (data not shown).

It is common that RNA-viruses accumulate mutations at high frequency since RNA-polymerase lacks proofreading activity. Indeed, there is evidence that also SARS-CoV-2 is evolving during time and mutations may occur in the target area of molecular tests used.

A previous report suggests that a mutation has already occurred in the target region of the cobas® SARS-CoV-2 E-gene test 10 . In this study, all specimens gave a positive signal for both target genes by cobas® test. Fortunately, the test uses a dual target approach, which means that the mutation proposed to be in the target area of the cobas® E-gene test does not compromise correct results. This highlights the importance of a dual target approach 11-13 . Many countries are now (in July 2020) moving from lock-down to "test, trace and isolate" strategy in their COVID-19 mitigation, which requires a large capacity for sensitive and reliable SARS-CoV-2 laboratory testing. Clinical laboratories need to be prepared for uninterrupted high-throughput testing during the coming months. In order to achieve this, laboratories need to deploy optional testing methods for the event of breakage in instrumentation, as well as prepare for the ongoing global shortage in testing supplies.

Therefore, it is of critical importance for clinical laboratories to maintain several simultaneous platforms in their SARS-CoV-2 nucleic acid testing, and continuously monitor the performance of the assays used. Table 2 The positive percent agreement (PPA), negative percent agreement (NPA) and concordance of the evaluated assays relative to the reference values ("consensus result" was used as a reference value and was defined as the result obtained by at least two of the three studied methods). The specimen had gone through two freezing-thawing cycles when anal yzed by Amplidiag® and only one with cobas® test and non with LDT 

Diagnostic Testing for Severe Acute Respiratory Syndrome-Related Coronavirus-2: A Narrative Review

Evaluation of commercial and automated SARS-CoV-2 IgG and IgA ELISAs using coronavirus disease (COVID-19) patient samples

Detection of 2019 novel coronavirus (2019-nCoV) by real-time RT-PCR

Kalorama Information, COVID-19 IVD Test Tracker

Evaluation of Saline, Phosphate-Buffered Saline, and Minimum Essential Medium as Potential Alternatives to Viral Transport Media for SARS-CoV-2 Testing

Development of a multiplex real-time PCR assay for detection of Mycoplasma pneumoniae, Chlamydia pneumoniae and mutations associated with macrolide resistance in Mycoplasma pneumoniae from respiratory clinical specimens

Rapid establishment of laboratory diagnostics for the novel coronavirus SARS-CoV-2 in Bavaria

Clinical evaluation of the cobas SARS-CoV-2 test and a diagnostic platform switch during 48 hours in the midst of the COVID-19 pandemic

Comparison of SARS-CoV-2 detection from nasopharyngeal swab samples by the Roche cobas 6800 SARS-CoV-2 test and a laboratory-developed real-time RT-PCR test

Failure of the cobas® SARS-CoV-2 (Roche) E-gene assay is associated with a C-to-T transition at position 26340 of the SARS-CoV-2 genome medRxiv preprint doi

The Finnish New Variant of Chlamydia trachomatis with a Single Nucleotide Polymorphism in the 23S rRNA Target Escapes Detection by the Aptima Combo 2

Comparison of two quantitative real-time CMV-PCR tests calibrated against the 1st WHO international standard for viral load monitoring of renal transplant patients

Comparison of a UL111a realtime PCR and pp65 antigenemia for the detection of cytomegalovirus

The laboratory staff doing the SARS-CoV-2 diagnostics at HUSLAB are thanked for the extra work they have done for this evaluation during the straining times.

Number of analyzed specimens