key: cord-0797927-b0qtzzac
authors: Zhen, Wei; Manji, Ryhana; Smith, Elizabeth; Berry, Gregory J.
title: Comparison of Four Molecular In Vitro Diagnostic Assays for the Detection of SARS-CoV-2 in Nasopharyngeal Specimens
date: 2020-07-23
journal: J Clin Microbiol
DOI: 10.1128/jcm.00743-20
sha: 23643abfb6f70046ee76ef5fadc17e8828eb4da4
doc_id: 797927
cord_uid: b0qtzzac

Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), the novel human coronavirus that causes coronavirus disease 2019 (COVID-19), was first discovered in December 2019 as the cause of an outbreak of pneumonia in the city of Wuhan, Hubei province, China. The clinical presentation of COVID-19 is fairly nonspecific, and symptoms overlap those of other seasonal respiratory infections concurrently circulating in the population. Furthermore, it is estimated that up to 80% of infected individuals experience mild symptoms or are asymptomatic, confounding efforts to reliably diagnose COVID-19 empirically. To support infection control measures, there is an urgent need for rapid and accurate molecular diagnostics to identify COVID-19-positive patients. In the present study, we evaluated the analytical sensitivity and clinical performance of the following four SARS-CoV-2 molecular diagnostic assays granted emergency use authorization by the FDA using nasopharyngeal swabs from symptomatic patients: the New York SARS-CoV-2 Real-time Reverse Transcriptase (RT)-PCR Diagnostic Panel (modified CDC) assay, the Simplexa COVID-19 Direct (Diasorin Molecular) assay, GenMark ePlex SARS-CoV-2 (GenMark) assay, and the Hologic Panther Fusion SARS-CoV-2 (Hologic) assay. This information is crucial for both laboratories and clinical teams as decisions on which testing platform to implement are made.

by coronaviruses can therefore typically be transmitted through several different routes, including the aerosol and fecal-to-oral routes, with fomites often playing an important role in the infection cycle (5) . Notably, coronaviruses possess a distinctive morphological feature, consisting of a ring of spike proteins on the outer surface of the virus, giving the appearance of a halo or corona. In addition to inspiring the name of the coronavirus genus, the spike proteins are also essential for infection of host cells. The SARS-CoV-2 spike protein recognizes and binds to human cellular receptor angiotensin-converting enzyme 2 (ACE2) and subsequently mediates fusion of the viral and host cell membranes, allowing the virus to gain entry (6, 7) . The ACE2 receptor is found on epithelial cells of the lungs and small intestines (8) .

Infection with SARS-CoV-2 can cause mild to severe respiratory illness, and symptoms include fever, cough, and shortness of breath. However, some populations experience severe, rapidly progressive, and fulminant disease. Among these populations are older adults and people who have serious underlying medical conditions (e.g., heart disease, diabetes, lung disease, and immunosuppression) (9) . Unfortunately, many elements, some intrinsic to the virus and others seasonal, have lessened the effectiveness of traditional infection control measures. The combination of high rates of human-to-human transmission (R 0 ϭ 2.0 to 2.5), stability of the virus in aerosols and on surfaces, the fairly nonspecific clinical presentation of COVID-19, and coincidence of the onset of COVID-19 with the active season of other respiratory viruses (e.g., influenza virus and respiratory syncytial virus [RSV] ) in many parts of the world presents a major challenge to efforts to stop the pandemic from spiraling into a more severe global health emergency (9, 10) .

During the early stages of the epidemic, both national and international agencies rushed to initiate the process of mass production of test reagents and issued an emergency use authorization (EUA) for the U.S. CDC COVID-19 real-time reverse transcriptase (RT)-PCR assay (11) . Despite the collective efforts, laboratories are still facing reagent supply shortages, lack of instrument access, an inability to perform highcomplexity testing, and increased staffing needs, leaving a gap in the ability of health care providers to rapidly diagnose and manage patients. The need to implement a sensitive, accessible, and rapid diagnostic test for the detection of COVID-19 is clear. In this study, we evaluated the analytical and clinical performance of four SARS-CoV-2 molecular diagnostic assays granted EUA by the FDA, including the modified CDC, DiaSorin Molecular, GenMark, and Hologic assays. These assays are authorized for the qualitative detection of SARS-CoV-2 in clinical specimens obtained from symptomatic patients and were evaluated using nasopharyngeal swab specimens.

Specimen collection and storage. Sterile nylon, Dacron, or rayon swabs with flexible plastic shafts were used to collect nasopharyngeal specimens (NPS) from symptomatic patients. After collection, swabs were placed into 3 ml of sterile universal transport medium (UTM; various manufacturers). Specimens were transported and tested as soon as possible after collection. Before testing, samples were subjected to vortex mixing for 3 to 5 s and a calibrated pipette was used to transfer the specimen volume specified by each manufacturer's instructions for use. The samples were kept for up to 72 h at 2 to 8°C after collection before initial testing. Following routine testing, samples were aliquoted and stored in a freezer at Ϫ80°C until comparator testing.

The New York SARS-CoV-2 Real-time Reverse Transcriptase (RT)-PCR Diagnostic EUA Panel (modified CDC assay). This assay is a modified version of the CDC 2019-Novel Coronavirus (2019-nCoV) Real-Time RT-PCR Diagnostic EUA Panel validated by the Wadsworth Center (Albany, NY) using the same primer and probe sets as the CDC assay for nucleocapsid gene 1 (N1) and N2 targets and the human RNase P gene (RP) but excluding the N3 primer and probe set. A 110-l volume of patient specimen was extracted by the use of a NucliSENS easyMAG platform (bioMérieux, Durham, NC) according to manufacturer's instructions, with a nucleic acid elution volume of 110 l. For each specimen, three Master Mix sets that included N1, N2, and RNase P were prepared, and 15 l of each master mix was dispensed into appropriate wells, followed by 5 l of extracted sample. Each run also included a no-template control (NTC), a negative extraction control, and a SARS-CoV-2-positive control. Amplification was performed on an Applied Biosystems 7500 Fast Dx real-time PCR system (Thermo Fisher Scientific, Waltham, MA). The results interpretation algorithm for reporting a positive specimen requires both N1 and N2 targets to be detected.

Simplexa COVID-19 Direct EUA (Diasorin Molecular LLC, Cypress, CA). Testing with the DiaSorin Molecular Simplexa COVID-19 Direct EUA assay was performed according to the manufacturer's instruc-tions for use. A 50-l volume of Simplexa COVID-19 Direct kit reaction mix (MOL4150) was added to the "R" well of the 8-well direct amplification disc (DAD) followed by addition of 50 l of nonextracted nasopharyngeal swab (NPS) sample to the "SAMPLE" well. Data collection and analysis were performed with LIAISON MDX Studio software. The assay targets two different regions of the SARS-CoV-2 genome, including the surface (S) gene and open reading frame 1ab (ORF1ab), differentiated with 6-carboxyfluorescein (FAM) and JOE fluorescent probes. An RNA internal control (Q670 probe) is used to detect RT-PCR failure and/or inhibition. The results interpretation algorithm for reporting a positive specimen requires only one of the two targets to be detected (S or ORF1ab gene).

GenMark ePlex SARS-CoV-2 assay EUA (GenMark Diagnostics, Carlsbad, CA). Testing with the ePlex SARS-CoV-2 was performed according to the manufacturer's instructions for use. Briefly, after vortex mixing was performed for 3 to 5 s, 200 l of the primary NPS sample was aspirated into the sample delivery device (SDD) provided with the ePlex SARS-CoV-2 panel kit and subjected to vortex mixing once again for 10 s. The entire volume of the SDD was dispensed into the sample loading port of the SARS-CoV-2 test cartridge, followed by firmly pushing down the cap to securely seal the sample delivery port. Each cartridge was bar-coded and scanned with the ePlex instrument and inserted into an available bay. Upon the completion of the assay run, the ePlex SARS-CoV-2 report was generated. The GenMark ePlex SARS-CoV-2 assay amplifies and detects the SARS-CoV-2 virus nucleocapsid (N) gene.

Hologic Panther Fusion SARS-CoV-2 EUA (Hologic Inc., San Diego, CA). The Fusion SARS-CoV-2 assay was performed according to the manufacturer's instructions for use. A 500-l volume of NPS was lysed by transfer to a specimen lysis tube containing 710 l lysis buffer. The input volume per sample for extraction is 360 l. Internal control S (IC-S) was added to each test specimen and to the controls via working Panther Fusion capture reagent S (wFCR-S). Hybridized nucleic acid was then separated from the specimen in a magnetic field. After wash steps, 50 l of purified RNA was eluted. Then 5 l of eluted nucleic acid was transferred to a Panther Fusion reaction tube already containing oil and reconstituted master mix. The Panther Fusion SARS-CoV-2 assay amplifies and detects two conserved regions of the ORF1ab gene in the same fluorescence channel. The two regions are not differentiated, and amplification of either or both regions results in a fluorescent ROX signal. The results interpretation algorithm for reporting a positive specimen requires only one of the two targets to be detected (the ORF1a or ORF1b gene).

Analytical sensitivity. Limit of detection (LoD) determinations were performed using a SARS-CoV-2 synthetic RNA quantified control (SARS-CoV-2 Standard) containing five gene targets (E, N, ORF1ab, RNA-dependent RNA polymerase [RdRP], and S genes of SARS-CoV-2) from Exact Diagnostics (catalog no. COV019; Exact Diagnostics, Fort Worth, TX). A starting concentration of 200,000 copies/ml control was used to generate a dilution panel. The control was diluted in Ambion RNA storage solution (catalog no. AM7001; Thermo Fisher Scientific, Waltham, MA) and aliquoted for testing in order to obtain a maximum of 12 replicates at the following concentrations: 20,000, 2,000, 1,000, 500, 100, 50, and 5 copies/ml. The LoD was determined by two methods: positive-rate analysis and Probit analysis. A positive rate was defined as the lowest dilution at which all replicates gave positive test results with a 100% detection rate. The LoD by Probit was defined as the lowest detectable dilution at which the synthetic RNA quantified control (measured in numbers of copies per milliliter) gave a positive result with a 95% probability of detection. The final LoD was based on results of Probit analyses and on each manufacturer's claimed results interpretation algorithm, which determines whether a specimen result is positive, negative, or inconclusive on the basis of the gene targets detected.

Study design. A total of 104 nasopharyngeal specimens (88 retrospective and 16 prospective) originally submitted during March and April of 2020 to Northwell Health Laboratories for routine COVID-19 testing on the GenMark were selected for this study. Of the 104 specimens analyzed, 51 of the samples were positive and 53 were negative. Retrospective frozen samples were thawed, pipetted into separate aliquots, and immediately tested by the modified CDC, DiaSorin Molecular, and Hologic assays in parallel. Prospective specimens were performed fresh on each platform in parallel at the time of patient testing. The study population included patients of all ages and both genders presenting with signs and/or symptoms of COVID-19 infection. Specimens selected for this study included positive specimens spanning the range of positivity and also specimens with low viral loads (characterized by high cycle threshold [C T ] values). In addition, specimens were selected to represent the true positivity rate determined by us at the time this study was performed (50% to 60%). The manufacturer's specifications are summarized in Table 1 .

Discordant analysis. Results were considered discordant when one molecular assay did not agree qualitatively (detected or not detected) with the other three assay results. In such cases, molecular testing was repeated for the discordant assay.

Workflow evaluation. Workflow was evaluated by the use of a stopwatch to measure the amount of time needed for each step being evaluated, including hands-on time (HoT), assay run time, and total 

Analytical sensitivity. A serial dilution of SARS-CoV-2 control panel was tested to determine the LoD, defined as the minimum concentration with detection values of 100% by positive rate and 95% by Probit analysis. The LoD established by percent positive rate ranged from 1,000 copies/ml by both the GenMark and the modified CDC assays to 50 copies/ml by the DiaSorin Molecular assay ( Table 2 ). The LoD results were further subjected to Probit analysis. The 95% detection limit values for the CDC assay were 779 Ϯ 27 copies/ml for the N1 gene and 356 Ϯ 20 copies/ml for the N2 gene. For the DiaSorin Molecular assay, the 95% detection limit values were 39 Ϯ 23 copies/ml for the S gene and 602 Ϯ 28 copies/ml for ORF1ab. For the Hologic assay, the 95% detection limit value was 83 Ϯ 36 copies/ml for ORF1ab. Probit analysis could not be performed for the GenMark assay ( Table 2 ). The final LoD values, according to the assay results interpretation algorithm from each manufacturer, ranged from 1,000 copies/ml by the GenMark assay to 39 Ϯ 23 copies/ml by the DiaSorin Molecular assay ( Table 3) .

Details of discordant sample analysis results are shown in Table 4 . A total of five discordant samples were found among three of the four platforms. One false-positive sample (sample A) had C T values of 38.9 and 39.6 the for N1 and N2 genes, respectively, on initial testing by the modified CDC assay. After repeating extraction and retesting, the sample was determined to be negative. Two samples (sample B and sample C) were considered false negative by GenMark but positive by the other three methods. After reprocessing and retesting, the GenMark assay was able to detect both samples as positive. Two additional false-positive samples (sample D and sample E) were found by the Hologic assay; the original samples were retested and were found to be positive and negative, respectively. Following retesting of the five discordant samples, the GenMark ePlex SARS-CoV-2 EUA panel showed an improvement of the PPA to 100% (51/51). Additionally, 100% NPA (53/53) was obtained for the CDC assay, while Hologic improved to 98% (52/53) ( Table 4 ). Workflow evaluation. HoT, run time, and overall TAT to results were assessed for all preanalytical, analytical, and postanalytical steps for all four platforms. Results of the workflow assessment are shown in Table 5 . The longest HoT was the Hologic assay at ϳ2 h followed by the modified CDC assay at ϳ1 h 30 min. Very comparable HoT results were found for DiaSorin Molecular and GenMark with times of 16 min and 12 min, respectively. The run time averaged 90 min for modified CDC, DiaSorin Molecular, and GenMark. Hologic was the exception, with 4 h 35 min of run time (Table 5) . Overall TAT assessment, from sample to results, showed DiaSorin Molecular with the lowest TAT to results, followed by the GenMark, modified CDC, and Hologic assays with the highest overall times (Table 5 ).

In this study, we compared four different platforms for the detection of SARS-CoV-2 in patient specimens collected during March and April of the 2020 COVID-19 outbreak in the United States. We were able to make several observations, including LoD, overall workflow comparisons, and how each test performed in a head-to-head clinical comparison. Accurate and actionable results have been at the core of medical decisionmaking during this current outbreak, both in the inpatient and the outpatient settings. For hospitalized patients, results are critical for clinical management as well as for infection control and cohorting for bed management. Likewise, results are just as critical in the outpatient setting as the basis for social distancing measures to slow the spread of infection. To that end, false-negative results are particularly troubling, since they inevitably lead to more exposures. TAT is also critical for allocation of limited resources, such as the limited availability of isolation rooms and real-time cohorting decisions. In addition, health care workers need rapid results to ensure that they are not exposing the patients whom they are treating. Moreover, levels of personal protective equipment (PPE) required by health care professionals also vary depending on whether a patient is COVID-19 positive, requiring a short TAT to preserve precious resources.

Considering the transmissibility of SARS-CoV-2, which has recently been estimated to have a basic reproduction number (R 0 ) of 2.2, meaning that on average, each infected person can spread the infection to an additional two persons, a false-negative result can be devastating (15, 16) . This is especially true in vulnerable patient populations such as the elderly (especially people living in a nursing home or long-term-care facility), the immunocompromised, and people with preexisting medical conditions (17, 18) . Our data suggest that all four PCR methods yielded comparable results ( Ն 0.96); however, we did observe a notable difference in the PPA of the methods during this large-scale evaluation of EUA in vitro diagnostic assays. Our study showed that the DiaSorin Molecular and Hologic Fusion assays outperformed both the modified CDC and GenMark assays with respect to overall LoD, with GenMark having the overall highest LoD of all four platforms evaluated. DiaSorin Molecular had the lowest LoD (39 Ϯ 23 copies/ml), closely followed by Hologic (83 Ϯ 36 copies/ml). The modified CDC assay showed a final LoD of 779 Ϯ 27 copies/ml based on the results interpretation algorithm. It is worth mentioning that this assay requires both targets to be fully detected; thus, clinical samples falling in this concentration range would be identified and testing repeated, potentially requiring additional TAT and laboratory labor. In contrast, GenMark was able to detect 100% of the replicates only at 1,000 copies/ml and was not able to reliably detect replicates below 1,000 copies/ml; thus, patient specimens below this concentration range could potentially be missed. One important limitation to mention is that sensitivity using Probit analysis could not be calculated for GenMark since C T values are not available as part of the ePlex system result interpretation.

The clinical correlation was also consistent with the LoD findings, where both the DiaSorin Molecular and Hologic assays had 100% PPA and detected all specimens deemed positive by the consensus standard (interpretation of three of four evaluated assays as the "gold standard"), whereas GenMark missed two positive specimens (which were subsequently detected by GenMark upon repeat testing). DiaSorin Molecular and GenMark showed 100% NPA, while the Hologic and CDC assays initially had two and one discordant results, respectively. Repeat testing of these three specimens showed that for Hologic, sample D repeated as positive a second time and was therefore potentially a false positive and sample E was negative upon repeat testing, meaning this result could have previously been a false positive as well. The continued discordant result from sample D could potentially be attributed to specificity issues, since DiaSorin Molecular exhibited a slightly lower LoD. Repeat testing of sample A on the modified CDC assay was negative. Considering the LoD of the modified CDC assay, coupled with the fact that both DiaSorin Molecular and Hologic gave negative test results for sample A, this was likely a false-positive result. While all 4 assays could reliably produce results for most patient specimens in our study, GenMark lacked sensitivity, initially missing two low-level-positive specimens, and this could easily have impacted patient diagnosis by missing true-positive patient results. With respect to the HoT and TAT of the four assays in this study, the throughput and workflow evaluation results are clearly shown in Table 5 and are based on lab technologist experience in our laboratory. As a routine real-time RT-PCR assay, the modified CDC assay requires nucleic acid extraction, master mix preparation and PCR setup, and standard PCR amplification, as well as interpretation of the results. This involves several manual steps, needing about 1 h 30 min HoT and an approximate overall TAT of 3 h for processing of 24 specimens. The DiaSorin Molecular and GenMark assays have comparatively similar HoT and TAT, based on processing 8 samples per disc on the DiaSorin LIAISON MDX and 6 cartridges per tower in the GenMark ePlex. Clinical laboratories may decide to purchase additional instruments to allow testing of more samples at a time in order to satisfy patient testing volume requirements. The Hologic Panther Fusion platform has more of an automated workflow, with five samples processed at a time after loading. The sample-to-answer time for the first five samples is 2 h 40 min, followed by 5 results every 5 min after loading 120 samples; the total assay run time for 120 specimens is approximately 4 h 35 min. Note that the Hologic platform has longer HoT, because the technologist has to load the primers, probes, and other consumables and because 120 clinical samples have to be manually transferred to sample lysis buffer tubes. These steps, especially the pipetting of the specimen into the lysis tube, can be somewhat labor-intensive and time-consuming, bumping the overall TAT for 120 specimens closer to the 7-h mark. It is important to emphasize that each platform has its advantages. For workflow, TAT, and ease of use, the three sample-to-answer platforms (DiaSorin Molecular, Hologic, and GenMark) outperformed the modified CDC assay, which is a manual assay requiring many steps, specialized personnel, and separate areas for processing and performing the test. The Hologic platform is more appropriate for high-volume testing, while the DiaSorin Molecular and GenMark systems both work well in an environment where rapid results and low-tomoderate testing volumes are required.

This study had several limitations that should be mentioned. First, this was a single-center study, and the majority of the specimens were frozen after initial testing on the GenMark assay. While these limitations were present, they were minimized by the fact that the GenMark assay (which was the least sensitive platform in the analysis) actually had a potential competitive advantage, since it was the assay initially performed on fresh specimens. Second, while the number of specimens included in the clinical correlation was only 104, the patient samples spanned the entire range of clinical positives (including inclusion of specimens with low viral loads) and reflected our overall true positivity rate, which was between 50% and 60% during this time period of the COVID-19 outbreak.

In summary, we have evaluated four molecular in vitro diagnostic assays for the qualitative detection of SARS-CoV-2 in nasopharyngeal specimens. The data from our evaluation suggest that the modified CDC, DiaSorin Molecular, Hologic, and GenMark assays performed similarly ( Ն 0.96) and that all but the CDC assay can function in a sample-to-answer capacity. The GenMark assay, however, was less sensitive and had a higher LoD than both the DiaSorin Molecular and Hologic assays. Considering the design of all four assays, differences that could affect assay performance could include characteristics such as input volume of initial specimen, RNA purification and elution volume differences, and overall differences in gene targets. The DiaSorin Molecular platform has a lower testing volume capability than the Hologic assay (8 specimens/ disc run versus 120 specimens loaded at once) but has a short TAT and requires less reagent/sample preparation. All of these parameters, along with patient care needs, may assist clinical laboratories to identify and choose the correct testing platform that best fits their needs for the diagnosis of patients infected with this novel human coronavirus.

Coronavirus disease (COVID-19) outbreak. World Health Organization

Locations with confirmed COVID-19 cases, global map

Coronavirus COVID-19 global cases by the

Hosts and sources of endemic human coronavirus

COVID-19: zoonotic aspects

Receptor recognition by the novel coronavirus from Wuhan: an analysis based on decade-long structural studies of SARS coronavirus

SARS-CoV-2 cell entry depends on ACE2 and TM-PRSS2 and is blocked by a clinically proven protease inhibitor

Tissue distribution of ACE2 protein, the functional receptor for SARS coronavirus. A first step in understanding SARS pathogenesis

WHO-China Joint Mission on Coronavirus Disease. 2020. Report of the WHO-China Joint Mission on Coronavirus Disease

Aerosol and surface stability of SARS-CoV-2 compared to SARS-CoV-1

Coronavirus (COVID-19) update: FDA issues new policy to help expedite availability of diagnostics

The measurement of observer agreement for categorical data

Interrater reliability: the kappa statistic

A table for the calculation of working probits and weights in probit analysis

Covid-19 -navigating the uncharted

SARS-CoV-2 Viral Load in Upper Respiratory Specimens of Infected Patients

Groups at higher risk for severe illness

Washington State 2019-nCoV Case Investigation Team. 2020. First case of 2019 novel coronavirus in the United States

We thank Diasorin Molecular LLC for providing the reagents used in this study. G.J.B. has previously given education seminars for Hologic, Inc., and received an honorarium.