key: cord-0925610-afhw0450
authors: Suhandynata, Raymond T; Hoffman, Melissa A; Kelner, Michael J; McLawhon, Ronald W; Reed, Sharon L; Fitzgerald, Robert L
title: Multi-platform Comparison of SARS-CoV-2 Serology Assays for the Detection of COVID-19
date: 2020-08-07
journal: J Appl Lab Med
DOI: 10.1093/jalm/jfaa139
sha: 7312fbd31e3f99a912a151c6140efdc6e77e27eb
doc_id: 925610
cord_uid: afhw0450

BACKGROUND: COVID-19 is caused by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), a novel beta-coronavirus that is responsible for the 2019 coronavirus pandemic. Acute infections should be diagnosed by polymerase chain reaction (PCR) based tests, but serology tests can demonstrate previous exposure to the virus. METHODS: We compared the performance of the Diazyme, Roche, and Abbott SARS-CoV-2 serology assays using 179 negative subjects to determine negative percent agreement (NPA) and in 60 SARS-CoV-2 PCR confirmed positive patients to determine positive percent agreement (PPA) at three different timeframes following a positive SARS-CoV-2 PCR result. RESULTS: At ≥ 15 days, the PPA (95% CI) was 100 (86.3–100)% for the Diazyme IgM/IgG panel, 96.0 (79.7–99.9)% for the Roche total Ig assay, and 100 (86.3–100)% for the Abbott IgG assay. The NPA (95% CI) was 98.3 (95.2–99.7)% for the Diazyme IgM/IgG panel, 99.4 (96.9–100)% for the Roche total Ig assay, and 98.9 (96.0–99.9)% for the Abbott IgG assay. When the Roche total Ig assay was combined with either the Diazyme IgM/IgG panel or the Abbott IgG assay, the positive predictive value was 100% while the negative predictive value remained greater than 99%. CONCLUSIONS: Our data demonstrates that the Diazyme, Roche, and Abbott SARS-CoV-2 serology assays have similar clinical performance. We demonstrated a low false positive rate across all three platforms and observed that false positives observed on the Roche platform are unique compared to those observed on the Diazyme or Abbott assays. Using multiple platforms in tandem increases the PPVs which is important when screening populations with low disease prevalence.

The 2019 worldwide coronavirus pandemic began with an unknown cause of pneumonia in the Wuhan province of China (1, 2) . The causative pathogen is now known as severe acute respiratory syndromerelated corona virus 2 (SARS-CoV-2), and is responsible for the COVID-19 disease (1,3). As of June 5 th , 2020, there have been over 6.8 million reported cases of COVID-19 (4) . Common symptoms include cough, shortness of breath, fever, and viral pneumonia (2, 5) . Although global efforts are focused on containing the virus (6-10), progress is hampered by the fact that asymptomatic carriers of are able to transmit the disease (11) (12) (13) (14) . A separate study indicates that the large number of undocumented COVID-19 infections play a major role in the rapid spread of the virus (15) .

The current standard for the diagnosis of an acute SARS-CoV-2 infection is real-time reverse transcriptase polymerase chain reaction (RT-PCR) performed on nasal or oral specimens (16, 17) .

However, RT-PCR tests are most effective in the early stages of infection and can be falsely negative as viral shedding decreases over the course of infection (18) (19) (20) (21) . In contrast, serology assays which probe for the presence of antibodies against SARS-CoV-2 are better suited for determining if an individual has been previously infected with COVID-19 (19, 22) . The accurate identification of prior SARS-CoV-2 infections is essential for epidemiologic studies, on-going surveillance, vaccine studies, understanding control measures (e.g. social distancing), and risk assessment (23) .

One of the major topics of debate that surround SARS-CoV-2 serology testing is the poor positive predictive value (PPV) of the currently available serology assays when used to screen for COVID-19 exposure in the general population (25) . On May 27, 2020, the centers for disease control and prevention (CDC) released interim guidelines for COVID-19 antibody testing (26) , which recommended only utilizing a test that has a specificity of greater than 99.5% or using an orthogonal testing algorithm in which positive results are confirmed on a secondary assay. Two assays with manufacturer reported specificity of greater than 99.5% are the Roche Elecsys Anti-SARS-CoV-2 and the Abbott SARS-CoV-2 IgG assays (27, 28) . In fact, several recent studies have reported independent validations of the Roche and Abbott SARS-CoV-2 serology assays and reported specificity ranging from 99.4 -99.90% for the Abbott assay and 98.7% -100% for the Roche assay (29) (30) (31) (32) . Furthermore, our group previously evaluated a third serology platform, the Diazyme 2019-nCoV IgM and IgG serology assays, and found them both to have specificity of 99.6% and 99.1%, respectively (33).

Here we evaluate three high specificity SARS-CoV-2 serology assays manufactured by Diazyme, Roche, and Abbott using an overall cohort of 239 subjects, 60 of which are SARS-CoV-2 PCR confirmed patients.

This is the first study that provides a direct comparison, using the same samples, of these SARS-CoV-2 serology assays and reveals unique characteristics that will improve the quality of SARS-CoV-2 serology testing.

339 excess serum and plasma samples from 239 subjects were collected in BD Vacutainer collection tubes (K-EDTA, lithium-heparin plasma separator tubes, and/or serum separator tubes) under UCSD IRB protocol 181656. This included serum or plasma samples from 60 patients (Supplementary Table 1) which tested PCR positive for SARS-CoV-2, 22 patients which tested PCR positive on a respiratory panel nucleic acid (RPNA) test infections other than SARS-CoV-2, 24 patients which tested positive for antinuclear antibodies (ANA) or anti-double stranded DNA (dsDNA), 3 patients with clinically elevated levels of IgM/IgG, 20 apparently healthy subjects (no respiratory symptoms per self-report), and 110 patient samples that had been stored frozen (-20 o C) since 2018. Three additional groups within the positive cohort were generated to calculate PPA and NPA for SARS-CoV-2 seropositivity at three timeframes relative to a confirmed positive SARS-CoV-2 PCR result: a ≤ 7 day group (n = 222), a between 8 -14 day group (n=210), and a ≥ 15 day group (n=204) (Supplementary Table 2 ). These three groups contained all the presumed negative subjects and one sample from each PCR positive patient who had a blood draw in the specified timeframe.

All 60 SARS-CoV-2 patients were confirmed positive for COVID-19 by a nucleic acid amplification test that had been clinically validated in our laboratory and had an emergency use authorization (EUA) listing with the US Food and Drug Administration. For the purposes of PPA and NPA calculations, all 60 SARS-CoV-2 confirmed patients were treated as a true positive for SARS-CoV-2 serology. All remaining samples were considered true negatives. 

To identify patient specimens containing other PCR confirmed microbes, the respiratory pathogen 

Precision was calculated across 4 days by running 5 batches of 5 replicates of the quality controls (QCs) supplied with the Abbott SARS-CoV-2 IgG kit and two patient pools (Positive and Negative) (n=25).

Positive patient pools were made by pooling SARS-CoV-2 PCR positive patient samples to create a sample pool, which was diluted with negative samples to be moderately positive. Negative patient pools were created by combining patient plasma (Li-heparin or K-EDTA) samples that were collected from healthy volunteers and verified to be negative on all three platforms.

Dilutional linearity was performed by mixing a positive patient pool with a negative patient pool in 10% increments prior to analysis. Samples were mixed thoroughly and immediately analyzed on both the Roche and Abbott platforms. Dilutional linearity has been previously reported for the Diazyme assays (33) .

Data was analyzed using R in Rstudio and linear regression analysis for all figures were performed in excel or Rstudio. Box and whisker plots were generated in Rstudio. Precision (%CV) was calculated by analysis of variance (ANOVA) and total precision (%CV) was calculated by the sum of squares. 95%

confidence intervals for PPA and NPA values were calculated using the MEDCALC online diagnostic test calculator (34) . PPA was defined as the number of positive test results divided by the sum of true positives and false negatives. NPA was defined as the number of negative test results divided by the sum of true negatives and false positive results.

The precision profiles (%CV) of the Roche and the Abbott assays can be found in Table 1 . Within run precision ranged from 1.6% -6.8% for the Roche assay and from 1.0% -5.6% for the Abbott assay.

Between-run precision ranged from 2.7% -12.1% for the Roche assay and from 12.3% -22.0% for the Abbott assay. Total precision ranged from 4.6% -13.9% for the Roche assay and from 11.6% -22.6% for the Abbott assay. The precision profile of the Diazyme platform has been published previously (33) .

Linear regression analysis was used to describe the relationship between observed and expected values following sample dilution (Supplementary Figure 1) . Dilution studies demonstrated that both the Roche (R 2 = 0.5231, y=0.7014x + 24.484) and the Abbott assays (R 2 = 0.405, y=0.6215x + 4.2263) were not linear (Supplementary Figure 1A and 1B) .

The cross-reactivity of the Diazyme, Roche, and Abbott platforms were evaluated against 49 patients, with other conditions or respiratory pathogens, including 7 with non-COVID-19 coronavirus ( Table 2) .

We observed no cross-reactivity for any of these samples on the Diazyme IgM, Diazyme IgG, Roche total Ig, or Abbott IgG SARS-CoV-2 serology platforms.

Patient samples grouped into three different time-frames relative to a positive PCR result were used to calculate the positive percent agreement (PPA) for all of the assays (Figure 1A and 

Diazyme, Roche, and Abbott SARS-CoV-2 serology assays were compared by correlating the raw quantitative values from each platform for all samples from PCR positive patients ( Supplementary   Figure 2A-C) . Linear regression analysis indicated weak positive correlation across all three platforms, with r 2 values ranging from 0.11-0.31. Since these are qualitative tests, we focused on samples that were discordant between the platforms ( Table 3) . Eighteen samples from 17 SARS-CoV-2 PCR positive patients were discrepant between the three platforms, and 4 discordant samples were discrepant in the 130 negative samples.

The impact of disease prevalence on the PPV and negative predictive value (NPV) for the Diazyme, Roche, and Abbott SARS-CoV-2 serology platforms were calculated using the PPA and NPA of the ≥ 15 day patient group ( Table 4 and Supplementary Table 4 ). The prevalence for COVID-19 in the ≥ 15 day patient group was 12.3%, and the observed PPV for the Diazyme IgM/IgG panel, the Roche total Ig, and the Abbott IgG assays were 89.3%, 96.0%, and 92.6%, respectively. The observed NPV for the Diazyme IgM/IgG panel, the Roche total Ig, and the Abbott IgG assays were 100%, 99.4%, and 100%, respectively.

Since the one Roche false positive result was independent of the other two assays, the combination of the Roche total Ig assay with either the Diazyme IgM/IgG panel or the Abbott IgG assay resulted in a PPV of 100% regardless of the prevalence, while maintaining NPV greater than or equal to 99% ( Table 5 ). In contrast, since the Abbott assay shared two false positive results with the Diazyme platform, the combination of the Diazyme IgM/IgG panel combined with the Abbott IgG assay resulted in a PPV that ranged from 17.6 to 95.7%, depending on the prevalence of the population being tested, and a NPV of 100% regardless of disease prevalence ( Table 5 ). The prevalence of 3.8% (256/6680) represents the percentage of positive SARS-CoV-2 PCR tests performed at UCSD as of June 6th, 2020.

We evaluated longitudinal samples from 16 SARS-CoV-2 PCR positive patients with the Diazyme, Roche, and Abbott serology platforms (Figure 2A -2D) . Observed median time to seropositivity and interquartile ranges for the Diazyme platform was 5.5 days for IgM (IQR: 4 -10 days) and 5 days for IgG (IQR: 3 -6.75 days) (Figure 2A, 2B 

The observed precision profiles of the Roche and the Abbott assays were appropriate for clinical use, with within-run precision ranging from 1.0% -6.8% across the Roche and Abbott platforms. The Abbott platform had between-run precision and total precision of 22.0% and 22.6%, respectively. However, these high levels of imprecision are for a negative result that is well below the cutoff value and have no negative impact when interpreting a qualitative assay. Patient samples did not dilute in a linear fashion on the Roche and Abbott assays, which is in contrast to the comparatively linear Diazyme SARS-CoV-2

IgM and IgG assay that were previously evaluated (33) .

We evaluated the PPA and NPA of the Diazyme, Roche, and Abbott SARS-CoV-2 serology assays using the same samples across all three platforms. The three assays evaluated had similar PPAs with overlapping confidence intervals in each of the three timeframes relative to a positive PCR result. The (Supplementary Figure 2) . We also compared the observed values in discordant samples across the three platforms ( Table 3) . Analysis of the discrepant samples showed that the false positive results were shared by both the Diazyme and Abbott assays, but that the Roche assay false positive was unique for this platform.

A major use of SARS-CoV-2 serology assays is for screening patient populations with low disease prevalence and determining the prevalence of infection/exposure to COVID-19. While an accurate assessment of the overall prevalence of COVID-19 in the United States is currently undetermined, the positivity rate of SARS-CoV-2 PCR tests performed at UCSD as of June 4, 2020 was 3.8%; with the prevalence in the general population assumed to be much lower. Assuming a disease prevalence of 1%, the highest PPV observed on a single platform belonged to the Roche total Ig assay at 62.2%, which is inadequate ( Table 4) . However, the PPV can be greatly improved by using two different assays which have unique sets of false positives. We used our cohort to calculate the PPV and NPV when combining the Roche total Ig assay and the Diazyme IgG or Abbott IgG assay ( Table 5) . Remarkably, at a disease prevalence of 1%, the PPV and NPV values would both be 100%. However, these improvements were not observed when combining the Diazyme IgM/IgG panel and Abbott IgG assay, with a PPV and NPV value of 46.2% and 100%. This data supports the recommendation of the CDC to combine independent assays to improve PPV (26) , and demonstrates the importance of directly comparing different assays in order to understand cross reactivity.

We analyzed longitudinal samples from 16 PCR positive SARS-CoV-2 patients on the Diazyme, Roche, and Abbott platforms. In these 16 patients, the Roche total Ig assay detected seropositivity for SARS-CoV-2 antibodies earlier (3.5 days) than either the Diazyme (5 days) or Abbott (4.5 days) platforms.

While an important observation, this minor difference is unlikely to be clinically significant, unless the assay is being used to confirm an infection at early stages of infection and disease progression.

Although this is one of the largest studies to be presented in the peer reviewed literature as of the current date, one limitation is the relatively small number of samples included. This is particularly important if SARS-CoV-2 serology tests will be performed on large populations. Our data suggests that the PPA of these assays is high when testing patients greater than 15 day post PCR positivity, but it would be naïve to quote PPA of 100%, as the confidence intervals range from 74 to 100%. In a similar manner, it would unwise to conclude that combining Roche with either Diazyme or Abbott would lead to 100% PPV. What we can conclude is that combining these assays change values in low prevalence populations from being unacceptably low to being meaningful.

In summary, our study demonstrates that the Diazyme, Roche, and Abbott SARS-CoV-2 serology platforms possess similar clinical performance and highlights several key differences. Our observations make a strong argument that when screening populations with low disease prevalence of COVID-19, laboratory professionals should consider the use of at least two orthogonal serology platforms to improve the poor PPVs that are expected from using one platform alone.

We would like to acknowledge all of the staff in the UC San Diego Health clinical laboratories for their help identifying specimens for sensitivity and specificity testing. We would also like to thank Amy

Diagnostics are acknowledged for funding the clinical chemistry fellowships for R.T. Suhandynata and M.A. Hoffman respectively. Observed PPA A) and NPA B) for the Diazyme IgM, Diazyme IgG, Roche total Ig and Abbott IgG SARS-CoV-2 serology assays. The number of unique patients in each group is shown below the X-axis. C) Distribution of AU/mL, COI, and S/C are shown in box and whisker plots which depict the median and interquartile ranges (IQR) observed for each platform in the indicated sample groups. Error bars are no greater than 1.5 times the IQR before Q1 and after Q3. Dashed line indicates the cutoff for each platform with values above representing positive results and values below indicating negative results.

Diazyme AU/mL values for A) IgM or B) IgG are plotted on the Y-axis on a semi-log scale and the number of days relative to a positive PCR test is shown on the X-axis. C) Roche COI values for total Ig or D) Abbott S/C index values for IgG are plotted on the Y-axis on a semi-log scale and the number of days relative to a positive PCR test is shown on the X-axis. The X-axis is set at the respective Diazyme, Roche, or Abbott cutoff value for a positive sample (Diazyme: 1.00 AU/mL, Roche: 1.00 COI, and Abbot: 1.4 S/C). The median and interquartile ranges for the observed time to seropositivity, relative to a positive SARS-CoV-2 PCR result, are shown for each platform.

Identification of a novel coronavirus causing severe pneumonia in human: a descriptive study

A Novel Coronavirus from Patients with Pneumonia in China

A pneumonia outbreak associated with a new coronavirus of probable bat origin

Live): 6,801,914 Cases and 396,681 Deaths from COVID-19 Virus Pandemic -Worldometer

Molecular and serological investigation of 2019-nCoV infected patients: implication of multiple shedding routes. Emerg Microbes Infect

Feasibility of controlling COVID-19 outbreaks by isolation of cases and contacts

Drive-Through Screening Center for COVID-19: a Safe and Efficient Screening System against Massive Community Outbreak

COVID-19 epidemic in Switzerland: on the importance of testing, contact tracing and isolation

Response to COVID-19 in Taiwan: Big Data Analytics, New Technology, and Proactive Testing

Countries test tactics in 'war' against COVID-19

Presumed Asymptomatic Carrier Transmission of COVID-19

Clinical characteristics of 24 asymptomatic infections with COVID-19 screened among close contacts in Nanjing

A systematic review of asymptomatic infections with COVID-19

Alert for non-respiratory symptoms of Coronavirus Disease 2019 (COVID-19) patients in epidemic period: A case report of familial cluster with three asymptomatic COVID-19 patients

Substantial undocumented infection facilitates the rapid dissemination of novel coronavirus (SARS-CoV-2)

Detection of 2019 novel coronavirus (2019-nCoV) by real-time RT-PCR

Molecular Diagnosis of a Novel Coronavirus (2019-nCoV) Causing an Outbreak of Pneumonia

Temporal profiles of viral load in posterior oropharyngeal saliva samples and serum antibody responses during infection by SARS-CoV-2: an observational cohort study

Profiling Early Humoral Response to Diagnose Novel Coronavirus Disease (COVID-19)

Viral Load Kinetics of SARS-CoV-2 Infection in First Two Patients in Korea

False Negative Tests for SARS-CoV-2 Infection -Challenges and Implications

Real-time RT-PCR in COVID-19 detection: issues affecting the results

Diagnostic Testing for Severe Acute Respiratory Syndrome-Related Coronavirus-2: A Narrative Review

Double-Edged SpikeAre SARS-CoV-2 Serologic Tests Safe Right Now?

Waiting for Certainty on Covid-19 Antibody Tests -At What Cost?

CDC. Information for Laboratories about Coronavirus

Elecsys® Anti-SARS-CoV-2

Performance Characteristics of the Abbott Architect SARS-CoV-2 IgG Assay and Seroprevalence in

Clinical performance of the Elecsys electrochemiluminescent immunoassay for the detection of SARS-CoV-2 total antibodies

Clinical Performance of the Roche SARS-CoV-2 Serologic Assay

Clinical Performance of Two SARS-CoV-2 Serologic Assays

Longitudinal Monitoring of SARS-CoV-2 IgM and IgG Seropositivity to Detect COVID-19

MedCalc's Diagnostic test evaluation calculator