key: cord-0827790-e2iwolez
authors: Ward, Matthew D; Mullins, Kristin E; Pickett, Elizabeth; Merrill, VeRonika; Ruiz, Mark; Rebuck, Heather; Duh, Show-Hong; Christenson, Robert H
title: Performance of four automated SARS-CoV-2 serology assay platforms in a large cohort including susceptible COVID-19 negative and COVID-19 positive patients
date: 2021-03-10
journal: J Appl Lab Med
DOI: 10.1093/jalm/jfab014
sha: b47c85c0d3703931924be19d685bd0d12764de95
doc_id: 827790
cord_uid: e2iwolez

BACKGROUND: Anti-SARS-CoV-2 serological responses may have a vital role in controlling the spread of the disease. However, the comparative performance of automated serological assays has not been determined in susceptible patients with significant co-morbidities. METHODS: In this study, we used a large number of COVID-19 negative patient samples (n = 2030) as well as COVID-19 positive patient samples (n = 112) to compare the performance of four serological assay platforms; Siemens Healthineers Atellica IM Analyzer, Siemens Healthineers Dimension EXL Systems, Abbott ARCHITECT, and Roche cobas. RESULTS: All four serology assay platforms exhibited comparable negative percent agreement with negative COVID-19 status ranging from 99.2-99.7%, and positive percent agreement from 84.8-87.5% with positive real-time reverse transcriptase polymerase chain reaction (RT-PCR) results. Of the 2142 total samples, only 38 samples (1.8%) yielded discordant results on one or more platforms. However, only 1.1% (23/2030) of COVID-19 negative cohort results was discordant whereas discordance was 10-fold higher for the COVID-19 positive cohort at 11.3% (15/112). Of the total 38 discordant results, 34 were discordant on only one platform. CONCLUSION: Serology assay performance was comparable across the four platforms assessed in a large population of COVID-19 negative patients with relevant comorbidities. The pattern of discordance shows that samples were discordant on a single assay platform, and discordance rate was 10-fold higher in the COVID-19 positive population. IMPACT STATEMENT: High negative percent agreement reinforces the reliability of serology testing especially in a cohort of at-risk patients. Serology platform discordance highlights the importance of a two-test strategy for properly identifying seroconverted patients.

The COVID-19 pandemic has challenged the capacity and capabilities of national medical systems and has highlighted the importance of laboratory and diagnostic testing to control the spread of the disease through timely diagnosis and robust contact tracing.

Nucleic acid amplification testing (NAAT) detecting the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is central to identifying individuals with active infection and a COVID-19 diagnosis (1). To enhance surveillance efforts and control the virus spread, serological testing has emerged as an opportunity to identify patients who may have been exposed to the virus, who have recently recovered from an infection whether or not they were symptomatic, or to assess or identify the durability of immune responses (2) .

The rapid emergence of serological assays has also challenged, and often out-paced, regulatory agencies as well as our understanding of COVID-19 serological responses.

The quality and performance of these assays was initially unknown, leading the US Food and Drug Administration (FDA) to generate a "removed" test list, for all assays considered for an emergency use authorization (EUA), but demonstrating poor clinical performance (3) . The overall utility of the assay, regardless of the manufacturer, has also been challenged, due to lack of supporting data for the proposed applications and the value of the test (4) . These questions surrounding test utility and assay performance remain unresolved.

Many serological tests have entered the healthcare market, most employing an immunoassay sandwich method using a SAR-CoV-2 envelope protein as an antigen 'bait' to detect immunoglobulins. The bait for these assays is often either recombinant-derived ectodomain of the spike (S) protein or the nucleocapsid (N), which is used as antigen to bind IgG, IgA, and/or IgM in a patient sample. Previous studies using commerciallyavailable ELISA kits, have shown that both S and N proteins have near equivalent performance characteristics in detecting serological responses (5) .

The present study was designed to compare head-to-head the performance characteristics of four automated analyzers, using a large and diverse population of patients. The study population was carefully selected to include a large number of COVID-19 positive patients (n=93; total samples, n=112) confirmed by reverse-transcriptase real-time polymerase chain reaction (RT-PCR), as well as a large group of COVID-19 negative patient samples (n=2030), collected in the United States (USA) prior to October 2019 and diagnosed with a respiratory or cardiovascular disorder that would otherwise increase risk for COVID-19 mortality (6).

Study samples belonged to two general groups, a COVID-19 negative group (Table 1) (Table 2) , diagnosed based on recent RT-PCR test. Both cohorts included remnant samples from patient plasma collected in lithium heparin collection containers.

To compare assay agreement using a COVID-19 negative cohort, 1 mL frozen aliquots were withdrawn from a sample bank maintained by the study sponsor (Siemens) .

Samples were identified based on an associated diagnosis or condition causing dyspnea.

To compare assay agreement using a COVID-19-positive cohort, remnant samples were Table 1 ). These platforms were validated, and shown to provide comparable results. Table 3 ). All instruments were operated and maintained according to each respective manufacturer's operations manual. Quality control and calibration materials were prepared using the Instruction for Use document, and control results were within acceptable ranges prior to acceptance of study samples.

Sample handling. Specimens used in this study were frozen prior to use. Frozen specimens were thawed, mixed thoroughly, then centrifuged to remove particulates prior to testing. All specimens were assayed on the same freeze thaw cycle and assayed in parallel on each of the four platforms in singlicate. 

The COVID-19-negative group included 2,030 unique patient samples (850 female, 1147 male, 33 not indicated; median age 69 years), including a range of pulmonary and cardiac diseases identified as conditions of breathlessness (Table 1 ). The COVID-19-positive group included 112 samples from 93 unique patients (41 female, 53 male; median age 55) ( Table 2) The apparent pattern of discordance between the four platforms was perceptibly unique to each instrument (Table 6) 

In this study, we reported serology testing of the largest cohort of COVID-19 negative, high risk patients with significant co-morbidities causing dyspnea which has not been previously shown. This large study population demonstrated a high degree of negative agreement in serological responses across the four commercial platforms tested. Central to this portion of the study was the inclusion of plasma samples collected prior to October 2019, intentionally including a population of COVID-19 negative patients with conditions of breathlessness, a group of co-morbidities that would otherwise have a higher mortality risk during the on-going pandemic (6) . We have also shown a high degree of positive agreement across these serology platforms as compared to a recent COVID-19 diagnosis. Seroconversion and antibody responses to infection are of vital interest in vaccination efforts and understanding effectiveness and persistence of immunity. This work joins several studies in describing the performance characteristics of rapidly emerging SARS-CoV-2 serological assays (11, 12, 21, (13) (14) (15) (16) (17) (18) (19) (20) .

By comparison, most similar studies have used a considerably smaller COVID-19 negative cohort from a collection of historic patient samples simply termed pre-COVID-19 or pre-pandemic samples without regard for co-morbidities (12, (15) (16) (17) 21) , or from healthy individuals (19) . The benefit of using pre-pandemic samples allows for definitive identification of any apparent false positives, since the virus was not known to exist prior to the year 2019. Other studies have collected contemporaneous patient samples during the pandemic with an associated negative RT-PCR test (18, 20) . We believe the use of COVID-19 negative patient samples with specific high-risk co-morbidities is most representative of patients likely to receive a COVID-19 associated test, including SARS-CoV-2 RT-PCR or SARS-CoV-2 serological test, during the pandemic. Without regard for cohort selection criteria, the majority of the platforms compared in these studies demonstrate sensitivities (positive percent agreement) and specificities (negative percent agreement) greater than 90%, using their select populations of COVID positive and COVID negative study participants.

Another important consideration in selection of COVID-19 negative samples is crossreactivity from similar viral respiratory infections or other coronaviruses. Several studies have previously shown minimal or no cross-reactivity with these other infectious diseases (15, 17, 19, 21) . That said, these issues will be studied systematically when the manufacturers seek FDA clearance or approval for their assays.

We avoided comments on specificity and sensitivity of the assay performance since a "gold standard" for SARS-CoV-2 serology has not been established. Therefore, discordant results were not considered false positive or false negatives for the purposes of this study, even though the FDA has applied a standard in which positive percent agreement is used as a surrogate for sensitivity and negative percent agreement for specificity (22) . It is important to note that RT-PCR positive COVID-19 patients with an undetectable serological response may always result as a 'false negative' using this scheme (12) . For this reason, the Center for Disease Control (CDC) does not recommend using serology testing to diagnose previous SARS-CoV-2 infection (23).

The cause of disagreement between serological testing and RT-PCR is summarized in CDC guidance for the use and interpretation of serological testing (23) . A negative serology test may not preclude a previous infection, as some infected patients may never develop antibodies. And, a positive serology test may not indicate a previous or current infection, because these antibodies may reflect an infection with a different virus from the same family of viruses.

Similar studies have analyzed how serological sensitivity changes over time using serial blood draws from the same patients (11) (12) (13) . One study has shown higher positive agreement between assays >14 days post first-positive RT-PCR test (13) . One head-tohead comparison study employed two assays both targeting the SARS-CoV-2 nucleocapsid (N) antibodies, displaying similar degrees of assay platform discordance with serial samples from the same patients (12) . Future studies investigating the complete serological response using multiple platforms and serial blood draws will identify if there is, or is not, a time point of convergence for positive agreement in these assays.

These results suggest that a two-test strategy to serology testing using two systems will improve overall agreement with RT-PCR results, as suggested previously (20) . Using this two-test strategy, a single positive result would indicate a positive result, increasing positive percent agreement with a modest decrement in negative percent agreement.

In this study, the detected immunoglobulin sub-type (IgG, IgM, IgA) and the antigen employed as bait, spike (S) or nucleocapsid (N), do not seem to be important factors based on the pattern of discordance observed in this study. Elucidating the explicit cause of discordance among the serology assays is complicated and was not part of this study's design. Because the virus was unknown to exist prior to 2019, the false positive results are thought to be the result of non-specific or cross-reactivity in antibody binding, perhaps with other related viruses. However, it is also possible that other unspecified interferences to each assay's individual reagents, measurement strategy and principles may have contributed to the observed discordance. It is noteworthy that all of the assays included here were available by EUA, and have not yet undergone the full rigor of FDA clearance or approval that, in part, includes thorough investigation of cross-reactivity and interferences."

We have completed the largest serology platform comparison to-date including 93 unique COVID-19 positive patients and 2030 COVID-19 negative patients, with samples run in parallel on four different platforms. We have shown a unique pattern of discordance across these platforms, which have implications in assay selection and suggests a two-testing strategy will improve overall agreement and reduce apparent false negative results. 

World Health Organization. Laboratory testing for 2019 novel coronavirus (2019-nCoV) in suspected human cases

The role of serological testing in the SARS-CoV-2 outbreak

COVID-19 Serology/Antibody Tests Should Not Be Used -Letter to Clinical Laboratory Staff and Health Care Providers | FDA. 2020;2-5

The role of antibody testing for sars-cov-2: Is there one?

Evaluation of nucleocapsid and spike protein-based enzyme-linked immunosorbent assays for detecting antibodies against SARS-CoV-2

Covid-19: Risk factors for severe disease and death

SARS -CoV -2 Total Antibody assay ( COV2T )

Total (COV2T) Assay for the Detection of Total Antibodies to SARS-CoV-2. 2020;Atellica I:1-26

Abbott ARCHITECT IgG SARS-CoV-2 assay

Elecsys Anti-SARS-CoV-2 assay

Clinical performance of two SARS-CoV-2 serologic assays

Comparison of two automated immunoassays for the detection of SARS-CoV-2 nucleocapsid antibodies

Clinical performance of the elecsys electrochemiluminescent immunoassay for the detection of SARS-CoV-2 total antibodies

An evaluation of COVID-19 serological assays informs future diagnostics and exposure assessment

Head-to-Head Comparison of Two SARS-CoV-2 Serology Assays

Multi-platform comparison of SARS-CoV-2 serology assays for the detection of COVID-19

Evaluation of three commercial automated assays for the detection of anti-SARS-CoV-2 antibodies

Diagnostic performances and thresholds: The key to harmonization in serological SARS-CoV-2 assays?

A comparison of four serological assays for detecting anti-SARS-CoV-2 antibodies in human serum samples from different populations

Comparison of four new commercial serologic assays for determination of SARS-CoV-2 IgG

Head-to-head evaluation on diagnostic accuracies of six SARS-CoV-2 serological assays

Food and Drug Administration U. EUA Authorized Serology Test Performance

Test for Past Infection | CDC. Centers Dis Control Prev