key: cord-1025918-0ru078yr
authors: Rychert, Jenna; Couturier, Marc Roger; Elgort, Marc; Lozier, Bucky Ken; La’ulu, Sonia; Genzen, Jonathan R; Straseski, Joely A; Delgado, Julio C; Slev, Patricia R
title: Evaluation of Three SARS CoV-2 IgG Antibody Assays and Correlation with Neutralizing Antibodies
date: 2020-10-16
journal: J Appl Lab Med
DOI: 10.1093/jalm/jfaa188
sha: 67e999b9b1647dca179aaf61a8634900e65e6e88
doc_id: 1025918
cord_uid: 0ru078yr

BACKGROUND: As serologic assays for SARS-CoV-2 become more widely utilized, it is important to understand their performance characteristics and correlation with neutralizing antibodies. We evaluated three commonly used SARS-CoV-2 IgG assays (Abbott, DiaSorin and EUROIMMUN) for clinical sensitivity, specificity, and correlation with neutralizing antibodies and then compared antibody kinetics during the acute phase of infection. METHODS: Three panels of samples were tested on every assay. Sensitivity was assessed using a panel of 35 specimens serially collected from 7 RT-PCR-confirmed COVID-19 patients. Specificity was determined using 100 sera samples collected in 2018 from healthy individuals prior to the outbreak. Analytical specificity was determined using a panel of 37 samples from individuals with respiratory illnesses other than COVID-19. RESULTS: Clinical sensitivity was 91.43% (95% CI 76.94%-98.20%) for Abbott, and 88.57% (95% CI 73.26%-96.80%) for both DiaSorin and EUROIMMUN. Clinical specificity was 99.00% (95% CI 94.55%-99.97%) for Abbott and DiaSorin and 94.00% (95% CI 87.40%-97.77%) for EUROIMMUN. The IgG assays demonstrated good qualitative agreement (minimum of 94%) and good correlation between the quantitative result for each combination of assays (r2≥0.90). The neutralizing antibody response did not necessarily follow the same temporal kinetics as the IgG response and did not necessarily correlate with IgG values. CONCLUSION: The three IgG antibody assays demonstrated comparable performance characteristics. Importantly, a qualitative positive IgG result obtained with any of the assays was associated with the presence of neutralizing antibodies; however, neutralizing antibody concentrations did not correlate well with signal to cutoff ratios.

Background: As serologic assays for SARS-CoV-2 become more widely utilized, it is important to understand their performance characteristics and correlation with neutralizing antibodies. We evaluated three commonly used SARS-CoV-2 IgG assays (Abbott, DiaSorin and EUROIMMUN) for clinical sensitivity, specificity, and correlation with neutralizing antibodies and then compared antibody kinetics during the acute phase of infection. Methods: Three panels of samples were tested on every assay. Clinical specificity was 99.00% (95% CI 94.55%-99.97%) for Abbott and DiaSorin and 94.00% (95% CI 87.40%-97.77%) for EUROIMMUN. The IgG assays demonstrated good qualitative agreement (minimum of 94%) and good correlation between the quantitative result for each combination of assays (r2≥0.90). The neutralizing antibody response did not necessarily follow the same temporal kinetics as the IgG response and did not necessarily correlate with IgG values. Conclusion: The three IgG antibody assays demonstrated comparable performance characteristics. Importantly, a qualitative positive IgG result obtained with any of the assays was associated with the presence of neutralizing antibodies; however, neutralizing antibody concentrations did not correlate well with signal to cutoff ratios.

The recent increase in the availability and use of serologic assays for detecting antibodies against SARS-CoV-2 is enabling a more thorough understanding of the broader epidemiology of COVID-19 infections and provides a more accurate estimate of local prevalence and disease incidence. However, the increase in testing options has also led to many questions. Clinicians and laboratorians are inundated with options for COVID antibody testing, ranging from rapid tests to laboratory based ELISAs and chemiluminescent immunoassays (CIAs). While the FDA has facilitated a more rapid pathway to the US market through updates to the EUA process, independent evaluation and verification of the performance characteristics of the remaining available assays is critical for guiding utilization. The aim of our studies was to two-fold. First, we aimed to evaluate the performance of three widely utilized SARS CoV-2 IgG commercial assays, the Abbott SARS-CoV-2 IgG, DiaSorin Liaison SARS-CoV-2 S1/S2 IgG, and the Anti-SARS-CoV-2 IgG ELISA from EUROIMMUN. Second, we wanted to determine whether these assays could be used to predict the neutralizing antibody response. The antibody response that develops in response to SARS CoV-2 infection consists of both binding and neutralizing antibodies. Neutralizing antibodies are crucial for preventing infection and are believed to play a key role in putative immunity. However, these assays are time consuming, labor intensive, require expertise, are expensive and are not amenable to mass screening protocols. The currently available commercial antibody assays measure both types of antibodies (binding and neutralizing) and are more readily implemented in the clinical laboratory. Therefore, determining correlation between the current high-throughput antibody assays and neutralizing antibodies, which are essential for any protective immunity, is critical.

A total of 35 samples were collected from 7 adults (age range 39-79, 29% female) confirmed positive for COVID-19 by an FDA authorized (EUA) SARS-CoV-2 RT-PCR assay at ARUP Laboratories (ARUP). Remnant serum or plasma samples from these individuals were collected from March to May 2020 based on availability and in accordance with IRB approved protocol 0007275.

A total of 100 healthy donor samples collected prior to August 2019, before the start of the outbreak, were used to evaluate clinical specificity. This included de-identified residual serum from 80 adult (age range 20-68 years old, 49% female) and 20 pediatric patients (age range 2-18 years old, 40% female). An additional 37 samples from individuals (age range 1-93, 49% female) with respiratory illnesses other than COVID-19 obtained between February and March 2020 were included to assess cross-reactivity (analytical specificity).

All specimens in each panel were tested using all three IgG immunoassays. The Abbott SARS-CoV-2 IgG assay was performed on the Abbott Architect i2000 (Abbott Laboratories Inc, Abbott Park, IL) according to the manufacturer's instructions. This is a qualitative chemiluminescent microparticle immunoassay that detects IgG to the nucleocapsid protein of SARS-CoV-2. According to the manufacturer, there is a direct relationship between the amount of SARS-CoV-2 specific IgG in the sample and the calculated signal of the sample divided by the signal of the calibrator (S/C) index. A result is considered positive if the S/C is greater than or equal to 1.4 

Liaison SARS-CoV-2 S1/S2 IgG assay (DiaSorin, Stillwater, MN) was performed on the DiaSorin XL according to the manufacturer's instructions. This is a qualitative indirect chemiluminescent immunoassay that detects IgG bound to recombinant spike protein S1 and S2 domains. Antibody concentration is expressed as arbitrary units (AU/ml) with an assay range up to 400 AU/ml. Results are interpreted as positive if the AU/ml are greater than or equal to 15.0. The EUROIMMUN assay was performed manually according to the manufacturer's instructions using the Anti-SARS-CoV-2 IgG Enzyme Linked Immunosorbent Assay (EUROIMMUN US, Mountain Lakes, NJ). The assay format is a 96-well microtiter plate coated with SARS-CoV-2 recombinant S1 spike protein produced in HEK 293 cells. Results are calculated as the ratio between the OD of the sample and the OD of the calibrator. This sample to calibrator ratio is interpreted 6 as negative when the ratio is below 0.8, borderline if the ratio is between 0.8 and 1.1, and positive if the ratio is greater than or equal to 1.1. Results were interpreted according to the manufacturer's cutoffs, unless otherwise indicated.

All specimens were tested for neutralizing antibody at Vyriad, Inc (Rochester, MN) using their Immuno-CoV assay (1). This is a quantitative test that measures neutralizing antibodies against a recombinant Vesicular Stomatitis Virus (VSV) encoding the spike glycoprotein of SARS-CoV-2. The neutralizing antibody concentration is expressed as virus neutralizing units (VNU) and is determined using a calibration curve consisting of an anti-SARS-CoV-2 monoclonal antibody spiked into pooled seronegative serum.

VNUs correlate with plaque reduction neutralization titer PRNTEC50 values.

Statistical analysis was performed using GraphPad Prism (version 8.4.2). Sensitivity was calculated using molecular testing as the reference method. Specificity was calculated using samples from healthy individual. Borderline results on the EUROIMMUN IgG assay were considered negative, unless otherwise indicated.

Sensitivity was calculated based on testing 35 specimens from 7 COVID-19 RT-PCR positive individuals. Differences in sensitivity observed between the three assays are summarized in Table 1 . Of the 10 samples collected less than 7 days post RT-PCR confirmed diagnosis, three samples were not detected by any of the assays. One additional sample, collected day 5 post RT-PCR, was positive on the Abbott assay but negative by DiaSorin and EUROIMMUN. The S/C ratio was 1.49 (positive cutoff for this assay is 1.4). The remaining 25 samples collected more than seven days post RT-PCR were positive on all three platforms. Considering all samples tested, regardless of timing post diagnosis by RT-PCR, the sensitivity of the Abbott assay was 91.43% (95% CI 76.94% to 98.20%), while the sensitivity of the DiaSorin and EUROIMMUN assays were both 88.57% (73.26% to 96.80%).

The clinical specificity for the three assays was calculated using 100 samples from selfreported healthy individuals collected prior to the pandemic are presented in Table 1 . Table 2) .

Given that the current prevalence of SARS-CoV-2 infection is low in some areas in the US, there is a high likelihood of false positives, even for assays that have high specificity. With this in mind, we calculated the positive and negative predictive values for each of the assays, assuming a prevalence of COVID-19 of 5% and using our sensitivity and specificity findings. Under this assumption, the negative predictive values were similar for all three platforms 99.55%, 99.40%, and 99.36% for Abbott, DiaSorin, and EUROIMMUN, respectively. The positive predictive values were similar for Abbott and DiaSorin (82.79% and 82.34% respectively), but much lower for EUROIMMUN (43.72%). To improve the positive predictive value of testing in low prevalence settings, the CDC currently recommends an orthogonal testing algorithm, such that samples that test positive by one assay are tested with another assay that has a different antigen or format (2) . Using the online calculator provided with the CDC guidance statement, at a prevalence of 5%, the estimated positive predictive value increases to >98% regardless of which two tests are used. In addition, less than 2% of samples would have discordant results using this type of algorithm with any two of these assays.

When compared against each other, the overall agreement for Abbott and DiaSorin was 98.26% with a positive percent agreement of 96.88% and a negative percent agreement of 98.57% ( Table 2) 

We next measured neutralizing antibody responses against SARS-CoV-2 using the same panels of samples to assess the relationship between neutralizing antibodies and IgG as determined by the three commercial assays. Among the 35 samples from COVID-19 RT-PCR positive individuals, 32 tested positive for neutralizing antibodies, two tested negative, and one had insufficient volume to perform testing. The two samples that tested negative were collected within a week of confirmation by RT-PCR and were negative for IgG on all three assays. All 100 samples, comprising the panel used to determine specificity, tested negative for neutralizing antibodies. Among the 37 samples included in the cross-reactivity panel, 29 tested negative, 6 tested positive for neutralizing antibodies and two had insufficient volume for testing. All six that tested positive had very low neutralizing antibody results, which were near the limit of detection of the assay. Five of these tested negative on all three of the IgG assays, the other was negative on Abbott (S/C = 0.04) and DiaSorin (AU/ml = 4.72) and indeterminate on EUROIMMUN (S/C = 1.0).

We also performed receiver-operating characteristic (ROC) area under the curve (AUC) analysis for the Abbott, DiaSorin, and EUROIMMUN IgG assays using the qualitative neutralizing antibody result as the reference standard. The area under the curve was high for all three assays (0.94, 0.97, 0.93 for Abbott, DiaSorin and EUROIMMUN respectively) (Figure 2) . At the manufacturer derived cutoffs ( Table 3) .

Because we had samples drawn from the same individuals at various time points, we were also able to analyze the antibody kinetics as determined by each assay. The range of days post RT-PCR confirmation was 0 to 21 days post RT-PCR. Importantly, once an assay detected an IgG response, all subsequent time points were also positive.

Across all three commercial IgG assays the magnitude of the IgG response was low within the first week, then generally increased over time, reaching a plateau around two weeks post RT-PCR. The neutralizing antibody responses did not follow the same temporal kinetics and the response was quite variable between different individuals and did not necessarily correlate with IgG serial values (Figure 3 ).

In this study we evaluated three commercially available SARS CoV-2 IgG serologic tests for sensitivity, specificity, and correlation with neutralizing antibodies. Sensitivity studies showed that there were no significant differences between the assays, as indicated by the overlapping confidence intervals. The overall sample set was small and the observed difference in sensitivity between the assays was due to one sample collected day 5 post RT-PCR that was detected by Abbott (borderline result) and failed to be detected by both DiaSorin and EUROIMMUN. Specificity studies showed EUROIMMUN had more false positive results in healthy individuals, as well as in the cross-reactivity study, even when considering borderline results as negative. An increased number of false positive results have been observed by our laboratory and others (3, 4) . Overall, data from this study showed that clinical sensitivity and specificity of each of the three commercial IgG assays evaluated were generally comparable and agreed with data provided by the FDA and others (4) (5) (6) (7) (8) (9) (10) . This is despite the fact that these assays use different methods (chemiluminescent immunoassay, ELISA) and different antigens (Abbott -nucleocapsid protein, DiaSorin -S1 and S2 domain of spike protein, EUROIMMUN -S1 domain of spike protein). While there has been a concern of false-positives due to cross-reactivity with the common human coronaviruses, few if any studies have shown this to be the case (11) . This may be because both the spike and nucleocapsid proteins only share about 30% amino acid identity with the common human coronaviruses (12) .

In low prevalence settings, even a test with high specificity will generate false-positive results. Based on our analysis, the Abbott assay would likely generate the fewest falsepositive results. To reduce the likelihood of reporting a false-positive, the CDC has suggested performing orthogonal testing. Based on the performance characteristics we and others have observed, this strategy increases the positive predictive value of testing while maintaining a low likelihood of discordant results. The downside of this approach is the additional cost and turnaround time (13) . However, there is also a possibility that discrepant results between assays may represent a true prior infection and may be due to differences in sensitivity between antibody assays, particularly, if the sample is collected too soon after symptom onset or a difference in the antigen used to capture antibodies. Seroconversion to some antigens, but not others has been observed in at least one study (8) . However, currently, there is no indication that using antigens representing the nucleocapsid or spike to detect IgG antibodies significantly affects assay performance.

In addition, although we detected seroconversion in all subjects by seven days post RT-PCR using all three assays, some studies have identified individuals in whom seroconversion was delayed (10, 14) or did not occur at all (4, 15, 16) . We and others found that once an individual seroconverts, the IgG response does increase over the first few weeks of infection and then plateaus (4, 10, (15) (16) (17) . As the coronavirus pandemic continues, additional data will need to be collected to determine the kinetics and longevity of the IgG response.

Finally, because neutralizing antibodies (Nab) are implicated in providing possible immunity, we aimed to correlate neutralizing antibodies with the IgG values obtained with the different IgG assays. The data demonstrated that in our study cohort, neutralizing antibodies developed early during the course of infection and the presence of IgG predicted a neutralizing antibody response. The concentration of neutralizing antibodies did not correlate well with the level of anti-SARS-CoV-2 IgG, as measured by either of the three assays. Others have also reported the early development of neutralizing antibody responses but did find a correlation between neutralizing antibody responses and IgG (3, 7, 12, 17) . Wide variation in the degree of neutralizing activity has also been reported (8, 18) . Furthermore, our data suggest that the current cutoffs for Abbott and DiaSorin are appropriate for discriminating neutralizing antibody-positive from neutralizing antibody-negative samples. However, based on our data, an adjustment of the cutoff for EUROIMMUN would be needed in order for this test to perform similarly. There is increasing evidence that neutralizing antibodies may mediate at least some degree of immunity. SARS-CoV-2 specific neutralizing antibodies have been detected in individuals who have recovered from COVID-19 and COVID-19 convalescent plasma (CCP) from recovered individuals has shown promise as a treatment (19) (20) (21) . In a Rhesus macaque model, animals developed both binding and neutralizing antibodies and were protected from reinfection by SARS-CoV-2 (22, 23) .

commercial assays is an important area of study, as it may implications for clinical decisions when a vaccine becomes available for example.

This study is limited due to the small number of samples from SARS-CoV-2 positive patients and the lack of samples from patients infected with other coronaviruses.

However, our results are consistent with other similar studies and with the manufacturers' performance claims. Another limitation is the retrospective study design and lack of clinical data, which does limit the analysis of the kinetics of the IgG and neutralizing antibody response. It is also not clear, based on our data, whether the variation observed between individuals is due to variation in severity of disease and/or immune response to different antigenic targets. Additional studies will be needed to clarify the factors that contribute to the variation in antibody responses and the development of a robust and protective immune response.

In conclusion, our study demonstrated that three EUA designated SARS CoV-2 assays had comparable performance characteristics that were consistent with manufacturer claims. Although not statistically significant due to our sample size, in our study borderline samples were considered negative. Our data also suggest that the cut-offs for Abbott and DiaSorin are appropriate for discriminating neutralizing antibody positive from neutralizing antibody negative samples. Importantly, we found no correlation between IgG Au or S/C values and concentration of neutralizing antibodies. Although, the findings in this study are consistent with manufacturer claims, the study provides useful information for laboratories that are still considering what platform to use for SARS CoV-2 antibody testing. Continued independent evaluation of SARS CoV-2 serologic assays is key as we learn more about individual variation in response to SARS CoV-2 infection, develop vaccines, and as testing guidelines evolve. 

Ratio of the signal to calibrator (S/C or AU/ml) are plotted relative to days post positive RT-PCR for (A) Abbott, (B) DiaSorin, (C) EUROIMMUN, and (D) neutralizing antibody. Dashed lines indicate the manufacturer derived cutoff for the assay. For neutralizing antibodies, the cutoff is 0 VNU. Each plotted line represents an individual patient and plotted using the same color for all graphs. 

Development and validation of immuno-cov™: A high-throughput clinical assay for detecting antibodies that neutralize sars-cov-2

Interim guidelines for COVID-19 antibody testing

Evaluation of commercial and automated sars-cov-2 igg and iga elisas using coronavirus disease (COVID-19) patient samples

Clinical performance of two sars-cov-2 serologic assays

EUA authorized serology test performance

Clinical and analytical performance of an automated serological test that identifies s1/s2 neutralizing igg in covid-19 patients semiquantitatively

Towards the next phase: Evaluation of serological assays for diagnostics and exposure assessment

Serological analysis of new york city covid19 convalescent plasma donors

Evaluation of the euroimmun anti-sars-cov-2 elisa assay for detection of iga and igg antibodies

Performance characteristics of four highthroughput immunoassays for detection of igg antibodies against sars-cov-2

Review of current advances in serologic testing for COVID-19

Severe acute respiratory syndrome coronavirus 2-specific antibody responses in coronavirus disease 2019 patients

Sars-cov-2 seroprevalence and neutralizing activity in donor and patient blood from the san francisco bay area

Symptoms and immunoglobulin development in hospital staff exposed to a sarscov-2 outbreak

Antibody responses to sars-cov-2 in COVID-19 patients: The perspective application of serological tests in clinical practice

Rapid generation of neutralizing antibody responses in COVID-19 patients

Neutralizing antibody responses to sars-cov-2 in a COVID-19 recovered patient cohort and their implications

Deployment of convalescent plasma for the prevention and treatment of COVID-19

Effectiveness of convalescent plasma therapy in severe COVID-19 patients

Treatment of 5 critically ill patients with COVID-19 with convalescent plasma

Reinfection could not occur in sars-cov-2 infected rhesus macaques

Sarscov-2 infection protects against rechallenge in rhesus macaques

We would like to thank Hailey Wells for her help with collecting and aliquoting samples.We would like to thank Abbott and DiaSorin for providing kits for evaluation.