key: cord-0698119-tn0799qx
authors: Dörschug, Anja; Schwanbeck, Julian; Hahn, Andreas; Hillebrecht, Anke; Blaschke, Sabine; Mese, Kemal; Groß, Uwe; Dierks, Sascha; Frickmann, Hagen; Zautner, Andreas E.
title: Comparison of Five Serological Assays for the Detection of SARS-CoV-2 Antibodies
date: 2021-01-06
journal: Diagnostics (Basel)
DOI: 10.3390/diagnostics11010078
sha: cb03fb764a91aea4955efcdbebd9ad7f08e3fa85
doc_id: 698119
cord_uid: tn0799qx

Serological assays can contribute to the estimation of population proportions with previous immunologically relevant contact with the Severe Acute Respiratory Syndrome Corona Virus 2 (SARS-CoV-2) virus. In this study, we compared five commercially available diagnostic assays for the diagnostic identification of SARS-CoV-2-specific antibodies. Depending on the assessed immunoglobulin subclass, recorded sensitivity ranged from 17.0% to 81.9% with best results for immunoglobulin G. Specificity with blood donor sera ranged from 90.2% to 100%, with sera from EBV patients it ranged from 84.3% to 100%. Agreement from fair to nearly perfect was recorded depending on the immunoglobulin class between the assays, the with best results being found for immunoglobulin G. Only for this immunoglobulin class was the association between later sample acquisition times (about three weeks after first positive PCR results) and positive serological results in COVID-19 patients confirmed. In conclusion, acceptable and comparable reliability for the assessed immunoglobulin G-specific assays could be shown, while there is still room for improvement regarding the reliability of the assays targeting the other immunoglobulin classes.

The Corona Virus Disease 2019 (COVID-19) pandemic, caused by Severe Acute Respiratory Syndrome Corona Virus 2 (SARS-CoV-2) and starting in Wuhan, China, in 2019 [1] , remains the most threatening global public health menace of the year 2020. On a global scale, diagnosis, containment and surveillance of the disease were considered issues of major concern.

For containment purposes, direct proof of virus RNA in respiratory samples is of central importance, so molecular tools for the detection of SARS-CoV-2 virus RNA were rapidly introduced and evaluated [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] at early stages of the pandemic. However, detectable amounts of virus RNA can quickly decline over the course of the disease [12, 15] , so infected individuals with lacking or mild symptoms have a good chance of going undetected if surveillance is just based on molecular diagnostic approaches.

To close this diagnostic gap, there was an early focus on the implementation of antibody-based surveillance. By doing so, a more realistic view on the real dimensions of the spread of SARS-CoV-2 in the population was aspired to and numerous benchtopbased and point-of-care-testing (POCT)-based serological assays were introduced . However, the limitations of this strategy rapidly emerged as well. While specificity was usually at least >95% in the geographic regions where the tests were developed, a broad variety of sensitivities, usually between 70% and 90% depending on the subpopulation assessed [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] [28] [29] and sometimes even lower [47] , were recorded with an optimum sensitivity two weeks after infection [29] and decreasing positivity rates afterwards [30] . Further, agedependency of serological sensitivity has been demonstrated [33] next to higher specificity but lower sensitivity of neutralizing antibodies compared to non-neutralizing ones [34] .

More than this, it has become obvious that immunologically relevant contacts with SARS-CoV-2, i.e., viral in-vivo replication leading to any adaptive immune response, can occur completely without the induction of specific antibodies but just with SARS-CoV-2-specific T cell responses [48] . This is well in line with observed low seropositivity in patients with previous PCR-confirmed COVID-19 as observed in a recent study [47] . It is discussed that specific T cells may provide protection against SARS-CoV-2 even in the absence of antibodies [49] .

However, the method-immanent imperfect sensitivity of diagnostic methods does not necessarily mean that they cannot be used for surveillance purposes. If diagnostic accuracy adjusted methods [50, 51] are applied, the true prevalence can be estimated even based on a test with imperfect but known test characteristics in epidemiological assessments. Accordingly, the evaluation of test characteristics of serological tests for antibodies against SARS-CoV-2 is still an issue of epidemiological relevance.

In the study performed here, five commercially available serological assays targeting SARS-CoV-2-specific antibodies were assessed. The comparison comprised previously described products such as the assays from EUROIMMUN (Lübeck, Germany) [26, 27, [35] [36] [37] [38] [39] [40] [41] [42] [43] [44] [45] [46] [47] 52, 53] , Roche (Basel, Switzerland) [54] [55] [56] [57] [58] [59] , Mikrogen (Neuried, Germany) [60] [61] [62] , and Virotech Diagnostics (Rüsselsheim am Main, Germany) [46, 55, 63] as well as a newly evaluated kit from Vircell (Vircell, Granada, Spain). As positive controls, residual serum samples from patients with PCR-confirmed COVID-19 were used, while samples from blood donors and patients with Epstein-Barr virus (EBV) were applied as negative controls. With this approach, test characteristics should be calculated to guide the application of the serological assays for both diagnostic and surveillance purposes. Surveillance purposes include population prevalence studies and diagnostic purposes include the confirmation of previous infections with SARS-CoV-2 in individual patients.

In the same way as described before [47] , three different serum sample collections were assessed comprising one collection of positive controls and two negative control collections. The positive control sample collection consisted of samples from 148 PCR-confirmed COVID-19 patients, on whom PCR had been performed from nasopharyngeal swabs. Due to limited sample volumes, between 100 and 148 samples were assessed with each assessed serological assay. For those samples from PCR-confirmed COVID-19 patients, the time between the positive PCR result and the acquisition of the serum samples was documented in 94 out of 148 (63.5%) instances, with time periods ranging from −2 to 120 days (median: 11 days, mean: 20.3 days, standard deviation (SD): 24.2 days). The 54 specimen donors with no clearly documented time between the positive PCR result and the acquisition of the serum sample for data protection reasons were candidates for convalescent serum donation who were referred via the Department of Transfusion Medicine of the University Medical Center Göttingen. However, the time between the positive PCR result and the acquisition of the serum sample from these specimens is at least 4 weeks.

The first negative control collection consisted of samples from 152 blood donors acquired in 2015 and thus well before the COVID-19 pandemic began. Sufficient sample volumes were available to allow testing of 50 to 152 out of those negative controls per assay.

The second negative control collection contained 32 Epstein-Barr virus (EBV)-positive serum samples, which had been collected at the beginning of 2020 when the likelihood of COVID-19 infections was still extremely low in Germany. Sufficient volumes for the testing of 30 to 32 samples out of those second negative control population per test assay were available. This third serum collection was included to assess the effects of polyclonal B cell stimulation. The used sample volumes were residual sample materials from routine diagnostic procedures performed at the University Medical Center Göttingen.

As the ethical board allowed only a completely anonymized use of sample materials for test comparison purposes, no patient-specific information can be provided, necessarily resulting in an unavoidable violation of the Standards for Reporting of Diagnostic Accuracy (STARD) criteria [64] .

The compared serological assays comprised:

The EUROIMMUN COVID-19 IgG/IgA assay (EUROIMMUN, Lübeck, Germany; referred to as "EUROIMMUN assay" in the following); •

The Roche Cobas Elecsys Anti-SARS-CoV-2 assay (Roche, Basel, Switzerland; referred to as "Roche assay" in the following); •

The Mikrogen recomWell SARS-CoV-2 IgG assay (Neuried, Germany; referred at as "Mikrogen assay" in the following); •

The Virotech Diagnostics assay VIROTECH SARS-CoV-2 IgA/IgM/IgG ELISA (Rüsselsheim am Main, Germany; referred to as "Virotech assay" in the following); •

The Vircell COVID-19 ELISA IgG/IgM+IgA assay (Vircell, Grenada, Spain; referred to as Vircell assay in the following).

All assays were exactly performed as demanded by the manufacturers' instructions.

To characterize the positive control samples, respiratory sample materials from the patients with suspected or confirmed COVID-19 were analyzed using real-time PCR for SARS-CoV-2 in a two-step procedure. In step one, screening was performed applying the Genesig Real-Time PCR Coronavirus (COVID-19) assay (Primerdesign Ldt., Chandlers Ford, UK). In a second step, first-time positive results were confirmed using automated Cepheid Xpert Xpress SARS-CoV-2 PCR (Cepheid, Sunnyvale, CA, USA). Both SARS-CoV-2-specific PCR assays were performed exactly as described by the manufacturers.

Due to the restricted number of samples, descriptive statistical analysis was performed only. With the positive control sample collection taken from patients with PCR-confirmed COVID-19, sensitivity was calculated. To assess the effect of the number of days between positive PCR results and the serum sample acquisition, Wilcoxon rank sum testing was calculated applying the software Stata/IC 15.1 for macOS 64-bit Intel (College Station, TX, USA).

With the two negative control sample collections taken from the blood donors and the EBV patients, specificity of the serological assays was assessed. Positive and negative predictive values were calculated for two exemplary populations with 1% and 10% prevalence, respectively.

Cohen's kappa [65] was calculated to assess the agreement between the test assays with the categories of poor (below 0.00), slight (0.00-0.20), fair (0.21-0.40), moderate (0.41-0.60), substantial (0.61-0.80) and almost perfect (0.81-1.00) for the immunoglobulin subclasses as well as across classes targeting the meta-structure "any SARS-CoV-2-related antibodies".

In line with common conventions of descriptive statistics, standard deviation (SD) was calculated for mean values and interquartile range (IQR) for median values. Additional calculation of median values next to mean values was performed to indicate left-or rightshifted distributions of values within the different groups-i.e., information which would have gone unreported otherwise.

Samples were not excluded if individual data points were missing due to insufficient amounts of sample material as stated above.

The study was ethically approved by the institutional ethics board of the University Medical Center Göttingen (Application number 21/05/20), allowing the use of residual sample materials for test comparison purposes.

Assessed by immunoglobulin classes, observed sensitivities of the evaluated test assays as recorded exclusively with samples from patients with previous positive results of SARS-CoV-2 PCR differed considerably. For IgG, sensitivities ranged from 63.0% to 81.9%, for IgA from 21.0% to 81.8%, for IgM from 17.0% to 20.0% and for the overarching assays measuring different immunoglobulin classes from 66.6% to 76.1%. The values slightly varied depending on whether borderline results were interpreted as positive or as negative. Details of individual assays are provided in Table 1 . 1 Borderline results were counted as positive. 2 Borderline results were counted as negative. N = numbers. CI = confidence interval. IgA/G/M = immunoglobulin A/G/M.

Significance for higher likelihood of detecting positive signals after prolonged duration of about 3 weeks between the first recorded positive PCR result and serum acquisition was calculated for all IgG-specific assays and Roche's immunoglobulin class-overarching assay. When focusing on the median instead of the mean numbers of days, significance for higher likelihood of positive results for SARS-CoV-2-specific IgG was detectable after about two weeks in most instances, indicating a left-shifted distribution of the recorded values. In contrast, the Virotech IgA assay was associated with a higher reliability after a short time period of little more than a week. For the other assays targeting IgA or IgM, no time-dependency could be confirmed. Details are provided in Table 2 .

Recorded specificity with blood donor samples, which had been collected prior to the COVID-19 pandemic, used as negative control samples ranged from 90.2% to 100%. Recorded specificities <95% were seen for the Mikrogen IgG assay only if borderline results were counted as positives. For the EUROIMMUN IgA assay, the Vircell IgM/IgA assay and Virotech IgM assay, specificity remained below this threshold even if borderline samples were considered as negative. Details are provided in Table 3 . 

Recorded specificity with samples from EBV-positive patients collected at the very beginning of the COVID-19 pandemic (used as negative samples) ranged from 84.3% to 100%. Recorded specificities <95% were seen for the Mikrogen IgG assay and the Vircell IgG assay only if borderline results were counted as positives. For the Vircell IgM/IgA assay, specificity remained below 90% even if borderline samples were considered negative. Details are provided in Table 4 . 1 Borderline results were counted as positive; 2 borderline results were counted as negative; N = numbers; CI = confidence interval; IgA/G/M = immunoglobulin A/G/M; n.e. = not estimable.

Based on the results as shown above, positive and negative predictive values were calculated in a mathematical modelling for two hypothetical exemplary populations with 1% and 10% prevalence of SARS CoV 2-specific antibodies. Over the different assessed assays, the negative predictive value was excellent with 99.1% till 99.8% for the 1% prevalence population but dropped to 91.3% till 97.9% for the 10% prevalence population. In contrast, for the 1% prevalence population, positive predictive values ranged from 3.9% till 100%, while this range was narrowed to 30.7% till 100% for the 10% prevalence population. Details are provided in Table 5 . 

Almost perfect agreement (0.81-1.00) between the compared assays according to the definitions by Landis and Koch [65] was observed for the IgG immunoglobulin class only. For immunoglobulin class-overarching comparisons, only moderate (0.41-0.60) to substantial (0.61-0.80) agreement could be seen with worse results if IgM was included. For the IgA immunoglobulin class, agreement even dropped to the fair level (0.21-0.40). Details are provided in Table 6 . 

The study was conducted to provide information on performance characteristics of commercially available serological assays. Thus, it contributes to previously described assessments [52] [53] [54] [55] [56] [57] [58] [59] [60] [61] [62] [63] and provides an additional piece of the puzzle in terms of interpreting the results of serological approaches for the retrospective diagnosis of infections with SARS-CoV-2.

One major result of the study is the confirmation of acceptable sensitivity and good specificity, associated with nearly perfect agreement, for the assays detecting SARS-CoV-2 antibodies of the immunoglobulin class IgG. While interpreting the less than perfect sensitivity, one has to consider the variance in the periods of time between the first recorded positive SARS-CoV-2 PCRs and sample acquisition for serological assessments. In case of longer periods of about 3 weeks, all IgG assays scored significantly better than in the case of shorter periods. This is well in line with previous reports [29] . Further, detectable immunoglobulins are not always detectable in individuals with confirmed immunologically relevant contact with SARS-CoV-2 [47] [48] [49] . Though a more detailed assessment of clinical data of SARS-CoV-2 infected patients without recorded seroconversion would have been desirable, the strict focus of the study design on the test comparison prevented this option, an undeniable limitation of this approach.

Focusing on other immunoglobulin classes such as IgM and IgA, the performance characteristics of the assessed assays were considerably worse, also confirming previous results [47] . This phenomenon was shown to affect sensitivity, specificity and also intertest agreement. A comparably good specificity as observed for the Virotech IgM and IgA assays was traded for particular poor sensitivity in these assays, while the other test producers seem to have aimed at a compromise between sensitivity and specificity. Thereby, sensitivity of the Virotech IgA assay was better in the early stages of infection about one week after the first positive PCR test, a phenomenon which could be shown for no other assay.

Polyclonal B-cell proliferation, as associated with EBV infection [66] , particularly affected the Vircell IgM/IgG assay. For the other assays, specificity with sera from blood donors and with sera from EBV patients was quite comparable.

The study has a number of limitations beyond the one stated above. Firstly, limited volumes of residual sample materials did not allow the assessment of all samples with all assays. Secondly, ethical considerations did not allow the inclusion of patient data which is an undeniable violation of the recommendations by the STARD guideline [64] . Thirdly, the assessed assays are not representative of all respective products available on the market. Fourthly, not all immunoglobulin classes were represented with equal numbers of assays, preventing the calculation of Cohen's kappa for immunoglobulin class M. Fifthly, economic restrictions limited the assessments to reasonable but still low sample numbers.

In spite of these limitations, the study provides another piece in the diagnostic puzzle, allowing a better interpretation of results of serological assays targeting antibodies against SARS-CoV-2.

This study indicates acceptable reliability of immunoglobulin class G-based serology for SARS-CoV-2-specific antibodies with a variety of test assays with increased sensitivities about 3 weeks after first positive PCR results compared with earlier time points. Assays for other immunoglobulin classes scored worse with less obvious associations to the time points of testing. 

A pneumonia outbreak associated with a new coronavirus of probable bat origin

Detection of SARS-CoV-2 by Use of the Cepheid Xpert Xpress SARS-CoV-2 and Roche cobas SARS-CoV-2 Assays

Test Agreement between Roche Cobas 6800 and Cepheid GeneXpert Xpress SARS-CoV-2 Assays at High Cycle Threshold Ranges

Multicenter Evaluation of the Cepheid Xpert Xpress SARS-CoV-2 Test

Multi-center evaluation of cepheid xpert ® xpress SARS-CoV-2 point-of-care test during the SARS-CoV-2 pandemic

Detection of low levels of SARS-CoV-2 RNA from nasopharyngeal swabs using three commercial molecular assays

Comparison of Cepheid Xpert Xpress and Abbott ID Now to Roche cobas for the Rapid Detection of SARS-CoV-2

Evaluation of the automated cartridge-based ARIES SARS-CoV-2 Assay (RUO) against automated Cepheid Xpert Xpress SARS-CoV-2 PCR as gold standard

Comparison of Abbott ID Now and Abbott m2000 Methods for the Detection of SARS-CoV-2 from Nasopharyngeal and Nasal Swabs from Symptomatic Patients

Comparison of Abbott ID Now, DiaSorin Simplexa, and CDC FDA Emergency Use Authorization Methods for the Detection of SARS-CoV-2 from Nasopharyngeal and Nasal Swabs from Individuals Diagnosed with COVID-19

Clinical Evaluation of Three Sample-to-Answer Platforms for Detection of SARS-CoV-2

Performance of Abbott ID Now COVID-19 Rapid Nucleic Acid Amplification Test Using Nasopharyngeal Swabs Transported in Viral Transport Media and Dry Nasal Swabs in a New York City Academic Institution

Trends and Innovations in Biosensors for COVID-19 Mass Testing

Type I and Type III Interferons-Induction, Signaling, Evasion, and Application to Combat COVID-19

A Method to Prevent SARS-CoV-2 IgM False Positives in Gold Immunochromatography and Enzyme-Linked Immunosorbent Assays

Diagnostic efficacy of anti-SARS-CoV-2 IgG/IgM test for COVID-19: A meta-analysis

Clinical evaluation of a rapid colloidal gold immunochromatography assay for SARS-Cov-2 IgM/IgG

Diagnostic performance of immunochromatography assay for rapid detection of IgM and IgG in coronavirus disease 2019

The different tests for the diagnosis of COVID-19-A review in Brazil so far

A new and rapid approach for detecting COVID-19 based on S1 protein fragments

Rapid Detection of IgM Antibodies against the SARS-CoV-2 Virus via Colloidal Gold Nanoparticle-Based Lateral-Flow Assay

Effect of heat inactivation of blood samples on the efficacy of three detection methods of SARS-CoV-2 antibodies. Nan Fang Yi Ke Da Xue Xue Bao

Clinical validation of an immunochromatographic SARS-Cov-2 IgM/IgG antibody assay with Japanese cohort

Diagnostic performance of a SARS-CoV-2 IgG/IgM lateral flow immunochromatography assay in symptomatic patients presenting to the emergency department

Diagnostic accuracy of Augurix COVID-19 IgG serology rapid test

Head-to-Head Accuracy Comparison of Three Commercial COVID-19 IgM/IgG Serology Rapid Tests

Concordance between two rapid diagnostic tests for the detection of antibodies against SARS-CoV-2

Diagnostic performance of COVID-19 serology assays

In vitro diagnostics of coronavirus disease 2019: Technologies and application

Diagnostics for SARS-CoV-2 detection: A comprehensive review of the FDA-EUA COVID-19 testing landscape

COVID-19 Serological Tests: How Well Do They Actually Perform? Diagnostics

Antibodies to coronaviruses are higher in older compared with younger adults and binding antibodies are more sensitive than neutralizing antibodies in identifying coronavirus-associated illnesses

Antigenic crossreactivity between severe acute respiratory syndrome-associated coronavirus and human coronaviruses 229E and OC43

Performance of six SARS-CoV-2 immunoassays in comparison with microneutralisation

Clinical Performance of Two SARS-CoV-2 Serologic Assays

Evaluation of the EUROIMMUN Anti-SARS-CoV-2 ELISA Assay for detection of IgA and IgG antibodies

Validation of a commercially available SARS-CoV-2 serological immunoassay

Evaluation of commercial and automated SARS-CoV-2 IgG and IgA ELISAs using coronavirus disease (COVID-19) patient samples

Performance Characteristics of Four High-Throughput Immunoassays for Detection of IgG Antibodies against SARS-CoV-2

Quantification of SARS-CoV-2 antibodies with eight commercially available immunoassays

Comparison of test performance of commercial anti-SARS-CoV-2 immunoassays in serum and plasma samples

Diagnostic performances and thresholds: The key to harmonization in serological SARS-CoV-2 assays?

Evaluation of three fully-automated SARS-CoV-2 antibody assays

Evaluation of six commercial mid to high volume antibody and six point of care lateral flow assays for detection of SARS-CoV-2 antibodies

Comparison of the diagnostic sensitivity of SARS-CoV-2 nucleoprotein and glycoprotein-based antibody tests

Evaluation of the Xiamen AmonMed Biotechnology rapid diagnostic test COVID-19 IgM/IgG test kit (Colloidal gold)

Intrafamilial Exposure to SARS-CoV-2 Induces Cellular Immune Response without Seroconversion

SARS-CoV-2-reactive T cells in healthy donors and patients with COVID-19

Estimating prevalence from the results of a screening test

On detection thresholds-a review on diagnostic approaches in the infectious disease laboratory and the interpretation of their results

Assessment of SARS-CoV-2 serological tests for the diagnosis of COVID-19 through the evaluation of three immunoassays: Two automated immunoassays (Euroimmun and Abbott) and one rapid lateral flow immunoassay

Evaluation of Abbott anti-SARS-CoV-2 CMIA IgG and Euroimmun ELISA IgG/IgA assays in a clinical lab

Clinical performance of different SARS-CoV-2 IgG antibody tests

Brief clinical evaluation of six high-throughput SARS-CoV-2 IgG antibody assays

Analytical and Clinical Evaluation of the Automated Elecsys Anti-SARS-CoV-2 Antibody Assay on the Roche cobas e602 Analyzer

Evaluation of the SARS-CoV-2-IgG response in outpatients by five commercial immunoassays

Weak correlation between antibody titers and neutralizing activity in sera from SARS-CoV-2 infected subjects

Comparison of eight commercial, high-throughput, automated or ELISA assays detecting SARS-CoV-2 IgG or total antibody

Kinetics of Nucleo-and Spike Protein-Specific Immunoglobulin G and of Virus-Neutralizing Antibodies after SARS-CoV-2 Infection

Antibody response against SARS-CoV-2 spike protein and nucleoprotein evaluated by four automated immunoassays and three ELISAs

Comparison of four new commercial serologic assays for determination of SARS-CoV-2 IgG

Differences of SARS-CoV-2 serological test performance between hospitalized and outpatient COVID-19 cases

STARD 2015 guidelines for reporting diagnostic accuracy studies: Explanation and elaboration

The measurement of observer agreement for categorical data

Epstein-Barr virus, infectious mononucleosis, and posttransplant lymphoproliferative disorders

Acknowledgments: Patient data collection was supported by the study nurse Irina Pavlova, Interdisciplinary Emergency Department, University Medical Center Göttingen.

The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.