key: cord-0792813-tj7iq4bj authors: Haselmann, Verena; Kittel, Maximilian; Gerhards, Catharina; Thiaucourt, Margot; Eichner, Romy; Costina, Victor; Neumaier, Michael title: Comparison of test performance of commercial anti-SARS-CoV-2 immunoassays in serum and plasma samples date: 2020-07-09 journal: Clin Chim Acta DOI: 10.1016/j.cca.2020.07.007 sha: 00f5685e354b2e8f549f7c5161362bfb446210c4 doc_id: 792813 cord_uid: tj7iq4bj Abstract Background For epidemiologic, social and economic reasons, assessment of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection prevalence and immunity are important to adapt decisions to current demands. Hence, immunoassays for detection of anti-SARS-CoV-2 antibodies are introduced rapidly without requiring FDA emergency use authorization approval. Thus, evaluation of test performance predominantly relies on laboratories. This study aimed to evaluate the test performance of recently launched commercial immunoassays in serum and plasma samples. Methods 51 serum samples from 26 patients with confirmed SARS-CoV-2 infection after end of quarantine and 25 control patients were analyzed using anti-SARS-CoV-2 IgG immunoassays from Roche, Euroimmun and Epitope to assess diagnostic sensitivity and specificity. 20 matching pairs of serum and plasma samples were included to analyze comparability between different specimens. Results Overall, a diagnostic sensitivity of 92.3%, 96.2-100% and 100% with a respective diagnostic specificity of 100%, 100% and 84-86% for the immunoassays from Roche, Euroimmun and Epitope were determined. In total, 84-96 % of samples were correctly classified as negative and 92.3-95.2% as positive. The level of concordance between plasma- and serum-based testing diverged between the assays (Epitope r2=0.97; Euroimmun r2=0.91; Roche r2=0.76). Conclusions The immunoassays from Euroimmun and Roche revealed a higher specificity than the Epitope assay without a substantial drop of diagnostic sensitivity. Significant differences between plasma- and serum-based testing highlights the need for determination of appropriate cut-offs per specimen type. Hence, there is an urgent need for test harmonization and establishment of quality standards for an appropriate use of COVID-19 serological tests. The infection with novel severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) was declared world pandemic by the World Health Organization (WHO) on 12 th of March 2020 [1] . Since then, the number of infections and the global spread continuously increased. As of May 21 st 2020, over five million confirmed SARS-CoV-2 infections have been reported worldwide and more than 330,000 people died due to SARS-CoV-2 caused acute severe respiratory disease, termed Coronavirus disease 2019 (COVID- 19) [2]. As long as no appropriate vaccination is available, the only possibility to reduce the rapid spread of SARS-CoV-2 represents quarantine of infected individuals along with social lockdown/restrictions and enhanced hygiene [3] . The diagnosis of acute infection relies on qRT-PCR based viral detection in respiratory material. As the identification of infected persons is hampered by the high percentage of oligo-or asymptomatic patients [4] and the shortage of test material [5, 6] , the number of infections worldwide is thought to be substantially underestimated [7] [8] [9] . For epidemiologic, social and economic reasons, determination and surveillance of the immune status within the population to estimate SARS-CoV-2 infection prevalence and herd immunity are of upmost importance [10] . Hence, immunoassays for detection of anti-SARS-CoV-2 immunoreactivity are gaining growing attention. Currently, over 100 SARS-CoV-2 antibody assays have been CE-marked under EU Directive 98/79/EC [11] . The available test systems can be discriminated into rapid diagnostic tests (RDT), either antigen-or antibody based, enzyme-linked immunosorbent assays (ELISA) and chemiluminescent immunoassays (CLIA). The global health non-profit Foundation for Innovative New Diagnostics (FIND) provides an overview of available test systems along with their respective market readiness as well as sensitivity and specificity data [12] . Additionally, information about immunological tests with approval in the United States can be obtained from the Center for Health Security of Johns Hopkins University [13] . As of May 21 st 2020, 11 serology tests have received FDA emergency use authorization (EUA) with performance data provided by FDA [14] . In detail, three RDTs (Cellex Inc., ChemBio, Autobio Diagnostics Co. [13] . In agreement with EUA, regulatory requirements have been reduced and FDA stated that in contrast to qRT-PCR based tests for viral detection, EUA approval is not mandatory for serology-based test systems -neither commercial nor laboratory-developed ones [15] . Consequently, laboratories are now forced to perform appropriate validation studies. Taken into consideration the high diversity of available immunoassays [16] , the different materials used for testing (e.g. serum, plasma, sputum), and the limited number of peer-reviewed publications addressing this issue, diagnostic accuracy and optimal use of serological anti-SARS-CoV-2 testing still need to be elucidated [17] . Therefore, we compared the test performance of recently FDA EUA approved immunoassays from Roche and Euroimmun with one for research and surveillance purposes only approved ELISA from Epitope Diagnostics. 26 patients with qRT-PCR confirmed COVID-19 disease after end of quarantine or hospitalization as well as 25 control patients were prospectively recruited at University Medical Center Mannheim, Medical Faculty Mannheim, University of Heidelberg, Germany. The study was approved by the Institutional Review Board and informed written consent was obtained from each subject prior to sample collection and analysis. For evaluation of the medical history, each subject answered a standardized questionnaire. Depending on results, control patients were assigned to five different cohorts: (i) atypical respiratory infection within last three months and either SARS-CoV-2 qRT-PCR negative or not performed (Control 1), (ii) other respiratory viral infection diagnosed (Control 2), (iii) chronic diseases (e.g. autoimmune disease) (Control 3), (iv) contact to a COVID-19 positive patient, but negative SARS-CoV-2 qRT-PCR and no clinical symptoms (Control 4), (v) healthy controls (Control 5). Patient characteristics are provided in Table 1 . From all patients, serum and lithium heparin blood samples were collected. Serum samples were stored at room temperature for at least one hour to allow appropriate clotting. Clotted serum samples and lithium heparin blood samples were centrifuged at 2000 g for 10 minutes at 18°C within 4 hours after sample collection. Serum and lithium heparin plasma was aliquoted and stored at -80°C. Elecsys Anti-SARS-CoV-2 (Lot:496298, Roche, Germany). All tests were performed using the same lot. Tests systems from Euroimmun and Epitope Diagnostics are enzyme-linked immunosorbent assays (ELISA) in a 96-well-plate format detecting IgG directed against the S1 domain of viral spike protein (Euroimmun) and IgG directed against full length nucleocapsid protein (Epitope), respectively. The Euroimmun assay reports the ratio of sample absorbance divided by calibrator absorbance and results are interpreted as positive (ratio ≥ 1.1), borderline (ratio ≥ 0.8 -< 1.1), or negative (ratio < 0.8). The Epitope assay reports the optical density (OD) of the sample corrected by OD of negative control. The cut-offs used for interpretation of assay results All statistical analyses were carried out using Abacus 2.0 (LABanalytics GmbH, www.labanalytics.de, 2016, Germany) and R version 3.0.1 (The R Foundation for Statistical Computing). Results of data analysis are presented as descriptive statistics by mean, 95% confidence interval (CI), standard deviation (SD), and coefficient of variation (CV) as appropriate. Verification studies were performed for all three test systems. Imprecision was determined in duplicates over 4 days, repeatability was assessed by analyzing a negative control, positive control and a patient sample near the positive cut-off for each assay twenty times. Linear and Passing Bablok regression as well as Bland-Altman analysis were determined for method comparison between serum and plasma samples. Between-group differences were assessed by Student´s t -test. For all statistical analyses, p-values less than 0.05 were considered statistically significant. For this study, 51 patients were prospectively enrolled and their serum samples as well as 20 matching lithium heparin plasma samples were analyzed in order to evaluate the test performance of three commercially available test systems for detection of anti-SARS-CoV-2 immunoreactivity. For method verification, within-and between-run imprecision were determined using a positive and negative control as well as a pooled patient sample near the positive cut-off of the respective assays. To assess within-run imprecision, all samples were analyzed as 20 replicates. Due to limited reagent availability, between-run imprecision was determined over 4 days exclusively for the positive and negative control. Results are provided in Table 2 . The Roche assay achieved the highest level of inter-assay precision, whereas the Euroimmun test revealed the highest intra-assay repeatability. Overall, all tests revealed an acceptable precision. Accuracy could not be determined as all three tests are qualitative assays without target values for the provided control samples. Importantly, agreement of qualitative results was 100% for all three immunoassays. In total, anti-SARS-CoV-2 antibodies were determined in 20 matching pairs of serum and lithium heparin plasma samples from 13 patients with previous confirmed SARS-CoV-2 infection and variable clinical presentation and 7 control patients. Results are listed in Supplemental Table 1 . Linear regression analysis of anti-SARS-CoV-2 antibody detection revealed a high level of correlation with r 2 =0.97 for the COVID-19 IgG ELISA from Epitope as depicted in Figure 1 . A good to moderate linear correlation with r²=0.91 and r²=0.76 were revealed for the Euroimmun and Roche test, respectively. In order to exclude a systematic error, a Passing Bablok regression and Bland-Altman analysis were conducted. Results are provided in Table 3 ; plots are displayed in Supplemental Figure 1 . Passing Bablok regression showed a strong to poor positive correlation for both testing specimens for the three different immunoassays with Kendall´s tau ranging from 0.82 (Epitope) to 0.60 (Roche). Additionally, a systematic or proportional error could be excluded for the two ELISAs (95%CI y-intercept= -0.1 -0.006; 95%CI slope= 0.997 -1.047 for Epitope; 95%CI y-intercept= -0.5 -0.2; 95%CI slope= 0.897 -1.301 for Euroimmun), but not for the ECLIA from Roche (95%CI y-intercept= -0.6 -0.1; 95%CI slope= 0.331 -0.775). In agreement with these results, Student´s t-test revealed a significant difference between the COI determined in matching serum and lithium plasma samples for the Elecsys Anti-SARS-CoV-2 assay (p<0.05). All tests were performed using the same lot for all three assays included in this study. As all three immunoassays are qualitative test systems, qualitative results (positive versus negative) were evaluated for both testing materials. This comparison revealed a 100% overall agreement for the tests system from Epitope and Roche, respectively. The overall agreement for the Euroimmun assay ranges from 90% to 95% depending on whether borderline results are considered positive or negative (Table 3 ). In detail, these discrepant results were seen in plasma samples, whereas no false-positive or false-negative result was revealed for serum. Importantly, in this case the COI of the Elecsys Anti-SARS-CoV-2 was 0.976 and thus marginally below the cutoff. The implementation of large-scale SARS-CoV-2 serological testing on a population level is heavily debated by governments and national regulators to adopt restriction regulations to current demands in countries where the number of infections is decreasing. Beside of supplementing primary diagnosis [18] , monitoring of immune response to vaccine candidates and evaluation of immunity duration [10] , seroepidemiologic studies represent the main application area of immunoassays. Here, serological tests promise to provide the greatest benefit as they may allow accurately assessing infection prevalence and establishing indicators of SARS-CoV-2 immunity. In this screening situation, oligo-or asymptomatic SARS-CoV-2 infection will represent the majority of cases. For evaluation of test performance, the study cohort should represent the intended test population. This has not been adequately addressed so far, as the majority of published studies comparing different immunoassays were performed on hospitalized patients with ongoing infection [19] [20] [21] [22] . This might affect the diagnostic test performance on two ways. First, the time of blood-draw after symptom onset substantially impacts the number of positive test results as reliable test results for anti-SARS-CoV-2 IgG testing can be obtained with a median of 14 days after symptom onset and sometimes with a delay of several weeks [13, 17, 18, [23] [24] [25] . Second, it has been shown that the IgG level of hospitalized patients is enhanced compared to a moderate to absent IgG immune response in mildly affected individuals [11, 23, 26, 27] . Therefore, our study cohort included patients after hospitalization or end of home quarantine representing the different severity of the disease course. All three commercial assays used in our method comparison study were verified according to our quality management requirements and in agreement with accreditation requirements according to ISO 15 189 . All assays revealed an acceptable inter-and intraassay variability with the lowest coefficient of variation (CV) seen for the Elecsys test. Most importantly, the diagnostic accuracy for all test systems was 100% using quality control samples. To the best of the author´s knowledge, this is the first study so far addressing systematically the impact of specimen type on the test performance of different immunoassays. For screening purposes or seroepidemiological studies, the available blood specimen types may vary. Often, only lithium heparin plasma is available as the most common residual material from patient samples archived in the laboratory. According to manufacturer´s instruction of Euroimmun and Roche, serum and lithium-heparin plasma can be used for anti-SARS-CoV-2 antibody testing and no different cut-offs have to be used for interpretation of test results. In contrast, the Epitope assay is restricted to serum as testing material. Our analysis of 20 matching serum and plasma samples revealed a high level of concordance between both testing modalities for assays from Euroimmun and Epitope by linear regression, Passing Bablok regression and Bland-Altman analysis demonstrating the interchangeability of anti-SARS-CoV-2 antibody detection in both blood matrices (r 2 =0.97 and r 2 =0.91). However, based on qualitative results the overall agreement was 100% for the Epitope test and 90-95% for the immunoassay from Euroimmun. Statistical evaluation of the Roche Elecsys assay demonstrated significantly higher values measured in plasma than in serum samples and correlation analysis proved that plasma-and serum-based testing are no interchangeable. In this context it is important to mention that all assays were performed with the same lot number and no technical problems from the manufacturer have been reported for this lot. Comparable results have been reported for neutralization antibody tests (NAT) [10] . Although no false-positive or false-negative qualitative result was noted for the Roche test, this might be the case for samples near the assays cut-off. Hence, our results for at least two FDA EUA approved assays demonstrate that optimal cut-offs and assay performance should be separately assessed for each testing material. To estimate test performance, manufacturers refer to test sensitivity and specificity or positive and negative percent agreement indicating that a non-reference standard was used for test evaluation. In case of anti-SARS-CoV-2-IgG testing, the respective manufacturer´s information is summarized by the Center for Health Security of Johns Hopkins University [13] . Here, for all three immunoassays included in this study a sensitivity of 100% is reported if patients are tested from 14 days after diagnosis onwards. The specificity is indicated by 100% for the tests from Euroimmun and Epitope diagnostics and 99.81% for the Roche assay. In our study, we compared the diagnostic test performance of these immunoassays for detection of anti-SARS-CoV-2 IgG antibodies on a panel of 51 serum samples including 26 patients with confirmed SARS-CoV-2 infection and 25 controls. Our analysis revealed a diagnostic sensitivity of 92.3%, 96.2-100% and 100% with a respective diagnostic specificity of 100%, 100% and 84-86% for the immunoassays from Roche, Euroimmun and Epitope Diagnostics. In total, 84-96 % of samples were correctly classified as negative and 92.3-95.2% as positive. For test sensitivity, these findings are clearly below manufacturer´s specifications. One explanation might be that we preferentially included patients with subclinical infections or a mild course of disease representing the majority of SARS-CoV-2 infections and for whom the IgG immune response is reported to be moderate [10, 23, [26] [27] [28] . In this context it is worth mentioning that one sample was classified as positive by two assays in agreement with the qRT-PCR confirmed SARS-CoV-2 infection, but was found negative by the Roche test. This underscores that, as expected, the high specificity of the Elecsys assay leads to a reduction in test sensitivity to a certain extent. Especially as no borderline results are reported by the Elecsys assay, patients with COIs marginally below the cut-off might need to be reevaluated after a certain time period to reduce false-negative test results. Egger et al. demonstrated that the diagnostic sensitivity strongly depends on time from symptom onset with a sensitivity ranging between 100% [29] and 89.36% [30] from day 15 onwards for the Elecsys test. Nevertheless, the false negative results revealed in our study are obtained 26 and 45 days after confirmation of SARS-CoV-2 infection. Overall, the diagnostic sensitivity of 92.3% revealed in our study is in line with those reported by others [29] [30] [31] . Comparable, findings for the assays from Euroimmun and Epitope diagnostics are in agreement with or even better than those reported in other studies [20] [21] [22] . The decision whether an assay is suitable for clinical use or which test might be preferred depends on the intended testing purpose. For seroepidemiologic studies and screening purposes, the identification of affected individuals is usually based on a screening test with high diagnostic sensitivity and a confirmatory test with high specificity. In case of anti-SARS-CoV-2, such a confirmatory test could be a neutralizing antibody test (NAT). However, these tests are time-consuming with a turn-around-time of 3-5 days [13] , require a biosafety level-3 laboratory and thus cannot be performed on a broad extent or for population-based studies. Hence, the estimation of the infection prevalence relies on serological tests without further confirmatory test and thus priority must be given to test specificity over sensitivity. Arguments for prioritizing test specificity include more far-reaching consequences in case of a false positive test result. Second, due to currently low prevalence of SARS-CoV-2 infections within the population that is estimated at 0.32% in the United States [32] , only with a test specificity approaching 100% an acceptable positive predictive value can be obtained [17, 21, 32] . Our study revealed that these requirements are fulfilled by the immunoassays from Euroimmun and Roche. However, as our study is limited by sample size further large-scale studies are warranted to address the specificity of the different assays in different subpopulations for whom a high level of test interference with immunoassays is reported, e.g. for patients suffering from chronic diseases. Another possibility to increase the positive predictive value of serological tests is to monitor individuals with a positive antibody test and only consider them positive in case of seroconversion defined as class-switch from IgM to IgG or a more than 4fold increased IgG titer formerly for SARS [33, 34] . Finally, it has to be mentioned that antibody detection might be due to cross reactivity with other coronaviruses [26] and even if specific does not necessarily equate with protective immunity [17, 35] . It remains to be elucidated whether antibodies against nucleocapsid or spike protein will prove superior in assessing immunity. True immunity studies will require direct comparison with NAT [17] . In summary, our study demonstrated a high specificity for both FDA EUA approved immunoassays and a good diagnostic sensitivity. Overall, these results were superior to the non-FDA EUA declared test included. This is underlining the suitability of the immunoassays from Roche and Euroimmun for seroepidemiologic studies and screening purposes. In case of positive results, either a follow-up of patient in order to confirm results by seroconversion or a NAT might be necessary. However, comparing different blood specimens significantly different results were obtained either on quantitative or qualitative evaluation. Hence, there is an urgent need for quality standards to be implemented, harmonization of tests allowing to compare test results and to guide further decisions. Appropriate cut-offs and deeper understanding of tests limitation are a prerequisite for the appropriate use of COVID-19 serological tests. VH, VC, MN designed the study. VH was responsible for data collection and management. CG, MT, MK enrolled patients. VC, RE, VH performed assays. VH was responsible for biostatistics analyses. VH, VC were responsible for interpretation of data. VH, MK prepared the tables and figures. VH was drafting the manuscript. All authors contributed to revision of the manuscript, and approved it for submission. The authors have declared no conflicts of interest.  FDA-EUA approved anti-SARS-CoV-2 tests compare favorably with others  Diagnostic sensitivity remains behind the manufacturer´s specifications  Plasma and serum cannot both be used equally for anti-SARS-CoV-2 IgG detection  Immune response relies on disease severity, thus need to be considered for validation Coronavirus disease (COVID-19) Pandemic. Geneva: WHO 2020 The paramount importance of serological surveys of SARS-CoV-2 infection and immunity Characteristics of and Important Lessons From the Coronavirus Disease 2019 (COVID-19) Outbreak in China: Summary of a Report of 72314 Cases From the Chinese Center for Disease Control and Prevention Why Only Test Symptomatic Patients? Consider Random Screening for COVID-19 The epidemiology, diagnosis and treatment of COVID-19 Characteristics of COVID-19 pandemic and public health consequences SARS-CoV-2 and COVID-19: The most important research questions Data-based analysis, modelling and forecasting of the COVID-19 outbreak Serological assays for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) Communication from the Commission -Guidelines on COVID-19 in vitro diagnostic tests and their performance FIND. COVID-19 Diagnostics Resource Center medicaldevices/emergency-situations-medical-devices/eua-authorized-serology-test-performance Vitro Diagnostic Assays for COVID-19: Recent Advances and Emerging Trends The Role of Antibody Testing for SARS-CoV-2: Is There One? Profiling Early Humoral Response to Diagnose Novel Coronavirus Disease (COVID-19) Early detection of SARS-CoV-2 antibodies in COVID-19 patients as a serologic marker of infection Test performance evaluation of SARS-CoV-2 serological assays Comparison of four new commercial serologic assays for determination of SARS-CoV-2 IgG Spread of SARS-CoV-2 in the Icelandic Population Correlation of Chest CT and RT-PCR Testing in Coronavirus Disease 2019 (COVID-19) in China: A Report of 1014 Cases Successful containment of Covid-19 outbreak in a large maternity and perinatal center while continuing clinical service Severe Acute Respiratory Syndrome Coronavirus 2-Specific Antibody Responses in Coronavirus Disease 2019 Patients Symptoms and immunoglobulin development in hospital staff exposed to a SARS-CoV-2 outbreak COVID-19: immunopathology and its implications for therapy Comparison of the Elecsys(R) Anti-SARS-CoV-2 immunoassay with the EDI enzyme linked immunosorbent assays for the detection of SARS-CoV-2 antibodies in human plasma Clinical Performance of the Roche SARS-CoV-2 Serologic Assay Clinical performance of the Elecsys electrochemiluminescent immunoassay for the detection of SARS-CoV-2 total antibodies SARS-CoV-2 Serology: Much Hype, Little Data Diagnostic performance of COVID-19 serology assays Serological assays for emerging coronaviruses: challenges and pitfalls Neutralizing Antibodies against SARS-CoV-2 and Other Human Coronaviruses We thank Sihem Aida, Marina Talamini and Cornelia Keup for aliquoting of samples, and Jan-Hendrik Haselmann for critical reading of the manuscript.