key: cord-0766142-9dd2giyn authors: Park, Younhee; Hong, Ki Ho; Lee, Su-Kyung; Hyun, Jungwon; Oh, Eun-Jee; Lee, Jaehyeon; Lee, Hyukmin; Song, Sang Hoon; Kee, Seung-Jung; Kwon, Gye Cheol; Kim, Su Hwan; Do, Hyeon-Nam; Kim, Ah-Ra; Lee, June-Woo; Kim, Sung Soon; Kim, Hyun Soo title: Performance Comparison of Five SARS-CoV-2 Antibody Assays for Seroprevalence Studies date: 2022-01-01 journal: Ann Lab Med DOI: 10.3343/alm.2022.42.1.71 sha: b9780b6a600dff3a2d78aec95320966bdea887d9 doc_id: 766142 cord_uid: 9dd2giyn BACKGROUND: Seroprevalence studies of coronavirus disease 2019 (COVID-19) cases, including asymptomatic and past infections, are important to estimate the scale of the disease outbreak and to establish quarantine measures. We evaluated the clinical performance of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) antibody assays available in Korea for use in seroprevalence studies. METHODS: The sensitivity, specificity, cross-reactivity, and interference of five SARS-CoV-2 antibody assays were evaluated using the following 398 serum samples from confirmed COVID-19 patients, 510 negative control samples from before 2018 (pre-pandemic), 163 serum samples from patients with SARS, Middle East respiratory syndrome (MERS), and other viral infections, and five samples for the interference study. RESULTS: The sensitivities of the five assays ranged from 92.2% to 98%, and their specificities, including cross-reactivity and interference, ranged from 97.5% to 100%. The agreement rates were excellent (kappa >0.9). Adjustment of the cutoff values could be considered through ROC curve analysis. The positive predictive values of the individual assays varied from 3.5% to 100% at a 0.1% prevalence but were as high as ≥95% when two assays were combined. CONCLUSIONS: The prevalence of COVID-19 in Korea is considered to be exceptionally low at present; thus, we recommend using a combination of two or more SARS-CoV-2 antibody assays rather than a single assay. These results could help select SARS-CoV-2 antibody assays for COVID-19 seroprevalence studies in Korea. Coronavirus disease 2019 (COVID- 19) , which originated in Wuhan, China in December 2019, is caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) [1] . More than Therefore, the actual number of COVID-19 positive cases has been underestimated. To determine the size of the infected population and to establish quarantine measures, accurate serological testing is required. Seroprevalence studies have been conducted in many countries, including the United States, the United Kingdom, Spain, and Korea [4] [5] [6] [7] [8] . In less than a year, several types of antibody assays have been developed worldwide. However, comparative studies on the performance of assays available in Korea to determine seroprevalence have not yet been conducted. The available antibody assays mainly use recombinant spike (S) proteins, nucleocapsid (N) proteins, receptor-binding domains, S1 antigens, and combinations of these antigens to detect IgG, IgM, and total antibody levels [9] [10] [11] [12] [13] [14] [15] [16] . We evaluated the clinical performance of COVID-19 antibody assays available in Korea for seroprevalence studies. We further estimated the positive predictive values (PPVs) of individual and two combined assays using the sensitivities and specificities obtained from this study and the expected prevalence in Korea. We also investigated cross-reactivity using serum samples from patients with antibodies to various viruses and bacteria, autoimmune disease, or monoclonal gammopathy. Serum samples, leftover from laboratory tests and designated to be discarded, from 398 patients diagnosed as having COVID-19 at two hospitals (Seoul Medical Center, Seoul, Korea and Hallym University Dongtan Sacred Heart Hospital, Hwaseong, Korea) and the Korea Disease Control and Prevention Agency (KDCA) were collected between March and September 2020 and stored at -70°C until analysis. The dates of symptom onset and hospital admission were obtained retrospectively from the medical records at the two hospitals. Serum samples of 510 negative controls, collected before 2018 (pre-pandemic period), were obtained from the National Biobank of Korea, the KDCA, and the High-Risk Human Serum Bank of Chung-Ang University (Seoul, Korea). A total of 168 samples were tested for cross-reactivity, including 136 residual serum samples of patients with antibodies to other viruses (human (h)CoV-229E, -NL63, -OC43, and -HKU1; adenovirus; influenza A virus; influenza B virus; human metapneumovirus; parainfluenza virus type 1/2/3/4; respiratory syncytial virus; rhinovirus; Mycoplasma pneumoniae; Chlamydia pneumoniae; hepatitis A/B/C virus; Epstein-Barr virus; cytomegalovirus; herpes simplex virus; mumps virus; and varicella zoster virus) collected from two hospitals (Severance Hospital, Seoul, Korea and Hallym University Dongtan Sacred Heart Hospital). Serum samples positive for fluorescence anti-nuclear antibody, rheumatoid factor, and monoclonal gammopathy were collected from the same two hospitals. Twenty-three serum samples positive for antibodies to SARS-CoV-1, Middle East respiratory syndrome (MERS)-CoV, hCoV-229E, -NL63, -OC43, and -HKU1, influenza B virus, respiratory syncytial virus, and adenovirus were purchased from Trina Bioreactives (Nänikon, Switzerland). Five serum samples from patients with monoclonal gammopathy were collected from the Korea Institute of Radiological and Medical Sciences Radiation Biobank (Seoul, Korea), and four MERS-CoV convalescent serum samples were provided by the International Vaccine Institute (Seoul, Korea) (Fig. 1) . The Public Institutional Review Board designated by the Ministry of Health and Welfare (http://public. irb.or.kr/) approved the study protocol and waived the need for informed consent (P01-202008-31-002). In August 2020, the KDCA received applications from 18 manufacturers to evaluate the performance of 21 SARS-CoV-2 antibody assays. It was not feasible to evaluate all 21 assays given the limited amount of serum samples; hence, a screening evaluation was required. After the Evaluation Committee did a preliminary assessment of the data submitted by the manufacturers, five assays were selected for performance evaluation: Elecsys Anti-SARS-CoV-2 assay on the Cobas e801 platform (Roche Diagnostics, Mannheim, Germany), Abbott Architect SARS-CoV-2 IgG assay on the Architect i2000SR platform (Abbott Laboratories, Abbott Park, IL, USA), Atellica IM SARS-CoV-2 Total assay on the Atellica platform (Siemens, Munich, Germany), SD Biosensor Standard E COVID-19 Total Ab assay (SD Biosensor, Suwon, Korea), and LG Chem AdvanSure SARS-CoV-2 IgG assay (LG Chem, Seoul, Korea). SD Biosensor and LG Chem ELISAs were performed using the Epoch Microplate Spectrophotometer and ELx50 Filter Microplate Washer (both from BioTek Instruments, Winooski, VT, USA). The principle, instrument, antibody detected, reagents used, cutoff value, sample volumes, and time to first results of all assays are listed in Table 1 . All assays were carried out according to the manufacturers' instructions. Most assays were performed at Hallym University Dongtan Sacred Heart Hospital by one laboratory technician and one scientific researcher, and the Atellica IM SARS-CoV-2 total assay was performed at Severance Hospital by one laboratory technician. The diagnostic sensitivity and specificity (and the 95% confidence intervals) of the five assays were calculated according to days after symptom onset using MedCalc software version 19.8 (MedCalc Software, Ostend, Belgium). The date of symptom onset was not available for some patients. The potential cross-reactivity with other coronaviruses, such as SARS-CoV-1, MERS, respiratory viruses, and other viruses, bac- teria, autoimmune diseases, and monoclonal gammopathy was evaluated using patient serum samples that were positive for antibodies against the relevant pathogen. For interference testing, Hb (10 mg/mL), bilirubin (0.4 ng/mL), a triglyceride mix (20 mg/mL), or zanamivir (10 mg/mL; all from Sigma-Aldrich) were spiked into serum samples that were negative for these antibodies. The results of spiked samples were compared with those of non-spiked samples and considered acceptable if the difference was < 15%. The agreement and correlations between the results of the five SARS-CoV-2 antibody assays were calculated using MedCalc software and were reported as correlation graphs. For the Sie-mens Atellica IM SARS-CoV-2 Total assay, antibody titers < 0.05 were considered 0, and those > 10 were considered 10. The upper limits of detection values were determined to be 4.0 for the SD Biosensor assay and 4.5 for the LG Chem assay. Positive, negative, and total agreement between assays were evaluated using Cohen's kappa statistics and categorized as poor ( < 0.00), slight (0.00-0.20), fair (0.21-0.40), moderate (0.41-0.60), substantial (0.61-0.80), or nearly perfect (0.81-1.00). ROC curve analyses and areas under the ROC curve (AUC) calculations for the five assays were performed using the MedCalc software. P ≤ 0.05 was considered statistically significant. For each ROC curve, we recalculated the cutoff values so that the Youden index ( = sensitivity+specificity-1) was maximized, and the sensitivity and specificity were determined from the calculated cutoff values. PPVs and negative predictive values (NPVs) for individual and two combined assays using varying seroprevalence settings PPVs and NPVs of individual and two combined assays using the calculated sensitivity and specificity values and the predictive seroprevalence of 10%, 5%, 2%, 1%, and 0.1% were calculated using a US Food and Drug administration online calculator [17] . The diagnostic sensitivities were 93.5% for the Roche assay, 92.2% for the Abbott assay, 95.7% for the Siemens assay, 98.0% for the SD Biosensor assay, and 97.0% for the LG Chem assay. The diagnostic specificities were 99.7% for the Roche assay, 99.4% for the Abbott assay, 100% for the Siemens assay, 99.3% for the SD Biosensor assay, and 97.5% for the LG Chem assay ( Table 1 ). The positivity rate was low in the first week after symptom onset but was nearly 100% three to four weeks after symptom onset. The SD Biosensor assay showed the highest sensitivity and the Siemens assay showed the highest specificity. The Abbott, Roche, and Siemens assays showed negative crossreactivity for 33 species of viruses and bacteria in 168 positive serum samples (Supplemental Data Table S1 ). The SD Biosensor and LG Chem assays each showed three false-positive results in the cross-reactivity study. No samples showed interference for Hb, a triglyceride mix, bilirubin, albumin, or zanamivir. The agreement among all assays was nearly perfect (kappa > 0.90) ( Table 2) ; however, correlations among the assays were non-linear (Supplemental Data Fig. S1 ), and several samples exceeded the upper limit of detection in the Siemens, SD Biosensor, and LG Chem assays. PPVs and NPVs for individual and two combined assays using determined sensitivity, specificity, and seroprevalence The lower the prevalence rate (from 10% to 0.1%), the lower is the PPV. The Siemens assay showed the highest specificity of 100% (95.2%; PPVs calculated using the lowest value of the 95% CI of the calculated specificity are shown in parentheses because the specificity was calculated as 100%, even at the lowest prevalence rate) among the five assays at a 10% prevalence and the highest specificity of 100% (15.1%) at a 0.1% prevalence (Table 4 ). When the predicted prevalence rate of 0.1% in Korea was considered, the PPV was as low as 24.2% for the Roche assay and 13.7% for the Abbott assay. However, the PPVs increased when two assays were both positive or when the orthogonal test algorithm (i.e., employing two assays in sequence when the first assay yields a positive result) [17] was used (Table 4 ). The sensitivities of the five assays were all > 93%, and the sensitivities were low in the first week after symptom onset but were nearly 100% three to four weeks after symptom onset. These results are similar to or better than those in previous studies [10, [12] [13] [14] [15] [16] [17] [18] [19] [20] . In a previous study, some SARS-CoV-2 antibody assays showed a sensitivity of 100%; however, that study used only samples collected two weeks after symptom onset [17] . The assay sensitivity may vary across studies depending on how many of the samples that are collected in the first week after symptom onset (especially on days 0-3) are included. In this study, the SD Biosensor and LG Chem assays had higher sensitivity but lower specificity than the Roche, Abbott, and Siemens assays. The Roche, Abbott, and Siemens assays use high cutoff values to increase specificity, according to the CDC guidelines [21] . The sensitivity and specificity and the cutoff may vary depending on the purpose of the assay. If the assay is used for diagnostic screening, a high sensitivity is preferred. However, the specificity should be high when the assay is used to investigate seroprevalence or the effect of vaccination. Laboratories can adjust assay cutoff values according to the intended purpose. We extensively investigated cross-reactivity using serum samples of patients with antibodies to various viruses and bacteria, autoimmune disease, or monoclonal gammopathy. The Roche, Abbott, and Siemens assays showed negative results for 163 samples tested for cross-reactivity, whereas the SD Biosensor and LG Chem assays showed false-positive results in three out of the 163 samples. These false-positive results seemed to be due to nonspecific reactions as only one sample yielded a positive result in each species. We found no cross-reactivity in serum with anti-SARS-CoV-1 antibody, which is considered the most similar to the anti-SARS-CoV-2 antibody, or in hypergammaglobulinemia serum. This study was the first to estimate the PPVs of individual and two combined assays based on the orthogonal test algorithm, using the sensitivities and specificities calculated in this study and the expected prevalence in Korea. As the prevalence in Korea is expected to be < 0.1% at present, if only one assay was used to determine the seroprevalence, the PPV was as low as 24.2% (8.2%) for the Roche assay, 13.7% (5.9%) for the Abbott assay, 100% (15.1%) for the Siemens assay, 57.5% (10.8%) for the SD Biosensor assay, and 3.5% (2.3%) for the LG Chem assay (Table 4 ). When two assays were combined, the PPV increased to > 95% for all combinations. We evaluated the performances of SARS-CoV-2 antibody as-says available in Korea during September 2020, and the assay reagents were still in development during this period; hence, the sensitivity and specificity of all assays could be improved, and the cutoff values could be adjusted. Presently, other SARS-CoV-2 antibody assays have been developed and are available in Korea. One limitation of this study is that neutralizing antibody assays that may represent personal immunity were not included [16] . Because most vaccines use the S protein as an immunogen, SARS-CoV-2 antibody assays targeting the N protein, such as the Abbott and Roche assays evaluated in this study, would not be useful to evaluate the vaccine response. However, the use of combinations of assays targeting the N and S proteins may allow discriminating natural infections from vaccinations. In summary, this study was performed to select appropriate SARS-CoV-2 antibody assays for implementation in a large-scale seroprevalence study in Korea. We estimated PPVs of individual and two combined assays based on the orthogonal test algorithm, using calculated sensitivity and specificity and expected prevalence of COVID-19 in Korea. Because the prevalence of COVID-19 in Korea is considered exceptionally low at present, we recommend using a combination of two or more assays rather than a single assay. A novel coronavirus from patients with pneumonia in China World Health Organization. Coronavirus disease (COVID-19) pandemic A systemic review of asymptomatic infections with COVID-19 Prevalence of SARS-CoV-2 antibodies in a large nationwide sample of patients on dialysis in the USA: a cross-sectional study High prevalence of asymptomatic COVID-19 infection in hemodialysis patients detected using serologic screening Prevalence of SARS-CoV-2 in Spain (ENE-COVID): a nationwide, population-based seroepidemiological study Seroprevalence of anti-SARS-CoV-2 antibodies among outpatients in Southwestern Seoul IgG Seroprevalence of COVID-19 among individuals without a history of the coronavirus disease infection in Daegu Serodiagnostics for severe acute respiratory syndrome-related coronavirus 2: a narrative review Antibody tests for identification of current and past infection with SARS-CoV-2 Antibody tests in detecting SARS-CoV-2 infection: a meta-analysis Diagnostic accuracy of serological tests and kinetics of severe acute respiratory syndrome coronavirus 2 antibody: a systematic review and meta-analysis Guidelines for Laboratory Diagnosis of Coronavirus Disease 2019 (COVID-19) in Korea Combination of a SARS-CoV-2 IgG Assay and RT-PCR for Improved COVID-19 Diagnosis Early Laboratory Preparedness of the Korea Disease Control and Prevention Agency and Response to Unknown Pneumonia Outbreak from Wuhan, China Comparison of SARS-CoV-2 Antibody Responses and Seroconversion in COVID-19 Patients Using Twelve Commercial Immunoassays EUA authorized serology test performance Evaluation of four commercial, fully automated SARS-CoV-2 antibody tests suggests a revision of the Siemens SARS-CoV-2 IgG assay National SARS-CoV-2 Serology Assay Evaluation Group. Performance characteristics of five immunoassays for SARS-CoV-2: a head-to-head benchmark comparison Centers for Disease Control and Prevention. Interim guidelines for COV-ID-19 antibody testing We appreciate Soyeon Yun for excellent technical support and data collection. The negative control samples used in this study were provided by the National Biobank of Korea, KDCA, and High-Risk Human Serum Bank (NRF-2017M3A9B8069581) at Chung-Ang University. The monoclonal gammopathy samples used in this study were provided by the Korea Institute of Radiological and Medical Sciences Radiation Biobank. MERS-CoV convalescent serum samples were kindly provided by Dr. Manki Song (International Vaccine Institute, Korea). Serum collection was funded by the Coalition for Epidemic Preparedness Innovations.