key: cord-0715181-jcqyt4jb authors: Barchuk, Anton; Shirokov, Daniil; Sergeeva, Mariia; Tursun­zade, Rustam; Dudkina, Olga; Tychkova, Varvara; Barabanova, Lubov; Skougarevskiy, Dmitriy; Danilenko, Daria title: Evaluation of the performance of SARS‐­CoV­‐2 antibody assays for a longitudinal population­based study of COVID‐­19 spread in St. Petersburg, Russia date: 2021-06-12 journal: J Med Virol DOI: 10.1002/jmv.27126 sha: e464ab52cbb51e06294c6f172bb11a68b44b0f39 doc_id: 715181 cord_uid: jcqyt4jb Geographical variation in severe acute respiratory syndrome coronavirus 2 (SARS­‐CoV­‐2) spread requires seroprevalence studies based on local tests, but robust validation is needed. We summarize an evaluation of antibody tests used in a serological study of SARS‐­CoV­‐2 in Saint Petersburg, Russia. We validated three different antibody assays: chemiluminescent microparticle immunoassay (CMIA) Abbott Architect SARS‐­CoV­‐2 immunoglobulin G (IgG), enzyme­ linked immunosorbent assay (ELISA) CoronaPass total antibodies test, and ELISA SARS­‐CoV‐­2‐­IgG‐­EIA‐­BEST. Clinical sensitivity was estimated with the SARS­‐CoV­‐2 polymerase chain reaction (PCR) test as the gold standard using manufacturer recommended cutoff. Specificity was estimated using pre­pandemic sera samples. The median time between positive PCR test results and antibody tests was 21 weeks. Measures of concordance were calculated against the microneutralization test (MNA).Sensitivity was equal to 91.1% (95% confidence intervbal [CI]: 78.8–97.5), 90% (95% CI: 76.4–96.4), and 63.1% (95% CI [50.2–74.7]) for ELISA Coronapass, ELISA Vector­Best, and CMIA Abbott, respectively. Specificity was equal to 100% for all the tests. Comparison of receiver operating characteristics has shown lower AUC for CMIA Abbott. The cut­off SC/O ratio of 0.28 for CMIA Abbott resulted in a sensitivity of 80% at the same level of specificity. Less than 33% of the participants with positive antibody test results had neutralizing antibodies in titers 1:80 and above. Antibody assays results and MNA correlated moderately. This study encourages the use of local antibody tests and sets the reference for seroprevalence correction. Available tests' sensitivity allows detecting antibodies within the majority of PCR­ positive individuals. The Abbott assay sensitivity can be improved by incorporating a new cut­off. Manufacturers' test characteristics may introduce bias into the study results. confirmed by polymerase chain reaction (PCR) and from healthy donors. The longterm performance of antibody tests is key to understanding the antibody kinetics. Previous validation studies have shown significant changes in the sensitivity of certain antibody tests in the longer followup. 15 This report summarizes the validation of antibody tests used in a representative populationbased serological study of SARS-CoV-2 in St. Petersburg, Russia, a densely populated city with more than 5.3 million inhabitants making it the fourth largest city in Europe. 6 We aimed at establishing test specificity (Sp) and sensitivity (Se) to correct estimates obtained through populationbased seroprevalence study. Petersburg is available in the detailed report. 6 In brief, we obtained a representative population sample through phone by using randomdigit dialing. The study started in May 2020. The first report was publicly available in August 2020. This study is underway and includes three rounds of blood sample collection with more than 2500 participants. This manuscript represents the evaluation of the performance of SARS-CoV-2 antibody assays using independent samples of sera taken from PCR positive cases and prepandemic sera that are not related to the populationbased serological study. We also assess the degree of agreement between the assays and neutralization tests using samples acquired through populationbased study. Test performance for CMIA Abbott is available from the manufacturer materials (Se = 100%, Sp = 99.6%). It was also evaluated in numerous independent studies. 9,16-18 ELISA Coronapass test manufacturer provides information on its official website (Se = 98.7%, Sp = 100%). ELISA VectorBest sensitivity and specificity is reported in one study published in Russian (Se = 100%, Sp = 99.8%). 19 We used CMIA Abbott and ELISA Coronapass for the first publication of a populationbased seroprevalence study in May June 2020. 6 The first round of the study analysis showed that the use of ELISA Coronapass results in a slightly higher seroprevalence estimation. This was the initial reason for the independent validation of the test performance. For sensitivity assessment, we also used nucleocapsid proteinbased CMIA Elecsys Anti-SARS-CoV-2, manufactured by Roche Diagnostics GmbH, Mannheim, Germany (CMIA Roche, S/CO ratio of 1.0 for positivity, Se = 99.5%, Sp = 99.8%; diagnostics. roche. com/ru/ ru/products/params/elecsys-anti-sars-cov-2. html). For the sensitivity assessment, we obtained 92 serum samples col- viruses. Written informed consent was obtained in all cases of blood donation and further sample preparation. We used 93 remnant sera samples from herd immunity studies that were kept frozen at 80°C. A neutralization test was performed for 365 positive samples for binding antibody samples (at least on one of three assays) and for 74 negative samples. TCID 50 based microneutralization test (MNA) was used to detect and titrate neutralizing antibodies. 20 First, serum samples were heated for 30 min at 56°C to avoid complementlinked reduction of the viral activity. This was followed by the preparation of serial twofold dilutions starting from 1:10 in 60 uL volume (tested in triplicate) of each serum specimen in culture medium (Alpha MEM containing antibiotics and 2% heatinactivated fetal bovine serum). Each dilution was mixed at equal volume with the live SARSCoV2 virus (60 µl, containing 25 TCID 50 /50 µl) and incubated for 60 min at 37°C in plastic microplates. Then 100 μl of the mix was transferred into 96-well microplates with monolayer Vero cells. The plates were incubated at 37°C in a 5% CO 2 atmosphere. Readings were evaluated 5-6 days later and neutralization was recorded if 100% of the cells in the well were preserved (no visible plaques or cytopathic effect). Serum neutralizing titer was expressed as the inverse of the higher serum dilution that exhibited neutralizing activity. All experiments were performed in a BSL3 laboratory. Clinical sensitivity was estimated with the SARS-CoV-2 PCR test as the gold standard for specificity in prepandemic sera samples. We Based on crossvalidation results, the sensitivity was equal to 91.1% (95% CI: 78. 8 57.3-81.9). Specificity was equal to 100% for all the tests (see Table 1 ). Using nonpaired samples in the validation of the tests did not dramatically change the sensitivity and point estimates but narrowed confidence intervals (see Table S1 ). Although the comparison of ROCs for three tests has shown that AUC is lower for CMIAAbbott, the difference was not dramatic-0.96 (95% CI: 0.90-1.00) for ELISA Coronapass and 0.90 (95% CI: 0.82-0.97) for CMIA Abbott, the difference was only significant when paired samples were compared (see Table 2 and Table S2 ). The cutoff SC/O ratio of 0.28 for CMIAAbbott resulted in a sensitivity of 80% at the same full level of specificity. The other test thresholds were optimal based on the ROC analysis of crossvalidation and nonpaired samples (see Figure 1 ). In less than one-third of the populationbased study participants with positive antibody total IgG test results, we detected neutralizing antibodies in titers 1:80 and above. NPA was between 28.0% (95% Table 3 ). The measures of concordance are presented in Table S3 . There was a moderate correlation between antibody assays results and MNA: 0.65 (95% CI: 0.59-0.71) for CMIA Abbott, 0.60 (95% CI: 0.54-0.67) for ELISA Coronapass, and 0.76 (95% CI: 0.72-0.81) for ELISA VectorBest (see Figure 2 ). Our validation study encourages the use of local antibody tests for populationbased SARS-CoV-2 surveillance and sets the reference for the seroprevalence correction. However, it discourages the use of Abbott Architect SARS-CoV-2 IgG for population based seroprevalence SARSCoV2 surveillance. The decrease in test sensitivity (63% with a cutoff of 1.4) is likely to be related to the long period between infection and blood sampling compared to the manufacturer's validation studies (it was on average 21 weeks after the reported positive PCR test). Previous studies also found similar results for samples taken at a longer follow up. 13, 15 These results may also explain the decline in antibody presence in longitudinal studies that may be incorrectly interpreted by the shortterm immune response to SARS-CoV-2. 22 Based on our validation study, the Abbott Architect SARSCoV2 IgG's sensitivity can be significantly improved by incorporating a new cutoff. Moving the S/CO ratio from 1.4 or 1.0 for positivity to 0.28 improved sensitivity from 63% or 71% to 80%, respectively, without loss in specificity. In another study 11 the optimized cutoff for Abbott CMIA was 0.91, but the mean cutoff in specificity cohorts ranged between 0.06 and 0.11 (below 0.28). The longer period between infection and blood draw in our study may explain the difference in the optimized cutoff. 25, 26 This finding has an important practical implication. Relying on test characteristics provided by manufacturers for correction of reported prevalence estimates introduces additional bias to the study. If the test reported prevalence is 5%, then with the test sensitivity of 65% and specificity of 100%, the true prevalence would be equal to approximately 8%. 27 If the crude prevalence based on the same test is 40%, then the true prevalence would be around 62%. The magnitude of bias in the population with a significant proportion of T A B L E 1 Sensitivity and specificity of tests based on crossvalidatation (paired serum samples) Abbreviation: CI, confidence interval. The paramount importance of serological surveys of SARSCoV2 infection and immunity Humoral immune response to SARSCoV2 in Iceland Prevalence of SARSCoV2 in Spain (ENECOVID): a nationwide, populationbased seroepi demiological study Estimating COVID19 antibody ser Analytical methods used in estimating the prevalence of HIV/AIDS from demographic and crosssectional surveys with missing data: a systematic review Seroprevalence of SARSCoV2 antibodies in Saint Petersburg, Russia: a populationbased study Comparison of sixteen serological SARSCoV2 immunoassays in sixteen clinical laboratories Stringent thresholds in SARSCoV2 IgG assays lead to underdetection of mild infections Evaluation of diagnostic accuracy of 10 serological assays for detection of SARSCoV2 antibodies Quantification of SARSCoV2 antibodies with eight commercially available immunoassays An evaluation of 4 commercial assays for the detection of SARSCoV2 antibodies in a predominantly mildly symptomatic low prevalence Australian population Two SARSCoV2 IgG immunoassays comparison and timecourse profile of antibodies response Six month antibody response to SARSCoV2 in healthcare workers assessed by virus neutralization and commercial assays SARSCoV2 seroprevalence worldwide: a systematic review and metaanalysis Longitudinal serological analysis and neutralizing antibody levels in coronavirus disease 2019 convalescent patients Clinical evaluation of serological IgG antibody response on the Abbott Architect for established SARSCoV2 infection A comparative evaluation between the Abbott Panbio™ COVID19 IgG/IgM rapid test device and Abbott Architect™ SARS CoV2 IgG assay Comparison of the clinical performances of the Abbott Alinity IgG, Abbott Architect IgM, and Roche Elecsys total SARSCoV2 antibody assays Sensitivity and specificity of JSC VectorBest assays for immunoglobulin of different classes to SARSCoV2 [Tchuvstvitel'nost' i specifichnost' naborov reagentov AO «Vektor-Best» dlja vyjavlenija immunoglobulinov raznyx klassov k SARSCoV2 Evaluation of SARSCoV2 neutralizing antibodies using a CPEbased colorimetric live virus microneutralization assay in human serum samples Bias, prevalence and kappa Longevity of seropositivity and neutralizing titers among SARSCoV2 infected individuals after 4 months from baseline: a populationbased study in the province of Trento. medRxiv Evolution of immune responses to SARSCoV2 in mildmoderate COVID19 COVID19 reinfection in the presence of neutralizing antibodies Antibody status and incidence of SARSCoV2 infection in health care workers Association of SARSCoV2 seropositive antibody test with risk of future infection Exact confidence limits for prevalence of a disease with an imperfect diagnostic test Specificity and positive predictive value of SARSCoV2 nucleic acid amplification testing in a lowprevalence setting Establishment and validation of a pseudovirus neutralization assay for SARSCoV2 Additional Supporting Information may be found online in the supporting information tab for this article.