key: cord-0997063-5xk52tvn authors: Perkmann, T.; Perkmann-Nagele, N.; Koller, T.; Mucher, P.; Radakovics, A.; Marculescu, R.; Wolzt, M.; Wagner, O. F.; Binder, C. J.; Haslacher, H. title: Anti-Spike protein assays to determine post-vaccination antibody levels: a head-to-head comparison of five quantitative assays date: 2021-03-08 journal: nan DOI: 10.1101/2021.03.05.21252977 sha: 423ac8ae99a2a2f870b524a0d78d8c6b59fb8223 doc_id: 997063 cord_uid: 5xk52tvn Background Reliable quantification of the antibody response to SARS-CoV-2 vaccination is highly relevant for identifying possible vaccine failure and estimating the time of protection. Therefore, we aimed to evaluate the performance of five different Anti-SARS-CoV-2 antibody assays regarding the quantification of anti-spike (S) antibodies induced after a single dose of BNT162b2. Methods Sera of n=69 SARS-CoV-2 naive individuals 21+/-1 days after vaccination with BNT162b2 (Pfizer/BioNTech) were tested using the following quantitative SARS-CoV-2 antibody assays: Roche S total antibody, DiaSorin trimeric spike IgG, DiaSorin S1/S2 IgG, Abbott II IgG, and Serion/Virion IgG. Test agreement was assessed by Passing-Bablok regression. Results were further compared to the percent inhibition calculated from a surrogate virus neutralization test (sVNT) by correlation and ROC (receiver-operating-characteristics) analysis. Results Individual values were distributed over several orders of magnitude for all assays evaluated. Although the assays were in good overall agreement (rho=0.80-0.94), Passing-Bablok regression revealed systematic and proportional differences, which could not be eliminated by converting the results to BAU/mL as suggested by the manufacturers. 7 (10%) individuals had a negative sVNT results (i.e. <30% inhibition). These samples were reliably identified by most assays and yielded low binding antibody levels (ROC-AUCs 0.84-0.93). Conclusions Although all assays evaluated showed good correlation, readings from different assays were not interchangeable, even when converted to BAU/mL using the WHO international standard for SARS-CoV-2 immunoglobulin. This highlights the need for further standardization of SARS-CoV-2 serology. ROC-AUC receiver-operating-charateristics area under the curve 60 All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. of becoming such substitute endpoints (6). 74 The first SARS-CoV-2 antibody testing systems were designed to distinguish individuals with 75 prior COVID-19 infection from those who were still naive to this new virus (7). Therefore, these 76 immunoassays were usually developed as qualitative rather than quantitative tests and were 77 designed by the manufacturer to achieve the highest possible specificity and high sensitivity. 78 High specificity was indispensable, especially at the beginning of the pandemic, because the 79 extremely low seroprevalence rates led to many false positives and low positive predictive values 80 even with tests having a specificity of 99% (8). In contrast, the sensitivity of SARS-CoV-2 81 testing was often reduced to assure the high specificities needed for these assays (9). The lower 82 All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted March 8, 2021. ; https://doi.org/10.1101/2021.03.05.21252977 doi: medRxiv preprint 5 antibody levels further aggravated suboptimal sensitivities in mild/asymptomatic infections and 83 during the pandemic by the natural decline in antibody levels (10-15). 84 Various antigens have been used for this purpose, but essentially two types can be distinguished: 85 nucleocapsid-and spike protein-based assays (16). Antibodies directed against SARS-CoV-2 86 specific nucleocapsid (NC) antigens are induced early and strongly in most infected individuals 87 due to the virus nucleocapsid's typical strong immunogenicity (17). Furthermore, a very high 88 specificity can be achieved by targeted modification of the nucleocapsid antigen so that no cross-89 reactivity is observed even with closely related viruses. The discriminatory properties of such 90 nucleocapsid-based antibody assays can therefore be excellent (18, 19) . The physiological 91 significance of these antibodies, on the other hand, is unclear, and these surrogate markers for a 92 previous infection are unlikely to be functionally relevant to confer protection or immunity. The 93 antibodies that react with the spike protein (S), however, act differently. At least a proportion of 94 these S-binding antibodies are likely to have the function of neutralizing antibodies (20). Thus, it 95 is not surprising that numerous studies have shown a correlation between spike protein binding 96 assays and various forms of functional virus neutralization assays (21-26). 97 In the context of SARS-CoV-2 vaccines, it is precisely these neutralizing antibodies that are of 98 paramount importance. The primary goal of active immunization is to induce many SARS-CoV-99 2-specific neutralizing antibodies that ideally prevent the pathogen's entry and thus infection or 100 stop the systemic spread to prevent disease (27). The functional virus neutralization assays are 101 not feasible everywhere: assays with live viruses require biosafety level 3, but variants such as 102 pseudotyped neutralization assays are also labor-intensive and cannot be performed at high 103 throughput (28-30). Classical antibody assays, which measure the reactivity of antibodies in 104 All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted March 8, 2021. ; https://doi.org/10.1101/2021.03.05.21252977 doi: medRxiv preprint 6 serum/plasma with defined antigens, can be performed very rapidly and in high throughput, in 105 contrast to neutralization tests. 106 Thus, anti-spike protein assays will play an important role in vaccine development, licensing, and 107 efficacy monitoring in the future. However, these test systems must be able to reliably quantitate 108 SARS-CoV-2 specific antibody levels, be comparable to each other, and have good to excellent 109 agreement with the presence of neutralizing antibodies. The comparability of antibody assays is 110 expected to be improved by the recent introduction of a first WHO International Standard for 111 anti-SARS-CoV-2 immunoglobulin (NIBSC code 20/136) with reference to neutralizing 112 antibodies. 113 In the present work, we aim to go a step further and characterize the vaccination response after 114 the first administration of the Pfizer/BioNTech btn162b2 vaccine using five commercial 115 quantitative anti-spike protein antibody assays (4 of them with manufacturer's correction factor 116 for the WHO standard) in a head-to-head comparison. 117 This prospective observational study includes sera taken from 69 individuals without a previous 120 SARS-CoV-2 infection taken 21±1 days (mean±standard deviation) after the first dose of the 121 Pfizer/BioNTech BNT162b2 vaccine. Further inclusion criteria were an age >18 years, whereas 122 an insufficient amount of serum would have led to exclusion from the study. The study protocol 123 was reviewed and approved by the Ethics Committee of the Medical University of Vienna 124 (EK1066/2021). All participants provided written informed consent to donate blood for the 125 evaluation of diagnostic test systems (EK404/2012). 126 All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. DiaSorin LIAISON (DiaSorin, Stillwater, USA). The quantification range is between 1.63 and 148 800 AU/mL. Intra-and interassay precision ranges between 0 and 5%. According to the 149 All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The DiaSorin LIAISON SARS-CoV-2 S1/2 CLIA (DiaSorin S1/2 IgG) detects IgG antibodies 152 against an S1/S2 combination antigen on a DiaSorin LIAISON (DiaSorin, Stillwater, USA). The 153 quantification range is between 3.8 and 400.0 AU/mL. Intra-and interassay precision ranges 154 between 0 and 4% and, according to the manufacturer, specificity among blood donors is 98.5% 155 (97.5 99.2) and sensitivity is 97.4% >15 days after diagnosis at a cut-off of >15 AU/mL, whereby 156 results between 12.0 and 15.0 AU/mL are considered borderline. 157 The Virion\Serion ELISA (enzyme-linked immunosorbent assay) agile SARS-CoV-2 IgG (Serion 158 IgG) (Institut Virion-Serion, Wuerzburg, Germany) was analyzed on a FilterMax F5 Multiplate 159 Reader (Molecular Devices, San José, USA) and quantifies IgG antibodies against total S-protein 160 between 3 and 250 U/mL. Intra-and interassay precision ranges between 1 and 4%. According to 161 the manufacturer, specificity is 99.2%, and sensitivity is 96.2% at a cut-off of 15 U/mL, with 162 values between 10 and 15 U/mL being considered borderline results. 163 If applicable, binding antibody units per milliliter (BAU/mL), which are traceable to the WHO 164 International Standard for anti-SARS-CoV-2 immunoglobulin, were calculated by applying the 165 following conversion factors, as suggested by the manufacturers: Roche S tAb = ‫כ‬ 1 , Abbott S 166 We excluded prior SARS-CoV-2 infection by using the Roche Elecsys® SARS-CoV-2 ECLIA 168 on the cobas® e801 analyzer (Roche), which detects total antibodies to the viral nucleocapsid 169 antigen. These antibodies are not induced by vaccination with bnt162b2. This assay yields high 170 diagnostic sensitivity (90%) and specificity (99.7%) for infections that occurred at least 14 days 171 All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. were performed using MedCalc 19.6 (MedCalc, Ostend, Belgium), and graphs were drawn using 187 GraphPad 9 (GraphPad, La Jolla, USA). 188 Measurement ranges differ between binding assays 190 29 female (42%) and 40 male (58%) participants with a median age of 42 years (29 -51) were 191 included. Results from the five different antibody binding assays are presented in Table 1 and 192 Figure 1 . The Abbott S IgG assay showed the highest values with a median of 1097.1 AU/mL 193 All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. TriS IgG and S1/2 IgG, Serion IgG). In Passing-Bablok regression, all systematic and 210 proportional errors were statistically significant: DiaSorin TriS IgG = 24.5 + 0.13*X, DiaSorin 211 S1/2 IgG = 34.5 + 0.02*X, Serion IgG = 6.2 + 0.04*X. 212 The DiaSorin TriS IgG assay showed an excellent correlation with the remaining two tests 213 (DiaSorin S1/2 IgG ρ =0.91, Serion IgG 0.94). In the Passing-Bablok regression, nevertheless, 214 All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted March 8, 2021. ; https://doi.org/10.1101/2021.03.05.21252977 doi: medRxiv preprint marked deviations became apparent: DiaSorin S1/2 IgG = 30.5 + 0.16*X, Serion IgG = -0.0 + 215 0.31*X. 216 Finally, the DiaSorin S1/2 IgG and the Serion IgG correlated at ρ =0.91, and the Passing-Bablok 217 regression equation was Serion IgG = -50.9 + 1.78*X. All described relationships, as well as 218 related residual plots, are presented in Figure 2 . 219 Furthermore, we assessed whether the classification of results into tertiles (0 -33.3%, 33.4 -220 66.7%, 66.8 -100%) was comparable, e.g., whether a sample yielding a result in the lowest 221 tertile of test A was also in the lowest tertile of test B. Cohen's kappa was between 0.60 and 0.80, 222 indicating a good agreement, for all but for one of the ten test combinations (Roche S tAb/Serion, 223 kappa = 0.59, see Table 2 ). 224 In conclusion, the results of the investigated test systems correlate well but are not necessarily 225 International Standard for SARS-CoV-2 immunoglobulin, as described in the methods section. 227 Next, we wanted to clarify whether comparing converted BAU/mL instead of arbitrary values 228 facilitates comparability. 229 Binding antibody units per milliliter (BAU/mL) were calculated for the Abbott S IgG, the 231 DiaSorin TriS IgG, and the Serion IgG, according to the recently proposed conversion factors. 232 Results from the Roche S tAb ECLIA did not require conversion as indicated by the 233 manufacturer. 234 As shown in Figure 3 , the BAU/mL recalculation did not solve the problem of high proportional 235 errors. The least proportional error could be observed for the relationship between Roche S tAb 236 All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. In a final step, the binding assays' results were compared to percent inhibition of a surrogate virus 240 neutralization assay (sVNT). In the sVNT, the tested samples yielded median values of 63% (50 241 -76), ranging from 6 to 92%. Figure 4a illustrates that all binding assays except the DiaSorin 242 S1/2 IgG showed a quadratic relationship with the sVNT. The binding assays also differentiated 243 those values clustered in the upper range of the sVNT. However, for the DiaSorin S1/2, the 244 quadratic curve approached a straight line, indicating a mostly linear relationship between this 245 binding assay and the sVNT within the observed range. 246 7 (10%) of the individuals yielded sVNT results below 30% inhibition, which is considered 247 negative according to the manufacturer (Figure 1 ). Binding assay results were compared between 248 positives and negatives in the sVNT by ROC-curve analysis. The resulting AUCs ranged between 249 0.84 and 0.93 (see Figure 4b) , however, the differences between AUCs were not statistically 250 significant. Optimal cut-offs according to Youden's Index, as well as other cut-offs observed 251 from the data and their respective sensitivities and specificities are given in Supplemental Table 252 1. For three out of the five assays (Abbott S IgG, DiaSorin S1/2 IgG, and Serion IgG), all 253 samples with a negative sVNT result could be correctly identified with a corresponding 254 sensitivity of approximately 60%. 255 Discussion 256 SARS-CoV-2 antibody assays become important tools to evaluate the proportion of people 257 affected by COVID-19 and identify those who are still at infection risk. Now with the first 258 All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted March 8, 2021. ; https://doi.org/10.1101/2021.03.05.21252977 doi: medRxiv preprint vaccines available a new field of use for SARS-CoV-2 antibody tests will open up. First, many 259 vaccinated individuals will be interested in confirming their own vaccination success based on 260 the detection of specific antibodies. Second, vaccination-induced antibodies may be used as 261 surrogate from which a protection correlate will be estimated. To date, only limited information 262 on the performance of quantitative SARS-CoV-2 antibody assays is available, since most 263 currently evaluated assays were developed in-house, as recently summarized by the CDC 264 COVID-19 response group (32). Only for a few commercially available quantitative CE-marked 265 test systems preliminary data on the performance are given in the literature (19, 22, (33) (34) (35) . 266 Although a protection correlate for immunity in SARS-CoV-2 has not been defined yet, it is 267 useful to begin this important preliminary work now (6). Therefore, in the present work, we 268 compared different commercial SARS-CoV-2 antibody assays with spike protein reactivity using 269 a vaccination cohort to give a first insight into the comparability of these assays. 270 With regard to the numerical results, we were able to determine a broad distribution of the result 271 values for each individual test system, so that these were presented on a logarithmic scale. This is 272 in line with recently published reports, showing the antibody response after a single dose of 273 BNT162b2 vaccine (3, 4) . Interestingly, in agreement with a study involving >500 participants in 274 an identical study setting, we observed very similar mean values for the measurements with the 275 DiaSorin S1/S2 IgG: 66.3 AU/mL versus 68.6 AU/mL (3). Therefore, it is reasonable to assume 276 that our cohort is representative despite the moderate number of participants. In addition, we 277 were able to show that the results of the different test systems varied by a factor of up to more 278 than 50. This leads to the initial conclusion that a direct comparability of the numerical results of 279 different test systems is unlikely to be given across the range of individual findings. Differences 280 also occurred with respect to measurement ranges, and upper measurement limits were exceeded 281 All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted March 8, 2021. ; https://doi.org/10.1101/2021.03.05.21252977 doi: medRxiv preprint in 2 out of 5 systems (DiaSorin TriS IgG and Serion IgG), although the study cohort reflects the 282 antibody response before the administration of the 2 nd dose of Pfizer/BioNTech vaccine in SARS-283 CoV-2 naïve individuals. However, it must be mentioned that it is not yet known up to which 284 level a differentiation of the obtained values is meaningful. A recently published paper shows that 285 an anti-S post-vaccination titer of 61.8 AU/mL for the Abbott S IgG was associated with 286 reinfection after a single dose of vaccination. Since this value is only just above the threshold for 287 positivity for this specific assay (50 AU/mL) and in our cohort >95% of all observed values were 288 far higher (5th percentile: 207.5 AU/mL), this finding is plausible and suggests vaccine failure 289 due to very low antibody production. Alternatively, a reinfection might have been caused in these 290 subjects by a virus variant where vaccination protection is mitigated. Nevertheless, it can be 291 assumed that the average values of completely vaccinated persons are significantly higher than 292 those in our collective and thus the upper measurement limits could frequently be exceeded in 293 most assays. If clinically relevant, this could make additional dilution steps necessary, which are 294 not yet taken into account by the manufacturers. 295 Despite the different levels of measurement, all systems showed good correlations with each 296 other. When the measured values of the individual antibody tests were assigned to tertiles, good 297 agreement was shown between the lowest third, the middle third and the highest third of the 298 results. Thus, one individual with known immunosuppressive therapy consistently showed no 299 formation of antibodies in all five antibody binding assays tested. With defined cut-offs for low 300 or high vaccination titers of the different test systems, at least a partial transferability of a result 301 from one to another test system may therefore be expected. 302 Such transferability of results could also be anticipated via referencing the antibody assays used 303 to an international reference standard (32). Indeed, a first WHO international SARS-CoV-2 304 All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted March 8, 2021. ; https://doi.org/10.1101/2021.03.05.21252977 doi: medRxiv preprint 15 antibody standard with the valence of 1000 BAU/mL has recently become available. This 305 standard was used by the manufacturers for four out of five of the assays studied. However, this 306 standardization was not introduced during the establishment of the test system, but post-hoc as a 307 reference material to define a conversion factor of their own units in BAU/mL. It is therefore not 308 surprising that, this subsequent correction did not reduce the existing systematic deviations 309 ( Figure 2 ) between the different tests. Only the Roche S tAb and Serion IgG tests were able to 310 approximate the equivalence line, although here a very wide scattering of values around the trend 311 lines was observed. 312 The in vitro binding of infection-associated antibodies to pathogen-specific antigens in an 313 antibody test are important markers to objectify a past infection or vaccination. However, these 314 do not necessarily say anything about the function of these antibodies (1) . In SARS-CoV-2 315 vaccination, an important goal is to induce neutralizing antibodies that will prevent the virus from 316 binding to the cellular receptor, the ACE2 receptor, via the surface spike protein (36, 37) . Tests 317 to neutralize live viruses can only be performed in very specialized laboratories and 318 unfortunately, in the case of SARS-CoV-2, are not standardized, making comparability almost 319 impossible. For this reason, we chose to use a well-characterized surrogate virus neutralization 320 test (sVNT) as a functional reference (38-40). In this assay, a simple ELISA format is used to 321 determine the inhibition of conjugated RBD protein by neutralizing antibodies to the plate-bound 322 ACE2 receptor. The manufacturer suggests a threshold for positivity of 30% inhibition. For the 323 Abbott S IgG, the DiaSorin S1/2 IgG, and the Serion IgG assay, all samples below this threshold 324 were identified at a corresponding sensitivity of about 60%. The corresponding criteria were 3 -325 17 times higher than the respective assays' thresholds for positivity. This implies that the cut-off 326 values given for the respective test systems are only valid for the diagnosis of a past infection, but 327 All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. Serology assays to manage covid-19 Three-quarters attack rate of sars-cov-2 in the brazilian amazon during a largely 382 unmitigated epidemic Sars-cov-2 seroprevalence 384 survey estimates are affected by anti-nucleocapsid antibody decline An original 386 multiplex method to assess five different sars-cov-2 antibodies t cell immunity in cases of covid-19 and sars, and uninfected controls epitope mapping on microarrays highlights strong immune-response to n protein region A comprehensive antigen production and characterization study for 396 Molecular 444 interaction and inhibition of sars-cov-2 binding to the ace2 receptor Structural basis for the recognition of sars-447 cov-2 by full-length human ace2 A simple 449 protein-based surrogate neutralization assay for sars-cov-2 Evaluation of a 451 sars-cov-2 surrogate virus neutralization test for detection of antibody in human, canine, 452 cat, and hamster sera Comparison of potency assays to assess sars-cov-2 neutralizing antibody capacity in 455 covid-19 convalescent plasma All rights reserved. No reuse allowed without permission.(which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. All rights reserved. No reuse allowed without permission.(which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.The copyright holder for this preprint this version posted March 8, 2021. ; https://doi.org/10.1101/2021.03.05.21252977 doi: medRxiv preprint