key: cord-0955307-g2gtz1lu authors: Bal, Antonin; Pozzetto, Bruno; Trabaud, Mary-Anne; Escuret, Vanessa; Rabilloud, Muriel; Langlois-Jacques, Carole; Paul, Adèle; Guibert, Nicolas; D’Aubarede-Frieh, Constance; Massardier-Pilonchery, Amélie; Fabien, Nicole; Goncalves, David; Boibieux, André; Morfin-Sherpa, Florence; Pitiot, Virginie; Gueyffier, François; Lina, Bruno; Fassier, Jean-Baptiste; Trouillet-Assant, Sophie title: Evaluation of high-throughput SARS-CoV-2 serological assays in a longitudinal cohort of patients with mild COVID-19: clinical sensitivity, specificity and association with virus neutralization test date: 2021-01-05 journal: Clin Chem DOI: 10.1093/clinchem/hvaa336 sha: bf64bf5a63e8a26065151a4b52543f389637e6f9 doc_id: 955307 cord_uid: g2gtz1lu BACKGROUND: The association between SARS-CoV-2 commercial serological assays and virus neutralization test (VNT) has been poorly explored in mild patients with COVID-19. METHODS: 439 serum specimens were longitudinally collected from 76 healthcare workers with RT-PCR-confirmed COVID-19. The clinical sensitivity (determined weekly) of nine commercial serological assays were evaluated. Clinical specificity was assessed using 69 pre-pandemic sera. Correlation, agreement and concordance with the VNT were also assessed on a subset of 170 samples. Area under the ROC curve (AUC) was estimated at 2 neutralizing antibody titers. RESULTS: The Wantai Total Ab assay targeting the receptor binding domain (RBD) within the S protein presented the best sensitivity at different times during the course of disease. The clinical specificity was greater than 95% for all tests except for the Euroimmun IgA assay. The overall agreement with the presence of neutralizing antibodies ranged from 62.2% (95%CI; 56.0-68.1) for bioMérieux IgM to 91.2% (87.0-94.2) for Siemens. The lowest negative percent agreement (NPA) was found with the Wantai Total Ab assay (NPA 33% (21.1-48.3)). The NPA for other total Ab or IgG assays targeting the S or the RBD was 80.7% (66.7-89.7), 90.3 (78.1-96.1) and 96.8% (86.8-99.3) for Siemens, bioMérieux IgG and DiaSorin, respectively. None of commercial assays have sufficient performance to detect a neutralizing titer of 80 (AUC<0.76). CONCLUSIONS: Although some assays show a better agreement with VNT than others, the present findings emphasize that commercialized serological tests including those targeting the RBD cannot substitute a VNT for the assessment of functional antibody response. The evaluation of the humoral immune response to SARS-CoV-2 with serological tests is crucial to further manage the coronavirus disease 2019 (COVID- 19) pandemic. Serological testing represents an easy to implement and cost-effective method allowing rapid identification of individuals exposed to the virus (1, 2) . Over the last few months, a large number of SARS-CoV-2 commercial assays have been evaluated for their ability to detect specific antibodies (3) (4) (5) (6) (7) (8) (9) . However, the detection of specific SARS-CoV-2 antibodies does not indicate whether or not the antibodies are functional for neutralizing the virus. In association with the assessment of other immune responses, such as cellular immunity, the exploration of the neutralizing antibody response is important to evaluate the protective immunity to SARS-CoV-2 after infection and therefore the risk of reinfection (10) (11) (12) (13) . To date, the association between SARS-CoV-2 commercial assay results and the presence of neutralizing antibodies has been mainly explored in hospitalized COVID-19 patients (14) (15) (16) (17) (18) . Because conflicting findings have been reported, it is unclear whether the commercial serological assays could be useful to assess the protective immunity against SARS-CoV-2. Virus neutralization test (VNT) is considered as the reference to assess the functional ability of antibodies to block the entry of the virus into human cells (19) . However, such an assay requires living virus manipulated in a biosafety level 3 facility that needs trained staff and specific equipment, and which is a tedious and time-consuming method. The first study exploring the association of commercial serological assays and VNT claimed that the Wantai Total Ab assay detecting total antibodies directed against the SARS-CoV-2 receptor binding domain (RBD) had the best characteristics to detect functional antibodies at different stages and severity of disease (14) . The RBD, within the sub-unit S1 of the spike protein, enables the viral entry into human cells by fixing to the angiotensin-converting enzyme 2 receptor (20) . As emphasized in this study (14) , there is an urgent need for further studies addressing the performance of alternative high-throughput assays in correlation with VNT among persons with mild COVID- 19 , which is the most common form of the disease. Thus, the aim of the present study was to evaluate widely-used SARS-CoV-2 serological tests and their potential association with VNT in a cohort of patients with mild mild COVID-19. A prospective longitudinal cohort study was conducted at the laboratory associated with the National reference center for respiratory viruses (University Hospital of Lyon, France) (21) . Healthcare workers (HCW) with symptoms suggesting a SARS-CoV-2 infection requiring a RT-PCR test were included (visit 1, V1). Clinical data including date of symptom onset were recorded for all included HCWs using an electronic case report form by trained clinical research associate. Patients with a positive RT-PCR result at inclusion (V1) returned weekly for 6 additional visits (V2 toV7). Serum samples were prepared from 5 mL of whole blood collected in BD Vacutainer® Serum Separating Tubes II Advance Tube (Beckon Dickinson). After collection tubes were shaken gently and serum were allowed to clot for a minimum 30 min at room temperature to obtain total coagulation, followed by centrifugation at 2,500 g for 10 min. Removed supernatants were frozen at -80°C until the serological assays. Written informed consent was obtained from all participants; ethics approval was obtained from the national review board for biomedical research in April 2020 (Comité de Protection des Personnes Sud Méditerranée I, Marseille, France; ID RCB 2020-A00932-37), and the study was registered on ClinicalTrials.gov (NCT04341142). A total of 439 serum specimens were longitudinally collected from 76 HCW. Among them, 74 had mild COVID-19 related symptoms (fever, cough, loss of taste or smell, diarrhea) and did not require hospital admission. Two out of 76 HCW were admitted to the hospital (not in intensive care unit, ICU) due to the severity of their symptoms . Among the 439 collected samples, 170 of them obtained at V2, V4, V7 from 57 patients were tested by VNT (for one patient the sample at V7 was missing). To compare neutralizing antibody titers between patients with mild and severe COVID-19 patients, 117 sera collected longitudinally from 44 patients with severe COVID-19 were also tested by VNT. These patients have been admitted to ICU at the University Hospital of Saint-Etienne and serum were collected longitudinally between March and May 2020. The median age was 70.4 y and 20% were female. All patients were sampled by nasal swab and had tested positive for SARS-CoV-2 RNA by RT-qPCR assay. In addition, to evaluate clinical specificity, we selected retrospectively 69 prepandemic sera (collected between April and July 2019) from 30 healthy volunteers (52% females, median age of 28 y, IQR: 21-34), 30 patients with autoimmune disorders, and 9 patients with a positive serological result for M. pneumoniae. The patients with autoimmune disorders (F: M ratio of 1.7; mean age 42.7 y) included 10 patients with antinuclear antibodies at a titer ≥ 320, 10 patients with rheumatoid factor associated (n=2) or not (n=8) with anti-citrullinated protein antibodies, and 10 patients with inflammatory bowel disease, 7 of whom exhibited antineutrophil cytoplasmic antibodies (n=2), anti-Saccharomyces cerevisiae antibodies (n=3) or antinuclear antibodies (n=2). The patients with a positive serological result for M. pneumoniae (55% female, mean age of 20 y) had IgM titers that ranged from 21 to > 27 UA/ml using the DiaSorin kit on the Liaison XL instrument. COVID-19 diagnosis at inclusion was performed by RT-PCR on nasopharyngeal swab using the cobas SARS-CoV-2 assay (Roche, Basel, Switzerland). A total of 9 serological assays (Abbott, DiaSorin, Siemens, Bio-Rad, Wantai Total and IgM, bioMérieux IgG and IgM, Euroimmun IgA) were investigated according to the protocol recommended by each manufacturer (characteristics are summarized in Table 1 ). Positivity was established according to the threshold value recommended by each manufacturer. A plaque reduction neutralization test (PRNT) was used for the detection and titration of neutralizing antibodies, as previously described (22) . Briefly, a ten-fold dilution of each serum specimen in culture medium (Dulbecco's Modified Eagle Medium containing antibiotics and 2% foetal calf serum) was first heated for 30 min at 56°C to avoid complement-linked reduction of the viral activity. Serial two-fold dilutions (tested in duplicate) of the serum specimens in culture medium were mixed at equal volume with the live SARS-CoV2 virus. After gentle shaking and a contact of 30 min at room temperature in plastic microplates, 150 µL of the mix was transferred into 96-well microplates covered with Vero E6 cells. The plates were incubated at 37°C in a 5% CO2 atmosphere. The reading was evaluated microscopically 5 to 6 days later when the cytopathic effect of the virus control reached 100 TCID50/150 µL. Neutralization was recorded if more than 50% of the cells present in the well were preserved. The neutralizing titer was expressed as the inverse of the higher serum dilution that exhibited neutralizing activity; a threshold of 20 was used (PRNT50 titer ≥ 20). All experiments were performed in a biosafety level 3 laboratory. The comparison of this VNT with a standardized assay using retroviruses pseudo-typed with the SARS-CoV-2 S viral surface protein found a high correlation and concordance (22) . For each test, the clinical sensitivity was estimated weekly after symptom onset considering SARS-CoV-2 RT-PCR results as the gold standard. For VNT, the overall positive and negative percent agreements (OPA, PPA, NPA) were determined for each commercial serological assay as previously described (23) . The correlation and concordance with the VNT were assessed using the Spearman and Cohen's Kappa coefficients, respectively. The concordance was classified as slight (Cohen's Kappa coefficient, Computing, Vienna, Austria). A p-value < 0.05 was considered as statistically significant. During the first week after the onset of symptoms the sensitivity for the detection of SARS-CoV-2 antibodies ranged from 6.6% (DiaSorin, Liaison) to 25 Table 2) . The neutralizing capacity of antibodies was determined at three time points for 57 patients (n=170 samples, Figure 1 ). No neutralizing antibodies were detected in 42.0% (21/50), 5.8% (3/51), and 8.7% (6/69) of samples collected between, respectively, 1-14, 15-28, and more than 28 days after symptom onset. Of note, three out of 57 patients had no neutralizing antibodies throughout their follow-up. For comparison, we also determined the titers of neutralizing antibodies in sera longitudinally collected from COVID-19 patients (n=44) admitted to an ICU (n=117 samples, Figure 1 In a longitudinal study of 76 HCW with RT-PCR-confirmed COVID-19, we found that the Wantai Total Ab assay had the best clinical sensitivity over the course of the disease. In particular, the sensitivity reached and remained at 100% as soon as week 3 post symptom onset. This finding observed in patients with mild COVID-19 is consistent with previous reports of excellent sensitivity of this test, notably in severe patients (3, 14) . Importantly, the sensitivity of the commercial tests can be higher in patients with severe COVID-19 in line with a stronger humoral immune response. In particular, we found lower neutralizing antibody titers in mild patients than in ICU patients in the present study. These findings are consistent with previous studies (25) (26) (27) (28) and raise questions about protective immunity after an infection although the immune response is not exclusively driven by the neutralizing antibody response. The immunological correlates of protection as well as the durability of natural immunity are still unknown, but patients with mild symptoms who had a low neutralizing antibody titer may be insufficiently protected to prevent a reinfection. The occurrence of reinfections in humans has been explored during a SARS-CoV-2 outbreak with a high attack rate (85.2%), which showed that individuals with preexisting neutralizing antibodies were not infected (10) . Further studies are needed to investigate the correlation between neutralizing antibody titers and the risk of reinfections in mild COVID-19 patients. In addition, following the FDA recommendation regarding the required titer for convalescent plasma donors (titer ≥ 160), the data presented here show that only a few patients with mild COVID-19 could be eligible. Thus, the ability of a commercial test to assess the neutralizing antibody response needs to be determined. With this aim, a prior study compared three commercial assays (Roche Total Ab, Abbott IgG, both tests targeting the N protein, and Euroimmun IgG assays targeting the S protein) to VNT on 66 specimens (17) , and found that the NPA was >90% for all assays only at a low neutralizing titer of 20 while the NPA dramatically decreased when higher neutralizing titers were used, making them imperfect proxies for neutralization. For instance, the NPA for neutralizing titers was <60% for all 3 the assays at a cutoff of 128. This study also suggested an increase in the manufacturer cutoff in order to improve the NPA, which can be very useful in a vaccination setting or for plasma donor screening (17) . Although the study design above (17) was different, notably regarding the disease severity of the patients enrolled, these findings are highly consistent with those of our present study that found the lowest NPA for the Wantai Total Ab assay (33%) and a NPA below 90% for all tests except for bioMérieux IgG and DiaSorin. Importantly we also found with an AUC analysis that all the commercial tested performed poorly at a neutralizing antibody titer of 80. Furthermore, the concordance between VNT and the Wantai Total Ab assay was only moderate while the concordance was substantial with bioMérieux IgG, DiaSorin, Siemens, Abbott, Euroimmun IgA and Bio-Rad. The low NPA and moderate concordance noticed for the Wantai Total Ab might be partially explained by the ability of this test to detect RBD-specific antibodies at the very early phase of infection, irrespective of their neutralizing properties in line with the delay required for antibody maturation (29) . The first study comparing VNT with commercialized tests (14) found that the Wantai Total Ab assay had the best characteristics to detect functional antibodies in different stages and severity of disease. However, the median interval between the onset of symptoms and sample collection was 43 days for the samples obtained from patients with mild symptoms (n=71 samples). Thus, the antibodies could be detected with both the Wantai Total Ab and the VNT assay at this time, explaining the high PPA values observed (14) . In our study as well as other reports (15, 16, 17) , a high PPA with VNT was also found for most of the commercial serological assays. Nevertheless, for determining the presence of neutralizing antibodies in serum specimen with commercial assays, the NPA should be maximized to avoid misinterpretation. Furthermore, as previously reported (29, 30, 31) , not all RBD-binding antibodies have neutralizing properties, which is consistent with our findings that the RBD-based assays do not have perfect concordance with VNT. Conversely, antibodies targeting a region other than the S protein may have functional activity (19, (32) (33) (34) . In the present study, the Abbott and Bio-Rad assays directed against the N protein presented a substantial concordance with VNT. Ndirected and RBD-neutralizing antibodies can be produced concomitantly over the course of the disease, which can also explain this finding. In addition to the different targeted antigens, the heterogeneity in assay performance found here could be related to various factors including the detected isotypes. Moreover, antibody concentrations may also be very different according to the time since symptom onset and according to clinical severity of the disease (25) . In our study, serum samples were collected longitudinally from disease diagnosis, enabling us to explore the early phase of the antibody response in a cohort of HCW, which constitutes one of the main strengths of the present study. The present study does, however, have certain limitations. For instance, clinical specificity was not extensively studied; yet, the Euroimmun IgA assay seemed to have the worst specificity, which is consistent with previous studies reporting a lack of specificity for this assay (4, 5, 14) . In addition, the performance of other notable commercial assays such as Euroimmun IgG or Roche Ig Total were not assessed. Second, not all the samples were systematically tested by VNT, in-line with the labor-intensive nature of this method. Finally, the size of the tested population remains small, contributing to wide CI, which limits the interpretation and extrapolation of the results. The results presented here confirm that the Wantai Total Ab assay showed the higher sensitivity for detecting SARS-CoV-2 antibodies after exposure. For the screening of neutralizing antibodies in serum specimens, an optimized cut-off maximizing the NPA needs to be established, as previously suggested for the Wantai Total Ab assay (14) . However, the data presented here suggest that other tests targeting the S protein as Siemens, DiaSorin or bioMérieux IgG might be more useful for this indication. These tests or others cannot substitute for a VNT for assessing functional antibody response; neutralizing assays remain the gold standard and easy-to-use tests, such as those based on pseudoviruses (5, 22, 35) , should be developed and standardized. Furthermore, the recent development of surrogate virus neutralization tests based on antibody-mediated blockage of the interaction between ACE-2 receptor and the RBD is very promising as they were designed in an ELISA format, enabling high-throughput testing (30, 36, 37) . In conclusion, our study provides original data concerning the performance of widely-used serological tests, which could help diagnostic laboratories in the choice of a particular assay according to the intended use. OPA, overall percent agreement; NPA, negative percent agreement, PPA positive percent agreement. Serology assays to manage COVID-19 The important role of serology for COVID-19 control Comparison of eight commercial, high-throughput, automated or ELISA assays detecting SARS-CoV-2 IgG or total antibody Diagnostic performance of seven rapid IgG/IgM antibody tests and the Euroimmun IgA/IgG ELISA in COVID-19 patients Validation of a commercially available SARS-CoV-2 serological immunoassay Diagnostic accuracy of serological tests for covid-19: systematic review and metaanalysis Performance characteristics of four highthroughput immunoassays for detection of IgG antibodies against SARS-CoV-2 Performance characteristics of five immunoassays for SARS-CoV-2: a head-to-head benchmark comparison Antibody response against SARS-CoV-2 spike protein and nucleoprotein evaluated by four automated immunoassays and three ELISAs Neutralizing Antibodies Correlate with Protection from SARS-CoV-2 in Humans during a Fishery Vessel Outbreak with a High Attack Rate SARS-CoV-2 T cell immunity: Specificity, function, durability, and role in protection A systematic review of antibody mediated immunity to coronaviruses: kinetics, correlates of protection, and association with severity Evaluation of three SARS CoV-2 IgG antibody assays and correlation with neutralizing antibodies An evaluation of COVID-19 serological assays informs future diagnostics and exposure assessment Performance of six SARS-CoV-2 immunoassays in comparison with microneutralisation Commercial serology assays predict neutralization activity Against SARS-CoV-2 Association between SARS-CoV-2 neutralizing antibodies and commercial serological assays Comparative performance of five commercially available serologic assays to detect antibodies to SARS-CoV-2 and identify individuals with high neutralizing titers Neutralizing Antibodies against SARS-CoV-2 and Other Human Coronaviruses A pneumonia outbreak associated with a new coronavirus of probable bat origin Assessment of serological techniques for screening patients for COVID-19 (COVID-SER): a prospective, multicentric study A longitudinal study of SARS-CoV-2 infected patients shows high correlation between neutralizing antibodies and COVID-19 severity Enabling Precision Medicine With Digital Case Classification at the Point-of-Care The measurement of observer agreement for categorical data Differences in antibody kinetics and functionality between severe and mild SARS-CoV-2 infections Convalescent plasma for COVID-19: male gender, older age and hospitalisation associated with high neutralising antibody levels Neutralizing antibody against SARS-CoV-2 spike in COVID-19 patients, health care workers and convalescent plasma donors Low serum neutralizing anti-SARS-CoV-2 S antibody levels in mildly affected COVID-19 convalescent patients revealed by two different detection methods Human neutralizing antibodies elicited by SARS-CoV-2 infection A SARS-CoV-2 surrogate virus neutralization test based on antibody-mediated blockage of ACE2-spike proteinprotein interaction Serological assays for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) Coronaviruses pandemics: Can neutralizing antibodies help? Perspectives on the development of neutralizing antibodies against SARS-CoV-2 Neutralizing and cross-reacting antibodies: implications for immunotherapy and SARS-CoV-2 vaccine development Establishment and validation of a pseudovirus neutralization assay for SARS-CoV-2 A SARS-CoV-2 serological assay to determine the presence of blocking antibodies that compete for human ACE2 binding SARS-CoV-2 assays to detect functional antibody responses that block ACE2 recognition in vaccinated animals and infected patients We thank all the personnel of the occupational health and medicine department of Hospices Civils de Lyon who contributed to the samples collection as well as the Centre de Ressources Biologiques of HCL for biobanking. We thank Fanny Joubert and the PMO team for their help for patient's inclusion. We thank Lucie Charreton; Kahina Saker and Khadija Sfouli for their excellent work concerning serological testing, all the technicians from the virology laboratory whose work made it possible to obtain all these data, as well as Amira Lachekhab and Naima Rolnin for their technical assistance. We thank Karima Brahami and all members of the clinical research and innovation department for their reactivity (DRCI, Hospices Civils de Lyon). We thank Philip Robinson (DRCI, Hospices Civils de Lyon) for his help in manuscript preparation.