key: cord-0777445-cdko13l3
authors: Santano, R.; Barrios, D.; Crispi, F.; Crovetto, F.; Vidal, M.; Chi, J.; Izquierdo, L.; Gratacos, E.; Moncunill, G.; Dobano, C.
title: Agreement between commercially available ELISA and in-house Luminex SARS-CoV-2 antibody immunoassays
date: 2021-03-10
journal: nan
DOI: 10.1101/2021.03.09.21252401
sha: 10174e3b424306d9bb534f49602e938e05190178
doc_id: 777445
cord_uid: cdko13l3

Serological diagnostic of the severe respiratory distress syndrome coronavirus 2 (SARS-CoV-2) is a valuable tool for the determination of immunity and surveillance of exposure to the virus. In the context of an ongoing pandemic, it is essential to externally validate widely used tests to assure correct diagnostics and epidemiological estimations. We evaluated the performance of the COVID-19 ELISA IgG and IgM/A (Vircell, S.L.) against a highly specific and sensitive in-house Luminex immunoassay in a set of samples from pregnant women and cord blood. The agreement between both assays was moderate to high for IgG but low for IgM/A. Considering seropositivity by either IgG and/or IgM/A, the technical performance of the ELISA was highly imbalanced, with 96% sensitivity at the expense of 22% specificity. As for the clinical performance, the negative predictive value reached 87% while the positive predictive value was 51%. Our results stress the need for highly specific and sensitive assays and external validation of diagnostic tests with different sets of samples to avoid the clinical, epidemiological and personal disturbances derived from serological misdiagnosis.

and sensitive in-house Luminex immunoassay in a set of samples from pregnant women and 23 cord blood. The agreement between both assays was moderate to high for IgG but low for 24 IgM/A. Considering seropositivity by either IgG and/or IgM/A, the technical performance of 25 the ELISA was highly imbalanced, with 96% sensitivity at the expense of 22% specificity. As 26 for the clinical performance, the negative predictive value reached 87% while the positive 27 predictive value was 51%. Our results stress the need for highly specific and sensitive 28 assays and external validation of diagnostic tests with different sets of samples to avoid the 29 clinical, epidemiological and personal disturbances derived from serological misdiagnosis. 30 KEYWORDS 31 COVID-19; SARS-CoV-2; IgA; IgG; IgM; ELISA; Luminex; antibody; coronavirus; immunity; 32 immunoassay; performance; sensitivity; specificity 33 . CC-BY-NC-ND 4.0 International license It is made available under a perpetuity.

is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

The copyright holder for this this version posted March 10, 2021. ; https://doi.org/10.1101 https://doi.org/10. /2021 INTRODUCTION 34 In the context of an ongoing pandemic, the importance of accurate diagnostic methods for 35 disease control and elimination has been underpinned. Coronavirus disease 2019 serological diagnostic based on antibody detection against the severe respiratory 37 distress syndrome coronavirus 2 (SARS-CoV-2) is a valuable tool that allows for the 38 determination of immunity development and surveillance of exposure to the virus (1,2). In the 39 case of COVID-19, it is thought that antibodies mediate protection via a myriad of functions. 40

(3). Therefore, good serological tests are required not only for epidemiological surveillance 41 and policy implementation, but also for helping elucidate the mechanisms involved in 42 protection and the susceptibility to reinfection after exposure to the virus (1,2). Additionally, 43 serological tests may be also useful for establishing vaccine induced protection and finding 44 blood donors that qualify to obtain plasma to be used as treatment for severe 2, 4) . 46

The definition of a good serological test in terms of technical performance is based on the 47 values of specificity (SP) and sensitivity (SE). The clinical relevance of a serological test is 48 defined by the negative (NPV) and positive (PPV) predictive values. These last two 49 parameters depend on the prevalence of the disease, while theoretically SP and SE do not. 50

Since achieving a high score of both SP and SE is generally difficult, prioritizing one of them 51 during the test development process has personal, social and clinical implications, especially 52 in the context of a pandemic where prevalence of disease might not be correctly defined. 53

Several antibody detection tests for SARS-CoV-2 are available in the market, such as rapid 54 diagnostic tests (RDT), enzyme-linked immunosorbent assays (ELISA), neutralization 55 assays, and chemiluminescent immunoassays (CLIA) (5). Here we compared the 56 performance of a commercially available Granada, 57 Spain) with the performance of an in-house fluorescence-based, high-throughput and 58 multiplex Luminex immunoassay (6). The Vircell COVID-19 ELISA (from now on ELISA) 59 detects specific IgG or IgM and IgA together (IgM/A) against the nucleocapsid (N) and spike 60 . CC-BY-NC-ND 4.0 International license It is made available under a perpetuity.

is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

The copyright holder for this this version posted March 10, 2021. ; https://doi.org/10.1101 https://doi.org/10. /2021 4 (S) antigens adsorbed on solid phase (7). The Luminex immunoassay has been optimized 61 for the detection of specific IgG, IgM and IgA separately against a multiplex panel of 5 62 antigens adsorbed on magnetic beads that are in suspension (6). 63

Thanks to the simplicity of the method and its automatization, ELISA is used in the clinical 64 practice (8-11). Internal validation from Vircell, S.L. reports a 88% SE and 99% SP for IgM/A 65 assay and a 85% SE and 98% SP for the IgG assay (7). External validation of widely used 66 tests during this COVID-19 pandemic is essential to assure correct diagnostics and 67 epidemiological estimations. We have previously reported that the in-house Luminex assay 68 reaches up to 100% SP and 95.78% SE (6). Our aim is to report the performance of the 69 ELISA IgG and IgM/A externally assessed using our highly specific and sensitive Luminex 70 immunoassay. 71

We analyzed 283 samples from peripheral blood from pregnant women and cord blood 74 (Table 1) . Of those, 168 mothers belonged to a larger cohort of pregnant women whose 75 samples were collected in the context of a study focused on the effects of COVID-19 on 76 pregnancy outcomes. The study population and sample collection methods have been 77 described elsewhere (12). The selection of samples in this report was based on the results 78 obtained by ELISA and the suspicion of low SP by the IgM/A assay. In particular, it was the 79 detection of 3 IgG and IgM/A positive tests in cord blood samples paired with mothers who 80 had evidence of past and/or present infection that led to further investigation. While IgG 81 crosses the placenta efficiently, a very limited amount of IgA passes from mother to fetus and 82 IgM does not cross the placenta during normal pregnancies (13). As a result, the presence of is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint samples were diluted and, in the case of the IgM/A assay, they were then incubated with an 93

IgG sorbent to eliminate IgG from plasma and any possible interference. Then, in both 94 assays, samples and controls from the kit were incubated at 37ºC for 45 min. After a washing 95 step, they were incubated with peroxidase-conjugated detection antibodies (anti-human IgG 96 or anti-human IgM+IgA) at 37ºC for 30 min. After another wash, they were incubated with 97 substrate solution for 20 min in the dark. Finally, stop solution was added and the optical 98 density was read at 450 nm. The cut-off value was established based on the manufacturer's 99 procedure. The ELISA IgG and IgM/A assays are based on the detection of specific 100 antibodies against the N and S antigens adsorbed on a solid surface. Quantification is based 101 on optical density after the reaction of the enzyme-linked secondary antibodies in contact 102 with a substrate, and it is detected in a spectrophotometer. 103

The in-house Luminex immunoassay has been optimized for the detection of specific IgG, 105

IgM and IgA separately against a multiplex panel of antigens adsorbed on magnetic beads 106 that are in suspension (6). Each magnetic bead region is characterized by a unique mix of 107 two fluorochromes that allows their identification by laser excitation. Each antigen is coupled 108 to a specific bead region and the multiplex panel included: the full-length N (N FL) antigen, a 109 specific C-terminal region of N (N CT) (14), the full-length S, the subunit 2 from the S antigen 110 (S2) and the receptor binding domain (RBD) in S1. Quantification is based on the detection 111 of fluorescence emission by a phycoerythrin-labelled secondary antibody and it is detected in 112 a Luminex FLEXMAP 3D instrument. 113 . CC-BY-NC-ND 4.0 International license It is made available under a perpetuity.

is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint 

Most of the IgG positive samples detected by ELISA were also classified as IgG positive by 116 each of the antigens included in the Luminex panel (Figure 1 and Table 2 ), and many of the 117 negative samples classified by ELISA were also classified as negative by each of the 118 antigens included in the Luminex panel. 119

For IgG, the percentage of agreement ranged from 56.10% for N CT up to 93.10% for S2 for 120 positive samples, and from 87.84% for N CT up to 98.61% for S2 for negative samples 121 ( Table 2 ). The antigen from the Luminex panel with the highest percentage of both positive 122 and negative agreement for IgG was S2, which is part of the S antigens included in the 123

Considering the classification obtained by the combination of IgG responses against all 125 antigens included in the Luminex multiplex panel, the positive agreement was 54% while the 126 negative agreement was 84.03% ( Table 2) . This resulted in 23 samples (all from mothers) 127 with discrepancy for IgG: 1 sample Luminex negative and ELISA positive (LM-E+) and 22 128 samples Luminex positive and ELISA negative (LM+E-) (Figure 1 and Table 2 ). Among 129 these discrepant samples, 19 out of the 22 LM+E-samples were asymptomatic and either 130 PCR negative or were not PCR tested, 1 sample was asymptomatic and PCR positive, 1 131 sample was symptomatic 1-2 months prior to sample collection and PCR negative, and 1 132 sample was symptomatic during the previous 7 days to sample collection and PCR positive. 133

The only LM-E+ sample was asymptomatic and PCR negative. The 3 cord blood samples 134 had concordant results between Luminex and ELISA. This most certainly reflects 135 transplacental transfer of maternal IgG, since the 3 mothers had detectable levels of specific 136

IgG detected by both tests and IgG is able to cross the placenta (13). 137

Since the in-house Luminex assay has previously been assessed and it demonstrated an 138 excellent performance (6), we used the results obtained by Luminex as the gold standard to 139 evaluate the performance of ELISA ( Table 3) . The usage of Luminex as gold standard is 140 . CC-BY-NC-ND 4.0 International license It is made available under a perpetuity.

is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

The copyright holder for this this version posted March 10, 2021. ; https://doi.org/10.1101/2021.03.09.21252401 doi: medRxiv preprint 7 further supported by the inclusion of several antigenic regions and three immunoglobulin 141 isotypes that allow capturing a variety of immunological responses that cannot be attained by 142 a PCR test, for example. Therefore, we evaluated the ELISA serological test based on a 143 serological reference (Luminex) assay. For IgG, the SE was 55%, while the SP was 99%. As 144 for the PPV and NPP, the percentages were 96% and 85%, respectively. Taking into account 145 only the subjects with a PCR test done (N= 110), for IgG, the SE was 62%, while the SP was 146 90%. As for the PPV and NPP, the percentages were 44% and 95%, respectively. Considering the classification obtained by the combination of all the antigens included in the 162 Luminex multiplex panel, the positive agreement was 23.29% for IgM and 30.14 % for IgA, 163 while the negative agreement was 17.65% for IgM and 17.74% for IgA ( Table 2) is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

The copyright holder for this this version posted March 10, 2021. Regarding the performance of ELISA (Table 3) using Luminex results as gold standard, the 186 SE was 97% for IgM and 94% for IgA, while the SP was as low as 18% for both IgM and IgA 187 due to the many false positive results. As for the clinical performance parameters, PPV was 188 23% for IgM and 30% for IgA, while the NPP was 96% for IgM and 88% for IgA. Regarding 189 the performance of ELISA based on subjects with a PCR test done, IgM/A reached a 100% 190 SE while the SP was as low as 23% due to the many false positive. As for the PPV and NPP, 191 the percentages were 15% and 100%, respectively. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

The copyright holder for this this version posted March 10, 2021. ; https://doi.org/10.1101/2021.03.09.21252401 doi: medRxiv preprint moment, there is no recognized standard serological test to assess the sensitivity and 196 specificity of other serological assays. Generally, neutralization assays and PCR test have 197 been considered as the gold standard in other studies (17-23), but other assays such as 198 CLIA have also been used (24). In this case, the use of PCR results as gold standard 199 showed a very similar technical performance (SE and SP) compared to that obtained by 200 using Luminex as stated above, although the set of samples used was slightly different since 201 not all samples were from individuals with a PCR test done. 202

Considering the results based on seropositivity by any of the isotypes included in the 204 immunoassays (Table 3) , the calculation of technical performance metrics resulted in a 205 highly imbalanced performance for ELISA, with 96% SE at the expense of a 22% SP. As for 206 the clinical performance, the NPV reached 87% while de PPV was 51%. 207

When prevalence of infection is low, the PPV of a test strongly relies on a high SP. For 208 example, given a 95% SE and 5% prevalence, a decrease in SP from 100% to 95% would 209 result in PPV dropping from 100% to 50%. On the contrary, in a scenario of 95% SE and 210 20% prevalence, a decrease in SP from 100% to 95% would result in a less steep fall of PPV 211 from 100% to 83% (25). Conversely, changes in SE within the same context of prevalence 212 do not affect so markedly the PPV and NPV. 213

In the case of ELISA, the low SP and PPV that we report here has important implications if 214 this test is to be used in hospitals for their screenings. In particular, a 51% PPV implies that is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint increased with days since onset of symptoms and disease severity (21, 22, 27) . In our case, 230 the SE for IgG does not reach the highest value reported by any of these papers. 231

A limitation of this study was the impossibility to stratify by days since onset of symptoms due 232 to the small sample size. Ideally, performance validation should be done stratifying by days 233 since onset of symptoms due to the kinetics of antibody production, which is delayed with 234 respect to the onset of the infection. This is especially relevant in the case of IgG, which is 235 the last immunoglobulin to develop during the course of an immune response. Therefore, the 236 performance of a test highly depends on the time passed since the onset of infection, which 237 can be monitored by the onset of symptoms or time since positivity of PCR in asymptomatic 238 cases. In fact, our Luminex assay and others have demonstrated that performance reaches 239 excellent levels at >10 o >14 days since onset of symptoms (6, 21, 22, 27) . Therefore, IgG SE 240 assessment in this report would probably be higher had we stratified by days since onset of 241 symptoms. 242

Concerning IgG SP, the same studies report the following ranges: 53% (22) is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

The copyright holder for this this version posted March 10, 2021. ; https://doi.org/10.1101/2021.03.09.21252401 doi: medRxiv preprint 11 we report here but also show a very poor performance due to an elevated number of false 251 positive samples. In fact, IgM is well-known for being a source of false positive results in 252 immunoassays for many other infectious diseases due to its high non-specific reactivity, 253 caused by cross-reactivity with other pathogens and the formation of rheumatoid factor (28). 254

However, the latter is solved in the ELISA IgM/A from Vircell by the use of an IgG sorbent 255 that removes rheumatoid factor complexes. 256

One explanation for the disagreement observed in some cases could be the source of 257 biological samples. One study has reported an unsatisfactory performance of an 258 immunochromatographic IgM/IgG rapid test in pregnant women due to an elevated false 259 positive rate, and argue that the complexity of the immunological changes during pregnancy 260 might be the underlying reason (24). Additionally, detection of cross-reactive antibodies 261 generated by other coronaviruses or infectious diseases has been described and could be a 262 source of false positive results not only for IgM (14, 22) . 263

In conclusion, we show a very low SP for the ELISA IgM/A compared to the high values 264 reported by Vircell. In addition, the SE for the ELISA IgG was also lower than expected, 265 although this value would probably be higher if only samples with >14 days since onset of 266 symptoms were considered. Our results stress the need for highly specific and sensitive 267 assays and external validation of diagnostic tests with different sets of samples. 268 is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

The copyright holder for this this version posted March 10, 2021. ; https://doi.org/10.1101/2021.03.09.21252401 doi: medRxiv preprint 

The important role of serology for COVID-19 control

Serodiagnostics for Severe Acute Respiratory Syndrome-Related Coronavirus 2 : A 290

Dissecting antibody-mediated protection against SARS-CoV-2

The placenta

immunoglobulins, and safety assessment of biopharmaceuticals in pregnancy

Immunogenicity and crossreactivity of antibodies to the nucleocapsid protein of SARS-336

CoV-2: utility and limitations in seroprevalence and immunity studies

Transport 339 of maternal immunoglobulins through the human placental barrier in normal 340 pregnancy and during inflammation

Transplacental transmission of SARS-CoV-2 infection

Performance of six SARS-CoV-2 immunoassays in comparison with 346 microneutralisation

SARS-CoV-2 serological tests for the diagnosis of COVID-19 through the evaluation of 350 three immunoassays: Two automated immunoassays (Euroimmun and Abbott) and 351 one rapid lateral flow immunoassay (NG Biotech)

Brief clinical evaluation of six 354 high-throughput SARS-CoV-2 IgG antibody assays

Evaluation of diagnostic accuracy of 10 358

It is made available under a perpetuity.is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprintThe copyright holder for this this version posted March 10, 2021. ; https://doi.org/10.1101 https://doi.org/10. /2021 19 is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprintThe copyright holder for this this version posted March 10, 2021. ; https://doi.org/10.1101/2021.03.09.21252401 doi: medRxiv preprint