key: cord-0779094-kno1k8k1
authors: Gdoura, M.; Halouani, H.; Mrad, M.; Sahli, D.; Chamsa, W.; Mabrouk, M.; Ben Salem, K.; Hogga, N.; Triki, H.
title: SARS-CoV2 serology assays: utility and limits of different antigen based tests through the evaluation and the comparison of four commercial tests
date: 2021-11-21
journal: nan
DOI: 10.1101/2021.11.19.21266615
sha: 3195d6fdcf7d83310acf26da138fe398a8cf214b
doc_id: 779094
cord_uid: kno1k8k1

Introduction: SARS-CoV2 serology testing is multipurpose provided to choose an efficient test. We evaluated and compared 4 different commercial serology tests, three of them had the Food and Drug Administration (FDA) approval. Our goal was to provide new data to help to guide the interpretation and the choice of the serological tests. Methods: Four commercial tests were evaluated: Cobas(R)Roche(R)(total anti-N antibodies), VIDAS(R)Biomerieux(R)(IgM and IgG anti-RBD antibodies), Mindray(R)(IgM and IgG anti-N and anti-RBD antibodies) and Access(R)Beckman Coulter(R)(IgG anti-RBD antibodies). Were tested: a positive panel (n=72 sera) obtained from COVID-19 confirmed patients and a negative panel (n=119) of pre-pandemic sera. Were determined the analytical performances and was drawn the ROC curve to assess the manufacturer's threshold. Results: A large range of variability between the tests was found. Mindray(R)IgG and Cobas(R) tests showed the best overall sensitivity 79,2%CI95%[67,9-87,8]. Cobas(R) showed the best sensitivity after D14; 85,4%CI95%[72,2-93,9]. The best specificity was noted for Cobas(R), VIDAS(R)IgG and Access(R) IgG(100%CI95%[96,9-100]). Access(R) had the lower sensitivity even after D14 (55,5% CI95%[43,4-67,3]). VIDAS(R)IgM and Mindray(R)IgM tests showed the lowest specificity and sensitivity rates. Overall, only 43 out of 72 sera gave concordant results (59,7%). Retained cut-offs for a significantly better sensitivity and accuracy, without altering significantly the specificity, were: 0,87 for Vidas(R)IgM(p=0,01), 0,55 for Vidas(R)IgG(p=0,05) and 0,14 for Access(R)(p<10-4). Conclusion: Although FDA approved, each laboratory should realize its own evaluation for commercial tests. Tests variability may raise some concerns that seroprevalence studies may vary significantly based on the used serology test.

(FDA-EUA) and an emergency use list (WHO-EUL), respectively (18) (19) (20) . Nowadays, 72 serology has become widely used with many indications (21, 22) . However, some tests may 73 lack sufficient clinical evaluation which made specialists establishing their own evaluation 74 and sharing concerns toward their performances(12, [14] [15] [16] [17] . 75

In the present study we conducted a head-to-head comparison of 4 different commercial 76 serology tests targeting either the N protein or the RBD protein or both of them; three of them 77 had an FDA-EUA. Our goal was to provide experimental data which help to guide the choice 78 of the serological tests according to their indication, as well as the interpretation of the 79 serology profiles. 80

The study was done in the Laboratory of Virology of the Institut Pasteur of Tunis, Tunisia IgM and IgG anti-N and anti-RBD antibodies and Access®Beckman Coulter® (CLIA) 96 detecting specific IgG anti-RBD antibodies. All sera, the 72 from confirmed cases and the 97 119 pre-pandemic sera were tested by the 4 tests and manipulations were carried out 98 according to the manufacturers' instructions. The intrinsic characteristics and the 99 manufacturer's performances of each test are summarized in Table 1 . 100

Were determined: the overall sensitivity, the sensitivity after D14, the specificity, the positive 101 predictive value PPV, the negative predictive value NPV, the area under the curve (AUC) 102 and the correlation between the results (in ratio index/cut-off index (ratios)) with the days 103 after COVID-19 confirmation. Based on the obtained ratios, Receiving operating 104 characteristic (ROC) curve for each test has been drawn using the obtained ratios, in order to 105 assess the manufacturer's threshold to achieve the best performances. Finally, the 4 tests were 106 compared between each other and pooled results of different tests were statistically studied. 107

Statistical Analysis: PPV and NPV were calculated using the FDA calculator available on its 108 website, accessed 22 July 2021 and arbitrary fixed the prevalence of the disease at 5%. Using 109 RT-PCR as the reference standard, sensitivity, specificity, AUC were calculated to assess the 110 performance of each assay and T-test was used to compare AUC for each test. The optimal 111 cutoff point was selected based on the point with the highest Youden index J. Agreement was 112 determined between all the tests, two by two, using the Cohen's Kappa statistic test. 113 Correlation between ratios was calculated by determining the Pearson Coefficient r. The 114 significance level was set at 5%, and a 95% confidence interval (CI95%) was reported for 115 each measure. All calculations were performed using MEDCALC®V18.2.1. 116

The performances of each test were evaluated by calculating the sensitivity, specificity, PPV, 118 NPV and AUC. All experimental results are shown in Table 2 . For IgM tests, VIDAS®IgM 119

and Mindray®IgM tests showed the lowest specificity and sensitivity rates for all the sera 120 All rights reserved. No reuse allowed without permission. perpetuity. preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in The copyright holder for this this version posted November 21, 2021. ; https://doi.org/10.1101 https://doi.org/10. /2021 and those collected after D14. The PPV and NPV were also the lowest. False positive tests 121 were obtained for 7 pre-pandemic patients for Vidas®IgM: 3 patients having rheumatoid 122 factors and 4 patients positive for Herpessimplex virus; the ratios ranged from 1,08 to 13,94. 123 False positive tests were obtained for 3 pre-pandemic patients for Mindray®IgM: 2 patients 124 having auto-immune disease and 1 patient positive for Herpessimplex virus; the ratios ranged 125 from 1,94 to 4,97. These two tests had good and similar accuracies (p=0,587). For IgG and 126 total antibody tests, Mindray®IgG and Cobas® tests showed the best overall sensitivity 127 79,2% CI95% [67, 8] (57 positive out of 72 true positive). Cobas® showed the best 128 sensitivity after D14 85,4%CI95% [72, 9] . For the other tests, the sensitivity increased 129 considerably after D14 except for Access® ( VIDAS®IgG and Mindray®IgG had very good and similar accuracy (paiwise comparison of 136 ROC curves for the 3 combinations p>0,05). However, Access® had an accuracy of 137 0,778CI95%[0,712-0,835] which is good but statistically lower than the other tests (p=0,587). 138 ROC curves for each test were drawn based on the obtained ratios ( Figure 1 ) as shown in 139

Tables 3. Retained cut-offs for a significantly better sensitivity and accuracy, without altering 140 significantly the specificity, were: 0,87 instead of 1 for Vidas®IgM (p=0,01), 0,55 instead of 141 1 for Vidas®IgG (p=0,05) and 0,14 instead of 1 for Access® (p<10 -4 ). For Cobas® and 142

Mindray®IgM and IgG, the new proposed cut offs did not give better analytical 143 performances than the original cut-offs (p>0,05, Table 3 ). 144 All rights reserved. No reuse allowed without permission. perpetuity. preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in The copyright holder for this this version posted November 21, 2021. ; https://doi.org/10. 1101 All experimental results obtained for the positive panel were summarized in Table 4  145 including concordance and discordance between the tests and in connexion also with the days 146 after confirmation. Overall, 43 ou of 72 sera gave concordant results (59,7% concordance). To further understand the differences between the tests results, we determined the agreement 157 between the results and the correlation between the ratios. In this section, the comparison was 158 focused on the isotype and on the antigen. For the IgM tests, Vidas® and Mindray® present 159 moderate agreement (k=0,570) and weak correlation (r=0,484). For the other tests, the results 160 of the agreement and correlation are grouped in Figure 2 . For tests detecting exclusively 161 antibodies against RBD, Vidas®IgG and Access®, concordance was important and 162 correlation is positive and strong. For Cobas®, the test that detects antibodies against N 163 exclusively, agreement was perfect with Vidas® and important with Access® and correlation 164 is positive and moderately strong with Vidas® but negative with Access®. Considering the 165 Mindray®, the test that detects both N and RBD specific antibodies, it presents a perfect 166 agreement with all tests except Access® and a positive correlation with all tests. 167 Figure 3 shows the scatterplots of the 4 tests' ratios against d. Figure 3 .a shows the 168 distribution of the IgM tests indexes (Vidas® and Mindray®), which is heterogenous et does 169 All rights reserved. No reuse allowed without permission. perpetuity. preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in The copyright holder for this this version posted November 21, 2021. ; https://doi.org/10.1101/2021.11.19.21266615 doi: medRxiv preprint not fit a specific pattern however it is obvious that high indexes were obtained during the first 170 21 days after infection. Figure 3 .b illustrates the distribution of total antibodies and IgG 171 antibodies tests indexes (Cobas®, Vidas®, Mindray® and Access®) emergence. Thanks to the the softened authorization procedure, they were rapidly 182 commercialized and used worldwide (18) (19) (20) . In this study, we evaluated and compared 4 183 serology commercial automated tests: Cobas®Roche® (ECLIA) detecting total anti-N 184 antibodies, IgG+++ (anti-N), VIDAS®Biomerieux® (ELFA) detecting specific IgM and IgG 185 anti-RBD antibodies, Mindray® (CLIA) detecting specific IgM and IgG anti-N and anti-186 RBD antibodies and Access®Beckman Coulter® (CLIA) detecting specific IgG anti-RBD 187 antibodies. Our evaluation revealed a gap between claimed and experimental analytical 188 performances in terms of sensitivity and specificity and, accordingly, we propose new 189 analytical criteria. In addition, the comparison between the evaluated tests showed a 190 significant divergence between the obtained qualitative results in 40,3% of the positive tested 191 sera (29 out of 72). Our findings suggest that the most sensitive test, after D14 is Cobas® 192 (85, 4%IC95%[72, 9] ) which detects high ratios until 4 months after primo-infection. 193

All rights reserved. No reuse allowed without permission. perpetuity. preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in The copyright holder for this this version posted November 21, 2021. ; https://doi.org/10. 1101 Besides, we found that combining RBD and N tests from different tests gives the best 194 accuracy. 195

We tested 72 RT-PCR confirmed patients and 119 pre-pandemic sera. Our work stands out 196 from the rest of the literature by studying tests of different antigen and having international Canada in order to harmonize the criteria of validation of the tests (23-26). In our study, the 209 evaluation of all tests gave lower performances than the claimed ones and did not respond to 210 the HAS validation criteria, the most flexible one, in terms of sensitivity (Table 1 ,2). 211

According to the HAS, the sensitivity of detecting IgG and total antibodies must exceed 90% 212 after D14 from disease onset while for IgM antibodies, the sensitivity must exceed 90% after 213 D7. All tests were studied specifically after D14, based on a large review published by 214

Cochrane on 15976 samples which found that all the results for IgG, IgM, IgA, total 215 antibodies showed low sensitivity during the first week after the symptoms onset (all less 216 than 30.1%), it rises in the second week and reaches its highest values in the third week (36). 217 All rights reserved. No reuse allowed without permission. perpetuity. preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in The copyright holder for this this version posted November 21, 2021. ; https://doi.org/10. 1101 In our series, the best sensitivity after D14 were the one of Cobas® (85, 4%CI95%[72, 9] ) followed by Vidas®IgG and Mindray®IgG (83, 3%CI95%[69, 5] ) and Access® 219 came in the last position(55,5% CI95% [43,4-67,3]) ( Table 2) . For IgM detection Mindray® 220 and Vidas® had very low sensitivity even after D14. High sensitivity for Cobas® confirms 221 the findings of other authors (12, 15, 16, 27) . This could be explained, first, by the used antigen 222 which is exclusively the N protein, known to be the most abundantly expressed immune-223 dominant protein (21). Second is the ability of Cobas® to detect all immunoglobulin classes 224 and such data was reported for Siemens Atellica® that detects total antibodies anti-225 glycoprotein S1 (28) and that has obtained the FDA-EUA and thus merits to be evaluated. 239

Regarding the specificity, Vidas®IgM and Mindray®IgM and IgG tests gave positive signals 240 for few pre-pandemic sera and this was also reported by other authors (31, 33, 34) . Cross 241 reactivity with pre-pandemic auto-immune disease patients sera was previously reported(37). 242 All rights reserved. No reuse allowed without permission. perpetuity. preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in The copyright holder for this this version posted November 21, 2021. ; https://doi.org/10. 1101 In contrast, cross reactivity with pre-pandemic pregnant women sera, and patients positive for 243

Herpessimplex virus is reported for the first time. Results of Vidas®IgM and Mindray®IgM 244 and IgG should be interpreted with caution; indeed, PPV of these tests are less than 50%, 245

which mean that half of tested patients are susceptible to be false positive. Here, PPV and 246 NPV were calculated by the FDA calculator fixing the prevalence at 5%. Each coutry is 247 invited to evaluate regularily the PPV according to the prevalence evolution. The tests 248

Cobas®, Access® and Vidas® IgG are the extremely specific (100%IC95% [96, ) and 249 the PPV is 100%, as reported by other studies (16,27,31) . 250

The pre-defined tests' thresholds were experimentally optimized and adjusted for an 251 improved sensitivity with very little loss in specificity. This approach is being widely used 252 and reported by many authors for better interpretation of commercial tests, for COVID-19 253 tests as well as other pathogens (31, 38, 39) . We found out that decreasing the cut-off signals 254 for Vidas®IgM, Vidas®IgG and Access® improves significantly the sensitivity and the 255 accuracy (Table3). As none of the tests propose a grey zone for borderline results, which is 256 unusual, we propose that any results between the proposed cut-off and the original cut-off 257 (i.e: [0,87 to 1] for Vidas®IgM, [0,55-1] for Vidas®IgG and [0, for Access®) should 258 be retested or, better, the patient should be re-sampled after 10 to 15 days to follow the 259 antibody kinetic. More generally, we suggest that any weak signal less than 2 times the cut-260 off, should be interpreted with caution. 261

Comparison between the four tests showed concordant results in 59,7% of the samples 262 collected in confirmed cases (43 out of 72) among which, 8 were negative by all the 4 tests; 263 they were sampled between D0 and D60, median=14. As the 4 different tests using different 264 antigens, gave negative results, it is suggested that this negativity is inherent to the 265 individuals. Indeed, this may be explained by either a late sero-conversion, or a rapid sero-266 reversion (40). Some authors has suggested that 5 to 10% of infected persons do not develop 267 All rights reserved. No reuse allowed without permission. perpetuity. preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in The copyright holder for this this version posted November 21, 2021. ; https://doi.org/10.1101 https://doi.org/10. /2021 antibodies at all (41). A non negligible proportion of discordant results was found in 40,3% 268 sera (n=29 out of 72). Access® was the test that fails the most to detect positive results (13 269 cases out of 29 discordant results, Table 4 In conclusion, although serological assays do not replace molecular tests in diagnosing active 291 infection, they are multipurpose provided to choose the most efficient test and to properly 292 All rights reserved. No reuse allowed without permission. perpetuity. preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in The copyright holder for this this version posted November 21, 2021. ; https://doi.org/10. 1101 interpret the results. We characterized the performance of four commercial antibody 293 platforms and found out and explained the variability between them. Although FDA 294 approved, each laboratory should realize its own evaluation for commercial tests, and health 295 professionals should be aware about false negative rate before 14 to 21 days after primo-296 infection. Finally, this variability may raise some concerns that seroprevalence studies may 297 vary significantly based on the used serology test. 

All rights reserved. No reuse allowed without permission.

perpetuity.

preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in

The copyright holder for this this version posted November 21, 2021. ; https://doi.org/10.1101 https://doi.org/10. /2021 perpetuity. preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in The copyright holder for this this version posted November 21, 2021. ;

1 

331 Evaluation of a Surrogate ELISA-Based Severe Acute Respiratory Syndrome Coronavirus 2 332 (SARS-CoV-2) cPass Neutralization Antibody Detection Assay and Correlation with IgG 333

Head-to-head evaluation on 335 diagnostic accuracies of six SARS-CoV-2 serological assays

Multicentre 338 Performance Evaluation of the Elecsys Anti-SARS-CoV-2 Immunoassay as an Aid

Determining Previous Exposure to SARS-CoV-2

Validation of a 341 commercially available indirect assay for SARS-CoV-2 neutralising antibodies using a 342 pseudotyped virus assay

Diagnostics for SARS-CoV-2 detection: A comprehensive 344 review of the FDA-EUA COVID-19 testing landscape

Diagnostics laboratory emergency use listing

Emergency Use Authorizations for Medical Devices

Practical guidance for clinical laboratories 357 for SARS-CoV-2 serology testing

organisation et de fonctionnement du système de santé nécessaires pour faire face à 360

épidémie de covid-19 dans le cadre de l'état d'urgence sanitaire -Légifrance

COVID-19 serological testing devices: Notice on sensitivity and specificity values

Target Product Profile: antibody tests to help determine if people have recent infection to SARS-370

CoV-2: Version 2

Positive IgG: n=71 out of 120