key: cord-0866671-dwcofvm2
authors: Tešija Kuna, Andrea; Hanžek, Milena; Vukasović, Ines; Nikolac Gabaj, Nora; Vidranski, Valentina; Ćelap, Ivana; Miler, Marijana; Stančin, Nevenka; Šimac, Brankica; Živković, Marcela; Žarak, Marko; Kmet, Marta; Jovanović, Marijana; Tadinac, Sanja; Šupraha Goreta, Sandra; Periša, Josipa; Šamija, Ivan; Štefanović, Mario
title: Comparison of diagnostic accuracy for eight SARS-CoV-2 serological assays
date: 2021-02-15
journal: Biochem Med (Zagreb)
DOI: 10.11613/bm.2021.010708
sha: 05d81e5bf9067780e2005cda1c1a63e221dd190c
doc_id: 866671
cord_uid: dwcofvm2

INTRODUCTION: Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) serological tests have been suggested as an additional diagnostic tool in highly suspected cases with a negative molecular test and determination of seroprevalence in population. We compared the diagnostic performance of eight commercial serological assays for IgA, IgM, and IgG antibodies to the SARS-CoV-2 virus. MATERIALS AND METHODS: The comparison study was performed on a total of 76 serum samples: 30 SARS-CoV-2 polymerase chain reaction (PCR)-negative and 46 SARS-CoV-2 PCR-positive patients with asymptomatic to severe disease and symptoms duration from 3-30 days. The study included: three rapid lateral flow immunochromatographic assays (LFIC), two enzyme-linked immunosorbent assays (ELISA), and three chemiluminescence immunoassays (CLIA). RESULTS: Agreement between IgM assays were minimal to moderate (kappa 0.26 to 0.63) and for IgG moderate to excellent (kappa 0.72 to 0.92). Sensitivities improved with > 10 days of symptoms and were: 30% to 89% for IgM; 89% to 100% for IgG; 96% for IgA; 100% for IgA/IgM combination; 96% for total antibodies. Overall specificities were: 90% to 100% for IgM; 85% to 100% for IgG; 90% for IgA; 70% for IgA/IgM combination; 100% for total antibodies. Diagnostic accuracy for IgG ELISA and CIA assays were excellent (AUC ≥ 0.90), without significant difference. IgA showed significantly better diagnostic accuracy than IgM (P < 0.001). CONCLUSION: There is high variability between IgM assays independently of the assay format, while IgG assays showed moderate to perfect agreement. The appropriate time for testing is crucial for the proper immunity investigation.

Tešija Kuna A. et al. Comparison of eight SARS-CoV-2 serological assays teins (S) are anchored. The interior of the particle contains nucleocapsid (N) proteins bound to the RNA molecule (4) . Spike glycoprotein is a transmembrane protein responsible for the crown-like appearance (corona) of the virus particle and contains two functional subunits: S1 and S2. The S1 subunit is responsible for binding to the host-cell receptor, angiotensin-converting enzyme 2 (ACE2) via receptor-binding domain (RBD), while the S2 subunit contains elements essential for fusion of the virus (1, 4) . The incubation period usually lasts from 4-6 days, with the onset of symptoms within 14 days in 95% of cases (1) . Clinical presentation varies from asymptomatic through mild and moderate symptoms which include cough, fever, shortness of breath, asthenia, arthralgia, myalgia, anosmia, and ageusia to the very severe and critical cases of severe pneumonia, septic shock, and acute respiratory distress syndrome (ARDS) (1, 5) .

Currently, the gold standard for diagnosis of SARS-CoV-2 infection is the detection of the viral sequence by reverse transcriptase polymerase chain reaction (RT-PCR) on respiratory tract specimens (usually from the upper respiratory tract taken by oropharyngeal or nasopharyngeal swabs). Unfortunately, the sensitivity of this method is highly dependent on preanalytical and analytical variables such as the type of the specimen (upper or lower respiratory tract), sampling technique, the timing of the sampling, or conditions during transportation of the sample, and false-negative results can be attributed to all of these (1, 6) . According to current opinion, diagnostic use of serological tests has limited use as an additional aid to molecular testing for patients who are highly suspected for infection but repeatedly negative on molecular testing and in deciding on the discharge of patients recovered from COVID-19 but still RT-PCR positive (7) . In an immunocompetent individual, the production of virus-specific host antibodies of IgA and IgM isotype is consistent with an acute phase infection, while IgG isotype appears with the later phase of infection. The diagnostic relevance of the serological test is highly dependent on the proper timing of sampling. However, the kinetics of immunologic response in SARS-CoV-2 infection is still unclear, as well as the duration of persistence and the protective role of circulating antibodies (8) . Highly variable median seroconversion times across studies have been reported; for IgM 5-17 days and for IgG 6-14 days, which can be attributed to different study populations, the timing of sample, but also to different performances of the used assays (9) .

In response to an urgency, a huge number of commercial serological tests in different formats have been rapidly developed with unknown or questionable clinical performance. This primarily refers to the numerous rapid easy-to-use devices offering combined detection of IgM and IgG antibodies to facilitate use outside of limited laboratory capacities. Unfortunately, many of these tests are not properly validated to accomplish the proposed role. There is an emerging number of articles with topics on the diagnostic performance of various SARS-CoV-2 serological assays including rapid test devices and newly developed assays of standard format like enzyme-linked immunosorbent assays (ELISA) and chemiluminescence immunoassays (CLIA) (10) (11) (12) (13) (14) .

Our study aimed to compare the diagnostic performance of eight different commercial serological assays for the detection of IgA, IgM, and IgG antibodies to the SARS CoV-2 virus with three assays compared for the first time to other assays included in the study.

The (16) . As ELISAs and chemiluminescence test results could be reported as a numerical value, we performed receiver operating characteristic (ROC) curve analysis to investigate whether acceptance of the cut-offs obtained by the ROC curve analyses could improve the diagnostic performance of the assays in comparison to manufacturer provided cut-off (17) . Statistical analysis was performed on the whole group of RT-PCR positive individuals and those with > 10 days of symptoms duration but not on those with < 10 days of symptoms duration due to the small number of cases (N = 18). P value less than 0.05 was considered statistically significant. Statistical analysis was performed using the Medcalc Statistical Software version 19.1.5 (MedCalc Software Ltd, Ostend, Belgium).

Overall sensitivity for IgM ranged from as low as 28% up to 80%, for IgG from 76% up to 91%, and for IgA 87%. When the duration of symptoms > 10 days was taken into consideration, improvement of sensitivity was obtained for all assays. Specificity for IgG ranged from 85% to 100% and was gen-erally higher in comparison to other isotypes from the same manufacturer, while for IgM ranged from 90% to 100%, and for IgA, it was 90%. Sensitivities and specificities for each assay, including the combination of isotypes, are presented in Table 2 .

When the combined measurement of IgG, IgM, and IgA was taken into consideration in comparison to individual measurement, the improvement of sensitivities occurred for both ELISA assays but at the expense of specificity compared to the determination of IgG alone ( Table 2) . For CLIA assay, the diagnostic accuracy of combined measurement was equal to IgG alone, while for all LFIC assays did not improve the sensitivity and lowered the specificity in comparison to IgG alone. Agreement in positive/negative result categorization between assays is generally poorer for IgM (Table 3) than for IgG ( tients with symptoms duration > 10 days were considered (Table 5 ). This especially refers to IgM, yet still significantly lower than IgA (P = 0.029).

To 

We aimed to compare diagnostic efficacy for detection of SARS-CoV-2 antibodies of eight commercially available serological assays of different formats. Our results confirmed the general poor utility of serological tests if performed in a period less than 10 days from the onset of the symptoms. The high variability between results of IgM assays was obtained, while the results for IgG showed good agreement and high diagnostic accuracy, es- pecially after 10 days of symptoms onset. Low sensitivity of IgM isotype for one of the tested rapid devices pointed out the necessity for verification of the test before the use in patient care.

Implementation of SARS-CoV-2 IgM and/or IgA detection as a marker of an early immune response has been suggested as an additional diagnostic tool to RT-PCR test in patients with symptoms highly suggestive of COVID-19 infection but with negative RT-PCR test (18, 19) . To warrant this role, such a marker has to have a high sensitivity in the early disease phase in order not to miss potentially infective individuals. On our RT-PCR-positive group, IgM showed generally lower sensitivity in comparison to IgG of the same manufacturer, irre-spective of symptoms duration. The exception is ENCODE COVID-19 Rapid test device for which IgM had the same sensitivity as IgG, and it was higher in comparison to other IgM assays. For this particular assay, sensitivities for both IgM and IgG obtained in our study group were very similar to those reported from the recent study (20) . Similarly, sensitivities for IgA and IgG Euroimmun ELISA and IgM and IgG Maglumi CLIA in patients with > 10 days of symptoms duration are close to the sensitivities reported by other authors (6, 10, 11) . In comparison to other assays, sensitivity for IgM Biozek COVID-19 LFIC assay was very low irrespective of symptoms duration. Similarly, Rudolf et al. also obtained a very low sensitivity of 19% (95%CI [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] [28] for Biozek IgM for the post-symptom period of 7-28 days (21) . The sensitivity of a serological test is influenced by several factors including antigen origin, antigen coating density, and serum dilution as well as observer visual error effect in the case of LFIC methods (22) . The manufacturers did not specify antigen origin for either of the tested LFIC assays but sensitivity for IgG Biozek COVID-19 LFIC assay was comparable to all other tests so it is unlikely that the antigen origin could be the reason for exceptionally low sensitivity of IgM isotype. Also, the observer error effect was set to a minimum. Therefore, we speculate that employed serum dilution with sample buffer in relation to antigen coating density in Biozek COVID-19 LFIC assay may not be optimal for the detection of antibodies of IgM isotype.

Within our RT-PCR-positive group, there was no case with isolated IgM positivity detected with any of the tested assays. This observation, although noticed on the small study population, together with lower specificity of IgM obtained in comparison to IgG in LFIC assays, which is in line with the recent report highlighted the limited utility of IgM isotype detection in the acute SARS-CoV-2 infection (23) . On the contrary, IgA showed better sensitivity in comparison to IgG when the whole RT-PCR-positive group was taken into consideration owing to the higher rate of positivity in the subgroup with < 10 days of symptoms duration. Within this subgroup, isolated positivity of IgA was detected in 6/18 (0.40) patients with the duration of the symptoms ranging from 4 to 9 days and in 2 asymptomatic patients tested 1 day after the RT-PCR-test. Additionally, the sensitivity of IgA rose to 96% after 10 days of symptoms, which confirmed the result from the recent study according to which IgA antibody can be reliably detected one week after the RT-PCR-confirmed SARS-CoV-2 infection (24) . Higher diagnostic accuracy of IgA in comparison to IgM in our study was further confirmed with the lack of added value of combined detection of IgA and IgM in comparison to IgA alone in spite the fact that the assay which detects a combination of the isotypes employs whole inactivated antigen while the other assay employs S1 only. The use of serological markers in the acute phase of infection is overweighted with the high variability of the time of seroconversion as well as the magnitude of the immune response, which is associated with the disease severity (22) . The high rate of false-negative results in the phase of the disease when patients are most infectious limits the role of serological analysis in diagnosing the acute infection (25) . Testing for SARS-CoV-2 antibodies is more likely to be useful for determination of seroprevalence in population and identification of highly reactive potential human donors of convalescent plasma for therapeutic use (26, 27 (14) . In the same study, no crossreactivity was observed for Abbott and Roche assays which are in line with 100% specificity for assays determined in our study. Crossreactivity examined by the manufacturer of the Vircell ELISA assay was performed on a small number of samples and did not include other coronaviruses, and yet confirmed some crossreactivity for IgM + IgA but none for IgG assay.

Diagnostic accuracy of the ELISA and chemiluminescence IgG assays did not significantly differ, including the Elecsys Anti-SARS-CoV-2 total antibodies assay which indicates the low added value of combined measurement of Ig isotypes in comparison to IgG, only. However, the number of RT-PCRpositive cases with symptoms duration less than 10 days included in our study is too small to assert this firmly.

It is worth noting that among 10 of our asymptomatic individuals sampled 1 day after RT-PCR confirmation, none was negative with all tests, and surprisingly, most of them were positive for both IgA/M and IgG with more than at least two tests of different formats.

Obtained sensitivities and specificities for five out of eight accompassed assays are similar to those declared by the manufacturers, taking into account the slight differences in the time from the onset of symptoms used for testing. Significant discrepancies were obtained for IgM isotype in two LFIC assays (Biozek and Lumiquick) and Maglumi CLIA assay. However, the time from symptoms onset used by these manufacturers for evaluation of sensitivities and specificities is unknown and can have a huge impact on comparability with our results.

Comparison of IgG serological assays in our study confirmed a low level of variability between different assays (including rapid tests), despite the differences in employed antigenic targets (S1 only, N only, or combination of both). On the contrary, variability between results of the IgM assays was high with only one acceptable agreement (between two rapid tests) according to the rule that kappa coefficient < 0.60 indicates inadequate agreement between methods to be equally employed in clinical laboratory practice (16) . The low agreement between IgM assays could be attributed to the above-mentioned cross-reactivity of SARS-CoV-2 IgM with other viruses which is unknown data for the rapid tests included in the study. Also, the impact of the rheumatoid factor (RF) as the common cause of cross reactivity is mostly unknown for rapid tests.

Improvement of diagnostic performance with the acceptance of ROC analysis provided cut-offs for some assays pointed out the importance of the verification of manufacturer-provided cut-off values on the local population. The choice of the cutoff is highly dependent on the intended use of the assay with the imperative for a minimum of falsepositives for detection of the acquired immunity (27) . Also, the results of our study suggest that the choice of the cut-off might be related to the time from the symptoms onset or RT-PCR test for asymptomatic individuals.

Despite the prejudices of the unreliability of SARS-CoV-2 rapid serological tests, the results of our study showed that the reliability is manufacturerdependent. LFIC assay format is easy to handle but the important drawback of this technology is the high impact of the test performer on the result, including the manipulation with the sample and buffer and, most important, the interpretation of the faint bands. Although we tried to minimize the bias in interpretation, significant discrepancies between manufacturer declared sensitivities and those obtained in our study for two LFIC assays could be partly attributed to this problem.

Advantages of rapid assays are easy handling, no need for extra instrumentation, low volume of sample, the possibility of the whole blood application, and fast results. On the other hand, advantages of conventional assays are automation, avoidance of the prolonged exposure to biological material, available quality control that assures batch-to-batch reproducibility and results that are not prone to subjective interpretation. Numeric results could provide added value in the assessment of the intensity of the immunologic response and also allow the adjustment of the cut-off according to ROC analysis performed on the representative population.

The strengths of this study are related to the experimental design and effort to compare eight differently designed assays (rapid and conventional, manually and fully automatic). The majority of studies presented sensitivities and specificities of different SARS-CoV-2 serological assays while our study additionally evaluates the agreement of the results between tested assays.

Although this is not the first study that compares several different serological SARS-CoV-2 assay methods, to the best of our knowledge, Lumiquick IgG/IgM and Biozek IgG/IgM as rapid assays and Vircell IgM+IgA/IgG as an ELISA method were not previously compared to other assays included in this study. The low number of participants, especially those with a duration of the symptoms less than 10 days is the limitation of the study. Additionally, to determine the specificity of the SARS-CoV-2 serological assay, the use of prepandemic samples would be the most correct approach.

In conclusion, our study confirmed that the selection of the appropriate time-frame for testing is crucial for the proper investigation of immunity, which can be very challenging among asymptomatic persons. There is high variability between IgM SARS CoV-2 serological assays independently of the assay format. On the contrary, IgG assays showed moderate to perfect agreement. We ob-served a higher diagnostic accuracy of IgA in comparison to IgM SARS CoV-2 serological assays.

Scientific information on COVID-19

A pneumonia outbreak associated with a new coronavirus of probable bat origin

The species Severe acute respiratory syndrome-related coronavirus: classifying 2019-nCoV and naming it SARS-CoV-2

Serological Assays for SARS-CoV-2 Infectious Disease: Benefits, Limitations and Perspectives

Evolving status of the 2019 novel coronavirus infection: Proposal of conventional serologic assays for disease diagnosis and infection monitoring

Analytical performances of a chemiluminescence immunoassay for SARS-CoV-2 IgM/IgG and antibody kinetics

European Centre for Disease Prevention and Control. Diagnostic testing and screening for SARS-CoV-2

A serological assay to detect SARS-CoV-2 seroconversion in humans

Evidence summary of the immune response following infection with SARS-CoV-2 (v2.0)

Assessment of immune response to SARS-CoV-2 with fully automated MAGLUMI 2019-nCoV IgG and IgM chemiluminescence immunoassays

Validation of a chemiluminescent assay for specific SARS-CoV-2 antibody

Comparison of the Elecsys® Anti-SARS-CoV-2 immunoassay with the EDI™ enzyme linked immunosorbent assays for the detection of SARS-CoV-2 antibodies in human plasma

Testing for SARS-CoV-2 (COVID-19): a systematic review and clinical guide to molecular and serological in-vitro diagnostic assays

Evaluation of Six Commercial Mid-to High-Volume Antibody and Six Point-of-Care Lateral Flow Assays for Detection of SARS-CoV-2 Antibodies

Comparison of eight SARS-CoV-2 serological assays

Detection of 2019 novel coronavirus (2019-nCoV) by real-time RT-PCR

Interrater reliability: the Kappa Statistic

Antibody Detection and Dynamic Characteristics in Patients With Coronavirus Disease

Clinical laboratory and SARS-CoV-2 infection: where do we stand?

Evaluation of eleven rapid tests for detection of antibodies against SARS-CoV-2

Clinical Characterisation of Eleven Lateral Flow Assays for Detection of COVID-19 Antibodies in a Population

Role of Serology in the Coronavirus Disease

Prevalence of SARS-CoV-2 in Spain (ENE-COVID): a nationwide, population-based seroepidemiological study

The role of serology for COVID-19 control: Population, kinetics and test performance do matter

Canadian Public Health Laboratory Network Statement on Point-of-Care Serology Testing in COVID-19

Antibody tests for identification of current and past infection with SARS-CoV-2. Cochrane

Antibody tests for CO-VID-19: drawing attention to the importance of analytical specificity

The authors thank Ph.D. Lovorka Đerek (the Head of the Clinical Department of Laboratory Diagnostics at University Hospital Dubrava, Zagreb, Croatia) who was a coordinator in samples distribution. The manufacturer's representatives provided inkind support assuring test kits for the evaluation but had no role in directing the study or influencing the study outcome.

None declared.