key: cord-0898985-ny2v1mco authors: Mylemans, Marnix; Van Honacker, Eveline; Nevejan, Louis; van den Bremt, Stefanie; Hofman, Laura; Poels, Jeroen; Cattoir, Lien; Boel, An; Van Hoovels, Lieve title: Diagnostic and analytical performance evaluation of ten commercial assays for detecting SARS-CoV-2 humoral immune response date: 2021-03-24 journal: J Immunol Methods DOI: 10.1016/j.jim.2021.113043 sha: 6f9da72ef90b280793fdf89d23ac615a77f97a5b doc_id: 898985 cord_uid: ny2v1mco OBJECTIVE: Analytical validation of newly released SARS-CoV-2 antibody assays in the clinical laboratory is crucial to ensure sufficient performance in respect to its intended use. We aimed to assess analytical and diagnostic performance of 8 (semi-)quantitative assays detecting anti-nucleocapsid IgG (Euroimmun, Id-Vet) or total Ig (Roche), anti-spike protein IgG (Euroimmun, Theradiag, DiaSorin, Thermo Fisher) or both (Theradiag) and 2 rapid lateral flow assays (LFA) (AAZ-LMB and Theradiag). METHODS: Specificity was evaluated using a cross-reactivity panel of 85 pre-pandemic serum samples. Sensitivity was determined at both the manufacturer's and a 95% specificity cut-off level, using 81 serum samples of patients with a positive rRT-PCR. Sensitivity was determined in function of time post symptoms onset. RESULTS: Specificity for all assays ranged from 92.9% to 100% (Roche and Thermo Fisher) with the exception of the Theradiag IgM LFA (82.4%). Sensitivity in asymptomatic patients ranged between 41.7% and 58.3%. Sensitivity on samples taken <10 days since symptom onset was low (23.3%–66.7%) and increased on samples taken between 10 and 20 days and > 20 days since symptom onset (80%–96% and 92.9%–100%, respectively). From 20 days after symptom onset, the Roche, Id-vet and Thermo Fisher assays all met the sensitivity (>95%) and specificity (>97%) targets determined by the WHO. Antibody signal response was significantly higher in the critically ill patient group. CONCLUSION: Antibody detection can complement rRT-PCR for the diagnosis of COVID-19, especially in the later stage, or in asymptomatic patients for epidemiological purposes. Addition of IgM in LFAs did not improve sensitivity. In December 2019 several cases of pneumonia of unknown cause occurred in Wuhan, Hubei Province, China. On January 7 2020, a novel betacoronavirus, severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) was isolated from the patients in Wuhan (1) . This virus is responsible for a viral pneumonia called coronavirus disease 2019 (COVID-19) (2, 3) . Although most people with COVID-19 disease have mild to moderate symptoms, the disease can cause severe medical complications such as acute respiratory distress syndrome, septic shock, bleeding and coagulation disorders, and can lead to death in pre-disposed people (4) . Due to a combination of high human-to-human transmissibility, absence of natural immunity in the population and a lot of international traffic, the virus has quickly spread around the world and evolved to a global pandemic (5, 6) . At the time of writing, the virus is globally disrupting society. Therefore, a rapid and correct identification of the virus is crucial, not only for the diagnosis of COVID-19 disease and subsequent correct treatment, but also to take necessary isolation precautions and thereby avoid further spreading. Also, vaccination will probably soon be a possible solution for reducing the spread of the virus by evoking humoral immunity in the vaccinated people. The current gold standard for the diagnosis of COVID-19 is the detection of viral RNA in respiratory tract samples with real-time reverse transcriptase-polymerase chain reaction (rRT-PCR) targeting SARS-Cov-2 specific sequences coding for spike (S), envelope (E), or nucleocapsid proteins (7) (8) (9) (10) (11) . rRT-PCR is highly sensitive and specific, especially in the acute phase of the infection (12,13). The sensitivity of the PCR test depends on the time of sample collection in relation to the diagnostic testing window. Negative PCR results may be caused by extremely low viral load when tested shortly after exposure or at late stages of infection. Maximum viral load in throat swabs was observed 2 days before until 5 days after symptom onset (14, 15) . The median [interquartile range] period between symptom onset and a negative rRT-PCR result has been reported to be 20 [17] [18] [19] [20] [21] [22] [23] [24] days (16) . Furthermore, higher viral load and a longer mean duration of viral detection in respiratory samples correlate with disease severity (17) . rRT-PCR can be false negative due to pre-analytical issues such as the sample collection technique. As to be expected, bronchoalveolar lavage (BAL) and sputum samples have shown to contain a higher viral load, and thus to remain positive for a longer time, compared to nasopharyngeal, nose or throat samples (8) . However, the Centers for Disease Control and Prevention (CDC) recommends using upper respiratory specimens for initial diagnostic testing, for logistical purposes and to limit invasive sampling procedures (18) . The nasopharyngeal swab (NP) is currently proposed as the gold standard sample for detection of SARS-CoV-J o u r n a l P r e -p r o o f Journal Pre-proof 2 due to the higher sensitivity for the detection of SARS-CoV-2 compared to oropharyngeal swabs and saliva samples (8) , restricting the latter sample types to specific screening strategies (19) . Serological assays have the potential to play a complementary role in the diagnosis of rRT-PCR-negative COVID-19 cases (20) . Seroconversion for SARS-CoV-2 is typically detected between 7-14 days post symptom onset (21) (22) (23) . Among the four SARS-CoV-2 structural proteins, the spike (S) and nucleocapsid (N) proteins are the most immunogenic (24, 25) . Different types of tests are available to detect anti-SARS-CoV-2 antibodies: rapid lateral flow assays (LFA) as point of care tests, enzyme-linked immunosorbent assays (ELISA) and automated immunoassays. The contribution of serological assays to seroprevalence studies and evaluation of the results of vaccine trials is currently under debate (26). The aim of this study is to compare the diagnostic performance of ten commercial SARS-CoV-2 antibody test assays: five ELISA"s, one fluoro-enzyme-immunoassay (FEIA), two rapid LFA"s, and two chemiluminescence immunoassays (CLIA) (Supplementary Material 1). The study was performed in cooperation with the Belgian Federal Agency for Medicines and Health Products (FAMHP) that had set up a validation scheme for serological SARS-CoV-2 assays, whereby positively evaluated laboratory assays are reimbursed by the national health insurance. This retrospective study was performed using 166 patient samples collected at the OLV Hospital Aalst, Table 3 . Sensitivity was assessed on a selection of 81 serum samples from 77 patients with a rRT-PCR confirmed SARS-CoV-2 infection on nasopharyngeal swab. rRT-PCR was performed using an in-house method complying with the WHO guidelines (7) . The time between symptom onset and sampling date was a) less than ten days (n=30) b) between 10-20 days (n=25) and c) more than 20 days (n=14). Also 12 samples of J o u r n a l P r e -p r o o f Journal Pre-proof asymptomatic patients were included. The median time between symptom onset and serum sampling was 11 days (range 1-51). The group consisted of 53 male and 28 female patients with a median age of 66 years (range 17-97). Of note, in case of multiple samples per patient, only the first sample per timecategory was used to assess sensitivity. All samples were stored at -20 °C until analysis. The protocol was approved by the local Ethics Committee OLV Hospital Aalst with Belgian registration number B126202000015. For all COVID-19 patients, disease severity status was collected. Patients were classified as a) "mild" if no hospital admission was required b) "moderate" in case of admission to a non-ICU ward, c) "critical" in case of admission to the ICU-ward or death and d) "asymptomatic". Serum samples of immunosuppressed patients (hematological malignancies, solid organ transplant) and patients younger than one year were excluded from the data set (specificity and sensitivity). Four new ELISA"s, one FEIA and two new rapid LFA"s were evaluated and compared to one established ELISA and two established CLIA"s. Analytical performance of each assay was assessed by calculating imprecision (coefficient of variation (CV), %) using the manufacturer"s internal quality control materials (iQC) and three patient serum samples with a low, medium and high SARS-CoV-2 Ab concentration. All iQC samples were measured before and after every run during 10 runs (CLSI EP5-A2) (27). Linearity was assessed by diluting a high level serum SARS-CoV-2 Ab sample with increasing amounts of a serum sample with very low levels of SARS-CoV-2 Ab (CLSI EP06-A) (28) . An overview of the demographic features of the different patient cohorts is shown in Supplementary Material 2a-d. In general, we retained no significant difference in gender distribution between the sensitivity and specificity patient cohorts (p=0.1174), but regarding age, the sensitivity patient group was significantly older (p<0.0001). J o u r n a l P r e -p r o o f Imprecision Results of the imprecision study are presented in Supplementary Material 3. For the ELISA"s of Euroimmun (EI-S & EI-N), the imprecision obtained for the patient sample iQC was higher than for the kit iQC which can be explained by the fact that the kit iQC"s are prediluted and their imprecision results didn"t include a predilution step. The latter is not true for the ELISA"s of Theradiag (TD-S & TD-SN) and Id-vet (Id-N), with comparable imprecision results for the kit and patient sample iQC. Assays based on ELISA format obtained the highest CV% results. No deviation from linearity was revealed for any of the assays, which is illustrated in Supplementary Material 4a-h. The lower results for TD-S are related to imprecision rather than to non-linearity. Specificity cohort (Table 1 , Table 2 ). The lowest specificity was obtained for the LFA TD-S IgM assay (82% [72.6-89.8]), with the highest cross-reactivity in the cohort of anti-nuclear antibody associated disease (AARD) and non-SARS- ). An overview of the aspecific reactivities is given for every SARS-CoV-2 antibody assay in Table 3 . The evaluated SARS-CoV-2 antibody assays showed a significant difference in diagnostic performance with the 81 serum samples selected from patients with a rRT-PCR confirmed SARS-CoV-2 infection. Data of all (semi-)quantitative assays are shown in Table 1 and corresponding receiver operating characteristic (ROC) curves in Figure 1 . The observed differences are mainly related to the diagnostic performance in the early phase of antibody detection. The diagnostic performance of the antibody tests was directly proportional to the time period after onset of symptoms: the longer this time period, the higher the diagnostic performance of all antibody tests and consequently, the lower the difference in diagnostic performance between tests. In addition, assays using a recombinant N-antigen revealed J o u r n a l P r e -p r o o f generally higher sensitivities compared to those targeting the S-antigen, although specificity results were generally comparable. Based on these results, the R-N assay showed the best diagnostic performance characteristics. [ Table 1 and Fig. 1 near here] Equally, all data on diagnostic performance characteristics of the rapid LFA are shown in Table 2 [ Table 2 near here] For all assays, Box and Whisker analysis revealed significantly higher antibody results in the "critically ill" patient cohort (n= 33) compared to the "moderately ill" cohort (n=33) (Supplementary Material 5; for all assays p<0.05). However, the proportion of samples collected ≥20 days after symptom onset was significantly higher in the "critically ill" patient cohort (36% versus 3%; p=0.0008), which could attribute to the higher antibody levels. No significant difference was observed when comparing antibody results between patient cohort "asymptomatic" (n = 12) and cohort "mildly ill" (n = 3) or "mildly ill" and cohort "moderately ill" patients for all assays (p>0.05). [ Table 3 near here] Since the start of the COVID-19 pandemic, an increasing number of serological SARS-CoV-2 assays have been introduced to the diagnostic market (29) . The expertise of laboratory professionals is critical in the validation of these diagnostic assays to ensure sufficient analytical performance in respect to the intended use (30) . This study evaluated the diagnostic performance of eight (semi-)quantitative (IgG/total Ig) and two rapid LFA (IgM and IgG) serological assays for the detection of SARS-CoV-2 (N/S protein). Taking into account the manufacturer"s threshold, the overall sensitivity in our cohort (n=81) ranged from 56.8% Antibodies against N protein are reported to appear earlier in infection than those against S protein (32) . Within the subgroup of "patients < 10 days of symptoms" and in the asymptomatic patient cohort we also revealed higher sensitivities for the N-based assays (range of respectively 46.7-50.0%, 50.0-58.3%) versus S-based assays (range of respectively 23.3-36.7%, 41.7-50%), with significantly different areas under the diagnostic (AUC) receiver operating curve (ROC) between some of the N and S-protein based assays. However, the Ag-source clearly appeared not to be the only factor attributing to diagnostic sensitivity (Table 1 ). Our data are in concordance with other head to head SARS-CoV-2 antibody comparison studies (Table 1 ). At this time-point, there are no significant differences in area under the diagnostic curve (AUC) between the serological tests (Fig. 1) . The sensitivity in the asymptomatic cohort was significantly lower than the overall sensitivity. Importantly, 9 of the 12 samples were taken < 10 days after positive rRT-PCR and 4 of those 9 serum samples tested negative in all antibody assays. Most likely, the lower sensitivity can be attributed to early infection or to a difference in Ab kinetics as described earlier in this patient category (40) . In the study of Jiang and colleagues, IgG/IgM titers and plasma neutralisation capacity were, at the time of virus clearance, significantly lower in recovered asymptomatic than in recovered symptomatic patients. Reinforced by the fact that a major part of asymptomatic and pauci-symptomatic patients is not even Regarding specificity, N protein-based serological assays were more often associated with cross-reactivity than the S-based assays (12). For the latter, recombinant and more standardized S1-based assays have shown to be more specific compared to assays to full viral antigens (42, 43) (Supplementary Material 1) . Overall specificity in the samples collected prior to the pandemic (n = 85) ranged from 82.4% (LFA TD-S) to 100% (R-N & TF-S) ( Table 1) . Cross-reactivity is mainly attributed to antigens well-conserved among different coronaviruses and to cross-reaction with antibodies of autoimmune diseases (44) . When using antibody assays on a population level, a high specificity is of utmost importance, as every small drop in specificity will seriously reduce the positive predictive value (45) . In the future, if antibodies prove to be protective, false positive results can potentially also have an important impact on the individual patient level if these results are used to decide whether or not to administer (re)vaccination or to use personal protective equipment. In accordance with preceding studies (23,26), we found that all (semi-)quantitative assays result in significantly (p<0.05) higher antibody levels in the "critically ill" patient cohort compared to the "moderately ill" cohort. However, the proportion of samples collected ≥20 days after symptom onset was also significantly higher in the "critically ill" patient cohort (36% versus 3%; p=0.0008), which could attribute to the higher antibody levels. Nevertheless, our observations are completely in line with earlier findings that antibody levels are associated with disease severity (46) . A strength of our study is the parallel evaluation of the diagnostic performance of several new serologic SARS-CoV-2 assays and assays with established diagnostic performance. Furthermore, we"ve performed a separate diagnostic performance analysis in asymptomatic people. In this subgroup, overall sensitivity revealed to be lower than the overall sensitivity obtained for the several assays, as mentioned above. This J o u r n a l P r e -p r o o f is not surprising taking into account the earlier mentioned difference in Ab kinetics in the asymptomatic population. A limitation of our study is that the samples used to evaluate specificity were all challenging. We thus expect a higher specificity in a routine laboratory setting. Another limitation is the limited sample size, which results in a small number of cases in the subgroup analyses concerning timing post symptom onset and severity of symptoms. Finally, the categorization of the patient cohorts "mild", "moderate", "critical" was only based on whether or not the patient was admitted to the hospital/intensive care unit. Information on duration and severity of symptoms of individual cases is lacking, due to the retrospective design of this study. We can conclude that, in this study, the R-N serological assay revealed the best overall performance. However, for the intended use of antibody detection (>20 days after symptom onset), the R-N, Id-N and TF-S assays all met the sensitivity (95-98%) and specificity (97-99%) targets determined by the WHO (47). No funding was received for conducting this study. AB and LVH have been consultants for Thermo Fisher Scientific. Data will be available from the author upon request. The protocol was approved by the local Ethics Committee OLV Hospital Aalst with Belgian registration number B126202000015. The procedures used in this study adhere to the tenets of the Declaration of Helsinki. Mycoplasma pneumoniae Systemic rheumatic disease (n = 10) mixed connective tissue disease 2 ------------1 1 rheumatoid arthritis 3 --1 --------1 2 2 systemic lupus erythematosus 3 -----------1 2 2 sjögren syndrome 2 -------------- Other pathogens (n = 21) Borrelia burgdorferi (IgG) 1 -1 -----------cytomegalovirus 3 --------1 -1 --epstein-barr virus 3 -------------hepatitis A virus 1 -----------1 1 1 hepatitis B virus (HBsAg) Concerning the "all sensitivity cohort" (n = 81), R-N showed significantly higher AUC than any other assay (p< 0.05); DS-S showed significantly lower AUC than any other assay (all p< 0.05) Id-N showed significantly higher AUC than TF-S (p=0.0465). B. In the sensitivity cohort "asymptomatic" (n = 12), DS-S showed significantly lower AUC than Id-N (p=0.0357), TD-S (p=0.0383), R-N (p=0.0056), TF-S (p=0.0062) and TD-SN (p=0.0475); R-N showed significantly higher AUC than EI S (p=0.0101) and TF-S (p=0.0286) and Id-N showed significantly higher AUC than TF-S (p=0.0465). C. In sensitivity cohort "<10 days after symptom onset", DS-S showed significantly lower AUC than any other assay (all p< 0.05); additionally, R-N showed significantly higher AUC than EI S (p=0.0147), EI N (p=0.0174), TF-S (p=0.0161) and TD-SN (p=0.0067). D. In sensitivity cohort "10-20 days after symptom onset", DS-S showed significantly lower AUC than Id-N (p=0.0195) and R-N (p=0.0197). E. Concerning sensitivity cohort "≥20 days after symptom onset" no significantly differences in AUC were revealed (all p> 0.05). J o u r n a l P r e -p r o o f Figure 1 A novel coronavirus outbreak of global health concern A novel coronavirus from patients with pneumonia in China Genomic characterisation and epidemiology of 2019 novel coronavirus: implications for virus origins and receptor binding Epidemiological and clinical characteristics of 99 cases of 2019 novel coronavirus pneumonia in Wuhan, China: a descriptive study Timeline of WHO"s response to COVID-19 The emergence of SARS-CoV-2 in Europe and North America Detection of 2019 novel coronavirus (2019-nCoV) by real-time RT-PCR Detection of SARS-CoV-2 in Different Types of Clinical Specimens Molecular Diagnosis of a Novel Coronavirus (2019-nCoV) Causing an Outbreak of Pneumonia World Health Organization. Laboratory testing of 2019 novel coronavirus ( 2019-nCoV) in suspected human cases: interim guidance Molecular, serological, and biochemical diagnosis and monitoring of COVID-19: IFCC taskforce evaluation of the latest evidence Serological Assays for SARS-CoV-2 Infectious Disease: Benefits, Limitations and Perspectives Laboratory Diagnosis of COVID-19: Current Issues and Challenges Virological assessment of hospitalized patients with COVID-2019 Potential sources, modes of transmission and effectiveness of prevention measures against SARS-CoV-2 Dynamic profile of RT-PCR findings from 301 COVID-19 patients in Wuhan, China: A descriptive study Lower nasopharyngeal viral load during the latest phase of COVID-19 pandemic in a Northern Italy University Hospital Interim guidelines for collecting, handling, and testing clinical specimens from persons for coronavirus disease 2019 (COVID-19) Saliva as a Noninvasive Specimen for Detection of SARS-CoV-2. McAdam AJ, editor Antibody Detection and Dynamic Characteristics in Patients With Coronavirus Disease Profiling Early Humoral Response to Diagnose Novel Coronavirus Disease (COVID-19) Sensitivity in Detection of Antibodies to Nucleocapsid and Spike Proteins of Severe Acute Respiratory Syndrome Coronavirus 2 in Patients With Coronavirus Disease Antibody responses to SARS-CoV-2 in patients with COVID-19 Serological assays for emerging coronaviruses: challenges and pitfalls Antibody responses to individual proteins of SARS coronavirus and their neutralization activities. Microbes Infect Evaluation of the Linearity of Quantitative Measurement Procedures: A Statistical Approach; Approved Guideline Antibody tests for identification of current and past infection with SARS-CoV-2. Cochrane database Syst Rev Lessons from the Belgian experience with regulatory control during the COVID-19 pandemic for the implementation of the European IVD regulation 2017/746 Antibody Responses to SARS-CoV-2 in Patients With Novel Coronavirus Disease SARS-CoV-2 serological analysis of COVID-19 hospitalized patients, pauci-symptomatic individuals and blood donors. medRxiv Evaluation of nine commercial SARS-CoV-2 immunoassays. medRxiv National SARS-CoV-2 Serology Assay Evaluation Group. Performance characteristics of five immunoassays for SARS-CoV-2: a head-to-head benchmark comparison SARS-CoV-2 infection serology validation of different methods: Usefulness of IgA in the early phase of infection Performance evaluation of two SARS-CoV-2 IgG/IgM rapid tests (Covid-Presto and NG-Test) and one IgG automated immunoassay (Abbott) Humoral Immune Response to SARS-CoV-2 Side-by-Side Comparison of Three Fully Automated SARS-CoV-2 Antibody Assays with a Focus on Specificity Antibody seroconversion in asymptomatic and symptomatic patients infected with severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) Clinical and immunological assessment of asymptomatic SARS-CoV-2 infections SARS-CoV-2 specific antibody responses in COVID-19 patients. medRxiv A serological assay to detect SARS-CoV-2 seroconversion in humans Cross-reaction of SARS-CoV antigen with autoantibodies in autoimmune diseases SARS-CoV-2 antibody performances: we need better criteria Humoral Immune Response to SARS-CoV-2 in Iceland World Health Organization. COVID-19 Target product profiles for priority diagnostics to support response to the COVID-19 Sensitivity is calculated at both the manufacturer"s cut-off (SN) and at the cut-off corresponding to a specificity level of 95% Abbreviations: anti-N, anti-nucleocapsid protein; anti-S, anti-spike protein AUC, area under the ROC curve, CI, confidence interval NS, not stated; ROC, receiver operating curve analysis Abbreviations: CI, confidence interval 19 IgG+IgM Thera We thank Euroimmun, Id-vet, AAZ-LMB, Theradiag and Thermo Fisher Scientific for the donation of the assays. We are very grateful to the laboratory technicians for their most appreciated efforts.