key: cord-0916980-w0k9cqzz
authors: Yassine, Hadi M.; Al-Jighefee, Hadeel; Al-Sadeq, Duaa W.; Dargham, Soha R.; Younes, Salma N.; Shurrab, Farah; Marei, Reham M.; Hssain, Ali Ait; Taleb, Sara; Alhussain, Hashim A.; Al-Nesf, Maryiam A.; Al-Khal, Abdullatif; Qotba, Hamda; Althani, Asmaa A.; Tang, Patrick; Abu-Raddad, Laith J.; Nasrallah, Gheyath K.
title: Performance evaluation of five ELISA kits for detecting anti-SARS-COV-2 IgG antibodies
date: 2020-10-27
journal: Int J Infect Dis
DOI: 10.1016/j.ijid.2020.10.042
sha: 42cb691ac367769425215eb26a35358754f1c18f
doc_id: 916980
cord_uid: w0k9cqzz

OBJECTIVES: To evaluate and compare the performances of five commercial ELISA assays (EDI, AnshLabs, Dia.Pro, NovaTec, and Lionex) for detecting anti-SARS-CoV-2 IgG. METHODS: 70 negative control samples (collected before the COVID-19 pandemic) and samples from 101 RT-PCR-confirmed SARS-CoV-2 patients (collected at different time points from symptoms onset: ≤7, 8-14, and >14 days) were used to compare the sensitivity, specificity, agreement, positive and negative predictive values of each assay with RT-PCR. A concordance assessment between the five assays was also conducted. Cross-reactivity with other HCoV, non-HCoV respiratory viruses, non-respiratory viruses, and nuclear antigens was investigated. RESULTS: Lionex showed the highest specificity (98.6%, 95%CI: 92.3-99.8), followed by EDI and Dia.Pro (97.1%, 95%CI: 90.2-99.2), NovaTec (85.7%, 95%CI: 75.7-92.1), then AnshLabs (75.7%, 95%CI: 64.5-84.2). All ELISA kits cross-reacted with one anti-MERS IgG positive sample except Lionex. The sensitivity was low during the early stages of the disease but improved over time. After 14 days from symptoms onset, Lionex and NovaTec showed the highest sensitivity at 87.9% (95%CI: 72.7-95.2) and 86.4% (95%CI: 78.5-91.7), respectively. The agreement with RT-PCR results based on Cohen’s kappa was as follows: Lionex (0.89)> NovaTec (0.70)> Dia.Pro (0.69)> AnshLabs (0.63)> EDI (0.55). CONCLUSION: The Lionex ELISA, which measures antibodies solely to the S1 protein, demonstrated the best performance.

public health concern all over the world [1, 2] . As of July 1, 2020, the virus has caused more than 10 million confirmed infections and over 500,000 reported deaths [3] . Currently, real-time reversetranscription polymerase chain reaction (RT-PCR) testing is the main technique used for the diagnosis of SARS-CoV-2 infection. However, false-negative RT-PCR results could occur in up to 30% of COVID-19 patients [4] [5] [6] . The reasons for this include poor sample collection techniques, sample collection too late after infection, or disease progression into the lower respiratory tract. Once an individual has been infected for at least seven days, the detection of antibodies is possibly more sensitive than RT-PCR for the diagnosis of COVID-19 [7] . Specific antibodies against SARS-CoV-2 can be detected as early as 4-7 days in approximately 40% of COVID-19 patients, with seroconversion rates reaching more than 90% by day 14 [7] . Therefore, serology could be used as a complementary test to RT-PCR to improve the diagnostic sensitivity, particularly in suspected COVID-19 individuals with negative RT-PCR results or those with no respiratory sample collected during the acute phase of illness [1, 2] .

Serology testing also has other advantages: it is easy to perform and interpret results; it is cheaper and quicker than RT-PCR; it indicates the patient immune status and infection stage (serosurvey studies); it facilitates the selection of the best candidate donors (with the highest antibody titers) for plasma exchange, and it aids in assessing the efficacy of vaccines that are in development. Due to urgency and demand in the current crisis, a large number of commercial serological tests have been developed and introduced into the global market, but often with insufficient validation on clinical samples. Hence, there is a pressing need for identifying reliable immunoassays with high sensitivity and specificity for serology testing and surveillance of SARS-CoV-2 infection as there remain persistent concerns regarding the accuracy and reliability of the available SARS-CoV-2 immunoassays.

In order to address this challenge, the present study evaluated the performance of five commercial CE-marked ELISA kits for detecting anti-SARS-CoV-2 IgG antibodies in samples from RT-PCR-confirmed COVID-19 patients. The sensitivity, positive predictive value, negative predictive value, positive percent agreement and Cohen's kappa were measured for each assay using samples collected from SARS-CoV-2 RT-PCR-positive patients at different times from symptoms onset (≤7, 8-14, >14 days). The specificity and cross-reactivity were evaluated using pre-pandemic serum samples collected from healthy blood donors. A concordance assessment was conducted to compare the agreement between the kits.

We evaluated the performance of five CE-marked ELISA assays (EDI, Dia.Pro, AnshLabs, NovaTec, and Lionex) for detecting anti-SARS-CoV-2 IgG antibodies. The performance was assessed using anonymous samples collected from RT-PCR-confirmed SARS-CoV-2 patients admitted to Hamad Medical Corporation (HMC), the main public healthcare provider and the nationally designated provider for COVID-19 healthcare needs, with different COVID-19 clinical outcomes. For the negative control group, pre-pandemic serum samples collected from blood donors before 2019, were selected. The IRB approvals for this study were obtained from HMC (HMC-IRB# MRC-01-20-145, HMC-IRB# MRC-05-003, and HMC-IRB# MRC-05-007) and

Qatar University (QU-IRB # QU-IRB 804-E/17).

A total of 101 case serum samples were selected from RT-PCR-confirmed COVID-19 patients, including ICU-admitted patients (n=35), hospitalized non-ICU patients (n=45), and J o u r n a l P r e -p r o o f convalescent samples collected from COVID-19-recovered patients by the Qatar Communicable Disease Center (CDC) at HMC (n=21). Clinical records of the patients were reviewed to determine the time from symptoms onset to collection and categorized into three groups: Group: 1-less than or equal to 7 days; Group 2: 8-14 days; Group 3: more than 14 days. RNA was extracted from nasopharyngeal swab specimens using Qiagen extraction kit. The extracted RNA was tested for SARS-CoV-2 using the SuperscriptIII OneStep RT-PCR kit as recommended by the manufacture instruction (Cat No. 12594100, ThermoFisher, USA). Each sample was tested using three sets of primers targeting the E gene for screening and confirmed with two different sets of primers targeting the RdRp gene as described in [8] . CT values below 32 were considered positive.

Characteristics of the 101 COVID-19 patients, including the demographic data and classification, are summarized in Table 1 For the control group, we utilized samples from healthy blood donors collected before 2019 and used in previous studies [9] [10] [11] [12] [13] [14] [15] [16] . The healthy blood donors had a median age of 36.0 (30.3-45.0) with 82.9% males and 8.6% females. Details about the collection, transport, and storage methods of the control samples were described elsewhere [9] [10] [11] [12] [13] [14] [15] [16] . The control group included samples that are seropositive for various viruses, including all other human coronaviruses (HCoV).

Further details about the control samples can be found in Table 1 and S2. Table S2 . All tests were carried out manually according to the manufacturers' instructions. A microplate reader, Epoch 2 microplate spectrophotometer (Bio-Tek, Italy) was used to read the optical density (OD) in all ELISA reactions. Borderline results were considered positive [1, 17] .

The sensitivity, specificity, positive predictive value, negative predictive value, positive percent agreement and Cohen's Kappa were calculated to assess the performance of each assay with the positive SARS-CoV-2 RT-PCR patients [18, 19] . Specificity and cross-reactivity of each assay were assessed using the pre-pandemic control samples. Data were summarized by number and percentage of positive results for each assay. Borderline results were considered positive.

Samples were categorized into three groups according to the time between collection and the onset of symptoms (≤7, 8-14, and >14 days), and all parameters were calculated for each group.

Concordance assessment between the ELISA kits was conducted to assess the agreement between the kits. These concordance measures include overall, positive, and negative percent agreement, as well as Cohen's kappa statistic. The latter measure is a standard and robust metric that estimates the level of agreement (beyond chance) between two diagnostic tests. Ranging between 0 and 1, a kappa value ≤0.40 denotes poor agreement, a value between 0.40 and 0.75 denotes fair/good J o u r n a l P r e -p r o o f agreement, and a value ≥0.75 denotes excellent agreement [20] . The significance level was indicated at 5%, and a 95% confidence interval (CI) was reported for each metric. All calculations were performed using Microsoft Excel 2016. Chi-square test was used to calculate the significance between the performances of ELISA kits. Significance (*) = p < 0.05; (**) = p < 0.01; (***) = p < 0.001. Further details about the statistical analysis and calculations can be found in Table S4 .

The diagnostic assessment of all ELISA kits according to each time-point after symptoms onset (≤7 days, 8-14 days, >14 days) is summarized in Table 2 and Figure 2 .

In the first week of symptoms onset (≤7 days), the sensitivity (95% CI) ranged from 57.1% 

All assays showed acceptable overall specificity ranging from 85. 7 (Table 3 and Figure 1 ). The specificity of each kit in relation to sample cross-reactivity with other viruses (Table 3 and S1) was also calculated. Only EDI and Lionex did not show cross-reactivity with non-HCoV respiratory viruses with 100% (15/15; 79.6-100) specificity for this subgroup. All assays showed some cross-reactivity with nonrespiratory viruses except Dia.Pro, which had 100% (33/33; 93.9-100) specificity. Finally, all assays showed no cross-reactivity with antinuclear antibodies samples except AnshLabs, which cross-reacted with one control sample [50.0% (1/2; 9.5-90.6)]. However, the sample size was very small, as only two specimens were positive for antinuclear antibodies were used. 

In this study, we evaluated the performances of five CE-marked ELISA kits using 101 (Table 2) . Also, the overall agreement and Cohen's kappa were calculated to compare the assays (Table 4 ).

Our results showed that most of the evaluated assays demonstrated a very good performance during the first week after symptoms onset compared to other studies [21] [22] [23] [24] . (Table S4, group 3 sample No. 4, 11, 15, and 26) . Surprisingly, one of the ICU-admitted patients did not show a detectable antibody response by all ELISA assays (Table S3 , group 3 sample No.33), which needs further investigation by other highly sensitive assays. Typically, if borderline results were obtained in ELISA testing, another sample is taken from the patient 1-2 weeks later for re-testing. However, this was not possible here as we were testing sensitivity and specificity in specific time frames. Considering that these borderline J o u r n a l P r e -p r o o f samples were collected from RT-PCR-positive patients, borderline results were considered positive, consistent with similar studies [1, 17] . Interestingly, only one pre-pandemic sample with positive anti-MERS-CoV IgG antibodies was found positive by four ELISA kits (Table 3 and S3), except Lionex, which demonstrated the highest specificity at 98.6% (Figure 1 ).

Another interesting finding is the heterogeneity of IgG antibody response in COVID-19

patients. Most developed serology assays target either the spike (S) or the nucleocapsid (N) protein of SARS-CoV-2. Previous studies performed on other HCoV suggested that the anti-nucleocapsid (anti-N) antibody response may appear earlier than the anti-spike (anti-S) response and may wane more rapidly [25, 26] . Here, we expect that the differences in sensitivity between ELISA kits depend on the targeted protein used in each assay. We noticed that there was a decline in the sensitivity of the ELISA kits targeting the N protein. However, the sensitivity increased in the kit that solely targets the S1 protein (Lionex). Therefore, a possible explanation for this is that the level of anti-N and anti-S antibodies may be similar during the acute phase of COVID-19 illness, but anti-N antibodies could be waning after the second week [25, 26] . Moreover, this could also explain the high specificity of Lionex compared to the other assays ( Figure 1 ). That is, Lionex targets the S1 protein, which is smaller and less conserved across different families of viruses than the N protein. Therefore, detection of anti-N antibodies may be useful in distinguishing more recent antibody response, while anti-S antibody may be used during the early and convalescent phases. Still, this does not explain why the sensitivity of NovaTec, which targets the N protein, remained steady after the second week compared to EDI which also targets the same protein (N).

Concordance assessment between the different assays showed good to excellent agreement between the kits. EDI and Dia.Pro had the best overall agreement (97.1%) and kappa index (0.94).

However, both assays demonstrated the lowest sensitivity in all time-points compared to the assays J o u r n a l P r e -p r o o f despite having a very high specificity (97.1%). Therefore, for diagnosis and clinical relevance, these two assays are the least recommended for such purposes. NovaTec and AnshLabs also showed an excellent positive percent agreement (91.2%) and a kappa index (0.82), where both assays had comparable overall sensitivity and specificity. Lionex, however, showed a variation in the agreement with the other ELISA kits, which could be due to the fact that Lionex is the only kit that targets S1 protein.

From an epidemiological perspective, the high sensitivity of the assay in combination with robust specificity is desirable. Here, Lionex and NovaTec ELISA kits showed the best overall performance in terms of specificity, sensitivity, agreement with RT-PCR, positive and negative predictive values compared to the other assays. The overall performance of both NovaTec and Lionex IgG manual ELISA was comparable to other detection methods, including automated tests, reported elsewhere [27, 28] . Both assays showed a diagnostic sensitivity of 87.9% after 14 days of symptoms onset compared to Abbott Architect (84.2%) and Cobas 6800 systems (95.2%). The specificity of Lionex (98.6%) was also comparable to the aforementioned automated assays (100% and 99.3%, respectively) [27, 28] .

A strength of this study is the use of a diverse control group to evaluate cross reactivity with antibodies against various viruses, including MERS, SARS-CoV, endemic coronaviruses, respiratory viruses, and other viruses. One of the limitations of this study is that the clinical details of the patients were not available which are important to understand why some of them did not develop an antibody response that is detectable by the evaluated kits. It would be very beneficial to perform a new study using a large sample size collected from patient with known disease severity outcomes. For instance, critical, severe, moderate, mild and asymptomatic to have a better implication about each assay performance and clinical practice relevance.

In conclusion, we believe that two ELISA kits (NovaLisa, and Lionex) showed promising overall results which could be used in the future for clinical testing. Further, all assays showed acceptable specificity ranging from 85.7-98.6%, except for the AnshLabs ELISA. Finally, although serological assays do not replace molecular tests in diagnosing active infection, they serve as an essential tool to accurately estimate the seroprevalence of SARS-CoV-2 in the general population and to quantify the level of herd immunity [29] . This could help ease the restrictions on human mobility and interactions without provoking a significant resurgence of transmission and mortality. However, it is still not clear whether positive results by serology reflect a protective immune response against infection [30] . Further studies are essential to distinguish functional antibodies from total binding antibodies using virus neutralization assays. 

All authors have no conflict of interest to declare J o u r n a l P r e -p r o o f

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. J o u r n a l P r e -p r o o f 

Diagnostic performance of seven rapid IgG/IgM antibody tests and the Euroimmun IgA/IgG ELISA in COVID-19 patients

Comparison of diagnostic accuracies of rapid serological tests and ELISA to molecular diagnostics in patients with suspected coronavirus disease 2019 presenting to the hospital

Estimating false-negative detection rate of SARS-CoV-2 by RT-PCR. medRxiv

False-negative results of real-time reverse-transcriptase polymerase chain reaction for severe acute respiratory syndrome coronavirus 2: Role of deep-learning-based ct diagnosis and insights from two cases. Korean journal of radiology

18 F-FDG PET/CT findings of COVID-19: a series of four highly suspected cases

Antibody responses to SARS-CoV-2 in patients of novel coronavirus disease 2019

Detection of 2019 novel coronavirus (2019-nCoV) by real-time RT-PCR. Euro surveillance : bulletin Europeen sur les maladies transmissibles = European communicable disease bulletin

Seroprevalence of hepatitis E virus among blood donors in Qatar

Prevalence and molecular profiling of Epstein Barr virus (EBV) among healthy blood donors from different nationalities in Qatar

Dengue and chikungunya seroprevalence among Qatari nationals and immigrants residing in Qatar

Prevalence of anelloviruses (TTV, TTMDV, and TTMV) in healthy blood donors and in patients infected with HBV or HCV in Qatar

Herpes Simplex Virus Type 2 Seroprevalence Among Different National Populations of Middle East and North African Men. Sexually transmitted diseases

Estimating seroprevalence of herpes simplex virus type 1 among different Middle East and North African male populations residing in Qatar

Measuring influenza hemagglutinin (HA) stem-specific antibody-dependent cellular cytotoxicity (ADCC) in human sera using novel stabilized stem nanoparticle probes. Vaccine

Comparative Serological Study for the Prevalence of Anti-MERS Coronavirus Antibodies in High-and Low-Risk Groups in Qatar

Validation of a commercially available SARS-CoV-2 serological Immunoassay. medRxiv

Evaluation of SARS-CoV-2 serology assays reveals a range of test performance

Journal of clinical virology : the official publication of the Pan American Society for Clinical Virology

The Measurement of Interrater Agreement, in Statistical Methods for Rates and Proportions

Evaluation of antibody testing for SARS-Cov-2 using ELISA and lateral flow immunoassays

Evaluation of the EDI enzyme linked immunosorbent assays for the detection of SARS-CoV-2 IgM and IgG antibodies in human plasma

Evaluation of ELISA tests for the qualitative determination of IgG, IgM and IgA to SARS-CoV-2. medRxiv

Evaluation of nine commercial SARS-CoV-2 immunoassays

Comparison of SARS-CoV-2 serological tests with different antigen targets. medRxiv

Serological differentiation between COVID-19 and SARS infections

Clinical evaluation of serological IgG antibody response on the Abbott Architect for established SARS-CoV-2 infection

Clinical Evaluation of the cobas SARS-CoV-2 Test and a Diagnostic Platform Switch during 48 Hours in the Midst of the COVID-19 Pandemic

The important role of serology for COVID-19 control. The Lancet. Infectious diseases

Protective immunity after COVID-19 has been questioned: What can we do without SARS-CoV-2-IgG detection?

Non-respiratory viruses

We would like to acknowledge Ms. Enas S. Al-Absi for her help in sorting the negative control samples.