key: cord-0921940-0frhiwcc
authors: Furuya, Andrea K.M.; Wagner, Christine; Connors, Julia; Bodnar, Janine; Miller, Thomas; Ryman, Nko Lea; Warszycki, Casey J.; Wong, Susan J.; Walsh, Anne C.; McDonough, Kathleen A.; Lee, William T.
title: Performance evaluation of antibody-based point-of-care devices intended for the identification of immune responses to SARS-CoV-2
date: 2020-12-24
journal: Diagn Microbiol Infect Dis
DOI: 10.1016/j.diagmicrobio.2020.115298
sha: 3881471a57cb4fc144a9b50b2778aef703a569be
doc_id: 921940
cord_uid: 0frhiwcc

The novel coronavirus outbreak caused by the severe acute respiratory syndrome coronavirus (SARS-CoV-2) was first identified in December of 2019 in Wuhan, China. The local outbreak quickly rose to pandemic level that has spread to more than 188 countries with more than 19 million cases and 732 467 deaths worldwide. The current recommendation for testing is RT-PCR based tests of nasopharyngeal or alternatively nasal- and/or oropharyngeal swabs that detect infection with SARS-CoV-2 to diagnose acute infection. However, there is an urgent need for a quick and accurate antibody-based point-of-care (POC) test method to quickly identify evidence of SARS-CoV-2 infection among people who might be missed through active case finding and surveillance efforts. Serology tests measure the presence of antibodies in serum after infection. Here we compared the performance characteristics of six commercially available antibody-based POC devices and their potential for identification of individuals infected at some time by SARS-CoV-2.

There is growing recognition of the importance for serology testing for SARS-CoV-2, not only for seroprevalence purposes, but also to help to guide "return to work" or other COVID-19 exposure decisions. Although most testing is performed in the laboratory environment, there have been a large number of rapid diagnostic serology tests that have been made available [1] .

Many of these tests have not been reviewed by the Food and Drug Administration (FDA).

Several more have been reviewed by the FDA and have received Emergency Use Authorization (EUA), although some of these tests have had their EUA revoked [2] . These instances point to the high variability in the performance of these "Point of Care" (POC) devices. Conceptually, POC devices offer many advantages such as the ability to detect IgM and IgG from a very small amount of blood, plasma or serum, and provide results within 10-15 minutes. We had the opportunity to evaluate the performance characteristics of a limited number of rapid test devices and compare them to a high complexity test used in our laboratory. from healthy, COVID-19 convalescent patients who were at least 25 days post symptom onset.

These specimens were all reactive at various levels in the comparator assay, New York SARS-CoV Microsphere Immunoassay (MIA). The MIA has FDA EUA and its development and performance characteristics are described in [3] . These specimens are labeled as "Positive by MIA" in this study. For specificity and control sera, we used 1) pre-COVID-19 sera purchased, submitted for clinical testing, or gifted to the Wadsworth Center that were characterized by positive identification for known pathogens or autoantigens; 2) sera obtained during the COVID-19 pandemic from individuals experiencing respiratory infections, but with a negative SARS-CoV-2 RT-PCR result (a kind gift from Dr. E. Hod, Columbia University Medical Center, NY).

Specimens were assessed for the presence of total antibody using the New York SARS-CoV MIA. For the present study, the original determination of COVID-19 positivity was based on reactivity to the highly cross-reactive SARS-CoV-1 nucleocapsid protein (N), however subsequent analyses showed that all the MIA reactive specimens had antibodies reactive with the SARS-CoV-2 N protein and all but one specimen also had antibodies reactive with the receptor binding domain (RBD) of SARS-CoV-2 spike protein. The assay detects total CoV-specific antibodies in human sera using an anti-human immunoglobulin reagent (reactivity to IgG, IgA, and IgM; Life Technologies, Grand Island, NY), but was also modified to separately identify IgG or IgM, where noted. Analysis was performed using a FLEXMAP 3D Analyzer (Luminex Corp.) and the results are provided as median fluorescence intensities (MFI). The specificity and sensitivity of the assay are 99% and 88%-93%, respectively [3, 4] .

Testing was performed at room temperature (20-25°C) according to manufacturer's instructions.

Briefly, serum (please refer to Table 1 for exact amounts) was directly pipetted into the sample well of the cassette, followed by 2-3 drops of buffer solution. The results in the form of pink/purple colored lines were read visually within 10-15 minutes. Test results were considered valid if the internal control line was visible. Weak signals for IgM and/or IgG was/were considered positive.

Specificity and sensitivity were calculated as follows:

Sensitivity % = 100*[True positive/ (true positive + false negative)] Specificity % = 100*[true negative/ (true negative + false positive)]

All sera used in this study were first characterized using the NYS SARS-CoV MIA. The serum samples used to measure the performance of the POCTs in this study were grouped as outlined in Table 2 .

Performance characteristics of POC tests based upon MIA reactivity. To determine the sensitivity of the POC devices in this study, we asked three fundamental questions: first, if there were any antibodies reactive with SARS-CoV-2 in the serum specimens, could the POC tests detect them; second, at what level must antibodies be present in the sera in order for the POC tests to be able to detect them; third, since all of the POC tests purport to detect, individually, both IgM and IgG, how well did they do so in IgM+/IgG+ sera? To address these questions, we analyzed sera from known convalescent COVID-19 positive individuals, as determined by RT-PCR positivity, and at least 21 days post-symptom onset. The sera were assessed using the NY SARS-CoV-2 MIA which provides a signal (MFI) that is proportional to antibody abundance.

Using reactivity to the N antigen as a standard, the sera were categorized as being low, intermediate, or high antibody sera ( Table 2 ). All but one of the sera also had varying amounts of antibodies reactive with the RBD antigen. In addition, all specimens were IgM and IgG positive, except for two specimens in the low and one specimen in the intermediate groups that lacked IgM. Table 1 , three of the POC tests detected IgM and IgG on a single device, while the other three POC tests required separate devices to singly detect IgM and IgG. IgM and IgG are scored separately regardless of whether the bands are on one device or separate devices.

Possible outcomes are no bands (nonreactive), IgM+/IgG-, IgM-/IgG+ or IgM+/IgG+. The overall sensitivity and the summary of IgM and IgG individually detected for each of the devices is shown in Table 3 . The POC devices had a large amount of variability in their detection levels (80%-97% over 30 samples), even in specimens with high antibody abundance. Overall, IgG was better detected than IgM. Even greater variability was seen when looking at IgM detection by itself (one band/device) or with IgG (two bands/devices), or, conversely, IgG detection by itself or with IgM. The distribution of IgM (or IgG) alone versus detecting both antibody classes is shown in Figure 1 . RayBiotech: without regard to isotype, this device could detect Ab in 80% (95% CI: 60.8-91.5) of the 30 specimens tested. At the lowest level of Ab abundance, based upon N antigen reactivity in the MIA, this device could detect Ab in 60% of the specimens. For separate detection of IgM versus IgG, both had a wide range in positive results over the different antibody abundance groups (IgM 63% -90%; IgG 60% -80%). We note that one specimen in each of the high and low Ab abundance groups did have a weakly reactive IgM band, which the kit considers inconclusive with a reflex to an alternative testing method.

SD Biosensor: without regard to isotype, this device could detect Ab in 96.70% (95% CI: 80.9-99.8) of the 30 specimens tested. At the lowest level of Ab abundance, based upon N antigen reactivity in the MIA, this device could detect Ab in 90% of the specimens. For separate detection of IgM versus IgG, IgM positivity had a larger variation among the Ab abundance groups (56% -100%) than did IgG positivity (80% -100%).

To further characterize the rapid diagnostic tests, we also examined their ability to detect antibody production in sera from early acute COVID-19 patients. Sera from 14 individuals, who had tested positive for SARS-CoV-2 RNA in an RT-PCR test, were tested at the Wadsworth Center using the MIA and additionally tested using the POC devices. Of these sera, the MIA detected antibodies at a high level in 3 specimens and did not detect antibodies in 11 specimens.

The failure to detect antibodies could be due to the sensitivity limit of the MIA or because the COVID-19 individuals had not (yet) produced antibodies. In addition, two of the sera had an indeterminate result in the MIA, which could be due to either a very low level of COVID-19specific antibodies in the sera or Ab cross-reactivity with other antigens. Given the recent infection with SARS-CoV-2, the former explanation is the likeliest. In testing the POC devices, we asked the question: in these instances of known COVID-19 infection, could the devices detect Abs when the MIA could not; i.e., would they perform better than the comparator assay?

As shown in Figure 2 , for any of the POC devices, Abs were only detected in the specimens which were reactive in the MIA. No Abs were detected in samples found to be either nonreactive or indeterminant using the MIA. Not all of the rapid tests, however, could detect in Abs in even the MIA positive specimens. The BioMedomics, BTNX, UCP Bioscience, and Zeus devices were Ab positive for all three specimens. However, while the first three devices detected both IgM and IgG in each of the three specimens, the Zeus devices detected IgG, but not IgM, in two of the three sera. This result is consistent with those obtained using convalescent sera, which showed the Zeus device to be the least sensitive in identifying IgM positive sera. Of the remaining devices, the RayBiotech device detected IgM and IgG in two of the three MIA positive specimens. We also note that two additional specimens had a weakly reactive IgM band, which the kit considers inconclusive with a reflex to an alternative testing method. Due to insufficient specimen volume, only one of the MIA reactive specimens could be tested using the SD Biosensor device and that specimen was found to be positive for both IgM and IgG.

Having determined the relative abilities of the POC tests to detect Abs that were present in positive sera, we next wished to determine if the devices provided positive results when COVID-19 Abs were not present. To assess specificity, we evaluated the devices using a panel known to be negative COVID-19, but to also have Abs to other infectious agents or autoimmune markers. Table 4 and Supplemental tables 1-6). We also had the opportunity to evaluate most of the POC devices using a small panel of sera from individuals which had recently been diagnosed, using a molecular respiratory pathogen panel test, as being infected with a non-SARS-COV-2 coronavirus (OC43 (n=2), NL63 (n=2) and HKU1 (n=1)). None of the specimens was reactive for any of the tests, except for a positive IgM result in the UCP Biosciences device with the serum from a recent infection with HKU-1.

Although the number of specimens tested was low (five, in total) it is encouraging that the false positivity rate is low with these related coronaviruses. We should note however, that, due to low specimen volumes, only three specimens NL63 (n=2) and OC43 (n=1) were tested on the SD Biosensor devices. Also, due to the limited number of devices, none of these specimens were tested on the RayBiotech devices.

The ease of use, fast turnaround time, and relative low cost make the use of POC tests an attractive option for serology (serosurveys and companions to molecular tests for determining exposures). However, as this small survey shows, there is great breadth in performance, both with respect to sensitivity and also specificity. In our opinion, the performance of many of these devices do not match up to high complexity tests and caution must be exercised in the choice of a particular rapid test for determining antibody responses to SARS-CoV-2.

Studies were performed on remnant sera from de-identified clinical specimens, submitted to the NYSDOH, as part of routine clinical testing for SARS COV-2. Testing was done in a CLIAcertified clinical laboratory. Testing of human specimens was in response to a declared Public Health Emergency and is not considered to be research. All procedures performed in this study involving human serum samples were done in accordance with the ethical standards of the institutional research committee of the NYSDOH. Acute serum specimens determined by RT-PCR 5 Table 2 . Groups of specimens used in evaluating antibody-based point-of-care (POC) testing. Table 4 . Summary of specificity results. * due to several inconclusive results of the RayBiotech devices two different sets of performance measures were calculated (please see Supplemental Table 4 ). **IgM results on the RayBiotech devices were inconclusive. Per kit instructions these samples should be tested using an alternative method to confirm results. 14 specimens were tested on the POC devices and the percentage of positive and negative IgM, IgG and IgM+IgG results were recorded. # Due to limited number of RayBiotech devices, only 12 specimens were tested. ## There were a total of 16 specimens tested on the SD Biosensor devices. PCR: ploymerase chain reaction; POC: point-of-care; Ig: immunoglobulin.

Insight into FDA's Revised Policy on Antibody Tests: Prioritizing Access and Accuracy

FDA) USFDa. FAQs on Testing for SARS-CoV-2

SARS-CoV-2 antibody characterization in emergency department, hospitalized and convalescent patients by two semi-quantitative immunoassays

We would like to acknowledge and thank members of the diagnostic immunology and virology 

Requires predilution of sample