key: cord-1018623-2f14q2u5
authors: Robosa, Roselle s.; Sandaradura, Indy; Dwyer, Dominic e.; O’sullivan, Matthew v.N.
title: Clinical evaluation of SARS-CoV-2 point-of-care tests
date: 2020-09-22
journal: Pathology
DOI: 10.1016/j.pathol.2020.09.002
sha: 784e720524d22e81d27f896811fbbd5bc29eee85
doc_id: 1018623
cord_uid: 2f14q2u5

The aim of this study was to assess the analytic and clinical performance of four rapid lateral flow point-of-care tests (POCTs) for identifying SARS-CoV-2-specific antibodies. A retrospective study was conducted between 22 January and 30 March 2020 on 132 serum samples for SARS-CoV-2-specific antibody detection referred to a tertiary referral hospital laboratory in New South Wales. Multiple sera were tested from 20 confirmed or suspected COVID-19 patients with SARS-CoV-2-specific antibodies detected by immunofluorescence (IFA) or neutralisation, and 71 SARS-CoV-2 uninfected individuals. We measured the sensitivity and specificity for detection of SARS-CoV-2 IgM and IgG antibodies for each POCT in comparison to positive SARS-CoV-2-specific IFA and viral neutralisation, our current laboratory benchmark tests. All POCTs were found to have a low analytic sensitivity for SARS-CoV-2 antibodies, ranging from 27.3% to 58.2%, with a specificity between 88.3% and 100%, and a low clinical sensitivity from 45% to 65%, with a clinical specificity between 87.3% and 100%. All POCTs had an increased sensitivity when specimens were collected more than 14 days from onset of symptoms. The detection using POCT of SARS-CoV-2-specific antibodies after disease onset lagged behind IFA by a range of 2–9 days. POCTs promise the benefit of providing quick easy testing for SARS-CoV-2-specific antibodies. However, their poor sensitivity and delayed antibody detection make them unsuitable as a diagnostic or screening tool alone.

The current coronavirus disease 2019 (COVID-19) outbreak caused by a novel coronavirus named severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), was first reported in Wuhan, China in December 2019. 1 The primary means of laboratory diagnosis for COVID-19 disease is nucleic acid testing (NAT) on deep nasal, nasopharyngeal and throat swabs, or lower respiratory tract specimens, using real-time reverse-transcriptase polymerase chain reaction (RT-PCR) during the acute symptomatic phase of illness.

Detection of SARS-CoV-2-specific antibodies is another method to identify recent or past infection with SARS-CoV-2. The SARS-CoV-2 envelope proteins trigger antibodies that are neutralising; the most important is thought to be the spike protein (S), which is responsible for attachment, fusion and viral entry into host cells, 2 and is an obvious target for serology test development. Other potential targets include the nucleocapsid protein (N). 3 Antibodies to SARS-CoV-2 are detected 7-10 days post-illness onset with studies showing the majority of patients seroconverting by weeks 2-3; this can vary depending on factors including the patient's immune status and disease severity. 4 Serology alone is not recommended for acute diagnosis of COVID-19, though is likely to be useful in the confirmation of recent or past COVID-19 infections (for example, in patients presenting seven or more days from symptom onset). Serology has proven useful in detecting convalescent cases to aid establishing epidemiological links between clusters. 5 It is uncertain whether the presence of SARS-CoV-2-specific antibodies indicates immunity from further infection, and how long antibodies persist following acute infection.

There is widespread interest in the use of point-of-care tests (POCTs). A media release by the Commonwealth Minister for Health in late March stated that the Australian Government had ordered 1.5 million POCTs to expand Australia's testing capacity for COVID-19 disease. 6 Potential benefits include a rapid turnaround time (as short as 15 minutes) and ease of performance, especially advantageous in remote and rural settings. 7 Most commercially available POCTs are based on detection of SARS-CoV-2 antigens or antibodies, and are generally rapid lateral flow assays (LFA) that detect IgM and/or IgG.

Twenty-two POCTs have been listed by the Australian Therapeutic Goods Administration (TGA) for use in Australia, and are undergoing an expedited post-marketing evaluation on the Australian Register of Therapeutic Goods. 8 Previous experience with antigen-detecting LFA for influenza have shown reduced sensitivity compared to NAT. [9] [10] [11] Concerns regarding the lack of robust validation of POCTs and the significant consequences of their misapplication has led to several bodies including the World Health Organization, the TGA, the Public Health Laboratory Network and The Royal College of Pathologists of Australasia to caution against their use for diagnosis of COVID-19 disease.

This study aimed to assess the analytic and clinical performance of POCTs in identifying SARS-CoV-2-specific antibodies, and so to help determine their role in the Australian setting.

We conducted a retrospective study evaluating the clinical sensitivity and specificity of four commercial lateral flow assay devices marketed as POCTs for the detection of SARS-CoV-2specific antibodies. All POCT were performed using serum according to the manufacturer's protocol. Test results were interpreted as negative, equivocal or positive depending on appearance of the band on J o u r n a l P r e -p r o o f the testing strip. Equivocal results were included as positive tests in our analyses, as the presence of a faint band is likely to be conservatively interpreted as positive by clinicians.

The in-house immunofluorescence assay (IFA) and neutralisation antibody assay were used as the benchmark. Compared to NAT-confirmed cases, the sensitivity and specificity of detecting one or more classes of SARS-CoV2 antibodies (IgG, IgM or IgA) by IFA when collected 14 days or more after symptoms onset was 91.3% (95% Cl 84.9-95.6) and 98.9% (98.4-99.8) respectively. 12 The principles of viral neutralisation and the IFA have been previously described. [13] [14] [15] 

Sixty serum samples were obtained from COVID-19 NAT-confirmed patients (n=17) or COVID-19 suspected patients without positive NAT tests (n=3) who were symptomatic household contacts of confirmed cases. The NAT method used was an in-house nucleic acid test targeting E gene and RdRp: the primers have been described elsewhere. 16 Both initial samples (n=20) and convalescent samples (n=40) were included. All confirmed and suspected COVID-19 patients had SARS-CoV-2-specific IgM, IgG and/or IgA antibodies detected by both IFA and neutralisation on one or more serum samples. Clinical data were available including date of symptom onset, SARS-CoV-2 NAT results and possible exposure history. In SARS-CoV-2 seropositive samples, the presence of SARS-CoV-2-specific IgM and/or IgA were regarded as suggestive of a recent infection. The detection of SARS-CoV-2specific IgG alone could not reliably be used to differentiate a past infection from a recent infection. SARS-CoV-2-specific IgG can often precede IgM or IgA following acute infection, and so the test should be interpreted within the context of the patient's date of symptom onset and potential exposure. This pattern has also been documented in SARS and SARS-CoV-2. [17] [18] [19] There were 18 patients with SARS-CoV-2-specific IgM and/or IgA detected by IFA, suggestive of a recent infection. A past infection was defined as a resolution of symptoms in addition to being 10 days or more from first positive NAT with the absence of a positive IFA IgM/IgA. Two patients had isolated SARS-CoV-2-specific IgG detected, with one patient likely having a past infection, as onset of symptoms predated serology by 50 days. The second patient likely had a recent infection with IgG development prior to appearance of IgM or IgA. Negative control samples (n=72) were collected from stored baseline testing for asymptomatic staff submitted at the start of the pandemic (n=9), symptomatic patients found to be SARS-CoV-2 NAT negative (n=21), serum samples submitted before the pandemic (June to August 2019) for respiratory pathogen testing found to be positive for Mycoplasma pneumoniae antibodies (n=8) or influenza A antibodies (n=19), and samples positive for rheumatoid factor (n=15). These negative control samples were negative (titre <10) for SARS-CoV-2-specific IgG, IgM and IgA by IFA and for SARS-CoV-2 neutralising antibody.

A total of 132 serum samples from 91 patients were tested by the OnSite COVID-19, Sinocare SARS-CoV-2, Standard Q COVID-19 Duo and Combo antibody tests. Only 100 samples from 60 patients (including 60 samples from confirmed or suspected patients and 40 negative control samples) were tested by the Innovita 2019-nCov Antibody Test, due to a limited availability of testing kits.

Tang et al. highlighted the importance of distinguishing 'analytic' and 'clinical' sensitivity when assessing methods to detect COVID-19, namely the ability of an assay to detect SARS-CoV-2-specific antibodies when present in sera, versus the ability of an assay to identify a patient's overall immune status to the virus. 20 The analytical sensitivity and specificity of each POCT was calculated independently for the presence of SARS-CoV-2-specific IgG and IgM in comparison to IFA. Clinical sensitivity was defined as the assessment of the overall development of antibodies in individual patients, and was measured as the detection of any antibody in any sample from a given patient. In addition, the clinical sensitivity and specificity of the test's ability to ascertain a correct immune status was assessed. Furthermore, we compared the sensitivity rate of specimens collected more than 14 days following onset of symptoms.

We also evaluated the median IFA antibody titre of the samples which were positive on each of the POCTs. Binomial 95% confidence intervals were calculated for all proportions.

We calculated the window periods from the onset of symptoms until antibody detection in three patients with serial collections who initially had negative serology.

Characteristics of all 91 patients including 20 COVID-19 confirmed or suspected patients and 71 uninfected patients are summarised in Table 1 .

When individually assessing the 60 samples from the 20 confirmed or suspected COVID patients, 55/60 samples had IgG detected and of those, 49 also had SARS-CoV-2-specific IgM and/or IgA detected. In 6/60 samples, only IgG was detected. The majority of samples (40%) were collected in week two post-onset of symptoms. Three patients had evidence of SARS-CoV-2-specific antibody seroconversion on serial testing and four-fold rise of IgG, a further three patients had a four-fold IgG rise (see Table 2 ).

Performance characteristics of the commercial POCT IgG and IgM assays compared to IFA are summarised in Table 2 . All had poor sensitivities for IgG and IgM, with relatively high specificities. Our evaluation had significantly lower sensitivity than the stated performances provided by the manufacturer for all POCTs (see Table 3 ). However, the Standard Q Duo and Combo tests had increasing sensitivity with symptom onset >7 and >14 days, which is a trend we observed in our analysis. The only false positive IgG using the OnSite assay was from a patient with known positive Mycoplasma pneumoniae IgM. The Sinocare test also demonstrated some cross-reactivity with samples positive on M. pneumoniae and influenza A serology. The Standard Q tests had two false positives: one was a patient with known rheumatoid factor, and the other from a SARS-CoV-2 uninfected patient. Furthermore, there were nine samples that were positive for IgA by IFA, but negative for IgM by IFA, OnSite and Innovita assays. The standard Q IgM assay detected two samples. All nine of these samples also had SARS-CoV-2-specific IgG detectable by IFA, yet none were detected by the Innovita IgG assay, only 3/9 were detected by both the OnSite IgG assay and Standard Q IgG assay and one detected by Sinocare assay.

There was a trend to improved detection rates with increasing titres of IgA, IgG and IgM by IFA ( Fig. 1 and 2) , and the window periods from the time of illness onset to detection of antibody were delayed by at least two days in each of the POCTs when compared to IFA in the three patients where serial serology samples were available (Table 4) .

When analysing samples collected greater than 14 days from onset of symptoms, there was an increase in sensitivity across all POCT devices in both IgG, IgM and total antibody (see Table 2 ).

Twenty patients had antibody detectable by IFA and neutralisation at one or more time points after illness onset. Thirteen of these had any antibody detectable at any time point by the OnSite assay (sensitivity 65%), 12 by the Standard Q assay (sensitivity 60%), nine by the Innovita POCT (sensitivity 45%) and 11 by the Sinocare POCT (sensitivity 55%) ( Table 2) .

Sinocare SARS-CoV-2 tests was undertaken to determine their efficacy in an Australian setting during the first wave of the COVID-19 pandemic.

The manufacturers of the POCTs have listed high sensitivities of 96.9% (OnSite), 87.3% (Innovita), 96.3% (Sinocare) and 99.1% (Standard Q) in their package information. Our findings contradicted these reports, as despite an increase in sensitivity in specimens collected more than 14 days post-onset of symptoms, all four POCTs had an unacceptably low analytic and clinical sensitivity for SARS-CoV-2-specific antibodies, with a delay in antibody detection.

A recent post-market evaluation of two LFA POCTs released by the Doherty Institute also shows poor total antibody sensitivity with 56.9% and specificity 95.6% for the OnSite test and 51.8% sensitivity and 97.8% for VivaDiag COVID-19 IgM/IgG Rapid Test. 21 They compared POCTs against both NAT and viral neutralisation. However, benefits of evaluating POCTs against IFA and viral neutralisation are that both tests are detecting and quantifying antibodies, and IFA (along with ELISA-based assays) have historically been the most widely used standard for serological diagnosis. 22 Our study is an independent validation of POCTs for COVID-19 infection in an Australian setting. Currently there are limited data reporting the diagnostic performance and independent validation of POCTs for diagnosis of SARS-CoV-2 in clinical samples. One of the largest studies used a LFA POCT device on 525 cases with a sensitivity of 89% and specificity of 91%. 23 A small study of 39 randomly selected patients from a high-prevalence area screening centre (German Red Cross), compared an LFA to NAT with a sensitivity of 36.4% and specificity of 88.9%. 24 The slight superiority in sensitivity of the combined IgG-IgM antibody test over individual IgG or IgM antibody test was seen in a previous study. 23 Since the appearance of IgG may pre-date IgA or IgM responses in SARS-CoV-2 infection as seen previously in SARS, [17] [18] [19] it is difficult to classify an individual as having recent or past infection from a serological test at a single time point. We attempted to assess whether a patient had a recent or past infection as part of clinical sensitivity, we determined that this was not possible using POCTs. We recommend serial measurements using a quantitative assay (like IFA or neutralisation) to detect seroconversion or a rising titre as a more reliable method for timing infection than looking for the presence of SARS-CoV-2 specific IgM. Furthermore, we found SARS-CoV-2-specific IgA was frequently present in the absence of SARS-CoV-2-specific IgM by IFA, and in cases when they were both present, IgA was present in a higher titre, making it a more reliable serological marker of SARS-CoV-2 infection. In patients with recent infections when analysed using IFA IgA as a benchmark, IFA IgG had a sensitivity of 87.3%. There was a small increase in sensitivity across all POCTs when assessing the presence of IgG, which tends to predate IgM and IgA (Supplementary Fig. 1 and Supplementary Table 1, Appendix A) .

There has been speculation regarding the application of POCTs to detect SARS-CoV-2 specific-antibody in combination with PCR to produce an 'immunity passport' to guide infection control and as part of strategies to ease lock-down measures. 25 This assumes that past SARS-CoV-2 infection conveys immunity. However, the degree of immunity postinfection, and the seroprevalence in the Australian community are both unknown. The current POCTs are not sufficiently sensitive to be utilised in this capacity and further development of POCTs is required.

The main study limitation is the relatively modest number of samples available for testing early in the pandemic in NSW. However, we were able to test these samples against four different devices. The timing of serum collection of initial and convalescent samples were not standardised, and not all patients had convalescent samples collected. Ideally, serial testing from well characterised serum panels would be used to most accurately define window periods.

In summary, there is a need for robust serological assays for SARS-CoV-2 to supplement diagnosis by NAT and help guide public health interventions. Despite potential benefits of POCTs, their current poor sensitivity and delayed SARS-CoV-2 specific antibody detection make them unsuitable to be utilised as a diagnostic tool, as an accurate marker of past infection, or in seroprevalence surveys. Fig. 1 Positive rate of IgG and total antibody point-of-care test results by SARS-CoV-2 specific IgG immunofluorescence assay titre in overall and in samples collected >14 days post-onset of symptoms. J o u r n a l P r e -p r o o f J o u r n a l P r e -p r o o f Table 3 Sensitivity reported by manufacturers

Early transmission dynamics in Wuhan, China, of novel coronavirus-infected pneumonia

Genomic characterisation and epidemiology of 2019 novel coronavirus: implications for virus origins and receptor binding

Diagnosis of acute respiratory syndrome coronavirus 2 infection by detection of nucleocapsid protein

Different longitudinal patterns of nucleic acid and serology testing results based on disease severity of COVID-19 patients

Connecting clusters of COVID-19: an epidemiological and serological investigation

Doorstop interview in Canberra about coronavirus (COVID-19) 2020

Impact of point-of-care testing for white blood cell count on triage of patients with infection in the remote Northern Territory of Australia

Post market review of COVID-19 point-of-care tests 2020

A highly specific rapid antigen detection assay for onsite diagnosis of MERS

Parallel validation of three molecular devices for simultaneous detection and identification of influenza A and B and respiratory syncytial viruses

Accuracy of rapid influenza diagnostic tests: a meta-analysis

The antibody response to SARS-CoV-2 Infection. Open Forum Infect Dis

Lennette EH, Lennette DA. Diagnostic Procedures for Viral, Rickettsial, and Chlamydial Infections

Lower immunity to poliomyelitis viruses in Australian young adults not eligible for inactivated polio vaccine

Immunofluorescence assay for serologic diagnosis of SARS

Detection of 2019 novel coronavirus (2019-nCoV) by real-time RT-PCR

Longitudinal profile of immunoglobulin G (IgG), IgM, and IgA antibodies against the severe acute respiratory syndrome (SARS) coronavirus nucleocapsid protein in patients with pneumonia due to the SARS coronavirus

Chronological evolution of IgM, IgA, IgG and neutralisation antibodies after infection with SARS-associated coronavirus

Antibody responses to SARS-CoV-2 in patients with COVID-19

The laboratory diagnosis of COVID-19 infection: current issues and challenges

Post-market evaluation of serology-based point of care tests

Detection of specific antibodies to severe acute respiratory syndrome (SARS) coronavirus nucleocapsid protein for serodiagnosis of SARS coronavirus pneumonia

Development and clinical application of a rapid IgM-IgG combined antibody test for SARS-CoV-2 infection diagnosis

Rapid point-of-care testing for SARS-CoV-2 in a community screening setting shows low sensitivity

The promise and peril of antibody testing for COVID-19

Standard Q COVID-19 Combo Test IgG and/or IgM Symptom onset <7

We thank the assistance of Andrew Sargeant and Christopher Kot from Point of Care Testing Services, New South Wales Health Pathology; the clinical and laboratory staff at the Centre for Infectious Diseases and Microbiology for sample collection, preparation and testing; and the ICPMR COVID-19 study group.

Wales Government Office for Health and Medical Research. The authors state that there are no conflicts of interest to disclose.

 Sinocare  1  11  13  14  13  13  2  12  14  21  14  14  3  13  16  16  13  16 a Patients with serial collections and initial negative serology.