key: cord-0890693-drrsil41
authors: Muhi, Stephen; Tayler, Nick; Hoang, Tuyet; Ballard, Susan A.; Graham, Maryza; Rojek, Amanda; Kwong, Jason C.; Trubiano, Jason A.; Smibert, Olivia; Drewett, George; James, Fiona; Gardiner, Emma; Chea, Socheata; Isles, Nicole; Sait, Michelle; Pasricha, Shivani; Taiaroa, George; McAuley, Julie; Williams, Eloise; Gibney, Katherine B.; Stinear, Timothy P.; Bond, Katherine; Lewin, Sharon R.; Putland, Mark; Howden, Benjamin P.; Williamson, Deborah A.
title: Multi-site assessment of rapid, point-of-care antigen testing for the diagnosis of SARS-CoV-2 infection in a low-prevalence setting: A validation and implementation study
date: 2021-03-02
journal: Lancet Reg Health West Pac
DOI: 10.1016/j.lanwpc.2021.100115
sha: ff61ae0c9ee5525a44d61e29f5d2d92491b937be
doc_id: 890693
cord_uid: drrsil41

BACKGROUND: In Australia, COVID-19 diagnosis relies on RT-PCR testing which is relatively costly and time-consuming. To date, few studies have assessed the performance and implementation of rapid antigen-based SARS-CoV-2 testing in a setting with a low prevalence of COVID-19 infections, such as Australia. METHODS: This study recruited participants presenting for COVID-19 testing at three Melbourne metropolitan hospitals during a period of low COVID-19 prevalence. The Abbott PanBio(TM) COVID-19 Ag point-of-care test was performed alongside RT-PCR. In addition, participants with COVID-19 notified to the Victorian Government were invited to provide additional swabs to aid validation. Implementation challenges were also documented. FINDINGS: The specificity of the Abbott PanBio(TM) COVID-19 Ag test was 99.96% (95% CI 99.73 - 100%). Sensitivity amongst participants with RT-PCR-confirmed infection was dependent upon the duration of symptoms reported, ranging from 77.3% (duration 1 to 33 days) to 100% in those within seven days of symptom onset. A range of implementation challenges were identified which may inform future COVID-19 testing strategies in a low prevalence setting. INTERPRETATION: Given the high specificity, antigen-based tests may be most useful in rapidly triaging public health and hospital resources while expediting confirmatory RT-PCR testing. Considering the limitations in test sensitivity and the potential for rapid transmission in susceptible populations, particularly in hospital settings, careful consideration is required for implementation of antigen testing in a low prevalence setting. FUNDING: This work was funded by the Victorian Department of Health and Human Services. The funder was not involved in data analysis or manuscript preparation.

Evidence before this study

The reported sensitivity and specificity of the Abbott PanBio TM COVID-19 Ag test (as reported by the manufacturer) is 91.4% and 99.8%, respectively. A small number of studies in high prevalence settings have demonstrated similar or reduced sensitivity but comparable specificity. Few studies have reported the use of SARS-CoV-2 antigen tests in a very low prevalence setting, and the obstacles to implementation in this setting require exploration.

This study reports the findings of antigen based SARS-CoV-2 testing in a low prevalence, hospital-based setting. We observed high specificity, but due to the low prevalence, no cases of RT-PCR-confirmed COVID-19 were newly diagnosed in the hospital arm of the study. Sensitivity, as determined from participants with known COVID-19, depended on the duration of symptoms. Numerous implementation challenges were identified; solutions to which may inform future pointof-care testing strategies in a low prevalence setting.

In a low prevalence setting, all positive SARS-CoV-2 antigen based tests should be confirmed with RT-PCR. Antigenbased tests may assist resource allocation while awaiting confirmatory testing. Considering the ability of the virus to spread rapidly in susceptible populations, particularly hospitals, careful consideration is required for implementation in a low prevalence setting.

The scale and speed of the coronavirus disease (COVID-19) pandemic caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is unprecedented [1] . In the current absence of a vaccine, public health responses have relied largely on a combination of population-level non-pharmaceutical interventions (NPIs) and individual-level diagnostic testing [2] . To date, diagnostic testing for SARS-CoV-2 has relied on highly sensitive reverse-transcriptase PCR (RT-PCR) assays performed in clinical laboratories. However, RT-PCR is relatively expensive and, depending on the setting, results may take 24-48 hours to return. Importantly, such delays in testing and contact tracing may lead to further transmission of disease [3] . Recently, rapid antigen tests have been proposed as a means of increasing population-level surveillance testing and enabling testing at, or near, the point of care (POC) [4] . As antigen tests detect viral protein rather than amplified nucleic acid, they are inherently less sensitive than RT-PCR assays. To offset this reduced sensitivity, it has been suggested that increasing the frequency of testing may enable rapid identification and isolation of infected individuals [ 4 , 5 ] .

Australia has one of the highest COVID-19 testing rates in the world; to date all diagnostic testing for COVID-19 in Australia has been performed using laboratory-based RT-PCR. These high testing rates, coupled with early and aggressive NPIs, including closure of the Australian border, have contributed to Australia having one of the lowest COVID-19 infection rates globally [6] . However, as societies begin to interact and international travel resumes, the requirement for rapid, scalable population-level testing may not be fully met by laboratory-based RT-PCR testing. Although antigen testing may have potential benefits in enabling widespread testing, the feasibility and utility of implementing this testing in a lowprevalence setting such as Australia have not been assessed. Here, we undertook a laboratory and multi-site clinical validation study of a commercially available rapid antigen test, the Abbott PanBio TM COVID-19 Ag test. At the time of study initiation, it was one of two rapid POC tests with emergency use authorised by the World Health Organisation and was approved for supply by the Australian Therapeutic Goods Administration [ 7 , 8 ] . Specifically, the aims of our study were to: (i) determine the performance characteristics of the Abbott PanBio TM COVID-19 Ag test against an RT-PCR reference standard; (ii) identify the implementation challenges associated with rapid antigen point-of-care testing, and (iii) develop a framework for the implementation of rapid antigen testing in a low-prevalence setting.

We undertook a prospective study across three academic hospitals located in Melbourne (population 4.97 million), which is the capital of the Australian state of Victoria. In Victoria, the COVID-19 pandemic has been characterised by two peaks of transmission, the first occurring between March and April 2020 (maximum 622 active cases) and the second between July and September (maximum 7,880 active cases) [9] . This study commenced in late September after significant public health interventions had controlled transmission, during which time the 14-day average in metropolitan Melbourne decreased from 20.3 to zero new cases per day [10] . The three participating hospitals were Monash Health (located in Melbourne's south-eastern suburbs, including Monash Medical Centre, Casey Hospital and Dandenong Hospital and servicing one quarter of Melbourne's population), Austin Hospital (located in Melbourne's north-eastern suburbs) and the Royal Melbourne Hospital (RMH) City Campus, with catchment extending to Melbourne's north-western suburbs. The study recruited participants at the RMH in a pre-pilot phase from 28 September 2020 and from all three sites from 19 October 2020 to 14 November 2020. Individuals presented for COVID-19 testing at each of these hospitals mainly due to: (i) the presence of symptoms consistent with COVID-19 or (ii) contact tracing / outbreak management responses. Participants underwent standard-of-care RT-PCR testing using a combined throat and deep nasal swab, plus simultaneous antigen testing using the Abbott PanBio TM COVID-19 Ag test if consent was provided (Supplementary Figure 1 ). All participants provided information about the presence, absence and duration of clinical symptoms, and possible exposures to SARS-CoV-2, which was recorded in a secure REDCap database hosted at the RMH [11] . For testing using RT-PCR, patients were swabbed in accordance with their local hospital procedure; all sites performed a single combined throat and deep nasal swab, of either one or both nasal cavities. For the Abbott PanBio TM COVID-19 Ag test, all sites performed a single deep nasal swab. Site investigators from the three sites met virtually each day to identify emerging logistical issues.

In addition to individuals presenting to screening clinics, RT-PCR and antigen testing were simultaneously performed on individuals with known COVID-19 infection notified to the Victorian DHHS who provided consent for additional sample collection. These samples provided additional material for validation of the Abbott PanBio TM COVID-19 Ag test and were tested at the Microbiological Diagnostic Unit Public Health Laboratory (MDU PHL), The University of Melbourne at the Doherty Institute, Australia. For these participants, RT-PCR and Abbott PanBio TM COVID-19 Ag tests were performed using a single deep nasal swab.

Swabs were collected using either a flocked Copan© Eswab in 1 ml liquid amies or flocked Kang Jian© swab in 3 ml universal transport media and tested using the preferred RT-PCR assay at each of the pilot sites. In brief, testing at RMH was performed using the Coronavirus Typing (8-well) panel (AusDiagnostics, Mascot, Australia), as previously described [12] or the Xpert® Xpress SARS-CoV-2 assay (Cepheid, Sunnyvale, USA) [13] . Testing at Monash Health was performed using the Respiratory Pathogens 12-well assay (AusDiagnostics, Mascot, Australia) [14] or Xpert® Xpress SARS-CoV-2 assay; testing at Austin Health was performed using the Coronavirus Typing (8-well) panel (AusDiagnostics, Mascot, Australia) [14] or the Xpert® Xpress SARS-CoV-2 assay, and testing at MDU PHL was performed using the Aptima R SARS-CoV-2 assay and the Panther Fusion R SARS-CoV-2 assay (Hologic, Marlborough, Massachusetts, USA) according to manufacturer's instructions [15] .

A workflow was developed to plan for the event of a positive antigen test result; in that event, the swab collected for RT-PCR would be expedited using the Xpert® Xpress SARS-CoV-2 RT-PCR assay. The purpose of this strategy was to provide rapid confirmatory RT-PCR results to enable clinical and public health action during the validation phase.

The Abbott PanBio TM COVID-19 Ag test is a lateral flow immunoassay that detects the SARS-CoV-2 nucleocapsid protein in nasopharyngeal swabs. In this study, deep nasal swabs were collected, in accordance with national testing guidelines [16] . All swabs were collected by trained healthcare professionals and swabs were placed into the accompanying sample elution buffer and tested immediately after sample collection. Results were read by two independent observers and interpreted according to manufacturer's instructions; RT-PCR results were not available to observers. The result was recorded in a REDCap database as either "positive", "negative" or "invalid" and a photograph of the result was also uploaded. In addition, qualitative data were collected regarding the attitudes towards ease-of-use and potential management implications of the Abbott PanBio TM COVID-19 Ag test in clinical practice.

To assess the analytical sensitivity of the Abbott PanBio TM COVID-19 Ag test, the limit of detection (LoD) was determined using heat-inactivated SARS-CoV-2. A SARS-CoV-2 (VIC01) isolate from a patient in Melbourne [17] was grown in Vero cells, quantified based on infective dose (50% Tissue Culture Infectious Dose [TCID 50 ]) using the method of Reed and Muench [18] , then heat inactivated at 60 °C for 15 min. All processing was performed at Biosafety Level 3 in the Department of Microbiology and Immunology at the University of Melbourne. Serial dilutions were prepared in triplicate in saline and 50 μl spiked into the Abbott PanBio TM COVID-19 Ag test collection tubes containing sample elution buffer (supplied with the assay). Testing was performed as described by the manufacturer. Analytical sensitivity was quantified in TCID 50 per mL. All testing was performed in triplicate to give nine replicates in total for determination of the LoD. In addition, a total of twenty replicates were tested at 0.5x, 1x and 2x LoD to determine assay precision and 95% confidence limits for the LoD. The ability of the Abbott PanBio TM COVID-19 Ag test proprietary extraction buffer to inactivate SARS-CoV-2 was also assessed. To do this, 30 μL of undiluted stocks of SARS-CoV-2 (concentration 10 5.3 TCID 50 /mL) was spiked into 300 μl of proprietary extraction buffer, or infection media (Minimal Essential Media [MEM] supplemented with 10 μM HEPES, 2 mM glutamine and antibiotics). One aliquot of extraction buffer did not have virus added and was used as a control for cell cytotoxicity. The presence and quantity of infectious virus was assessed using a TCID 50 assay after 30 s, five minutes and ten minutes incubation, with all timepoints at room temperature, as previously described [19] .

Sensitivity, specificity and positive and negative predictive values (PPV and NPV) were calculated by comparing the results of the Abbott PanBio TM COVID-19 Ag test with RT-PCR. Where appropriate, results were reported with 95% confidence intervals. Statistical analyses were performed using SPSS statistical software package version 27 (SPSS Inc., Chicago, IL, USA), or GraphPad Prism, version 9.0 (San Diego, CA, USA).

Ethics review and study approval was provided by Monash Health Human Research and Ethics Committee (RES-20-0 0 0 0-678A) and local Governance approval was provided by Melbourne Health and Austin Health Offices for Research.

This work was funded by the Victorian Government, Victoria, Australia. The funder was not involved in data collection, analysis or manuscript preparation.

In total, 2,418 individuals underwent both antigen testing and RT-PCR testing at the hospital sites (899 at the RMH; 528 at Monash Health; 991 at Austin Health) and 26 as part of additional testing from patients with known COVID-19 ( Table 1 ; Supplementary Table 2 ). Five patients were excluded (two at RMH due to missing data during pre-pilot phase and one each at RMH, Monash Health and Austin Health with no RT-PCR available) leaving 2,413 for analysis. The median age of patients across the three clinical sites was 35 years (range 1 to 93 years), and 44.2% were male. 124 (5.1%) were asymptomatic, and of those who were symptomatic, the median symptom duration was 2 days (range 0 to 36 days) ( Table 1 ) .

Of the 2,413 individuals tested in hospitals (or associated screening clinics), no individuals tested positive using RT-PCR, and one individual tested positive using the Abbott PanBio TM COVID-19 Ag test, giving a clinical specificity of 99.96% (95% CI 99.73 -100%). This participant was asymptomatic and tested negative via RT-PCR (which was taken simultaneously). There was no interobserver variability reported and 81% of staff reported the test to be "very easy" or "easy" to use. Nearly 40% of staff reported that if the result was accurate, it would not result in a change in clinical management. This is likely because all symptomatic patients are advised to return to their residence and isolate until complete symptom resolution regardless of the result, although over one quarter of test users reported they would send the patient home with different advice (Supplementary Table 1) .

Of the 26 individuals with notified COVID-19, the time from symptom onset ranged from 1 to 33 days. Seventeen individuals tested positive using the Abbott PanBio TM COVID-19 Ag test and 22 tested positive using both the Aptima R SARS-CoV-2 assay and Table 2 ).

A framework was iteratively developed by investigators to enable implementation of antigen testing in a low prevalence setting (Supplementary Figure 2) . Specifically, this framework was designed to mitigate the clinical and public health impact of false positive and false negative results and ensure the safety of testing staff. Further, a list of key logistical and implementation challenges related to the use of antigen tests was developed by investigators ( Table 2 ).

Using serial dilutions of heat inactivated SARS-CoV-2, the analytical sensitivity of the Abbott PanBio TM COVID-19 Ag test was 250 TCID 50 (equivalent to 175 plaque forming units [pfu]/ml). In contrast, the analytical sensitivity of the Aptima and Panther Fusion SARS-CoV-2 assays were 0.01 TCID 50 (0.007 pfu/ml). The elution buffer was not shown to be completely viricidal following ten minutes of exposure at the concentrations examined, although there was a significant reduction in infectious SARS-CoV-2 titre (Supplementary Figure 3 ).

Here, we describe the implementation of a rapid antigen test for COVID-19 in a low prevalence setting in Melbourne, Australia. Although the relatively small number of cases limited our analysis of clinical sensitivity, we identified a range of logistical and implementation challenges that will inform future roll-outs of antigen testing, particularly in low-prevalence settings.

As expected, the analytical sensitivity of the antigen test was less than RT-PCR, with a detection limit of 175 pfu/mL, compared to 0.007 pfu/ml by RT-PCR. Our findings are similar to a recent study by Corman et al. that observed an analytical sensitivity of 88 pfu/mL when using SARS-CoV-2-infected cell culture supernatants with the Abbott PanBio TM COVID-19 Ag test [20] . Further, our findings are in keeping with the manufacturer's stated analytical sensitivity of 2.5 × 10 1.8 TCID 50 /mL of SARS-CoV-2 [21] . Importantly however, there is limited standardisation of protocols between studies; for example, cell culture methods may vary and the reference strain of virus may also differ between studies [22] . Accordingly, there is a clear need for agreed, standardised laboratory protocols to enable accurate comparison of sensitivity across different antigen test kits. Both the analytical and clinical (within 7 days of symptom onset) specificity of the antigen test were high (100 and 99.96%, respectively), in keeping with other studies [23] [24] [25] . However, even with a highly specific test, in a low prevalence setting, the majority of positive results are likely to represent false positives. As such, confirmatory testing of positive antigen test results by RT-PCR is critical. Similar to another recent study, we also found that the proprietary test buffer was not virucidal, even after ten minutes of exposure to the buffer [24] . Accordingly, appropriate biosafety measures should be in place when undertaking testing outside a laboratory setting. In our study, individuals collecting and performing the test wore personal protective equipment consisting of gown, gloves, a N95 respirator mask, and eye protection. It is likely that different settings implementing antigen testing will require specific risk assessments to minimise the risk of exposure to infectious virus.

In addition to assessing the performance characteristics of the test, we also worked through a range of practical challenges with test implementation. For example, to enable test tracking and 'traceability' a custom REDCap database was used to input patient data and upload photographs of each test as a record of each result. However, implementing this testing into public health responses at scale will require a more systematic and streamlined approach to data integration (e.g., the use of QR codes for test registration and tracking; cloud-based interfacing with laboratory information systems). Currently in Australia, there are regulatory restrictions for the use of POC COVID-19 testing such that antigen tests can only be supplied to accredited laboratories, medical practitioners, healthcare professionals working in residential and aged care facilities, or health departments [8] . Best practice guidelines for POC testing in Australia recommend training and competency assessments for staff performing this testing, in addition to an overarching quality framework to ensure appropriate quality control and quality assurance [26] . The supply of COVID-19 kits for self-testing at home is presently prohibited in Australia [8] . However, in countries that are currently experiencing a much higher disease prevalence than Australia, widespread deployment of lateral flow antigen kits is underway, including self-testing [27] . Importantly, outbreaks of COVID-19 can emerge and spread quickly, as evidenced by the rapid emergence of a 'second wave' in Melbourne between July and September [9] . As such, appropriate planning to overcome the logistical, governance, regulatory and implementation challenges we identified for scaled-up antigen testing is valuable preparation, especially in low-prevalence settings.

There were some limitations with our study. Most obviously, the study was conducted in a setting where disease prevalence was extremely low, although this experience is shared with other members of the Western Pacific region. Furthermore, in order to Table 2 Pre-analytical, analytical and post-analytical issues identified in this study to address in implementing SARS-CoV-2 antigen testing. 

Understand the performance on the test based on the clinical characteristics of the patient, e.g., duration of symptoms

Consider role of external quality assurance program and incorporate into quality management system

Ensure data is captured, accurate, confidential and stored securely (e.g., photograph result of lateral flow Ag assay)

Consider public health implications in the context of above, e.g., false negative result in aged care worker

Handle all specimens using appropriate PPE and consider testing as close to patient as feasible to reduce transport needs evaluate sensitivity in a low prevalence setting, all positive Abbott PanBio TM COVID-19 Ag tests were obtained from patients with known COVID-19, limiting a comprehensive appraisal of utility. Our study was conducted specifically in a hospital setting, predominantly amongst symptomatic individuals. Given the propensity for SARS-CoV-2 to spread in healthcare facilities [ 28 , 29 ] , RT-PCR remains the gold standard high sensitivity clinical diagnostic test in this context. However, antigen tests may have utility as a rapid screening tool in the hospital setting that can help triage symptomatic patients and improve patient flow while awaiting confirmatory RT-PCR testing. Further work is required to establish the optimal framework for antigen testing as a surveillance tool in the community setting, particularly in areas with a low disease prevalence and in asymptomatic individuals. In summary, we describe the validation and implementation of a widely available antigen test in a low prevalence setting. We identified several practical challenges to scaling up this testing, mostly related to pre-and post-analytical stages of testing. Our findings will help inform the responsible use of antigen tests in other low-prevalence countries.

Writing -review & editing. Fiona James: Data curation, Investigation, Resources, Supervision, Writing -review & editing. Emma Gardiner: Data curation, Investigation, Resources, Writing -review & editing. Socheata Chea: Formal analysis, Validation, Writing -review & editing. Nicole Isles: Data curation

Writing -review & editing. George Taiaroa: Formal analysis, Methodology, Validation, Writing -review & editing

Epidemiology and clinical features of COVID-19: a review of current literature

Association of public health interventions with the epidemiology of the COVID-19 outbreak in Wuhan, China

Impact of delays on effectiveness of contact tracing strategies for COVID-19: a modelling study

Rethinking COVID-19 test sensitivity -A strategy for containment

Test sensitivity is secondary to frequency and turnaround time for COVID-19 surveillance

Modelling transmission and control of the COVID-19 pandemic in Australia

World Health Organisation Emergency Use Listing for in Vitro Diagnostics (IVDs) Detecting SARS-CoV-2

Australian Government Therapeutic Goods Administration COVID-19 Test Kits Included in the ARTG for Legal Supply in Australia

COVID-19) in Australia

Victorian Department of Health and Human Services Victorian COVID-19 Data

Research Electronic Data Capture (REDCap) -a metadata-driven methodology and workflow process for providing translational research informatics support

Implementation and evaluation of a novel real-time multiplex assay for SARS-CoV-2: in-field learnings from a clinical microbiology laboratory

Test Resources

Clinical evaluation of AusDiagnostics SARS-CoV-2 multiplex tandem PCR assay

Hologic SARS-CoV-2 Assays

Public Health Laboratory Network PHLN guidance on laboratory testing for SARS-CoV-2 (the virus that causes COVID-19)

Isolation and rapid sharing of the 2019 novel coronavirus (SARS-CoV-2) from the first patient diagnosed with COVID-19 in Australia

A simple method of estimating fifty per cent endpoints

Validation of a single-step, single-tube reverse transcription loop-mediated isothermal amplification assay for rapid detection of SARS-CoV-2 RNA

Comparison of seven commercial SARS-CoV-2 rapid point-of-care antigen tests

Abbott PanBioTM COVID-19 Ag rapid test device Instructions for use

Laboratory diagnosis of SARS-CoV-2-A review of current methods

Detection of 2019 novel coronavirus (2019-nCoV) by real-time RT-PCR

Real-life validation of the Panbio COVID-19 antigen rapid test (Abbott) in community-dwelling subjects with symptoms of potential SARS-CoV-2 infection

Validation report: SARS-CoV-2 antigen rapid diagnostic test

Australian Government Department of Health Guidelines for point of care testing (PoCT) First Edition

United Kingdom National Health Service A guide for healthcare staff self-testing for coronavirus using a Lateral Flow Device (LFD)

A hospital-wide response to multiple outbreaks of COVID-19 in health care workers: lessons learned from the field

Containment of a healthcare-associated COVID-19 outbreak in a university hospital in Seoul, Korea: a single-center experience

SM and KB are supported by Postgraduate Scholarships from the National Health and Medical Research Council (NHMRC) of Australia ( GNT1191368 and GNT1191321 ). KG and JCK are supported by an Early Career Fellowship Grants from the National Health and Medical Research Council (NHMRC) of Australia ( GNT1120816 , GNT1142613 ). BPH is supported by a Practitioner Fellowship from the National Health and Medical Research Council (NHMRC) of Australia ( APP1105905 ). DAW is supported by an Investigator Grant from the National Health and Medical Research Council (NHMRC) of Australia ( APP1174555 ). We thank the Victorian Infectious Diseases Reference Laboratory for providing reference material. We also thank study staff at each hospital for collection of swabs from patients.

All authors confirm that they had full access to all the data in the study and accept responsibility to submit for publication. Deidentified data is available from the time of publication and available for five years following article publication. Requests should be directed to the corresponding author.

Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.lanwpc.2021.100115 .