key: cord-0906272-l07bfzgn authors: Rahman, H.; Carter, I.; Basile, K.; Donovan, L.; Kumar, S.; Tran, T.; Ko, D.; Alderson, S.; Sivaruban, T.; Eden, J.-S.; Rockett, R.; O’Sullivan, M. V.; Sintchenko, V.; Chen, S.C-A.; Maddocks, S.; Dwyer, D. E.; Kok, J. title: Interpret with caution: An evaluation of the commercial AusDiagnostics versus in-house developed assays for the detection of SARS-CoV-2 virus date: 2020-06-30 journal: Journal of Clinical Virology DOI: 10.1016/j.jcv.2020.104374 sha: b299ca7945f2a6a41300cf6a722a35d6fea4629c doc_id: 906272 cord_uid: l07bfzgn Abstract Introduction There is limited data on the analytical performance of commercial nucleic acid tests (NATs) for laboratory confirmation of COVID-19 infection. Methods Nasopharyngeal, combined nose and throat swabs, nasopharyngeal aspirates and sputum was collected from persons with suspected SARS-CoV-2 infection, serial dilutions of SARS-CoV-2 viral cultures and synthetic positive controls (gBlocks, Integrated DNA Technologies) were tested using i) AusDiagnostics assay (AusDiagnostics Pty Ltd); ii) in-house developed assays targeting the E and RdRp genes; iii) multiplex PCR assay targeting endemic respiratory viruses. Discrepant SARS-CoV-2 results were resolved by testing the N, ORF1b, ORF1ab and M genes. Results Of 52 clinical samples collected from 50 persons tested, respiratory viruses were detected in 22 samples (42 %), including SARS CoV-2 (n = 5), rhinovirus (n = 7), enterovirus (n = 5), influenza B (n = 4), hMPV (n = 5), influenza A (n = 2), PIV-2 (n = 1), RSV (n = 2), CoV-NL63 (n = 1) and CoV-229E (n = 1). SARS-CoV-2 was detected in four additional samples by the AusDiagnostics assay. Using the in-house assays as the "gold standard", the sensitivity, specificity, positive and negative predictive values of the AusDiagnostics assay was 100 %, 92.16 %, 55.56 % and 100 % respectively. The Ct values of the real-time in-house-developed PCR assay targeting the E gene was significantly lower than the corresponding RdRp gene assay when applied to clinical samples, viral culture and positive controls (mean 21.75 vs 28.1, p = 0.0031). Conclusions The AusDiagnostics assay is not specific for the detection SARS-CoV-2. Any positive results should be confirmed using another NAT or sequencing. The case definition used to investigate persons with suspected COVID-19 infection is not specific. On 31 December 2019, the Chinese Center for Disease Control and Prevention reported to the World Health Organization (WHO) a series of patients with pneumonia of uncertain aetiology in Wuhan city, Hubei province, China [1] . The pathogen responsible for this outbreak was subsequently identified as a novel group 2B betacoronavirus, designated as Severe acute respiratory syndrome coronavirus-2 (SARS CoV-2). It shares approximately79 % homology to the Severe Acute with New South Wales accounting for 2734 infections. (Department of Health, Australia). Expeditious and accurate laboratory diagnosis of persons with COVID-19 infection followed by appropriate infection control measures is key to preventing further spread of infection, particularly in the absence of effective antiviral therapy. The genetic sequence of SARS-CoV-2 was released on 10 January 2020, and the WHO subsequently recommended several assays for the detection of SARS-CoV-2 [3] . Within NSW, the Centre for Infectious Diseases and Microbiology Laboratory Services, in the NSW Health Pathology-Institute of Clinical Pathology and Medical Research, was designated as the public health laboratory responsible for specific testing of SARS-CoV-2 during the earliest phase of the outbreak. During this initial phase, commercial assays were not available in Australia. However, on 31 January 2020, a commercial assay for the specific detection of SARS-CoV-2 (AusDiagnostics Pty Ltd, Mascot, NSW, Australia) was announced [4] . We evaluated this assay's performance against real-time PCR (RT-PCR) assays using SARS-CoV-2 gene targets recommended by the WHO [5] . Samples were collected from persons with suspected SARS-CoV-2 infection according to the case definition outlined in the Communicable Diseases Network Australia (CDNA) National Guidelines for Public Health Units for managing COVID-19. A suspect case was defined as meeting both clinical criteria, fever ≥ 38°C or history of fever or acute respiratory infection (e.g. cough, shortness of breath, sore throat) and epidemiology criteria which includes international travel in the 14 days prior to symptom onset to a country known to have cases of COVID-19 or close contact with a confirmed or probable case. This case definition continues to be updated and is (available at: https://www1.health.gov. au/internet/main/publishing.nsf/Content/cdna-song-novelcoronavirus.htm). Upper respiratory tract (URT) samples [including nasopharyngeal swabs (NPS), combined nose and throat swabs (NTS) and nasopharyngeal aspirates (NPA)] were collected under appropriate infection control measures, placed in 1−3 mL of viral or universal transport media, transferred to the laboratory and tested within 12 h of specimen receipt. As aerosol generating procedures (collection of induced sputum, tracheal aspirates, or bronchoalveolar lavage) were not performed in patients not requiring invasive ventilatory support, sputum was the only lower respiratory tract (LRT) sample available for testing. Serial sampling of the URT and/or LRT was performed in patients with confirmed SARS-CoV-2 infection. The specimen types, collection, and processing used in this study were in accordance with those recommended by the AusDiagnostics Assay package insert. Viral cultures using Vero E6 cells were performed on samples where SARS-CoV-2 nucleic acid was detected. Cells were inspected daily for cytopathic effects, and when observed, this was confirmed by nucleic acid testing (NAT) of cell culture supernatant, using the ORF1b target. Serial dilutions of a patient's SARS-CoV-2 cell culture supernatant (neat to 10 −7 ) and synthetic positive control (neat and 10 -9 , gBlocks gene fragments covering E, RdRp and N genes, Integrated DNA Technologies, Coralville, Iowa, USA) were also tested by NAT as outlined below. RNA extraction was performed using the BioRobot EZ1 and EZ1 Virus Mini Kit v2.0 (Qiagen, Valencia, CA, USA) according to the manufacturers' instructions, using 200 u L of specimen. Nucleic acid amplification and detection was then performed using the methods outlined below. From January 22 to January 31, 2020, SARS-CoV-2 was detected using an in-house developed gel-based NAT targeting the E gene as described by Corman et al. [6] as primers but not probes were available. From 1 February 2020 onwards, SARS-CoV-2 detection was performed using RT-PCR assays targeting both the E and RdRp genes. The RdRp gene P2 probe is specific for SARS-CoV-2 and does not detect SARS-CoV [7] . Specimens collected prior to 31 January 2020 previously tested using the gel-based NAT were retrospectively tested using the same RT-PCR assays. Samples were also tested using the AusDiagnostics assay (Coronavirus Typing version 01) a research use only assay, which is a RT-PCR assay that distinguishes between SARS-CoV-2 and other endemic coronaviruses (HKU-1, OC43, 229E and NL63), SARS-CoV and MERS-CoV. Data analysis is performed using proprietary software (RealTime_PCR v7.7) with positive, negative or inhibited results provided following interpretation. Typically, a positive result satisfies predefined criterion for cycle threshold (Ct) and melting temperature (Tm) for the amplified gene target. Where discrepant AusDiagnostics assay results were encountered (relative to the in-house assays targeting the E and/or RdRp genes), the cycling and melt curves of each sample were manually examined, followed by further testing of four other WHO recommended targets (the N, ORF1b, ORF1ab and M genes) [5] for discrepant analysis, with the definitive result determined by the consensus of the assays. In addition to SARS-CoV-2-specific testing, all samples were also tested using an in-house multiplex respiratory virus panel, which detects influenza A, influenza B, respiratory syncytial virus [RSV], parainfluenza viruses [PIV] 1−3, human adenovirus and human metapneumovirus [hMPV], as previously described [8] .The primers and probes, mastermix composition and cycling conditions used in the inhouse assays specifically detecting SARS-CoV-2 are outlined in Table 4 . Three full-length sequences of the SARS-CoV-2 genome from clinical samples and/or viral cultures were obtained by amplicon-based Illumina sequencing (with and without enrichment using the Illumina Pan Viral Kit), and submitted to GISAID, accession IDs: EPI_ISL_407893, 408976 and 408977. The median age of the persons under investigation was 31.5 years of age (range 0-84) and 30 were males (58 %). Samples were collected from the URT and LRT (sputa) from adults. Nine of 50 (18 %) persons were children (< 16 years old), with six providing NTS or NPS, and three providing NPA. Of the 52 specimens tested, respiratory viruses were detected in 22 (42 %) samples, including SARS-CoV-2 (n = 5), rhinovirus (n = 7), enterovirus (n = 5), influenza B (n = 4), hMPV (n = 5), influenza A (n = 2), PIV-2 (n = 1), RSV (n = 2), CoV-NL63 (n = 1) and CoV-229E (n = 1). Coinfections were detected in four samples (one sample each of SARS-CoV-2 plus enterovirus, rhinovirus plus PIV-2, RSV plus rhinovirus, and rhinovirus plus enterovirus). SARS−COV-2 was detected in nine samples by the AusDiagnostics assay, but this was only confirmed by in-house assays targeting E, RdRp genes in five samples. In the remaining four samples, the E, RdRp, N, ORF1b, ORF1ab and M gene results were negative, and these were deemed as false positive results. Using the in-house assays as the "gold standard", the sensitivity, specificity, positive and negative predictive values of the AusDiagnostics assay 100 %, 92.16 %, 55.56 % and 100 % respectively. Further interrogation of the cycling and melt curves of the false positive samples revealed a relatively flat, non-sigmoidal amplification curve (Fig. 1) . By manual interpretation, the results of such amplification curves would have been called negative, despite the melt curves suggesting a positive test result. Of note, no other coronaviruses (HKU-1, OC43, 229E, NL63, SARS-CoV and MERS-CoV) were detected by the AusDiagnostics assay in these four samples. However, the AusDiagnostics assay detected other coronaviruses (NL63 and 229E) in two other samples. The results of testing by the in-house and AusDiagnostics assays are shown in Tables 1-3. In all clinical samples and serial dilutions of a SARS-CoV-2 culture isolate and gBlock positive synthetic controls tested, the Ct values of the in-house RT-PCR assay targeting the E Gene was significantly lower than the corresponding RdRp gene assay (mean 21.75 vs 28.10, p = 0.0031 by Mann Whitney test) (Fig. 2) . Our experience highlights important considerations for laboratory preparedness in the early phase of the COVID-19 epidemic. Laboratory confirmation of SARS CoV-2 infection is important for individual patient care and public health management to limit the spread of infection by quarantine (which may include their close contacts). In this regard, the clinical laboratory is vital in providing accurate and timely diagnostics to confirm or exclude infection. As SARS-CoV-2 was rapidly identified as the causative pathogen of the cases of pneumonia of unknown aetiology in Wuhan, laboratories had to quickly develop and evaluate the analytical performance of diagnostic NATs. Several assays targeting different regions of the SARS-CoV-2 genome has been proposed [3] , but there are limited data on the analytical performance of these assays. In the early phases of the outbreak, no commercial assays were available for testing within Australia. Assays detecting SARS-CoV-2, whether in-house developed or commercial, should be thoroughly evaluated to ensure they are fit for purpose. The initial NAT in each laboratory may also be limited by the availability of testing reagents, as evidenced in our laboratory initially using a gel-based assay targeting the E gene in the absence of probes. Following probe procurement, the NAT was changed to RT-PCR assays and initial samples tested using gel-based assay were re-tested using RT-PCR. This is the preferred method as viral loads (and hence viral kinetics) may be determined in a semi-quantitative fashion by correlating the Ct values of positive tests. In the present study, the Ct values of the E gene assay were consistently lower than the corresponding values in the RdRp assay in SARS-CoV-2 positive clinical samples, serial dilutions of SARS-CoV-2 cultures and synthetic positive controls, highlighting the superior sensitivity of the E gene assay as a screening test. In the early phases of the outbreak, it was suggested that laboratories use a pan-coronavirus NAT followed by sequencing of amplicons from non-conserved regions for characterization and confirmation of SARS-CoV-2, or amplification and detection of SARS-CoV-2 specific sequences without further sequencing [3] . The use of sequencing (although valuable for confirmation of NAT) in the diagnostic algorithm increases laboratory turnaround times and may not be ideal when there are substantial numbers of specimens that require testing. Furthermore, although sequencing may be used to confirm the diagnostic accuracy of newly developed tests for novel pathogens, this technology may not be readily available in all diagnostic laboratories. The use of sequencing Take-off cycle E-gene Ct RdRp-gene Ct 36 F 0 NTS ND ND ND ND 11 F 0 NTS ND ND ND ND 61 F 0 NPS Influenza B ND ND ND 30 M 5 NPA ND ND ND ND 4 M 2 NPA ND ND ND ND 32 F 2 NPS ND ND ND ND 33 M 5 NPA Influenza B ND ND ND 28 M 11 NTS ND ND ND ND 21 F 8 NPS ND ND ND ND can assist with investigation of cases with no clear links at a local level and sharing of sequencing data to platforms such as GISAID can contribute to the understanding of the viral evolution. The optimal type of respiratory sample that should be collected to confirm or exclude SARS-CoV-2 needs to be determined. In our study, four patients were diagnosed with SARS-CoV-2 infection. One patient had SARS-CoV-2 detectable from his LRT (sputum) but not URT sample on day eight of infection following symptom onset. This discrepant result may reflect differences in sampling quality or progression of infection from the URT to the LRT, and supports the recommendations to test LRT samples where available [PHLN] , similar to our previous experience with A(H1N1)pdm09 infection [9] . The detection of SARS-CoV-2 in LRT samples is suggestive of active viral replication within the LRT, particularly in patients with deteriorating respiratory function and abnormal radiological imaging. Together with Ct values, this may be useful in guiding the clinical management of such patients. The significance of viral co-infection in COVID-19 disease on clinical outcomes is unknown. Other samples where SARS-CoV-2 have been detected by NATs include saliva, stool, rectal swabs and urine [CCDC, To] . This study has several limitations. Non-respiratory tract samples were unavailable for testing. There is emerging evidence that there may be prolonged viral shedding in stools from persons with COVID-19 infection, similar to those infected with SARS-CoV [10, 11] . The specific gene target(s) of the AusDiagnostics assay are not known, so we were not able to determine the exact reason for the reduced analytical specificity. The respiratory multiplex assay used in this study is an in-house assay that does not include other endemic coronaviruses such as NL-63, OC43, 229E and HKU-1 so we were unable to verify the results of non-SARS-CoV-2 coronaviruses tested using the AusDiagnostics assay. In conclusion, we determined that the commercial AusDiagnostics assay was less reliable than an in-house RT-PCR using WHO recommended gene targets for the detection of SARS-CoV-2. Assay validation and verification are required to ensure that commercial assays used to detect SARS-CoV-2 are fit for purpose. Further data are awaited regarding the use of assays with different SARS-CoV-2 targets, viral loads and kinetics to better guide individual patient management and infection control measures. The optimal testing algorithm to detect SARS-CoV-2 for diagnostic pathology providers and public health reference laboratories will depend on their healthcare systems, laboratory capacity and capability, particularly with the anticipated substantial increase in testing demands as the epidemic progresses and revised case definitions broaden the indications for testing. The kits for the AusDiagnostics assay were supplied to the laboratory by AusDiagnostics Pty Ltd free of charge, no funding was provided for this study. All work including study design, execution, analysis, write up and editing was undertaken independently by the authors listed. Formal analysis, Resources, Data curation, Project administration. K. Basile: Validation, Formal analysis, Data curation, Writing -original draft, Writing -review & editing A novel coronavirus from patients with pneumonia in China Genomic characterisation and epidemiology of 2019 novel coronavirus: implications for virus origins and receptor binding Novel coronavirus (SARS-CoV-2) technical guidance: Laboratory testing for SARS-CoV-2 in humans World Health Organization Detection of 2019 novel coronavirus (SARS-CoV-2) by real time RT-PCR Diagnostic Detection of SARS-CoV-2 by Real-time RT-PCR Pandemic clinical case definitions are non-specific: multiple respiratory viruses circulating in the early phases of the 2009 influenza pandemic in New South Wales, Australia Rapid-test sensitivity for novel swine-origin influenza A H1N1 virus in humans Molecular and serological investigation of 2019-nCoV infected patients: implication of multiple shedding routes Persistent shedding of viable SARS-CoV in urine and stool of SARS patients during the convalescent phase This study was supported by the Prevention Research Support Program funded by the New South Wales Ministry of Health and the NHMRC Centre for Research Excellence in Emerging Infectious Diseases and the Australian Partnership for Preparedness Research on Infectious Disease Emergencies.