key: cord-310140-h7uwl0pb
authors: Templeton, K.E.; Forde, C.B.; Loon, A.M. van; Claas, E.C.J.; Niesters, H.G.M.; Wallace, P.; Carman, W.F.
title: A multi-centre pilot proficiency programme to assess the quality of molecular detection of respiratory viruses
date: 2005-07-12
journal: J Clin Virol
DOI: 10.1016/j.jcv.2005.05.005
sha: 
doc_id: 310140
cord_uid: h7uwl0pb

OBJECTIVES: To assess the quality of molecular detection of respiratory viruses in clinical diagnostic laboratories. STUDY DESIGN: Respiratory virus proficiency panels were produced from diluted stocks of respiratory viruses provided and tested by four reference laboratories. The panels consisted of strong positive, positive, low positive and negative samples for influenza viruses A and B, respiratory syncytial virus, parainfluenza viruses 1 and 3, adenovirus serotypes 4 and 7, human rhinovirus serotypes 16, 72 and 90, human coronaviruses OC43 and 229E. The panels were sent to 17 participants; results and information on methodology was collected. RESULTS: All laboratories returned results, of which five submitted complete data sets. So, for analysis all results were combined. Samples were correctly identified by participants in 93.75%, 76.75% and 47.03% for the high positive, positive and low positive samples, respectively. One false positive was reported for all data sets (1.1%). The overall score for all assays using different methodologies was 78.8%. Laboratory performance was not dependant on methodology as all in-house methodologies could achieve optimal results, but dependant on careful optimisation and procedures specific to the laboratory. CONCLUSIONS: The first proficiency panel showed that in general all participants performed well. Although, it also highlights areas for improvement for all participants in order to generate robust results for use in clinical diagnostics.

Detection of respiratory viruses in clinical samples is important for effective patient management and infection control. Many viruses are involved in respiratory infection and include: influenza viruses A and B, respiratory syncytial virus (RSV), parainfluenza viruses (PIV) 1-4, adenoviruses (ADV), human rhinoviruses (hRV), human coronaviruses (hCoV: OC43, 229E, NL-63, NL-Hong Kong and severe acute respiratory syndrome coronavirus (SARS-CoV)) and human metapneumoviruses (hMPV). Many of the infections are indistinguishable by clinical features alone and require rapid laboratory investigation for optimal patient management and infection control. Detection of respiratory viruses is becoming clinically important as the possibilities for antiviral treatment increase. Some of the viruses (hRVs and hCoVs) were initially considered as causing mild infections but are now more frequently found to be associated with severe infections (Ison et al., 2003; van Elden et al., 2004) . Viral culture is the gold standard for laboratory diagnosis of respiratory viruses. However, some viruses grow poorly in cell culture, and therefore routine diagnosis is sub-optimal as, for example, with hMPV (van den Hoogen et al., 2004) . Furthermore, culture is relatively slow which affects the clinical value, and therefore alternatives are employed. Rapid antigen detection tests are available for influenza viruses A and B, PIV 1-3 and RSV and adenoviruses, which are widely used in routine laboratories. These have been shown to be less sensitive and less specific than culture, particularly in adult or elderly populations (Casiano-Colon et al., 2003; Effler et al., 2002; Storch, 2003) . Amplification assays are now increasingly being used in the clinical laboratory and have proven to be more sensitive and specific than culture (Syrmis et al., 2004; van Elden et al., 2002; Weinberg et al., 2002 Weinberg et al., , 2004 . Newer real-time PCR formats also enable rapid test results (Templeton et al., 2004) while some conventional formats can detect a large number of respiratory viruses in a single test (Puppe et al., 2004; Coiras et al., 2004; Gruteke et al., 2004) PCR is often the most appropriate means to diagnose slow growing viruses (van Elden et al., 2004; van den Hoogen et al., 2004) due to the difficulties sometimes observed with culture.

Currently, some commercial assays are available for respiratory virus testing (Henrickson, 2004) but the majority of assays applied in clinical diagnostic laboratories have been developed in-house and standardisation is problematic for all formats. Laboratories performing respiratory molecular tests want to report accurate and reliable results regardless of the type of assay in use and one of the best ways to assess performance is to participate in proficiency programmes, enabling laboratories to evaluate their performance (Schirm et al., 2002; Schloss et al., 2003; Noordhoek et al., 2004; Verkooyen et al., 2003) .

A pilot study for the evaluation of nucleic acid amplification technologies (NATs) for detection of respiratory viral pathogens was organised by Quality Control for Molecular Diagnostics (QCMD) (www.qcmd.org). The aim of the pilot programme was to perform a comparative study of the current NATs and protocols used for the molecular detection of a variety of respiratory viruses. The aim of this study was to focus on the sensitivity and specificity of the NAT protocols used, as well as the methodology employed for molecular testing.

A project group comprising QCMD and four European laboratories was established to co-ordinate and produce materials for the proficiency programme panel. Each laboratory was responsible for the production of bulk stocks for a number of defined viruses: Specialist Centre for Virology (SVC), Gartnavel General Hospital, Glasgow, UK (influenza viruses A and B, RSV A); University Medical Centre Utrecht (UMCU), The Netherlands (hRV 16, 72 and 90); Erasmus Medical Center (EMC), Rotterdam, The Netherlands (229E and OC43, hMPV); Leiden University Medical Center (LUMC), Leiden The Netherlands (PIV 1 and 3, ADV 4 and 7).

At SVC stocks of influenza A virus (A/New Caledonia/20/99 (H1N1)) and influenza B virus (B/Victoria/504/00) were prepared on MDCK cells in Medium 199. A RSV A clinical isolate (03.413667) was cultured on HEp-2 cells in MEM (+2%, v/v, FCS) . Viral nucleic acid extraction was carried out using the BioRobot Blood DNA kit (Qiagen, Crawley, UK). Real-time PCR with TaqMan probes was used to amplify the influenza virus A NS gene, influenza virus B matrix gene and the RSV nucleocapsid gene on the i-Cycler IQ real-time detection system (Biorad, Hemel Hempstead, UK).

Clinical isolates of ADV 4 and 7 were cultured on A549 cells in MEM (+10%, v/v, FCS) and the subtype was confirmed by serotyping and sequence analysis. Clinical isolates of PIV 1 and 3 were cultured on LLC-MK2 cells and the subtype was confirmed by monoclonal antibody staining and sequence analysis. Virus nucleic acid extraction was carried out with QIAamp Blood DNA mini spin protocol for ADV and the QIAamp viral RNA Mini spin protocol for PIV (Qiagen, Hilden, Germany). Real-time PCR with molecular beacons were used to amplify the ADV hexon gene and the PIV haemagluttinin-neuramididase gene on the i-Cycler IQ real-time detection system (Biorad, Veenendaal, The Netherlands).

ATCC strains of 229E and OC43 and the hMPV isolate 00-1 (van den Hoogen et al., 2001) were cultured on Vero cells in MEM (+10%, v/v, FCS). Virus nucleic acid extraction was carried out using the MagNAPure LC total nucleic acid kit (Roche Applied Science, Almere, The Netherlands). Realtime PCR was performed with TaqMan probes to amplify and detect the nucleocapsid gene of OC43 and 229E and the nucleoprotein of hMPV on the ABI Prism 7700 sequence detection system (Applied Biosystems, Nieuwerkerk aan de IJssel, The Netherlands).

The human rhinoviruses (hRV) were donated by Dr. Tapani Hovi (KTL, Helsinki, Finland) and were originally obtained from ATCC, hRV type 16), RIVM (Bilthoven, The Netherlands, hRV type 72) and Janssen Pharmaceutica (Beerse, Belgium, hRV type 90). They were grown on Hela Ohio cells in roller tube cultures at 33 • C, and characterised by acid-lability testing, serotyping and sequence analysis. For production of stocks for the pilot proficiency panel, viruses were grown in human diploid fibroblast cells. Virus nucleic acid extraction was carried out using the MagNAPure LC total nucleic acid kit and real-time PCR was performed on the ABI Prism 7700 sequence detection system with Taq-Man probes to amplify the 5 non-coding region (NCR) of rhinoviruses.

Each laboratory prepared a small-scale dilution series from their bulk stocks and determined the lower limit of detection (LLD) for each virus stock, based on real-time cycle threshold (C t ) values (Table 1) . This process was repeated by a second laboratory. Thereafter, the project group selected three dilutions of each virus for inclusion in the proficiency panel. Bulk dilution stocks were then prepared in virus transport medium (VTM) except for influenza virus B, which was prepared in Medium 199 (+10%, v/v, FCS). The bulk dilution stocks were repeat tested using the same methodology (data not shown), dispensed into 1 ml aliquots and shipped on dry ice to QCMD for coding, packaging and distribution to participants.

The panel samples were randomised by QCMD, labelled, packed and distributed to participants on dry ice along with a panel receipt form, an instruction leaflet, a reporting form and a questionnaire for technical details. The panel was distributed in the form of six sub-panels: Panel A: containing influenza viruses A and B; Panel B: PIV 1 and 3; Panel C: RSV A and hMPV; Panel D: hRV 16, 72 and 90; Panel E: ADV 4 and 7; Panel F: hCoV 229E and OC43. Each sub-panel contained a three-member dilution series for each virus, a mixed sample (except for Panel C) and a negative sample.

Laboratories who had expressed an interest to QCMD in participating in a proficiency programme for molecular detection of respiratory viruses were asked to complete a questionnaire detailing technical aspects of the assays they had applied. Each participating laboratory was assigned a unique, confidential code. Participants were requested to return the result form and technical questionnaire, either by fax or by e-mail, before the closing date for return of the results, which was 6 weeks post-distribution. The majority of participants did submit results prior to the closing date, but as a number of participants requested additional time to complete testing, the date for return of results was extended to 10 weeks. Results and questionnaire data was collated and validated by QCMD and participants were recontacted, if further information was required. Result codes letters were sent to all participants after all results had been received.

Results were scored: 2 points for a correct result, 0 for an incorrect and 1 for an equivocal result on a positive sample or for detection of only one sample in a mixed sample in the influenza viruses A and B, PIV 1 and 3 and Coronavirus 229E and OC43 panels. The mixed samples in the adenovirus and rhinovirus panels were not scored, as they required typing results and this was not requested.

The results of the molecular testing of the samples in the proficiency panel are shown in Table 2 . Three dilutions of each of the 13 different viruses were included in the QC panel. Five of the 17 (29%) participants provided complete data sets on all viruses and none of these five reported 100% correct. The high positive sample was on average correctly identified by 93.8% (range 72.7-100%) by the participants. The mean Only 2 of 11 participants achieved a 100% score for the hRV panel. Three participants did not detect any of the hRV 72 samples, although they could detect the other rhinovirus types 16 and 90, even in the low positive. Three participants who detected hRV 72 as the low positive did not detect both hRV 16 and 90 in the low positive. Four of the 14 participants scored 100% in the adenovirus panel. All participants detected the strong positive adenovirus 4 (subtype E) sam-ple and 10 out of the 14 participants detected the low positive sample. In contrast, only 12 out of the 14 participants detected the strong positive adenovirus 7 (subtype B) sample and 8 out of the 14 participants detected the low positive adenovirus 7 sample. In the hCoV panel, all participants detected the strong positive samples but none of the participants detected the low positive sample for OC43 in contrast, four out of eight participants detected the low positive for 229E.

The performance of the 17 laboratories is shown in Table 3 . The mean overall score was 78.8% with a range from 46.4% to 96.9%. Ninety-four assays were used for all different targets of which 42 (44.5%), 27 (29%) and 25 (26.5%) were realtime PCR, nested PCR and other methodologies, respectively. Six participants used a combination of methodologies. The scores obtained for each of the individual panels are shown in Table 3 . The hMPV panel was performed best with 12 out Methodology used as indicated by integers in superscripts: 1 real-time PCR; 2 nested PCR; 3 nucleic acid sequence amplification (NASBA); 4 single PCR; 5 multiplex reverse line blot (PCR amplification with detection by reverse line blot); 6 RV ® Chip assay (manufactured by BCS Biotech SPA, Cagliari, Italy); 7 only assay for 229E performed, nt = not tested, ns = not scored. of 14 participants scoring 100% whereas the PIV panel had only one participant scoring 100% and 10 out of 15 scoring <70%. Six participants performed real-time PCR, as the only NAT technology, and their overall rank was 2, 4, 5, 7, 13 and 17. The only commercially available assay was the RV ® Chip assay and this had an overall performance of 57.1%. The other assays were all in-house protocols and consist of many different parameters and technical aspects. The correlation between these aspects and a 100% score is given in Table 4 and this shows that a maximum score in the panel could be achieved with any combination of methodology and protocol.

The different genes targeted in the NAT assay are shown in Table 5 . Four out of the five NATs targeted to the matrix protein of influenza virus detected the weak positive sample, whereas only two of the five non-structural protein NATs detected the low positive sample. The NAT assays for the HN protein of PIV detected the weak positive samples, whereas the NAT targeting the nucleoprotein and fusion protein only detected the strong positive sample.

The study described here is the first large-scale external quality assessment scheme for the molecular detection of respiratory viruses. Application of NATs for respiratory virus diagnosis is increasing as it is recognised as the best method for rapid, sensitive and specific diagnostic results (van Elden et al., 2002) . Sixteen of the 17 participants in this study have developed their own 'in-house' NAT assays using a large variety of NAT protocols. In order for laboratories to assess the quality of these in-house assays an external QA programme has been developed. The performance in external QA panels is an integral part in accreditation and quality management of clinical diagnostic laboratories.

The preparation of a suitable panel that is reproducible and of a high quality to assess the performance of the NATs is demanding. In this study, samples were grown in cell culture and dilutions were made so sensitivity and limited specificity of assays for these viruses could be assessed. The panels were assessed as a small-scale dilution series prior to distribution in a second laboratory to determine the detection limit after a freeze-thaw step. Thereafter, the same stock viruses were used to prepare the final panels. These samples were not assessed by more than one reference centre prior to distribution so there have been some differences between the initial small-scale dilution series and the bulk panels sent to all participants. The lowest positive dilution used for OC43 was detected by none of the participants, so this dilution was too low for inclusion in further panels. However, for many of the panels, a number of the NATs used have detected all dilutions of the viruses in the panels, showing that the panels employed gave a good indication of NAT performance. In the future, the panel could be produced by freeze-drying the samples and include more pre-distribution testing to ensure more reproducible panels, as employed in other QCMD schemes (Schirm et al., 2002; Schloss et al., 2003) . The panels employed in this distribution used samples that reflected the limit of detection according to NATs used in two reference laboratories. The clinical relevance of these detection levels was not assessed; as it is not known what detection limit is required to detect clinically relevant respiratory virus infections. The limit of detection employed in these panels probably reflects a good guide to assess the sensitivity of NATs. At present, there is no international reference material for respiratory virus NAT's and these panels can provide the basis for new internationally recognised stocks.

One false positive was detected in the negative sample in the HRV panel. No false positives were detected in any other panels, giving a 1.1% false positive rate in the whole distribution. This shows a high level of technical skill in the participating laboratories. In other QA programmes in which many 'in-house' methods were used, the false positivity rates have been much higher (Schloss et al., 2003) . The reduction of the false positivity rate may also be as a result of the use of real-time PCR, which reduces the chance of contamination.

In general, correct results were obtained with a mean of 93.8% of the strong positive samples, with participants failing to detect a mean of 23.2% and 53.0% of the positive and low positives, respectively. The false negative results observed were either results of samples below the detection limit of the assays, or of primer and probes not detecting the viruses at all. The selection of primers and probes for respiratory viruses is essential as a single NAT is employed for a specific virus that needs to detect many variants. Adenoviruses comprise of six different serotypes that show large sequence variation and RNA viruses show even greater sequence variation, as they lack proof reading capability. The virus included in the hMPV panel was a very well characterised (van den Hoogen et al., 2001) , and therefore this information would have been used to generate NATs whereas the PIVs were clinical isolates with no publicly available sequence information. This knowledge of sequence information would affect the design of the NATs, and hence one of the reasons why the hMPV panel had good performance scores and the PIV poor.

There were some assays that did not detect a particular virus at all, e.g. PIV 1 and RSV while other NAT's by the same laboratory performed well. This is a result of poor primer and probe design to a non-conserved region of the virus. Most of the panels only contained one virus type; however, the panel for hRV contained three subtypes and the panel for ADV contained two subtypes. These two panels showed that some assays were unable to detect certain subtypes, even in the high positive sample, but were able to detect the weak positive with other subtypes. This is a result of primer and/or probe selection, which were unable to detect these viruses owing to the sequence variation seen in these strains. Therefore, participants who missed these strains would need to adapt their NAT in order to detect the possible variations. These results indicate the problem of using NAT methods for detection of respiratory viruses, in that they are not catchall methods and careful evaluation of the primers and probes needs to be performed to ensure that no false negatives are obtained. All respiratory viruses detected in clinical samples will have sequence variation. So, in future panels more viruses to determine the ability of assays to detect clinical isolates might be included. In the case of influenza A and B, the strains included could be the most common circulating strains from the previous winter. In this way, the quality of the NATs can be monitored for detection of a wide variety of viral strains.

The other reason that assays failed to detect viruses in the weak positive was the lack of sensitivity of the NAT or procedures in the laboratory. There are a lot of factors that affect NAT sensitivity but this study gives some indications as to ways to improve sensitivity with PCR format, selection of gene target for the PCR and design of primers and probe. Firstly, all the individual components of the 83 in-house NAT could achieve maximal sensitivity in one or more panels, so the difference in sensitivity is probably due to inter-laboratory variation. Another factor that affects sensitivity is the methodology. However, all methodologies, except the one laboratory using the RV ® Chip assay, could obtain maximum scores. So, again, the differences are probably due to inter-laboratory variations. A wide range in performance could be seen with the same methodology, e.g. real-time PCR.

The selection of the gene target was also shown to affect the sensitivity, in that some targets result in a more sensitive NAT. Comparisons of different targets have been shown with hMPV where either the polymerase or the nucleoproteins were shown to be more sensitive than matrix, fusion and phosphoprotein (Cote et al., 2003; Maertzdorf et al., 2004) .

The results here suggest that for influenza A the M protein is a good target and the HN for PIV 3. The conservation of the target is also important as the primers and probe need to detect all the virus types, so in some cases, a primer with two or three mismatches may work for a particular virus although the sensitivity will be affected. Therefore, assays, which only detected the strong positive dilution using a good sensitive methodology, may need to check for a mismatch of the primer and probe sequence and potentially include further forward or reverse primers to achieve good sensitivity. Inclusion of additional primers or probes to improve the range of detection has been shown in the case of enteroviruses and rhinoviruses previously (Deffernez et al., 2004; Nijhuis et al., 2002) .

Respiratory virus detection by culture has limitations owing to the sensitivities of cell lines to clinical isolates, this is particularly apparent with hRV and hCoV. Although reference strains can be used to check sensitivity of a particular cell line this does not necessarily mean that it will detect all clinical isolates. This problem can be equally true for poorly evaluated NATs. Nevertheless, NAT is becoming a valuable diagnostic tool, and hence the need for laboratories to perform quality control to evaluate sensitivity.

The production and distribution of this first respiratory virus panel showed that 92.4% of participants detected the strong positive and 47% of participants detected the low positive correctly. The panel also gives a very good indication to participant as to where to improve the assay's design as well as providing a good measure for the integrity of the NAT employed by a laboratory.

Lack of sensitivity of rapid antigen tests for the diagnosis of respiratory syncytial virus infection in adults

Simultaneous detection of fourteen respiratory viruses in clinical specimens by two multiplex reverse transcription nested-PCR assays

Comparative evaluation of real-time PCR assays for detection of the human metapneumovirus

Amplicon sequencing and improved detection of human rhinovirus in respiratory samples

Enhancing public health surveillance for influenza virus by incorporating newly available rapid diagnostic tests

Practical implementation of a multiplex PCR for acute respiratory tract infections in children

Advances in the laboratory diagnosis of viral respiratory disease

Rhinovirus infections in hematopoietic stem cell transplant recipients with pneumonia

Real-time reverse transcriptase PCR assay for detection of human metapneumoviruses from all known genetic lineages

Rapid and sensitive routine detection of all members of the genus enterovirus in different clinical specimens by real-time PCR

Multicentre quality control study for detection of Mycobacterium tuberculosis in clinical samples by nucleic amplification methods

Evaluation of a multiplex reverse transcriptase PCR ELISA for the detection of nine respiratory tract pathogens

External quality assessment program for qualitative and quantitative detection of hepatitis C virus RNA in diagnostic virology

An international external quality assessment of nucleic acid amplification of herpes simplex virus

Rapid diagnostic tests for influenza

A sensitive, specific, and cost-effective multiplex reverse transcriptase-PCR assay for the detection of seven common respiratory viruses in respiratory samples

Rapid and sensitive method using multiplex real-time PCR for diagnosis of infections by influenza a and influenza B viruses, respiratory syncytial virus, and parainfluenza viruses 1, 2, 3 and 4

van den Hoogen BG, Osterhaus DM, Fouchier RA. Clinical impact and diagnosis of human metapneumovirus infection

Polymerase chain reaction is more sensitive than viral culture and antigen testing for the detection of respiratory viruses in adults with hematological cancer and pneumonia

Frequent detection of human coronaviruses in clinical specimens from patients with respiratory tract infection by use of a novel real-time reverse-transcriptase polymerase chain reaction

Reliability of nucleic acid amplification methods for detection of Chlamydia trachomatis in urine: results of the first international collaborative quality control study among 96 laboratories

The value of polymerase chain reaction for the diagnosis of viral respiratory tract infections in lung transplant recipients

Superiority of reverse-transcription polymerase chain reaction to conventional viral culture in the diagnosis of acute respiratory tract infections in children

Seventeen European laboratories were invited by QCMD to take part in the respiratory virus pilot programme: