key: cord-1040598-g0xnbul3 authors: Forman, Michael S.; Advani, Sonali; Newman, Christina; Gaydos, Charlotte A.; Milstone, Aaron M.; Valsamakis, Alexandra title: Diagnostic performance of two highly multiplexed respiratory virus assays in a pediatric cohort date: 2012-07-24 journal: J Clin Virol DOI: 10.1016/j.jcv.2012.06.019 sha: 9be84e01a32ee5dbe44a6f2f93ee3b4933a495a3 doc_id: 1040598 cord_uid: g0xnbul3 BACKGROUND: Rapid detection of respiratory viruses is important for management and infection control in hospitalized patients. Multiplex nucleic acid tests (NATs) have begun to replace conventional methods as gold standards for respiratory virus detection. OBJECTIVE: To compare the performance of two large multiplex NATS, ResPlex II (RPII) and Respiratory Virus Surveillance kit with electrospray ionization mass spectrometry (RVS/MS) using nasopharyngeal aspirates (NPAs) from hospitalized children who had been tested previously with conventional methods. STUDY DESIGN: Stored residual NPAs (N = 306) were tested concomitantly by RPII and RVS/MS. Alternate NATs were used to adjudicate discordant results. RESULTS: More viruses were detected with multiplex NATs (RPII, 110; RVS/MS, 109) than conventional assays (86); diagnostic gain was primarily for fastidious viruses (coronaviruses and enteroviruses [EVs]/human rhinoviruses [HRVs]). Total positive and negative agreement between the multiplex NATs for all viruses detected was quite high (86% positive agreement, 99% negative agreement). Most individual viruses were detected with fairly equivalent accuracy by the multiplex NATs, except for adenoviruses (RPII sensitivity 40%) and human metapneumovirus (RVS/MS sensitivity 42%). RPII had the advantage of detecting EVs and HRVs, however, it demonstrated considerable EV/HRV cross-reactivity (29 HRV-positive specimens by real-time PCR were positive for EV by RPII and 21 specimens positive for HRV only by RT-PCR were dual positive for EV/HRV by RPII). RPII also had reduced sensitivity for HRV detection (in 36 specimens, HRV was detected by RT-PCR but not by RPII). CONCLUSIONS: Both multiplex NATs were promising, but had notable limitations. Timely respiratory virus detection in hospitals is important for patient management and infection control. Rapid, highly sensitive nucleic acid tests (NATs) are replacing conventional methods as the new gold standard. Respiratory virus NATs vary from singleplex to multiplex PCR and in either conventional or real-time formats. 1 Multiplexed respiratory NATs are often preferable since they eliminate the need for separate NATs to identify respiratory viruses that often cause overlapping clinical presentations. ResPlex II (RPII) and Respiratory Virus Surveillance kit with electrospray ionization mass spectrometry (RVS/MS) are two large multiplex NATs with different amplification/detection designs. RPII contains a low concentration primer pair that is complementary to viral targets and has a heterologous sequence. A second, high concentration primer pair contains sequences complementary to this heterologous sequence. This design favors specific target amplification early in PCR and amplicon amplification at later cycles, improving amplification efficiency and reducing background. Respiratory syncytial virus (RSV) A and B, influenza (INF) A and B, parainfluenza 1-4 (PIV1-4), human metapneumovirus (hMPV), coxsackieviruses/echoviruses (EV herein), human rhinovirus (HRV), adenovirus B and E (ADV), coronaviruses NL63, HKU1, 229E, and OC43 (referred to collectively as CoV), and bocavirus are detected. The RVS/MS assay utilizes conventional PCR and electrospray ionization mass spectrometry (ESI/MS) to detect and identify respiratory viruses. 4 ESI/MS is used to convert amplicon duplexes to single strands, and to determine nucleotide composition by measuring single strand masses. 5 Observed masses are compared to a library of expected masses to identify amplified products. Each PCR reaction contains an exogenous internal control that can also be used to estimate target analyte concentration. RVS/MS detects similar viruses as RPII except it is designed to detect a broader array of ADVs (genotypes A-F) and it does not detect EVs and HRVs. The purpose of this study was to compare the ability of RPII and RVS/MS to detect viruses in nasopharyngeal aspirates (NPAs) from clinically well-characterized, hospitalized pediatric patients that had previously been tested by conventional methods. NPAs were collected from infants and toddlers (N = 306) upon admission to the hospital from December 2007 to March 2008 and submitted for conventional virologic testing. 2 This retrospective cohort was selected for testing because it has been well characterized clinically. Residual samples were stored at −70 • C until nucleic acid testing; NATs were performed simultaneously. Cell pellets from processed NPAs were spotted onto slides and stained with antibodies to INF A and B, RSV, PIV 1-3, ADV, and hMPV (D3 Ultra Screening and ID reagents and hMPV reagent, Diagnostic Hybrids, Inc [DHI], Athens, OH). Supernatants were cultured by shell vial (R-Mix Too, DHI) and tube culture. R-Mix Too coverslips were stained at 48 h with D3 Ultra Screening reagent then with specific ID reagents when appropriate. Tube cultures were screened for cytopathic effect for 3 weeks; confirmatory staining was performed when appropriate. Total nucleic acid was extracted from NPAs using the QIAsymphony SP automated instrument, QIAsymphony Virus/Bacteria Mini Test Kit reagents, and the Complex 200 protocol (Qiagen Germantown, MD). Input and elution volumes were 350 l (200 l processed) and 110 l. Amplification and detection reactions were performed according to manufacturer's research use only protocol. 3 Process controls added by the QIASymphony during extraction included an internal control and reagents to detect human genomic DNA (proprietary target) that ascertained sample quality. Signal detection/raw data analysis Signals were detected with the Luminex 200 IS System using QIAplex MDD Software. In-house determination of the cutoff value that could be applied to distinguish positive results was required since software lacks this interpretive criterion. Four different cut-off values were studied including 3× background (bkgd), ≥50, ≥100, or ≥150 mean fluorescence intensity (MFI) values. For 3× background calculation, the negative control MFI for each assay was multiplied by three and used as the cutoff to define positive samples within that run. To decide which cut-off value was most appropriate, the results obtained with each cutoff were compared to RVS/MS, discrepant analysis with alternate PCR assays, and conventional test results. Two cutoffs, ≥100 MFI and 3× bkgd MFI, showed the best correlation with aggregate data from other test methods. The 3× bkgd cut-off was selected because it demonstrated excellent overall agreement with the other methods and it allowed inter-run data normalization by accounting for runto-run variability in background fluorescence. For the purposes of this study, RSV A and B were not differentiated and bocavirus was not included in the analysis since its importance as a respiratory pathogen has not been firmly established. 3.5. RVS/MS (Ibis Biosciences, Inc., a part of Abbott Molecular, Des Plaines, IL) Target amplification was performed as described previously 4 using a Mastercycler ep gradient S (Eppendorf, Hamburg, DE). Amplification products were desalted and analyzed using the T5000 universal biosensor, a time-of-flight mass spectrometer. 5 Two different real-time RT-PCR assays for EVs 6 and HRVs 7 were used as comparators for RPII since these viruses were not detected by RVS/MS. Assays were performed as described with the following modifications: the QIAsymphony was used for extraction and realtime PCR was performed with QuantiTect Virus Kit (Qiagen) and 7500 Real Time PCR system (50 • C for 30 min, 95 • C for 15 min and 45 cycles of 95 • C for 15 s and 60 • C for 1 min). Discordant results were adjudicated with alternative NATs (Table 1) . Highly sensitive singleplex assays (5-50 copies/reaction as demonstrated by published or in-house data) were selected for use. Multiplex NAT (xTag RVP, Luminex, Austin, TX) was used when singleplex assays that met the above criteria were not readily available (Table 1) . Data were stored in Excel (Microsoft, Redmond, WA) and analyzed using Stata, version 11.0 (StataCorp, College Station, TX). Positive and negative agreement (by virus and by total positives and negatives) were calculated using RPII as the test assay and RVS/MS as the comparator assay. Since this agreement analysis highlights deficiencies in virus detection by RPII but minimized detection deficiencies by RVS/MS, an assessment of sensitivity and specificity Likewise, PIVs were analyzed as a single group (PIV) since the combined total number of PIV 1, 3 and 4 detected was four. CoVs were analyzed as a group since there is currently no clinical utility to distinguishing these viruses individually. EV and HRV discordant results were not adjudicated by a third assay, therefore, only agreement was determined. Confidence intervals were calculated using a web-based calculator. 8 Positive signal statistical data provided in Supplemental data tables were calculated using Excel. The number of viruses detected by RPII and RVS/MS was similar (110 vs. 109, Table 2 ) and greater than the number detected by conventional methods (N = 86). Viruses were detected in 101 (33%), 98 (32%), and 86 (29%) of 306 samples by RPII, RVS/MS, and conventional methods respectively (data not shown). RSV was most commonly identified by all methods ( Table 2 ). The second and third most commonly detected viruses varied by multiplex. INF A, hMPV, and CoVs were the next most prevalent viruses found by RPII (INF A, 5.2% prevalence overall and 15% of all identified viruses; hMPV and CoVs, 4.2% prevalence overall and 12% of all identified viruses), while ADV and INF A were found by RVS/MS (ADV, 5.5% prevalence overall and 16% of all identified viruses; INF A, 4.9% prevalence overall and 14% of all identified viruses). CoVs were not detected by conventional methods but were fairly common (∼4% prevalence overall, ∼12% of all identified viruses) and similarly detected by both multiplex NATs. The relative diagnostic performance of the two multiplex NATs in individual samples was high and fairly equivalent for most viruses, as determined by agreement analyses and by assay sensitivity/specificity ( Table 3 ). The lowest % positive agreement was observed for ADV; lowest % negative agreement was observed for hMPV. Resolution of discordant results with either an alternate NAT or conventional result further delineated the disparity in detection of these two viruses by the multiplex NATs (sensitivity of 40% for ADV by RPII and 42% for hMPV by RVS/MS). Aside from samples that were negative for both viruses by RPII and RT-PCR, there was little agreement of EV/HRV results in individual samples (Table 4 ). No samples were dual positive for HRV plus EV by RPII and by RT-PCR, only one sample was positive for EV by RPII and RT-PCR, and only 7 samples were positive for HRV by RPII and by RT-PCR. RPII detected EVs in a greater number of samples than RT-PCR (67 vs. 16). These data suggest cross-reactivity between RPII EV and HRV detection reagents rather than enhanced sensitivity of EV detection since 50/67 EV positive samples (21 dual positive and 29 EV only) contained only HRV by RT-PCR. Furthermore, a large proportion of RPII EV only/RT-PCR negative samples would be expected if EVs were more effectively detected by RPII, however only two such samples were identified. In contrast to EV detection, RPII detected HRV in fewer samples than RT-PCR (29 vs. 107). Notably, 36 samples were found to contain HRV RNA by RT-PCR but were negative by RPII ( Table 4 ). The mean Ct value for these HRV-positive samples was 36, the highest of any category found to contain HRV by RT-PCR in the 4 × 4 analysis (Supplemental data Table 1 ), suggesting that these samples contained low target RNA levels. Fourteen samples were positive for HRV and EV by RT-PCR, but were only positive for EV by RPII. In a qualitative concordance analysis, these samples would appear to be false positive for RT-PCR results, however analysis of signal strength suggests the opposite. The mean and median HRV Ct values were the lowest of any category of samples found to contain HRV by RT-PCR (mean 24, median 21; see Supplemental data Table 1 ). These data suggest erroneous EV detection by RPII and RT-PCR in samples containing high HRV concentrations. A simple 2 × 2 agreement table in which the specific virus or combination of viruses is not differentiated (Table 4 , bottom panel, EV/HRV Not Specified) was constructed given the high degree of sequence concordance between EVs and HRVs and the absence of definitive sequence data for the studied samples. Positive agreement in this analysis was fairly high (72/109, 66%). Even in this analysis however, singleplex real-time RT PCRs were more sensitive since they detected EV/HRV in 37 samples that were negative by RPII, largely due to the number of samples that were positive by RT-PCR for HRV but negative by RPII. Co-infections with viruses other than EV/HRV were found in relatively few samples (RPII, 6/306, 1.9%; RVS/MS, 8/306, 2.6%) and all confirmed co-infections contained CoVs (specifically HKU1, Supplemental data Table 2 ). Co-infections with enteroviruses (EV and HRV) and other viruses were found in 18 samples by RPII (Supplemental data Table 3 ). NATs are powerful tools in the diagnosis and management of respiratory virus infections due to their enhanced sensitivity and rapid time to result. Deep multiplexing also has the important advantage of allowing identification of not only the usual pathogens (RSV, INF A, ADVs) but also unsuspected, typically lower virulence viruses (PIVs, HRVs, CoVs) that can cause lower respiratory tract disease and can therefore be significantly pathogenic in highly susceptible hosts. In this performance comparison study on samples from hospitalized children, the highly multiplexed RPII and RVS/MS assays detected most viruses with fairly equivalent accuracy. The exceptions to this were hMPV and ADV which were more effectively detected by RPII and RVS/MS respectively. A low ADV detection rate by RPII compared to a different extensively multiplexed respiratory virus NAT (Seeplex) has been previously reported. 9 Independent published confirmation of low hMPV detection rates by RVS/MS compared to an alternate NAT are lacking. Given that ADV and hMPV are highly prevalent and can cause serious disease, these potential diagnostic shortcomings are significant and represent opportunities for performance enhancement in subsequent generations of these assays. Unfortunately the prevalence of PIVs was too low to make any definitive conclusions regarding the relative efficacy of RPII and RVS/MS. It should also be noted that the study design precludes conclusions regarding true (vs. relative) accuracy and sensitivity. Therefore, while the accuracy of detection for most viruses was comparable and quite high, testing with gold standard NATs for each analyte might reveal additional detection deficiencies common to RPII and RVS/MS. For RPII, the manufacturer does not supply a cutoff MFI value that allows differentiation of positive and negative results. Our method comparison study design allowed for analysis of multiple cutoffs as defined above. A variable cutoff based on intra-run background signal functioned well and offered the theoretical advantage of correcting for inter-run background variability. Additional studies with a larger cohort are required to conclusively establish its superiority. To have maximal clinical utility, respiratory virus NATs should detect the broadest range of pathogenic targets. RPII provides better diagnostic potential than the current version of RVS/MS since it can detect EV/HRV. In fact, in this study, the prevalence of EV/HRV was highest of all viruses (∼37%, including RPII and RT-PCR positives). However, RPII has limitations with regard to EV/HRV detection. First, the data suggest cross-reactivity between RPII EV and HRV detection reagents as suggested by previous studies that noted discordant EV/HRV results in a limited number of samples 10 and a high proportion of EV/HRV dual positives. 9 Our data imply that crossreactivity is more extensive than previously appreciated given the number of discordant (RPII EV+/RT-PCR HRV+) and RPII dual+/RT-PCR HRV only specimens. One curious result was the number of specimens found to be dual positive by RT-PCR but positive only for EV by RPII. On a qualitative basis, these results would appear to suggest false positive HRV RT-PCR results. However signal strength data point to a different conclusion. The low mean and median HRV RT-PCR Ct values suggest that most of these samples likely contained HRV only, further implying issues with EV detection specificity (by RPII and RT-PCR) and sensitivity of HRV detection (by RPII). Published data 6 and our unpublished observations with the EV RT-PCR reagents suggest a propensity for cross-reactivity. An internal study of EV and HRV RT-PCR reagents demonstrated HRV in 69/507 NPAs; 4/69 were also amplified by EV RT-PCR. VP1 sequence data was obtainable from 1/4 samples and demonstrated HRV. HRV RT-PCR specificity should be much greater than EV RT-PCR; the latter was developed by comparative analysis of a more extensive sequence library compared to EV RT-PCR and was shown to cross-amplify EV only in artificial samples spiked with high concentrations of synthetic transcripts. 7 Improved EV/HRV concordance between RPII and RT-PCR was observed when positive results were grouped (EV/HRV unspecified), which may be the most accurate approach for reporting RPII results. However even with grouped reporting, it was apparent that RPII missed approximately one third of infections detected by singleplex RT-PCR. A similar defect in EV/HRV detection compared to an alternate NAT has been reported. 10 Subsequent versions of this assay would benefit from improved HRV sensitivity given reports of HRV causing clinically significant disease. 11, 12 Few co-infections with viruses other than EV/HRV were observed. Strikingly, all confirmed non-EV/HRV co-infections included HKU-1 as a co-pathogen. The biological significance of this is unclear; no data are available in the literature. Among EV/HRV coinfections, no dominant non-enteroviral pathogen was observed. The identified viruses reflect the spectrum commonly observed in young children. The greatest diagnostic gain of the extended panel NATs was largely for viruses that are more challenging to detect by conventional methods. Viruses such as RSV and FluA that are readily detectable by DFA and culture were equivalently detected by NATs and conventional methods. PIV prevalence was too low to make any conclusive statements regarding the relative efficacy of conventional vs. NAT panels but the data suggest improved detection by NATs. Deeply multiplexed respiratory virus NAT replaced cell culture for use in immunocompromised populations in our center in the wake of the 2009 influenza pandemic due to the perceived benefit that a rapid, highly sensitive result would provide for patient management. After several years' experience with this testing, it has become apparent that the characteristics of the optimal respiratory virus NAT include ability to detect broadest range of respiratory virus pathogens, highly accurate detection of each virus, low labor requirement, results available within a single shift, and low contamination rates. The increasing availability of deeply multiplexed commercial respiratory virus NATs offers the opportunity to compare performance characteristics. Our studies with RPII and the current version of RVS/MS suggest that they each have strengths but both demonstrated some limitations with regard to fitting the profile of the perfect assay. This work was funded by R-Baby Foundation. Reagent support was kindly provided by Qiagen and Ibis Biosciences/Abbott Molecular. C. Gaydos has received research funding from Abbott/Ibis. A. Valsamakis has received research funding from Abbott/Ibis and Qiagen; she has served on a scientific advisory board for Diagnostic Hybrids Inc. This study was approved by the Johns Hopkins School of Medicine Institutional Review Board (Study #NA 00021591). Nucleic acid amplification tests for detection of respiratory viruses Epidemiology of respiratory viruses in children admitted to an infant/toddler unit Sample and Assay Technologies Qiagen Rapid identification viruses from nasal pharyngeal aspirates in acute viral respiratory infections by RT-PCR and electrospray ionization mass spectrometry Rapid identification of emerging infectious agents using PCR and electrospray ionization mass spectrometry Evaluation of real-time PCR versus PCR with liquid-phase hybridization for detection of enterovirus RNA in cerebrospinal fluid Real-time reverse transcription-PCR assay for comprehensive detection of human rhinoviruses Evaluation of multiple commercial molecular and conventional diagnostic assays for the detection of respiratory viruses in children Evaluation of commercial ResPlex II v2.0, MultiCode-PLx, and xTAG respiratory viral panels for the diagnosis of respiratory viral infections in adults Severity of human rhinovirus infection in immunocompromised adults is similar to that of 2009 H1N1 influenza Severe human rhinovirus outbreak associated with fatalities in a long-term care facility in Ontario Pring-Akerblom P. Rapid and quantitative detection of human adenovirus DNA by real-time PCR Real-time reverse transcriptase PCR assay for detection of human metapneumoviruses from all known genetic lineages Molecular diagnosis of respiratory virus infections The authors thank the staff of the Johns Hopkins Hospital Clinical Virology Laboratory for excellent technical support. Supplementary data associated with this article can be found, in the online version, at http://dx.doi.org/10.1016/j.jcv.2012.06.019.