key: cord-0874468-55r01n0x
authors: Lemmon, Gordon H; Gardner, Shea N
title: Predicting the sensitivity and specificity of published real-time PCR assays
date: 2008-09-25
journal: Ann Clin Microbiol Antimicrob
DOI: 10.1186/1476-0711-7-18
sha: 2a98b0a9e3e89d1aff14a6f64e25374b4e2855a8
doc_id: 874468
cord_uid: 55r01n0x

BACKGROUND: In recent years real-time PCR has become a leading technique for nucleic acid detection and quantification. These assays have the potential to greatly enhance efficiency in the clinical laboratory. Choice of primer and probe sequences is critical for accurate diagnosis in the clinic, yet current primer/probe signature design strategies are limited, and signature evaluation methods are lacking. METHODS: We assessed the quality of a signature by predicting the number of true positive, false positive and false negative hits against all available public sequence data. We found real-time PCR signatures described in recent literature and used a BLAST search based approach to collect all hits to the primer-probe combinations that should be amplified by real-time PCR chemistry. We then compared our hits with the sequences in the NCBI taxonomy tree that the signature was designed to detect. RESULTS: We found that many published signatures have high specificity (almost no false positives) but low sensitivity (high false negative rate). Where high sensitivity is needed, we offer a revised methodology for signature design which may designate that multiple signatures are required to detect all sequenced strains. We use this methodology to produce new signatures that are predicted to have higher sensitivity and specificity. CONCLUSION: We show that current methods for real-time PCR assay design have unacceptably low sensitivities for most clinical applications. Additionally, as new sequence data becomes available, old assays must be reassessed and redesigned. A standard protocol for both generating and assessing the quality of these assays is therefore of great value. Real-time PCR has the capacity to greatly improve clinical diagnostics. The improved assay design and evaluation methods presented herein will expedite adoption of this technique in the clinical lab.

Real-time PCR assays are gaining popularity as a clinical tool for detecting and quantifying the presence of both viral and bacterial pathogens, as reviewed in [1] . Com-pared to traditional culturing methods used in identification, real-time PCR is fast and cost effective. In addition, it can be quantitative and sensitive, in some cases greatly exceeding the sensitivity for conventional testing meth-ods. Commercially distributed kits are available for PCRbased pathogen diagnostics, and PCR is no longer thought of merely as confirmatory to culture. However real-time PCR assays are limited by the quality of the primers and probes chosen. These primers and probes must be sensitive enough to match all target organisms yet specific enough to exclude all others.

A common approach to developing a primer/probe combination is by using commercial software such as PrimerExpress ® (Applied Biosystems, Foster City, CA, USA). This software asks the user to upload a DNA sequence file, and then finds possible primer/probe sets that meet the assay criteria. Generally a researcher will provide as input a gene region conserved throughout the taxa that the assay is being designed to detect. The software then provides possible primer/probe sets. The researcher chooses a representative signature. If there are single nucleotide polymorphisms (SNPs) within the chosen conserved region, a signature with consensus primers and probes is often chosen. Next a BLAST [2] search is performed to ensure that the primers are not hitting other targets. Finally the signature is verified in vitro with laboratory strains.

While this design approach may work acceptably well in the research laboratory, the clinical laboratory calls for a more thorough analysis to ensure detection of novel, diverse, and uncommon strains. These may appear, for example, as a result of spread by foreign travel or migration. Whole genome based automated signature design [3] presents a great improvement to the common method. However, in addition to better design strategies, methods for automated signature evaluation are needed. As additional sequence data becomes available, it is necessary to regularly reassess the predicted efficacy of a given signature. This analysis must include the predicted false negative and false positive rates for the developed signatures, and consider all available public sequence data.

We have analyzed a number of real-time PCR assays found in the literature based on public sequence data. Herein we report how well these signatures performed, offer a revised approach to PCR assay design, and use this approach to produce new assays predicted to have higher sensitivity and specificity.

The literature was combed for recently published articles reporting real-time PCR assays for the clinical detection of bacterial and viral taxa. The primer and probe sequences were accumulated, with a preference for TaqMan assays. However, 3 intercalating dye assays were also selected. Papers reporting nucleotide sequences that could not eas-ily be copied from an online source were avoided. In total, 112 signatures from 32 papers were analyzed.

Local Oracle databases have been constructed from the complete genome sequence data available at NCBI Genbank, TIGR, EMBL, IMG (JGI), and Baylor HGSC. We used our "all_virus" and "all_bacteria" databases to find signature matches and predict false negatives and false positives. These databases were designed to contain only whole genomes and whole segments from segmented genomes. However, the heuristics used to separate whole genomes from partial sequences are not fail-proof due to inconsistency in sequence annotation within the public databases. Consequently many sequences in these databases may show up as false negatives when they are actually just a section or segment of a genome that is not expected to contain the signature, and we manually sorted these sequences into true or false negatives.

A freely available real time PCR analysis tool called TaqSim [4] was used to find public sequences that would match the primer/probe assay in question. TaqSim uses BLAST searches to find sequences that match both forward and reverse primers and probe. To be reported as a "hit" the primers and probe must match in the required orientations relative to one another and the primers must be in sufficiently close proximity. The forward/reverse primers may fall on either the plus or minus strand, so long as the orientation relative to one another is appropriate. There may not be mismatches at the 3' end of either primer. For each hit TaqSim calculates the primer and probe melting temperatures as bound to the candidate hit sequence (accounting for mismatches) based on reaction conditions (reagent concentrations and hybridization temperature), and returns sequences predicted to be amplified. Instead of replicating the various exact reaction conditions reported in each paper, very lenient settings were applied in all cases, essentially removing the screen for primer/probe vs. candidate hit Tm by setting this threshold to 0°K, and instead checking for specificity by requiring that hits have fewer than 3 mismatches per primer or probe.

TaqSim's predicted sequence hits were compared with sequences listed under a given set of NCBI taxonomy tree nodes. For instance, if a signature was reported to detect Hepatitis B, then its set of TaqSim hits would be compared with the set of sequences under node 10407, corresponding to Hepatitis B virus. Sequences in both sets were considered true positives, sequences in the TaqSim output that were missing from the chosen taxonomy nodes were considered false positives, and sequences that were in the taxonomy tree but missing from the TaqSim output were considered false negatives. Test statistics such as specificity and sensitivity (power) were then calculated. In this paper we define sensitivity and specificity as follows:

The primary research articles were read carefully to determine what the authors had designed their primers/probes to detect. NCBI Taxonomy nodes were chosen to represent these target organisms. This was not a trivial task, since many articles lack clarity as to which taxa, specifically, their assay should detect. For instance, the cytomegalovirus assay did not detect all sequences in the cytomegalovirus genus (taxonomy node 10358), but rather all sequences in the human herpesvirus 5 species (taxonomy node 10359). None of the articles specified a taxonomy node for their signatures.

Perl scripting [5] was used to help compare BLAST hits and taxonomy node sequences, and count false negative, false positive and true positive sequence matches. However some sequences required hand sorting due to the wide array of sequence types and annotations. These often represented segmented genomes, in which case many of the would-be false negative sequences simply represent a different segment than that on which the signature lands, so we manually tabulated them as true negatives. They may also represent plasmids. In these situations, a careful review of the Genbank entry, and sometimes of the primary article cited by Genbank, was necessary to determine if the sequence of interest was truly a false negative.

Although we attempt to include only complete genomes in our sequence database, because of inconsistencies in the annotation of sequence data some partial sequences nevertheless make it into our databases. Any of these partial CDS's documented as containing the target gene on which the signature was supposed to land were counted as false negatives, but those partial CDS's not documented to contain the target gene were eliminated from the false negative pool because it is possible the signature could land on the unsequenced section with the target gene. Our database also contains "glued fragments", which represent draft genomes "glued" together with hundreds of "N"s as a simple way to keep the separate contigs associated as part of the same genome. While we report false negatives from these draft genomes, it is possible that the signatures could land on gaps between the contigs, and that finished sequencing could result in re-classification as a true positive.

Tables 1, 2, 3, 4, 5 summarize our analysis of various DNA signatures. Details of all true positive, false positive and false negative sequences are available from the authors. Note that these results are in silico results; no laboratory testing was performed for verification, so that by stating that an organism is "detected" we mean that this is our prediction based on sequence data. A few notes of interest concerning the data in the tables are described below:

The two human corona virus strains, 229E and OC43 are a frequent cause of the common cold [6] . A Taqman assay for 229E was predicted to perform perfectly, while an assay for OC43 turned up a number of false positives, all of which were animal corona viruses. A coxsackie B3 virus assay [7] performed well, but a coxsackie B4 assay [8] hit many other human coxsackie, echo, and entero viruses. Four out of 5 false negatives for a Marburg virus assay [9] were of the Lake Victoria variety. False negatives associated with a yellow fever signature [10] included Trinidad, French neurotropic, French viscerotropic, and vaccine strains. The Filoviridae (Ebola/Marburg) assay [10] detected only Ebola viruses.

Staphylococcus aureus [11] and Enterobacteriaceae assays [12] had low sensitivity. An Escherichia coli assay [12] hit Shigella and Vibrio sequences.

Many of these signatures [13] had high sensitivity. Combining several of them into a multiplex assay would probably improve sensitivity further.

These signatures were designed using a minimal set clustering approach [14] . While individual signatures have decent sensitivity, combining several signatures in one assay, as advocated in the publication greatly improved sensitivity. The signatures for Hepatitis A are currently undergoing laboratory screening by the FDA, and are performing well (G. Hartman, personal communication).

Several reported signatures produced no predicted hits. These include assays for several flaviviruses [10, 15, 16] , and 16S rRNA assays [12] for several bacteria. Examination of BLAST output showed that in these cases either a primer or internal oligo (probe) did not have BLAST hits to target, there were too many mismatches per primer or probe sequence above the threshold specified in our anal- All but three are Taqman signatures. The three intercalating dye type assays are bolded. yses, or there were mismatches at the 3' end of a primer relative to target. It is possible that if the sequences of the samples used in the laboratory differ from available genomic data, or if the PCR reaction conditions are performed at low stringency (e.g. low annealing temperatures or high salt concentrations) these assays could in fact work in the laboratory. However, according to the genomic data available, a better match of primers and probes to target is possible and is usually desired for high sensitivity detection.

Targeting a number of the organisms for which currently published signatures were predicted to perform poorly, as well as some for which additional signatures may be desired (even though published signatures may perform well), we generated new signatures using Minimal Set Clustering (MSC) according to methods previously described [14, 17] . MSC begins by removing non-unique regions from consideration as primers or probes from each of the target sequences relative to a database of nontarget bacterial and viral sequences. The remaining unique regions of each target sequence are mined for all or many candidate signatures, without regard for conservation among other targets, yet satisfying user specifications for primer and probe length, T m , GC%, avoidance of long homopolymer runs, and amplicon length. All candidate signatures are compared to all targets and clustered by the subset of targets they are predicted to detect. Signatures within a given cluster are equivalent, in that they are predicted to detect the same subset of targets, so by clustering we reduce the redundancy and size of the problem to finding a small set of signatures that detect all targets. Nevertheless, finding the optimal solution of the fewest clusters to detect all targets is an NP complete problem, so for large data sets we use a greedy algorithm to find a small number of clusters that together should pick up all targets. In the supplementary table, we often provide more than one alternative signature to detect a given equivalence group of genomes to serve as a backup should a signature perform poorly in laboratory testing. Some of the signatures may have mismatches to some of their intended targets, although these mismatches are not predicted to reduce the T m of primer/probe hybridizing to target below typical TaqMan reaction conditions. None of these computationally predicted signatures have been screened in the laboratory, as this is beyond the scope of this paper. 

Year Target Pseudomonas aeruginosa, Escherichia coli, and Neisseria meningitidis.

As expected we found that false negatives were much more common than false positives. Though signatures are generally based on conserved gene regions, they often fail to take into account all of the variation within a target set of organisms. This may be because the signatures were developed using sequence data from a handful of strains, rather than a thorough study of all strains publicly available.

These false negatives may also represent sequences that have become available since the publication of the given signature. Since new sequence data is made available at an ever increasing rate, there is great benefit in re-evaluating clinically used DNA signatures regularly. When new sequence data leads to false negative predictions for a signature, one of two explanations can be given. The new sequences either represent recently recognized variation that has been around since the time the signature was published, or new variation, the result of mutation and natural selection. In either case, an improved or additional signature should be designed.

High false positive or false negative rates do not necessarily indicate a "bad" DNA assay. The quality of an assay must be considered in light of the milieu in which the testing will take place. In the clinical laboratory, a signature with high sensitivity but perhaps low specificity may be preferred over a test with lower sensitivity in cases where the putative pathogen requires immediate treatment or may spread quickly. The case of antibiotic resistant bacteria probably falls in this category. On the other hand, the nation's BASIS and BioWatch programs insists on zero false positives, so as to avoid public disturbances due to false alarms, while still aiming for zero false negatives [18] .

One must also consider the type of false negative and false positive results to determine their relevance. For instance, in this article an assay for human corona virus OC43 [6] What about such a match in a clinical lab in Africa? On the other hand, the echovirus sequences that the coxsackie B4 assay [8] can detect could produce misleading results in any clinical lab.

The false negative and false positive rates presented in this study may vary substantially from those seen empirically. This is because the strains available in a laboratory may differ significantly from the sequence data available, or because the empirical protocol is more or less stringent than the sequence-based requirements we imposed, which allowed no more than 2 mismatches per primer or probe for detection.

We believe that as more target sequences become available, our predicted false negative rates will tend to increase for a given published signature both as a result of better sampling of diversity and as a result of failure to detect newly evolved variants. It has been estimated that a minimum of 3-4 genomes are needed in order to computationally design TaqMan PCR signatures likely to detect most strains, with those isolates chosen for sequencing that have been selected to span gradients of geographic, phenotypic, and temporal variation [19] . Even more than 4 genomes are needed for particularly diverse organisms. Thus, older signatures may not perform as well as newly developed signatures from the most up-to-date sequence data. A future study of interest would be a longitudinal look at how these rates continue to change over time as additional sequences become available. This study could be performed retrospectively, since sequence submission dates are easily obtained from public databases.

We also hypothesized that the wider the intended scope of a signature, the lower its sensitivity would tend to be. The point is illustrated loosely in our data tables. Twenty-six of the 28 signatures with less than 10 publicly available target sequences had sensitivities of 1 (i.e. zero false negatives), while signatures with 10 or more targets had an average sensitivity of 0.710116. However this approach only considers scope in the context of sequence data available.

We tried to demonstrate the relationship between specificity and scope at a more fundamental level by grouping signatures by the taxonomic level of their target as shown in Figure 1 . However the results are misleading. In virology, taxonomic level is not a good indicator of nucleotide diversity. For instance, there is more diversity in the influenza A species then there is in the entire Filoviridae family, which consists of only two known genera: Ebola-like viruses and Marburg viruses. A better approach might be to calculate nucleotide diversity as a function of phylogenetic branch length or shared k-mer clusters within a target taxonomy node.

Finally, we averaged the sensitivities of microbes by genome type as shown in Table 6 . Note that the ssRNA-RT category includes only HIV-1. This chart demonstrates that creating signatures with high sensitivity becomes more difficult for target organisms with high mutation rates. 

Current real-time PCR assay design approaches produce signatures with sensitivities generally too low for clinical use. We suggest that a rigorous approach involving false positive and false negative analysis should be the standard by which an initial assessment of signature quality is made. Signatures must also regularly be reassessed as sequence data becomes available. For targets with wide nucleotide diversity, it becomes necessary to develop a set of signatures, for which we suggest a minimal set clustering approach that may also include signatures with degenerate/inosine bases.

NewRealTimePCRSigatures. Fifty Seven TaqMan PCR primer/probe combinations we predict to have higher sensitivity/specificity than current published assays.

Click here for file [http://www.biomedcentral.com/content/supplementary/1476-0711-7-18-S1.doc] Sensitivity by taxonomy level Figure 1 Sensitivity by taxonomy level. Each colored diamond represents a real-time PCR assay examined in this paper.

Black bars indicate the mean, grey bars indicate the median. Top and bottom of each box indicates 75th and 25th percentiles, and grey lines at whisker ends denote min and max values. The wide ranging sensitivities demonstrate both inconsistency in genetic diversity at a given taxonomy level, and inconsistency in signature design approaches.

Japanese encephalitis virus

Too many mismatches in either forward or reverse primer. Several strains have 3 mismatches at 3' end of forward primer in addition to internal mismatches

Reverse primer only has a blast hit to one strain (Angola71)

Saint Louis encephalitis virus 2007 Too many mismatches in the reverse primer, with 3 mismatches at 3' end as well as at other locations

Dengue virus 1 2006 Reverse primer does not have any BLAST hits to target

Dengue virus 2 2006 Forward primer has 3 or 12 mismatches at 3' end for most strains, the probe has BLAST hits to only 7 of the 57 genomes available, and reverse primer only has a BLAST hit to 1 genome

Dengue virus 4 2006 Too many mismatches in forward primer and in some cases the probe

Too many mismatches in forward primer. However, they are at the 5' end, so assay could still work for some strains

Pseudomonas aeruginosa 2004 No blast hits of probe to Pseudomonas aeruginosa

Probe is not between or even in close proximity to the forward and reverse primers

Stenotrophomonas maltophilia 2004 No blast hits of probe to Stenotrophomonas maltophilia

Probe only matches in 17 of 22 bases, which is unlikely to give a strong signal, since probe is unlikely to bind prior to the primers as desired for real time TaqMan chemistry. References 1

Basic local alignment search tool

Comprehensive DNA signature discovery and validation

The Perl Directory

Frequent Detection of Human Coronaviruses in Clinical Specimens from Patients with Respiratory Tract Infection by Use of a Novel Real-Time Reverse-Transcriptase Polymerase Chain Reaction

The Interferon Inducer Ampligen Markedly Protects Mice against Coxsackie B3 Virus-Induced Myocarditis

Coxsackievirus B4 Infection of Human Fetal Thymus Cells

Rapid detection protocol for filoviruses

Rapid Detection and Quantification of RNA of Ebola and Marburg Viruses, Lassa Virus, Crimean-Congo Hemorrhagic Fever Virus, Rift Valley Fever Virus, Dengue Virus, and Yellow Fever Virus by Real-Time Reverse Transcription-PCR

A 5' Nuclease PCR (Taq-Man) High-Throughput Assay for Detection of the mecA Gene in Staphylococci

Algorithm for the identification of bacterial pathogens in positive blood cultures by real-time LightCycler polymerase chain reaction (PCR) with sequence-specific probes. Diagnostic Microbiology and Infectious Disease

Development of quantitative gene-specific real-time RT-PCR assays for the detection of measles virus in clinical specimens

Limitations of TaqMan PCR for detecting divergent viral pathogens illustrated by hepatitis A, B, C, and E viruses and human immunodeficiency virus

Development of Mulitplex Real-Time Reverse Transcriptase PCR Assays for Detecting Eight Medically Important Flaviviruses in Mosquitoes

Development of Real-Time Reverse Transcriptase PCR Assays To Detect and Serotype Dengue Viruses

Comparative genomics tools applied to bioterrorism defense

Rapid Development of Nucleic Acid Diagnostics

Sequencing needs for viral diagnostics

Design and Validation of an H5 TaqMan Real-Time One-Step Reverse Transcription-PCR and Confirmatory Assays for Diagnosis and Verification of Influenza A Virus H5 Infections in Humans

Lion T: Real-Time Quantitative PCR Assays for Detection and Monitoring of Pathogenic Human Viruses in Immunosuppressed Pediatric Patients

Rapid reverse transcription-PCR detection of hepatitis C virus RNA in serum by using the TaqMan fluorogenic detection system

Rapid Detection of West Nile Virus from Human Clinical Specimens, Field-Collected Mosquitoes, and Avian Samples by a TaqMan Reverse Transcriptase-PCR Assay

Development of a Quantitative Real-Time Detection Assay for Hepatitis B Virus DNA and Comparison with Two Commercial Assays

Sensitive and accurate quantitation of hepatitis B virus DNA using a kinetic fluorescence detection system (TagMan PCR)

Comparison of two quantitative CMV PCR tests, Cobas Amplicor CMV Monitor and TaqMan assay, and pp65-antigenemia assay in the determination of viral loads from peripheral blood of organ transplant patients

Differentiation of herpes simplex virus types 1 and 2 in clinical samples by a real-time taqman PCR assay

Development of a flurogenic polymerase chain reaction assay (TaqMan) for the detection and quantitation of varicella zoster virus

Rapid and sensitive detection of mumps virus RNA directly from clinical samples by real-time PCR

Development of a Real-Time Reverse-Transcription PCR for Detection of Newcastle Disease Virus RNA in Clinical Samples

Transfer and Evaluation of an Automated, Low-Cost Real-Time Reverse Transcription-PCR Test for Diagnosis and Monitoring of Human Immunodeficiency Virus Type 1 Infection in a West African Resource-Limited Setting

Rapid Detection of Enterovirus RNA in Cerebrospinal Fluid Specimens with a Novel Single-Tube Real-Time Reverse Transcription-PCR Assay

Use of Applied Biosystems 7900HT Sequence Detection System and Taqman Assay for Detection of Quinolone-Resistant Neisseria gonorrhoeae

Comparison of a New Quantitative ompA-Based Real-time PCR TaqMan Assay for Detection of Chlamydia pneumoniae DNA in Respiratory Specimens with Four Conventional PCR Assays

A LightCycler TaqMan assay for detection of Borrelia burgdorferi sensu lato in clinical samples

Detection of Medically Important Ehrlichia by Quantitative Multicolor TaqMan Real-Time Polymerase Chain Reaction of the dsb Gene

We thank Beth Vitalis, Jason Smith, and Tom Slezak for helpful discussion and for encouraging this work, and Kari Allmon for entering the references. We gratefully acknowledge financial support from the Intelligence Technology Innovation Center. Lawrence Livermore National Laboratory is operated by Lawrence Livermore National Security, LLC, for the U.S. Department of Energy, National Nuclear Security Administration under Contract DE-AC52-07NA27344.

The authors declare that they have no competing interests.

GL found real time PCR signatures in the literature, wrote Perl scripts, and performed the analysis of published signatures. SG conceived of the research, designed new signatures, and provided guidance throughout the study.

Publish with Bio Med Central and every scientist can read your work free of charge http://www.ann-clinmicrob.com/content/7/1/18