key: cord-0684697-qgv0p1v2
authors: MacLean, A. R.; Gunson, R.
title: Automation and standardisation of clinical molecular testing using PCR.Ai – A comparative performance study
date: 2019-11-30
journal: Journal of Clinical Virology
DOI: 10.1016/j.jcv.2019.08.005
sha: 05175c253f3716d1f0cc02d6b106a426cadd49d6
doc_id: 684697
cord_uid: qgv0p1v2

Abstract Background We undertook a prospective clinical study to evaluate PCR.Ai’s (www.pcr.ai) accuracy and impact when automating the manual data-analysis and quality control steps associated with routine clinical pathogen testing using real-time PCR (qPCR). Objectives We evaluated the impact of PCR.Ai when used as the final interpretation/verification step for routine in-house qPCR tests for respiratory pathogens and for norovirus for a total of 22,200 interpretations. Study Design We compared PCR.Ai to our existing manual interpretation, to determine accuracy and hands-on time savings. PCR.Ai was accurate. Results and Conclusions There was 100% concurrence between validated respiratory virus and norovirus detection by our manual routine analysis method and PCR.Ai. Furthermore, there were significant routine savings with PCR.Ai of 45 min/respiratory run and 32 min/norovirus run. Our conclusion is that PCR.Ai is a highly accurate time-saving tool that reduces complexity of qPCR analysis and hence the need for specialists and hands-on time. It demonstrated capabilities to enable us to get results out more quickly with lower costs and less risk of errors.

qPCR has now become a core component of most diagnostic virology laboratories' testing repertoires. It remains the most popular method for molecular detection of disease and is used globally for pathogen detection. It is considered the gold standard for molecular detection due to its high assay specificity and sensitivity and in our lab comprises 75% of sample testing. However, qPCR methods are still largely in-house developed and maintained. Almost all of this testing method is now automated by off-the-shelf machinery and consumables. However data-analysis and associated quality control still need to be done manually. Manual interpretation is also required for most commercial qPCR assays (e.g. Focus Diagnostics, CDC kits, etc.). This final interpretation process is complex requiring expertise and rigorous oversight, so is a burden on laboratory staffing. The complex interpretation process can be off-putting to laboratories considering implementing such assays.

Automated solutions for this stage have been attempted, however most have been unsuccessful. For example, some qPCR platforms will set a threshold and then any traces (output curves) that cross this threshold will be deemed positive. However, such systems are in danger of either missing low level positives or can result in false positives, consequently, these automated systems still need to be checked by a user. Furthermore, these systems do not offer whole system interpretation of results (i.e. interpret runs in relation to the control results) and QC (e.g. interpretation of runs in relation to laboratory-based rules such as Westgard).

As a result, most laboratories using qPCR methods and commercial kits still interpret and validate results manually. The methods used by laboratories are likely to differ. In our laboratory this is a multi-stage process that incorporates multiple members of staff, and systems for every run, following specific per-test standard operating procedures (SOPs) -(detailed description in Materials and Methods).

Due to its complex nature, mistakes can be made and there may be a lack of standardisation between users and across labs. It frequently requires the additional input of expert, often senior, staff. Large amounts of paperwork are required to document the processes. In the era of pressures and "doing more with less" anything that simplifies or automates this aspect of qPCR methodology would be welcomed. Clinical qPCR labs still use significant resources to ensure consistent manual analysis and reporting. Processes remain similar to those described some ten years ago by the Mayo Clinic [1] : T SOPs are required for each qPCR test;

• Identifying a technical expert to provide one-on-one training for qPCR is critical;

• Ensuring proficiency in analysis and use of qPCR instrument software, including for samples with unusual results.

In addition, most tests require review by two (or more) scientists. Regular proficiency testing, including ensuring accurate manual analysis of data, is also necessary.

PCR.Ai (www.pcr.ai) is a novel system created by diagnostics.ai (59a Brent Street, London, UK; www.diagnostics.ai) that is designed to automatically and rapidly interpret qPCR curves. It is carried out directly on the qPCR platform after the run is complete, using either an on-site or internet server, and interfaces with the Abbott Laboratory Information Management System (LIMS) to ensure results are downloaded. The system is described in further detail below (Study Design).

The aim of this prospective clinical study was to compare PCR.Ai to the West of Scotland Specialist Virology Centre (WOSSVC) existing interpretation/verification practice, to determine its accuracy and hands-on time from initial qPCR output to result download. Two inhouse-developed, multiplexed, qualitative qPCR tests were used in the study: one for the detection of respiratory pathogens (run on the ABI7500) and the other for detection of norovirus (run on the Rotor-Gene Q). These tests were chosen for this study as they are commonly used, high-throughput assays, which are run on two commonly used qPCR platforms and in one month produced 22,200 test results for comparison purposes. Fig. 1 below compares and contrasts the data analysis methods for WOSSVC and the PCR.Ai system. For each task in the figure it is indicated whether the task was completed by the technician (ie manually) or done by the application (ie automated). Both systems were interfaced with the Telepath LIMS system.

3.1.1.1. Description of the PCR.Ai analysis process. PCR.Ai is a novel tool that automatically interprets qPCR amplification curves providing rapid and standardised results. PCR.Ai identifies positive and negative amplification curves and highlights ambiguous results requiring further analysis, by analysing each curve shape both individually and in the context of run/assay history. The system automatically validates qPCR runs by assessing positive controls using user-defined rules such as Westgard Medlab QC 3 St Dev rule, interprets the IC values and examines negative controls. It can also verify qPCR runs by assessing quantitation standards and will quantify positive traces by comparing them to standard curves. After analysis is complete PCR.Ai interfaces with the LIMS to enable result download. PCR.Ai was accessed via a physical server connected directly to the qPCR machine (in this case the ABI7500 and Rotor-Gene Q), although the company also offers a cloudbased product, and automatically interpreted data directly on the qPCR platform following run completion. Conveniently, PCR.Ai is able to interpret multiple tests on one 96 well plate.

For interpretation of each assay, PCR.Ai is calibrated to determine positive, negative or indeterminate samples by analysing endpoint dilution series of the targets in the multiplex to be used. It is further calibrated by using a large number of qPCR amplification curves (typically containing at least 200 positive and 200 negative results) from previous runs that have been assessed on the PCR platform to be used. If necessary, this is then further refined by analysing a number of prospective runs, and comparing outcomes to the manual method. On completion of this calibration, PCR.Ai can then be used as often as required for the specific test to provide automated analysis.

Conventional interpretation of qPCR output using qPCR instrument software (we used both the ABI7500 and Rotor-Gene Q in this study) is a complex manual process. It requires analysis by at least two scientists for each run -a junior for initial interpretation and a senior to analyse ambiguous results and validate the run. Fig. 1 outlines the multiple manual steps and software packages involved in conventional analysis of qPCR instrument output data for clinical reporting. Assays used

The WOSSVC respiratory assay screens for 16 pathogens plus an internal control in 5 multiplex qPCR assays using the ABI7500 instrument. For each multiplex assay, a one-step qPCR reaction was performed as described previously [2] .

Norovirus is screened using a single qPCR multiplex assay [3] . These multiplexes and number of runs/samples for each are described in Tables 1 and 2 .

Currently, these assays are performed and manually analysed/interpreted using the qPCR instrument software (and with other QC packages such as MedLab QC).

Over a period of 1 month (mid April-mid May 2017), 20 respiratory and 20 norovirus runs were analysed by the WOSSVC biomedical scientists by both the normal manual methods outlined above and by PCR.Ai. For both systems, results were analysed in terms of positive controls passing Westgard (1:3S rules), negative controls showing no contamination, internal control failures and the positive/negative status of each sample. Once this was carried out the results generated by both methods were compared for accuracy. The same biomedical scientist (in total seven to minimise bias) carried out both analyses on each run; carrying out the manual analyses first as these were the results downloaded into LIMS, followed immediately by PCR.Ai. Each result was analysed only once. For the comparison of both methods all the final results were reviewed by one experienced clinical scientist to check the results of the biomedical scientist. Here the manual and PCR.Ai results were analysed on separate days to prevent bias. Where there were discrepant results, a second experienced clinical scientist reanalysed the run and the two clinical scientists decided the final status of these discrepant results and which method had correctly analysed these discrepant results.

Time auditing was carried out by recording the time taken for various stages involved in both manual and PCR.Ai analysis. To obtain this timing data, the following data was recorded by the biomedical scientist/laboratory technician carrying out the test analysis:

• System (ABI/Rotor-Gene/PCR.Ai). 

Tables 3 and 4 details results of the comparison (described above) between WoSSVC and PCR.Ai qPCR evaluation methodology for the 22,200 amplification curves (20,400 respiratory assay and 1800 norovirus) that were assessed by both methods. The figures refer to the number of positive traces in the ABI7500 or rotorgene analysis with the discrepancy referring to the different number of positives detected by PCR.Ai.

In no cases was the PCR.Ai analysis incorrect. There were a small number (5) of results where the original manual analysis was wrong There were a small number of differences in results when comparing Westgard checks (0.5% for respiratory and norovirus) and internal control failures (5 more for PCR.Ai for norovirus and 15 more for respiratory), which reflects positively for PCR.Ai (see discussion for details).

Over the 20 runs there were time savings of 45 min per run for the respiratory screen and 32 min per run for norovirus (see Tables 5 and  6 ).

qPCR methods in clinical virology are mostly in-house developed. Various adaptations have aided automation, but result interpretation and verification are still largely performed manually. Manual interpretation is also required for most commercial qPCR assays (e.g. Focus Diagnostics, CDC kits, etc.). Consequently, the complex interpretation process can be off-putting to laboratories considering implementing such assays.

qPCR interpretation can take time and requires expert staff, and is therefore a cost burden for the laboratory. This can also impact the test turnaround-time (TRT) and throughput, which in turn can affect the clinical management of a patient. Therefore, any successful developments that simplify/automate this process would be welcome.

PCR.Ai is a new system that aims to automate qPCR interpretation. We compared it to our current manual interpretation process (outlined above) for the interpretation of two typical in-house qPCR assays, carried out on different qPCR platforms. The main findings are discussed below.

Our data shows that PCR.Ai is as accurate as our manual-based system and is unaffected by the qPCR platform used. We assessed PCR.Ai prospectively on a total of 22,200 clinical results over a onemonth time period, with 100% concordance between the PCR.Ai and conventional manual interpretation. In general, interpretation of the positive and negative controls was also concordant as was the validation/verification decision/outcome. Slight differences in control interpretation arose, with PCR.Ai being more accurate as it used current data (from the start of the PCR.Ai study) to set Westgard ranges, whereas manual analysis used historical data from the introduction of the control to set ranges.

Slight differences were noted in IC interpretation between PCR.Ai and the manual method, with PCR.Ai detecting slightly more samples as inhibited in comparison with the manual method. It is likely that this is a reflection of the different methods used to assess sample inhibition. The manual method compares the Ct value of the internal control to a retrospective rule based on the first 20 runs with that batch of internal control. In our opinion the PCR.Ai method is more likely to be accurate, as unlike the manual method it automatically calculates the average based on samples in that run, giving a tighter range. In addition PCR.Ai analyses IC trace quality as well as Ct. As a result, PCR.Ai is likely to pick up inhibited samples that could be missed by the rule-based system. As this was a study the results of PCR.Ai were not allowed to influence clinical release of samples results.

Although the study showed similar accuracy between the two methods where the experienced biomedical scientists were performing optimally, in practice, PCR.Ai is likely to be more accurate than lab staff since it will be unaffected by operator fatigue or time pressure (e.g., end of the day or on-call interpretation) and level of operator expertise. Furthermore, it should be noted that the WoSSVC has strict qPCR run interpretation guidelines with in-built double checks that may be over and above that in place in other laboratories (perhaps no double checks etc). PCR.Ai was also able to interpret large mixed qPCR plates (i.e. plates containing > 1 multiplex assay with multiple controls), as encountered when assessing the respiratory qPCR. When questioned, staff found it to be easy to use and required minimal training (data not shown).

Our data also showed the PCR.Ai system to be quicker than the established manual system. For the respiratory qPCR, there was an average time-saving of 45 min per run. A saving of 5 min per run was achieved for the initial interpretation process (i.e. setting the threshold and identification of problem curves). A far larger saving of 40 min per run was achieved for the verification/validation steps (e.g. second check and entering control data into Westgard). This probably reflects the fact that PCR.Ai does this automatically, whereas the manual system is reliant on the second checker reviewing the whole run. During our review, further delays were observed when the second checker was unavailable to carry out checks and Westgard entry (e.g. breaks, other runs to interpret, meetings, end of shift, absence etc). Applying these savings to a whole year of respiratory service would result in a time saving of 160 h per year based on one run per day over a 5 day week. In reality, we tend to run more than one respiratory qPCR per day and offer our service 7 days a week. Similar savings of 32 min per run were achieved for the norovirus qPCR (the lower time savings likely reflect that this is a more straightforward assay to interpret due to it having less targets and associated controls). If we were to apply PCR.Ai to the remainder of our qPCR tests (> 20) this would significantly increase the overall savings to the laboratory. We would expect that in laboratories with less qPCR expertise and efficiency, time savings would be greater.

PCR.Ai is straightforward to learn and use. Expertise is only required for the final check of each run (i.e. are ambiguous traces negative, positive or indeterminate). Currently in the WOSSVC we use highly experienced biomedical scientists for the entire process. The ease of use of PCR.Ai may allow us to use less skilled staff to do the initial qPCR process download. We would then only require more skilled staff for the second check. This could lead to further staff savings and would allow us to redistribute skilled staff within the department. It would also have a clinical impact as tests will have better TRT and a higher throughput, as more samples can be processed per day. This will lead to further cost benefits relating to improved patient care. It should increase the number of tests which can be performed on our outof-hours service. It should also enable labs that do not have qPCR specialists to run qPCR assays if ambiguous results are repeated (or reviewed by an off-site expert).

• In our study, PCR.Ai calibration required the user to send a large database of historical qPCR run data to PCR.Ai for analysis. This includes clinical samples and/or dilution series. PCR.Ai analyses the run data to learn to identify positive, negative and ambiguous samples. Depending on the assay complexity, this can take time (weeks to months). PCR.Ai have already begun to introduce methods that use less clinical data and a small number of dilution series to calibrate the learning machine and we will evaluate the performance of this in due course.

• The assays assessed here are qualitative assays. We wish to assess PCR.Ai on quantitative assays to determine how well it works on these more complex assays.

• PCR.Ai works with most qPCR machines. It would be useful to see how well PCR.Ai works with other platforms (e.g. Roche LightCycler or Bio-Rad CFX).

• Data examined here all were produced by one primer/probe lot. We plan to investigate PCR.Ai's performance with lot-to-lot reproducibility and this may need to be included in our validation of new primer/probe lots.

Although not often discussed in the literature, we find that in practice different in-house qPCR tests vary in their ease of interpretation and therefore different tests may be more burdensome for staff than others. This could be due to assays being "messy" and having many ambiguous traces to interpret. Furthermore, in our laboratory, quantitative assays are often considered by staff to be more complex than qualitative assays and take longer to interpret. We have also encountered differences between the software used by different qPCR platforms, which can in turn affect the ease of result and run interpretation. Consequently, we find that some staff attempt to avoid particular assays, whereas others can become stuck on one particular assay as the resident expert. Taken together such issues can result in variable expertise between the "expert" staff members. Use of PCR.Ai, should standardise the interpretation of all in-house qPCR assays within the laboratory irrespective of test complexity or platform used. Tests will no longer be only under the auspices of particular staff. Furthermore, its ease of use should allow laboratories to use less skilled and experienced staff to interpret such assays. This is beneficial to laboratories already with such assays and could lead to restructuring of staff within a department. It is also beneficial to laboratories without such assays or staff expertise, as now they will be easier to implement and interpret, and lead to better access to qPCR technology (either in-house or commercial kits). If PCR.Ai was to become widely adopted, there would be better service and standardisation by and between labs -improving tests and patient care by region, nationally and internationally.

PCR.Ai offers numerous other benefits to the laboratory. It can link directly with laboratory LIMs, reducing manual steps still further. This should lead to a reduction in transcription errors and improved test TRT. PCR.Ai stores all the qPCR run data on hardware/in the cloud, thus providing long-term records of each qPCR run as well as all related decisions. For example, for each run it is possible to see the user who carried out the run, any decisions relating to samples or controls (and who made these decisions), as well as the outcome. Presently it would be difficult for most laboratories to keep such information. This aspect is excellent for various accreditation systems (e.g. ISO15189 as now mandated by UKAS). Furthermore, as all the PCR.Ai data is stored in a single database, it is simple to automatically produce reports that currently are done manually -for example validation reports for new assays, controls, EQA panels or other qPCR reagents. In future, it could possibly be used to highlight unusual result patterns relating to particular users (which could aid targeted training), contamination or unexpected outbreaks.

Use of PCR.Ai will result in reduced paper use within laboratories. For example, we perform most interpretation using paper. Thus many pages are printed per run, and run data is stored for 12 months until storage is full. Keeping such data in the related PCR.Ai cloud also provides excellent security should qPCR platforms break down, reduces storage requirements, is eco-friendly and reduces costs.

We have determined that PCR.Ai is a highly accurate qPCR interpretation tool. It is a valuable time-saving tool, by reducing the complexity of qPCR interpretation and validation and as such the need for specialists and hands-on time. In practice it should enable us to get results out more quickly, at a lower cost and with improved standardisation.

Dr Alasdair MacLean carried out the majority of the laboratory work.

Dr Rory Gunson initiated the study and was extensively involved in data analysis. 

Real-Time PCR in clinical microbiology: applications for routine laboratory testing

During the summer 2009 outbreak of "swine flu" in Scotland what respiratory pathogens were diagnosed as H1N1/2009?

Increased norovirus activity in Scotland in 2012 is associated with the emergence of a new norovirus GII.4 variant

Both authors work for NHS Greater Glasgow and Clyde.

None.