key: cord-0692760-7943cib4
authors: Asai, Satomi; Seki, Akira; Akai, Yasumasa; Tazawa, Hiromitsu; Kakizoe, Hidehumi; Ravzanaaadii, Mend-Amar; Miyachi, Hayato
title: Nationwide external quality assessment of SARS-CoV-2 nucleic acid amplification tests in Japan
date: 2021-11-17
journal: Int J Infect Dis
DOI: 10.1016/j.ijid.2021.11.022
sha: 52b5da791d468866fd893f14585cb5bbcf54269f
doc_id: 692760
cord_uid: 7943cib4

Objectives We conducted a nationwide external quality assessment (EQA) study of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) nucleic acid amplification testing in Japan. Methods A total of 563 public health and private sector laboratories participated. The EQA samples comprised 6 RNA and full-process controls. Results The overall agreements were 99.3% and 97.9% for the RNA and full-process controls, respectively. A total of 530/563 (94.1%) laboratories reported correct results; public health laboratories had the highest accuracy. Thirty-three laboratories reported at least one incorrect result (26 laboratories of medical facilities, 5 commercial laboratories, 1 public health laboratory, and 1 other). Sixteen laboratories of medical facilities that used a fully automated assay system failed to detect the presence of the full-process control, due to inherent insufficiency in the limit of detection (LOD). Other causes of incorrect results included failure to ensure the LOD (n = 13), error in result judging or reporting (n = 3), and error in sample handling (n = 1). Conclusions Performance was mostly dependent on the laboratory category and assay evaluation, particularly the LOD. Guidance should be developed based on these results, particularly in the phase of new entry into laboratory services for SARS-CoV-2.

In the ongoing outbreak of the coronavirus disease 2019 , access to reliable diagnostic assays and adequate testing capacity for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection is essential for preparedness and response strategies worldwide (World Health Organization 2021b). In Japan, the standard method for the detection of SARS-CoV-2 RNA has been an in-house polymerase chain reaction (PCR) method developed by the National Institute of Infectious Diseases (NIID) (National Institute of Infectious Diseases 2020). However, governmental and public health laboratories have a limited ability to perform the volume of tests that is needed. As a result, diagnosis and isolation of patients, contact tracing, and even treatment of symptomatic patients cannot be performed adequately. Therefore, rapid expansion of diagnostic testing for emerging pathogens into private sector laboratories has been demanded, which allows an increase in test volume and shorter turnaround times Matheeussen et al. 2020b ).

The government has promoted the use of nucleic acid amplification tests (NAATs), such as PCR, by implementing new measures such as emergency approval of in vitro diagnostic devices (IVDs) and coverage by national health insurance. A number of assay methods are covered, provided that the assay performance is comparable to that of the standard method, regardless of whether the test is an IVD or a laboratory-developed test (LDT) . However, NAATs are used in various facilities, with a number of new entries into laboratory services, and the quality and diagnostic performance of these tests has not been adequately validated. Further, the nucleic acid extraction methods, NAAT reagents, and thermocyclers used differ among laboratories.

External quality assessment (EQA) is a fundamental element to ensure reliable test results, especially when using Emergency Use Authorization diagnostic kits for newly emerging pathogens (Matheeussen et al. 2020a; Sung et al. 2020; Wang et al. 2020; Fischer et al. 2021; US Food and Drug Administration 2021) . The World Health Organization (WHO) has encouraged laboratories to participate in EQA schemes for this novel virus (World Health Organization 2021a). Here, we present the results of the first nationwide EQA on SARS-CoV-2 NAATs, which was a commissioned project by the Ministry of Health, Labour and Welfare of Japan. We also present data from a questionnaire survey that assessed the molecular testing capability of clinical laboratories conducting SARS-CoV-2 NAATs throughout Japan.

This nationwide EQA was conducted as a commissioned project by the Ministry of Health, Labour and Welfare of Japan. The Japanese Committee for Clinical Laboratory Standards was responsible for the EQA scheme plan and evaluation of the EQA results. All public health laboratories conducting SARS-CoV-2 tests were requested to participate, and these laboratories were included in the study by default. Private sector laboratories conducting SARS-CoV-2 tests were also recruited as participants.

A total of 128 public laboratories performing SARS-CoV-2 tests participated, including 65 regional health and environment institutes, 14 national quarantine stations, and 49 public health stations, along with 310 laboratories in medical facilities, 107 commercial laboratories (83 registered and 24 temporary), and 18 others, for example, research laboratories. EQA samples were shipped on November 15 and 16, 2020. All participants were asked to submit all data, including test results and other information, by November 21, 2020.

A positive RNA control and a pseudovirus-containing positive RNA full-process control were used as EQA samples to evaluate the process of amplification and detection and the entire process of extraction, amplification, and detection, respectively. The AcroMetrix TM Coronavirus 2019 (COVID-19) RNA Control (Thermo Fisher Scientific Inc., Fremont, CA, USA) was used as a positive RNA control sample. It was prepared by formulating genomic RNA from SARS-CoV-2 into a proprietary buffer and quantified using Bio-Rad (Hercules, 6 CA, USA) Droplet Digital TM PCR (ddPCR). The AccuPlex SARS-CoV-2 Verification Panel (SeraCare LifeSciences Inc., Milford, MA, USA) was used as a positive full-process control sample. Non-replicative encapsulated recombinant viruses were used to evaluate test proficiency and accuracy through the entire process, including extraction and amplification, and viral RNA was quantified using ddPCR. The viruses were formulated in viral transport media that consisted of Tris-buffered saline with glycerol, antimicrobial agents, and human proteins. This material must undergo extraction, similar to that of the patient sample.

In order to ensure the quality of EQA samples, all samples were handled as specified by the manufacturers, which were certified under ISO 13485 (Medical devices -Quality management systems -Requirements for regulatory purposes).

All samples were stored at −70°C until shipment. To validate the stability of EQA samples, one random panel was tested for confirmation of the property as specified by the manufacturers in the EQA round, i.e., 2 days after thawing (anticipated maximum shipping time) and after further storage at 4°C for 3 days (maximum expected storage time before testing at participating laboratories). To validate the homogeneity of EQA samples, three random panels were tested for confirmation of the property without significant differences among the first, the middle and last vials dispensed prior to distribution.

The EQA panel was composed of six samples. SARS-CoV-2 RNA control samples were diluted with a proprietary buffer to concentrations of 20 copies/μL (no. 1) and 10 copies/μL (no. 2). SARS-CoV-2 full-process control samples were diluted with a proprietary diluent to concentrations of 10 copies/μL (no. 4) and 20 copies/μL (no. 5). Nuclease-free water was used as a negative control (no. 3 and no. 6).

Protocols for SARS-CoV-2 NAATs varied among the participating laboratories regarding the assay components, including the RNA extraction platform, assay reagent, and instrument for amplification and detection. Participants were asked to choose one method used as a mainstay of routine tests in the laboratory and to test the material as they would clinical samples using their routine molecular assay workflows, with a minor modification for the EQA.

To ensure consistency of the EQA samples across this variety of combinations of assay components, three core protocols were prepared based on RNA extraction platforms, as follows. For column-based RNA extraction followed by amplification (i.e., PCR and Loop-Mediated Isothermal Amplification, LAMP), a set of SARS-CoV-2 RNA controls at total loads of 100 copies (20 copies/μL) and 50 copies (10 copies/μL) and one negative sample were input to evaluate the amplification process, and a set of SARS-CoV-2 full-process control samples at a total load of 2800 copies (20 copies/μL) and one negative sample were used to evaluate the entire process. For so-called direct PCR without extraction and purification followed by PCR amplification, the set of RNA controls and the set of fullprocess controls at a total load of 50 copies (10 copies/μL) each were input. For fully automated assay systems, SARS-CoV-2 full-process control samples at a total load of 2800 copies (20 copies/μL) with one negative sample were input. For SmartGene (MIZUHOMEDY Co. Ltd., Tosu, Japan), μTAS WAKO COVID-19 (FUJIFILM Wako Chemicals Co., Osaka, Japan), Aptima SARS-CoV-2 (Hologic, Inc., Tokyo, Japan), and TRCReady® SARS-CoV-2 (Tosoh Co., Tokyo, Japan) systems, a total load of 2800 copies was directly input, since the extraction efficiency of these assays was not disclosed at that time.

Sysmex Corp. (Kobe, Hyogo, Japan), which is accredited by the Japan Accreditation Board as a proficiency testing provider under ISO/IEC 17043, was responsible for the preparation and transport of the EQA material and aggregation of the EQA results. EQA samples were 8 shipped on dry ice with temperature monitoring by Timestrip UK Ltd (Cambridge, UK).

Delivery to all laboratories was completed within 12-36 h. Upon receipt, the samples were stored at 2-8°C until use within 7 days, according to the manufacturer's instructions. After thawing, refreezing was not allowed.

Participants were asked to submit their results and the workflow, including those of the extraction/amplification reagents and instruments, through the EQA program website (https://crm.fjcl.fujitsu.com/certification/login_ini.do; CRMate, Fujitsu Ltd., Tokyo, Japan).

In addition, information on routine quality management in the laboratory was also collected, including laboratory accreditation, qualification of personnel, standard operating procedures, evaluation of performance parameters at implementation, and routine quality control.

For qualitative evaluations, EQA participants were only asked to report whether the samples were SARS-CoV-2-positive or not. Correct results reported for each sample were evaluated for each peer group and laboratory category. A peer group was defined as over 10 laboratories using the same combination of assay kit and instrument. The laboratory categories performing SARS-CoV-2 tests were classified into public laboratories (regional institutes of health and environment, national quarantine stations, and public health stations), laboratories in medical facilities, commercial laboratories (registered and temporary), and others, for example, research laboratories.

Statistical analyses were conducted to examine whether the overall EQA performance correlated with the specific technical details provided by the EQA participants. Statistical analyses were performed by means of chi-square test using a standard software package (StatFlex, version 7.0; ARTEC Co. Ltd. (Osaka, Japan). When needed, the information about workflows and protocols provided by the EQA participants or manufacturers was reviewed to 9 be conclusive regarding applied kits or tests. If needed, participants were contacted to confirm or specify the provided data and information. However, the reported information was incomplete for all the participants. Therefore, the utilized datasets differed slightly in size among the statistical analyses of the different variables.

The extraction platforms in routine use included direct PCR (183, 32.5%), columnbased (138, 24.5%), proprietary systems (120, 21.3%), simple direct for LAMP (54, 9.6%), magnetic bead-based (53, 9.4%), isopropyl alcohol-based (7, 1.2%), and unknown (7, 1.2%).

Assay reagents and instruments with regulatory approval were used in 304 (54.0%) and 300 (53.3%) laboratories, respectively. The rates of validation and verification of the assays performed at implementation were 56.0% (145/259) and 48.0% (146/304), respectively. The limit of detection (LOD) was evaluated in some regional institutes of health 

The overall accuracies were 1243/1252 (99.3%) for the RNA control and 1168/1193 (98.0%) for the full-process control. The positive agreement in each sample ranged from 96.4% to 99.8%, and the negative agreement ranged from 99.4% to 99.8% (Table 1) There were a total of 18 peer groups with over 10 laboratories using the same combination of assay kit and instrument (Table 3) .

Inconsistent results in duplicate assays of the set of SARS-CoV-2 RNA control samples (nos. 1 and 2) were reported in 1.3% (2/150) and 5.3% (8/151) of tests, respectively.

Assay reagents used included the Loopamp (SARS-CoV-2) detection kit (n = 4), SARS-CoV-2 Direct Detection RT-qPCR Kit (n = 3), and Ampdirect 2019-nCoV detection kit (n = 1).

A total of 33 laboratories reported at least one incorrect result. All 16 laboratories of one peer group, which were categorized as laboratories of medical facilities, used a fully automated cartridge assay system (μTAS WAKO g1 and μTAS WAKO COVID-19) and failed to detect the presence of the SARS-CoV-2 full-process control (no. 5) due to inherent insufficiency in the LOD. Therefore, we performed further assessment of the remaining 17 laboratories for qualitative analysis after omitting the results from these laboratories.

Analysis of submitted data and information acquired from interviews with the individual laboratories revealed that causes of incorrect results included error in pre-or postexamination processes, including errors in sample handling (n = 1) and result judging or reporting (n = 3). In 13 laboratories with incorrect results, a failure to ensure the LOD with or without resultant inappropriate judgement criteria led to incorrect results. Seven laboratories had false-negative results for EQA sample no. 2, including laboratories using the Loopamp EXIA/LF-160 and Loopamp (SARS-CoV-2) detection kit (n = 3), the Takara Dice/System and SARS-CoV-2 Direct Detection (n = 1), the cobas z 480 and LightMix Modular SARS-CoV (COVID-19) (n = 1), the QuantStudio5/5Dx and LightMix Modular SARS-CoV (COVID-19) (n = 1), and the QuantStudio5/5Dx and NIID (n = 1). The three assay systems yielding false-negative results for EQA sample no. 4 included the Loopamp EXIA/LF-160 and Loopamp (SARS-CoV-2) detection kit (n = 1), the 7500Fast (Thermo Fisher Scientific) and SARS-CoV-2 Direct Detection (n = 1), and the LightCycler 96/TaqMan48 and Ampdirect 2019-nCoV detection kit (n = 1). The assay system yielding a false-negative result for EQA sample no. 5 was the Smart Gene and Smart Gene nCoV Detection test cartridge, and two false-positive results for EQA sample no. 6 were produced by the Takara Dice/System and SARS-CoV-2 Direct Detection (n = 1) and the LightCycler 96/TaqMan48 and Ampdirect 2019-nCoV detection kit (n = 1). Most of these laboratories used a Ct cutoff of 40, following the manufacturer's instructions (5/7, 71.4%).

In the ongoing outbreak of COVID-19, reliable detection of SARS-CoV-2 is essential for patient care and interruption of transmission chains. This increase in the need for testing has led to an expansion in the assay systems and laboratories used to perform tests, making continuing quality improvement essential. To this end, we conducted a nationwide EQA of SARS-CoV-2 nucleic acid amplification testing in Japan, which was a commissioned project by the Ministry of Health, Labour and Welfare. The EQA in this report is unique because of its nationwide scale, including both public health and private sector laboratories conducting SARS-CoV-2 testing. The result of this EQA was well-guided to support the responses of participant laboratories and others to minimize the ongoing outbreak. EQA of assay performance allows participating laboratories to assess the quality and identify the weaknesses and strengths of the currently used diagnostic methods (Matheeussen et al. 2020a; Sung et al. 2020; Wang et al. 2020; Fischer et al. 2021 ; US Food and Drug Administration 2021). Based on our results, a guidance document for internal process control in laboratory practice was developed (Ministry of Health, Labour and Welfare of Japan 2021).

This study included both public health and private-sector laboratories. The proportion of correct results was the highest in public health laboratories, followed by private sector laboratories. The intended purposes of tests in the public health and private sector laboratories are different. Public health laboratories have long been responsible for performing diagnostic tests for newly emerging pathogens and are familiar with using LDTs.

By contrast, private sector laboratories, such as laboratories of medical facilities, have a history of performing diagnostic tests using IVDs approved by a regulatory body, since laboratory tests performed using IVDs are covered by the national health insurance in Japan.

Therefore, these laboratories are not as familiar with LDTs as public health laboratories. The assay performance evaluation of LDTs and IVDs requires validation and verification, respectively. Performance was mostly dependent on the laboratory category and assay evaluation, particularly the LOD. In this context, the laboratory evaluation of the LOD was found to be significantly lower in private sector laboratories than in public health laboratories.

Most of the laboratories used a Ct cutoff value of 40, following the manufacturer's instructions. However, the Ct value can vary depending on the extraction and assay reagents and the instrument (Engelmann et al. 2021; Fischer et al. 2021) . This could increase the risk of incorrect results in specimens with lower viral loads. Therefore, laboratories must evaluate the assay performance, including the appropriate cutoff value.

As the performance in an EQA is the outcome of many steps in the entire workflow, each step of the workflow should be assessed. These steps include pre-examination, examination, and post-examination, for example, resuspension of the panel samples, extraction method/kit used, change in nucleic acid concentration during extraction, and type 13 of NAAT assay, including the genomic target and the number of different genomic targets or tests performed. In this EQA, a positive RNA control and a positive full-process control were used as EQA samples to evaluate the process of amplification and detection and the entire process of extraction, amplification, and detection, respectively. The RNA control sample with a lower viral load yielded more unstable results. This suggests that reliability in terms of repeatability of the assay comes partly from the amplification process itself.

Thirty-three laboratories reported at least one incorrect result; most were privatesector laboratories. Our results indicated that performance was mostly dependent on the assay evaluation, particularly on the LOD at implementation.

We developed guidelines for laboratory practice based on the results of the EQA as well as a questionnaire survey that assessed the molecular testing capability of clinical laboratories conducting SARS-CoV-2 NAATs. The content of the guidelines focused on compliance with requirements of the Medical Care Service Act amendment for quality assurance of laboratory tests, assay performance evaluation at implementation, quality monitoring, standard operating procedure, personnel training on quality assurance, and competence of molecular laboratories (Ministry of Health, Labour and Welfare of Japan 2021; Quality Control for Molecular Diagnostics 2020). Laboratories that reported incorrect results on their qualitative tests were asked to take corrective actions by reevaluating their nucleic acid extraction protocols and internal quality control processes according to the laboratory guidance. A follow-up EQA has been planned to confirm improvement to acceptable results.

Fully automated systems are commercially available as point-of-care tests. The LOD can differ significantly among these systems. Therefore, although the extraction efficiency was not disclosed for such systems, a total load of 2800 copies was directly input for this EQA. Nevertheless, all 16 laboratories of one peer group using the fully automated cartridge assay system μTAS WAKO COVID-19 failed to detect the presence of the SARS-CoV-2 full-process control. The LOD for this system reported by the manufacturer was 150 copies/assay. Therefore, a cause of the false-negative results was attributed to inherent insufficiency in the LOD. In this study, SeraCare's reagents were used as the full-process control as a pseudovirus, which is classified as an alpha virus different from SARS-CoV-2.

This could have affected the EQA results.

In Japan, a person does not qualify for free-cost testing unless presenting with COVID-19 symptoms or deemed to have been in close contact with a confirmed case.

Therefore, asymptomatic persons must pay for a test out of pocket in the clinic, which generally costs $200-400. Meanwhile, a number of low-cost facilities have become available, offering no-frills testing for a fraction of the price at a regular clinic. Since the service is provided with low cost and ease of use, it is proving very popular. However, whether the test has been adequately implemented and operated is unknown, and the quality of the test results is questionable. Those facilities were not the targets of this EQA. An EQA should enroll such low-cost facilities to assess their actual performance to confirm acceptable results.

Continuous improvement of test quality of all laboratories involved in SARS-CoV-2 diagnostic testing, expansion of tests, and even a broad screening program for the population are needed to control the ongoing COVID-19 pandemic.

In conclusion, this report summarizes the nationwide EQA of SARS-CoV-2 NAATs, which included both public health and private sector laboratories. Performance was mostly dependent on the category of the laboratory and assay evaluation, particularly the LOD. This study indicates that EQAs should be performed for all laboratories involved in SARS-CoV-2 diagnostic testing on a regular basis for the evaluation of potential weaknesses in SARS-CoV-2 molecular testing procedures. Development and use of guidance for laboratory practice based on survey results should be considered, particularly during new entry into 15 laboratory services for emerging pathogens, such as SARS-CoV-2. This action will help to increase the quality of test results and thus contribute to the quality of patient care and interruption of transmission chains.

We are especially grateful to Sysmex Corporation, KBBM Inc., and Fujitsu Co. Ltd. for their cooperation with this work.

This work was financially supported as a commissioned project by the Ministry of Health, Labour and Welfare of Japan in collaboration with the Japanese Committee for Clinical Laboratory Standards . The funder had no roles in the study design; in the collection, analysis or interpretation of data; in the writing of the report; or in the decision to submit the article for publication.

HM: consultancies for BM, Inc. and SRL, Inc.; AS: consultancies for RMA; HT: stock ownership of SRL, Inc. All other authors have no conflicts of interests to declare.

The requirement for ethical approval was not applicable because patient identifiable information and patient specimens were not used. Written informed consent was not obtained. EQA: external quality assessment; cps: copies; polymerase chain reaction (PCR): polymerase chain reaction

Detection of 2019 novel coronavirus (2019-nCoV) by real-time RT-PCR

Preanalytical issues and cycle threshold values in SARS-CoV-2 real-time RT-PCR testing: should test results include these?

Variable sensitivity of SARS-CoV-2 molecular detection in European expert laboratories: external quality assessment

RECOVER project and collaborating networks. International external quality assessment for SARS-CoV-2 molecular detection and survey on clinical laboratory preparedness during the COVID-19 pandemic

Preparedness of European diagnostic microbiology labs for detection of SARS-CoV-2

Manual for quality control of the detection of Pathogen 2019-nCoV.; 2021

Manual for the detection of Pathogen 2019-nCoV ver.2.6, 2020

Quality control for molecular diagnostics (QCMD), Respiratory Special

Nationwide external quality assessment of SARS-CoV-2 molecular testing

COVID-19) EUA information

Limits of detection of 6 approved RT-PCR kits for the novel SARS-Coronavirus-2 (SARS-CoV-2)

Coronavirus disease (COVID-19) technical guidance: laboratory testing for 2019-nCoV in humans

World Health Organization. 2019 novel coronavirus. nCoV. strategic preparedness and response plan