key: cord-1009310-ccx61owl authors: Kapitula, D. S.; Jiang, Z.; Jiang, J.; Zhu, J.; Chen, X.; Lin, C. Q. title: Performance & Quality Evaluation of Marketed COVID-19 RNA Detection Kits date: 2020-05-01 journal: nan DOI: 10.1101/2020.04.25.20080002 sha: 2e5a0bb1180a893dcf76a4e55f4d479ef7933135 doc_id: 1009310 cord_uid: ccx61owl Compared to other coronaviruses, COVID-19 has a longer incubation period and features asymptomatic infection at a high rate (>25%). Therefore, early detection of infection is the key to early isolation and treatment. Direct detection of the virus itself has advantages over indirect detection. Currently, the most sensitive and commercially validated method for COVID-19 testing is RT-qPCR, designed to detect amplified virus-specific RNA. Reliable testing has proven to be a bottleneck in early diagnosis of virus infection in all countries dealing with the pandemic. Significant performance and quality issues with available testing kits have caused confusion and serious health risks. In order to provide better understanding of the Quality and performance of COVID-19 RNA detection kits on the market, we designed a system to evaluate the specificity (quantitation), sensitivity (LOD) and robustness of the kits using positive RNA and pseudovirus controls based on COVID-19 genomic sequence. We evaluated 8 Nucleic Acid qPCR Kits approved in China, some of which are also approved in the US and EU. Our study showed that half of these 8 kits lack 1:1 linear relationship for virus RNA copy: qPCR signal. Of the 4 with linear response, 2 demonstrated sensitivity at 1 Copy viral RNA/Reaction, suitable for early detection of virus infection. Furthermore, we established the best RNA extraction, handling and qPCR procedures allowing highly sensitive and consistent performance using BGI qPCR kits. Our study provides an effective method to assess and compare performance quality of all COVID-19 nucleic acid testing kits, globally. Since the novel Coronavirus (COVID- 19) was first identified in Wuhan, China, December, 2019, Chinese scientific communities have been making important contributions to the understanding of this virus from genomic sequence to protein structures, infection mechanism to clinical strategies and post-mortem pathology, etc. [4] [5] [6] Most significantly, Chinese epidemiologists have done an impressive job in contact tracing, testing, quarantining of all potentially infected people in China, the majority of whom came from Wuhan, and implementing a strict lock-down, effectively minimizing further spread to the rest of China. One of the biggest challenges in effective control of the COVID-19 outbreak is the fact that compared to similar viruses, it is more contagious due to its environmental stability; having longer incubation, higher rate of non-symptomatic infections as demonstrated from multiple independent sources 7 , making early detection of virus infection one of the most critical tactics in the battle against the virus. Recognizing the criticality and need early, the China Center for Drug Evaluation (CDE) announced a "Green Channel" for reviewing and approving COVID-19 testing kits and other bio-medical interventions such as clinical trial drugs in January 2020. At the time of submission, 23 diagnostic kits have been approved in China, of which 8 are based on quantitative Polymerase Chain Reaction (qPCR) using COVID-19 viral RNA sequence as templates and fluorescence detection. See Table 4 in the Tables section. These detection kits provided the Chinese CDC and related clinics the ability to test for the presence of infection. However, significant technical and performance quality issues with these testing kits have become evident. In China, the reliability of detecting COVID-19 using approved qPCR methods have been reported by first line healthcare professionals to range from 35-50% 8 . This situation resulted in the initial under-reporting of COVID-19 infected cases in China, which in turn triggered the change of diagnostic criteria by China CDC on Feb 11 from solely the RNA test result to relying on a combination of clinical symptoms including lung CT scan and RNA test 9 . These issues are now becoming evident in the pandemic management in the world. Continued global spread of the epidemic highlights the continual and present demand for sensitive and reliable virus detection without exception, including countries considered as leading bio-medical countries (i.e. EU nations and the US). Notably, the initial failure in quality virus testing kits distributed by US CDC is viewed as a critical mistake in the management of the crises by the US government resulting in compromised ability to test and trace the earlier infected population clusters before they spread to wider populations through community contacts. The failure in providing a sufficient amount of reliable quality testing tools has since been identified as a critical factor in the rapid spread of the virus in many parts of the world which is paralyzing healthcare systems and claiming hundreds of thousands of lives so far. Our study aims at providing objective evaluation and comparison of the quality and performance characteristics of 8 of the currently marketed COVID-19 nucleic acid detection kits in China based on qPCR and fluorescence detection. The parameters assessed include specificity (quantitation) of the testing kit to COVID-19 gene sequences, sensitivity (LOD), as well as the RNA extraction reagents and processing methodologies. The data (CT#) in Table 1 for each kit was assessed to determine the number of ACs providing a positive result (when performed per manufacturer's instruction). Kits were reviewed for Positive results for all relevant target genes, as well as non-target. Multiple and Single Positive results for the ORF1a/b regions were used to determine suitability as a robust quantitative analytical method. See Table 1 in the Tables section. The results show that kits from Shanghai Bio-Germ Medical Technology Co., Ltd, Beijing AB Technology Co., Ltd, Sansure Biotech Co., Ltd and BGI anneal on single area on ORF1a/b. The others amplified from multiple areas of ORF1a/b, indicating non-specific annealing and resulting in a non-linear sensitivity for virus RNA copy number (i.e. not one-to-one). Furthermore, only the kits from Zhongshan Daan Gene Co., Ltd and Maccura Biotech Co., Ltd could amplify from the Chinese National Standard (CSC). See Table 4 in the Tables section. The data (CT#) for each kit was assessed ( Table 2) to determine the number of positive results for both H (10 Copies) and L (1 copy) (when performed per manufacturer's instruction) and/or a single positive result for replicates. Only Kits from Sansure Biotech Co., Ltd and BGI can achieve low concentration (1 copy/reaction), Bio-Germ, LiveRiver and Maccura can achieve 10 Copies/rxn., and the rest were not reactive at or below 10 Copies/rxn. See Table 2 in the Tables section. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 1, 2020. . The results (Table 3) show that for the Tiangen and Qiagen extraction kits, RNA extraction and stability are consistent as claimed. The BGI qPCR COVID-19 Detection kit demonstrated highest fidelity in detection of both high and low copies/rxn. See Table 3 in the Tables section. In Vitro Diagnostic kits (IVD) are usually approved by regulatory authorities around the world under the category of medical devices. Commercialization of IVDs typically requires 3-5 years of product development, followed by a 1 to 2-year regulatory review period. Evaluation criteria include scientific rationale, technical considerations, product quality and safety, and consistency of performance for the intended purpose and markets including manufacturing capability (supply). The COVID-19 RNA kits approved under EUA had much shorter time for development and regulatory review, in some cases resulting in higher variability and quality issues. These issues often resulted in delays in diagnosis requiring repetitive testing using different testing kits and methods in order to confirm a positive infection as a result of the high rate of "False Negatives" observed in many of the approved testing kits. IVD industry leaders (i.e. Roche Diagnostics) are equally affected, although having one of the first COVID-19 IVD approved by the US FDA under EUA, the determined sensitivity, when compared head-to-head with other kits in our study, did not achieve single copy/reaction detectability. ROCHE reagents used in Cobas-SARS-CoV-2 10 achieved quantitative detection of the viral RNA, consistent with product claim (LOD of 10 copies/reaction) and confirmed in our study (data not shown), while both BGI and Sansure kits demonstrated LODs at 1 copy/reaction. Therefore, the existing RNA testing tools world-wide are at best insufficient in providing consistent performance suitable for reliable early detection and proper isolation, a key-criteria central to the control of the pandemic before an effective vaccine or treatment is available. In addition, lack of reliable correlation of clinical treatment with viral load could also affect decisions on treatment and post-recovery patient management. Our study provides an elegant design to define the most important performance characteristics of the RNA detection kits for COVID-19, which are specificity (quantitative), sensitivity (LOD), and robustness. Using this approach, all COVID-19 RNA detection kits can be compared with the currently approved kits to aid the decision in regulatory approval as well as clinical applications. Use of the E, N genes and ORF1a/b region of the virus genome as RNA based assay and internal controls in our study takes into consideration that these represent the most likely regions (coding for structural and nonstructural proteins, including those essential to virus functions) for Primer and Probe design 11 . This study design is viable for comparing and screening all new COVID-19 virus detecting reagents in the world to provide objective evaluation of the performance and quality of the new IVD tools. In conclusion, Jade Biomedical (Suzhou) Co, Ltd has identified the Best Total Solution (BTS) for COVID-19 RNA testing, which involves using Tiangen or Qiagen RNA extraction and BGI qPCR kits. This combination allows reliable detection of viral RNA at 1-10 Copies/reaction, which equates to 95-1000 Copies of viral RNA in 1 mL of test article. Coincidentally, BGI Real-Time Fluorescent RT-PCR kit from China is the only one of its kind to have obtained US FDA and EMA emergency approval so far 12, 13 . The Quality Control strategy and reliable performance by Jade Biomedical Laboratories demonstrated in this study will continue to allow contribution to the global need for more reliable and easily accessible COVID-19 testing. Direct clinical validation of our BTS is currently being planned in order to help with the on-going global clinical treatment strategy and intervention development by providing such specific, sensitive, and reliable viral load testing. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 1, 2020. . https://doi.org/10.1101/2020.04. 25.20080002 doi: medRxiv preprint The COVID-19 RT-PCR test kits are real-time reverse transcriptase polymerase chain reaction (rRT-PCR) tests for the quantitative detection of nucleic acid from SAR-CoV-2. The tests are shown to support detection in upper and lower respiratory specimens (e.g.nasopharyngeal or oropharyngeal swabs, sputum, lower respiratory tract aspirates, etc.). They are designed with specific primer and probe sets to detect from one to three regions in the COVID-19 genome, 1) Nucleocapsid (N) gene, seq. 28274-29533; 2) ORF1a and ORF1bvarious functional proteins, seq. 266-21555; and 3) Envelope Protein (E) gene, seq. 26245-26472 (Table 4) . Quantitative real-time qPCR is performed using the Roche LightCycler 480 real-time PCR System (LC480) 14 . RNA isolated from specimens is reverse transcribed to cDNA and subsequently amplified using one of the kits in Table 4 . The specific primer target(s) (amplicon) is/are amplified via Reverse Transcription (RT), denaturation, annealing, and extension of the primers followed by detection of the fluorescent reporter dye(s), per manufacturer's instructions. See Table 4 in the Tables section [15] [16] [17] [18] [19] [20] [21] [22] . To enable the objective performance evaluation of the kits, and due to non-disclosure by some kit manufacturers, a total of 8 synthetic RNA controls (AC) were synthesized, purified, and individually packaged into Lentivirus for RNA extraction studies, Pseudovirus (IC). The sequence design of these controls was based on RNA length supported by the Lentivirus, (Refer to Table 5 below for sequence specifics). Furthermore, to facilitate the ability to support single and multi-plex assay designs, each of the controls also contained the full-length N and E genes, as well as a specific sequence with partial overlap of the ORF1a/b sequence on both the 3' and 5' ends ( Table 5 ). The partial overlap was designed to provide sufficient detectability in the event the primers/probe combinations of the commercial kits were designed to anneal near the boundaries of the included ORF1a/b segment. In addition, the Chinese National Standard, a synthetic RNA (CSC) similar in construct of the AC was also included as a screening element. See Table 5 in the Tables section. SARS-nCoV-2 genomic sequence (MN908947.3) 3 was acquired from NCBI. All gene fragments used in this study were assembled by overlapping synthesized oligonucleotides. The whole ORF1a/b sequence was divided evenly into 8 fragments with 150bp overlaps in between each other. Genes E and N were added to the 3' end of each ORF1a/b fragments, then cloned into RNA producing plasmid pUC57 (Thermo Scientific, Carlsbad, CA) or Lentivirus transfer plasmid pLEX-MCS (Addgene, Cambridge, MA). To prepare DNA templates for IVT, the synthesized SARS-nCoV-2 fragments, flanked by T7 promoter at 5' end and 30 poly(A)s at 3' end, were amplified by Phanta Max Super-Fidelity DNA Polymerase (Vazyme, Nanjing). IVT of RNAs were using RNA TranscriptAid T7 High Yield Transcription Kit (Thermo Scientific, Carlsbad, CA) with extended DNase I treatment (1 hour). Then RNAs were purified by Monarch RNA Cleanup Kit (NEB, Ipswich, MA) and diluted to 1 X 10 6 copies/mL based on OD260 readout by Nanodrop One (Thermo Scientific, Carlsbad, CA). For Lentivirus packing, SARS-nCoV-2 sequence containing pLEX-MCS (Addgene, Cambridge, MA), psPAX2 (Addgene, Cambridge, MA) and pVSV-G (Addgene, Cambridge, MA) were co-transfected by RNAi-Mate (GenePharma, Suzhou) into HEK293T cells. Virus was harvested from culture medium 72 hours after transfection by filtration through 0.45μm filters (Merck Millipore, Nottingham, UK). The flow through was moved into 100kd Centrifugal Filters (Merck Millipore, Nottingham, UK) and spun @8000rpm in 4°C for virus concentration. Virus was treated by DNase I (Thermo Scientific, Carlsbad, CA) for 30 minutes before RT-qPCR titration. To quantify genomic RNA in SARS-nCoV-2 fragment replaced Lentivirus, the virus was lysed by Ezol (GenePharma, Suzhou) and extracted with chloroform. Purified RNA was then reverse-transcribed into DNA by M-MLV Reverse Transcriptase (Thermo Scientific, Carlsbad, CA). RT-qPCR reactions using Primer set S-LV5-FO-3 (5'-AGAGCGTCAGTATTAAGCGGG-3') and S-LV5-RE-3 (5'-TGTTTCTAACAGGCCAGGATT-3') were performed in Applied Biosystems 7500 Real-Time PCR System (Thermo Scientific, Carlsbad, CA). Copy number of virus was calculated by standard curve using plasmid DNA. Real-time fluorescence quantitative general reagent (Cat# GMRS-001 GenePharma, Shanghai) was used to detect copy number of RNAs. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted May 1, 2020. The kits were evaluated using a stepwise three-stage approach. All assays were performed in accord with the manufacturer's instruction using the LightCycler (Roche) (adjusted for color compensation for the specific Reporter Dye(s). Performed to verify the ability of the assay to be used for quantitative analysis and to determine the target region of the primer(s) and probe(s) for each manufacturer's kit. Only AC (RNA) was used for this stage of the study. Each assay included kit provided Positive and Negative Controls. As all commercial kits used have a claimed LOD of ≤ 1000 copies/reaction), Master Mix was prepared to which *1000 copies of AC were added, (note the AC, GP5 was spiked at 10 8 copies). Additionally, separate samples using CSC were run in parallel. Those kits demonstrated suitable for quantitative analysis (from Stage 1) were assayed with the relevant single ACs using two concentrations of spiked AC. Each assay included kit provided Positive and Negative Controls. Master Mix was prepared to which 2 copies/µl (@10 copies/rxn., (H) and 0.2 copies/µl @1 copy/rxn., (L) of identified (Stage 1) AC were added with samples being run in duplicate. For those kits with an established LOD at 1 Copy/rxn, the robustness of the testing was evaluated using 3 available RNA extraction kits, Tiangen 23 , Qiagen 24 , and Roche 25 . This stage was performed to identify the best combination of extraction reagent, method, storage and processing conditions, and RT-PCR reagents providing the most sensitive and reliable detection of low levels of COVID-19 infection, termed "Best Total Solution" (BTS) COVID-19 RNA Detection Method. All extractions were performed in accord with the manufacturer's instructions. The relevant IC (from Stage 1) was used for each of the kits assessed. Eight (8) samples were individually extracted for each kit. A Positive and Negative Control (included in the kits) were also prepared. Pseudovirus RNA Extraction was performed using IC @1000 Copy/mL Based on 90% extraction efficiency per manufacturer's claims, final RNA Extraction concentrations were assessed to be ~2.1 copy/µl. 5 µl of extracted solution (10 Copies/rxn.) were used for all assays and all samples were run in duplicate. Covid-19: four fifths of cases are asymptomatic, China figures indicate Clinical characteristics of 24 asymptomatic infections with COVID-19 screened among close contacts in Nanjing A new coronavirus associated with human respiratory disease in China Structure of the SARS-CoV-2 spike receptor-binding domain bound to the ACE2 receptor Novel Coronavirus Pneumonia Emergency Response Epidemiology Team. The Epidemiological Characteristics of an Outbreak of 2019 Novel Coronavirus Diseases (COVID-19) in China Zhong Hua Liu Xing Bing Xue Za Zhi Coronavirus disease 2019(COVID-19) situation report Available at Comparison and analysis of detection performance of 6 domestic novel coronavirus nucleic acid detection reagents National Health Commission of the People's Republic of China. Diagnosis and treatment of novel coronavirus pneumonia Genome composition and divergence of the novel coronavirus (2019-nCoV) originating in China The Real-Time Fluorescent RT-PCR Kit for Detecting SARS-2019-nCoV Letter of Authorization LightCycler 480 Real-time PCR Instrument Handbook 2019-nCoV (ORF1ab/N gene) Nucleic Acid Detection Kit Instruction book 16. XABT. 2019-nCoV ORF1ab/N Gene Nucleic Acid Detection Kit Instruction book 2019-nCoV Nucleic Acid Detection Kit Instruction book 2019-nCoV Nucleic Acid Detection Kit Instruction book 2019-nCoV Nucleic Acid Detection Kit Instruction book 2019-nCoV Nucleic Acid Detection Kit Instruction book 2019-nCoV ORF1ab/E/N Gene Nucleic Acid Detection Kit Instruction book 22. BGI. 2019-nCoV Nucleic Acid Detection Kit Instruction book 23. TIANGEN. TIANamp Virus RNA Kit instruction book High Pure Viral RNA Kit Handbook Shanghai Bio-Germ Medical Technology Co Table 1 -CT Number of 8 Kits Using GP and CSC RNA Controls * Sansure Pos. and Neg. control did not test in this assay. / represents no amplification