key: cord-1003061-y0zskvgi
authors: Petrillo, Mauro; Querci, Maddalena; Tkachenko, Olga; Siska, Ioana-Raluca; Ben, Enrico; Angers-Loustau, Alexandre; Bogni, Alessia; Brunetto, Antonino; Fabbri, Marco; Garlant, Linda; Lievens, Antoon; Munoz, Amalia; Paracchini, Valentina; Pietretti, Danilo; Puertas-Gallardo, Antonio; Raffael, Barbara; Sarno, Eleonora; Tregoat, Virginie; Zaro, Fabrizio; Van den Eede, Guy
title: The EU one-stop-shop collection of publicly available information on COVID-19 in vitro diagnostic medical devices
date: 2020-11-03
journal: F1000Res
DOI: 10.12688/f1000research.27308.1
sha: b3360104a284187175ab3e9666351e98de15c5e6
doc_id: 1003061
cord_uid: y0zskvgi

The JRC COVID-19 In Vitro Diagnostic Devices and Test Methods Database, aimed to collect in a single place all publicly available information on performance of CE-marked in vitro diagnostic medical devices (IVDs) as well as in house laboratory-developed devices and related test methods for COVID-19, is here presented. The database, manually curated and regularly updated, has been developed as a follow-up to the Communication from the European Commission “Guidelines on in vitro diagnostic tests and their performance” of 15 April 2020 and is freely accessible at https://covid-19-diagnostics.jrc.ec.europa.eu/.

The Communication from the Commission "Guidelines on in vitro diagnostic tests and their performance" 1 , published on 15 April 2020, states the following under the Further Actions Needed section: "The Commission, supported by the ECDC, health technology assessment experts and in vitro diagnostics competent authorities, will assist Member States with a centralised overview of available information on test performance and act as a single point of contact for management of this information. Taking stock of the state of the art on a regular basis will support Member States' informed decisions on national testing strategies, as well as support the continuous development of devices by manufacturers."

As an initial step in collecting performance information of devices and in house methods, to address the above need, European Commission services (Directorate-General for Health and Food Safety [DG SANTE], Directorate-General Joint Research Centre [DG JRC], Directorate-General for Research and Innovation [DG RTD]), together with the European Centre for Disease Prevention and Control (ECDC), several experts from in vitro diagnostics competent authorities and from the European Network for Health Technology Assessment (EUnetHTA) 1 , published the working document "Current performance of COVID-19 test methods and devices and proposed performance criteria" 2 on 16 April 2020.

The JRC capitalised on its expertise in knowledge management to conduct the literature review as part of this work and, as a follow-up action to the need identified in the Communication, committed to make the information broadly accessible and to update the compilation as new data become available.

The outcome of these actions is the JRC COVID-19 In Vitro Diagnostic Devices and Test Methods Database presented here, a single place collection of all publicly available information on performance of CE-marked in vitro diagnostic medical devices (IVDs) as well as in house laboratorydeveloped devices and related test methods for COVID-19. The database is freely accessible at https://covid-19-diagnostics.jrc. ec.europa.eu/.

The initial information is gathered following the strategy for documentation on test methods and devices indicated in Section 3 "Methodology used" of the Current performance of COVID-19 test methods and devices and proposed performance criteria -Working document of Commission services 2 .

The retrieved information is manually verified and curated. In particular:

• For CE-marked devices, only the information that the manufacturer has chosen to make publicly available is included in the database. Full information on the manufacturer's performance evaluation of the device is contained in the technical documentation required by EU legislation 3 . As manufacturer technical documentation is usually not publicly available and is therefore not included, a form is available to stakeholders to update and integrate the information: manufacturers are invited to submit performance information on new devices, which are not yet listed in the database, or to provide data not available to the authors at the time of the last update. The submitted information, once verified against the source provider, is taken into consideration for updating the database.

• Performance details (as retrieved from manufacturers' web pages) are provided only for devices commercially available with CE-IVD mark. Products labelled as for 'research use only' or 'under development' as well as products fulfilling regulatory frameworks other than the one in place in the EU are listed for information only. The correctness of information, such as performance data of the listed devices, has not been confirmed by checking raw experimental data or full technical documentation of the manufacturer nor by own laboratory verification or by any clinical validation studies.

• A team of JRC experts regularly takes care of updating the Scientific Literature section, which includes scientific articles reporting about the use and performances of devices and related test methods. This section regards also the so-called in house 2 or laboratory developed devices, which are used in healthcare institutions but are not commercially available. These are not included in the list of devices, as they are not easily identifiable by name, but scientific publications on such devices and related test methods are included in the section.

• Information on publicly available results of validation studies and on publications by national/regional health technology assessment bodies and Joint Action EUnetHTA 4 is included. These reports are produced in response to requests from national authorities and support national policy and decision-making on testing for COVID-19.

In silico simulations of NAAT methods, on available high-quality and full length sequenced SARS-CoV-2 genomes (provided as collaboration with GISAID 5 ) are performed by using ThermonucleotideBLAST 6 . The tool is run on each assembled genome with default parameters, except for the following ones: -e=30, -E=20, --max-gap=4, --max-mismatch=4.

Putative not redundant target amplicons are then extracted to build the final target dataset that is added to the database.

At the time of writing, high-quality (i.e. <1% Ns and <0.05% unique amino acid mutations) and full length (i.e. >29,000 bp) sequenced SARS-CoV-2 genomes have been manually downloaded from the GISAID website. To simplify the sharing of information and the related analysis, a formal agreement is under finalisation between GISAID and JRC.

The database is structured in a way aimed to:

• facilitate sharing of information among researchers, in line with the principle of FAIR sharing;

• link the devices, and their features, to scientific articles that reported their use and performance, in order to have a clear, transparent and fully open data source where users can look at and make the right choice in the selection of the device or method to use;

• verify and monitor if natural occurring SARS-CoV-2 mutations give rise to "false negative".

Data records are currently organised and available in three main sections: Finally, a Submit your device section provides forms for manufacturers and NAAT developers to submit information on devices not yet listed in the database or to provide performance data not available to the authors at the time of the last update. The submitted information, once verified against the source provider, is taken into consideration for updating the database.

At the time of writing, the database includes 869 devices and 382 selected articles, easily linked to each other, as shown in Figure 1 . Considering the rapidly evolving situation in relation to the development and commercialisation of diagnostic devices for COVID-19, the completeness of the information is limited to the time of the last update as indicated in the database for each individual item. For this reason, manufacturers are invited to submit and update the information through the corresponding "Submit your device" section's form.

With respect to NAAT methods, currently eight NAAT methods are available in the database: seven are from the World Health Organization (WHO) support to COVID-19 and are in-house PCR protocols assays posted online on the WHO website 7 while one was developed by the JRC in the context of the production of EURM-019, a universal positive control material to be used in the testing of SARS-CoV-2 by RT-PCR (described in reference 8). The possibility to evaluate whether variations occurring overtime in the viral genomes can affect PCR-based detection methods is fundamental to guarantee reliability of the detection. The aims of the SARS-CoV-2 target regions section in the database are therefore both to detect potential target regions of the NAAT methods and to highlight possible differences with the expected reference sequence that might affect the performance of NAATs. As of 01 May 2020, about 10% of the retrieved genomes were found to be not detectable in silico by at least one of the eight NAAT methods, as shown in Figure 2 . As an example, a case of variation potentially affecting NAAT methods detectability is shown in Figure 3 .

As for devices, considering the rapidly evolving situation in relation to COVID-19, NAAT methods developers are invited to submit updated information or new developed ones through the corresponding "Submit your device" section's form.

According to our knowledge, it is the first time that a repository provides and links information on COVID-19 in vitro diagnostic medical devices and the scientific articles testing and reporting their performance.

To date, more than 100 requests, either for addition of new in vitro diagnostic medical devices or for updating the information, have been received through the forms complied by manufacturers, as demonstration of the strong need of having a structured data sharing point for this kind of information. However, it is important to highlight that this resource does NOT represent a list of devices approved or authorised for use either by the European Commission or by Member States' national authorities. There is no central approval system for in vitro diagnostic medical devices in the EU. The currently applicable legislation for placing COVID-19 diagnostic devices on the market in the EU is Directive 98/79/EC on in vitro diagnostic medical devices 3 . Under the current Directive 3 , for COVID-19 devices that are designed for professional use, the manufacturer may affix the CE-mark to This graph visualises the relations (grey lines) among devices (coloured according to the corresponding detection principle) and the scientific articles (black spots) in which they are mentioned. The large "corona" near the centre and the group near the top right corner are clustered around two documents that refer to 172 9 and 23 10 devices, respectively. It is quite evident that publications are specialised with respect to devices detection principle (i.e. the used technology): few publications (only three) mention different detection principles' devices. Please note that devices without references are not represented. The size of the symbol representing the device is proportional to the corresponding number of references (min 1, max 8).

As of 01 May 2020, more than 16,000 high quality (<1% Ns and < 0.05% unique amino acid mutations) and full length (> 29,000 bp) viral genomes were made available by GISAID 5 . By performing in silico PCR simulations with eight NAAT methods, the graph shows that about 10% of the 16,000 selected genomes (cumulative numbers) were found to be not detectable in silico by at least one of the eight NAAT methods. Blue area represents the (cumulative) number of genomes in silico detected by all eight NAAT methods; red area represents the (cumulative) number of those not detected by at least one NAAT method. Figure 3 . Genomic variations that can affect NAAT methods target detectability. As of 01 May 2020, more than 16,000 high quality (<1% Ns and < 0.05% unique amino acid mutations) and full length (> 29,000 bp) viral genomes were made available by GISAID 5 . An example of identified genomic variations that can affect NAAT methods target detectability is here reported and it regards the first method developed by the US Centers for Disease Control and Prevention and supported by WHO in 7. 1) On the top, the target sequence (in red the left primer, in green right primer, and in azure the probe annealing region) that has been found in several genomes. With respect to the expected reference target, a variation (C/T transition, marked by *) is present at the beginning of the probe annealing region that can potentially affect the detection method performance. 2) Below, the table with details about the genomes where this variation of the target sequence has been found: the two columns with # indicate the number of genomes with such a sequence, aggregated by region and country, respectively. In the table, the dates of first and last appearance of the variation are reported for each country, in order to provide information on the "spread" of the variation.

the product after having ensured the compliance of the device with the Directive and drawn up a declaration of conformity. For COVID-19 devices that are designed for use by lay persons (self-tests), the manufacturer must also apply to a third party body called a notified body that will do additional verification and issue a certificate. As a consequence, the JRC should not be deemed responsible for the validity of such data.

As recently reported by Guglielmi 12 , "even once a test is working beautifully in the lab, it still faces an arduous journey to mass usage. The first challenge is to verify performance, because quality can vary". We believe that the here presented database contributes to the required prompt handling of the on-going public health COVID-19 situation. It makes sharing information easier and helps medical professionals, scientists and laboratories understand and properly assess COVID-19 medical devices quickly. The overview of the market may also help manufacturers develop and improve their own devices, in line with the current EU legislation.

The database is freely accessible at https:// The 'SARS-Cov2 target regions' database is enabled using third party data from GISAID, subject to GISAID's Terms and Conditions. The database is freely accessible at https://covid-19-diagnostics.jrc.ec.europa.eu/amplicons and GISAID data are available directly from GISAID at https://www.gisaid.org/ 5 . Access to GISAID data requires registration and agreement to the conditions for use at https://www.gisaid.org/registration /register/.

Code used to perform scientific literature mining is freely available at https://github.com/ec-jrc/JRC_COV19_IVD-DEVs-TEMs_DB. 

from the Commission Guidelines on COVID-19 in vitro diagnostic tests and their performance 2020/C 122 I/01C/2020/2391. Reference Source 2. Current performance of COVID-19 test methods and devices and proposed performance criteria -Working document of Commission services

on in vitro diagnostic medical devices and repealing directive 98/79/EC and Commission Decision 2010/227/EU, OJ L 117, 5.5

GISAID: Global initiative on sharing all influenza data -from vision to reality

Improved assay-dependent searching of nucleic acid sequence databases

Immunological assays for SARS-CoV-2: an analysis of available commercial tests to measure antigen and antibodies

Diagnostic Testing for Severe Acute Respiratory Syndrome-Related Coronavirus 2: A Narrative Review

PubMed Abstract | Publisher Full Text | Free Full Text

Directive 98/79/EC of the European Parliament and of the Council of 27

October 1998 on in vitro diagnostic medical devices

The explosion of new coronavirus tests that could help to end the pandemic

ec-jrc/JRC_COV19_IVD-DEVs-TEMs_DB: First release of JRC_COVID-19_Literature_search

We gratefully acknowledge the Authors and the Originating laboratories where the clinical specimen or virus isolate was first obtained and the Submitting laboratories, where sequence data have been generated and submitted to GISAID, on which this research is based.We acknowledge all the manufacturers and repository website owners of the devices that have been analysed; all the authors, publishers and literature repositories of the papers that have been analysed.

The manuscript presents a significant work aimed to collect publicly available information on test methods for COVID-19. Due to the amount of devices and methods available, the database described offers a useful tool to be addressed in the field. I find of special interest the example on genomic variations that can affect detectability. The manuscript is well presented and written. I find in the manuscript only minor points to be addressed:As a reader, I would find in the article an example of the output obtained when the database is searched. 

Competing Interests: No competing interests were disclosed.Reviewer Expertise: Bioinformatics, biochemistry, molecular biology I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.The benefits of publishing with F1000Research:Your article is published within days, with no editorial bias • You can publish traditional articles, null/negative results, case reports, data notes and more • The peer review process is transparent and collaborative • Your article is indexed in PubMed after passing peer review • Dedicated customer support at every stage • For pre-submission enquiries, contact research@f1000.com