key: cord-0849601-u20wsdyg
authors: Maury, E.; Boldi, M.-O.; Greub, G.; Chavez, V.; Jaton, K.; Opota, O.
title: An automated Dashboard to improve laboratory COVID-19 diagnostics management
date: 2021-03-24
journal: nan
DOI: 10.1101/2021.03.20.21253624
sha: 0d2519507e404128b33abca382d5878c99a8280e
doc_id: 849601
cord_uid: u20wsdyg

Background: In response to the CoVID-19 pandemic, our microbial diagnostic laboratory located in a university hospital has implemented several distinct SARS-CoV-2 RT-PCR systems in a very short time. Thanks to our automated molecular diagnostic platform, more than 140 000 SARS-CoV-2 RT-PCR tests were achieved over 12 months, with peaks higher than 1 500 daily tests. A dashboard was developed to give access to Key Performance Indicators (KPIs) to improve laboratory operational management. Methods: RT-PCR data extraction of four respiratory viruses - SARS-CoV-2, influenza A and B and RSV - from our laboratory information system (LIS), was automated. Important KPIs were identified and the visualization was achieved using an in-house dashboard based on the open-source language R (Shiny). Information is updated every 4 hours. Results: The dashboard is organized into three main parts. The Filter page presents all the KPIs, divided into five sections: i) general and gender-related indicators, ii) number of tests and positivity rate, iii) cycle threshold and viral load, iv) test durations, and v) not valid results. Filtering allows to select a given period, a dedicated instrument, a given specimen, or a requester for instance. The Comparison page allows a custom charting of all the available variables, which represents more than 182 combinations. The Data page gives the user access to the raw data in table format, with the possibility of filtering, allowing for a deeper analysis and data download in Excel format. Conclusions: The dashboard, that gives a rapid access to a huge number of up-to-date information, represents a reliable and user-friendly tool improving the decision-making process, resource planning and quality management. The dashboard represent an added value for diagnosric laboratories during a pandemic, where rapid and efficient adaptation is mandatory.

is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

The copyright holder for this this version posted March 24, 2021. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

The copyright holder for this this version posted March 24, 2021. represents one of the pillars of the diagnosis of COVID-19. Indeed, RT-PCR is also the heart of 67 the patient care and epidemic control process and will be the mainstay of several clinical studies. 68

Although our laboratory has extensive experience in the development of RT-PCR, the 69 introduction of this new parameter represented a challenge in terms of speed of development 70 [7] . It is also the first time that an introduced parameter has been used on such a large scale in 71 such a short time; more than 10'000 tests were carried out in one month in Spring [6] and even 72 in a single week during fall 2020. This was possible thanks to automation and digitalization, to 73 allow high throughput and acceptable time to results [7] . In this context, the IMU set strategies 74 to ensure the quality and reliability of the RT-PCR. This included the monitoring of key 75 performance indicators (KPIs) for quality management such as the proportions of positive tests 76 or the virus load, both per day, per instruments, and per requester. These indicators aimed for 77 instance to identify variations not explained by epidemiological changes. Indeed, abnormal 78 variations could be synonymous with pre-analytical problems (sampling problem, transport 79 medium, etc.) or even analytical problems (mutation in the target sequences of PCRs associated 80 with losses of sensitivity or specificities). The IMU also defined KPIs for operations management 81 such as the time to results [8] . 82

Before COVID-19, such indicators were monitored periodically, for example in the context of an 83 annual report or retrospective studies. At the beginning of the COVID-19 outbreak, the IMU 84 decided to follow these indicators frequently. Because the needed manual analyses were time-85 consuming, the monitoring of analytical and operational KPIs was carried out once a week 86 initially and then, twice a week depending on the period. These analyses were also prone to 87 . CC-BY-NC-ND 4.0 International license It is made available under a perpetuity.

is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint In this paper, we present the design, the development, and the use of a dashboard targeted to 128 a laboratory located in a teaching hospital, in charge of PCR test, following the COVID-19 129 outbreak, such as the IMU. This includes i) to define the need, ii) to build the dashboard, iii) to 130 deploy the tool, and iv) to demonstrate the added value in terms of the quality and operations 131 management for the laboratory mission. This research focuses on aspects other than 132 epidemiological matters (patient type, pathogen, period of the year, etc.), and which can explain 133 some variation of results in the laboratory. We split these aspects into two main categories: 134 quality issues and management issues. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint Ct values, the viral quantification (in copy per milliliter, cp/mL), and whether the analysis had to 159 be repeated. 160

A sample is related to one patient but that a patient may be tested several times. The analysis 161 codes correspond to a test, which is performed for a specific virus, on a specific device (machine 162 used to perform the test), for a targeted gene. 163 Some cleaning and data wrangling were performed before building the dashboard. Using a 164 matching table shared by the IMU, the analysis codes were renamed using more user-friendly 165 structure (NOM.VIRUS_TYPE.ANALYSE_APPAREIL_GENE). Then, different measures are 166 extracted, especially on date-time data: the reception duration is the difference between the 167 sampling time and the reception time at the laboratory, the test duration is the difference 168 between the reception time and the results validation time and the total duration is the sum of 169 the last two. Using the date of birth, patients were categorized into age groups with a 10-year 170 window. Similarly, the type of sampling was recoded using wider groups categories, the most 171 present being NPS (nasopharyngeal secretions). Then, each analysis is assigned a "Virus" and 172 a "Device", corresponding to the non-empty analysis codes described above. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint Stacked Column charts, Scatter plots, Line charts and Boxplots. The latter has the advantage 197 of displaying many information at once, and the end-users are accustomed to this format. 198

Finally, for a quicker adoption and an optimal usage, the dashboard was built in French, the 199 mother tongue of the end-users. 200

The data were obtained during a quality enhancement project at our institution. According to 202 national law, the performance and publishing the results of such a project can be done without 203 asking the permission of the competent research ethics committee. The current version of the dashboard contains three main pages detailed in figure 1: i) "Filter", 219 ii) "Comparison" and iii) "Data". The Filter page allows the user to select inputs such as a date 220 range, the gender, the age, the test result, the hospitalization status, the device used, the 221 confirmation status, the type of sample, and the type of requesters. In the Comparison page, the 222 user selects variables to appear on graphs, but also filters the dataset to narrow down 223 comparison subjects. Finally, the Data page also lets the user filter the dataset to look at 224 individual observations. Overall, the user therefore has a role in filtering observations and in 225 deciding which information to be represented (Figure 1 and suppl. Table 1) . 226 . CC-BY-NC-ND 4.0 International license It is made available under a perpetuity.

is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

The copyright holder for this this version posted March 24, 2021. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

The copyright holder for this this version posted March 24, 2021. ; https://doi.org/10.1101/2021.03.20.21253624 doi: medRxiv preprint copy/millimeter (cp/mL) since April 9th, 2020, date from which the laboratory started to keep this 259 record. These are shown in a scatterplot crossed by type of sampling (blood, in nasopharyngeal 260 secretions, etc.) as well as in a summary statistics table (Suppl. Figure 3) . 261

Finally, a descriptive table of the cases for which the result of the analysis was "NOT VALID" is 263 displayed. The user can then identify any issue and investigate further. For sake of readability 264 only some variables are shown (Suppl. Figure 4) . (Table 1) . 271

Upon this choice, the absolute and the relative number of tests are plotted, such as shown in 272

Suppl. Figure 5 for gender and week of reception. The following plot shows data about the test 273

duration. An example in Figure 4 .B shows the average test duration per instrument, depending 274 on the result of the test. This is especially useful to control the speed of some devices. Finally, 275 the Ct information is displayed using boxplots (Figure 6 .A), a familiar representation to the end-276 users. All the described figures are common to all viruses. For SARS-Cov-2, an additional 277 boxplot of the viral load in log 10 scale is displayed (Figure 6 .B). This additional boxplot appears 278 only for this virus whose load can vary from 1000 to 14000000000 cp/mL. The log10 scale 279 provides a more readable graph. 280

On the Data page, the users can have a look at the raw data in a table format, after filtering if 282 wanted, thus deepening the analysis of an issue located with the previous pages tools. They 283 can then download the data and explore it in Excel. This simple feature appeared to be 284 surprisingly useful to the users who can then extract data more quickly than with a data base 285 request from Molis. 286

The general KPIs (e.g., number of daily tests, average number of tests per patient, etc.), such 288 as Suppl. Figure 2. A, appearing at the very top of the dashboard are used to continuously 289 reassess the current situation. Seeing the evolution of the past few days or weeks helps the staff 290 . CC-BY-NC-ND 4.0 International license It is made available under a perpetuity.

is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint issues. For instance, the dashboard led to the discovery of a huge amount of samples received 295 at the end of the business day, and helped tune the process of deliveries at the laboratory. This 296 is central to meet the needs of the hospital, of the requesters and, at the end, of the patients 297 themselves. The tool also supports the team in charge of identifying potential causes of delay: 298 by being able to look into various combination and level of details, it is possible to point out these 299 causes during the whole process (from sampling, to testing, to result sending). They hence can 300 then be investigated individually (e.g., by requester) or more globally (e.g., by day of the week). 301 The dashboard also comforted some intuitions the institute had from its experience: it helped 304 identify the peak hours, adapt the distribution of the employees, and make recommendation to 305 major requesters in order to smooth the operational activity during the day. 306

Finally, the daily tracking of these KPIs allowed the inventory monitoring of materials and 307 avoided an unexpected shortage of scarce test resources. 308

Positivity rate 310

Having the possibility to observe the positivity rate by instrument can help diagnose any failure, 311 such as contamination. For instance, an increase in the positivity rate could be due to a 312 contamination of the test reagent. Easily accessing this information, (Suppl. Figure 2 .B for an 313 example) being able to quickly analyze it and linking it to Ct values provides a critical advantage 314 in maintaining high quality results over the course of the outbreak. Since the users of the 315 dashboard can also look at the data per requester, or requester type, in case of a sudden 316 variation in the positive rate test results, it allows them to look for non-epidemiologic reasons. 317

Possible explanations include change of sampling methodologies, change of patient type (e.g., 318 from mostly children to elderly patients), addition of new facilities in some requester categories. 319

They can also investigate the geographical origin of the test and share relevant information with 320 the appropriate authorities. 321

Finally, being able to verify the number of positive specimens by age category is critical, as the 322 patients' age is playing a key role in the pandemic (spreading role of children, disease evolution 323 among elderly, etc.). Figure 3 .A shows the number of tests per age group, which can be filtered 324 for specific periods of time, or different requesters. 325 is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint Among various data in the dashboard, the viral load especially helps guarantee the accuracy of 327 the test or conversely identify and solve analytic problems. For instance, any sudden drop or 328 jump in the daily or weekly median viral load raised a warning and calls for explanation (e.g., 329

change of testing strategy, population target). If no explanation is found, one could suspect a 330 problem at the analytical stage (i.e.: pre-analytical stage, virus mutation etc.). A major viral load 331 drop could mean a decrease in analytic sensitivity, which could be dramatic during an outbreak. 332 A decrease in the median viral load was observed in April 2020 which led to a modification of 333 the testing strategy, namely a universal testing strategy [35] . 334

Similarly, tracking the virus loads per patient types helps addressing the test sensitivity. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

The copyright holder for this this version posted March 24, 2021. ; https://doi.org/10.1101/2021.03.20.21253624 doi: medRxiv preprint test has been used with such a high throughput soon after its introduction. In this, the statistical 361 analyzes proposed by the dashboard help to improve the reliability of these tests. For example, 362 the real-time monitoring of the positivity rate of the tests or of the viral load in the clinical samples 363 enables to highlight analytical problems, which could lead to false positives or false negatives. 364

Using the dashboard, the positivity rate was compared according to the different instruments to 365 check whether one instrument might be associated with a systematic error. However, cautious 366 interpretation of the data available through the dashboard is mandatory since differences might 367 also be due to various other factors. For example, because highly symptomatic subjects 368 admitted at the emergency ward of the hospital were tested with the GenXpert rapid RT-PCR, 369 the tests performed with that instruments were more often positive. The dashboard allows to dynamically monitor the time of arrival of samples at the laboratory and 382 thus to adjust the work organization. Typically, we added human resources at the end of the 383 afternoon to allow more same-day results. 384 385 In its current version, the code is specific to the practice of our laboratory. Thus, it is difficult to 386 generalize to other laboratories or users. So far, the app is still in a development version and 387 suffers from limited performances: slow to refresh, and unlikely to support database growth on 388 the long term. 389

On the analytical part, there is no multivariate analysis feature. It is thus, for example, not 390 possible to correlate KPIs of several viruses altogether. Furthermore, no automated detection 391 has been implemented yet. And, in the same vein, no efficient forecasting method is available. 392

We can expect that this type of tool will also help to consolidate the user's intuitions on the long 394 term. This is linked to the predictive aspect of the dashboard, a topic not addressed in this paper, 395 but a subject of future research. The data accumulation allows to develop models leading to 396 . CC-BY-NC-ND 4.0 International license It is made available under a perpetuity.

is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

The copyright holder for this this version posted March 24, 2021. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

The copyright holder for this this version posted March 24, 2021. ; https://doi.org/10.1101/2021.03.20.21253624 doi: medRxiv preprint Comparison page and Data page. On the filtering page, key indicators are available such as the 562 number of samples, time to results, positivity rate, percent of invalid tests, etc. These indicators 563 are provided by default for the whole dataset, but are also available for subgroups, according to 564 the filtering criteria applied to the whole dataset. Thus, it is possible in a click to observe 565 specifically the tests done during a given period or done on a dedicated instrument. It is also 566 possible to select only the analysis performed for a given requester. The Comparison page offers 567 more than 182 combinations of the KPI.The Data page, gives access to the raw data in a table 568

format that can be downloaded; filters can be applied to shoose a subset of data of interest. 569 570 571 . CC-BY-NC-ND 4.0 International license It is made available under a perpetuity.

is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint

The copyright holder for this this version posted March 24, 2021. ; https://doi.org/10.1101/2021.03.20.21253624 doi: medRxiv preprint

Dashboard (business). (Wikipedia, The Free Encyclopedia

The Landscape for Performance dashboards. In: Performance 431 dashboards: measuring, monitoring, and managing your business

Visual 434

Analytics: Definition, Process, and Challenges. In: Information Visualization: Human-435

Illuminating the Path: The Research and Development Agenda 438 for Visual Analytics

Productivity of Information Systems in the 440

Improving Health Care Management in Hospitals 443

Through a Productivity Dashboard

Balanced scorecard: application in the General Panarcadian 445

Interactive dashboards: Using visual analytics for 448 knowledge transfer and decision support

Ease of Healthcare Analysis Through Use of Data Visualization Dashboards. Big data

Applications of business analytics in healthcare

Digital dashboard design 456 using multiple data streams for disease surveillance with influenza surveillance as an 457 example

An interactive web-based dashboard to track COVID-19 459 in real time. The Lancet infectious diseases

is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprintThe copyright holder for this this version posted March 24, 2021. is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprintThe copyright holder for this this version posted March 24, 2021. ; https://doi.org/10.1101/2021.03.20.21253624 doi: medRxiv preprint