key: cord-253958-8lhzgqfr authors: Clark, T. W.; Brendish, N. J.; Poole, S.; Naidu, V. V.; Mansbridge, C.; Norton, N.; Wheeler, H.; Presland, L.; Ewings, S. title: Diagnostic accuracy of a host response point-of-care test for identifying COVID-19 date: 2020-06-02 journal: nan DOI: 10.1101/2020.05.27.20114512 sha: doc_id: 253958 cord_uid: 8lhzgqfr Rationale Management of the COVID-19 pandemic is hampered by long delays associated with centralised laboratory PCR testing. In hospitals this leads to poor patient flow and nosocomial transmission and rapid, accurate diagnostic tests are urgently required. The FebriDx is a point-of-care test that detects an antiviral host response protein in finger prick blood within 10 minutes, but its accuracy for the detection of COVID-19 is unknown. Objectives To evaluate the diagnostic accuracy of FebriDx in hospitalised patients during the first wave of the pandemic. Methods Measures of diagnostic accuracy were calculated based on FebriDx results compared to the reference standard of PCR, and stratified by duration of symptoms. A multivariable predictive model was developed and underwent internal validation. Results FebriDx was performed on 251 patients and gave a valid result in 248. 118 of 248 (48%) were PCR positive for COVID-19. Sensitivity of FebriDx for the identification of COVID-19 was 93% (110/118; 95% CI 87 to 97%) and specificity was 86% (112/130; 95%CI 79 to 92%). Positive and negative likelihood ratios were 6.73 (95%CI 4.37 to 10.37) and 0.08 (95%CI 0.04 to 0.15) respectively. In the multivariate model diagnosis of COVID-19 was not significantly influenced by clinical symptoms and signs, and FebriDx accuracy was not improved by restricting testing to those with duration of symptoms of less than seven days. Conclusions During the first wave of the pandemic, FebriDx had high sensitivity for the identification of COVID-19 in hospitalised adults and could be deployed as a front door triage tool. The management of COVID-19 cases in secondary care is severely hampered by the long turnaround times of centralised laboratory PCR testing, which can take up to several days to generate results. In acute hospitals this leads to poor patient flow though acute areas, as suspected patients are held in assessment cohort areas until their results are available. In addition, lack of single occupancy rooms means that COVID-19 negative patients cohorted in these assessment areas may acquire infection from positive patients before test results are available. Rapid, accurate diagnostic tests are therefore urgently required. Molecular point-of-care testing (POCT) for COVID-19 may mitigate this situation but regulatory requirements for new POCT assays and difficulties in rapidly upscaling manufacture mean that the availability of these tests is currently severely limited. Alternative diagnostic solutions are therefore urgently required. FebriDx (Lumos diagnostics, Sarasota, Florida, US) is a CE-marked POCT that detects two host response proteins, Myxovirus resistance protein A (MxA) and C reactive protein (CRP), in finger prick blood samples, and is designed to distinguish viral from bacterial respiratory infection. [1] [2] [3] [4] [5] MxA is a marker of interferon-induced antiviral host response and in our previous work, the detection of MxA by FebriDx had high sensitivity for the identification of Influenza in hospitalized adults, during influenza season. 6 MxA levels are also likely to be significantly elevated in patients with COVID-19 but the diagnostic accuracy of FebriDx in this situation is currently unknown. FebriDx is a low cost, instrument-free, disposable POCT system and if sensitive for the detection of COVID-19 could be rapidly deployed across the NHS and other healthcare systems as a rapid triage tool, and help mitigate the long 5 was collected at enrolment and outcome data collected retrospectively from case note and electronic systems, using an electronic case report form. For this sub-study the first 266 patients enrolled were consecutively approached for testing using the FebriDx host response POCT on finger prick blood samples, taken at the same time as the nose as throat swabs for PCR. Detailed instructions for use of FebriDx are provided via this link: https://www.febridx.com/how-to-use#testing but in brief; after puncture of the skin with the integral lancet, 5 microliters of blood are drawn into the blood collection tube via capillary action by placing it against a blood drop. The blood is then transferred to the lateral flow section of the device and reagents released by pressing a button. The test is read after 10 minutes. The FebriDx system generates results in the form of the presence or absence of were read and recorded before PCR results were available). . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted June 2, 2020. . https://doi.org/10.1101/2020.05.27.20114512 doi: medRxiv preprint For the reference standard of PCR, the QIAstat-Dx PCR system gives a readout of positive or negative for the detection of targets including SARS-CoV-2. 7 The PHE laboratory RdRp and E gene assay PCR results are also provided as a binary result, RNA detected or RNA not detected, and the detection of RNA by either assay is considered a positive result. For this study PCR detection by either QIAstat-Dx or laboratory PCR or both was considered as positive for COVID-19. The sample size for this sub-study was driven by consideration for estimating sensitivity of FebriDx MxA for the identification of COVID-19 (as defined by the reference standard of detection of SARS-CoV-2 by PCR on respiratory samples) and of the methods proposed by Riley et al 12 for multivariable predictive models. In order to estimate a sensitivity of 85% to within +/-8%, based on the score method for a 95% confidence interval, 13 88 positive cases are required. Assuming a prevalence of COVID-19 of around 40% in those tested, 220 people are required, and to achieve an 80% chance of obtaining enough cases, this number was increased to 236. This was further increased to 266 to allow for patients who decline or are unable to undergo finger prick testing (estimated from the rates in our previous work). Full justification for the samples size in the multivariable model is provided in File E1 in the online data supplement. Baseline characteristics are summarised for all those recruited to the study, where data are available, and presented for the whole sample and by COVID-19 status. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted June 2, 2020. As the antiviral host response to SARS-CoV-2 would be expected to wane over time, but detection of viral RNA by PCR may persist for several weeks, a secondary analysis was pre-specified to assess diagnostic accuracy in those who have had duration of less than seven days. Basic measures of diagnostic accuracy (e.g., sensitivity, specificity) are presented separately for those with duration less than or more than seven days. A multivariable model, pre-specified to be the same as that for the full model, was also run in those with duration less than seven days. This study was prospectively registered with the ISRCTN14966673 on the 18 th March 2020. The first 266 patients recruited to the CoV-19POC study were enrolled and FebriDx was performed in 251. 12 of 251 (5%) tests failed to give a valid result on the first test and a subsequent valid result was obtained on re-testing in 9 of 12, so that overall 248 patients had a valid FebriDx result. Flow of study participants is shown in Figure 1 . 118 of 248 (48%) were COVID-19 positive by PCR (for SARS-CoV-2) on . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted June 2, 2020. . Table 1 . Patients positive for COVID-19 were less likely to be ethnically White British, had been unwell for longer (median duration of illness prior to presentation of 8 days), had a higher temperature and respiratory rate and lower oxygen saturations, compared to COVID-19 negative patients. In addition, COVID-19 positive patients had higher CRP levels, lower white cell counts and were more likely to have radiological evidence of pneumonia, shown in Table 1 . Compared to the reference standard of detection of SARS-CoV-2 RNA by PCR on respiratory samples, the sensitivity of FebriDx MxA detection for the identification of COVID-19 was 93% (110/118; 95%CI 87 to 97%). Specificity in this cohort was 86% (112/130; 95%CI 79 to 92%). Positive predictive value (PPV) was 86% (110/128; 95% CI 79 to 91%) and negative predictive value (NPV) was 93% (112/120; 95% CI 87 to 97%). Overall accuracy was 90% (222/248; 95%CI 86 to 93%), shown in Table 2 . As detection of MxA is not specific for COVID-19 the diagnostic accuracy of FebriDx MxA detection for the identification of all respiratory viruses is shown in Table 3 . Of . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted June 2, 2020. . the 11 patients with positive FebriDx MxA but who were PCR negative for SARS-CoV-2 and other viruses, 7 of 11 (63%) had classic radiological features of COVID- Using the calculated performance of FebriDx MxA for the identification of COVID-19 to estimate PPV and NPV at different disease prevalence, a prevalence of COVID-19 of 20% would give a PPV of 63% (95%CI 52 to 72%) and NPV of 99% (95%CI 96 to 99%). At a prevalence of 10% the PPV would be 43% (95%CI 33 to 53%) and the NPV would be 99% (95%CI 98 to 100%). Measures of diagnostic accuracy according to duration of symptoms are shown in Table 4 . Although direct comparison of accuracy was not possible due to samples size, the sensitivity, specificity and positive and negative predictive values of FebriDx appeared comparable for patients presenting before and after 7 days of symptom duration. A total of 201 participants contributed data to the multivariable model, with symptoms being the main contributor to missing data. With the possible exception of cough (12% missing data in those without COVID-19 versus 6% with), missingness of data appeared unrelated to outcome. All continuous variables (age, respiratory rate and temperature) were mean-centred and assessed for a non-linear relationship with outcome using restricted cubic splines, following visual assessment and using Bayesian Information Criterion values (where smaller values indicate a more parsimonious model), all were entered in the model on the assumption that a linear . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted June 2, 2020. . relationship with the log odds of having COVID-19 was appropriate. All variables were entered into the multivariable model with no variable selection. Model results are given in Table 5 , alongside adjusted estimates. The overall results suggested that the addition of the patient and clinical characteristics do not add significantly to the diagnostic accuracy compared to using The model underwent internal validation, using bootstrap resampling with 500 samples following the same model development process, to estimate optimism and provide a basis for adjusting model coefficients and the C-statistic. The calibration slope suggests the model tended to estimate probabilities that are too extreme and that if the model were to be used externally, the original multivariable regression coefficients should be adjusted by the value of the calibration slope (the shrinkage factor) to adjust for optimism. The C-statistic was also adjusted for optimism, but remained very high (0.90) suggesting the model would still perform well outside of the current sample. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted June 2, 2020. . The multivariable model was rerun with anosmia; a total of n=164 were included in this analysis. Conclusions for the other covariates did not change, and there was little change in discrimination results (area under the curve, sensitivity and specificity at the Youden index, Brier score) nor other measures of fit such as Nagelkerke's R 2 ; anosmia itself had an estimated odds ratio of 4.0 (95% CI: 0.7 to 22; p=0.11) suggesting a potentially useful diagnostic indicator, though this conclusion is hampered by the uncertainty in the estimate. Lastly, the multivariable model (excluding anosmia) was also run in the subgroup of participants who presented within 7 days of symptom onset. Although the odds ratio for FebriDx increased (190), the uncertainty of this effect was large (95% CI 26 to 1404). There was little change in model performance with regard discrimination and model fit. As detailed above, some FebriDx tests failed to give a valid result on the first attempt. Tests could fail by blood clotting in the sample collection tube so that no blood would pass onto the lateral flow strip. There were no adverse events seen with FebriDx testing. A single patient suffered brief epistaxis following nasal swabbing for PCR testing. In this large, real-world study we have shown that that the detection of the antiviral host response protein MxA using the FebriDx point-of-care test had high sensitivity for the identification of COVID-19 during the first wave of the pandemic. Although . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted June 2, 2020. . https://doi.org/10.1101/2020.05.27.20114512 doi: medRxiv preprint there were differences in baseline characteristics between COVID-19 positive and negative patients, the multivariate predictive model results showed that clinical features did not contribute significantly to the effect of the FebriDx MxA result in distinguishing these groups. MxA detection is a marker of antiviral host response and is not specific for SARS-CoV-2 and so the high specificity of FebriDx MxA for COVID-19 seen in this study reflects the low levels of other respiratory viruses circulating during the study which will change with the seasonal circulation of other viruses such as influenza. It is now recognised that the diagnostic accuracy of PCR on upper respiratory tract samples is suboptimal and that patients presenting with COVID-19 often initially test falsely negative when compared to serial PCR testing or CT scanning. [16] [17] [18] This means that our reference standard was likely to be suboptimal and it is notable that several of the patients in our study with positive MxA results and negative PCR for viruses (ie those considered to be false positive MxA) had classical radiological changes of COVID-19, strongly suggesting that they had the disease despite negative PCR results. The calculated specificity of FebriDx MxA for the detection of COVID-19 in the first wave was therefore likely to have been an underestimate. The current reliance on centralised laboratory PCR for the diagnosis of COVID-19 with its prolonged turnaround time, leads to long delays in identification of positive and negative patients, with subsequent nosocomial transmissions, poor patient flow and reduced operational capacity in hospitals. As FebriDx is a low cost (£12 per test), rapid and highly scalable test these findings suggest that FebriDx could be rapidly deployed in hospitals and urgent care centres to be used as a front door triage tool. Because of the high sensitivity and negative predictive value, MxA negative patients . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted June 2, 2020. . being admitted to hospitals could be rapidly directed to non COVID-19 areas whilst MxA positive patients could be moved to assessment areas whilst awaiting confirmation or exclusion of COVID-19 positive status by PCR or CT scanning. The manufacturers of FebriDx state that it is intended for use in patients aged over two years of age who have had respiratory symptoms for less than or equal to 7 days and/or fever for 48 hours duration or less. In this study we have demonstrated that MxA detection by FebriDx remains accurate for the identification of COVID-19 beyond this period, presumably due to a very strong and persistent antiviral host response to SARS-CoV-2. There are several limitations to the generalisability of our study. As the study was performed in almost exclusively immunocompetent adults the results cannot be applied to immune-compromised patients or to children and separate evaluations should be urgently undertaken in these important patient groups. In addition the findings of our study cannot be extrapolated to community dwelling patients including those who are infected but asymptomatic or pauci-symptomatic, as it uncertain whether their antiviral host response would be comparable to hospitalised patients. The diagnostic accuracy of FebriDx reported in this study is in the context of a very high disease prevalence and as we have shown the performance characteristics will change as the disease prevalence drops, with implications for how the test could be used. In particular the positive predictive value will decrease now that the first wave is subsiding, although the negative predictive value will increase further meaning that a negative FebriDx will remain a useful rule-out test. As MxA detection is a marker of antiviral host response and is not specific for COVID-19, the calculated test . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted June 2, 2020. . specificity will be lower when other viruses such as influenza are circulating. As early identification of influenza alongside COVID-19 will remains vitally important in hospitals, a rapid triage tool that detects both influenza and COVID-19 is likely to be of utility in the coming winter months. Finally, as this was a single centre study the diagnostic accuracy of FebriDx MxA for COVID-19 should be external validated in other centres. In summary, detection of MxA by FebriDx had a high sensitivity for the identification of COVID-19 in the context of a high prevalence during the first wave of the pandemic. FebriDx could be rapidly deployed in secondary care settings as a triage tool to address the current problems of delayed diagnosis with laboratory PCR. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted June 2, 2020. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted June 2, 2020. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted June 2, 2020. is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted June 2, 2020. . is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted June 2, 2020. . It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted June 2, 2020. T a b l e 2 . M e a s u r e s o f d i a g n o s t i c a c c u r a c y o f F e b r i D x M x A f o r i d e n t i f i c a t i o n o f C O V I D -1 9 , c o m p a r e d t o t h e r e f e r e n c e s t a n d a r d o f P C R p o s i t i v i t y , n = 2 4 8 . . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted June 2, 2020. . T a b l e 3 . M e a s u r e s o f d i a g n o s t i c a c c u r a c y o f F e b r i D x M x A d e t e c t i o n f o r a n y r e s p i r a t o r y v i r u s , c o m p a r e d t o t h e r e f e r e n c e s t a n d a r d o f P C R p o s i t i v i t y , n = 2 4 8 . . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted June 2, 2020. . T a b l e 4 . M e a s u r e s o f d i a g n o s t i c a c c u r a c y o f F e b r i D x M x A f o r i d e n t i f i c a t i o n o f C O V I D -1 9 , a c c o r d i n g t o d u r a t i o n o f i l l n e s . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted June 2, 2020. . . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted June 2, 2020. . MxA-based recognition of viral illness in febrile children by a whole blood assay Diagnosis of viral infections using myxovirus resistance A (MxA) Diagnostic accuracy of FebriDx: A rapid test to detect immune responses to viral and bacterial upper respiratory infections A prospective, multi-centre US clinical trial to determine accuracy of FebriDx point-of-care testing for acute upper respiratory infections with and without a confirmed fever A Rapid Diagnostic Test for Differentiating Bacterial and Viral Aetiologies in Acute Respiratory Infections Two-sided confidence intervals for the single proportion: comparison of seven methods by Robert G. Newcombe Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal Regression modelling strategies for improved prognostic prediction Correlation of chest CT and RT-PCR testing in coronavirus disease We would like to acknowledge and gives thanks to all the patient who kindly participated in this study and to all the clinical staff at University Hospital Southampton who cared for them. We would also like to give thanks to Richard Poole for assistance in data management.