key: cord-0311637-hrrnkxm8 authors: Harding, C.; Pompei, M.; Burmistrov, D.; Pompei, F. title: Overlooked bias with thermometer evaluations using quickly retaken temperatures in EHR: applications to axillary, oral, temporal artery, and tympanic thermometry date: 2020-11-28 journal: nan DOI: 10.1101/2020.11.24.20237958 sha: c331ccb266bcf628fc33ae1bd0f2df7e111e3891 doc_id: 311637 cord_uid: hrrnkxm8 Studies related to COVID-19 increasingly use electronic health records (EHR) to obtain large-scale evidence. However, EHR-based research must be performed with care because it can involve new study design problems that are unfamiliar to much of the medical community. Haimovich et al. (2020) sought to inform COVID-19 practice by evaluating temporal artery thermometers (TATs). They retrospectively searched EHR for temperatures measured twice within 15 minutes, including once with a TAT. The TAT often disagreed with reference measurements, so Haimovich et al. concluded TATs perform poorly. Here, we extended Haimovich et al.'s study design to all other major thermometer types using the eICU Collaborative Research Database. We retrospectively identified 80,065 pairs of quickly retaken temperatures from 24,765 adult U.S. critical care patients treated in 2014-2015. We found that oral, tympanic, and axillary thermometers disagreed with reference measurements as much as TATs did. Moreover, all thermometer types showed unprecedentedly worse agreement than observed in research using other study designs. Every thermometer type broke {+/-}0.9{degrees}F ({+/-}0.5{degrees}C) limits of clinically acceptable agreement by >2-fold and no type satisfied basic standards for repeatability. A natural explanation for these findings is that clinicians often retook temperatures within minutes because of user or patient errors during measurement, such as probe misplacement or patient movement. This means that quickly retaken EHR measurements do not reflect device accuracy or precision in correct use and, contrary to Haimovich et al.'s conclusions, should not be used to evaluate thermometer performance or revise COVID-19 fever thresholds. Our study provides an illustrative example of unexpected study design problems that can undermine EHR-based research. . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint The copyright holder for this this version posted November 28, 2020. Axillary, central, oral, temporal, and tympanic sites were studied. Each has strengths and weaknesses in terms of accuracy, safety, and convenience. 5 Central temperature was defined as pulmonary artery, urinary bladder, esophageal, rectal, or core (subtype unspecified). Pulmonary artery is the only true gold standard, 6 but is highly invasive and rarely taken (0.6% of temperatures), so all central temperatures were used as a reference standard instead. Site was recorded in 1-4 words of free text. We recoded entries appearing ≥ 10 times, amounting to 99.98% of entries. We evaluated agreement between paired temperatures using Bland-Altman analyses. Analyses were cluster bootstrapped by patient to address within-patient nonindependence (replicates=20,000). Re-running analyses with only 1 measurement pair per patient also produced similar results. Patients with quickly retaken temperatures had mostly similar characteristics to the overall Every thermometer site had similar, very low agreement with reference temperatures (Figure 1 ). No site satisfied the ±0.9°F (±0.5°C) limits of agreement often used to define clinical acceptability. 6 When analyzing temperatures retaken at the same site, every site showed unprecedentedly low repeatability ( Figure 2 ). Retaken central temperatures were often anomalously cold (9.9% <95°F and 4.2% <92°F). . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint The copyright holder for this this version posted November 28, 2020. observed this for TATs and we observed it for all major non-invasive thermometer types. Anomalously cold central temperatures were also common in both studies. 1 A natural explanation is that clinicians often retook temperatures within minutes because of user or patient errors during measurement. Common errors include patient movement, inadvertent thermometer activation, and insufficiently deep rectal or esophageal probes. This last error produces artificially cold temperatures, but correcting it discomforts patients, so clinicians sometimes try non-invasive thermometers instead. The explanation means that quickly retaken EHR measurements do not reflect device accuracy or precision in correct use. To detect similar problems in other EHR-based studies, researchers should place themselves in the clinicians' shoes and carefully consider why actions listed in EHR were taken. Funding: This study was funded by Exergen, Corp., a manufacturer of thermometers, including of the temporal artery type. Online-only supplement: None. (Not allowed for the journal's brief reports.) . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint The copyright holder for this this version posted November 28, 2020. Performance of temporal artery temperature measurement in ruling out fever: implications for COVID-19 screening ICU Collaborative Research Database (version 2.0). PhysioNet Scripts for analyzing quickly retaken temperatures in eICU datasets The eICU collaborative research database, a freely available multi-center database for critical care research Best practice in the measurement of body temperature A systematic review of the accuracy of . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprintThe copyright holder for this this version posted November 28, 2020. Confidence intervals are 95%. Mean differences (mean biases) are shown in red and limits of agreement (mean difference ± 2 standard deviations) are shown in blue. Central temperatures include pulmonary artery, esophageal, urinary bladder, rectal, and core.. It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.