key: cord-0833801-y0xlza90 authors: Claessens, J. L.; van Egmond, J.; de Boer, J. H.; Wisse, R. P. L. title: The evaluation of a novel tool to remotely assess visual acuity in chronic uveitis patients during the COVID-19 pandemic date: 2021-04-20 journal: nan DOI: 10.1101/2021.04.14.21255457 sha: daca8135e98f990b218bebe76155a6118b8e0f85 doc_id: 833801 cord_uid: y0xlza90 Background: Restrictions due to the recent COVID-19 pandemic catalysed the deployment of telehealth solutions. A novel web-based visual acuity test, validated in a healthy population, may be of great value in the follow-up of uveitis patients. Objective To determine the measurement accuracy of the unsupervised remote Easee web-based visual acuity test in uveitis patients, when compared to a conventional in-hospital assessment. Methods: Cross-sectional diagnostic accuracy study. Between April 2020 and September 2020, consecutive adult uveitis patients were invited for the web-based visual acuity test (index test) within two weeks prior to their conventional in-hospital assessment (reference test). Results: A total of 269 patients were invited by mail, of whom 84 visited the website (31%). Ultimately 98 eyes met the criteria for statistical analysis. The mean difference between the two tests was low and non-significant: 0.02 logMAR (SD 0.12, P = 0.085). The 95% limits of agreement ranged from -0.21 to 0.26 logMAR. No relevant differences were identified in clinical characteristics between subgroups with a good performance (i.e. difference between the tests <=0.15 logMAR) or underperformance (i.e. difference >0.15 logMAR) on the web-based test. Conclusion: The web-based visual acuity test is a promising tool to remotely assess visual acuity in the majority of uveitis patients, especially relevant when access to ophthalmic care is limited. No association between patient- or uveitis-related variables and (under)performance of the test were identified. These outcomes underline the potential of remote vision testing in other common ophthalmic conditions. A proper implementation of this web-based tool in health care could be of great value for revolutionizing teleconsultations. The COVID-19 pandemic impacted almost every aspect of life and put a lot of stress on health care systems. Most elective outpatient care had to be cancelled or rescheduled. 1 At the ophthalmological department of our academic hospital, the University Medical Centre Utrecht (UMCU), the Netherlands, a telehealth approach was introduced to assist with triaging urgency of appointments. 2 As such, patients were requested to perform a remote unsupervised visual acuity test, of which the validity was recently investigated by our research team just prior to Uveitis is an important cause of blindness in both developed (7-15%) and developing countries (24-25%). [4] [5] [6] It describes a heterogeneous group of inflammatory conditions of the eye with multiple clinical presentations, classified by The Standardization of Uveitis Nomenclatur (SUN) Working Group. [7] [8] [9] Common symptoms of disease activity include blurred vision, ocular pain, eye redness, and photophobia. 10, 11 Despite therapeutic strategies aiming to reverse visual loss in early disease stages, advanced disease complications, such as macular oedema, glaucoma, or retinal scars, can result in irreversible visual loss. 12, 13 In order to minimise the damage and incidence of complications, early detection and timely treatment are of great importance. 14 The UMC Utrecht is a Center of Excellence and national referral clinic for the diagnosis and treatment of uveitis. 15 Web-based self-tests can remove barriers to care and have the potential to enrich teleconsultations. Tools for self-testing of visual acuity these have already been successfully implemented in screening programs of visual impairment. 16, 17 We hypothesize that remote self-assessment tools could reduce frequencies of hospital visits, thereby lowering costs for patients for travelling, parking, taking time off from work, as well as saving valuable hospital resources and staff. Especially considering the acute reduction in access to health care during the pandemic, telehealth solutions are of great value to continue (eye) care delivery. The Amsterdam-based med-tech start-up company Easee BV, offers the world's first Conformité Européenne (CE) certified web-based visual acuity test that complies with all relevant International Organization for Standardization (ISO) standards. The accuracy of the webbased test by Easee BV to assess visual acuity has been validated in healthy individuals 3 and in keratoconus patients. 18 Promising results were found, especially in higher visual acuity ranges. 18 We hypothesize that an unsupervised self-assessment of visual acuity is a reliable substitute for the conventional in-hospital assessment, also in more complex patients. Uveitis patients are a challenging domain, since our institution hosts a particularly complex population, with a favourable age distribution regarding digital proficiency. Study design and recruitment of data This study was conducted at the University Medical Center Utrecht (UMCU), the Netherlands, a uveitis Center of Excellence and expertise center. Adult uveitis patients were invited by telephone or by mail to do the web-based test at home, prior to their hospital consultation, between April 2020 and September 2020. For this study, we included the patients who performed the Web-based visual acuity test one to fourteen days before the conventional in-hospital visual acuity assessment. Patients were instructed to redo the web-based eye exam, or reach out to the study team, whenever they experienced a change in visual acuity in the interval between the two assessments. We excluded patients who were unable to complete the Web-based test successfully, who did not have a reference test outcome assessed within the 14-days interval, who coincidentally changed their glasses or contact lenses, or who reported a change in subjective visual acuity between the two assessments without redoing the web-based eye exam. For each patient, the following data were collected from the electronic health record: sex, age, uveitis classification, medical history, use of medication, symptoms associated with uveitis, any coexisting ophthalmic disease at time of the appointment, and visual acuity outcomes. All ophthalmologists in the UMCU practice according the SUN classification criteria. 9 Disease activity was classified based on the vitreous haze (VH), anterior chamber cell count (ACC), optical coherence tomography and fluorescent angiography and dichotomized into, 'inactive' (ACC and VH ≤ 0.5 and not considered active by the expert) or 'active' (ACC or VH > 0.5 or considered as active by the expert). Web-based assessment (index test) and conventional visual acuity assessment (reference test) Patients were invited to perform the web-based test in their home environment, prior to their hospital appointment. The test is accessible via a dedicated URL. Users need a computer or tablet, and a smartphone to do the test. In short, the smartphone functions as a remote control by which the user submits input from a distance of 3 meters to the computer screen. The set-up is depicted in Figure 1 below. The screen displays a sequence of optotypes that the user must correctly identify. Audio instructions guide the user through the test. Patients are asked to wear their habitual spectacles or contact lenses, if applicable. Patients can choose to measure one or both eyes. We instructed patients to assess both. Conventional measurements were assessed during routine consultations by ophthalmologists, residents in training, or optometrists, using a Snellen chart (the gold standard in our hospital). The health care providers were blinded for the previous web-based test results of the subjects. Our main outcome is the measurement accuracy of the web-based visual acuity test, as compared to the conventional visual acuity assessment at hospital consultations within 14 days afterwards. The web-based outcome was reported in logMAR. The Snellen decimal score of the hospital assessments was converted to the logarithm of the minimum angle of resolution (logMAR). The measurement accuracy is expressed by mean differences between the two assessments, and 95% Limits of Agreement (95%LoA). The mean difference can be interpreted as the systematic difference between the assessments (bias) and the 95%LoA as the range within 95% of the differences between one assessment and the other are included (random error). This methodology was first introduced by Bland and Altman, and is commonly used in method comparison studies to evaluate the agreement between two measurements on a continuous scale. 19 . CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted April 20, 2021. ; Variable outcomes when repeatedly performing visual acuity tests in an individual are common. [20] [21] [22] We will consider a difference of more than ±0.15 logMAR clinically relevant. This is supported by an authorative cross-sectional study in a large eye clinic using various charts and observers. 20 The minimum visual acuity (VA) score that can be determined by the web-based test is 0.05 Snellen decimal (1.3 logMAR). Lower scores will be described as "<0.05 Snellen decimal" (>1.3 logMAR). As their exact test value was unknown, patients with these poor VA scores were not compared in the Bland-Altman analysis, but descriptively analysed as a subgroup. We did an additional subgroup analysis to investigate associations between clinical characteristics and the agreement between the index and reference test: patients with differences greater than 0.15 logMAR (i.e. underperformance of the web-based test), vs. patients with differences smaller than or equal to 0.15 logMAR (i.e. good performance of the web-based test). Chi-squared and independent sample t-tests were used to assess differences between these groups. To correct for bilaterality (both eyes of the same patient included), the statistical tests for age, sex, and interval between the two tests, were presented per eye separately. A multivariate generalized estimating equation (GEE) assessed the association between independent variables and visual acuity outcomes of both tests. The GEE model was constructed to correct for bilaterality, age, sex, use of mydriatics, visual acuity influencing comorbidities, symptoms associated with uveitis activity and the interval in days between the two assessments. The study was performed in accordance with Dutch privacy laws and the Declaration of Helsinki. The data collected by Easee BV is stored on GDPR and HIPAA compliant servers in the EU. All included patients were registered in the registry 'Ocular inflammation, METC 17-363', that contains a written informed consent for the use of data. A total of 269 patients were invited by mail to execute the web-based exam at home, prior to their hospital visit. Ultimately, 84 patients visited the Easee website, and 72 patients finished the test successfully. Participants did not structurally report their reason for non-participation, though. Patients who visited the URL were significantly younger than those who did not (47.2 vs. 53.3 years, p=0.012). The participation flowchart can be found in the supplemental data. Included patients A total of 59 patients met all inclusion criteria, and 98 eyes were included for analysis (not all patients followed the instruction to assess both eyes). The clinical characteristics of the study population are summarized in Table 1 . Most of the included patients were female (67.8%). The mean interval between the two tests was 4.8 days ±2.7 days. At time of the appointment, symptoms of potentially active uveitis, such as ocular pain, floaters, photophobia, or visual loss, were present in 26.5% of the eyes. Based on the SUN classification 9 only 27.1% of the patients had uveitis anterior and 96.6% had a chronic course. At time of the appointment, 24.5% of the eyes were described as "active inflammation" of uveitis, while 74.5% of the eyes were "inactive". . CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted April 20, 2021. ; is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted April 20, 2021. ; Measurement accuracy of the Web-based visual acuity test The mean web-based VA was 0.12±0.25 logMAR (0.86±0.37 Snellen), compared to 0.10±0.25 logMAR (0.89±0.32 Snellen) for the conventional Snellen chart. The Bland-Altman plot in Figure 2 visualizes the comparisons between the two tests for 91 comparisons (seven patients had a web-based test-score of <0.05 Snellen). The mean difference was 0.02 ±0.12 logMAR (p=0.085). The 95%LoA ranged from -0.21 to 0.26 logMAR, without indications for a proportional bias. Overall, a majority of comparisons (76.9%, n=70) fell within the predetermined accepted deviation limit of ±0.15 logMAR. The cumulative distribution of the differences demonstrates that most of the differences were close to zero (i.e. no difference). Subgroup analysis of poor visual acuity outcomes The visual acuity scores of seven patients were lower than the minimum detectable value of the webbased test. For six of these patients, the visual acuity was correctly determined by the web-based test: these patients also had a visual acuity test score of <0.05 Snellen on the conventional assessment. Surprisingly, one patient had a reference test score of 0.4 Snellen decimal. Upon inquiry, the patient reported that the test was technically difficult to perform, and that the outcome therefore was not representative for the actual visual acuity status. The results of the subgroup analysis are shown in Table 2 . Multiple trends were visible in the 'underperformance' group (n=21 eyes) when compared to the 'good performance' group (n=70 eyes), though none of these differences reached statistical significance. In the 'underperformance' subgroup, mean age was higher (for OS: 50.67 vs. 43.89 years, p=0.250). Also, there were more visual acuity . CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted April 20, 2021. ; influencing comorbidities at the time of appointment (42.9% vs. 24.3%, p=0.098), and less anterior uveitis (14.3% compared to 34.3%, p=0.078) in the eyes that showed poorer agreement between two tests. Furthermore, a higher percentage of inactive uveitis (85.7% compared to 71.0%, p=0.176) was seen in the underperformance group. Generalized estimating equation analysis The GEE analysis revealed no significant associations between the independent variables and the visual acuity outcomes of the tests. There are no factors that seem to influence visual acuity outcomes of both tests individually, or the difference between the two tests. The studied associations and their outcomes can be found in the Supplemental data. . CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted April 20, 2021. ; is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted April 20, 2021. c: according to the SUN classification 9 d: When ACC and VH is <= 0.5 and not called active by the ophthalmologist e: When ACC or VH is >=1 or called active by ophthalmologist f: invalidated, because >20% expected count less than 5 . It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted April 20, 2021. This study determined the diagnostic accuracy of the novel Easee web-based test to remotely selfassess visual acuity in complex uveitis patients, and compared these outcomes to a conventional inhospital assessments shortly afterwards. Based on our findings, the web-based test seems a promising instrument to estimate visual acuity of uveitis patients in their home-environment, especially relevant when access to health clinics is limited. We found a clinically negligible mean difference of 0.02 ±0.12 logMAR between the web-based test and the conventional chart assessment. This mean difference is evidently lower than the predetermined accepted deviation limit of ±0.15 logMAR. The distribution of the differences, expressed by the 95%LoA, ranged from -0.21 to 0.26 logMAR, which slightly exceeds the predetermined accepted deviation limit, meaning that some of the individuals had higher differences between the two assessments than expected based on normal measurement variation. Notwithstanding, the agreement between the two assessments was good for the majority of the comparisons (77%). Sub-group analyses identified no significantly clinical or patient factors influencing the agreement between the two tests. When comparing visual acuity assessments, there are two important phenomena that should be taken into account. First, there is always variation when repeatedly assessing visual acuity within the same individual, even in the absence of a clinical change. A study that focused on test-retest variability of Snellen visual acuity charts reported 95%LoA ranging from ±0.18 logMAR (single letter method) to ±0.33 logMAR (line assignment method). 21 The line assignment method of the Snellen chart (where the test is terminated when half or more of the letters are misread) remains the most popular method in clinical practice, despite the introduction of more reliable alternatives such as the ETDRS chart 22 (with reported test-retest variabilities ranging from ±0.07 to ±0.18 logMAR 23, 24 ). Convenience (familiarity, short test duration) seems to outweigh accuracy in large and busy eye clinics. When comparing the abovementioned test-retest 95%LoA to the range found in our study, the remote webbased exam does not appear to underperform compared to the Snellen chart's line assignment method, the most used VA test in our hospital. A second phenomenon, is that there is always variation when two different types of visual acuity charts are compared. 20, 25, 26 The web-based test used a presentation of optotypes based on tumbling-E and proprietary optotypes, while the conventional exam was a Snellen letter chart. A conversion effect contributes partly to the observed differences in outcomes between the Web-based test and the conventional chart. Before this study, our team had evaluated the accuracy of the web-based eye exam of Easee in a cohort of healthy individuals 3 and a cohort of keratoconus patients. 18 When comparing our current results to previous findings, the measurement accuracy of the tool appears to be improved: while the mean differences in VA were already comparable, the current distribution of VA differences is evidently smaller (e.g. in healthy individuals the 95%LoA were ±0.43 logMAR from the mean difference). Clearly, the measurement accuracy has been improved as the algorithm and test flow of the tool have been updated over time. An important difference with the previous studies, however, is that in this study we focused on corrected visual acuity, whilst previously, uncorrected visual acuities were assessed. For uncorrected VA assessments, there will be individuals in the lower VA ranges due to unmitigated refractive errors. Accuracy of assessments is known to be suboptimal in these lower VA ranges, especially for Snellen charts. 22 In higher visual acuity ranges, web-based tools appear very reliable, as demonstrated by this study. . CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted April 20, 2021. ; This was the first study from our group to evaluate the web-based tool assessments in a completely unsupervised condition; in patient's own home environment, where lightning and test set-up were only controlled by the algorithm itself. We value this greatly, as it best reflects the true observations after implementing these tools in clinical practice. Notably, the unsupervised conditions did not seem to materially impact diagnostic accuracy. A limitation of the study design was that only patients were included who were willing to participate, and who were able to successfully complete the web-based eye exam in their home environment. The study design induced a participation bias of digitally competent patients, often of a younger age, which might affect the generalizability of the study outcomes. Diagnostic accuracy might be poorer in less digitally competent patients, as remote exams might be performed incorrectly. However, it is important to bear in mind that our inclusion criterion of successful completion of the web-based eye exam does not imply adequate performance. In other words: a patient can 'finish' an exam (at home) without conducting it correctly. In this study, these patients were included in our analyses as well, as information of outliers is important for interpreting the measurement accuracy and identifying patient characteristics that relate to bad performance. Another possible limitation of the design is the interval between the two assessments. Ideally, two compared VA assessments should be conducted within a time interval as short as possible, to prevent clinical changes to impact observed differences. We consider the mean interval in our interval to be fairly short (mean 5 days ± 3 days). In addition, patients were instructed to redo the web-based exam or contact the research team if they experienced a change in visual acuity between performing the web-based eye exam and their hospital appointment. We therefore do not expect the interval to have strongly impacted the observed differences. The results of this study indicate that an unsupervised remote web-based visual acuity test is a valid replacement of an in-hospital assessment for the patients who are willing and able to perform a selftest. It has the potential to enhance teleconsultations in ophthalmic care and creates opportunities for chronic uveitis patients to self-assess their visual acuity at home, when they are suspecting that their visual function deteriorates. From our own patient board we learned that, this self-control is considered very important for reassurance and psychological comfort. Most patients seem to be welcoming this tool. Potentially, a web-based eye exam in combination with telephonic-or videoconsultations, might increase future intervals between periodic follow-up measurements in chronic (and mostly stable) patients, reducing the burden and costs of frequent hospital visits. We strongly recommend introducing telehealth tools like these on individual levels and in close dialogue with the patient. Personalised risk analyses and adequate patient-instructions are essential. We do not claim this web-based tool to be a replacement of an ophthalmic investigation nor that it is by itself sufficient for adequate follow-up in all uveitis patients. A remote web-based eye exam will obviously never replace complete ophthalmic examinations, including split lamp examinations, essential for cell counting. Importantly, not all uveitis patients will notice visual loss a first symptom loss of increasing activity. [27] [28] [29] Based on our subgroup analyses, it is not possible to preselect uveitis patients for whom this web-based test is considered unsuitable. The sample sizes of the subgroups were small, so true effects might have been missed due to a lack of power. We hypothesize that the test performance will be mostly influenced by individual factors such as digital competence or environmental test conditions (such as lighting and test set-up). In general, we expect remote visual acuity assessments to be universally applicable for both healthy individuals, as well as individuals with ophthalmic conditions. We propose that the measurement accuracy that we found in this study will not be materially different for similar-aged patients with eye conditions other than uveitis. An interesting avenue would be to enrich the web-based platform with a near vision test (reading distance) to internally validate the VA results. Another interesting feature would be the remote assessment of contrast sensitivity and stray light, since both are related to quality of vision and might influence the visual acuity measurements. Unsupervised telehealth tools require a certain level of digital skills of the intended user. We assume that a lower digital competence influences the willingness to perform the test and that it might impact the diagnostic accuracy of unsupervised remote exams (as exams might be performed incorrectly). We recommend future usability testing and clinical research to focus the performance of the web-based tool in older patients, prior to implementation of these tools in clinical practice. The web-based Easee visual acuity test shows a reliable diagnostic accuracy in the large majority of the studied uveitis patients. No uveitis-related factors influencing accuracy of the web-based test were identified. A proper implementation of this web-based tool in health care could be of great value for enriching teleconsultations, especially when access to hospital care is limited. . CC-BY-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted April 20, 2021. Multivariate analysis to identify the associations between independent variables and the visual acuity outcome for the difference in logMAR between the web-based BCVA and manifest BCVA, the web-based BCVA in logMAR and the manifest BCVA in logMAR. LogMAR difference between the BCVA of the web-based and manifest assessment a BCVA in logMAR of the web-based assessment BCVA in logMAR of the manifest assessment B: beta value a: absolute difference between BCVA calculated in logMAR (web-based minus manifest) b: analyzed using an Generalized Estimating Equations to correct for inclusion of 2 eyes of one patient c: reference female d: reference no use of mydriatics e: visual acuity influencing comorbidities such as: (secondary) cataract, keratitis, scleritis, corneal lesion or history of pars plana vitrectomy f: reference no symptoms associated with uveitis (pain, floaters, photophobia, visual loss) at time of the manifest assessment g: interval in days between the web-based assessment and manifest assessment . It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted April 20, 2021. ; https://doi.org/10.1101/2021.04.14.21255457 doi: medRxiv preprint Gezondheid en zorg -Ziekenhuizen Emerging AI and Big Data Applications in Ophthalmology -A preview of innovations and ideas Validation of an independent web-based tool for measuring visual acuity and refractive error via the MORE (Manifest Versus Online Refractive Evaluation) trial: Prospective open-label noninferiority clinical trial The possible impact of uveitis in blindness: A literature survey The natural history of uveitis Review on the worldwide epidemiology of uveitis Prim Care -Clin Off Pract A glimpse into uveitis in the aging eye Standardization of uveitis nomenclature for reporting clinical data. Results of the First International Workshop Long-term clinical outcome and causes of vision loss in patients with uveitis Impact of Macular Edema on Visual Acuity in Uveitis Evidence-based treatment for uveitis Validation of Peek Acuity application in pediatric screening programs in Paraguay Smartphone-based screening for visual impairment in Kenyan school children: a cluster randomised controlled trial The evaluation of a web-based tool for measuring the uncorrected visual acuity and refractive error in keratoconus eyes: a prospective open-label method comparison study. medRxiv Statistical methods for assessing agreement between two methods of clinical measurement Variability of measurements of visual acuity in a large eye clinic The development of a "reduced logMAR" visual acuity chart for use in routine clinical practice Prospective evaluation of visual acuity assessment: A comparison of Snellen versus ETDRS charts in clinical practice (an aos thesis) Development of a clinically feasible logMAR alternative to the Snellen chart: Performance of the "compact reduced logMAR" visual acuity chart in amblyopic children Visual acuity scored by the letter-by-letter or probit methods has lower retest variability than the line assignment method Development and validation of a smartphone-based visual acuity test (peek acuity) for clinical practice and Community-Based Fieldwork Is conversion of decimal visual acuity measurements to logMAR values reliable? Graefe's Arch Clin Exp Ophthalmol Understanding uveitis: The impact of research on visual outcomes Long-term follow-up of patients with birdshot retinochoroidopathy treated with systemic immunosuppression Contrast Sensitivity Among Patients With Birdshot Chorioretinopathy | IOVS | ARVO Journals