key: cord-0941050-6ie9fwb5
authors: Garcia Moraes Pagano, Cassia; de Campos Moreira, Tais; Sganzerla, Daniel; Matzenbacher, Ana Maria Frölich; Faria, Amanda Gomes; Matturro, Lucas; Cabral, Felipe Cezar; Rucks Varvaki Rados, Dimitris; Decavata Szortyka, Anelise; Falavigna, Maicon; Vinadé Chagas, Maria Eulalia; Harzheim, Erno; Gonçalves, Marcelo; Umpierre, Roberto; Lutz de Araujo, Aline
title: Teaming-up nurses with ophthalmologists to expand the reach of eye care in a middle-income country: Validation of health data acquisition by nursing staff in a telemedicine strategy
date: 2021-11-30
journal: PLoS One
DOI: 10.1371/journal.pone.0260594
sha: eb4a357ec4dd6858278bcf9daf802a45524e8238
doc_id: 941050
cord_uid: 6ie9fwb5

Telemedicine can be used to conduct ophthalmological assessment of patients, facilitating patient access to specialist care. Since the teleophthalmology models require data collection support from other health professionals, the purpose of our study was to assess agreement between the nursing technician and the ophthalmologist in acquisition of health parameters that can be used for remote analysis as part of a telemedicine strategy. A cross-sectional study was conducted with 140 patients referred to an ophthalmological telediagnosis center by primary healthcare doctors. The health parameters evaluated were visual acuity (VA), objective ophthalmic measures acquired by autorefraction, keratometry, and intraocular pressure (IOP). Bland-Altman plots were used to analyze agreement between the nursing technician and the ophthalmologist. The Bland-Altman analysis showed a mean bias equal to zero for the VA measurements [95%-LoA: -0.25–0.25], 0.01 [95%-LoA: -0.86–0.88] for spherical equivalent (M), -0.08 [95%-LoA: -1.1–0.95] for keratometry (K) and -0.23 [95%-LoA: -4.4–4.00] for IOP. The measures had a high linear correlation (R [95%CI]: 0.87 [0.82–0.91]; 0.97 [0.96–0.98]; 0.96 [0.95–0.97] and 0.88 [0.84–0.91] respectively). The results observed demonstrate that remote ophthalmological data collection by adequately trained health professionals is viable. This confirms the utility and safety of these solutions for scenarios in which access to ophthalmologists is limited.

Introduction Visual impairment is a serious problem in many different populations [1] . According to the World Health Organization (WHO), it was estimated that in 2015 there would be 36 million blind people in the world and millions more people with visual impairment lacking adequate treatment [2] . In Tropical Latin America, which includes Brazil, 1.64% of the population have visual problems considered severe or moderate [3] . The WHO developed a global action plan to reduce avoidable visual impairment and offer treatment to people with visual impairment. One of the action plan's goals was to increase investment in scientific research, with the objective of developing cost-effective, evidence-based diagnostic techniques and interventions with adequate strategies for eye health care, particularly for mid and low-income populations [2] .

Employing telemedicine in ophthalmology is one way of facilitating patient access to specialist care, particularly for those in localities where there is a dearth of ophthalmologists, reducing travel costs, time off work, and waiting times for consultations [4] . This technology offers real-time interaction between doctors and patients and enables examinations to be conducted at a distance [5] . Telemedicine can be used to conduct ophthalmological assessment of patients, facilitating screening for diseases such as diabetic retinopathy and glaucoma [6] and enabling correction of refractive errors [7] .

Teleophthalmology models currently in use require data collection support from other health professionals, such as technology professionals, nurses, or nursing technicians. These trained professionals provide in-person support for collection of parameters, which may be conducted under ophthalmologist supervision, or unsupervised, and may be synchronous or asynchronous [7] [8] [9] .

An important challenge to be overcome in implementing teleophthalmology is to guarantee the quality of the data collected by these health professionals, which is generally based on training of human resources, standardization of routines, and regular assessments of the performance required to provide the reliable data needed for distance diagnosis [8, 10] . In this scenario, assessment of the fundamental parameters measured by the professionals involved in telediagnosis is extremely relevant to achieving development and ensuring the reliability of telemedicine.

The coronavirus disease 2019 pandemic illustrates the increasingly important role of telemedicine as a method of clinician-patient interaction. In this scenario, investigating interobserver agreement between specialist doctors and support professionals when acquiring health data for transmission and remote interpretation is fundamental for the validity of remote diagnosis. The objective of this study was to assess agreement between the nursing technician and the ophthalmologist in acquisition of health parameters that can be used for remote analysis as part of a telemedicine strategy. The health parameters evaluated in this study were visual acuity and objective ophthalmic measures acquired by autorefraction, keratometry, and air-jet intraocular pressure measurement.

A cross-sectional study was conducted during the months from October to December of 2017 at a teleophthalmology center in the city of Porto Alegre, Brazil. A total of 140 patients with ophthalmological complaints who had been referred to the ophthalmological telediagnosis center by primary care (PC) doctors were enrolled consecutively on the study. The teleophthalmology center (TeleOftalmo Project) consists of two remote examination rooms where visual acuity tests and automated refractometry and keratometry acquisition are conducted, photos are taken of the anterior segment of the eye under a slit-lamp, and retinography and non-contact air-jet tonometry are performed by a nursing team (nursing technicians and nurses) with support from ophthalmologist doctors who are responsible for overseeing and supervising the examinations via video links. The off-site ophthalmologist remotely evaluates the images and data simultaneously or immediately after its collection by the nurse technicians.

The objective of the ophthalmological telediagnosis model adopted is to diagnose eye conditions that are prevalent among patients cared for on the Brazilian National Health Service (SUS-Sistema Ú nico de Saúde). Doctors working in PC refer patients with nonspecific low visual acuity, refractive errors, strabismus, eyelid lesions, conjunctival lesions, and cataracts without prior indication for surgery and also diabetic patients for diabetic retinopathy screening. Patients in need of emergency treatment or with indications for medical or surgical treatment (e.g. laser, intraocular corticoids) were not eligible for the study [11] .

Data collection was conducted independently, in two remote examination rooms equally equipped with an ophthalmology specialist experienced in ambulatory consultations and by nursing technicians (NT) who had been trained in advance to collect the data involved. The NT had undertaken a training program with ophthalmological doctors, 10 hours of which were dedicated to theoretical-practical training on acquisition of examinations conducted at the TeleOftalmo Project [11] and another 14 hours of which were dedicated to practical training with the equipment used. At the time of data collection, the NT had been working at the project for 4 months and had had no previous experience with teleophthalmology.

Patients were randomized to be examined first by the NT or by an ophthalmologist. The randomization list was generated electronically and a nurse supervisor who did not take part in the examinations was responsible for patient allocation.

All examiners followed a standardized assessment protocol used for consultations at the ophthalmological telediagnosis center and data were collected using standardized and pretested forms. After each patient had undergone all of the examinations, the NT recorded the results on an on-line platform, used to store patients' registration data, in accordance with the normal flow of care. The ophthalmology specialist recorded the same information for each patient on an identical electronic form. Both sets of data collection were performed on the same day, in random order, and in sequential time. On average, patients were seen by the nurse technicians and by the ophthalmologist between 30 minutes and 1 hour apart.

The following data were collected: visual acuity (VA) without correction, using the modified Bailey-Lovie chart, refractomety and automated keratometry (Visuref 1001, Zeiss, Germany), and intraocular pressure (IOP), measured by non-contact air jet tonometry (Visuplan 5001, Zeiss, Germany).

Visual acuity was measured one eye at a time, covering the left eye (LE) when testing the right eye (RE) and the opposite when testing the LE, with the VA chart placed 4 meters away from the patient. During each test, the patient read the letters shown on each line out loud, or indicated the position of the tumbling E in the case of illiterate patients, starting with the line of the largest optotypes. For each measurement, the smallest line on which at least 3 of the 5 optotypes were correctly identified was noted. The measure was recorded as a Snellen fraction together with the number of optotypes not identified on the same line (for example, 20/25-2) or identified on the following line (for example, 20/25 +1). The autorefraction, keratometry, and IOP measurements were recorded as the mean of three automated measurements performed by the machines. Examinations were conducted one eye at a time, starting with the RE.

For the purposes of analysis and interpretation of results, the following cutoffs were predefined, as specified in Table 1: • Normative values for parameters: The usual cutoff points employed in clinical practice were adopted, for analysis of the capacity of each test to detect suspected or abnormal cases. The use of cutoff points in this study was based on a theoretical scenario in which each test, in isolation, would be indicative of a given clinical condition or suspicion of that condition (for example, IOP exceeding 21mmHg as indicative of ocular hypertension, glaucoma, or suspected glaucoma). For visual acuity, a cutoff point of 20/40 was adopted, in line with the new WHO classification of visual impairment (ICD-11).

• Expected variability: For the VA measurement, variability between repeated measures was expected to be in line with a previous test-retest study in the literature [12] . For autorefraction, automated keratometry and IOP, the assumed variability was that indicated by the manufacturer of the equipment, if available, or if not available, data from previous studies in the literature using similar equipment were used [13] [14] [15] .

Since we have four different measures, with same importance for study objectives, with uncertainty for dispersion parameters, the sample size estimate was based on effect size. Assuming a design effect (DEFF) of 1.7 [16] , the study would need a sample size of 262 pairs (131 patients) to achieve 80% power and a 5% significance level (two tailed) to detect a standardized mean difference of 0.23 (Cohen's d) between pairs, that is considered a small to non-important effect size in health sciences [17] . Allowing for possible losses, a sample of 140 patients would be sufficient. Continuous variables were expressed as means and standard deviations, or medians with interquartile ranges, while categorical variables were expressed as absolute and relative frequencies, with cumulative frequencies when appropriate. To perform the statistical analysis comparing the two assessments, the VA measurements were converted into the logarithm of the minimum resolution angle (logMAR), which corresponds to a continuous measure of VA with a linear scale, suitable for quantitative comparison of the mean VA. Visual acuities that were not covered by the values contained in the table employed (for example, finger counting or light/dark perception) were excluded from the analysis. The components of refraction measurement (sphere, cylinder, and axis) were converted to power vectors using the method described by Thibos and Horner [18] , to obtain the independent parameters M, J0, and J45, in diopters (D). Cohen's Kappa was calculated to measure the inter-rater reliability between the nursing technician and the ophthalmologist measurements in categorical variables. Bland-Altman plots were used to analyze agreement between the nursing technician and the ophthalmologist, in addition to scatter plots of comparison of assessments. The differences between measures taken by the nursing technicians and the ophthalmologists were assessed using mixed generalized linear models, to consider the dependence between the measurements and eyes. Sensitivity, specificity, Cohen's Kappa and 95% confidence intervals [95%CI] were calculated for VA and IOP, using the ophthalmologists' results as the reference standard. For visual acuity, a cutoff point of 20/40 was adopted, in line with the new WHO classification of visual impairment (ICD-11) and a cutoff point of � 21mmHg as indicative of ocular hypertension, glaucoma, or suspected glaucoma. Statistical analyses were conducted using R Core Team statistical software [19] . Informed consent was obtained from participants after explanation of the nature of the study and its possible consequences. The study was approved by the Research Ethics Committee at the Hospital de Clínicas de Porto Alegre (CAAE 64499316.1.0000.5327, consolidated opinion number 2.757.077). A printed consent form was applied and the participants signed in 2 copies, one kept by the researchers and the other delivered to the participants. The study did not include underage participants. Twenty-six of the visual acuity measurements were off the acuity chart, because vision was worse than the lower limit (visual acuity measurements using finger count or light perception), and were therefore excluded from the analysis. Twenty-six, 10, and 2 autorefraction, keratometry, and intraocular pressure measurements, respectively, could not be collected because of equipment limitations (for example, keratometry "out of range") or because of anatomic ocular abnormalities that prevented measurement (for example, an opaque cornea). The Bland-Altman plot ( Fig 1A) showed a mean bias equal to zero for the VA measurements taken by the doctors and the nursing technicians, and the limits of agreement showed that the differences between the two measures were less than -0. 25 (Fig 2A) . Therefore, 210 pairs (82.7%) exhibited agreement within the expected inter-test variability (up to 0.10 log-MAR). Considering the preestablished criterion for low VA (>0.3 logMAR), 108 eyes (42.5%) were detected with low VA according to the doctors' measurements and there was disagreement for 20 eyes (7.87%). Agreement for detection of low acuity was 92. Table 2 .

Considering the order of tests, patients' performance was better in the second VA, irrespective of whether the measurement was taken by the technicians or the doctors (data shown in Supporting Information). No differences were observed in agreement related to laterality (RE or LE).

The difference between means for technicians and doctors for the autorefraction spherical equivalent (0.00 [95%CI]: -0.13-0.13) was not statistically significant (p = 0.690).

The Bland-Altman plot (Fig 1B) illustrates the data for spherical equivalent (M) results. Bias was 0.01D between doctors and nursing technicians and the limits of agreement indicate that the differences between the two measures fall within the range of -0.86 to 0.88D. The correlation between measurements was 0.97 ([95%CI]: 0.96-0.98; p<0.001).

The spherical equivalent M results were in agreement for 77 assessments (30.3%), while 166 assessments (65.4%) had up to 0.5 D disagreement (Fig 2B) . No differences in the agreement assessment were observed in relation to laterality or order of assessment (doctor first or technician first).

The mean doctor-assessed keratometry result (K) was 43.8D ([95%CI]: 42.8-45.0). The difference between means for technicians and doctors (-0.13 [95%CI]: -0.25-0.00) was significant (p<0.001), but was not clinically relevant.

The Bland-Altman plot (Fig 1C) illustrates the data for keratometry (K) results in diopters (D). Bias was -0.08D between doctors and nursing technicians and the limits of agreement indicate that the differences between the two measures fall within the range of -1.11 to 0.95D. The correlation between measurements was 0.96 ([95%CI]: 0.95-0.97; p<0.001).

For the keratometry assessments, results for 59 pairs of eyes (21.9%) were in agreement and results for 203 pairs of eyes (75.2%) disagreed by less than 1D (Fig 2C) . The test's diagnostic properties could not be evaluated because just 1 participant had a value above the limit of normality (>48D) according to measurements taken by the doctor and none were outside of the 

The mean ophthalmologist-assessed IOP of patients was 15.0 mmHg ([95%CI]: 13.0-18.0). The Bland-Altman plot (Fig 1D) illustrates the data for IOP in mmHg. Bias between doctors and nursing technicians was -0.23 mmHg and the limits of agreement indicate that the differences between the two measures fall within the range of -4.44 to 4.00 mmHg. The correlation between measurements was 0.88 ([95%CI]: 0.84-0.91; p<0.001).

The difference between means for technicians and doctors (0.00 [95%CI]: -1.00-1.00) was not statistically significant (p = 0.085). A total of 72 IOP measurements (25.9%) were in agreement and 163 (58.6%) disagreed by up to 2 mmHg (Fig 2D) . Table 3 .

No differences in the agreement assessment were observed in relation to left or right eyes or order of assessment (S1 

Our results show that there were small discrepancies between the data collected by the doctors and by technicians. However, these discrepancies did not have major clinical relevance that could impact the ophthalmologist's decision on the patient's diagnosis. These findings provide evidence of the safety of teleophthalmology strategies with in-person support from nursing technicians.

Some methodological aspects of our study merit more in-depth discussion. Part of the variability in visual acuity assessments can be explained by the inherent effects of repeated measures in patients and would also be observed in repeated assessments conducted by a single professional.

In relation to visual acuity, patients may memorize the letters shown on the Snellen chart because of their repeated use during ophthalmological examinations [20] . This explains the fact that patients performed better during the second administration and, as a result, analysis of agreement between measures revealed greater variability, albeit still acceptable. The acuity charts used for data collection displayed the same optotypes, which is a limitation of the study that could have facilitated memorization by the patients. Additionally, measurement of acuity requires adequate training, experience with data collection, and support from qualified professionals to be conducted satisfactorily [21] . The nursing technicians who took measurements at the project only had 4 months' experience and this may also have been an important factor in some of the measurement discrepancies.

The autorefraction measure, expressed as the spherical equivalent M, exhibited satisfactory agreement in our study. The eye's refraction is subject to physical accommodation fluctuations, with frequency of 1 to 5 Hz and amplitude of approximately ± 0.1 to ± 0.25 D of ideal accommodation [22, 23] Subjective refraction is considered the gold-standard method for determination of the refractive status of the eyes, because it takes into consideration important factors not accounted for by autorefractors, such as neural processes and binocular equilibrium, although it too exhibits interobserver variability [24] . Because of this variability, refraction errors are not normally assessed to precision exceeding ± 0.25 D for smaller refraction errors or ± 0.50 D for larger refraction errors [25] . The keratometry measurements exhibited good agreement in the data from our study, but the spectrum of patients was restricted to healthy patients, so the data do not reveal how good performance would be in examinations of patients with keratoconus or other diseases of the cornea. Since it is a disease that does not reveal symptoms or clinical signs, it is extremely important to diagnose keratoconus during the initial stages of the disease [26] . Measurements of refraction and visual acuity are less sensitive for keratoconus than topography, but changes in these measurements are nevertheless important signs of the progression of this disease [27] . Our results cannot therefore be generalized to patients with suspected keratoconus.

The variability of examinations such as intraocular pressure measurement is dependent on minor variations such as patient positioning and breathing. In our study, agreement between tonometry measurements taken by nursing technicians and ophthalmology specialists was acceptable, but discrepancies did affect the classification of diagnostic suspicion for some patients. A number of important factors can interfere with this measure. In this study, patients were examined using Zeiss non-contact tonometers, with an expected variability of up to 4 mmHg, according to the manufacturer [28] and since the variability between professionals was within this limit, we can infer that the nursing technicians' assessment did not exhibit variability, considering expected variability. Non-contact tonometers (NCTs) have the advantage of pushing back the cornea with an air jet, reducing the possible risk of cross-infection, but are not always precise [29, 30] . Repeated readings with contact tonometers are more reproducible than other automated measurements [29] and are the gold standard for IOP measurement. Kotecha et al. [31] published studies showing that ophthalmologist doctors achieve better agreement in IOP measurements using contact tonometers and nursing technicians have better agreement with non-contact tonometers. Other variables can also interfere with IOP measures, including astigmatism, corneal thickness, biomechanics, accommodation, breathing, heart rate, and daily variation, resulting in a maximum clinically acceptable variation of ± 3.0 mmHg [32] .

Although the study has some limitations, such as the nursing technicians' length of experience with assessment of ophthalmological parameters (4 months), use of air-jet tonometry measurements only, and use of acuity charts showing the same optotypes, the results observed show that assessment by nursing technicians exhibits acceptable agreement.

One relevant aspect of the setting in which this study was conducted is that there is no specific professional category for ophthalmological assistants in Brazil and so very often nursing technicians are trained to perform this role, as is the case at the service where the study was conducted. Similar to Brazil, other countries in Latin America don't have standard professionals who can assist ophthalmologists. In Peru, for example, there are few certified ophthalmological nurses, the majority of them being trained in their work center only. In Honduras, healthcare professionals who have a role assisting in ophthalmologic procedures do not have formal training. The optometrist is only formally recognized in Peru, Honduras and Mexico. Meanwhile, countries such as Chile and Uruguay have ophthalmological technicians who collect refraction data [33] .

It is known that continuous training and continuing education lead to establishment of standards for the quality of the assessments conducted by these professionals. In other countries, ophthalmology assistant professionals have helped in screening and follow-on ophthalmological diseases such as glaucoma and diabetic retinopathy in primary care [34] [35] [36] .

Even though some studies have shown good accuracy in telemedicine diagnosis in the ophthalmology field [37] [38] [39] . The evaluation of the data necessary to perform a synchronous remote diagnosis, collected by technicians who are not specialists in ophthalmology, is important in order to validate the care provided by a non-medical professional for ophthalmological diagnosis by telemedicine.

The results found demonstrate that actions employing distance ophthalmological data collection by adequately trained health professionals with secondary or technical education are feasible options for providing care to populations that do not have access to specialist services. With support from adequately trained health professionals, ophthalmological healthcare provided via telemedicine enables access to eye health to be extended to populations with limited access to traditional services. 

Magnitude, temporal trends, and projections of the global prevalence of blindness and distance and near vision impairment: a systematic review and metaanalysis

Draft action plan for the prevention of avoidable blindness and visual impairment 2014-2019

Prevalence and causes of vision loss in Latin America and the Caribbean in 2015: magnitude, temporal trends and projections

Applying RE-AIM to evaluate two community-based programs designed to improve access to eye care for those at high-risk for glaucoma

The use of telemedicine by physicians: still the exception rather than the rule

Telemedicine in ophthalmology in view of the emerging COVID-19 outbreak. Graefe's Arch Clin Exp Ophthalmol

Accuracy of vision technicians in screening ocular pathology at rural vision centres of southern India

Clinical competency of 1-year trained vision technicians in Andhra Pradesh

Philadelphia telemedicine glaucoma detection and follow-up study: methods and screening results

Modeling a telemedicine screening program for diabetic retinopathy in Iran and implementing a pilot project in Tehran suburb

The use of telemedicine to support Brazilian primary care physicians in managing eye conditions: The TeleOftalmo Project

A modified ETDRS visual acuity chart for European-wide use

Precision (repeatability and reproducibility) and agreement of corneal power measurements obtained by Topcon KR-1W and iTrace

Repeatability and comparability of automated keratometry: the Nikon NRK-8000, the Nidek KM-800 and the Bausch and Lomb keratometer

Comparison of corneal powers obtained from 4 different devices

Graefe's Arch Clin Exp Ophthalmol = Albr von Graefes Arch fur Klin und Exp Ophthalmol

Effect sizes for interpreting changes in health status

Power vector analysis of the optical outcome of refractive surgery

R Foundation for Statistical Computing. R Core Team. A language and environment for statistical computing. 2020

Factors contributing to discrepancy between visual acuity fractions derived from a Snellen chart and letter scores on the early treatment diabetic retinopathy study chart

Uveal effusion: clinical features, management, and visual outcomes in a retrospective case series

Assessment of subjective refraction with a clinical adaptive optics visual simulator

Spherical subjective refraction with a novel 3D virtual reality based system

Refraction and visual acuity measurements: what are their measurement uncertainties?

Pentacam top indices for diagnosing subclinical and definite keratoconus

Keratoconus natural progression: a systematic review and metaanalysis of 11 529 eyes

Guia de instruções de uso Visuplan 500 Tonô metro sem contato

A comparison of four methods of tonometry: method agreement and interobserver variability

Systematic review of the agreement of tonometers with Goldmann applanation tonometry

Interobserver agreement using Goldmann applanation tonometry and dynamic contour tonometry: comparing ophthalmologists, nurses and technicians

Clinical evaluation of the Canon TX-10 noncontact tonometer in healthy eyes

Field Testing Project to Pilot World Health Organization Eye Health Indicators in Latin America

Early experience with Technology-Based Eye Care Services (TECS): a novel ophthalmologic telemedicine initiative

Implementation of medical retina virtual clinics in a tertiary eye care referral centre

Interobserver agreement using Goldmann applanation tonometry and dynamic contour tonometry: comparing ophthalmologists, nurses and technicians

Evaluation of the effectiveness of diagnostic & management decision by teleophthalmology using indigenous equipment in comparison with in-clinic assessment of patients

Accuracy of a technologyassisted eye exam in evaluation of referable diabetic retinopathy and concomitant ocular diseases