key: cord-1034291-cgpgw2v4 authors: Marques, Sandro; Cruz, June Alisson Westarb; da Cunha, Maria Alexandra Viegas Cortez; Tuon, Felipe Francisco; de Moraes, Thyago Proença; Daiane Zdziarski, Alaís; Bomher, Sean T.; Donnelly, Lane F.; Capasso, Robson title: Patient and family experience with telemedicine and in-person pediatric and obstetric ambulatory encounters throughout 2020, during the COVID-19 epidemic: the distance effect date: 2022-05-16 journal: BMC Health Serv Res DOI: 10.1186/s12913-022-08037-8 sha: 5a1b3fc6fb1d8b6cf75f6b04221b7b70699ce236 doc_id: 1034291 cord_uid: cgpgw2v4 BACKGROUND: Telemedicine has grown significantly in recent years, mainly during the COVID-19 pandemic, and there has been a growing body of literature on the subject. Another topic that merits increased attention is differences in patient and family experience between telehealth and in-person visits. To our team’s knowledge, this is the first study evaluating pediatric and obstetrics outpatients experience with telemedicine and in-person visit types in an academic maternal and children’s hospital, and its correlation with geographic distance from the medical center throughout 2020, during the COVID-19 crisis. METHODS: We aim to evaluate and compare patients’ telemedicine and in-person experience for ambulatory encounters based on survey data throughout 2020, during the COVID-19 pandemic, with particular focus on the influence of distance of the patient’s home address from the medical facility. A total of 9,322 patient experience surveys from ambulatory encounters (6,362 in-person and 2,960 telemedicine), in a maternal and children’s hospital during 2020 were included in this study. The percentage of patients who scored the question “Likelihood to recommend practice” with a maximum 5/5 (top box) score was used to evaluate patient experience. The k-means model was used to create distance clusters, and statistical t-tests were conducted to compare mean distances and Top Box values between telemedicine and in-person models. Logistic regression analysis was used to evaluate the correlation between Top Box scores and patients’ distance to the hospital. RESULTS: Top Box likelihood to recommend percentages for in-person and telemedicine were comparable (in-person = 81.21%, telemedicine = 81.70%, p-value = 0.5624). Mean distance from the hospital was greater for telemedicine compared to in-person patients (in-person = 48.89 miles, telemedicine = 61.23 miles, p-value < 0.01). Patients who live farther displayed higher satisfaction scores regardless of the visit type (p-value < 0.01). CONCLUSIONS: There is a direct relationship between the family experience and the distance from the considered medical center, during year 2020, i.e., patients who live farther from the hospital record higher Top Box proportion for “Likelihood to Recommend” than patients who live closer to the medical center, regardless of the approach, in-person or telemedicine. The term Telemedicine is used to define the exchange and use of medical information to deliver medical services from different sites, by using electronic communication networks. [1, 2] . Although telemedicine is not a novel concept [3] [4] [5] , it has grown significantly in recent years, in particular during the COVID-19 pandemic when non-face-to-face healthcare became essential [6, 7] . Consequently, there is increased interest on its outcomes and indications, and there has been a growing body of literature on the subject, as shown on Fig. 1 . Based on data obtained from the Scopus (Elsevier) and Web of Science databases, the pediatrics is one area with few publications (470), as showed at Fig. 2 A. Obstetrics has even less (253), and it is not showed at picture Fig. 2 A. There has been an increase in publications for these two specifics area, mainly in recent years: in pediatrics, it is noted an increase from 55 publications in 2019 to 174 in 2020, while in obstetrics, it increased from 36 in 2019 to 92 in 2020 (Fig. 2 B) . Another topic that merits increased attention is data from surveys of patient and family experience, but specifically for the two specialties considered here, there are just publications about telemedicine and patient experience. While the ideal methodological approach to measure it is still a matter of discussion [8, 9] , proxies such as patients' experience survey scores are still the most commonly utilized tools. Literature reviews and development of frameworks for telemedicine satisfaction measurements have been proposed [10, 11] . However, to our team's knowledge, this is the first study evaluating pediatric and obstetrics outpatients experience with telemedicine and in-person visit types in a large US-based academic medical center, and its correlation with geographic distance from the medical center during the COVID-19 crisis. There is a direct relationship between the family experience and the distance from the considered medical center, during year 2020, i.e., patients who live farther from the hospital record higher Top Box proportion for "Likelihood to Recommend" than patients who live closer to the medical center, regardless of the approach, in-person or telemedicine. Keywords: Telehealth, Patient Experience, Quality improvement, In-person visits, Likelihood to Recommend, Maternal and children's hospital, Logistic regression This is a retrospective analysis of patient's experience survey data administered at a maternal and children's hospital and associated clinics. Questionnaires were collected from January to December 2020 for in-person and telehealth visit types. Data included primary care and subspecialized pediatric and obstetric outpatient services. As part of our institutional oversight of quality, safety, and service -patient experience survey scores are monitored to determine where we are improving and what targeted actions may need to be taken. Following our institutional guidelines, this project met criteria as quality improvement activity and was not considered human subjects research, and as a result did not require approval by our Institutional Review Board. The 16-question patient experience survey was electronically collected from patients and families who underwent to medical services, in-person or via telehealth. The same 16-question patient experience survey (Table 1) was used in other studies developed at this hospital unit: "The survey is a tailored version of an ambulatory survey created in cooperation with a vendor (Press Ganey, South Bend, IN), validated and used at multiple healthcare systems throughout the United States" [12] . Each question was evaluated on a scale of 1-5. A grade of five (5/5) was considered Top Box [12] . The organization considers the percentage of Top box scores as a standard metric for patient experience. For the purposes of this study, the answers to the question 13-Likelihood of your recommending our practice to others (LTR) were considered for further analysis. This metric has been extensively studied, and is frequently utilized in patient experience improvement efforts [13] [14] [15] . Patient's distance to the medical facility was calculated by an interface with Google Maps service utilizing the zip code for each patient's residence. The Top Box data was then stratified by micro-regions, and the averages were weighted by the number of responses from each zip code. Zip codes were clustered according to distance to the medical center, and a k-means model, a clustering algorithm that associates an identification to each group of elements that share the same cluster, was used [16] . The distance between the elements and each centroid is calculated using a cartesian plane with clusters on the x-axis and distances on the y-axis. The number of centroids defines the number of desired clusters. If only one centroid is considered, we will have only one cluster. With two or more defined centroids, each element is associated with the centroid closest to it. Since the number of centroids (or clusters) desired must be indicated in the algorithm, a calculation was made to determine the ideal number of clusters to minimize the sum of the distances of each element to the centroids. In a range from 1 to 15 centroids, the amount that minimizes this sum and, therefore, the ideal number of clusters, was three [3] . As all patients living within the same zip code were considered to have the same distance to hospital, distance outliers were eliminated. To compare mean values Our concern for your privacy for Top Box and distance for telehealth and in-person groups, a statistical t-test was conducted. To evaluate the relationship between distance and visit type (independent variables) with Top Box (dependent variable), a logistic regression model was used. All statistical analysis was made using R Studio [17] . A total of 9,322 patient experience surveys (for 6,362 in-person and 2,960 telemedicine encounters) were analyzed, from pediatric and obstetrics outpatients encounters during year 2020. They were answered by patients or families living in 30 American states, 406 cities, and 694 zip codes. Table 2 shows that most micro-regions were located less than 193.77 miles from the center, with a mean of 248.16 and a median of 93.44 miles. This big difference between mean and median is influenced by 126 individual patient surveys, which addresses have the farthest 93 zip codes, with distance greater than 421.61 miles (limit value to be statistically considered an outlier) and although few, significantly influenced the means. Therefore, we excluded the evaluations from these 93 zip codes as they were considered outliers, leaving 9,196 individual patient encounters. Therefore, 98.6% of the patient encounters and 86.6% of the original zip codes were retained in the study. For clarity, clusters were renamed to represent the average distance from the cluster to the hospital as "Close," "Intermediate," and "Far". According to their distances to the hospital, the cluster compositions are presented in Fig. 3 , which is a representation of zip codes and clusters. For "Close", the median distance from the zip code to the hospital was 46.1 with a range of 0.0 to 104.3 miles. For "Intermediate", the median distance from the zip code to the hospital was 159.0 with a range of 105.7 to 229.3 miles. For "Far", the median distance from the zip code to the hospital was 288.0, with a range of 233.7 to 419.3 miles. The distribution of zip codes and the associated mean score for "likelihood to recommend" are presented in Fig. 4 , where each zip code is represented by a circle, and the size of the circle represents the number of evaluations in that micro-region. Note that most experience scores in the "Close" cluster are above the value of 50% for Top Box LTR. Average patient experience by visit type (in-person or telemedicine), and number of completed surveys are shown in Fig. 5 . Despite monthly variations in the Top Box percentages for each modality, the t-test comparing the mean Top Box between two visit types, showed that there was no statistical difference between the average of telemedicine and in-person service evaluations (in-person = 81.21%, The relationship between average distance from the patient's home and the healthcare center by visit type is shown in Fig. 6 . Comparing the average distance of consultations by telemedicine relative to in-person, t-test analyses demonstrate a statistical difference (in-person = 48.90 miles, telemedicine = 61.24 miles, p-value < 0.01) confirming that the average distance of telemedicine services was greater than average distance of patients on in-person visit type, as shown in Table 3 . Considering that distance is a relevant factor for telemedicine (the prefix 'tele' , from the Greek 'telos' , implies distance [18] ), we analyzed the relation between visit type, distance and Top Box proportion for LTR for patient surveys during 2020. The variable Top Box was created from transformation of variable LTR: if the LTR was equal to 5, Top Box value was defined to 1; if the LTR was less than 5, the Top Box value was defined to 0. Figure 7 shows that there is a statistically significant relation between average patient's distance from hospital and Top Box proportion for LTR. The greater the average distance, the greater the proportion of Top Box. This trend is valid for both telemedicine and in-person services. The Top Box proportion for the "Close" cluster was 78.2% for in-person and 79.3% for telemedicine. For the "Intermediate" it was 86.7% for in-person and 84.7% for telemedicine, and for the "Far" cluster, the proportion of Top Box was 91.8% for in-person and 89.3% for telemedicine. Logistic regression confirmed that there was a statistically significant difference between the Top Box proportion, used as dependent variable, and patient distance In our geographic region, location is highly related to median household income [23] . Our healthcare center is located in the middle of Silicon Valley in which many zip codes have high median household incomes. Patients who travel farther distances to our medical center often come from the Central Valley (Fig. 3) . Many of the zip codes in this region have a lower median household income. It is possible that patients and families from areas with lower median household incomes are more appreciative and provide higher patient experience scores. This is speculative but one of the limitations to our evaluation of differences Further studies are required on this topic. In conclusion, our analysis revealed a direct relationship between the proportion of Top Box evaluations, as a proxy for patient and family experience with pediatric and obstetric services, and the distance from the considered medical center, during year 2020, i.e., patients who live farther from the hospital record higher Top Box proportion for "Likelihood to Recommend" than patients who live closer to the medical center, regardless of the approach, in-person, or telemedicine. The possible reason for this direct relationship was not evaluated and could be analyzed in future research. translation from Portuguese to English and thorough paper revision on all versions. MC worked on manuscript strategy planning, ideas validation and Portuguese version revision. FT worked on idea validation, Portuguese version revision and structured the paper to medical journal submission. TM worked on manuscript strategy planning, ideas validation, manuscript revision on Portuguese and English versions. AZ supported all organization, and description of telemedicine and quality research, Portuguese version text revision and formatting, figures and tables revision and final validation. SB worked on data collection, validation of data usage and analysis, paper revision on English version. LD worked on paper strategy planning, ideas validation, thorough paper revision on English version. RC (Senior author) worked on research planning; manuscript strategy; ideas validation; intermediary data and final analysis validation; thorough manuscript revision on all versions. All authors read and approved the final manuscript. This research did not have any form of funding from the university or third parties, being fully funded by its authors. The datasets generated and/or analyzed during the current study are not publicly available due to Stanford Medicine's institutional policies, as quality/ performance data access is limited, but are available from the corresponding author on reasonable request. Ethics approval and consent to participate All methods were carried out in accordance with relevant guidelines and regulations. All experimental protocols were approved by Stanford University's Research Compliance Office. Following our institutional guidelines, this project met criteria as quality improvement activity and was not considered human subjects research by Stanford University's Research Compliance Office (our Institutional Review Board), and as a result, received exemption of approval documented by e-mail. What is telemedicine? A collection of 104 peer-reviewed perspectives and theoretical underpinnings Albany: University At Albany Exploring the Determinants and Experiences of Senior Stroke Patients with Virtual Care Inside Look at Using Telemedicine During COVID-19 Pandemic American Academy of Family Implementing Telemedicine in Response to the 2020 COVID-19 Pandemic Telemedicine and patient satisfaction: current status and future directions Systematic review of studies of patient satisfaction with telemedicine Towards a Conceptual Framework for the Evaluation of Telemedicine Satisfaction Conceptual framework to evaluate health care professionals' satisfaction in utilizing telemedicine Comparing patient experience survey scores between telehealth and in-person ambulatory pediatric subspecialty visits Unlocking the Doors to Patient Satisfaction in Pediatric Orthopaedics Physician impact on the patient and family experience in a pediatric otolaryngology practice Becoming a leader in patient satisfaction: Changing the culture of care in an Academic community Hospital Introduction to Deep Learning: From Logical Calculus to Artificial Intelligence RStudio: Integrated Development Environment for R Telemedicine: history, applications, and impact on librarianship Evaluation of teleconsultation system in the urological patient during the COVID-19 pandemic Family Satisfaction: Differences Between Nursing Homes and Residential Care Facilities Evaluation of Neonatal Services Provided in a Level II NICU Utilizing Hybrid Telemedicine: A Prospective Study Rapid Implementation of Telemedicine During the COVID-19 Pandemic: Perspectives and Preferences of Patients with Cancer American Community Survey Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations Not applicable.Authors' contributions SM (main author) conducted all planning, data preparation, interfaces with Google Maps services, data analysis, and final version writing and formatting. JC worked on manuscript strategy planning, ideas validation, organized paper • fast, convenient online submission Ready to submit your research Ready to submit your research ? Choose BMC and benefit from:? Choose BMC and benefit from: This manuscript does not contain any individual person's data or image. The authors declare that they have no competing interests.