key: cord-0712236-hfjgpbce authors: Özkeskin, Mehmet; Özden, Fatih; Ekmekçi, Özgül; Yüceyar, Nur title: The reliability and validity of the Turkish version of the Telemedicine Satisfaction and Usefulness Questionnaire (TSUQ) and Telemedicine Patient Questionnaire (TPQ) in individuals with multiple sclerosis date: 2021-08-03 journal: Neurol Sci DOI: 10.1007/s10072-021-05519-1 sha: 01bb186180a244ace052cf1dfb090676a4b6fa81 doc_id: 712236 cord_uid: hfjgpbce OBJECTIVE: The study was aimed to translate and adapt the Telemedicine Satisfaction and Usefulness Questionnaire (TSUQ) and Telemedicine Patient Questionnaire (TPQ) into Turkish, thereafter to analyze the psychometric properties of both questionnaires. METHODS: A total of 149 multiple sclerosis (MS) patients were recruited in the study. For 4 years, all patients were supervised by a department clinician using telemedicine. Cronbach’s alpha coefficient was used to assess internal consistency. By evaluating the scores of 41 retested patients 1 week later, the test-retest reliability was determined using the intraclass correlation coefficient (ICC). Pearson’s correlation coefficient was used to assess the construct validity (r). RESULTS: A total of 149 patients (103 women, 46 men) with a mean age of 40.9±10.9 years were included in the study. MS duration of the patients was 9.15±6.24 years. Internal consistency of all items and the total score of the TSUQ were excellent (>0.80; ranged 0.971–0.974). On the other hand, the internal consistency of all items and total score of the TPQ was excellent, either (>0.80; ranged 0.878–0.890). The ICC of the TSUQ’s and TPQ’s total score was excellent (>0.80). The correlation between TSUQ and TPQ was strong (r=0.734, p<0.01). In addition, there was moderate relationship between the TSUQ and the Beck Depression Scale (BDS) (r=−0.363, p<0.01). On the other hand, there was low correlation between TPQ and BDS (r=−0.217, p<0.05). CONCLUSION: The Turkish version of the TSUQ and the TPQ is valid and reliable in individuals with MS. Telehealth services are common in neurological assessment and rehabilitation, including stroke, multiple sclerosis (MS), Parkinson's disease, and epilepsy [1] [2] [3] [4] . MS patients need specific healthcare services due to different neurological deficits. Individuals with MS living in rural areas with low socioeconomic levels and mobility problems experience difficulties receiving neurological care. It has been reported that at least 30% of individuals with MS do not have access to healthcare. In addition, MS patients who receive neurological care have more opportunities to receive specific health services such as physiotherapy and occupational therapy [5] . Disease-modifying therapies and neurorehabilitation are essential in the effective management of MS and are provided by neurorehabilitation specialists with proper coordination [6] . Robb et al. compared telemedicine and face-toface visits in individuals with MS. It has been stated that the telemedicine service provided in the virtual environment for individuals with MS is low cost and saves time. Moreover, it has been reported that it is feasible and preferable for the clinician and that individuals with MS have a high level of satisfaction [7] . In both telemedicine and face-to-face interviews, patient and caregiver satisfaction has been reported to be similar. Individuals suffering from neurological diseases can receive fast and accessible healthcare services with telemedicine service and reduce the cost of healthcare services [8] . The services provided within the scope of telemedicine service in individuals with MS have been reported as functional assessment, treatment follow-up, treatment compliance, cognitive assessment, cognitive rehabilitation, mindfulness, and telerehabilitation [7, [9] [10] [11] [12] [13] . In a systematic review on this subject, evidence about the positive effects of telerehabilitation on fatigue, function, and quality of life of individuals with MS has been presented [14] . Moreover, it was stated that, with telerehabilitation, people with MS increased their participation in exercise and their level of satisfaction was good [2] . In order to get the maximum benefit from telehealth technologies, the telehealth application system can be used effectively for clinicians and patients [15] . In addition to the quality and cost of telemedicine services, the acceptance of this service by individuals, its availability, and satisfaction level should be evaluated [16] . The availability of telemedicine services can be made by observation, questionnaire, interview, and different recording methods [17, 18] . Clinical results of telemedicine services can be measured with questionnaires, quality of life, and different tools specific to the disease [19] . In the literature, the expectations, satisfaction, and usability of individuals who receive telemedicine services are evaluated with different surveys: Telemedicine Satisfaction and Usefulness Questionnaire (TSUQ) and Telemedicine Patient Questionnaire (TPQ) [20, 21] . The psychometric properties of the Turkish versions of both questionnaires were not investigated. Besides, there is no other tool that shows the availability and satisfaction of Turkish-speaking individuals' telehealth services. Thus, it is not possible to control the usability of telehealth applications and receive patient feedback in terms of satisfaction in a standard way. The present study aims to translate and adapt the TSUQ and the TPQ into Turkish, and to analyze the psychometric properties of both questionnaires. Permission was obtained from Bakken and Demiris, who developed the Telemedicine Satisfaction and Usefulness Questionnaire (TSUQ) and Telemedicine Patient Questionnaire (TPQ), respectively. The cultural adaptation of both questionnaires was carried out by taking into account the internationally accepted translation procedures in order to carry out the language translation stages [22, 23] . During the advanced translation phase, the original English was translated into Turkish by four academicians, native speakers of TSUQ and TPQ, who have a good command of English. Expert committee members in the field identified and recorded linguistic and cultural problems. In the second phase, the expert committee members created correction notes of TSUQ and TPQ for translations and discussed. The English translations of both questionnaires were synthesized and finalized. In the third stage, both questionnaires were independently translated into Turkish by two native English translators. Original English versions of both questionnaires were rejected and compared with the English versions in the fourth step, regardless of whether the final Turkish versions were conceptually and linguistically correct. In the last stage, draft versions of both questionnaires were created for use in pilot testing. At the last stage, a pilot study was conducted to question whether the Turkish versions of TSUQ and TPQ are suitable for understandability. A pretest was conducted on a 5-point Likert-type scale with 30 Turkish-speaking elderly individuals. Understandability was excellent in the pilot study. Therefore, no additional changes have been made. The latest Turkish versions of TSUQ and TPQ have been created. The sample size was determined using statistical suggestions and power analysis. Fayers and Machin proposed that in cross-cultural adaptation experiments, the sample size should be at least 100 and, preferably, seven times the number of questionnaires [24] . TSUQ and TPQ have 21 and 17 items, respectively. The psychometric study required a total of 147 participants due to the outnumbered TSUQ items. The minimum sample for the reproducibility was calculated by G* power 3 software (Heinrich-Heine-Universität, Düsseldorf, Germany) with an effect size of 0.40, a probability of error (α) = 0.05, and the power (1-ß) of 0.80. Thirtyfour patients were required for the reproducibility. As a result, 149 MS patients were tested in the first round, and 41 patients were refilled the TSUQ and TPQ, 1 week later for retest reliability [25] . The study was conducted as a prospective cross-sectional design by "X" University Neurology Department on individuals with MS. Inclusion criteria of the study: Individuals with MS who are older than 18 years old and have been monitored with telemedicine for at least 4 years, individuals who have not had an MS attack in the last month, individuals with MS who can read and understand Turkish. The study's exclusion criteria are individuals who have had an MS attack the last month. Telemedicine services (e.g., medication modifications, monitoring of symptoms, exercise management, follow-up of relapses) conducted to the patients were carried out through several operators or through web-based applications, phone calls, messaging, teleconference, within the needs and possibilities of the patients. Socio-demographic and physical characteristics of individuals with MS were recorded. Expanded Disability Status Scale (EDSS) of the patients was scored by the neurologist [26] . In the first evaluation, 149 individuals with MS completed by thoroughly answering all the items of the Turkish versions of TSUQ, TPQ, and Beck Depression Scale (BDS) [20, 21, 27] . Due to the psychological criterion of pleasure was questioned, convergent validity was investigated by comparing it to BDS. In addition, since the expectations and benefits of the patients from the telemedicine service were also questioned, it was thought that it would have positive effects on their psychological state. A retest evaluation was carried out for individuals with MS 1 week later. In the retest evaluation, 41 individuals with MS answered the Turkish versions of TSUQ and TPQ. TSUQ was developed by Bakken et al. in 2006 . It consists of 21 items. With this questionnaire, the satisfaction levels and usability of individuals using telemedicine service (from the software or system from which they receive treatment or other therapy services) are evaluated. A 5-point Likert scale is used in TSUQ evaluation (1 = strongly disagree, 5 = strongly agree). The total score ranges from 21 to 105 [20] . TPQ was developed by Demiris et al. It consists of 17 items in total. A 5-point Likert-type scale is used for TPQ evaluation. It evaluates individuals' telemedicine service, expectation, satisfaction, and usability. Items 2, 3, 7, 15, and 17 are scored by reversing. The total score is scored between 17 and 85 [21] . The Turkish version of the scale was proved for validity and reliability. BDS evaluates depression severity. BDS consists of 21 items. Each item is pointed between 0 and 3. It is assessed over the sum score. A higher score shows high depression. The highest score is 63 [27] . SPSS for Windows v25.0 was used for statistic interpretations in all of the analyses (SPSS Inc, Chicago, IL, USA). The quantitative data was assigned with a mean and a standard deviation. For the qualitative results, percentages were used. To detect whether the data in the sample had a homogeneous distribution, the Shapiro-Wilk test was used. The correlation coefficients were agreed with 95% confidence intervals. The Cronbach alpha coefficient was calculated to determine the TSUQ and TPQ's internal consistency. TSUQ and TPQ, which have a higher alpha coefficient, suggest more consistent results. An alpha of ≥0.6 was deemed acceptable; ≥0.8 was ranked excellent consistency [28] . The two-way random-effect model single-measure reliability study of Shrout and Fleiss was chosen. Reproducibility was measured using the intraclass correlation coefficient (ICC, 95% confidence interval). ICC was calculated for the total score and items of the TSUQ and TPQ. An ICC ≥0.8 shows excellent test-retest reliability [29] . The construct validity of TSUQ and TPQ was tested by Pearson's correlation coefficient (r). The total of the TSUQ, TPQ, and BDS was compared with each other. A high correlation coefficient was presumed in terms of convergent validity. In addition, low correlation was considered divergent validity. If the coefficient was greater than 0.5, it was considered strong; if it was between 0.5 and 0.35, it was considered medium; and if it was less than 0.35, it was considered low [30] . A total of 149 patients (103 women, 46 men) with a mean age of 40.9±10.9 years were recruited in the study. MS duration of the patients was 9.15±6.24 years. Table 1 shows the absolute values of the patients. TSUQ and TPQ were excellently understandable in linguistic and cultural concepts in a pilot study conducted with 30 Turkish-speaking individuals. As a result, after the pretest, no additional changes were required ( Table 2) . The study's reliability findings are summarized in Tables 3 and 4 . Cronbach's alpha coefficient was determined for the TSUQ and TPQ items and overall ratings. Internal consistency of all items and the total score of the TSUQ were excellent (>0.80; ranged 0.971-0.974). On the other hand, the internal consistency of all items and total score of the TPQ was excellent, either (>0.80; ranged 0.878-0.890). The ICC of the TSUQ total score was excellent (>0.80). The ICC score of items for TSUQ was ranged from 0.581 to 0.889. The reproducibility of the total score for the TSUQ was excellent (>0.80). The test-retest reliability of the items of the TPQ was ranged from 0.214 to 0.873. The validity results are presented in Table 5 . The correlation between TSUQ and TPQ was strong (r=0.734, p<0.01). In addition, there was moderate relationship between the TSUQ and the total score of the BDS (r=−0.363, p<0.01). On the other hand, there was low correlation between TPQ and BDS (r=−0.217, p<0.05). This research aimed to demonstrate the TSUQ's and TPQ's translation, cross-cultural adaptation, reliability, and validity in MS patients who were monitored with telemedicine by the neurology department for 4 years. As a result, the Turkish versions of the TSUQ and TPQ have been proven reliable and valid patient-reported outcome measures for testing Turkish-speaking, multiple sclerosis cases. To adapt the questionnaires to Turkish patients, a scientific, textual, and cultural adaptation procedure was followed [22, 23] . Psychometric properties of the TSUQ and TPQ were also successfully presented. Since there are no Turkish screening methods for telehealth providers, both questionnaires are a one-of-a-kind assessment tool for Turkish-speaking patients. It is evident that the importance of telemedicine applications during the COVID-19 era further increased [31] . The importance of telemedicine in terms of providing care and increasing accessibility in patients with MS during the COVID-19 period has been emphasized in a recent study [32] . In addition, it has been stated that telemedicine has vital importance in pandemic situations due to possible limited staff, limited outpatient working hours, and increasing outpatient needs [33] . In this study, psychometric properties of TSUQ and TPQ were analyzed in individuals who had been undergoing a telerehabilitation program for 4 years, but more intensively for the last 6 months, by the neurology outpatient clinic. In this way, it is proposed to provide an analysis to use these patients reported outcome measures in future studies safely. TSUQ is one of the most popular questionnaires frequently used in telemedicine studies. English and Spanish versions of the TSUQ are available [20] . On the other hand, TPQ is available only in its original English version [21] . The Turkish version of both questionnaires could greatly lead to clinical practice and other case-control or intervention trials, given the dramatic rise in telemedicine and telerehabilitation in recent years and the role of remote health services in the COVID-19 process [34] [35] [36] . The consistency of the telemedicine program and level of satisfaction could be compiled and tracked in a more detailed and systematic way using the quantitative data derived from patient reviews. Both questionnaires found to be practical, easy to use, and thus can be completed in a limited period of time. The phrase "telemedicine program" was changed as "method" in the TSUQ and TPQ translation adaptation processes. The translation committee anticipated this technological revolution, speculating that telemedicine programs would no longer be restricted to the "system." Furthermore, the term "clinician" was favored over "healthcare professional" in TPQ. The translation committee introduced this modification because the "clinician" identified a possible telemedicine provider in all health sciences to improve conformity with the questionnaire's interprofessional mode. The TSUQ and TPQ were found to be appropriate in terms of comprehensibility in a pilot analysis. As a result, no other changes to the pretest were needed. The internal consistency for the total score and item scores of the TSUQ and TPQ was [21] . Their calculation results were conformed to our study (α> 0.80). Cronbach's alpha scores were not presented in the TSUQ development study [20] . The test-retest reliability for the total scores of the TSUQ and TPQ was excellent (ICC>0.80). In addition, all items (except item 2) of the TSUQ have moderate to excellent reproducibility (>0.60). The ICC value of only the second items of TSUQ was less than 0.6. In this article, the answer to the question given to the sentence "My health is better than it was before I used the technology" is sought. In the answer given by the patients to this question, we emphasized the possibility of not understanding telemedicine service with the word technology. In addition, the ICC score in items 1, 2, 5, and 7 in TPQ was below 0.60. However, despite this, having the ICC score of the total score above 0.80 is sufficient for the questionnaire's reproducibility. The ICC was not mentioned in the development work of both questionnaires. Our study is also unique in this respect. Construct validity was analyzed by comparing the TSUQ, TPQ, and BDS scores. The correlation between TSUQ and TPQ was strong (r=0.734, p<0.01). In addition, there was moderate relationship between the TSUQ and the total score of the BDS (r=−0.363, p<0.01). On the other hand, there was low relationship between the TPQ and the total score of the BDS (r=−0.217, p<0.05). In particular, the authors used BDS mainly within the scope of divergent validity. In other words, we were predicting that there would be a low to moderate correlation of BDS with TSUQ and TPQ. According to the validation results of our study, both questionnaires were found to be valid. It is also necessary to note the study's limitations. The current research did not look at responsiveness analysis. Furthermore, the "System Usability Scale" was not used to determine validity [37] . The construct validity could be further clarified using a 10-item System Usability Questionnaire. However, it could be time consuming to ask the participants more questions. It can also obstruct the processing of specific data. Furthermore, in both surveys, testing and validating more psychometric properties in various case types would be more helpful to clinical practice. Ultimately, the Turkish versions of the TSUQ and TPQ are determined to be reliable and valid for measuring usability and satisfaction in MS cases. Both questionnaires are accurate patient-reported outcome measures for patients with multiple sclerosis. Both questionnaires also would contribute significantly to both science and clinical practice, given that telemedicine is a trendy era due to the rapid development in healthcare technologies. Data availability All data generated or analyzed during this study are included in this published article. Ethical approval The permission for the translation for the Turkish version of the Telemedicine Satisfaction and Usefulness Questionnaire (TSUQ) and Telemedicine Patient Questionnaire (TPQ) was acquired from the developers of the original questionnaires. The study was carried out in accordance with the ethical principles and the Helsinki Declaration. Informed consent of the patients was obtained. The study protocol was approved by the ethics committee of Ege University (No: 21-3T/29). Informed consent Informed consent of the patients was obtained. The authors declare no competing interests. Emerging subspecialties in neurology: telestroke and teleneurology Effectiveness of telerehabilitation interventions in persons with multiple sclerosis: a systematic review Evidence brief: video telehealth for primary care and mental health services Neuro-telehealth for fragile patients in a tertiary referral neurological institute during the COVID-19 pandemic in Milan, Lombardy Access to health care for people with multiple sclerosis Consensus opinion of US neurologists on practice patterns in RIS, CIS, and RRMS: evolution of treatment practices Comparison of telemedicine versus in-person visits for persons with multiple sclerosis: a randomized crossover study of feasibility, cost, and satisfaction Telemedicine in neurology: telemedicine work group of the American Academy of Neurology update Telerehabilitation benefits patients with multiple sclerosis in an urban setting The evaluation of an online mindfulness program for people with multiple sclerosis: study protocol An initiative to improve care in the community: multiple sclerosis at home access (MAHA) Feasibility study for remote assessment of cognitive function in multiple sclerosis Response heterogeneity to home-based restorative cognitive rehabilitation in multiple sclerosis: an exploratory study Telerehabilitation for persons with multiple sclerosis. Cochrane Database Syst Rev CD010508 The Italian technical/administrative recommendations for telemedicine in clinical neurophysiology The most used questionnaires for evaluating telemedicine services A systematic review of the methodologies used to evaluate telemedicine service initiatives in hospital facilities Usability in telemedicine systems-a literature survey The Model for Assessment of Telemedicine (MAST): a scoping review of empirical studies Development, validation, and use of English and Spanish versions of the telemedicine satisfaction and usefulness questionnaire A questionnaire for the assessment of patients' impressions of the risks and benefits of home telecare Cross-cultural adaptation of health-related quality of life measures: literature review and proposed guidelines Guidelines for the process of cross-cultural adaptation of self-report measures Sample size: how many patients are necessary? Statistical power analyses using G* Power 3.1: tests for correlation and regression analyses Rating neurologic impairment in multiple sclerosis: an expanded disability status scale (EDSS) Beck Depression Inventory II: evaluation of the psychometric properties and cut-off points in a Turkish adult population Evaluating the fit of structural equation models: tests of significance and descriptive goodness-of-fit measures Confidence limits for intraclass reliability coefficients How to develop and validate a new quality of life instrument: quality of life and pharmacoeconomics in clinical trials Telemedicine in the era of COVID-19 Telemedicine during the Coronavirus Disease (COVID-19) Pandemic: a multiple sclerosis (MS) outpatients service perspective Implementation of telemedicine in a tertiary hospital-based ambulatory practice in detroit during the COVID-19 pandemic: observational study Virtual ophthalmology: telemedicine in a COVID-19 era Telemedicine technologies for confronting COVID-19 pandemic: a review How about actively using telemedicine during the COVID-19 pandemic SUS-A quick and dirty usability scale