key: cord-0735070-ketvdk7l
authors: Ghadrdoost, Behshid; Sadeghipour, Parham; Amin, Ahmad; Bakhshandeh, Hooman; Noohi, Feridoun; Maleki, Majid; Peighambari, Mohammad Mehdi; Haghjoo, Majid; Alemzade-Ansari, Mohammad-Javad; Sadeghvashveshady, Mohammad; Mohebbi, Bahram
title: Validity and reliability of a virtual education satisfaction questionnaire from the perspective of cardiology residents during the COVID-19 pandemic
date: 2021-08-31
journal: J Educ Health Promot
DOI: 10.4103/jehp.jehp_32_21
sha: 030322debb4bd4c76b50a8778245d0282a7d5aa3
doc_id: 735070
cord_uid: ketvdk7l

BACKGROUND: The coronavirus disease 2019 (COVID-19) pandemic has prompted the further virtualization of medical education. The satisfaction level of specific users such as cardiology residents with virtual education can augment its quality; hence, the significance of a valid and reliable questionnaire to obtain feedback is needed. This study aimed to design and measure validity and reliability of a satisfaction questionnaire for virtual education of cardiology residents during COVID-19 pandemic. MATERIALS AND METHODS: In this cross-sectional study, a self-administered questionnaire was developed by the faculty members of Rajaie Cardiovascular Medical and Research Center. Reliability was tested utilizing Cronbach's alpha and intercorrelation which was tested using Pearson's correlation coefficient test (ICC). Factor analysis was done by the Kaiser–Meyer–Olkin measure of sampling adequacy and Bartlett's sphericity test. The statistical analyses were performed with the SPSS software version 22. RESULTS: The face validity index was determined via an assessment of the relevance, clarity, and simplicity of each item, and values >0.79 were accepted. The total Cronbach's alpha coefficient was calculated 0.93. Concerning test–retest reliability, the correlation between two rounds of evaluation was >80 (P > 0.001) and ICC was 0.99 (P = 0.001). The content validity evaluation yielded an index of 0.95 and a ratio of 0.91. The principal component factor analysis, conducted to investigate construct validity, generated four domains. CONCLUSIONS: The study results confirmed the validity and reliability of the designed questionnaire to evaluate the level of satisfaction of cardiology residents with virtual learning in COVID-19 pandemic.

T he coronavirus disease 2019 pandemic has significantly impacted educational systems the world over. Medical training with medical educationists now, faced with the unenviable task of altering hospital-based education given the contagious circumstances where clinical mentors and residents work. [1, 2] Virtual learning has proven itself to be the solution of choice during the current pandemic; nonetheless, from the perspective of medical training, reservations have been raised vis-à-vis familiarity with technology, face-to-face interactions, and control. [3] Indeed, unless virtual learning takes heed of the attitude of its users, it cannot be evolved. Specialized medical educational programs must always reflect the ever-changing and ever-increasing demands of society, at large, and the medical community, in particular. [4] Indubitably, fulfilling such needs in the COVID-19 era, in which the hitherto practiced methods of medical training-for instance, as regards angiography and echocardiography-have been rendered well-nigh unfeasible, requires the optimal use of virtual learning. [4] Beside several benefits that virtual learning offers to educational system, some detriment points make immediate concerns among medical students, residents, and their attends such as impossibility of face-to-face interaction between the professor and the student, less control of the professors over the learners, and unfamiliarity of some professors with the online education system. [3] The development of virtual learning will not be successful without considering the attitude of users toward this system. Success in virtual learning in medical fields depends, to a large extent, on the satisfaction of residents and professors and their willingness to use it. The quality of specialized medical education programs has not changed in line with the changing needs of society, changing the face of disease, and changing people's expectations of the medical community and healthcare systems. [4] The idea that virtual education was a new learning domain is crucial. It means that virtual education is a domain in its own right, and it can be combined in a course with both face-to-face and distance learning. [5] Motivation and learning behavior are two very important factors in determining students' learning achievement. Perceived usefulness, management support, self-efficacy, technical support, and training are some factors of learning satisfaction. [6] Aspects of learner motivation and learning behavior constitute a key factor in the achievement of competency standards in educational process. Final process of motivation is completing an action that can provide satisfaction. [7] Several studies show that both intrinsic and extrinsic motivation serves to reinforce attitudes toward behavior, or motivation can make a connection between attitudes and behavior. [8, 9] Collected data from a valid and reliable questionnaire could improve quality of virtual education via promoting residents' motivation.

Research has shown that a systematic perspective toward receiving feedback from medical residents can promote higher quality standards of specialized education. [5] Given the exigencies of the COVID-19 pandemic, it seems advisable that a valid and reliable questionnaire be devised to obtain medical residents' feedback on virtual education. To that end, we endeavored to develop and validate a questionnaire on the level of satisfaction of cardiology residents with virtual learning in the COVID-19 pandemic.

This cross-sectional study evaluated the validity and reliability of a satisfaction questionnaire for cardiology residents regarding virtual education during the COVID-19 pandemic in a tertiary teaching hospital in the Iranian capital, Tehran, in 2020.

A self-administered questionnaire was devised by the faculty members of Rajaie Cardiovascular Medical and Research Center on the basis of scientific resources and expert opinion. The instrument consisted of three parts. The first part comprised 23 questions about various virtual training-related factors such as the quality and content of virtual education, adequacy of teaching hours, interactions between professors and residents, interactions between residents themselves, internet facilities, willingness to learn practical skills such as angiography and echocardiography, and eagerness to conduct research. This part of the questionnaire evaluated the above areas through the format of a five-point Likert scale encompassing "strongly disagree," "disagree," "neutral," "agree," and "strongly agree" to allow respondents to express how much they agreed or disagreed with a particular statement. Thus, the scores obtained in this section ranged between 23 and 115. The second part consisted of four questions each demanding a choice between two options: "face-to-face learning" and "virtual learning." The third part comprised two visual scales to enable respondents to specify their level of satisfaction with both "face-to-face learning" and "virtual learning" on a continuum from zero, denoting minimum satisfaction, to 10, indicating maximum satisfaction.

Content validity was determined using qualitative and quantitative methods. Qualitative content validity was assessed by 14 cardiologists, who commented on the questionnaire's grammar, sentence structure, and placement of phrases in the appropriate place, while quantitative content validity was evaluated using the coefficients of the content validity ratio (CVR) and the content validity index (CVI). For CVR evaluation, the expert group was asked to rate each question on a three-part "essential" scale: "essential;" "useful, but not essential;" and "not necessary." The acceptable CVR, which depends on the number of specialists commenting on the instrument, was considered 0.49 in the present study. [6, 7] Construct validity was evaluated using principal component factor analysis to determine the domains of the designed questionnaire in the form of multiple-choice questions. For this purpose, a factor analysis using a varimax rotation was conducted, and the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy was employed. Interpretation optimization was ensured through the use of a Promax rotation.

External reliability was assessed by five cardiology fellows in different cardiovascular fields. The fellows measured the reproducibility of the answers to the questions at 2 weeks' intervals via the test-retest method; then, the correlations between the results were calculated. Correlations exceeding 0.7 were considered to be reliable features.

Internal consistency varies from zero to 1 which was assessed using Cronbach's alpha. Scales with internal consistency coefficients >0.7 were regarded as acceptable results.

The study was approved by the Ethics Committee of Rajaie Cardiovascular Medical and Research Center (Ethical Code Number: IR.RHC.REC.1399.100). The purpose of the study was informed to participants and written informed consent was obtained. Confidentiality was assured by informing that the information recorded was used for research purposes only and that no personal details would be recorded or produced on any documentation related to the study.

Reliability was tested for internal consistency utilizing Cronbach's alpha, and inter-item correlation was tested using Pearson's correlation coefficient. In addition, the inter-rater consistency of the raters was determined by applying the intraclass correlation coefficient (ICC), the benchmark for which was as follows: >0.75: excellent; between 0.40 and 0.75: moderate; and <0.40: poor. [8] Internal consistency as regards the interscale correlation was examined using Cronbach's alpha. Cronbach's alpha coefficient for the entire questionnaire was considered to be 0.78 (0.73-0.77). Descriptive analysis was conducted to describe the data. For the purposes of factor analysis to investigate construct validity, two tests were applied: the KMO measure of sampling adequacy and Bartlett's sphericity test. The former is a statistical measure that denotes the proportion of variance among variables likely generated by underlying factors, with values of close to 1 generally taken to indicate the possibility of the usefulness of a factor analysis with the data and values less than 0.50 taken to indicate the inadequacy of the factor analysis results. The latter compares the observed correlation matrix with the identity matrix.

Overall, values >0.50 in the KMO measure of sampling adequacy and P < 0.05 in Bartlett's sphericity test are considered acceptable. [9, 10] The significance level was set at a P < 0.05. The statistical analyses were performed with the SPSS software, version 22, for Windows (SPSS Inc., Chicago, Illinois, U.S.A).

As a measure of internal consistency, for the first part of the questionnaire (23 questions), Cronbach's alpha coefficient for each question ranged from 0.75 to 0.91 and the total Cronbach's alpha coefficient was 0.93, indicating the goodness of the overall reliability of the instrument. Furthermore, Cronbach's alpha coefficient for reliability was 0.95, denoting the stability of the whole questionnaire. Apropos the test-retest reliability, the correlation between the two evaluation rounds was more than 80 (r > 0.80 and P > 0.001) and ICC was 0.99 (P = 0.001), which demonstrated relatively good stability for the questionnaire.

With respect to the content validity of the questionnaire, the results showed a CVI of 0.95 and a CVR of 0.91. For face validity, the relevance, clarity, and simplicity of each item were evaluated, and values >0.79 were accepted.

The appropriateness of the instrument for evaluating the intended purpose, the difficulty level of the instrument for the target group (cardiology residents), and the adequacy of the questionnaire construction, which constitute the parameters that show the content validity of a questionnaire, were given high scores by the respondents. The cardiology residents in the present study had no difficulty understanding the questions; it can, therefore, be concluded that the content validity of this questionnaire was also high.

Principal component factor analysis for investigating construct validity was carried out to determine the domains of the designed questionnaire in the form of multiple-choice questions. Kaiser's criterion (eigenvalue = 1.0), considered to determine the main factors of the questionnaire, found four domains. The values of Bartlett's sphericity test were statistically significant (χ 2 = 2393.63, df = 435; P < 0.001). The KMO measure of sampling adequacy yielded a value of 0.921, indicating that the degree of common variance among the 23 items was acceptable. The results of the promax rotation, performed to optimize interpretation, are presented in Table 1 . The first domain comprised 10 items on the quality and content of virtual education, the second domain consisted of eight items on hours/days in the week allocated to virtual education and also the facilities required for e-learning, the third domain contained three questions on practical learning for cardiology residents, and the fourth domain comprised two items on willingness to do research activity.

This study aimed to design a valid and reliable questionnaire to evaluate the level of satisfaction of cardiology residents with virtual education in the COVID-19 pandemic.

A viable management strategy as regards satisfaction creation in a workforce is to prevent dissatisfaction, [11] which requires the regular measurement of these concepts with the aid of valid and reliable instruments. The fact that the training of medical specialists, as valuable human resources in the healthcare system, is a significant goal pursued in medical education prompted us to design a questionnaire with a view to evaluating satisfaction with virtual education among cardiology residents.

Reliability refers to the degree of the replicability of the results obtained by a measurement instrument. However, its significant contribution to assess validity of a questionnaire notwithstanding, it deemed a sufficient condition for the validity of the questionnaire. [12] The validity of a questionnaire, on the other hand, certifies the ability of its scales to measure the intended concept. [13] The principal component factor analysis revealed four strong, clinically relevant domains. The first domain covered the quality and content of virtual education for cardiology residents, the second domain enquired about hours/days in the week allocated to virtual education and also the facilities required for e-learning, the third domain investigated the practicality of learning for respondents, and the fourth domain determined residents' eagerness to do research activity. Factor loadings were all within the acceptable range (0.61-0.89), transcending the recommendations of other reference studies (at least 0.4 for factor loadings). [14] The results from the current study demonstrated that our questionnaire had good reliability and validity. Cronbach's alpha coefficient was greater than 0.9, indicative of good internal uniformity. In other words, the respondents had almost the same perception of the instrument. Moreover, with respect to the reproducibility and test-retest of our questionnaire, the results indicated not only an appropriate level of reliability but also the reproducibility, stability, and internal coherence of the items. Alternatively stated, if the measuring instrument is repeated in the same conditions, similar results will Eagerness to conduct research in the field of "COVID-19" 0.64 be obtained. Finally, the results also showed that the questionnaire had good validity in this study population.

In summary, five distinctive attributes of virtual education are (1) many-to-many (group communication);

(2) any place (place-independence); (3) any time (time-independence); (4) text-based (enhanced by multiple media); and (5) computer-mediated messaging. [15] However, since face-to-face learning also has many benefits especially for cardiovascular residents who need clinical training, awareness of their willingness and satisfaction in changing methods of training after the end of COVID-19 pandemic and even permanent changes in their educational curriculum as a combined virtual and face-to-face method, considering a suitable instrument developed in the same field in various fellowship fields of cardiology seems important. Consequently, this questionnaire as a reliable and valid instrument can be used to assess satisfaction level of cardiology residents in virtual residents. This will help clinical professionals improve virtual educational plan for better training of cardiology residents.

The study results confirmed the validity and reliability of our Persian questionnaire, designed to evaluate the level of satisfaction of cardiology residents with virtual learning in the COVID-19 pandemic. Validated questionnaires that report medical residents' feedback on virtual education, especially in the period of the coronavirus outbreak, can assist educationists in opting for the most efficient medical training methods. We believe that the questionnaire introduced herein can help health policymakers and planners effectively devise comprehensive, yet easily comprehensible, educational programs.

Transition to virtual learning during the coronavirus disease-2019 crisis in Iran: opportunity or challenge? Disaster Med Public Health Prep

COVID-19 impact and virtual medical education

Nursing teacher's perception of the challenges of clinical education and solutions

Predicting students' satisfaction through service quality in higher education

An evaluation of a patient satisfaction survey: Validity and reliability

Validity and reliability properties of the Persian version of perceived health competence scale among patients with cardiovascular diseases

Testing Psychometrics of Healthcare Empowerment Questionnaires (HCEQ) among Iranian Reproductive Age Women: Persian Version

Reliability of classification systems for subaxial cervical injuries. Evid Based Spine Care

Principal Component Analysis and Factor Analysis: differences and similarities in Nutritional Epidemiology application

Validity, reliability and factor analysis of Persian version of schizophrenia quality of life scale

Reliability and validity of Job Satisfaction Survey questionnaire in military health care workers

Validity and reliability of the research instrument

How to test the validation of a questionnaire/survey in a research

The study of factors involved in job satisfaction of worker who works in Najmie subspecialty hospital in

Psychometric Testing of the Persian Version of the Conditions of Work Effectiveness Questionnaire-II (CWEQ-II-PV)

Factors influencing student satisfaction and perceived learning in online courses

The authors express their gratitude to Sara Mortaz Hejri, MD, PhD, and all other participated in the study. We would also like to express our special gratitude to deputy of research of Rajaie Cardiovascular Medical and Research Center for facilitation implementation of this study with registered number 99074.

Nil.

There are no conflicts of interest.