key: cord-0799710-jsbs6n3e authors: da Cunha de Sá-Caputo, Danúbia; Sonza, Anelise; Bachur, José Alexandre; Bernardo-Filho, Mario title: Development, validation and reliability of a questionnaire to evaluate changes on the level of physical exercises and psychological impact due to COVID-19 pandemic social distancing date: 2020-09-07 journal: Acta Biomed DOI: 10.23750/abm.v91i3.9888 sha: a2106647a5b96d062d9c71ceb6035554011cd18c doc_id: 799710 cord_uid: jsbs6n3e BACKGROUND AND AIM: Physical inactivity is expected to happen during the COVID-19 pandemic through home quarantine measures. The aim of this study was to develop, validate and perform the reliability of the questionnaire “Physical exercise (PE) level before and during social isolation (PEF-COVID19)” to evaluate the level of PE during the social distancing due to the COVID-19 pandemic and to try to identify changes in the daily life of the individuals. METHODS: This transversal study was developed to measure psychometric properties of the questionnaire PEF-COVID19. The survey was divided into 4 sections including subjects’ characterization, social isolation update and physical exercise performed, pain, anxiety and stress before and during COVID-19 pandemic. After the survey construction in Portuguese language (Brazil), the survey was transferred to an online digital platform (Google® forms). The Construct, Clarity and Relevance Validation strategy was judged by a panel of experts and the validity index (VI) were calculated. The reliability was evaluated through the test-retest interrater reliability and measured through the intraclass correlation coefficient (ICC) and Kappa coefficient (KC). RESULTS: Twenty-five experts participated of the survey validity and 34 respondents from the target population participated of the test-retest reliability. The general average measures for VI were all above 0.84 and test-retest ICC and KC were 0.89 and 0.88, respectively. CONCLUSIONS: This survey was considered valid and reliable to be applied to the general population over 18 years-old to investigate the PE practice and psychological aspects during the social distancing due to the COVID-19 pandemic, a public health problem. (www.actabiomedica.it) Self-administered questionnaires or surveys are tools commonly used in science health area for health research (1, 2) . Nevertheless, it is necessary to consider (i) the advantage is related to the practicality and num-ber of persons that can be reached with the application and (ii) the disadvantage is the dependence of the subject interpretation, since a health professional will not conduct the questionnaire in case of any doubt. If a questionnaire is subject dependent, the process to develop this instrument must be careful and consistent, following quality standards and requires verification of its usefulness before implementation (1) . World Health Organization (WHO) classified COVID-19 as a pandemic on March 11th 2020 (World Health Organization, 2020), caused by a respiratory virus called severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), described first time in Wuhan, China, in December 2019 (4) . The first COV-ID-19 case in Brazil was confirmed on February 26th 2020 (5) . As a consequence of the COVID-19 characteristics, fast and well done strategies to reach useful and important information for the health area community is important and required. A survey that can be easily spread in many countries is an alternative to achieve this goal (6, 7) . WHO advices the importance of the social distancing to prevent a rapid spread of the disease in the population at the same time; also, to avoid a collapse in the worldwide health systems with the goal to prevent the death of many people (8, 9) . A consequence of the lockdown is the reduction of the level of physical activity because the individuals reduce their daily activities outside. This condition is not good for heath in general, because it can contribute to sedentary behaviours (4) and in few months this can represent a public health problem. Authors have pointed the importance of physical exercise (PE) practice to prevent overall mortality (10, 11) , as well as cardiovascular disease-related mortality ( Je et al, 2013) , or cancer-relate mortality (12) (13) (14) . Considering that the physical exercise (PE) practice is committed and a fast source of information is required, the rationale of this study was to develop and validate the questionnaire "Physical exercise level before and during social isolation (PEF-COVID19)" to evaluate the level of physical activity during the social distancing due to the COVID-19 pandemic and to try to identify changes in the daily life of the general population over 18 years-old. Moreover, these findings could permit to identify undesirable behaviours of the population, and to program guidance strategies to minimize the bad consequences of the COVID-19 pandemic. This is a transversal methodological study that was carried out in the end of March 2020, and the development of the "Physical exercise level before and during social isolation (PEF-COVID19)" survey, followed three main steps: reliability, validation and feasibility. This project was approved by the Ethics Committee of the Hospital Universitário Pedro Ernesto (HUPE), Universidade do Estado do Rio de Janeiro (UERJ), under protocol number CAAE 30649620.1.0000.5259. For this survey development, a seven-step scale design proposed by Artino et al. (1) was followed: 1) literature review; 2) conduct focus groups; 3) literature synthesis and discussion in the focus groups; 4) items were developed; 5) expert validation were conduct; 6) cognitive interviews with few respondents to check if they understood what was proposed; 7) pilot testing. A literature review on 30 th March 2020 was conducted to identify other validated questionnaires in the topic of interest and, to the best of our knowledge, no validated surveys were found at the time of the search (search strategy on Supplementary material). The searches were carried out in important databanks and the search strategy is presented as a supplementary material. This stimulated PhD doctors started to build the questionnaire after to set the objectives. The goals of the questionnaire were to investigate the physical exercise practice, changes in the exercise habits, presence of pain, anxiety and stress before and during the COVID-19 pandemic social distancing. The experts decided to build a self-administered questionnaire considering the worldwide pandemic. The challenge was to write clear and unambiguous questions using a proper vocabulary to the target population (1) . The inclusion criteria to define the target sample were people from general population over 18 years old. The exclusion criterion was age below as 18 years old. Considering the format, some questions were open ended, but the majority close ended with only one possibility of answer, to have answers easier to administer and analyze. Also, Likert scales were used to measure levels of pain, anxiety and stress. Items should be simple, short, and written in language familiar to the target respondents. The survey was divided into 4 sections, presented on Figure 1 . Table 1 presents the questionnaire characteristics. After concluding the survey construct in Portuguese language (Brazil), the survey was transferred to an online digital platform (Google® forms) that was automatically hosted and a pilot testing was done to find language mistakes and errors in the questions format. The URL link data access is password protected through a unique study ID that ensures confidentially of all self-reported data, secured using a "Cloud" database (15) . In the sequence, data was automatically scaled and scored by the platform and downloaded to Excel sheets, to analyse the results with a statistical software of choice. A variety of methods for analyzing the validation of quantitative data collected exists, but independent of the method used, acceptability of an item or scale and its criterion should be determined in advanced (16) . The validity of an instrument is related to the extent to which the interpretations of a test are warranted (17) . An instrument is valid when its construction and applicability allow the accurate desired measurement (18) . In general, there are some kinds of validity as content validity, appearance validity, clarity validity and validity of construct. Content validity, one of the types of validation used in this investigation, is based on judgment of a panel of experts in a specific area of interest (19, 20) . This means that content validation determines if an instrument effectively exploits certain phenomenon to be investigated. In this study, together with content analysis, the judges with extensive experience on the subject in question that composed the panel of experts scored four sections of the questionnaire with their specific goals to be achieved. The Construct, Clarity and Relevance Validation strategy was also judged by the panel of experts, even though it is considered a subjective and unsophisticated technique, as it provides only judgment on the relevance and adequacy of the items. The scores from Likert scales and grades from the panel of experts compound the validity index (VI) determination (21) . A flowchart of the items selection and validation is presented on Figure 2 . The Judgement and Quantification phase included two evaluations. The first assessment consisted of an expert panel that focused on the relevance and clarity of the questions as well as the significance and completeness of responses. The second evaluation comprised an online discussion with coworkers that focused on the understandability, completeness, plausibility and management of the instrument. The Panel of experts were composed by doctors in science health who work in practice and science with physical exercise. The instrument presented to the panel of experts was divided into 4 sections, according to the objectives of the instrument and for the questions, a Likert scale was used for Content, Construct, Clarity and Relevance Validation. The judges should give a score for each question from 1 to 10 (1-4 = not relevant/ incomplete/unclear/meaningless; 5-7 = partially relevant/ partially well-constructed/ partially clear/ partially relevant; 8-10 = highly relevant/clear/complete/ meaningful) (20, 22) . The reliability of a questionnaire can be considered the ability to reproduce a result consistently across time and space, or from different observers (19) . As measurement error is present in respondents, the reliability can be evaluated using its internal consistency, test-retest reliability (stability test), and inter-rater reliability, respectively (23) . The use of these psychometric properties depends on the kind of questions adopted and the application method (with or without raters). The guideline for reporting reliability and agreement studies (GRRAS) was followed in this study (24) , as suggested by the Equator Network guideline for the study type. Considering the specificities of the current questionnaire, the stability test was applied to measure the instrument reliability. The intraclass correlation coefficient (ICC) has become the preferred index to measure the test-retest reliability, as it reflects both correlation and agreement (25) . Also, Kappa coefficient quantifies the level of agreement of categorical variables and measures inter-or intra-rater reliability or test-retest reliability in epidemiologic and clinical instruments (26) . A convenience sample from the target population based on the inclusion criteria was invited to answer the survey through social media. Fifteen days after the first time, these respondents were kindly asked to answer the survey again, to avoid the memory interference. An interval of 2 to 14 days can be used, however it depends on the attribute being measured (27) . The data from the first and second sets of answers were tabulated in electronic sheets (Excel®, Microsoft, 2010) and the ICC and Kappa was calculated, depending on the nature of the questions if unranked (nominal) or ranked (ordinal). The first part of the feasibility study was made through the panel of experts (25 doctors specialists in the field of physical exercise). For each section of the survey, the doctors answered through a likert scale, the clarity of the questions and they also could give an opinion about how to improve the questions. The second part of feasibility study was made based on the guidelines proposed by ARTINO et al. (1) where cognitive interviews with few respondents were applied to check if they understood what was proposed. Three respondents from the target population were invited to fill the survey and to register the time to finish the questionnaire. The respondents answered 3 questions: 1. "Was the survey clear, regarding its questions?"; 2. "Did you have some doubt during the survey filling?"; 3. "How many minutes did you spend to fill the survey?". A validity index (VI) was calculated after the analysis of a panel of experts, doctors in the health area and human movement. The VI was calculated as the number of experts providing a score of above 8 in the Likert scale (scores were between 1 and 10) divided by the total number of experts. With more than 5 experts, the VI should not be lower than 0.78 (20, 22) . The following cutoff points were considered for VI: ≥0.78: valid; <0.6 medium validity and ≤0.40 low validity (20) . For the reliability, respondents from the target population participated of the test-retest measurement. The agreement between the answers was meas-ured through the intraclass correlation coefficient (ICC) and Kappa coefficient (KC). The two-way mixed effects model was used considering people effects are random and measures effects are fixed for the ICC. To calculate the ICC and KC values, the IBM SPSS Statistics for Windows, version 20.0 (IBM® Corp., Armonk, N.Y., USA) was used. The cutoff points for ICC analysis was based on the guidelines which state that, the ICC below .40, the reliability is poor; between .40 and .59 is fair; between .60 and .74, the reliability is good; and when it is between .75 and 1.00 is excellent (28) . The agreement between nominal answers was calculated through Kappa coefficient analysis. Researchers (29, 30) have proposed the following as standards for strength of agreement for the KC with slightly different descriptors. According to Brennan and Silman (1992) , if KC is below .20, the agreement is poor; between .21 and .40 is fair; between .41 and .60, the agreement is moderate; between .61 and .80 is good and between .81 and 1.00 is very good (29) . Another proposition from Landis and Koch (30) ≤0=poor, .01-.20=slight, .21-.40=fair, .41-.60=moderate, .61-.80=substantial, and .81-1=almost perfect. The literature review strategy at the time of the survey construction (Supplementary material) did not find any validated survey on the topics of interest (physical exercise, pain, stress and anxiety before and during COVID-19 pandemic). Few manuscripts were related about the topics of interest on physical activity (31) or physical exercise (32) , and psychological impact (32, 33) . These articles were used to provide ideas to the survey questions. The results obtained in each of the phases of the validity and reliability process were reported descriptively. Twenty-five experts participated of the survey validity and 34 respondents answered the same survey fifteen days after the first time for the test-retest reliability. The respondents were Brazilians with mean age (±SD) of 46.6(±13.6) years, body mass 76(±17.7), height 1.63(±0.3), 82% highly educated (with masters or doctorate), 70.1% were married and 61.8% works in the health area (physiotherapy, veterinary medicine, nutrition, holistic therapies, nursing). The diagnosis was made on the basis of a sum of points (score) and the means of the scores for the different sections of the questionnaire. To determine the validity index (VI) the scores provided from Likert scales and grades from the panel of experts, the VI was calculated and the results are presented on Table 2 . The absolute agreement definition for the 4 sections of the instrument are presented on Table 3 . The general average measures ICC test-retest for the questionnaire was 0.99 and Kappa 0.88. Considering the cutoff points, the 4 sections of the instrument were considered reliable. The scores from the feasibility study gave by the panel of experts is presented on Table 2 (Clarity indexes). The second part of the feasibility study, with the target population, showed that for the interviewers the questionnaire was clear, they did not have any doubt about the questions and the mean time to fill the survey was about 4 minutes. This study reported the construction and evaluated the validity, reliability and feasibility of a selfadministered questionnaire to investigate the level of physical exercise and some undesirable conditions related to the stress, pain and anxiety during the social distancing due to the COVID-19 pandemic, a public health problem. Moreover, it is expected that this survey can permit to identify undesirable behaviours of the population, and to provide information to the definition of program guidance strategies to minimize the bad consequences of the COVID-19 pandemic. The COVID-19 disease crisis has drastically changed people's normal routines, since the SARS-CoV-2 virus is still spreading around the world with impact in the global economy and in the population health (34) . Many countries decided to lockdown and other social distancing and this all happened over a very short period of time (35) , including the different regions from Brazil. A sedentary life tends to increase with the social distancing and the established virtual contact to the work and community relationships (36) . Such changes require a flexible adaptation to new circumstances to avoid sedentary behaviour, and mental illness and this goes in agreement with the purpose of the development of the current survey. Therefore, the need to perform physical exercise is an evidence as the need to guarantee the proper safety of individuals, during this period of crisis caused by the current pandemic (37) . All the steps involving the development of this survey were followed according to Artino et al., (2014) . To the best of our knowledge, no validated surveys were found at the time of the instrument construction. Other self-administered questionnaires considering COVID-19 in different topics were also developed, as the desire for parenthood (38) , the impacts on anxiety (39, 40) , the sleep disturbances (41), psychological outcomes (42) , telemedicine (43) and so on, considering the worldwide pandemic. The findings of this study showed that clear and unambiguous questions were created using a proper vocabulary to the target population as the validity index (clarity) for each section of the survey, evaluated by a panel of experts. About the questions format (the majority close ended), other survey also used similar format (44) . Likert scales were used in the PEF-COVID19 to measure levels of pain, anxiety and stress; a study about knowledge, attitude, anxiety and perceived mental healthcare need in Indian population during COVID-19 pandemic also used the same format (40) . This study used the Google® forms digital platform as it is a free and reliable platform (15) , as other studies (40, 43, 45) . Evaluating validity (20, 22) and reliability (19,23) is important to know the accuracy of an instrument and if similar results are reproduced under the same methodological conditions. One of the ways of evaluating the validity is through a panel of experts and the reliability of self-administered questionnaires is through test -retest. High levels of reliability and validity of a health assessment instrument, denote that it is an instrument that can be used at different social, economic and cultural levels in a given population (46) . This fact will allow an excellent identification of the health problems in that population and, consequently, the more adequate elaboration and decision making to the reality. Related to the validity, this instrument was considered valid by the panel of experts. They evaluated content, construct, clarity and relevance. Content relevance, representativeness, and technical quality are required to achieve content validity and this is mainly assessed through the panel of experts (18) . The content validity is considered as the "theoretical analysis" (47) . Boateng et al, 2018 suggest the same validation strategy used in this study and Goni et al., (2020) and Zamanzadeh et al., (2015) used them to develop their questionnaire. Reliability shows how the measurement tool is reproducible and determines its internal consistency (49) . The test-retest reliability with a diagnosis made on the basis of a sum of points (score) for each section of the questionnaire thought ICC and Kappa coefficient, depending on the nature of the question showed how reliable the instrument is. Technically the PEF-COVID19 is considered an instrument ready to be used to the general population and to answer the proposed goals. The strength of this study is, to the best of our knowledge, this is the first survey to evaluate the level of physical exercise and psychological impact as stress and anxiety and pain of the Brazilian population during the COVID-19 outbreak. Moreover, it is expected that with the comparison of some parameters before and during the outbreak, the results can aid in the definition of policies to help the Brazilian population. As a limitation, the answers that are given on the second occasion of the test-retest reliability, sometimes can be influenced by the ratings of the first measurement, threaten the assumption of independence. This survey construction and validation followed a systematic, seven-step design process with the goal to develop a high-quality questionnaire. During a pandemic occurrence, a specific-related questionnaire requires agility and commitment between the involved researchers because the instrument can be applied only during this event. This instrument was considered valid and reliable to be applied to the general population over 18 years-old to investigate the physical exercise practice and psychological aspects during the social distancing due to the COVID-19 pandemic. Funding: This study was financed by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES) -finance code 001 and The Brazilian National Council for Scientific and Technological Development (CNPq). Each author declares that he or she has no commercial associations (e.g. consultancies, stock ownership, equity interest, patent/licensing arrangement etc.) that might pose a conflict of interest in connection with the submitted article Search strategy to build the questionnaire PEF-COVID19 The literature search was made in order to obtain the articles for the topic of interest synthesis. The Searches werecarried out through the electronic index databases (PubMed, SCOPUS, EMBASE) on 3rd March 2020. The data collection used the following keywords/descriptors/MESH in the English, Spanish and/or Portuguese languages according to the databases instructions. The general search string was: ("covid-19" OR "COVID-19 pandemic") AND ("physical exercise" OR "physical activity") AND ("stress and anxiety" OR "mental health" OR "psychological impact") AND ("pain" OR "pain perception") AND ("survey and questionnaire" OR "crosssectional study") were used. The aim of the literature search was to identify validated surveys in the literature to investigate the effects of the physical exercise practice, pain, stress and anxiety before and during the pandemic. The studies that fulfilled the eligibility criteria (Table 1) were considered for the analysis and to serve as a base to build a new questionnaire in case of survey inexistence based on the proposed goal. Os dados deste formulário são confidenciais e a sua proteção e tratamento será assegurada pelos investigadores responsáveis. O questionário demora aproximadamente 5 minutos para ser preenchido e foi elaborado apenas para fins de investigação. A Organização Mundial da Saúde, em março de 2020, passou a considerar a COVID-19 como uma pandemia. Com a finalidade de evitar que a COVID-19 venha alcançar dimensões cada vez mais alarmantes, o confinamento das pessoas em suas residências, passou a ser uma recomendação da OMS e dos dirigentes da maioria dos países. Em consequência, essa condição favorece o comportamento sedentário, e várias consequências indesejáveis, como doenças físicas e comprometimento psicológico. Esse estudo objetiva avaliar efeitos do confinamento no estilo de vida das pessoas, visando contribuir para apresentar sugestões para serem oferecidas as pessoas para minimizar efeitos indesejáveis devido ao confinamento. Este questionário é anônimo, consequentemente, as informações recolhidas no PEF-COVID19 são mantidas de forma estritamente confidencial. Este instrumento foi aprovado pelo Comitê de Ética do Hospital Universitário Pedro Ernesto (HUPE), Universidade do Estado do Rio de Janeiro (UERJ), com protocolo número CAAE 30649620.1.0000.5259. O registo das respostas ao questionário que contém qualquer informação sobre a sua identidade, serão convertidos em um código numérico para que não seja identificado. O código é gerido numa base de dados separada e não há forma de relacionar os códigos dos participantes com as respostas dadas. Ao responder SIM, você concorda em participar anonimamente deste estudo. ( )SIM ( )NÃO Developing questionnaires for educational research: AMEE Guide No Guidelines for developing, translating, and validating a questionnaire in perioperative and pain medicine Statement on the second meeting of the international health regulations (2005) emergency committee regarding the outbreak of novel coronavirus (2019-nCoV) The positive impact of lockdown in Wuhan on containing the COVID-19 outbreak in China How Brazil can hold back COVID-19 Multidisciplinary research priorities for the COVID-19 pandemic: a call for action for mental health science. The Lancet Psychiatry A framework for identifying regional outbreak and spread of COVID-19 from oneminute population-wide surveys A Review of Coronavirus Disease-2019 (COV-ID-19) Systematic review and meta-analysis of reduction in all-cause mortality from walking and cycling and shape of dose response relationship Physical Activity and All-cause Mortality: An Updated Meta-analysis with Different Intensity Categories Association between physical activity and mortality in colorectal cancer : A meta-analysis of prospective cohort studies Physical activity for cancer survivors : meta-analysis of randomised controlled trials Prevention of Low Back Pain A Systematic Review and Meta-analysis Administer and collect medical questionnaires with Google documents: a simple, safe, and free system Ensuring content validity: an illustration of the process Validity and reliability of measurement instruments used in research Best Practices for Developing and Validating Scales for Health, Social, and Behavioral Research: A Primer Propriedades psicométricas na avaliação de instrumentos: avaliação da confiabilidade e da validade Development and evaluation of the content validity, practicability and feasibility of the Innovative dementia-oriented Assessment system for challenging behaviour in residents with dementia Design and Implementation Content Validity Study: Development of an instrument for measuring Patient-Centered Communication Determination and quantification of content validity Guidelines for developing, translating, and validating a questionnaire in perioperative and pain medicine Guidelines for reporting reliability and agreement studies (GRRAS) were proposed Foundations of Clinical Research: Applications to Practice Dependence of Weighted Kappa Coefficients on the Number of Categories Health Measurement Scales: A Practical Guide to Their Development and Use Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology Statistical methods for assessing observer variability in clinical measures The measurement of observer agreement for categorical data Coronavirus disease (COVID-19): The need to maintain regular physical activity while taking precautions Physical exercise as therapy to fight against the mental and physical consequences of COVID-19 quarantine: Special focus in older people People with Suspected COVID-19 Symptoms Were More Likely Depressed and Had Lower Health-Related Quality of Life: The Potential Benefit of Health Literacy Environmental perspective of COVID-19 The Socio-Economic Implications of the Coronavirus and COVID-19 Pandemic: A Review A tale of two pandemics : How will COVID-19 and global trends in physical inactivity and sedentary behavior affect one another? Coronavirus Pandemic (SARS-COV-2): Pre-Exercise Screening Questionnaire (PESQ) for Telepresential Exercise Desire for parenthood at the time of COVID-19 pandemic: an insight into the Italian situation COVID19 pandemic impacts on anxiety of French urologist in training: Outcomes from a national survey Study of knowledge, attitude, anxiety & perceived mental healthcare need in Indian population during COV-ID-19 pandemic Sleep disturbances among medical workers during the outbreak of COVID-2019 A multinational, multicentre study on the psychological outcomes and associated physical symptoms amongst healthcare workers during COVID-19 outbreak Use of self-administered surveys through QR code and same center telemedicine in a walkin clinic in the era of COVID-19 US Public Concerns About the COVID-19 Pandemic From Results of a Survey Given via Social Media Personal Safety during the COVID-19 Pandemic: Realities and Perspectives of Healthcare Workers in Latin America Content validity of the Geriatric Health Assessment Instrument. Einstein (São Paulo) Scale development: ten main limitations and recommendations to improve future research practices Development and validation of knowledge, attitude and practice questionnaire for prevention of respiratory tract infections among Malaysian Hajj pilgrims Reliability and Validity of Survey Instruments to Measure Work-Related Fatigue in the Com relação ao seu estado de saúde atual, possui alguma doença dentro desses grandes grupos?