Article

Translation and Cross-Cultural Adaptation of the System Usability Scale to Brazilian Portuguese

Traducción y adaptación transcultural de la System Usability Scale al portugués de Brasil

Tradução e adaptação transcultural da System Usability Scale para o português do Brasil

10.5294/aqui.2022.22.2.8

Douglas Fabiano Lourenço ¹
Elenice Valentim Carmona ²
Maria Helena Baena de Moraes Lopes ³

¹ 0000-0001-8305-8950 Nursing School, Unicamp, Brasil, d210044@dac.unicamp.br

² 0000-0001-9976-3603 Nursing School, Unicamp, Brasil carmona@unicamp.br

³ 0000-0001-7747-1140 Nursing School, Unicamp, Brasil, mhbaena@unicamp.br

Received: 12/05/2021
Sent to peers: 17/01/2022
Approved by peers: 19/03/2022
Accepted: 22/03/2022

Theme: Promotion and prevention.

Contribution to the discipline: It is expected that the translation and cultural adaptation of the System Usability Scale will contribute to teaching and researching health sciences, leading to new knowledge, and make available tools widely used in the healthcare field to improve the measurement of the usability of systems, software, and applications. Furthermore, it is essential to further research in this field, aiming to translate usability scales into other languages.

Para citar este artículo / To reference this article / Para citar este artigo: Lourenço DF, Carmona EV, Lopes MHBM. Translation and Cross-Cultural Adaptation of the System Usability Scale to Brazilian Portuguese. Aquichan. 2022;22(2):e2228. DOI: https://doi.org/10.5294/aqui.2022.22.2.8

Abstract

Objective: To adapt culturally the System Usability Scale to Brazilian Portuguese and assess its internal consistency and structural construct validity.
Materials and methods: This methodological study of a measurement instrument's cultural adaptation included the initial translation, the translations' synthesis, the back-translation, the evaluation by a committee of eight experts, testinh with a sample of 100 students, who evaluated the usability of the WhatsApp application, and the evaluation of structural construct validity by exploratory factor analysis.
Results: The judges validated the Brazilian version of the System Usability Scale in the second round. It was revealed that, following the structural construct validation, the version had a unidimensional structure and an acceptable level of reliability (Cronbach's alpha of 0.76). Besides, in the usability test, no suggestions for change were made.
Conclusions: It is worth noting that the Brazilian version of this scale was semantically, idiomatically, conceptually, and culturally equivalent to the original English version and showed adequate reliability and structural construct validity.

Keywords (Source DeCS): Translating; mobile applications; software; data accuracy; evaluation of research programs and tools.

Resumen

Objetivo: realizar la adaptación cultural de la System Usability Scale al idioma portugués, variante brasileña, y evaluar su consistencia interna y su validez de constructo estructural.
Materiales y métodos: se trata de un estudio metodológico de adaptación cultural de un instrumento de medición, que contempló traducción inicial, síntesis de traducciones, retrotraducción, evaluación por ocho expertos, prueba con una muestra de 100 estudiantes, quienes evaluaron la usabilidad de la aplicación WhatsApp, y evaluación de la validez de constructo estructural mediante análisis factorial exploratorio.
Resultados: los expertos en la segunda ronda validaron la versión brasileña de la System Usability Scale. Se reveló que, luego de la validación de constructo, la versión presentó una estructura unidimensional y un nivel aceptable de confiabilidad (alfa de Cronbach de 0,76). Además, en la prueba de usabilidad, no hubo sugerencia de cambio.
Conclusiones: se informa que la versión brasileña de la escala presentó equivalencia semántica, idiomática, conceptual y cultural con la versión original en inglés, así como una confiabilidad y validez de constructo estructural adecuadas.

Palabras clave (Fuente DeCS): Traducción; aplicaciones móviles; programas informáticos; exactitud de los datos; evaluación de programas e instrumentos de investigación.

Resumo

Objetivo: realizar a adaptação cultural da System Usability Scale para a língua portuguesa do Brasil e avaliar a sua consistência interna e a validade estrutural de constructo.
Materiais e método: trata-se de um estudo metodológico, de adaptação cultural de instrumento de medida, que contemplou a tradução inicial, a síntese das traduções, a retrotradução, a avaliação por comitê de oito especialistas, o teste com uma amostra de 100 estudantes, que avaliaram a usabilidade do aplicativo WhatsApp, e a avaliação da validade de constructo estrutural por análise fatorial exploratória.
Resultados: validou-se a versão brasileira da System Usability Scale pelos juízes na segunda rodada. Revela-se que, após a validação de constructo estrutural, a versão apresentou estrutura unidimensional e nível aceitável de confiabilidade (alfa de Cronbach de 0,76). Acrescenta-se que, no teste de usabilidade, não houve sugestão de mudança.
Conclusões: informa-se que a versão brasileira dessa escala apresentou equivalência semântica, idiomática, conceitual e cultural com a versão original em inglês, bem como adequada confiabilidade e validade de constructo estrutural.

Palavras-chave (Fonte DeCS): Tradução; aplicativos móveis; software; confiabilidade dos dados; avaliação de programas e instrumentos de pesquisa.

Introduction

It is known that the use of software and applications for mobile devices is growing daily ⁽¹⁾, and one of the fields in which this advancement stands out is healthcare ⁽²⁾. When creating a computer system for use on computers or mobile devices, the developer is expected to evaluate the usability, that is, investigates the ease with which users can perform specific tasks when interacting with such systems. Therefore, it is necessary to use appropriate methods and tools to improve what has already been developed and provide broader access to these resources ⁽³⁾. More than 242 million mobile phone lines currently exist in Brazil ⁽⁴⁾, demonstrating the need to develop solutions based on mobile devices and advancing studies to measure usability, specifically for computer tools ⁽⁵⁾.

Considering the significant amount of software and applications, it is necessary to evaluate their usability by instruments that can be widely used and adapted to the Brazilian Portuguese language and culture. The usability scales employed internationally include the Usability Measurement Inventory software, which measures user perception of usability and compares software versions ⁽²⁾; the Standardized User Experience Percentile Rank Questionnaire, which evaluates usability, trust, appearance, and loyalty ⁽⁶⁾, and the Questionnaire for User Interaction Satisfaction, which considers subjective user satisfaction with specific aspects of the human-computer interface.

In Brazil, the Questionnaire for User Interaction Satisfaction and the System Usability Scale measure software usability; both are used through free translation ⁽⁷⁾, lacking professional translation and cross-cultural validation procedures.

Therefore, it is relevant to translate and cross-culturally validate into Brazilian Portuguese, with adaptation to the Brazilian context, a usability scale with psychometric properties previously evaluated in other contexts and that proves to be valid and reliable, as well as simple and easy to use. We found that a scale with such characteristics exists, used internationally to measure software usability, including the nursing field ⁽⁸⁾: the System Usability Scale ⁽⁹⁾, which has already been translated and validated into European Portuguese ⁽¹⁰⁾. This version has also been used in Brazil.

It should be noted that the System Usability Scale is simple, consisting of only ten objective statements. They provide an overview of the user's usability evaluation and can be applied to several products and services, such as websites, mobile applications, and clinical systems.

Within the usability evaluation methods described in the literature, questionnaires assume significant importance for collecting self-reported qualitative data on users' characteristics, thoughts, feelings, perceptions, behaviors, or actions ⁽¹⁰⁾. This scale has drawn the attention of researchers for its benefits, such as ease of use, evaluation of different tasks within the same interface (system screens), compar­ison of versions of the same system, and comparison of competing implementations ^(8-11).

A growing number of health care researchers, especially in nursing, have been involved in the development of software, systems for the electronic registration of care, and mobile devices for use by professionals, patients, and the public. However, it is crucial to evaluate the usability of these systems and applications, depending on their type or purpose. The literature indicates that usability assessment though the System Usability Scale questionnaire is a valuable strategy to verify the interaction between application and system users and used in several fields, including nursing ⁽¹²⁾. It is worth noting that the System Usability Scale can be applied to evaluate usability in any system or application, not only in healthcare.

Thus, this study aims to perform the cultural adaptation of the System Usability Scale for the Brazilian Portuguese language and evaluate its internal consistency and structural construct validity.

Materials and methods

Type of study

This methodological study aims to translate and culturally adapt the System Usability Scale into Brazilian Portuguese.

Population and sample

In the cultural adaptation stage, a committee of specialists ⁽¹³⁾ evaluated the translated version. Therefore, a minimum of five and a maximum of ten people are recommended for content validation ⁽¹⁴⁾. Thus, the committee was composed of eight specialists: three methodologists, one linguist, one IT professional, two translators, and one representative of the target audience (a professional who uses software and applications, in this case, a nurse).

Following the System Usability Scale translation, the structural construct validation and testing of the scale were carried out with students from the Systems Analysis and Development and Nursing courses attending a private university in the state of São Paulo, Brazil; this was a convenience sampling.

For the test, 30 to 40 participants are recommended ⁽¹⁴⁾. Based on literature recommendations ⁽¹⁵⁾, We verified that 100 participants would be necessary for structural construct validity: ten per scale item.

Criteria to select specialists and undergraduates

For selecting specialists, we considered training and qualificationd ^{(13, 16)}. Thus, English language knowledge was assumed, as stated by all the invited specialists, with no need for certification, and reading skills were expected of them. Knowledge of the culture of the country where the scale was built was assumed for the invited translators (private English teachers who had lived for one year or more in England, the native country of the System Usability Scale's author); a minimum of two years' experience in computer science for information technology professionals (analysts and system developers with expertise and currently working in this sector); a minimum of two years' experience in linguistics for linguists (professionals with a degree in linguistics and presently working in this sector) and, for methodologists, the conduct and publication of research about validation. Specialists who failed to carry out the evaluations in full were excluded.

To evaluate the System Usability Scale translated into Brazilian Portuguese, we included Nursing and Systems Analysis students. The evaluation by Nursing students is vital, as the objective is to use this scale to evaluate the usability of systems and applications in the health field, especially nursing. The participation of System Analysis and Development students is relevant since they can identify some critical aspects of the evaluated system that are not contemplated in the System Usability Scale. The students included used WhatsApp on mobile devices daily to send and receive text or audio messages and images or photos. Those who failed to complete any item of the System Usability Scale, even with the primary researcher's support and guidance, were excluded.

System Usability Scale

The scale presents items with sentences that denote positive evaluations and other items with negative evaluations alternately to minimize the possibility of responses that lack reflection by those who apply the System Usability Scale to evaluate the usability of a given system or tool. For each statement present in this scale, a five-point Likert scale is used, where one means "strongly disagree" and five means "strongly agree."

To obtain the final score, the researcher must perform the calculation described as follows: for the odd-numbered statements (1, 3, 5, 7, and 9), the individual score is obtained by subtracting one from the score given by the evaluator. For even-numbered statements (2, 4, 6, 8, and 10), the individual score is obtained by subtracting five from the score given by the evaluator. The obtained values are added, and the result of this sum is multiplied by 2.5, obtaining the total value of the System Usability Scale. Regarding this total value, it is possible to consider the evaluated system as follows: worst imaginable (up to 20.5); poor (21 to 38.5); average (39 to 52.5); good (53 to 73.5); excellent (74 to 85.5); best imaginable (86 to 100) ^{(8, 9)}.

Cross-cultural adaptation process

For a reliable cultural adaptation, the literature recommends the following steps: initial translation, synthesis of translations; back-translation; evaluation by a committee of specialists, and testing ⁽¹³⁾. In the initial translation stage, two translators independently performed the translation, both native to the country to which the scale was translated (Brazil) and proficient in the original scale language (British English), in addition to being knowledgeable of the culture of the country where the instrument was produced (England). Two Portuguese scale versions were obtained: translation 1 (T1) and translation 2 (T2).

Then, the translations were synthesized, producing a single version in Portuguese, the T1-2. This version was forwarded to two native translators . from the scale's country of origin (England) with proficiency in the language and culture of both countries involved in the adaptation process (Brazil and England) for the back-translation process. The objective was to verify if the T1-2 version reflected the content of the original English version. The material produced was referred to as "back-translation 1" (RT1) and "back-translation 2" (RT2).

A committee of eight specialists carried out the evaluation stage. To this end, each of them received the original instrument, the two initial translations (T1 and T2), the synthesis of the translations (T1-2), and the two back-translations (RT1 and RT2). The specialists' committee met twice to reach a consensus on whether semantic, idiomatic, conceptual, and cultural equivalences were present ⁽¹³⁾.

It is known that the pre-test is the final stage of the cultural adaptation process. This stage included a sample of 100 undergraduate students ⁽¹⁴⁾, distributed between the System Analysis and Nursing courses, to evaluate the comprehension of the Portuguese version of the System Usability Scale. From the demographic point of view, the sample consisted of undergraduate students aged 17 to 45 years, with a mean age of 22.68, a variance of 32.02, and a standard deviation of 5.66. Among the participants, 61 undergraduates were Nursing students, and 39 were System Analysis students. Among the undergraduate Nursing students, 50 were male, 11 were female, and among the undergraduate Systems Analysis students, 30 were male, and nine were female. The participants were invited and oriented to use this scale to evaluate the WhatsApp application for mobile devices.

Construct validity and internal consistency of the System Usability Scale — Brazilian version

The structural construct validity was evaluated using the exploratory factor analysis ⁽¹⁵⁾, using two factor extraction methods: principal components and common factors. The common factors method is more suitable when the objective is to identify latent factors in the analyzed variables. In contrast, the primary com­ponents method is more convenient when the aim is to obtain the minimum number of factors necessary to summarize most of the total variance of the variables ⁽¹⁶⁾. By definition, the common factors method may be the most suitable when the objective is the structural construct validity of a measurement instrument. Still, this method may present issues in its application, for instance, factor indeterminacy ⁽¹⁶⁾. Since several studies in the literature apply the two methodologies, the decision made was to apply the two extraction methods to compare the results obtained. For the analysis, we used the Varimax rotation method for the primary components, and the Direct Oblimin rotation for common factors. This choice was defined by analyzing the correlation matrix between the factors. Thus, in the case of the principal component method analysis, correlations lower than 0.32 were observed between factors ⁽¹⁷⁾. In the case of the common factor method analysis, values higher than 0.32 were observed. We employed the Statistical Package for the Social Sciences (SPSS) software, version 23, to implement these analyses.

The Kaiser-Meyer-Olkin (KMO) index was initially calculated to verify the data fitting to the factor analysis, applying Bartlett's Test of Sphericity. In this study, KMO values < 0.50 are considered unacceptable; > 0.50 and < 0.60 are poor, but still acceptable; > 0.60 and < 0.70 are mediocre; > 0.70 and < 0.80 are average; > 0.80 and < 0.90 are good; and > 0.90 are excellent ⁽¹⁸⁾.

To define the number of factors extracted, we observed the factors that presented eigenvalues greater than 1.0 if they represented at least 60 % of the total variance explained. Factor load values greater than or equal to 0.50 were considered satisfactory ⁽¹⁵⁾. The System Usability Scale's internal consistency was evaluated using Cronbach's alpha. An adequate good alpha value ranges from 0.70 to 0.90 ⁽¹⁵⁾.

Ethical aspects

The author of the System Usability Scale was previously contacted, and formal authorization was obtained for the translation and cultural adaptation of the instrument. The Research Ethics Committee og the University of Campinas (UNICAMP) approved the research under the Certificate of Ethical Appraisal no. 95580318.2.0000.5404 and Legal Opinion no. 3.004.013/2018. After the necessary clarifications and orientations, the specialists and undergraduate students agreed to participate and signed the informed consent form, keeping one copy in their possession. The study was conducted between November 2018 and August 2019.

Results

Two rounds of evaluation with the experts were necessary. In the first round, four items required discussion to reach a consensus. Item 1 of the System Usability Scale, "I think that I would like to use this system frequently," translated as "Eu gostaria de usar esse sistema frequentemente," was considered non-equivalent to the original by one of the specialists for lacking the phrase "eu acho que." Item 4, "I think that I would need the support of a technical person to be able to use this system," demanded adjustments, as the Portuguese version included the word "eu" twice, just as the English version, which was considered unnecessary by a specialist. Regarding item 7, "I would imagine that most people learn to use this system very quickly," approaching semantic equivalence and considering the grammar (partitive collective), the consensus among researchers and specialists led to the wording of "Eu imagino que a maioria das pessoas pode aprender a usar esse sistema muito rapidamente." For item 9, "I felt very confident using the system," the final wording was "Eu me senti muito seguro usando o sistema."

Following the adjustments made by the specialists, the modified version was sent to everyone for a second round and approved. This version, considered the pre-final version of the System Usability Scale in Brazilian Portuguese, is presented in Table 1 and was used in the test. At the end of this process, the proposed title for the adapted version was "Escala de Usabilidade de Sistema - versão brasileira" (System Usability Scale - Brazilian version).

Table 1. Pre-final version of the System Usability Scale, translated into Brazilian Portuguese

Source: Own elaboration.

It is reported that no suggestions for changes or comments regarding difficulties in comprehension were made. The Brazilian version of the System Usability Scale presented an acceptable level of reliability, with Cronbach's alpha equal to 0.76.

A KMO value of 0.73 for the instrument and a p-value lower than 0.0001 in Bartlett's Test of Sphericity were obtained for structural construct validity, indicating that the exploratory factor analysis was adequate. The results suggested a structure composed of three domains, which combined would explain 64.64 °% of the variance in the principal component analysis and 51.55 °% in the common factor analysis (Table 2).

Table 2. Values of explained variance obtained using the principal component and common factor extraction methods.
Campinas (SP), Brazil, 2019

Source: Own elaboration using research data (2019).

Table 3 presents the factor loadings of the items in each factor of the principal component analysis after applying the Varimax rotation method. Table 4 presents the factor loadings of the structure matrix and the standard matrix obtained by the common factor method after applying the Direct Oblimin rotation. The results obtained suggest the same distribution of items among the three factors.

Table 3. Factor loadings of the System Usability Scale items - Brazilian version, obtained using the principal component extraction method.
Campinas (SP), Brazil, 2019

^* Varimax rotation method.
^** Factor loading values lower than 0.30 were removed from the table.
Source: Own elaboration using research data (2019).

Table 4. Factor loadings of the System Usability Scale items - Brazilian version, obtained using the common factor extraction method.
Campinas (SP), Brazil, 2019

^* Direct Oblimin rotation method.
^** Factor loading values lower than 0.30 were removed from the table.
Source: Own elaboration using research data (2019).

In Table 5, where the communality values obtained for each of the methods used are presented, it can be noted that some items have low communality values.

Table 5. Communality values of the System Usability Scale items - Brazilian version, using the principal component and common factor extraction methods.
Campinas (SP), Brazil, 2019

Source: Own elaboration using research data (2019).

We verified that the evaluations of the factorial solutions obtained were not satisfactory, either through the indexes obtained or through the domain structure formed, which was not adequate from a practical point of view. Based on this, the decision was to keep the instrument's original version (unidimensional).

The internal consistency was also evaluated through Cronbach's alpha coefficient. This coefficient varies from 0 to 1, where values greater than 0.7 indicate reliability between measures.

Discussion

The translation process carried out with the method recommended by the literature enabled the achievement of a final version entirely understandable by the test participants. This study also confirmed the unidirectionality of the scale and an indication of the well-defined structure as proposed initially.

We believe that this study came across the need for a Brazilian version of the System Usability Scale, as it is a helpful instrument that has been applied in the Brazilian context ^{(3, 8)}, but without a version that had formally undergone the cross-cultural adaptation process. The specialists' committee proved to be consistent with the study's objectives and the context of the System Usability Scale, as recommended by the literature ⁽¹³⁾.

The psychological test adaptation procedure was performed to translate the System Usability Scale into Polish ⁽¹⁸⁾. A Cronbach's alpha value of 0.805 was also identified, with high correlations between the Polish and English versions' items and the total score.

The Malaysian version of the System Usability Scale ⁽¹⁹⁾ was developed from a front-to-back translation of the scale. The final version was reached through consensus and cross-cultural adaptation with ten specialists in mobile application development. The content validity index was found to be 0.91, indicating the relevance of the ten items of the System Usability Scale to evaluate the usability of mobile device applications. Face validity was also analyzed with ten cell phone users, obtaining a value of 0.94, indicating that the scale was easily understood. Reliability was assessed among 54 cell phone users, reaching a Cronbach's alpha value of 0.85 (95 % confidence interval; 0.79-0.91), which denotes a reliable version for its proposed assessment ⁽¹⁹⁾.

On the one hand, the System Usability Scale provides a usability score easily understood by many people: project managers, computer programmers, and application users. It becomes apparent, on the other hand, as its author rightly puts it, that the scores for the items are not meaningful by themselves ⁽⁹⁾. Therefore, the usability evaluation should be understood based on the set of answers to the items of the System Usability Scale. It has been noted; however, that software users and IT professionals tend to ignore this recommendation provided by the scale's author ^(20-22) and often consider, in isolation, information regarding the quality of specific aspects of usability, viewing the evaluation obtained through some items of the scale. This study confirmed the one-dimensionality of the System Usability Scale, which reaffirms the original author's recommendation that its items should not be analyzed separately.

A recent study ⁽²³⁾ has found that the use of mobile devices by nurses is increasing, considering a sample of 24 studies (19 master's theses and five dissertations) that verify the development and use of applications: 16 (66.7 %%) of them with a care focus, six (25 %%) with an educational focus, and two (8.3 %%) with a management focus.

From the automotive sector to space and industrial research, from people with different disabilities to voters, many sectors are interested in conducting usability research ^{(9, 10)}. In nursing care and in light of the swift technological development in healthcare, it is believed that this scale can be used in different contexts.

Conclusions

The System Usability Scale cross-cultural adaptation process into Brazilian Portuguese was conducted according to the steps recommended by the literature. For the adapted version, the title "Escala de Usabilidade de Sistema — versão brasileira" (System Usability Scale — Brazilian version) was adopted. The cultural adaptation process demanded only a few adjustments in some terms concerning the original version. The Brazilian version of the System Usability Scale was user-friendly and easy to understand, confirming its functionality. It also presented construct validity, being equivalent to the original English version.

As a limitation, the test was carried out at a single private university, with students from two courses, and focused only on the WhatsApp application. Studies involving the testing of other systems and other target audiences are suggested to confirm this version's adequacy, considering the country's cultural diversity. Another limitation of the study is that the participants were not randomly selected. Furthermore, as the results suggested a one-dimensional factor structure, it is recommended that the structural construct validity be assessed again using the item response theory technique in a future study.

Therefore, it is believed that the System Usability Scale can be used to evaluate new technological tools, such as computer programs and software for teaching and care in healthcare and nursing.

Conflicts of interest: None declared.

References

1. Odgers C. Smartphones are bad for some teens, not all. Nature. 2018554(7693):432-4. DOI: https://doi.org/10.1038/d41586-018-02109-8

2. Pérez LSV, Tornés AFG, Riverón EMF, Hernández AMS, Galeana NIP. Software quality methodology to train engineers as evaluators of information systems development Tools. Universal J Educ Res. 2018;6(12):2942-51. DOI: https://doi.org/10.13189/ujer.2018.061230

3. Hodrien A. A review of post-study and post-task subjective questionnaires to guide assessment of system usability. J Usability Stud. 2021;16(3):203-32. http://usir.salford.ac.uk/id/eprint/60928/4/Hodrien_JUS_May2021.pdf

4. Agência Nacional de Telecomunicações. Histórico: acessos [internet]. Brasília: Anatel; 2017. https://www.anatel.gov.br\dados-abertos\PDA\historico_acessos\historico_acessos.csv

5. Barra DCC, Paim SMS, Sasso GT, Colla GW. Methods for developing mobile apps in health: na integrative review of the literature. Texto Contexto Enferm. 2017; 26(4):e2260017. DOI:  https://doi.org/10.1590/0104-07072017002260017

6. Reichenpfader D, Glauser R, Dugas M, Denecke K. Assessing and improving the usability of the medical data models portal. Stud Health Technol Inform. 2020;271:199-206. DOI: https://doi.org/10.3233/shti200097

7. Narcizo JCM, Silva LA. Qualidade de software: melhorando a qualidade de serviços computacionais, baseado na percepção do usuário [dissertação de mestrado] [internet]. Anápolis: Centro Universitário de Anápolis; 2018. http://repositorio.aee.edu.br/bitstream/aee/1103/1/TCC2_2018_02_Josimar_Candido_e_Lucas_Andrade.pdf

8. Padrini-Andrade L, Balda RCX, Areco KCN, Bandiera-Paiva P, Nunes MV, Marba STM et al. Evaluation of usability of a neonatal health information system according to the user's perception. Rev Paul Pediatr. 2019;37(1):90-6. DOI: https://doi.org/10.1590/1984-0462/;2019;37;1;00019

9. Teshnizi SH, Alipour J. Usability evaluation of hospital information system: A cross-sectional study. Appl Med Inform. 2020;42(2):118-25. https://ami.info.umfcluj.ro/index.php/AMI/article/view/773

10. Martins AI, Rosa AF, Queirós A, Silva A, Rocha NP. European Portuguese validation of the System Usability Scale (SUS). Procedia Comput Sci. 2015;67:293-300. DOI: https://doi.org/10.1016/j.procs.2015.09.273

11. Dowding D, Merrill JA, Barrón Y, Onorato N, Jonas K, Russell D. Usability evaluation of a Dashboard for Home Care Nurses. Comput Inform Nurs. 2019;37(1):11-9. DOI: https://doi.org/10.1097/cin0000000000000484

12. Lopes JP, Dias TMR, Carvalho DBF, Oliveira JF, Cavalvante RB, Oliveira VC. Evaluation of digital vaccine card in nursing practice in vaccination room. Rev Latino-Am Enfermagem. 2019;27:e3225. DOI: 10.1590/1518-8345.3058.3225

13. Beaton DE, Bombardier C, Guillemin F, Ferraz MB. Recommendations for the cross-cultural adaptation of health status measures. Spine (Phila Pa 1976). 2000; 25(24):3186-91. DOI: https://doi.org/10.1097/00007632-200012150-00014

14. Hair JF, Black WC BJ, Anderson RE. Multivariate data analysis. 9a ed. Hampshire: Cengage Learning; 2019.

15. Lynn MR. Determination and quantification of content validity. Nurs Res. 1986;35(6):382-5.

16. Hair Junior JF, Black WC, Babin BJ, Anderson RE, Tatham RL. Análise multivariada de dados. 6a ed. Porto Alegre: Bookman; 2009.

17. Tabachnick BG, Fidell LS. Using multivariate statistics. 5a ed. Upper Saddle River, NJ: Pearson Allyn & Bacon; 2007.

18. Alexandre NMC, Coluci MZO. Content validity in the development and adaptation processes of measurement instruments. Ciênc Saúde Colet. 2011;16(7):3061-8. DOI: https://doi.org/10.1590/S1413-81232011000800006

19. Marôco J. Análise estatística com o PASW statistics (ex-SPSS). Lisboa: ReportNumber; 2010.

20. Gao M, Kortum P. M. Borkowska Aj Jach K. Multi-language toolkit for the system usability scale. Int J Human-Comput Interact. 2020;36(20):1883-901. https://doi.org/10.1080/10447318.2020.1801173

21. Marzuki MFM, Yaacob NA, Yaacob NM. Translation, cross-cultural adaptation, and validation of the malay version of the system usability scale questionnaire for the assessment of mobile apps. JMIR Hum Factors. 2018;5(2):e10308. DOI: https://doi.org/10.2196/10308

22. Bloom BM, Pott J, Thomas S. Usability of electronic health record systems in UK EDs. Emerg Med J. 2021;38(6):410-5. DOI: https://doi.org/10.5061/dryad.7h44j0zsq

23. Lima CSP, Barbosa SDFF. Aplicativos móveis em saúde: caracterização da produção científica da enfermagem brasileira. Rev Eletr Enferm. 2019;21. DOI: https://doi.org/10.5216/ree.v21.53278

Home