key: cord-0852634-zopy1vlw
authors: Ong, Ardvin Kester S.; Chuenyindee, Thanatorn; Prasetyo, Yogi Tri; Nadlifatin, Reny; Persada, Satria Fadil; Gumasing, Ma. Janice J.; German, Josephine D.; Robas, Kirstien Paola E.; Young, Michael N.; Sittiwatethanasiri, Thaninrat
title: Utilization of Random Forest and Deep Learning Neural Network for Predicting Factors Affecting Perceived Usability of a COVID-19 Contact Tracing Mobile Application in Thailand “ThaiChana”
date: 2022-05-17
journal: Int J Environ Res Public Health
DOI: 10.3390/ijerph19106111
sha: ba3d9bf655faf77db1c8449e1812952c64a3c618
doc_id: 852634
cord_uid: zopy1vlw

The continuous rise of the COVID-19 Omicron cases despite the vaccination program available has been progressing worldwide. To mitigate the COVID-19 contraction, different contact tracing applications have been utilized such as Thai Chana from Thailand. This study aimed to predict factors affecting the perceived usability of Thai Chana by integrating the Protection Motivation Theory and Technology Acceptance Theory considering the System Usability Scale, utilizing deep learning neural network and random forest classifier. A total of 800 respondents were collected through convenience sampling to measure different factors such as understanding COVID-19, perceived severity, perceived vulnerability, perceived ease of use, perceived usefulness, attitude towards using, intention to use, actual system use, and perceived usability. In total, 97.32% of the deep learning neural network showed that understanding COVID-19 presented the most significant factor affecting perceived usability. In addition, random forest classifier produced a 92% accuracy with a 0.00 standard deviation indicating that understanding COVID-19 and perceived vulnerability led to a very high perceived usability while perceived severity and perceived ease of use also led to a high perceived usability. The findings of this study could be considered by the government to promote the usage of contact tracing applications even in other countries. Finally, deep learning neural network and random forest classifier as machine learning algorithms may be utilized for predicting factors affecting human behavior in technology or system acceptance worldwide.

The COVID-19 pandemic has been present for almost 2 years since March 2020. Despite the vaccination programs evident in different countries, the continuous rise of infected 1.

Would the integrated Protection Motivation Theory and Technology Acceptance Model holistically measure perceived usability of a health-related application for COVID-19 contact tracing? 2.

Could a machine learning algorithm solely measure and predict factors affecting human behavior, specifically measuring perceived technology usability? 3.

Are the integrated deep learning neural network and random forest classifier enough to highlight the significant factors affecting perceived usability of a technology? 4.

Could the proposed model and methodology be applied and extended to different studies involving human behavior?

To measure the usability of tracing applications such as Thai Chana, frameworks such as Protection Motivation Theory and Technology Acceptance Model could be utilized. Protection Motivation Theory is a fear and coping appraisal theory that is utilized to measure health-related measures [14] . Technology Acceptance Model on the other hand is utilized for measuring the usage of a product or technology [15] . Both studies of Ong et al. [14] and Prasetyo et al. [15] integrated the respective theories to holistically measure an individual's intention or usage behavior. Protection Motivation Theory solely measures a person's perception of vulnerability and severity [16] . Van Bavel et al. [17] considered Protection Motivation Theory to measure the minimization of risk and exposure to improve online security behavior. Mousavi et al. [18] considered Protection Motivation Theory for privacy protection behavior on social networking sites. Their results showed that privacy assurance played a significant role in people's usage of a system.

Consequently, the Technology Acceptance Model was considered by several studies for actual usage of a system. Tomczyk et al. [19] integrated health behavior changes and the Technology Acceptance Model to measure the predicted adoption intentions of the German contact tracing application. Their study considered hierarchical regression modeling and showed that there was only a marginal increase in the predictive value. Moreover, Velicia-Martin et al. [7] considered Technology Acceptance Model regarding the contact tracing application in the United Kingdom. However, their study utilized partial least square-Structural Equation Modeling. According to Fan et al. [20] , Structural Equation Modeling alone cannot measure the most significant factor due to the causal relationship among the framework considered. The farther the independent variables from the dependent variables may cause low to no significance. In addition, Woody [21] stated how a mediating effect may be present which hinders the importance and significance level of latent variables due to connections present in a framework. Thus, it would be best to consider utilizing machine learning algorithms such as deep learning neural network and random forest classifier to have high accuracy of prediction among factors affecting human behavior [22] .

This study aimed to determine the perceived usability of the COVID-19 contact tracing mobile application in Thailand, Thai Chana. This was achieved through the integration of Technology Acceptance Model and Protection Motivation Theory to measure the perceived usability of Thai Chana as a contact tracing application in Thailand. This was measured using deep learning neural network and random forest classifier to predict factors affecting the perceived usability of Thai Chana. Specifically, factors such as understanding COVID-19, perceived severity, perceived vulnerability, perceived ease of use, perceived usefulness, attitude towards using, intention to use, actual system use, and perceived usability were considered in this study. This is the first study that considered deep learning neural network and random forest classifier for contact tracing applications. The results of this study would be beneficial for contact tracing applications in different countries as a theoretical foundation for new mobile applications for disease control. Lastly, this framework could also be utilized for another application's usability among different technologies and its overall acceptance worldwide.

With the trend of research focusing on human behavior dealing with integrated multivariate tools and machine learning algorithms [22] [23] [24] [25] [26] [27] [28] , this study highlighted how machine learning algorithm alone can accommodate analysis involving technology usability. Little to no studies were found to deal with studies that considered sole machine learning algorithm tools in analyzing human behavior, specifically perceived usability of technology with integrated theories as the framework. In addition, this study is one of the first studies that provided evidence for analyzing factors affecting technology usability using combined random forest classifier and deep learning neural network. The flow of the paper is as follows: (1) Introduction that covers the literature review, gap, and background of the study, (2) related studies and theoretical framework, (3) methodology, (4) results and validation, (5) discussion, and (6) conclusion.

The machine learning algorithm has been widely utilized due to the availability of bulk data nowadays. It is a tool used for predicting, classifying, and recognizing patterns among different datasets. Several studies [22] [23] [24] have utilized machine learning algorithm tools such as decision tree with random forest classifier and neural networks. Milani et al. [25] have utilized random forest classifier in classifying factors affecting professional child removal based on parental factors. Their results presented how random forest classifier could be utilized in analyzing factors affecting human behavior. Moreover, Chen et al. [22] considered random forest classifier for predicting risk evaluation of flood disasters in China. The different studies have proved how random forest classifier compared to the normal decision tree provided better classification accuracy [24] [25] [26] .

On the other hand, neural networks have been utilized to determine pattern recognition. Neural networks have been developed utilizing an algorithm based on how the neurons transfer information to the brain. It is said to be advantageous since it produces state-of-the-art results based on large datasets [26] . Yariyan et al. [27] utilized Artificial Neural Network focusing on risk assessment in Iran. Oktarina et al. [28] considered neural networks in predicting damages and casualties of people in Indonesia. However, simple neural networks such as artificial neural networks have reduced capabilities to predict higher accuracy due to limited processing of factors. Utilizing a lot of factors would consider more complex calculations. To which, deep learning neural network would be beneficial since it considers more hidden layers for further processing and calculation of output [26] . However, the disadvantage of which is the optimization process to determine the best activation function, optimizer, and the number of nodes since artificial neural networks are classified as a black box.

Compared to the traditional statistical analysis and multivariate tools such as Structural Equation Modeling, machine learning algorithm has been said to have several advantages that suffice the limitations of Structural Equation Modeling. Fan et al. [20] explained how the traditional Structural Equation Modeling has limitations due to the indirect effect the framework has considered. This reduces the level of significance, and may even bring non-significant results due to the connections among dependent and independent variables. In addition, Woody [21] explained how the mediating effects brought by partial and full mediation cause different significant and insignificant results. Therefore, Structural Equation Modeling could not relatively predict the most significant factor affecting human behavior. As suggested by several studies, machine learning algorithms such as random forest classifier and deep learning neural network could be utilized to predict highly significant factors affecting human behavior [23] [24] [25] [26] [27] [28] . Thus, this study opted to highlight how machine learning algorithms such as deep learning neural network and random forest classifier could be utilized to evaluate human behavior, specifically factors affecting perceived usability of a technology with integrated frameworks of Protection Motivation Theory and Technology Acceptance Theory.

Presented in Figure 1 is the theoretical framework considered in this study. The integration of Protection Motivation Theory and the Technology Acceptance Model following factors such as understanding COVID-19 (U), perceived severity (PS), perceived vulnerability (PV), perceived ease of use (PEU), perceived usefulness (PU), attitude towards using (A), intention to use (IU), and actual system use (AU) were considered to measure perceived usability (PUS). The basic Technology Acceptance Model factors were considered such as PEU, PU, A, IU, and IU. Protection Motivation Theory is a framework used to measure the threat and coping appraisal influencing the behavioral intention of an individual, dealing with health-related topics [13] . Under Protection Motivation Theory, only those under the threat appraisal factors were considered such as U, PV, and PS. Since "Thai Chana" is utilized for the mitigation of threats brought by the COVID-19 pandemic, the coping appraisal was not included in the framework of this study.

Factors under Protection Motivation Theory when dealing with threats consider different factors such as U, PV, and PS [13] [14] [15] . Li et al. [29] explained how the perception in choosing and using a mobile application for COVID-19 tracing is caused by people's risk perception, readiness to use the system, and socialization. With knowledge regarding health-related negative effects, people would often look for ways to mitigate them [14] . Understanding of COVID-19 in this study considers the perception of what the virus is, how it is transmitted, affects, and health protocols. On the other hand, PV is how susceptible the individual is to contract the virus. PS is the perception of how critical the effect of contracting COVID-19 is. Martins et al. [30] explained that knowing the positive effect of mitigation through the use of technology would to an increase in perceived usability. Mingxing et al. [31] also highlighted that when the perception of risks is increased, there is also an increase in the perceived usability of a technology such as contact tracing applications. Thus, it was hypothesized that: Hypothesis 1 (H1). U has the most significant effect on PUS.

Hypothesis 2 (H2). PS has the most significant effect on PUS. "Thai Chana" is utilized for the mitigation of threats brought by the COVID-19 pandemic, the coping appraisal was not included in the framework of this study. Factors under Protection Motivation Theory when dealing with threats consider different factors such as U, PV, and PS [13] [14] [15] . Li et al. [29] explained how the perception in choosing and using a mobile application for COVID-19 tracing is caused by people's risk perception, readiness to use the system, and socialization. With knowledge regarding health-related negative effects, people would often look for ways to mitigate them [14] . Understanding of COVID-19 in this study considers the perception of what the virus is, how it is transmitted, affects, and health protocols. On the other hand, PV is how susceptible the individual is to contract the virus. PS is the perception of how critical the effect of contracting COVID-19 is. Martins et al. [30] explained that knowing the positive effect of mitigation through the use of technology would to an increase in perceived usability. PEU is the perception that using a system or technology is free of effort [32] . In addition, PU is defined as the beneficial effect of the technology in the daily activities of an individual [33] . Zheng and Li [34] presented how both PEU and PU are significant latent variables in the acceptance and utility of an application. Mohammadi [32] added that PEU is an influential latent variable that leads to the behavioral intention in using an application. This indicates that if the application is useful and easy to use, then people would have a positive perception of its usability. Especially in health-related applications, Walrave et al. [35] presented how the benefits of an application such as contact tracing would lead to a significant effect on the perceived usability. Therefore, the following were hypothesized: Hypothesis 4 (H4). PEU has the most significant effect on PUS.

Hypothesis 5 (H5). PU has the most significant effect on PUS.

Attitude, as one of the latent factors in the Technology Acceptance Model, indicates the positive or negative behavior the individual establishes upon using the technology. Parasuraman and Colby [36] explained how the contact tracing application is a new technology, thus the people's acceptance and perception of usability may be affected due to their attitude in adaptation. Moreover, Li et al. [29] showed how there are negative perceptions in contact tracing adoption due to a pessimistic attitude in using a system. However, the study of Ong et al. [37] explained that when health is at stake, people would consider utilizing technologies, which will lead to a positive perception of how highly usable a technology is. Thus, it was hypothesized that: Hypothesis 6 (H6). A has the most significant effect on PUS.

IU in this study pertains to the purpose of using an application, while AU is the definite utility of an application [38] . Dehghani et al. [39] studied enabling technologies, their benefits towards health-related concerns, and the goods the technology may provide. Their study showed how both IU and AU affect the perceived usability among individuals when dealing with health-related applications. Following the study of Pal and Vanijja [33] , the actual system has been seen to have a significant effect preceded by usability. In addition, Ong et al. [37] explained how people who have high IU would lead to AU due to their perception of usability and usefulness. Moreover, to measure the usability, this study adapted the questions under the System Usability Scale. System usability scale is said to be the widely utilized tool with standardized questions in assessing perceived usability [40] . To which, the following were hypothesized: Hypothesis 7 (H7). IU has the most significant effect on PUS.

. AU has the most significant effect on PUS.

Presented in Table 1 are the constructs considered in this study. Different kinds of literature were considered for the adaptation of the different items utilized. Under Understanding of COVID-19 (U), there were 6 items, perceived vulnerability (PV) and perceived ease of use (PEU) has 5, perceived severity (PS) and perceived usefulness (PU) has 7, attitude towards using (A) and intention to use (IU) has 5, actual system use (AU) has 6, and perceived usability (PUS) has 10. The data were available from answering a 5-point Likert Scale. Prior to the full distribution of the questionnaire, a pilot test was conducted among 150 responses. The responses collected underwent validation using Chronbach's alpha test. The result presented a value of 0.836, indicating that the questionnaire is valid for full distribution [52, 53] .

Through convenience sampling, a total of 800 respondents voluntarily answered the survey for the perceived usability of the "Thai Chana" COVID-19 tracing application. Utilizing an online survey, the questionnaire was distributed through different social media platforms due to the strict COVID-19 lockdown implemented. Before responses were collected, a question regarding their utility of the Thai Chana application was asked. Only those who utilized the Thai Chana mobile application were considered since other respondents (250) answered they utilized the paper documents. Thus, 800 valid data were considered. From the collected data, a total of 37,600 datasets were considered (respondents and their responses) to run the deep learning neural network and random forest classifier.

Presented in Figure 2 is the methodological flowchart utilized in this study. Data acquisition was carried out through an online survey. Data pre-processing is considered a correlation analysis to determine significant indicators for each latent variable. It was seen that 47 total constructs were collected from the study. Following the study of Kuo and Zulvia [54] , those with values less than 0.20 correlation coefficient and p-value greater than 0.05 were removed due to insignificance. After which, data aggregation of the 8 latent variables were considered as input parameters for the machine learning algorithm. A min_max scalar normalization technique was utilized for the algorithm [54] . Running the random forest classifier and deep learning neural network, parameter optimization was conducted. Following the study of Chen et al. [22] , parameters for criterion, splitter, training and testing ratio, and depth were considered in this study. For the neural network section, the activation functions, optimizer, number of nodes, number of epochs, and number of hidden layers were optimized [55] [56] [57] [58] [59] . The following sections provide a detailed explanation for each algorithm.

Following the parameter optimization is the testing for accuracy using cross validation techniques. Following several studies [56] [57] [58] [59] , a 60% threshold was set for the accepted accuracy rate of the classification model. After the creation of the final classification model, validation was conducted to test the created models. After which, interpretation was conducted. As support for the utility of machine learning algorithms, no computational complexity was seen. Liu et al. [60] explained how the utilization of complex machine learning algorithms would reduce computation time, complexity, and would have higher accuracy. This study considered Python 3.8 to run all algorithms with SKLEARN and Tensorflow packages. Justus et al. [61] explored the computational complexity of using highly utilized resources and showed that an increase in computation time would be reduced with the current up-to-date technology. From the packages utilized, the main cost of computational complexity would vary from the features used, input data, model complexity, and feature extraction [62, 63] . Moreover, the more complex the model is, the more training time [63] . From this study, it was seen that one combination of random forest classifier spent only 0.146 s while deep learning neural network considered 0.480 s. This presents little to no computational complexity with the utilization of the ACER NITRO 5 with core i5 processor, 8 GB RAM, 1 TB Hard disk, Nvidia GeForce GTX 1050 Graphics, running on a Windows 10 system.

The purpose of using the random forest classifier is to create a classification model that considers different features coming from the constructs which represents the unobserved variables utilized in this study. Following the study of Gao et al. [64] , the random forest classifier creates a tree model that considers features that are unified, predicts multi-class dimensions, and presents only significant factors for the classification. The advantage of this algorithm is to present only those significant factors and generate a higher accuracy rate compared to other simple classification models [22] . With the aim to predict and classify factors affecting perceived usability of a new technology among users, random forest classifier may be used to create a classification model for extension and application of predicting influential factors with the integrated theories of Technology Acceptance Model and Protection Motivation Theory. For the utilization of random forest classifier, data preprocessing was completed by inspecting missing data. The SPSS 25 indicated no missing data. Following this, data cleaning considering correlation analysis was performed. This study considered a threshold for the correlation value of greater than 0.20 with a p-value less than 0.05. From a total of 47 constructs, 34 were considered significant. To which, the data were aggregated to focus on the different factors that influence the usability of Thai Chana application. The factors of U, PV, PS, PEU, PU, A, IU, and AU served as the input nodes for the deep learning neural network. Moreover, PUS served as the output during training.

Data normalization was done and the random forest classifier was utilized considering different parameters such as the criterion (gini and entropy), splitter (best or random), training and testing ratio (60:40, 70:30, 80:20, 90:10) , and tree depth (4, 5, 6, 7) . Utilizing SKLEARN Packages in Python 4.5, 100 runs of each combination were considered for a total of 6400 optimization runs. It was seen that gini, best, and a 5-tree depth utilizing an 80:20 training ratio presented the highest average accuracy of 92% with 0.00 standard deviation.

Deep learning neural network is known to be 'the best model' for predicting factors or recognizing patterns due to its ability to assess and calculate several perceptions [65] . Daube et al. [65] stated how this algorithm resonates with the human-level performance in terms of real-world classification. Luceri et al. [66] added how deep learning neural network could best predict human behavior, social interaction, subjective thoughts, and feeling. Utilizing deep learning neural network in this study would support the result presented from random forest classifier. Since random forest classifier only predicts significant factors, using deep learning neural network could predict and classify the most impactful factors affecting perceived usability of technology in using the Thai Chana COVID-19 contact tracing mobile application.

Deep learning neural network preprocessing considered data cleaning using correlation analysis, similar to the set conditions with random forest classifier. After data normalization, different activation functions for the hidden layer (sigmoid, tanh, and swish) and output layer (sigmoid and softmax), together with the optimizer (Adam, SDG, RMSProp) was considered. Moreover, the number of nodes was also included for the 80:20 training and testing ratio. To which, a total of 6300 runs were conducted for the feed-forward deep learning neural network process; 10 runs per combination with 150 epochs [56] . This was conducted to determine the best parameters for the deep learning neural network model.

The collected data comprised 51.88% male, 45.62% female, and 2.50% others. The majority of which were within 15-24 years old (77.87%), 10.50% were within 25-34 years old, and the rest were 35 years old and above with salaries/allowances less than 15,000 Thailand Bhat (THB) ($454) (41.25%) and 43.12% within THB 15,000-30,000 (USD 454-USD 909.85). Following the suggestion of Ong et al. [14] , younger generations are the ones that are most active online, thus justifying that the majority of the respondents are within this age group. In Thailand, most of the younger generation considers using technology easy compared to the older generation that utilizes the document form, since they do not have mobile phones [57] . Lastly, 60.50% were enrolled in insurance and the rest were not. Presented in Table 2 is the summary of the descriptive statistics of the demographics.

In addition, the descriptive statistics of the responses are presented in Table 3 . The mean and standard deviation of each indicator is presented. Figure 3 presents the optimum decision tree considering random forest classifier. It could be seen that at 5-tree depth, gini as the criterion, and best as splitter, the random forest classifier produced the highest average accuracy at 80:20 training and testing ratio. With 92% average accuracy from testing and 0.00 standard deviation, this tree was considered for the interpretation of results. The result for random forest classifier was set to very low, low, neutral, high, and very high usability following the 5-point Likert Scale from the survey. To which, these correspond to strongly disagree (1) to strongly agree (5). Figure 3 presents the optimum decision tree considering random forest classifier. It could be seen that at 5-tree depth, gini as the criterion, and best as splitter, the random forest classifier produced the highest average accuracy at 80:20 training and testing ratio. With 92% average accuracy from testing and 0.00 standard deviation, this tree was considered for the interpretation of results. The result for random forest classifier was set to very low, low, neutral, high, and very high usability following the 5-point Likert Scale from the survey. To which, these correspond to strongly disagree (1) to strongly agree (5) .

The result indicates that PV (X1) will present as the parent node which will lead to perceived usability. The parent will then consider U (X0) if true, which will consider both PV and U, leading to very high PUS. If U will not be satisfied, it will consider PS (X2) and PEU (X3) which will lead to high PUS. Thus, it could be deduced that if people understand COVID-19 and see the vulnerability, Thais would consider Thai Chana very usable. On the other hand, if people know the severity and there is an ease in using the application, then Thais would agree that Thai Chana is usable. Therefore, these factors would lead to the perception of usability of Thai Chana as a COVID-19 contact tracing application. However, the most significant factor could not be indicated since the tree only indicates the path leading to the classification of significant factors. To verify the significance level of the factors, neural network may be applied to show the score of importance among the different significant factors. Abiodun et al. [56] explained how applications of artificial neural network such as deep learning would help in predicting the most significant factor affecting cognitive computing. Presented in Table 4 is the summarized result for the initial optimization of deep learning neural network. With a 60% threshold for the average accuracy [55-59] of the The result indicates that PV (X 1 ) will present as the parent node which will lead to perceived usability. The parent will then consider U (X 0 ) if true, which will consider both PV and U, leading to very high PUS. If U will not be satisfied, it will consider PS (X 2 ) and PEU (X 3 ) which will lead to high PUS. Thus, it could be deduced that if people understand COVID-19 and see the vulnerability, Thais would consider Thai Chana very usable. On the other hand, if people know the severity and there is an ease in using the application, then Thais would agree that Thai Chana is usable. Therefore, these factors would lead to the perception of usability of Thai Chana as a COVID-19 contact tracing application. However, the most significant factor could not be indicated since the tree only indicates the path leading to the classification of significant factors. To verify the significance level of the factors, neural network may be applied to show the score of importance among the different significant factors. Abiodun et al. [56] explained how applications of artificial neural network such as deep learning would help in predicting the most significant factor affecting cognitive computing.

Presented in Table 4 is the summarized result for the initial optimization of deep learning neural network. With a 60% threshold for the average accuracy [55] [56] [57] [58] [59] of the deep learning neural network, only U, PS, and PV were considered to be significant within the threshold set. The results obtained from the initial optimization undergo Analysis of Variance (ANOVA) to determine the significant differences among the different factors (U, PS, and PV). The factor causing the significant difference is considered for the final optimization to determine the average accuracy for predicting factors affecting perceived usability of Thai Chana COVID-19 tracing application. U is seen to be the most significant factor with no overfitting.

The final optimization is conducted with parameters of swish for the hidden layer activation function (30 nodes and 10 nodes) and softmax for the output layer. With adam as the optimizer, the deep learning neural network was run with 200 epochs, 50 times each, for both 70:30 and 80:20 training and testing ratios [59] . The results indicate the highest average accuracy of 97.32% with 1.632 standard deviations for U with an 80:20 training testing ratio. Figure 4 presents the training and validation loss of the final optimization. Based on the result, no overfitting is present [59, 64] . Walczak and Cerpa [58] considered this result as relatively high for acceptability with human behavior studies. Figure 5 represents the final deep learning neural network utilized in this study. Based on the figure, the input layer considered 8 nodes (factors) with 30 and 10 nodes for the hidden layers. The activation function of swish and sigmoid with adam as optimizer produced the highest average testing accuracy of 97.32%. It could be deduced that U is the most significant factor affecting the perceived usability of Thai Chana COVID-19 tracing application. activation function (30 nodes and 10 nodes) and softmax for the output layer. With adam as the optimizer, the deep learning neural network was run with 200 epochs, 50 times each, for both 70:30 and 80:20 training and testing ratios [59] . The results indicate the highest average accuracy of 97.32% with 1.632 standard deviations for U with an 80:20 training testing ratio. Figure 4 presents the training and validation loss of the final optimization. Based on the result, no overfitting is present [59, 64] . Walczak and Cerpa [58] considered this result as relatively high for acceptability with human behavior studies. The score of importance is presented in Table 5 . Based on the results, U had the highest score of importance (100%), followed by PS (87.5%), PV (77.2%), then PEU (66.0%). Other factors were considered significant but presented a significantly lower score of importance (>60%). The score of importance is presented in Table 5 . Based on the results, U had the highest score of importance (100%), followed by PS (87.5%), PV (77.2%), then PEU (66.0%). Other factors were considered significant but presented a significantly lower score of importance (>60%).

To verify the findings, Pearson's correlation analysis was conducted, utilizing SPSS V26. Table 6 presents the result from the correlation analysis and presented that U, PS, PV, and PEU were highly correlated with PUS. All of the correlation coefficients presented significant values (p-value < 0.05). 

This study utilized deep learning neural network and random forest classifier for predicting factors influencing the perceived usability of Thai Chana COVID-19 tracing application. Factors such as understanding COVID-19 (U), perceived severity (PS), perceived vulnerability (PV), perceived ease of use (PEU), perceived usefulness (PU), attitude towards using (A), intention to use (IU), and actual system use (AU) were considered to measure perceived usability (PUS). Based on the result, a 97.32% average accuracy from deep learning neural network was seen and a 92% average accuracy with a 0.00 standard deviation from random forest classifier. Deep learning neural network showed that U had the highest significant effect, followed by PS, PV, and PEU. Consistent with the result from random forest classifier, U and PV led to very high PUS while PS and PEU led to high PUS.

Understanding COVID-19 was seen to be the most significant factor affecting PUS for both deep learning neural network and random forest classifier. The indicators included Thais' understanding of the COVID-19 virus, its incubation period, symptoms, protocols, when to get the vaccine, and which hospital to go to. The comparison of Asian counties was considered by Wang et al. [67] . Their study showed that Thais have an 89.8% positive belief regarding the knowledge about COVID-19 and that they know its effect when they are in contact. Caldwell et al. [68] explained how the knowledge with the minimum health standard could help in the reduction of COVID-19 transmission. In addition, Ning et al. [69] explained how the success of healthcare professionals regarding interventions would have a positive effect when people know, accept, and understand the disease. Relating to this study, the more people would understand and know about the COVID-19 virus, the more likely they will have the intention to prevent any contraction.

Second, PS was seen to be significant towards PUS among Thais. The indicators included Thais finding of the seriousness of the disease, affects people's mental health, prolong the outbreak, and that Thailand is more severe than other ASEAN countries. In relation to the study of Fragkaki et al. [70] , when people have a higher perception of severity and high government satisfaction, the more likely they will exhibit behavior change. Based on the results, PS was seen to be the second-highest significant factor. Thus, it could be instigated that Thais a have higher perception of severity, leading to the significant factor affecting the PUS of contact tracing application for reduction of exposure. Mant et al. [71] explained that when people have high PS, they would change their behaviors to mitigate COVID-19 transmission. For this study, people will have high PUS when they have high PS of the COVID-19 virus. As a support, this result is also consistent with the findings from Trkman et al. [72] .

Third, PV was seen to be a significant factor affecting PUS. The indicators included Thais' belief in their vulnerability to COVID-19 personally (self, friends, and family), location, and that their country is more vulnerable compared to other ASEAN countries. With that, people have high levels of PUS for the Thai Chana COVID-19 tracing application since they know it will help mitigate the contraction of COVID-19. Boyraz et al. [73] explained that PV positively affects people's worries, traumatic experiences, and stress. To which, the more vulnerable the perception is, the higher the will to mitigate contraction. This is in line with the results from De Coninck et al. [74] from Belgium. They explained how greater health measures were seen in protecting the population when there is a greater belief in PV. Similar to the results of Ong et al. [37] , when people understand the risk, they would highly consider the PV. This would lead to the mitigation in reducing any health-related risks.

Lastly, PEU had a significant effect on PUS among Thai. They believe that Thai Chana as an application can provide clear and understandable information related to COVID-19, successful usage of the application every time, and that the application is easy to use. When utilizing an application, the PEU is usually considered a significant factor [15] . Prasetyo et al. [15] explained how PEU can affect the intention to use a certain application. PEU was also seen to be one of the most significant factors affecting the acceptance and usage of technology [75, 76] .

Interestingly, PU and A were not considered significant factors affecting PUS. Guillon and Kergall [77] explained how the attitude of a person directly correlates with their belief towards the advantage of quarantine. In addition, Guillon and Kergall [77] explained how trust in the government and health consequences would lead to a high willingness to utilize contact tracing applications. In relation to this study, as long as there is high PS and PV, people will continue to utilize the Thai Chana contact tracing application despite the PU and A. Thus, the advantage of mitigation could be said to revolve around the risk and severity of contracting the virus rather than PU and A. This also supports why IU and AU were not significant. The reason why contact tracing applications are utilized is because of the advantages when it comes to health-related concerns [78, 79] . The reason for mitigation upon utilizing the contact tracing application among Thais is for reduction of COVID-19 virus contraction, reduce exposure, and safety in general.

Overall, it could be deduced that when people understand COVID-19, as well as its severity and vulnerability, it would affect their perception of the usability of the Thai Chana COVID-19 tracing application. It could be generalized that when there is risk and health-related concerns, people would understand the benefit of utilizing contact tracing applications. Moreover, as long as there is perceived ease of use, then people would continuously utilize the application. Therefore, this should be considered to promote the utilization of contact tracing applications, not only in Thailand, but may also be applicable to other contact tracing applications worldwide.

The utilization of the machine learning algorithm for human behavior was seen to have different advantages. Machine learning algorithms, such as the artificial neural network, is a type of artificial intelligence that mimics how the body sends a signal to the brain through different neurons that create an output [79] . Deep learning neural network is a type of neural network that has two or more hidden layers that can process the information further and produce higher accuracy [54] . Moreover, Vasilev et al. [80] deep learning neural network has a higher power when it comes to calculation. Ais has been utilized during the COVID-19 pandemic to help in screening, tracking, and predicting future events [80, 81] . In addition, Jamshidnezhad et al. [82] utilized a machine learning algorithm for the transmission rate for COVID-19 outbreaks in Iran. Thus, it could be inferred that utilizing machine learning algorithms, such as the deep learning neural network or random forest classifier, may contribute to help mitigating and reducing contraction during the COVID-19 pandemic.

With the utilization of deep learning neural network, it was seen that there was a high accuracy of 97.32% in predicting perceived usability on contract tracing applications. Thus, it could be deduced that the accuracy of prediction is highly reliable. In addition, the 92% accuracy of the srandom forest classifier having consistent results with deep learning neural network further justified the findings. Juarez-Orozco et al. [83] and Chen et al. [22] explained how random forest classifier can be a powerful predictive machine learning algorithm tool for human behavior with higher accuracy since it determines the optimum tree among other decision trees produced. Combining both the result for neural network and random forest classifier would therefore be beneficial in predicting human behavior [22] .

It was seen from the validation that the results presented well-grounded output in using machine learning algorithms to assess and predict factors affecting human behavior, specifically perceived usability of a technology. Academically, the methodology may be applied and extended in related fields of studies, such as assessing behavior and predicting factors applying machine learning algorithms for classification. It could be deduced that integrating multivariate tools with machine learning algorithms or solely using a machine learning algorithm may be utilized to holistically measure and predict human behavior. Recognizing patterns may be evaluated with the consideration of constructs, and measure items to assess unobserved variables.

Applying the findings of this study, the government may consider implying the severity and vulnerability of COVID-19 in Thailand, comparing it to other countries, and help people understand the COVID-19 virus [84] . This would promote how advantageous Thai Chana as a contact tracing application is. In addition, the usability of Thai Chana was seen to be easy among Thais. This means that there may be only a few changes needed to be made to promote usage. Based on the findings, it could be seen that the highlight on severity and vulnerability may be capitalized on to enhance the perception of usability, increase motivation, and enhance the Thai Chana COVID-19 contact tracing mobile application's applicability every day. People already understand the implication, side-effects when infected, negative effects, and health complications when infected with the virus. Thus, there is only a need for highlighting the vulnerability and severity in promoting the mobile application. Moreover, the government may enhance motivation by indicating the benefits, both community-wise and health-wise. In addition, the government may also promote the utilization of Thai Chana by campaigning and addressing the public's concerns. The findings of the study from Munzert et al. [43] showed that people questioned the effectiveness of different contact tracing applications when they are not promoted properly. Therefore, the need to highlight the use, its intention, and application would help people in using the Thai Chana COVID-19 contact tracing mobile application.

This study may have attained a high accuracy rate for both machine learning algorithms, however, there are still limitations. First, this study was conducted only in Thailand and focused on the main contact tracing application, Thai Chana. Other contact tracing applications were not considered. Future researchers may consider other contact tracing applications and compare them to determine the significant factors to generalize the findings. In this way, the contact tracing application may be generalized to enable promotion and usage [82] . Second, this study only considered two machine learning algorithms, deep learning neural network and random forest classifier. Other machine learning algorithm tools may be considered and compared to promote the utility of other algorithms applicable [65] . Classification tools such as support vector machine, and optimization techniques such as particle swarm optimization may be applied. In addition, the location was not considered. The urban and rural areas may have different results depending on their understanding of COVID-19 [85] [86] [87] , perceived severity, and perceived vulnerability towards utilizing the Thai Chana contact tracing application. Thus, clustering may also be applied to enhance the findings of the study, such as KMeans algorithm or Fuzzy CMeans. Lastly, the majority of the age group considered in this study was between 15 and 24 years old only. Due to the COVID-19 lockdown implemented, an online questionnaire was only utilized. As explained by Ong et al. [14] , most of the generation utilizing social media platforms are at a younger age. Thus, it is recommended to consider in-person data collection to consider broader and wider age groups. Moreover, findings may be different based on the effect of the virus, perception, and even health-related disease outcomes among the different age groups. Moreover, interviews may be conducted to consider the qualitative measures applicable in measuring perception, knowledge, understanding, and usability errors. A qualitative-quantitative approach may then be utilized to highlight and generalize the perceived usability of Thai Chana as a COVID-19 contact tracing mobile application. This may produce results that may be utilized for workshops, and even create possible techniques that are applicable to other studies.

The progress in mitigating the COVID-19 contraction has been slow. Due to new variants and the mutation of the COVID-19 virus, the vaccination program was not able to keep up with the progression. To which, countries implemented contact tracing applications to help reduce exposure to the COVID-19 virus [88] [89] [90] [91] . However, there is a lack of study regarding the different applications available. Specifically, the Thai Chana COVID-19 contact tracing application has been underexplored. Thus, this study aimed to predict factors affecting the perceived usability of Thai Chana in Thailand integrating Protection Motivation Theory, Technology Acceptance Model, and System Usability Scale using deep learning neural network and random forest classifier. Specifically, factors such as understanding COVID-19 (U), perceived severity (PS), perceived vulnerability (PV), perceived ease of use (PEU), perceived usefulness (PU), attitude towards using (A), intention to use (IU), actual system use (AU), and perceived usability (PUS) were considered in this study.

With the 800 voluntary Thais participants in the survey, a total of 37,600 datasets were considered. Applying the deep learning neural network, the results produced an average accuracy of 97.32% and 92%, with 0.00 standard deviation for random forest classifier. From the results, U was seen to be the most significant factor, followed by PV, which leads to high PUS. Moreover, PS and PEU were considered significant with high PUS among Thais in using the Thai Chana contact tracing application. When people understand the COVID-19 virus, their perception of severity, and perceived vulnerability would lead to a high perception of contact tracing usability. This means, in order to promote the utilization, the government should instill awareness of the severity and vulnerability among people of the COVID-19 virus. This will also help people to understand how the virus can affect them, leading to the continuous usage of Thai Chana. With that, the government may capitalize on the finding of this study to promote motivation for continuous usage of the mobile application. When people see how the mobile application may help promote positive health-related behaviors and highlight the effect of reduced vulnerability and severity, an increase in motivation for the utility would be applied. As explained by Siddiqui et al. [5] , people who are knowledgeable will practice promoting mitigation and intention to reduce the negative effects of COVID-19. In addition, Chuenyindee et al. [13] explained how implementing the utilization of technology does not necessarily promote positive usage. Thus, with the proposed suggestions, people will be highly motivated to use the system.

The consideration of the machine learning algorithm may be highlighted in this study. With high accuracy rates, it was validated that the machine learning algorithms may be utilized for assessing and predicting factors affecting human behavior, not just in assessing technology usability. The disadvantage was the time it took to finish the optimization process for the different parameters to enhance the accuracy of the models considered. In addition, the advantage is the model and parameter setting that was considered, which could be utilized for other studies considering human behavior. In addition, despite several factors considered, the high-quality machine learning algorithms were able to assess the study effectively. Thus, machine learning algorithms could be used to assess perceived usability of technology and other related studies such as natural disasters and even education [90] .

The framework and findings of this study may be considered by other countries for their independent contact tracing application. It was seen that when the citizen understood the impact of the virus, how it could affect the health of the people negatively, and how the contact tracing mobile application would help in mitigating the negative side-effects, then people would be more likely to consider and utilize a system to reduce it. It could be deduced that the integrated framework holistically measured health-related technology, its application, and usability among user. Moreover, this may be considered to promote the utilization of contact tracing applications, not only in Thailand [91] , but may also be applicable to other contact tracing applications worldwide. Lastly, future researchers may consider the method and framework utilized in this study for the evaluation of applications and systems worldwide. 

The data presented in this study are available on request from the corresponding author.

Review of COVID-19 variants and COVID-19 vaccine efficacy: What the clinician should know?

Working conditions and anxiety levels of individuals who have to work during the COVID-19 pandemic

Burnout and anxiety level of healthcare professionals during the COVID-19 pandemic in Turkey

Risk of COVID-19 infection in workplace settings and the use of personal protective equipment

Knowledge and practice regarding prevention of COVID-19 among the Saudi Arabian population

Quality and Adoption of COVID-19 Tracing Apps and Recommendations for Development: Systematic Interdisciplinary Review of European Apps

Researching COVID-19 tracing app acceptance: Incorporating theory from the technological acceptance model

Learnings from the design and acceptance of the German COVID-19 tracing app for IS-driven crisis management: A design science research

A Survey of COVID-19 Contact Tracing Apps

Crisis Communication of Website and Application Developed for COVID-19 Pandemic Situation in Thailand

Ten commonly asked questions about COVID-19 and lessons learned from Thailand

An examination of Thailand's health care system and strategies during the management of the COVID-19 pandemic

Factors Affecting the Perceived Usability of the COVID-19 Contact-Tracing Application "Thai Chana" during the Early COVID-19 Omicron Period

Factors affecting intention to prepare for mitigation of "the big one" earthquake in the Philippines: Integrating protection motivation theory and extended theory of planned behavior

Determining Factors Affecting Acceptance of E-Learning Platforms during the COVID-19 Pandemic: Integrating Extended Technology Acceptance Model and DeLone & McLean IS Success Model

Factors affecting perceived effectiveness of Typhoon Vamco (Ulysses) flood disaster response among Filipinos in Luzon, Philippines: An integration of protection motivation theory and extended theory of planned behavior

Using protection motivation theory in the design of nudges to improve online security behavior

Effectiveness of privacy assurance mechanisms in users' privacy protection on social networking sites from the perspective of protection motivation theory

Utilizing Health Behavior Change and Technology Acceptance Models to Predict the Adoption of COVID-19 Contact Tracing Apps: Cross-sectional Survey Study

Applications of structural equation modeling (SEM) in ecological studies: An updated review

An SEM Perspective on Evaluating Mediation: What Every Clinical Researcher Needs to Know

A Machine Learning Ensemble Approach Based on Random Forest and Radial Basis Function Neural Network for Risk Evaluation of Regional Flood Disaster: A Case Study of the Yangtze River Delta

Customer switching behavior analysis in the telecommunication industry via pushpull-mooring framework: A machine learning approach

Real-time prediction of nuclear power plant parameter trends following operator actions

Familial risk and protective factors affecting CPS professionals' child removal decision: A decision tree analysis study

Neural Networks and Deep Learning: A Textbook

Earthquake risk assessment using an integrated Fuzzy Analytic Hierarchy Process with Artificial Neural Networks based on GIS: A case study of Sanandaj in Iran

Artificial neural network for predicting earthquake casualties and damages in Indonesia

What makes people install a COVID-19 contact-tracing app? Understanding the influence of app design and individual difference on contact-tracing app adoption intention

Understanding the Internet banking adoption: A unified theory of acceptance and use of technology and perceived risk application

An Empirical Study on Consumer Acceptance of Mobile Payment Based on the Perceived Risk and Trust

Investigating users' perspectives on e-learning: An integration of TAM and IS success model

Perceived usability evaluation of Microsoft Teams as an online learning platform during COVID-19 using system usability scale and technology acceptance model in India

What drives students' intention to use tablet computers: An extended technology acceptance model

Ready or Not for Contact Tracing? Investigating the Adoption Intention of COVID-19 Contact-Tracing Technology Using an Extended Unified Theory of Acceptance and Use of Technology Model

An Updated and Streamlined Technology Readiness Index

Investigating the acceptance of the reopening Bataan nuclear power plant: Integrating protection motivation theory and extended theory of planned behavior

Toward a better understanding of behavioral intention and system usage constructs

Exploring the motivational factors on continuous usage intention of smartwatches among actual users

The system usability scale: Past, present, and future

Factors affecting perceived effectiveness of COVID-19 prevention measures among Filipinos during Enhanced Community Quarantine in Luzon, Philippines: Integrating Protection Motivation Theory and extended Theory of Planned Behavior

Research on robustness of five typical data-driven fault diagnosis models for nuclear power plants

Tracking and promoting the usage of a COVID-19 contact tracing app

Protection Motivation and the COVID-19 Virus

Method and Acceptance of COVID-19 Website Users in Indonesia

Attitudes Toward Using COVID-19 mHealth Tools Among Adults with Chronic Health Conditions: Secondary Data Analysis of the COVID-19 Impact Survey

Determination of factors affecting the response efficacy of Filipinos under Typhoon Conson 2021 (Jolina): An extended protection motivation theory approach

Cutting tool quality improvement: A case study of electronic parts manufacturer

Perceived usability evaluation of learning management systems: Empirical evaluation of the System Usability Scale

Choosing a package carrier during COVID-19 pandemic: An integration of pro-environmental planned behavior (PEPB) theory and service quality (SERVQUAL)

Automatic clustering using an improved artificial bee colony optimization for customer segmentation

Landslide susceptibility assessment and factor effect analysis: Backpropagation artificial neural networks and their comparison with frequency ratio and bivariate logistic regression modelling

Factors influencing the Thai elderly intention to use smartphone for e-Health services

State-of-the-art in artificial neural network applications: A survey

A deep learning approach for automatic recognition of seismo-volcanic events at the Cotopaxi volcano

An Adaptive CU Size Decision Algorithm for HEVC Intra Prediction Based on Complexity Classification Using Machine Learning

Predicting the computational cost of Deep Learning Models

Computational Performance

A softmax classifier for high-precision classification of ultrasonic similar signals

Grounding deep neural network pre-dictions of human categorization behavior in understandable functional features: The case of Face Identity

Analyzing and inferring human real-life behavior through online social networks with social influence deep learning

The impact of COVID-19 pandemic on physical and mental health of Asians: A study of seven middle-income countries in Asia

Understanding COVID-19 dynamics and the effects of interventions in the Philippines: A mathematical modelling study

The impacts of knowledge, risk perception, emotion and information on citizens' protective behaviors during the outbreak of COVID-19: A cross-sectional study in China

Human responses to COVID-19: The role of optimism bias, perceived severity, and anxiety. Pers

Canadian university students' perceptions of COVID-19 severity, susceptibility, and health behaviours during the early pandemic period. Public Health Pract. 2021, 2, 100114

The impact of perceived crisis severity on intention to use voluntary proximity tracing applications

COVID-19 and traumatic stress: The role of perceived vulnerability, COVID-19-related worries, and social isolation

Perceived vulnerability to disease and attitudes towards public health measures: COVID-19 in Flanders

Effect of perceived ease of use on librarians' e-skills: Basis for library technology acceptance intention

The Effects of Perceived Usefulness and Perceived Ease of Use on Continuance Intention to Use E-Government

Attitudes and opinions on quarantine and support for a contact-tracing application in France during the COVID-19 outbreak

Exploring the competing influences of privacy concerns and positive beliefs on citizen acceptance of contact tracing mobile applications

Neural network models and deep learning

Python Deep Learning: Exploring Deep Learning Techniques and Neural Network Architectures with Pytorch, Keras, and TensorFlow

Artificial Intelligence (AI) applications for COVID-19 pandemic

The role of ambient parameters on transmission rates of the COVID-19 outbreak: A machine learning model

The machine learning horizon in cardiac hybrid imaging

The Evaluation of Preference and Perceived Quality of Health Communication Icons Associated with COVID-19 Prevention Measures

Putting the world back to work: An expert system using big data and artificial intelligence in combating the spread of COVID-19 and similar contagious diseases

Factors influencing job satisfaction among dentists during the new normal of the COVID-19 pandemic in Indonesia: A structural equation modeling approach

Theory of planned behavior as a conceptual framework for the willingness to self-isolate during the COVID-19 pandemic: A regional cross-sectional study

Recommending Suitable Smart Technology Applications to Support Mobile Healthcare after the COVID-19 Pandemic Using a Fuzzy Approach

Information Disclosure Contents of the COVID-19 Data Dashboard Websites for South Korea, China, and Japan: A Comparative Study

Digital Health Platforms in Saudi Arabia: Determinants from the COVID-19 Pandemic Experience

Applications of Artificial Intelligence, Machine Learning, Big Data and the Internet of Things to the COVID-19 Pandemic: A Scientometric Review Using Text Mining

Infectivity Upsurge by COVID-19 Viral Variants in Japan: Evidence from Deep Learning Modeling

Blackboard e-learning system acceptance and sat-isfaction among Filipino high school students: An extended technology acceptance model (TAM) approach

Factors Influencing the Perceived Effectiveness of COVID-19 Risk Assessment Mobile Application "MorChana" in Thailand: UTAUT2 Approach

The researchers would like to extend their deepest gratitude to the respondents of this study despite the current COVID-19 inflation rate.

The authors declare no conflict of interest.