key: cord-1021986-vb60rpxb authors: Panchal, M.; Singh, S.; Rodriguez-Villegas, E. title: Analysis of the Factors Affecting the Adoption and Compliance of the NHS COVID-19 Mobile Application date: 2021-03-08 journal: nan DOI: 10.1101/2021.03.04.21252924 sha: a231beb5b5f71a6ad4686a503e4e58aef973ba15 doc_id: 1021986 cord_uid: vb60rpxb Objectives: To conduct an independent study investigating how adults perceive the usability, and functionality of the NHS COVID-19 app. This study aims to highlight strengths, and provide recommendations to improve adoption of future contact tracing developments. Design, Setting, and Participants: 1036 adults in England and Wales completed a 60-item, web-based observational survey. This survey was disseminated through social media outlets and email-lists. Primary Outcome Measures: To evaluate the compliance and public attitude towards the NHS COVID-19 app, regarding its functionality and main features. This included whether participants expectations were met, and their thoughts on the privacy and security of app data. Furthermore, to distinguish how usability, perception, and adoption differed with varying demographics and user values. Results: Fair rates of app compliance were identified across participants, with the app meeting the expectation of 59.7% of participants who downloaded it. However, participants finding the interface challenging were less likely to read information in the app and had a lesser understanding of its functionality. Furthermore, lack of understanding regarding the apps functionality and privacy preserving features were behind the reasons why users did not download it. A readability analysis of the text revealed that the information of the app was conveyed at a level which might only be accessible to under 60% of the population. The study highlighted issues related to the high probability of false positives caused by the design choices in the Check-In feature. Conclusion: This study showed that while the NHS COVID-19 app was viewed positively, however, there remained issues regarding participants perceived knowledge of the app with privacy and functionality, which appeared to affect compliance. Therefore, we have recommended improvements regarding the delivery and presentation of the apps information, and highlighted the potential need for the ability to check out of venues to reduce the number of false positive contacts. • This is the first study assessing the perceived usability and functionality of the "NHS COVID-19" app directly from app users. • Diverse sample size, with representation from all major regions of England. • Statistical analysis to compare perceived app usability and functionality across demographics and the participants values regarding privacy and information. • Study was conducted during lockdown where app use may have been minimal. • Study may have missed potential participants who were not active on the relevant social media channels and email mailing lists which were used to disseminate the survey. The , has become widespread, resulting in over 95,000 UK citizen deaths over the last 10 months [1] . Many unprecedented measures have been imposed on citizens day to day lives such as full isolation of all non-key workers, social distancing, and mask wearing. Contact tracing strategies, aiming at early identification and preventative quarantining of subjects who had been in close proximity of a confirmed virus carrier, have been up-taken by public health authorities to help further prevent the spread of the disease. It has been estimated that if 80% of contacts could be identified, stopping the progression of the pandemic would be extremely likely [2] . Contact tracing strategies have been attempted across the globe, with varying degrees of success [3, 4] . Although in an ideal scenario, one hundred percent of close contacts of one hundred percent confirmed cases would be promptly identified and socially isolated, in practical terms there are limitations to any method trying to achieve this. These limitations include: lack of professional human resources for both interviewing patients and following up with the identified contacts; reliance on patients' memories for up to a two week time span; significant time delays between diagnosis and isolation of contacts; and potential violation of privacy. Furthermore, the resources required for manual contact tracing are significant, with projected costs of the UK's national contact tracing solution (Track and Trace), surpassing £10 Billion [5] . The Track and Trace system was released as the UK began to exit its first national lockdown, requiring citizens to manually check in to venues by providing their name and contact information [6] . Citizens could then be alerted if an outbreak was reported at a venue they recently entered. Logistically, this approach meant that the staff in the venue were responsible for holding citizens accountable for providing their information upon entering. Although practical, this approach was clearly privacy intrusive, as businesses were provided with personal information of all visiting individuals. This process was also seen as incomplete, as the Track and Trace method is ineffective in public areas (Public Transport, Parks, etc.) where citizens cannot check in, and will likely encounter significantly more contacts than they can remember. To this end, smartphone-based digital contact tracing presented itself as a faster and potentially more efficient method of contact tracing, especially at larger scales [7, 8] . Countries such as South Korea, Singapore, and Taiwan showed promise early on in the use of digital solutions to combat COVID-19 [9] . In South Korea, for example, the location of every subject is continuously monitored via their mobile phone, allowing the possibility of identifying who has been in contact with a newly diagnosed individual and for what period of time preceding that diagnosis [10] . The positive effects of this approach, in terms of modulation of the epidemic curve, are proven by the fact that South Korea is the country in which the rate of transmission, after reaching a critically large number of infections, was the lowest in the world early in the pandemic [11] . But, adopting a location based approach poses many ethical questions [12, 13] . Whilst this may be effective and justified in stages of the pandemic in which health services are under risk of collapsing; many public concerns have been raised regarding these system's due to their lack of personal privacy. For example, South Korea's digital contact tracing solution provides the government with access to GPS locations, and card transaction logs of each citizen [14] . This type of approach drew many concerns in the UK, which were clear from early surveys suggesting citizens would not endure a contact tracing solution that is privacy and location intrusive [15, 16] . Thus, to improve uptake of the app, the UK government, in common with many European governments, adopted different design approaches in the development of the "NHS COVID-19" app, focusing on providing UK citizens with a more privacy preserving digital contact tracing solution. The "NHS COVID-19" app aims to be a one-stop system, allowing users to book a COVID-19 test, to self-report a positive COVID-19 result, and also alerts them if they encountered another app user who tested positive for COVID-19. This app uses Bluetooth technology paired with a QRcode based venue check in system to dictate close contacts between users. Furthermore, the app has additional features: informing of the latest COVID-19 restrictions, providing a risk level associated with the users' postal code, and a symptom checker. To combat controversy surrounding privacy, this app has utilized a decentralized approach of data storage, claiming to take as little information as possible, and only requiring the users postal code when downloading [17, 18] . Thus far, the "NHS COVID-19" application has received over 10 million downloads. However, it is predicted that uptake by approximately 40 million people is required for this app to have a significant impact on the pandemic [19, 20] . Despite respectable initial uptake, and claims of implementing a privacy preserving design, the development of the app has faced scrutiny due to development pitfalls, privacy, and security concerns which may have affected citizens overall perceptions of digital contact tracing [21, 22] . Furthermore, due to the use of the Google/Apple API, this app is unfortunately inaccessible for many older phone models. The aforementioned factors may have been potential reasons for the "NHS COVID-19" app not having a higher adoption, which is ultimately crucial to maximize the effectiveness of the app [20] , but no formal study has been carried out exploring to what extent they may have affected this. All studies on UK citizens perception of digital contact tracing occurred before the release of the "NHS COVID-19" app [15, 16] . However, one surveying the public attitudes post-release has not taken place. This is the first independent study that investigates how efficiently the features of the app are being used by the public, in relation to their original intended use; in addition to assessing how citizens perceive the app, both in regards to its functions and usability. The findings of this study highlight the strengths and shortcomings of the app based on users' perceptions and opinions. The "NHS COVID-19" app ( Figure 1 ) contains a variety of features designed to support contact tracing within the population, booking testing services, and giving users an indication of the risk level in the area they live in. [17, 18] Contact tracing is performed through two distinct services: an automatic proximity based system utilising the Bluetooth Low Energy Protocol (Bluetooth Contact Tracing); and a manual venue "check-in" system using the QR Code scanning features of mobile devices (Check-In). Automatic proximity based contact tracing is performed using an API developed jointly by Apple . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted March 8, 2021. ; https://doi.org/10.1101/2021.03.04.21252924 doi: medRxiv preprint and Google, which requires each mobile device to constantly transmit a unique randomised identifying code (which is changed over a period of time) using the Bluetooth Low Energy Protocol. Each mobile device is also constantly scanning for nearby identifying codes from other devices from which distance can be estimated through the measure of Received Signal Strength Intensity (RSSI). This allows each mobile device to maintain an automatic log of other devices it has been within a certain distance from, which can then be used to inform users of those devices to self-isolate, if the user reports a positive test. Users have control on whether or not proximity contact tracing is enabled. The other method of contact tracing involves users manually scanning QR codes at venues they visit, to then store a log of the venues including the times visited. Users are checked out of venues when they either check into another venue or at midnight of the same day they checked in. Other users whose checked in periods in the venue overlap in time will be notified of the need to self-isolate, if one of them reports a positive test. The app also contains a symptom checker tool, designed to inform the users of what to do next depending on the symptoms they are experiencing, and allowing them to book a COVID-19 test. Finally, the app provides users with information on their COVID-19 infection risk within their general location, requiring the first half of the user's postcode, as well as links to the latest COVID-19 advice. Our survey was designed to: 1) Determine how citizens perceived the usability of various features included within the "NHS COVID-19" app; 2) Ascertain the level of usage and compliance with these features; 3) Evaluate the level of users understanding of the app. To design our survey, we partly relied upon methodologies from various reviews and analyses of mobile health usability studies, such as the Systems Wide Analysis of mobile health-related technologies (SWAT) [23] . The vast majority of these usability studies, however, apply a methodology across a wide range of health-related mobile apps, including questions which are either too general or refer to topics not relevant for this study. We therefore determined the common themes used to describe user perceived application usability within these studies, and generated focused questions specific to the "NHS COVID-19" application. The main themes used within our study are summarised in Table 1 . To facilitate both the completion of the survey and also its analysis we used multiple choice questions, and, where further detail was required, free text answer boxes. All questions were optional. If participants did not download the "NHS COVID-19" app they were still able to complete the survey since this, on its own, helped to understand the reasons why they did not engage with the application to start with. Conditional formatting in the survey, however, ensured that participants usability-based questions could not be answered by those who stated not to have downloaded the app. These participants were instead directed to a different set of questions that focused on trying to find out the reasons why they did not download it, and if any suggested changes would have changed their mind. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) Focused on how often users would use the previously outlined functions of the app within a given week or day, as well as identifying user habits that would facilitate the need to use these features. Elicit the participant's feelings toward several issues that may influence their decision to use the app and its various features. These included participants own views on privacy as well as their beliefs on the privacy protection, security, and usefulness of the app itself. Sought to understand whether the "NHS COVID-19" application provided enough information to users about each feature within the app but also on the user's current status regarding their possible COVID-19 exposure, and the actions they should take based on this. Encompassed the user's views on their ability to use the features of the app, whether these were simple or difficult to use, their ability to navigate the application and any technical difficulties they may have faced. Elicit participants understanding of the basic functions and claims of the app. The study received ethical approval by the Science Engineering Technology Research Ethics Committee of Imperial College London (SETREC ref.: 20IC6427). The survey (Supplementary File) consisted of 60 questions which were implemented and released through the Qualtrics Xm survey platform. It was distributed for 45 consecutive days during Decemeber 2020 and January 2021 using different social media platforms (LinkedIn, Twitter, Reddit, Facebook, and NextDoor), emailing lists, and word of mouth. Survey respondents needed to consent to take part in the study, and were required to be above the age of 18. We determined mathematically the required sample size that needed to be collected by our survey in order to be able to generalise the findings of our study to the population of the UK with statistical confidence. Assuming 95% confidence level, and a margin level of 3%, a sample size close to 1068 respondents was required. The vast majority of our results are uni-variate, reducing our ability to apply statistical tests for these questions. Therefore, only the proportion of responses for each category within these questions are presented in the Results section below, and analyzed using descriptive statistics. Additionally, we performed comparisons between questions, in order to identify significant correlations between certain user demographic/values/app knowledge, with how users . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted March 8, 2021. ; https://doi.org/10.1101/2021.03.04.21252924 doi: medRxiv preprint perceived the app and its features. Within these comparisons we performed the Chi-Squared test for independence on comparisons between categorical data and the Kruskal-Wallis H test between categorical and ordinal data with more than two groups. We evaluated the p-values for these comparisons and determined statistical significance at the level of α=0.05, performing Bonferroni correction where appropriate. The analyses were carried out using the Python programming language with the commonly used Pandas and SciPy Python packages. The Flesch Reading Ease test was used to assess the overall readability of the explanatory body text on the on-boarding screens, and the "About" section of the app. This test is based on the sentence and word lengths present in a body of text and has been used effectively in other studies involving the presentation of medical information [24] . No participants, patients, or members of the general public were involved in the overall design, conduct, reporting, or distribution of the research. In total, 1036 participants completed the study, with the sampling of participants being representative of all age groups, and regions in the UK. Due to the multiple educational credentials submitted, education was grouped based on whether the participant's highest educational achievement was above a UK A-level, or A-level and below. Demographic data is shown in Table 2 . A much larger proportion of participants (848, 82.1%) had downloaded the app, compared to those that did not (185, 17.9%). As per the reason the 185 (17.9%) participants did not download the app, 68 (26.7%) of them "Did not feel safe downloading", 67 (26.6%) "Did not see any benefit", 29 (11.5%) stated "Due to place of work", and 19 (7.5%) were unable to, due to the app not being compatible with their device. Furthermore, 74 (34.7%) of these participants responded that they would have downloaded the app" If it was completely anonymous", despite the application already claiming to be completely anonymous. When asked if participants changed their behavior based on the assigned risk-level to their area, 519 (63.1%) users marked that they had, which indicated that the app had had some positive effect in terms of enabling users to make informed behavioral decisions, as a function of the local evolution of the pandemic. The complete participant results outlining compliance can be found in the Tables S1 (Supplementary materials). . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) Among the 848 (82.1%) participants that did download the app, 722 (85.1%) were still using the app at the time of the survey, and 715 (84.3%) participants had had the app downloaded for more than two months. Most participants would use the app minimally; 626 (73.8%) participants would open the app once a week, or not at all. This, in isolation, would not be a concerning result as the app's Bluetooth Proximity Contact tracing feature does not require regular app foreground use in order to function effectively, due to its use of the Apple/Google API. However, it also indicates that other features of the app, which require foreground use, may not have been used as frequently as they were designed to be, such as the manual venue "Check-In" system and the "Read Latest Advice" tools. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted March 8, 2021. ; https://doi.org/10.1101/2021.03.04.21252924 doi: medRxiv preprint Regarding the contact tracing features of the app, the majority of the participants (631 74.4%) stated that they always kept the Bluetooth "Contact Tracing" feature enabled. While this is an encouragingly high proportion of users, this number not being 100% ultimately means that the number of app downloads may not be indicative of the amount of users who can be contact traced. When asked how many venues were visited on average, the majority of participants (447, 52.5%) stated they visited 5 or less venues a week. However, of the participants able to visit venues, more than half (494, 69.7%) indicated they used the "Check-In" feature 75% of the time or less. Moreover, if participants forgot to check in, only 109 (17.4%) said they were reminded by a staff member, 450 (71.9%) indicated they were not reminded, and 67 (10.7%) used another method available to check-in. Thus despite participants more often than not using the "CheckIn" feature at venues, these results indicate that a potential increase in venues engaging with the user to utilize this feature may have further increased compliance. This can be further supported by the fact that 236 (32.1%) participants indicated that the NHS QR codes were accessible about half the time or less when visiting a venue. Furthermore, when participants were asked how many venues on average they visited a day, the majority said either zero (411, 48.3%) or one (311, 36.6%), with the rest indicating two or more (129, 15.2%). In addition to this, only 21 (2.5%) of participants indicated that they would visit a venue for more than three hours on average. However, as the venue "Check-In" feature only checks users out when they visit another venue or at midnight on the same day, this could result in a large number of users being checked into venues much longer than they were actually present. This in turn may lead to a false number of encounters when cross-referencing between app users and venues they visited, thus possibly decreasing trust and confidence in the application's ability to trace contacts accurately. Of those participants who developed symptoms, many participants (151, 67%) entered these into the symptom checker tool within the app to obtain advice on how to proceed. This indicates that this feature was well used, thus contributing to more effective triage and management of users at a large scale. Participant results outlining user values, including their expectations of the app and views on privacy, can be found in Tables S2, S3 (Supplementary materials). . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) Overall, 814 (78.9%) participants stated that they believed it is necessary for the app to protect the identities of individual users, with 531 (51.5%) participants noting that privacy in general is extremely important to them. Regarding the perceived privacy of the features, 589 (69.4%) and 613 (71.9%) participants were comfortable with the "Check-In" and the COVID19 self-report features, respectively, in regards to privacy, and 531 (82.9%) respondents who downloaded the app noted they would self-report a positive COVID-19 test in the app if they were to test positive for COVID-19. This indicates that privacy is an important value for users, and the approach of the "NHS COVID-19" application, in promoting and utilising anonymous data, has led to the vast majority of users being content with using the features it provides. Participants views on privacy, in general, were statistically analysed against various themes of the study. A statistical significance was shown between 531 (51.5%) subjects that indicated privacy was extremely important, and with subjects that did and did not download the app in this study (p < 0.0001). Of the participants that downloaded the app, 47.8% indicated privacy is extremely important to them. However this percentage increased for participants that did not download the app, with 68.4% indicating they consider privacy very important. This shows that despite the app not collecting any personal information, there may still be concerns affecting trust in the app. In addition to users' views on privacy, participants also indicated that they strongly believed this app should not be legally enforced upon users, with 567 (54.9%) participants stating they do not think they should be legally obligated to download the app. This shows that users' value the autonomy to choose whether or not they engage with nationwide automated contact tracing programs. Participants responses regarding their expectations of the App and the resulting weighted analysis can be found in Tables S4, and S5 (Supplementary materials) . Overall, over half (503, 59.7%) of the participants marked that the app met their expectations, with 340 (40.3%) stating otherwise. This was further analysed to compare the user perceived adequacy of information within the app with whether the application met the user's expectations. We found a statistically significant difference between the expectations of those who found there was enough information on how close contacts were derived and those who did not (p < 0.0001). These data indicate that if the user found the information adequate then they would be more likely to say the app met their expectations (77.7%); compared with those who thought there was not enough information (41.6%). In regards to usability, relationships were drawn between user's expectations and if the user felt the text was simple and easy to read (p < 0.0014); if they felt the app was easy to navigate through (p < 0.0016); and how they rated the overall style/interface of the app (p < 0.0015). 67.8% and 71.5% of app users whose expectations were met, felt the app was simple and easy to . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted March 8, 2021. ; https://doi.org/10.1101/2021.03.04.21252924 doi: medRxiv preprint read, and intuitive to navigate through respectively. Conversely, only 31.2% and 28.4% of users whose expectations were not met found the text on the app easy to read, and intuitive to navigate. This potentially indicates that the design and interface of the app had a sizeable effect on the user experience. Participant results including the participants views on the apps information content can be found in Tables S6, S7, S8 (Supplementary materials) . With regards to the perceived information content of the app we identified whether users felt adequate information was presented for them to garner an understanding of the app's major features. Overall a small majority of participants believed the amount of information presented for the features within the app was adequate for their needs, with 436 (51.4%) and 327 (38.5%) participants finding the information adequate to understand the "Check-In" and "Enter Test Result" features respectively. Furthermore, many participants believed they were not provided adequate information regarding how the app dictated whether they had encountered another app user with COVID-19, with 320 (37.7%) feeling the information was inadequate and 296 (34.9%) feeling the information was neither inadequate or adequate. We also enquired as to whether participants ever required information from outside sources to better understand the app's function and features; 204 (44.3%) participants responded that this was the case. The aforementioned points raise a possible concern, as a lack of information, and reliance on outside sources may increase the likelihood of misconceptions regarding the app and decrease users trust in the reliability and accuracy of its features. There was also a statistically significant difference between the expectations of users who obtained information from outside the app with those who did not (p < 0.0001). Participants who did not require external information were more likely to have found the app met their expectations (54.1%), compared with those who did have to use external sources of information (37.8%). Many statistical relationships were drawn when grouping between users that did, and did not read all the information presented in the app. It appeared that a higher percentage of participants aged 50+, totaling 89 (64.9%) participants did not read all information presented in the app (p < 0.0001) compared to the younger groups. Furthermore, more respondents that did not read all the information, also did not find the text as easy to read (p =0.0012), and the app as intuitive (p =0.0029), compared to the participants that read all available information. Lastly, a higher percentage of participants (57.4%) had a better understanding of what their current status regarding their COVID-19 exposure was that read all the information compared to those that did not (p =0.0049). . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted March 8, 2021. ; https://doi.org/10.1101/2021.03.04.21252924 doi: medRxiv preprint A relationship was drawn between users that both read all the information in the app, and are still using the app (p < 0.0001), with a slightly higher percentage of users that read all information continuing to use the app (57.7%) compared with users who did not read all the information. Ultimately, it appeared that participants that read all the information had increased trust in the app. However, for a smaller proportion of users it may not have been as straight forward and intuitive due to the interface/design of the app, and the complexity of the text. In addition to whether or not the user read all of the information within the app, we found that the text within the app scored on average a value of 68.8 on the Flesch Readability Ease test. This level is only achieved by 57% of the UK General Public [25] . This leads to the possibility that users may have not engaged fully with the information within the app due to difficulty in understanding and interpretation. Participants views regarding the perceived usability, and their overall understanding of the app can be found in Table S9 (Supplementary materials). Of the participants that downloaded the app, many were satisfied with the design aspects, as 591 (69.6%) stated the app's navigation was intuitive. Furthermore, many participants felt the style/interface was consistent across the app most of the time (377, 52.7%), and the views were generally favourable on how participants viewed the style/interface, with 377(44.4%), 269 (31.7%), and 75 (8.83%) indicating they somewhat liked, neither liked nor disliked, or somewhat disliked the interface of the app respectively. Regarding the status indicating if the participant had encountered another app user with COVID-19, 554 (65.2%) participants marked it was clear what their current status was. However the proportion of users who were not always aware of their current status was not negligible and therefore this may contribute to uncertainty regarding their exposure. In regards to power usage, only 115 (13.6%) felt the app was significantly impacting the performance of their battery. Participants responses regarding their knowledge of the app functions and weighted analysis can be found in Tables S4, S5 , and S10 (Supplementary materials). In total, only 400 (37.7%) participants believed the app does not require the user's personal information to function. Furthermore, on the multi-choice question asking what technology is used to identify close contacts, 298 (15.9%) falsely chose GPS. 887 (47.2%) and 573 (30.5%) correctly chose Bluetooth and self-check in logs respectively, which in fact are both used for detection. 197 (20.6%) participants chose "Yes" when asked if they believed the app could track them. 681 (74.8%) participants correctly indicated that venues are not provided with information when using the "Check-In" feature. Hence, there was a significant misunderstanding regarding . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted March 8, 2021. ; https://doi.org/10.1101/2021.03.04.21252924 doi: medRxiv preprint some of the key features of the app, despite explanations for each of these features in the app itself. This may reflect on the app's ability to effectively present and relay information to the user. Out of all participants, 242 (26.3%) chose all answers correctly, 226 (24.5%) answered three correctly, 168 (18.2%) answered two correctly, 188 (20.4%) answered one correctly, and 96 (10.4%) did not answer any correctly. Based on these groups, a relationship was shown between test scores, and Android vs iOS users; with a higher percentage (30.1% vs. 23.1%) of Android users scoring perfect, and a lower percentage of Android users scoring a zero (8.3% vs. 12.6%). Out of participants that did not download the app, only 16.4% of these participants answered either three or all questions correctly, compared to users who did download the app; with 57.5% of users answering either three or all answers correctly (p < 0.0016). This is likely due to users that downloaded the app simply being presented with more information compared to users that did not. However, this could also elude to the possibility that misconceptions within the population may be the reason for a reduced uptake of the app. In regards to usability, comparisons were drawn between app knowledge, and the simplicity to read the text (p < 0.0007). Out of participants that scored higher (3 or 4), 61% and 60.4% respectively, found the app's text easy to read and intuitive to navigate through. This again, seems to hint to the possibility of the app design being unable to effectively convey information to a smaller subset of participants, which may have hindered their ability to understand the fundamentals of the app. Out of all of the participants that downloaded the app, 286 (33.7%) noted that they had experienced technical issues with the app, with 208 (24.4%) of these participants further explaining their difficulties in detail. These free text responses regarding what participants considered technical issues to them were broadly classified as shown in Table 3 . It can be seen that the most common issue experienced was false exposure notifications (98, 47.1%) and loading notifications. Given how the notification system is a critical feature of the app with regards to informing users of potential contacts, failures of this system may have led to reduced confidence in the contact detection system. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted March 8, 2021 In total, 467 (45.1%) comments were submitted when asked if participants had any general suggestions on the app. 326 (69.8%) of these responses could be grouped together into eight different classification categories as seen in Table 4 . . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted March 8, 2021 Users mentioned they would benefit with additional information in relation to how a close contact is calculated, and where and how the data is being stored. Other suggestions included a more concise privacy policy with bullet points, an FAQ in the app, an in-app introductory video, and metrics pertaining to daily vaccination and COVID case updates for their specific zone. Many users lacked trust when being told they needed to self-isolate, as they argued they came in contact with a minimal amount of others, or they did not leave their house. Furthermore, users suggested the app provide information regarding the duration of the contact, and/or the venue it occurred in. Suggestions included having a counter that refreshes each day showing how many encounters the app has registered, and having in-app support with NHS staff members to provide them with more certainty regarding COVID-19 exposure. Many users were unaware of their current status, which caused uncertainty regarding whether the app was working or not. Suggestions included incorporating an indicator in the home-screen that the app has yet to identify adequate COVID-19 exposure to the user, and to its best knowledge that they are safe. Users were also uncertain whether the app was, or was not running in the background, with some users indicating the dynamic scanning icon on the home screen led them to believe the app only scans when open. Suggestions included the app provide more reminders that the app will in fact work in the background for less tech-savy users. Many users were unaware that the "NHS COVID-19" app uses the Google/Apple API, and suggested that the Google/Apple API be used to ensure privacy is preserved and to increase public trust. Many users were confused and uncertain when sent a false exposure notification, indicating that when they tapped the notification, they were not provided with more information. It was determined these notifications were sent by the Google/Apple API, however, users suggested this be addressed in the app introduction, and/or provide the user with a log of their exposure notifications. Allow for location change for users that travel to different zones 22 (7.6%) Users that work across various locations in the UK, or moved during the pandemic did not find changing their inputted postal code to be trivial, and re-downloaded the app. Suggestions include allowing users to change zones in the app. Table 4 : Summary of participant recommended additions to the "NHS COVID-19" app. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) Overall, the results demonstrated that users valued their privacy, and strongly believed that their identities should remain anonymous. To this end, many participants saw value in the privacy preserving digital contact tracing solution, with most users finding it useful, and meeting their expectations. Fair, although not optimum, compliance was noted regarding many of the apps features such as the venue "Check-In", and enabling Bluetooth contact tracing. However, in order to maximize the benefits of digital contact tracing to fight the pandemic, maximizing users uptake is crucial, and there were clear indicators that usability, information content, and misconceptions of the app may have been responsible for the lack of understanding, and overall compliance. It appeared usability was adequate for a subset of participants, as many participants that read all the information on the app found the text easy to read, and navigation to be intuitive. However, there was an inverse relationship for those that did not read all the information. This may be related to the readability score of the text only being comprehensible for roughly 57% of the population. There was a similar link with participants that found the app more usable, as they had an improved understanding of the app. Ultimately, it may be expected that by improving the overall user journey, and more effective conveyance of information suited for the whole population, overall understanding and compliance with app features may be improved. It was also clear that users that did not download the app had a poor understanding of key app features, with many believing the app uses GPS, and that it requires personal information. Although this is not clearly linked to the usability of the app, it could be linked to prior development setbacks, which may have ultimately affected public trust, or to the lack of information present in external sources such as newspapers, social media, ads, etc. Moreover, the possibility of falsely identifying contacts, with the venue "Check-In" feature was evident, as once a user checks into a venue, they are not checked out until they visit another venue, or at midnight that same day. Many participants highlighted that their typical time spent at venues was often less than an hour and the number of venues visited within an average day was close to one. Hence, this is potentially a sizeable problem. A possible recommendation to help reduce this potential pitfall is to employ a "Check-Out feature, allowing the user and/or venue to dictate when the user has left the premise, resulting in a more accurate representation of contacts encountered by the user. Another aspect of the Venue "Check-In" feature that can be improved upon is the engagement of both the venue and the user in the process of checking in. The current system relies mostly on the user to initiate the check in process, which can ultimately lead to unconscious noncompliance. By designing a system that requires both parties (venue and user) to engage with the process, compliance could be driven by either party. Lastly, it was identified from commented suggestions that false exposure notifications, and aspects of the user interface were causing uncertainty in terms of users exposure; and this played a role in users confidence in the tool if they were told to self-isolate. Although a false exposure notification bug issue was addressed in the past (well before launching this study), it was clear that the app had limited information regarding this. Going forward, many participants suggested . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted March 8, 2021. ; https://doi.org/10.1101/2021.03.04.21252924 doi: medRxiv preprint the app should provide an increased amount of information through optional FAQs, an introductory video, and increased feedback. As our work endeavored to investigate the compliance and usage of the main features and tools contained within the app, we ideally required the survey to be active at the same time as users were able to use the app outside their home. However due to the rapidly changing nature of the COVID-19 pandemic, not long after the "NHS COVID-19" app was released, the UK was placed into a 2nd and 3rd lockdown period with restrictions on movement and outside activity. We therefore tried to mitigate the effect of this on the results of our study by specifically referring users to how they would have used the app in the period between the first and second lockdown in all compliance questions involving the venue "Check-In" feature, which understandably was only able to be used during this time period. Our study also specifically focused on the "NHS COVID-19" app rolled out in England and Wales. Hence, not all of our conclusions might be applicable to other contact tracing apps rolled out around the world, not just because of their potentially different designs but also due to the different demographic characteristics of their particular geography. Despite our statistical calculation of an appropriate sample size required for the study, the study failed to attract an adequate cohort of participants from Wales. We also noticed that, as our study was advertised as an "Usability and Compliance" study, we received a significantly higher proportion of responses from people who had downloaded the app compared with those who had not. While this did not impact the usability and compliance sections of our research, a greater proportion of respondents who had not downloaded the app would have allowed us to derive stronger conclusions regarding reasons why members of the public had not installed the application, as well as the general views and beliefs this section of society holds. This work reports an independent study demonstrating how the public perceived and used different features of the "NHS COVID-19" app through analysis of an UK wide survey. The results of this study can be used to inform future contact tracing development efforts in order to optimize the adoption, compliance and overall efficacy of digital tools. This project has received no funding. All the work has been done by the authors on a voluntary basis. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted March 8, 2021. ; https://doi.org/10.1101/2021.03.04.21252924 doi: medRxiv preprint Due to restrictions in the ethics document, data beyond what is presented in this publication cannot be distributed. If any questions, please email the corresponding author (ss7719@ic.ac.uk). Daily summary -Coronavirus in the UK Feasibility of controlling COVID-19 outbreaks by isolation of cases and contacts. The Lancet Global Health COVID-19 epidemic in Switzerland: On the importance of testing, contact tracing and isolation First cases of coronavirus disease 2019 (COVID-19) in France: Surveillance, investigations and control measures Government to spend £10 billion on test and trace -Metro News Quantifying SARS-CoV-2 transmission suggests epidemic control with digital contact tracing Digital technologies in the public-health response to COVID-19 The Use of Digital Tools to Mitigate the COVID-19 Pandemic: Comparative Retrospective Study of Six Countries Contact transmission of Covid-19 in South Korea: Novel investigation techniques for tracing contacts. Osong Public Health and Research Perspectives COVID-19) Latest Updates Cases in Korea South Korea is reporting intimate details of COVID-19 cases: has it helped? China uses mass surveillance tech to fight spread of coronavirus In defence of digital contact-tracing: human rights, South Korea and Covid-19 Public attitudes towards COVID-19 contact tracing apps: A UK-based focus group study. Health Expectations. 2021 1:hex.13179 Acceptability of app-based contact tracing for COVID-19: Cross-country survey study. JMIR mHealth and uHealth GitHub -nhsx/covid19-app-system-public: COVID19 app backend GitHub -nhsx/covid-19-app-android-ag-public: COVID19 Android app NHS COVID-19 app has been downloaded over 10 million times -GOV.UK Automated and partly automated contact tracing: a systematic review to inform the control of COVID-19 UK may ditch NHS contact-tracing app for Apple and Google model -Technology -The Guardian Government's coronavirus tracing app facing questions on cyber security and privacy Criteria for assessing the quality of mHealth apps: A systematic review Consent information leaflets -readable or unreadable Skills for Life Survey: A Survey of Literacy, Numeracy and ICT Levels in England The authors would like to express their sincere gratitude to those that assisted with the deployment of the survey, including techUK, and /r/coronavirusuk. MP, and SS led data collection, and survey distribution. MP and SS analyzed the data. MP, SS and ERV wrote the manuscript. ERV led the work. The authors declare no competing interests.