key: cord-0852332-dtlky5yv
authors: Helou, R. I.; Waltmans–den Breejen, C. M.; Severin, J. A.; Hulscher, M. E. J. L.; Verbon, A.
title: Use of a smartphone app to inform healthcare workers of hospital policy during a pandemic such as COVID-19: A mixed methods observational study
date: 2022-01-05
journal: PLoS One
DOI: 10.1371/journal.pone.0262105
sha: cd9f99755c3aa1a14209d99c19048a141295f2c4
doc_id: 852332
cord_uid: dtlky5yv

OBJECTIVE: To evaluate the use of a COVID-19 app containing relevant information for healthcare workers (HCWs) in hospitals and to determine user experience. METHODS: A smartphone app (Firstline) was adapted to exclusively contain local COVID-19 policy documents and treatment protocols. This COVID-19 app was offered to all HCWs of a 900-bed tertiary care hospital. App use was evaluated with user analytics and user experience in an online questionnaire. RESULTS: A total number of 1168 HCWs subscribed to the COVID-19 app which was used 3903 times with an average of 1 minute and 20 seconds per session during a three-month period. The number of active users peaked in April 2020 with 1017 users. Users included medical specialists (22.3%), residents (16.5%), nurses (22.2%), management (6.2%) and other (26.5%). Information for HCWs such as when to test for SARS-CoV-2 (1214), latest updates (1181), the COVID-19 telephone list (418) and the SARS-CoV-2 / COVID-19 guideline (280) were the most frequently accessed advice. Seventy-one users with a mean age of 46.1 years from 19 different departments completed the questionnaire. Respondents considered the COVID-19 app clear (54/59; 92%), easy-to-use (46/55; 84%), fast (46/52; 88%), useful (52/56; 93%), and had faith in the information (58/70; 83%). The COVID-19 app was used to quickly look up something (43/68; 63%), when no computer was available (15/68; 22%), look up / dial COVID-related phone numbers (15/68; 22%) or when walking from A to B (11/68; 16%). Few respondents felt app use cost time (5/68; 7%). CONCLUSIONS: Our COVID-19 app proved to be a relatively simple yet innovative tool that was used by HCWs from all disciplines involved in taking care of COVID-19 patients. The up-to-date app was used for different topics and had high user satisfaction amongst questionnaire respondents. An app with local hospital policy could be an invaluable tool during a pandemic.

a1111111111 a1111111111 a1111111111 a1111111111 a1111111111

In the first months of the COVID-19 pandemic, and even more thereafter, an overwhelming number of publications appeared on COVID-19. Especially in the first months, clinically relevant new findings were published on a daily basis while healthcare workers (HCWs) and hospital management struggled with an enormous influx of COVID-19 patients in the hospitals. Guidelines which assist HCWs in clinical decision making have shown to have a beneficial effect on therapy, length of hospital stay, mortality and overall patient safety [1, 2] . However, for COVID-19, these had to be changed regularly and often on a daily basis. Since clear, centrally directed communication during a healthcare crisis is key, hospitals should ensure correct and current policy is disseminated in a concise manner to all HCWs [3, 4] . HCWs have to be able to rely on the latest information provided, thereby ensuring good quality outbreak response.

Information technology such as clinical decision support systems (CDSS) can positively impact healthcare by facilitating diagnostic [5] , therapeutic [6] and prognostic [7] processes. CDSS is software that facilitates clinical decision making, either by comparing individual patient characteristics with a clinical knowledge database or by providing relevant information which is then combined with the knowledge of the clinician [8] . Such systems also have shown value in the COVID-19 outbreak management setting [9] [10] [11] . A multitude of smartphone applications (apps) for patients have been developed and evaluated during the COVID-19 pandemic with focus on contact tracing [12, 13] , self-testing [14] , self-reporting [15] and education [16] . While these apps can help inform the public, professionals, and policy-makers, they are dependent on user uptake and continued engagement. In addition, ethical issues must be addressed to enable professionals and policy-makers to effectively use the data collected while protecting the privacy of app users. Few apps specifically for HCWs, some including CDSS, have been studied and these were limited to instruction videos [17] , policy documents [18] or the addition of COVID-19 content to an existing platform containing an array of clinical medical content [19] . Although apps may have high potential to distribute the latest vital hospital information and policy on COVID-19 amongst HCWs, few studies evaluated the use of such apps focusing solely on COVID-19. The aim of this study is to evaluate the use of a COVID-19 app containing relevant information (such as local infection prevention guidelines, clinical guidelines, instructions on the use of personal protective equipment (PPE) and frequently asked questions) for HCWs in hospitals and to determine user experience.

This study was conducted in the Erasmus MC University Medical Center (Erasmus MC), Rotterdam, The Netherlands, an academic, tertiary care hospital with approximately 900 beds and 14,359 HCWs including 2,512 nurses and 935 medical specialists. The overall number of saturation was reached. The interview questions were based on the Flottorp framework, which is designed to evaluate barriers and facilitators of an intervention [22] . For analysis of the interviews the software Atlas.ti was used. Based on the Flottorp framework, the findings of the interviews and additional literature, the current survey was developed. The preliminary data has not yet been published as it is part of the abovementioned trial. Questions of the COVID-19 app survey focused on content and app use in general, especially during a busy period caused by SARS-CoV-2 / COVID-19 (Tables 3, 4 and 5, S1 Appendix).

During the installation process of the app, users were informed that user analytics would be collected and anonymously analyzed on group level. Additionally, users were asked to select their role in the hospital. Data of process indicators was stored in Data Studio, Google. Responses on the questionnaire were stored in LimeSurvey which was hosted on secure, password-protected hospital servers. In the questionnaire basic demographic information such as age, gender, role and department was asked. Data obtained from the questionnaire was collected and analyzed anonymously. Descriptive analyzes and Fisher's exact test to assess questionnaire responses association were performed using IBM SPSS Statistics 25.0 for Windows (IBM, Armonk, NY, USA).

Ethical approval for this study was obtained from the institutional review board of the Erasmus MC, Rotterdam, The Netherlands, number MEC-2020-0406. All data was anonymized.

The first COVID-19 patient was admitted to our hospital at the end of February 2020. After a peak of admissions in (mainly the second half of) March 2020 with 135 patients with a positive SARS-CoV-2 PCR, the number of patients with a positive SARS-CoV-2 PCR declined in April (111), May (20) , June (5) until July 8 (1). In total, 272 patients with a positive SARS-CoV-2 PCR were admitted until July 8, 2020. Similarly, on a national level COVID-19 admissions in February 2020 were few (9) , peaked in March 2020 (6373) and decreased in the following months (4845 in April, 904 in May, 205 in June and 166 in July) [26] .

The COVID-19 app was introduced on March 30, 2020. From the introduction until the end of study app content was updated 156 times. The total number of users from March 30, 2020 to July 8, 2020 was 1,168 of the hospital's 14,359 HCWs (8.1%) which included 27.9% of all medical specialists and 10.3% of all nurses. COVID-19 app users were predominantly in the role of medical specialist (22.3%), nurse (22.2%), resident (16.5%), management (6.2%), medical student (4.0%), fellow (2.1%), infection prevention expert (0.4%) and other (26.5%) (Fig 1,  S2 Appendix) . The group 'other' consisted primarily of technicians (such as radiologic, laboratory, operating room, medical, IT, anesthetic, maintenance and pharmacy technicians) (38.4%), administrative professionals (30.8%), clinicians (16.1%), and research and education professionals (9.0%). The number of active users varied during the study period from 128 (March), 1017 (April), 594 (May), 487 (June) to 368 (July). The total number of sessions was 3903 with peak accessions at nine o'clock in the morning (Fig 2, S1 Fig) . Nurses had the highest number of sessions throughout the day ( Table 1) . The average session time was 1 minute and 20 seconds and most users used the app for 2-4 sessions during the study period (Table 2) .

Overall, the most frequently accessed advice was information for HCWs such as when to test for SARS-CoV-2 (1214), latest updates (1181) and the COVID-19 telephone list (418). An advice can contain several screens with information. The most accessed screens were HCWs with possible COVID-19 symptoms (596), no / unknown unprotected physical contact with COVID-19 patients (472) and HCW without fever (461) (S3 Appendix). Among nurses, medical specialists and residents the most frequently accessed advice was information for HCWs such as when to test for SARS-CoV-2, latest updates and information regarding patients.

The initial app push message to invite users to respond to the questionnaire yielded 11 respondents. Three subsequent e-mails were sent as a reminder and yielded an additional 24, 22 and 14 respondents, respectively.

Of the COVID-19 app users, 71 users with a mean age of 46.1 years from 19 different departments completed the questionnaire of which three users partially completed the questionnaire. Most respondents were female (53/71; 75%), and nearly all respondents agreed that use of smartphone apps can contribute to their work (68/71; 96%) and indicated to already use different apps for their work on a daily (21/71; 30%) or weekly (18/71; 25%) basis (Table 3) .

Questionnaire respondents considered the COVID-19 app clear, easy-to-use, fast, and useful (Table 4 ). One percent (1/70) of the respondents did not trust the app.

The most frequent reasons to use the app were checking the latest SARS-CoV-2 / COVID-19 updates, infection prevention and control policy, looking up/dial telephone numbers and (7) Medical student (5) Physician assistant (35) Fellow (6) Physician assistant (4) Infection prevention expert (3) Infection prevention expert (2) Infection prevention expert (1) https://doi.org/10.1371/journal.pone.0262105.t001 

Thoughts on smartphone app use for work, n (%)

Apps can contribute to my work 68 (96) Apps don't contribute to my work 1 (1) No opinion 2 (3)

A few times a month 18 (25) A few times a week 18 (25) Every day 21 (30) I don't use smartphone apps for work 14 (20) Familiarity with advice in COVID-19 app a , n (%) (8) Resident (75) Resident (21) Resident (4) Management (26) Management (10) Management (3) Medical student (24) Fellow (5) Fellow (1) Physician assistant (8) Physician assistant (4) Infection prevention expert (0) Fellow (6) Medical student (3) Medical student (0) Infection prevention expert (1 (Table 5) . Eighty-seven percent (62/71) of all questionnaire respondents thought that app use was moderately or fully culturally accepted in their work environment. However, none of the nurses in age group 20 to 29 (0/5) felt app use was fully accepted compared to 76% (13/17) of nurses in the other age groups. Age group was associated with the perception that app use was culturally accepted in the work environment (p<0.01). For example, most respondents in the age group 20-29 felt this was moderately accepted, while the majority of respondents of all other age groups felt it was fully accepted (S4 Appendix). Additionally, gender was associated with the perception that app use was culturally accepted in the work environment (p<0.01). None of the female respondents felt it was not accepted. Contrarily, male respondents did feel this way (S3 Appendix). There was a relationship between role and whether respondents would use the COVID-19 app adjacent to patients (p<0.01). None of the residents found this unprofessional, although medical specialists did (S4 Appendix). App use adjacent to patients was no problem for half of the respondents (36/68; 53%), a quarter had doubts to do so (19/68; 28%) while others, predominantly nurses (7/8), felt it was not right (8/68; 12%) and some thought it was unprofessional (5/68; 7%). Colleagues had no negative influence on app use and most respondents would (38/68; 56%) or might (27/68; 40%) recommend the COVID-19 app to their peers. Only 3% (2/68) of the respondents would not use a similar app with information and guidelines if it was to be introduced in our hospital in the future.

Here, we demonstrated that a smartphone app can be a very useful tool to disseminate up-todate information and hospital policy to HCWs during a healthcare crisis such as the COVID- 19 pandemic. Our COVID-19 app was implemented in a short period of time, was adopted easily by HCWs and cost little. Over the course of a 3⅓ month period, even after the peak of admitted patients, the app had 1168 users and 3903 unique sessions. The app was primarily used to access hospital policy regarding information for HCWs, such as when to test for SARS-CoV-2, latest updates, the SARS-CoV-2 / COVID-19 guideline and the COVID-19 telephone list. Considering the novelty of the virus and disease and changing insights it is not surprising information for HCWs regarding when to test for SARS-CoV-2 was accessed the most. Respondents of the questionnaire reflected the female:male ratio (75:25) of HCWs in our hospital (72: 28) . Respondents were positive about the influence apps had on their work and experienced an overall high user satisfaction using the COVID-19 app. One of the advantages was that the app can be used to quickly check information, while 'on the go' or in meetings, and predominantly had no effect on time or saved the user time. Additionally, previous non-users Table 5 . COVID-19 app use (N = 71). In what situation did you use the app, n (%)

In meetings to look up advice 7 (10)

When walking from A to B 11 (16) When no computer available 15 (22) Slow computer 2 (3)

To look up / dial phone numbers 15 (22) To quickly check something 43 (63) Other 7 (10)

App use saved me time 20 (29) App use neither saved me nor cost me time 43 (63)

App use cost me time 5 (7) https://doi.org/10.1371/journal.pone.0262105.t005

of apps seem to have a positive attitude towards this new technology for their work and are willing to adapt apps like the COVID-19 app. Three months after its launch our app was downloaded by 1168 users, which is more than most other studies have reported for apps containing hospital policy. The dynamic work situation during the COVID-19 outbreak could explain the relatively high number of downloads of our app. The number of total app downloads in other studies varied from 53 times after 3 months, 1233 times in 4 months, 2013 times after 10 months, 990 times after 12 months to 3056 times after 14 months [19, 23, 25, 27, 28] . However, total app sessions (3903) seemed below average compared to other studies which ranged from 9259 sessions over 14 months, >16,000 sessions over 10 months to 18,860 sessions over 12 months and could be due to the short study period and novelty of the tool within the hospital [25, 28, 29] . Additionally, the app was introduced after the peak in COVID-19 admissions in the hospital and showed a corresponding decline in monthly use, similar to another study [16] . One study that was conducted during the start of the pandemic saw an increase in users, whereas our study conducted after the peak saw a decrease in COVID-19 app users, similar to the decrease in accessing COVID-19 related content in another study [18, 19] . We did not consider or intend the COVID-19 app to replace communication of the hospital policy via the intranet. However, since smartphones and apps are used extensively we aimed to augment the main channel of communication to increase the chance that SARS-CoV-2 / COVID-19 hospital policy reached HCWs. An article on the COVID-19 app was published on the intranet. Similar to survey respondents of our COVID-19 app, other studies found HCWs experienced their studied app as positive, easy and useful [17, 18] .

Due to the very frequent updates in the COVID-19 app content we decided not to send push notifications to inform users of an update. This may have been a missed opportunity to engage more users as shown in another study [19] . Adherence to the COVID-19 content information in the app as well as the impact of the app on outcomes in patient care remain unclear and could be the focus of future research [18, 19] .

Our COVID-19 app was used by all kind of HCWs. Indeed, hospitals are not only staffed with clinical workers such as physicians and nurses but also with other HCWs including technicians, administrative professionals and research and education professionals. During the COVID-19 crisis extra effort was asked of every HCW. Many HCWs got their COVID-19 information, such as risk of acquiring and transmitting SARS-CoV-2, from social media which may lead to poorer understanding or misunderstanding on several topics [30, 31] . Additionally, unclear and frequently changing guidelines can cause confusion amongst HCWs [32] . Therefore, hospital management needs to ensure that all HCWs receive accurate and up-todate information in a timely fashion. In this technological era a smartphone app is an accessible, easy-to-use and an universal tool to facilitate dissemination of information.

Remarkably, none of the nurses in the youngest age category felt app use was fully accepted in their work environment contrary to the older age groups. This may be associated with a junior position of these HCWs who might worry that others perceive their smartphone use as not work-related [33] . Age group and gender were associated with the perception that app use was culturally accepted in the work environment. Additionally, a new group of app users who previously did not use apps for their work were enthusiastic to use our app during the COVID-19 crisis (13/71; 18%). Most would use or consider use of a similar future app which illustrates the willingness to adapt a new way of working in order to combat a new, not fully understood virus and disease.

Most questionnaire respondents felt app use for work was somewhat (12/71; 17%) or fully (50/71; 70%) accepted in their work environment and did not think it would be a problem to use the app adjacent to patients if necessary. Role was associated whether respondents would use the COVID-19 app adjacent to patients. Seven percent (5/68) of the respondents considered app use adjacent to patients unprofessional and 12% (8/68) stated that it did not feel right. Although we do not know the underlying reasons for this, as this was not explored, there are several possibilities. Mobile phones are known to be contaminated with a variety of pathogens [34] . Mobile phones in the hospital should therefore be used with application of basic infection prevention measures, but this may not be known by HCWs which might have made them reluctant to use the app. Additionally, HCWs might think they give the impression using their phone for leisure in front of the patient. Similarly, other studies showed that HCWs may feel uncomfortable (20%) using an app near colleagues or patients [23, 33] . Increasing awareness of the use of smartphone apps as medical tools remains necessary to allow them reaching their full potential in healthcare institutions.

A strength of this study was that we evaluated the use of a relatively novel tool in a hospital during a healthcare crisis, which has been studied very limited. A possible limitation is the short period of this study which started after the peak and during the decline of COVID-19 in our hospital, and this has to be weighed against the corresponding onboarding, plateau phase and decline in app use. HCWs also may have been already familiarized with hospital's intranet as the go-to source of SARS-CoV-2 / COVID-19 information at the start of the study. Data on how often hyperlinks to source documents were clicked were not available, and we could not determine if app users had any interest in the hyperlinks. Additionally, the questionnaire used in this pilot study was based on preliminary data from an ongoing study. A limited number of app users from all categories of HCWs participated in the questionnaire and users with a positive experience might be more inclined to do so, leading to a bias in appreciation of the app.

In a healthcare crisis HCWs need to be able to access an up-to-date hospital policy at all times in order to fulfill their jobs effectively and correctly. Nowadays, most HCWs carry a smartphone providing an ideal medium to receive critical information through apps during a pandemic such as COVID-19. Short implementation time makes it an excellent hospital resource to have in dire times. Our COVID-19 app proved to be a relatively simple yet innovative tool that was used by HCWs from all disciplines involved in care of COVID-19 patients. The upto-date app was used for different topics and had a high user satisfaction amongst questionnaire respondents. The short implementation time makes it an attractive tool to disseminate information on topics such as infection prevention and control, use of PPE and patient care. An app with local hospital policy can be a invaluable tool during a pandemic. 

Adherence to guidelines for hospitalized community-acquired pneumonia over time and its impact on health outcomes and mortality

Excess mortality and guideline-indicated care following non-ST-elevation myocardial infarction

Which recommendations are considered essential for outbreak preparedness by first responders?

Selection of key recommendations for quality indicators describing good quality outbreak response

Clinical Decision Support for Recognition and Management of Hypertension: A Randomized Trial

Clinical Decision Support Systems and Their Role in Antibiotic Stewardship: a Systematic Review

Impact of a clinical decision support tool on prediction of progression in early-stage dementia: a prospective validation study

An overview of clinical decision support systems: benefits, risks, and strategies for success

Rapid response to COVID-19: health informatics support for outbreak management in an academic health system

Clinical decision support tool and rapid point-of-care platform for determining disease severity in patients with COVID-19

A COVID-19 Risk Assessment Decision Support System for General Practitioners: Design and Development Study

Stemming the flow: how much can the Australian smartphone app help to control COVID-19? Public Health Res Pract

A New System for Surveillance and Digital Contact Tracing for COVID-19: Spatiotemporal Reporting Over Network and GPS

Smartphone-Based Self-Testing of COVID-19 Using Breathing Sounds. Telemed J E Health

Real-time tracking of selfreported symptoms to predict potential COVID-19

Using eHealth to Support COVID-19 Education, Self-Assessment, and Symptom Monitoring in the Netherlands: Observational Study

Increasing global awareness of timely COVID-19 healthcare guidelines through FPV training tutorials: Portable public health crises teaching method

A Mobile Health Platform to Disseminate Validated Institutional Measurements During the COVID-19 Outbreak: Utilization-Focused Evaluation Study

Using mHealth to Increase the Reach of Local Guidance to Health Professionals as Part of an Institutional Response Plan to the COVID-19 Outbreak: Usage Analysis Study

Study protocol for an international, multicentre stepped-wedge cluster randomised trial to evaluate the impact of a digital antimicrobial stewardship smartphone application

A checklist for identifying determinants of practice: a systematic review and synthesis of frameworks and taxonomies of factors that prevent or enable improvements in healthcare professional practice

An analysis of the development and implementation of a smartphone application for the delivery of antimicrobial prescribing policy: Lessons learnt

A mixed methods pilot study to investigate the impact of a hospital-specific iPhone application (iTreat) within a British junior doctor cohort

Attitudes and behaviours to antimicrobial prescribing following introduction of a smartphone App

COVID-19 development in graphs 2021

Impact of a smartphone app on prescriber adherence to antibiotic guidelines in adult patients with community acquired pneumonia or urinary tract infections

Implementation of a Mobile Clinical Decision Support Application to Augment Local Antimicrobial Stewardship

Implementation of an institution-specific antimicrobial stewardship smartphone application

YouTube as a source of information on COVID-19: a pandemic of misinformation?

Knowledge and Perceptions of COVID-19 Among Health Care Workers: Cross-Sectional Study

Working experiences of nurses during the Middle East respiratory syndrome outbreak

The role of the smartphone in the transition from medical student to foundation trainee: a qualitative interview and focus group study

Bacterial flora on cell phones of health care providers in a teaching institution