key: cord-1025308-3p4bf9ik authors: Lai, Lucinda; Sato, Rintaro; He, Shuhan; Ouchi, Kei; Leiter, Richard; Thomas, Jane deLima; Lawton, Andrew; Landman, Adam B.; Zhang, Haipeng Mark title: Usage Patterns of a Web-Based Palliative Care Content Platform (PalliCOVID) during the COVID-19 Pandemic date: 2020-07-27 journal: J Pain Symptom Manage DOI: 10.1016/j.jpainsymman.2020.07.016 sha: 226c0ebf73161b16a9a8c60fca0699807b5ec809 doc_id: 1025308 cord_uid: 3p4bf9ik BACKGROUND: The COVID-19 pandemic has highlighted the essential role of palliative care to support the delivery of compassionate, goal-concordant patient care. We created the web-based application, PalliCOVID (https://pallicovid.app/), in April 2020 to provide all clinicians with convenient access to palliative care resources and support. PalliCOVID features evidence-based clinical guidelines, educational content, and institutional protocols related to palliative care for COVID-19 patients. It is a publicly available resource accessible from any mobile device or desktop computer that provides clinicians with access to palliative care guidance across a variety of care settings, including the emergency department, hospital ward, intensive care unit, and primary care practice. OBJECTIVE: The primary objective of this study was to evaluate usage patterns of PalliCOVID to understand user behavior in relation to this palliative care content platform during the period of the local peak of COVID-19 infection in Massachusetts. DESIGN: We retrospectively analyzed de-identified usage data collected by Google Analytics from the first day of PalliCOVID’s launch on April 7, 2020 until May 1, 2020, the time period that encompassed the local peak of the COVID-19 surge in Massachusetts. MEASURE: ments: User access data was collected and summarized by Google Analytics software that had been integrated into the PalliCOVID web application. RESULTS: 2,042 users accessed PalliCOVID and viewed 4,637 pages from April 7 to May 1, 2020. Users spent an average of 2 minutes and 6 seconds per session. 81% of users were first-time visitors, while the remaining 19% were return visitors. The majority of users accessed PalliCOVID from the United States (87%), with a large proportion of users coming from Boston and the surrounding cities (32% of overall users). CONCLUSIONS: PalliCOVID is one example of a scalable digital health solution that can bring palliative care resources to frontline clinicians. Analysis of PalliCOVID usage patterns has the potential to inform the improvement of the platform to better meet the needs of its user base and guide future dissemination strategies. The quantitative data presented here, although informative about user behavior, should be supplemented with future qualitative research to further define the impact of this tool and extend our ability to deliver clinical care that is compassionate, rational, and well-aligned with patients’ values and goals. The novel coronavirus (COVID-19) pandemic has strained the healthcare system beyond its usual capacity to accommodate such a large influx of seriously ill patients. 1, 2 This has created the need to bring palliative care medicine practices to the frontline across specialties. 6 Many clinicians who have not been specially trained in palliative care do not feel adequately prepared to facilitate difficult conversations with patients or their family members about goals of care. 7 Providing these clinicians with convenient access to focused education, clinical reference materials, and specialist palliative care support may improve end of life care for patients. 1 As the COVID-19 surge approached Boston in April 2020, our interdisciplinary team of emergency physicians, palliative care specialists, and digital health innovators at Partners HealthCare recognized the need to rapidly increase the capacity to deliver primary palliative care at the frontline. Partners HealthCare is a large academic integrated healthcare system that was founded by Massachusetts General Hospital and Brigham and Women's Hospital in 1994, and now consists of a core network of eleven hospitals in the New England area. Our goal was to create a centralized resource of information that would support Partners healthcare staff to provide goal-concordant care and expert symptom management to patients who were ill with suspected COVID-19 infection. Given the challenges of distributing and maintaining the quality of up-to-date information in such a fast-changing environment, we sought out a digital solution. We aimed to develop an online, centralized compendium of clinical reference materials, focused educational content, and institutional protocols intended to seamlessly integrate palliative care across a variety of care settings, including the emergency department, hospital ward, intensive care unit, and primary care practice with respect to Partners-specific resources and established protocols. We built a progressive web application (PWA) called "PalliCOVID" for the purpose of sharing digital content related to the delivery of palliative care during the COVID-19 pandemic. PWAs have previously been noted to have significant untapped potential in the field of healthcare to improve work efficiency and quality of care. 8 The main advantages of PWAs is that they provide the user with a similar experience to using a native application, but are accessed via a web browser, do not require download from an app store, can be shared via a URL link, and can be found via a web search engine. Users around the world can access PalliCOVID via web browser on any mobile or desktop device, thus allowing for wide dissemination of the digital content posted by the authors. At the same time, the authors are able to centrally manage the content and make instantaneous updates as needed, which is an important mechanism for quality control considering the rapidly changing information environment of the pandemic. The clinical guidelines and content featured on PalliCOVID were sourced, produced, and reviewed by palliative care specialists at Brigham and Women's Hospital and Dana-Farber Cancer Institute and emergency physicians from Brigham and Women's Hospital. This interdisciplinary team of nurses, physicians, and physicians assistants met weekly to discuss updates to the content. We prioritized the publication of guidelines that were succinct, specific, and commensurate with the best available scientific evidence about the COVID-19 disease process and end of life care. We established several key criteria for content posted on PalliCOVID: • Accurate: Content was reviewed by palliative care experts to reflect the best available scientific evidence • Practical: Recommendations were designed to be useful and implementable by nonpalliative care clinicians in a variety of care settings • Accessible: Content was presented in a format that was optimized for viewing on both mobile devices and desktop computer screens • Applicable: Content was specific to the care of patients with confirmed or suspected COVID-19 infection and took into account the need to limit face-to-face interactions due to enhanced infection control measures and restricted visitor policies We have included examples of PalliCOVID resources that respectively address the three major domains of palliative care expertise with regards to symptom management, goals of care discussions, and family support: 6, 9 opioid dosing recommendations for the treatment of dyspnea and pain at the end of life (Figure 1 ), a conversation guide for rapid code status determination in the peri-intubation setting (Figure 2 ), and a tip sheet for responding to difficult questions from family members of dying patients ( Figure 3 ). Although certain features of PalliCOVID are restricted to users within the Partners HealthCare System, such as one-click access to the hospital paging system, we have made the majority of the content on PalliCOVID open to public access. The primary objective of this study was to collect and analyze usage data from PalliCOVID as a way to better understand user behavior and gain insights about the population of users accessing this palliative care content platform. The secondary objective was to use the insights derived from user engagement behavior to formulate ideas about future improvements to the platform and future dissemination strategies. We retrospectively analyzed de-identified usage data that was collected by Google Analytics from the first day of PalliCOVID's launch on April 7, 2020 through May 1, 2020. Google Analytics is a free tool that provides quantitative data on website usage and has previously been used in health research for process evaluation and quality improvement. 15 This project was undertaken as a Quality Improvement Initiative at Brigham and Women's Hospital and as such was not formally supervised by the Institutional Review Board per their policies. We followed the SQUIRE guidelines for quality improvement reporting. 10 Evaluating Usage We evaluated usage data using Google Analytics (Google, LLC, Mountain View, CA), which was installed on PalliCOVID and used to track user data from April 7, 2020 through May 1, 2020). 11 This allowed collection of data related to user behavior emanating from a user's interaction with the application. The data could come from avenues such as the URLs of the pages the user viewed, the location of the user, and the type of device being used to access the application. Google Analytics also collected information about the nature of the visit such as the content viewed and length of the session. Google Analytics did not collect any personally identifiable information and presented all collected data in aggregate form, thus mitigating the ethical concerns that could arise from user behavior research. 12, 13 We evaluated user engagement by examining "sessions". A session was defined as a series of interactions by a user that took place within a predetermined time frame (30 minutes), which represented the period of time that the user was actively engaged with the application. The number of returning users referred to the number of sessions visited by the same client ID. A relatively high proportion of returning users has been noted in web-based mental health intervention research to be a marker of user engagement. 15, 17, 18 The number of pages per session refers to the number of webpages within the platform that the user viewed in a single session, and the mean session duration referred to the mean duration of time that users spent on the platform (reported in minutes and seconds). This kind of user traffic data provides an approximation of the degree of exposure that users had to the content being hosted on the platform. 15 Graphics demonstrating the geographic distribution of users were created by Google Analytics. Tables demonstrating the distribution of sessions, visits, most viewed pages, and device types were created using Microsoft Excel, Version 16.36 (Microsoft Corporation, Seattle, WA). There was a total of 2,042 users and 4,637 page views during the period of April 7, 2020 to May 1, 2020. 81% of users were first-time visitors to the application, with the remaining 19% representing return visitors ( Figure 4 ). 75% of sessions were accessed by first-time or secondtime visitors to the application (Table 1) . Users spent an average of 2 minutes and 6 seconds per session on PalliCOVID, with 77% of sessions lasting less than one minute (Table 2 ). The most viewed page was the home page (96.5% of total page views), followed by Quick Guides to Symptom Management (an example of which is included as Figure 1 ), the page with indexed links to COVIDprotocols.org, the section on Emergency Department resources, and the collection of videos simulating different types of difficult conversations (Table 3) . Almost all users used either mobile devices (56%) or desktop computers (42%) to access the application (Table 4 ). Only 2% of users accessed PalliCOVID by tablet device. The majority of users (87%) accessed PalliCOVID from the United States, with 32% of all users located in Boston and the surrounding cities. The remainder of users accessed PalliCOVID from 29 other countries outside of the U.S. In the response to the large influx of seriously ill patients during the peak period of the COVID-19 pandemic in Boston, PalliCOVID was created as a scalable digital solution to increase access to palliative care educational resources and specialist support. Analyzing user engagement behavior, we found that three-quarters of sessions (75%) consisted of first-time and second-time visitors to the application. The remaining 25% of sessions represented usage by "frequent users," defined as those who have visited the application three or more times. A previous study on usage patterns of a mobile palliative care application found that a small subset of users (13% of users) comprised the majority of all activity in the application (66% of activity). 14 One possible explanation for this distribution are frequent users who coming back to the application as a "just in time" clinical reference tool, whereas infrequent users were accessing the application for more passive educational purposes. 14 PalliCOVID content features both types of information: "just in time" clinical reference material such as medication dosing recommendations, as well as focused educational content such as difficult conversation simulation videos, so this may explain the different distribution of frequent versus infrequent users. Furthermore, we limited our data collection period to less than four weeks of operation, which may not be enough time to observe trends in users returning to the application. Average duration of sessions on PalliCOVID was short, at less than 2 minutes per session for mobile users and less than 3 minutes per session for desktop users. Some may interpret the short session duration to indicate a degree of user disengagement, but an alternative explanation may be that users were able to quickly find what they were looking for. Sessions in which users access the link to the hospital paging directory from the PalliCOVID home page, for example, would cause the user to exit the application and end the session relatively quickly. In that case, a short session duration would have indicated that PalliCOVID had successfully carried out a function that it was designed to perform. The platform can be improved based on user behavior. Future iterations of the application will focus on improving user engagement with the content pages for "Nursing Resources" and "Hospice Care Referrals", which did not rank among the top five most-viewed pages. The lower ranking of these pages may be a reflection of the original dissemination strategy, which focused more heavily on reaching physicians and advanced practice providers (APP) rather than the nursing staff. Future dissemination strategies should specifically target nurses or have product design specifications to specific user populations as well. Population and user specific design may be critical as different roles such as nursing spend the majority of their time at patients' bedsides and are more likely to be aware of patients' specific symptoms and end of life needs than perhaps physicians or APPs. There was an overall even split in device types between mobile and desktop, arguing against a stronger user preference for one type of device over the other. As we add future content to the platform, we should continue to optimize viewing for both smaller screens and larger screens with vector graphics, responsive web design, and screen size flexibility. When observing the location of the users, 32% of users were found to be from Boston and surrounding cities. This finding aligns with our implementation science focus on engaging users about the platform primarily within the Partners institutions. In addition, a subset of the content was Partners institution-specific and paging directory access was restricted to members of the Partners network. Future reorganization of the content into internal-and external-facing sections may make the application more inviting to users outside of the Partners enterprise. The majority of the clinical content contained within the guides would be applicable to patient care regardless of the particular institution, so future improvements to the platform may include better sign-posting of content as broadly applicable and not Partners specific. Although Google Analytics has provided promising data about the usage patterns of PalliCOVID, the data should be interpreted cautiously. Google Analytics collects and presents data with a marketing and e-commerce framework for understanding web-based behavior, rather than a health research frame. Aggregation of the data, in addition to potential discrepancies in how users are counted when users delete browser cookies, switch devices, or create a new client ID by changing browsers, creates bias in the data. Nonetheless, we believe there is still value in tracking overall engagement numbers and observing trends in usage over time in order to continually evaluate and improve the performance of the platform. As the volume of patients with COVID-19 infection rises and falls through different phases of the pandemic, the need for a digital solution such as PalliCOVID may also fluctuate over time. 15, 16 Because our team focused our efforts on the development of a resource that would be maximally useful to clinicians within the Partners HealthCare System, the sample size was limited to its distribution within this institution and geographic area. We considered uptake of the resource outside of the Partners enterprise to be a secondary benefit. Future studies should include a more detailed analysis of longitudinal user data or a mixed method assessment with focus groups, surveys, or in-depth interviews, to supplement Google Analytics data and allow for a more comprehensive quality improvement study. This study described usage patterns of a web-based palliative care content platform over its first month of operation in April 2020. We have demonstrated that, even in the midst of a global pandemic, it is possible to rapidly design and implement a digital solution in response to an unprecedented healthcare challenge. We have also demonstrated the use of a free, open-access tool such as Google Analytics to evaluate patterns of user behavior, consequences of the dissemination strategy, and aspects of the platform that may be amenable to future improvements. Quantitative data should be combined with qualitative research to provide more accurate interpretations of user behavior. We hope that clinicians who access the content on PalliCOVID may feel empowered to deliver care that is focused on dignity, symptom control, and avoidance of unnecessarily invasive or non-beneficial interventions whenever possible. Goal: Make rapid, patient-centered intubation recommendations for patients who may be at high-risk for poor outcomes. After establishing that advance directive does not exist, complete the following steps: In 24 to 48 hours, ICU team will discuss with you how (your mother) is responding to treatment. Can she still hear us? Encourage family (even if by phone) to talk with patient as they normally would. We think so, and many believe that hearing is the last sense people retain. I'd encourage you to talk to her as you normally would. The Importance of Addressing Advance Care Planning and Decisions About Do-Not-Resuscitate Orders During Novel Coronavirus 2019 (COVID-19) Creating a Palliative Care Inpatient Response Plan for COVID19 -The UW Medicine Experience What matters most in end-of-life care: perceptions of seriously ill patients and their family members Should Doctors Withhold Ventilator? The New York Times Fair Allocation of Scarce Medical Resources in the Time of Covid-19 Pandemic palliative care: beyond ventilators and saving lives Emergency Medicine Physicians' Perspectives of Providing Palliative Care in an Emergency Department Towards PWA in Healthcare Barriers to Access to Palliative Care. Palliative Care: Research and Treatment SQUIRE 2.0 ( Standards for QUality Improvement Reporting Excellence) : revised publication guidelines from a detailed consensus process: Table 1 A Process Evaluation of a Web-Based Mental Health Portal (WalkAlong) Using Google Analytics Evaluating information seeking and use in the changing virtual world: the emerging role of Google Analytics An objective approach to evaluating an internet-delivered genetics education resource developed for nurses: using Google Analytics TM to monitor global visitor engagement Usage Patterns of a Mobile Palliative Care Application Response and role of palliative care during the COVID-19 pandemic: A national telephone survey of hospices in Italy Palliative Care Pandemic Pack: a Specialist Palliative Care Service response to planning the COVID-19 pandemic