key: cord-0047975-mhs5y2dp
authors: Guney, Senem; Daniels, Chrissy; Childers, Zach
title: Using AI to Understand the Patient Voice During the Covid-19 Pandemic
date: 2020-04-30
journal: NEJM Catal Innov Care Deliv
DOI: 10.1056/cat.20.0103
sha: 7cd0da531fc334f03e415dfd9c940b6f8959b31a
doc_id: 47975
cord_uid: mhs5y2dp

Responses to Press Ganey patient experience surveys show increasing numbers of comments related to Covid-19. Patients express concerns through primarily negative comments regarding cleanliness and logistics regarding tests and treatments. The artificial intelligence–supported analysis of unstructured text reveals that despite the persistent negative trend in the patients’ experiences of receiving tests and treatments, patients are commenting increasingly positively about their interactions with clinicians as the Covid-19 pandemic continues.

Much of what is being reported about Covid-19 is from the 10,000-foot view (e.g., rate of rise of diagnosed cases) or from administrative data (e.g., percentage of patients requiring ICU care). A different type of insight can be gained from patients themselves through the comments they include on surveys about their care experience. When patients write comments, they describe every aspect of care that matters to them, regardless of whether it is the focus of a survey question. Artificial intelligence (AI) is a valuable tool for analyzing these comments to identify positive and negative trends in patients' perceptions of care as well as important themes that can help providers best meet patients' evolving needs. 1 -4 For this analysis, we used NarrativeDx (a Press Ganey Solution) -the AI platform developed specifically to analyze patient experience comments. This technology categorizes comments and detects positive and negative sentiment in unstructured text at the phrase level, based on sentence structure and state-of-the-art machine learning algorithms that are trained on annotations by native speakers of English. This design principle would allow the AI to detect the sentiment in a comment like "The doctors were kind, I would go back" as positive; and it would identify the sentiment from the same words used in a different sentence structure such as "Were the doctors kind, I would go back" as negative.

Using this technology, we studied Covid-19-related comments from patient experience surveys from February 1 to April 4, 2020. These comments referenced Covid-19, regardless of diagnosis code, and were collected as part of routine patient experience surveys administered by Press Ganey for hospitals and other health care providers across the United States. Total number of comments collected in Press Ganey surveys in the same time period was 7,039,094. Because the AI platform can pull multiple insights per comment, our analysis of 34,552 comments referencing Covid-19 generated 124,135 insights (54,325 positive and 69,810 negative), enabling the following observations: the frequency of Covid-19-related comments increased rapidly; a higher proportion of the Covid-19-related comments was classified as negative, compared with non-Covid-19-related comments collected during the study period; patients communicated negative perceptions about test and treatment logistics; patients voiced concerns about the cleanliness of the care setting and the hygiene practices of care providers; and patients voiced awareness and gratitude for caregiver and staff professionalism under these challenging circumstances.

From February 1 to April 4, 2020, we saw a continuous increase, not only in the number of comments, but also in the number of locations across the United States where patients talked about their experiences related to Covid-19. Not surprisingly, early comments were mainly from patients in Washington state5 and in New York, Boston, and other cities with high volumes of early cases, but they became more ubiquitous as infections and anxiety spread.

Attentiveness, kindness, and empathy 'in the midst of a very stressful time' were recurring aspects of positive care experiences."

Historically, higher ratios of positive to negative insights across all categories have been identified in patient experience comments using the same AI platform. The platform also has a benchmark based on the expected range of positive insights for general patient experience comments. Ratios of positive to negative insights that fall out of this range -either by exceeding or falling below itrepresent a noteworthy signal in the comment data.

More than half of the Covid-19-related comments (52.3%) were classified as negative during the study period. In particular, a negative trend in patients' experiences with tests and treatments began to spike in early March 2020 and was 77.3% negative through the 9-week period. We also observed increases over this period in negative comments about cleanliness (63.5%) and administration logistics, which addresses matters related to billing and insurance (59.7%), although there was a lower volume of comments on these topics compared with the tests and treatments " category ( Figure 1 ). During the study period, positive comments about tests and treatments as well as cleanliness fell noticeably below the platform's benchmark.

There were no regional differences in the distribution of positive and negative insights from patients' comments during the study period. The largest share of negative insights related Covid-19 came from patients age 18 to 34 years (69.6% of insights from this demographic group were negative). Patients age 80 years and older gave the largest share of positive insights about their experiences related to Covid-19 (42.9% of insights from this demographic group were positive). The younger demographic groups complained about wait times and reliability of services, which drove the negative trend in their comments. Comments by the older patients mentioned appreciation for the caregivers. For the older age group, positive interactions with caregivers appeared to matter more than negative issues they might have encountered in their care.

Patients commonly talked about potential exposure to infection." Across all age groups, the analysis of positive comments showed patients' appreciation of the courtesy and respectfulness of physicians and staff "under the circumstances of Covid-19." For " example, patients took note of physicians' explanations for why they were not able to shake hands under these circumstances. Patients also remarked on the reception staff 's professionalism in instances where the staff directed them to areas of the waiting room with "fewer patients coughing." Attentiveness, kindness, and empathy "in the midst of a very stressful time" were recurring aspects of positive care experiences. A physician's apology for the extreme delay and the staff making special parking arrangements for a patient contributed to positive perceptions regarding the reliability and scheduling of care delivery during this pandemic (Figure 2 ).

This analysis showed a greater ratio of negative to positive insights for cleanliness as well as tests and treatments than what would be expected for both categories (based on the platform's benchmark). When patients talked about cleanliness, their largely negative comments described the staff 's handwashing, the availability of masks, hand sanitizers, and wipes, as well as how certain objects in the care setting posed threats to hygiene, such as "nasty germ-covered iPads defeat[ing] the purpose of trying to be hygienic in the first place."

The largest volume of negative insights was about tests and treatments (16,258 negative out of 21,043 total in that category). Complaints included the time patients waited to be seen by a physician; how the physician had to fight for the patient to get a Covid-19 test; wait times for different types of services, such as ultrasounds and X-rays; and the lack of attention to hygiene by the staff delivering tests and treatments (Figure 3) . Patients commonly talked about potential exposure to infection -for example, by roommates in patient rooms who "coughed on [them] for 6 days."

Our analysis indicates that patients are very much aware of the constraints on health care providers due to the pandemic and appreciate the courtesy and respect of providers as they face increased questions and anxieties. Further, patients' frustrations with the processes of care delivery, such as increased delays and unavailability of tests, do not appear to extend to the staff delivering care. Patients observe the strain produced by the extraordinary circumstances and express their appreciation for the care they receive in such comments as, "The office staff and [doctor] look so extremely tired and worn out due to [Covid-19] but they all were kind and explained everything to the fullest." (Figure 4 , Figure 5 )

What is noteworthy in the data is the positive trend going up for both staff and doctors in the middle of March 2020 as the intensity of the pandemic increases. This finding may indicate the caregivers' adjusting to the "new normal" in their interactions with patients. It also shows the patients' growing appreciation of the work the caregivers are doing against the magnitude of the pandemic.

Comments as data provide insights into the areas of experience that matter to patients. These data also give us windows into what a specific area of experience, such as cleanliness, means from the patient's perspective. Patients' Covid-19-related comments about cleanliness and hygiene practices also included negative observations about the care givers' professionalism, respect for the patient, and consideration for safety. While patients' attention to cleanliness may be expected during a pandemic, patients' perceptions of professionalism or courtesy and respect of the caregiver may evolve to include cleanliness and proper hygiene as a long-term impact.

Patients' concerns and frustrations are largely driven by the processes, and not the people, involved in getting tests and treatments done."

In this first phase of our ongoing analysis of patients' comments related to Covid-19, the ratio of positive to negative insights about staff and doctors continued to fall within the expected range of values for patient experience comments, although the recent jump in positive responses is noteworthy. Patients' persistent positivity and appreciation about their caregivers at this moment of crisis should be noted amid the call to address the signals in these data for learning and improvement. A strong negative signal from the comment data is the need to improve test and treatment logistics. In describing negative experiences, patients distinguish between processes and people. Patients' concerns and frustrations are largely driven by the processes, and not the people, involved in getting tests and treatments done.

Compassion for clinicians is a strong positive signal in the voice of patients. We need to keep listening to learn more and adapt our actions more quickly while showing clinicians that their hard work in these pressing times is being acknowledged with gratitude.

Disclosures: Senem Guney, Chrissy Daniels, and Zach Childers have nothing to disclose.

Unmet Needs: Hearing the Challenges of Chronic Patients with Artificial Intelligence

Analytical insights from patient narratives: the next step for better patient experience

Understanding patient satisfaction with received healthcare services: A natural language processing approach

Leveraging technology to sustain extraordinary care: a qualitative analysis of meaningful nurse recognition

Insights on the Role of Technology and Leadership in Covid-19 Care Delivery, from the Team that Treated America's First Case