key: cord-0855779-7zg4jdqg authors: Chonde, Daniel B.; Pourvaziri, Ali; Williams, Joy; McGowan, Jennifer; Moskos, Margo; Alvarez, Carmen; Narayan, Anand K.; Daye, Dania; Flores, Efren J.; Succi, Marc D. title: RadTranslate™: An AI-powered intervention for urgent imaging to enhance care equity for patients with limited English proficiency during the COVID-19 pandemic date: 2021-01-31 journal: J Am Coll Radiol DOI: 10.1016/j.jacr.2021.01.013 sha: 8f8f5d30fd8cc7c590a43872538933cb37f77bae doc_id: 855779 cord_uid: 7zg4jdqg BACKGROUND Disproportionally high rates of COVID-19 have been noted among communities with limited English proficiency (LEP), resulting in an unmet need for improved multilingual care and interpreter services. To enhance multilingual care, we created a freely available web app (RadTranslate™) that provides multilingual radiology exam instructions. The purpose of this study was to evaluate the implementation of this intervention in radiology. METHODS The device-agnostic web app leverages artificial-intelligence text-to-speech to provide standardized, human-like spoken exam instructions in the patient’s preferred language. Standardized phrases were collected from a consensus group consisting of technologists, radiologists, and ancillary staff. RadTranslate was piloted in Spanish for chest radiographs (CXR) performed at a COVID-19 triage outpatient center that served a predominantly Spanish-speaking Latine community. Implementation included a tablet displaying the app in the CXR room. Imaging appointment duration (IAD) was measured and compared between pre- and post-implementation groups. RESULTS In the 63-day test period following launch, there were 1267 app uses, with technologists voluntarily switching exclusively to RadTranslate for Spanish-speaking patients. The most used phrases were a general explanation of the exam (30% of total) followed by instructions to disrobe and remove any jewelry (12%). There was no significant difference in the IAD, 11±7 min (mean ± standard deviation) and 12±3 for standard-of-care versus RadTranslate, respectively; however, variability was significantly lower when RadTranslate was used (p=0.003). CONCLUSION AI-aided multilingual audio instructions were successfully integrated into imaging workflows, reducing strain on medical interpreters and variance in throughput resulting in more reliable average exam length. Controlling for other factors, patients with limited English proficiency (LEP), individuals for whom English is not their primary language and have difficulty communicating effectively in English, are at increased risk for medical complications and poor outcomes [1] . As such, the role of healthcare provider-patient language concordance is critical and continues to be a topic of interest in medicine [2] . Language concordance is most commonly achieved through real-time direct translations services via certified medical interpreters, including in-person and remote (telephonic or video) interpretation [3] . While the benefits of language concordant medical care have been well documented, it has typically been studied in medical office settings or within emergency departments. Many LEP interventions, including the use of remote interpreters, may not be operationally feasible for urgent and emergent radiology settings where there is expected rapid throughput of patients and where both interpreters (in-person or remote) and imaging equipment are limited resources. While LEP has been shown to impact image quality during CT and MRI [4, 5] , interpreters may not be routinely utilized during standard radiography where exam times are substantially shorter, with an average duration on the order of 4 minutes [6] . The COVID-19 pandemic increased demand for chest radiography as well as interpreter services. Communities of color and immigrant populations were disproportionally affected by COVID-19, with non-white individuals more likely to test positive and have more severe disease at time of presentation [7] . To assist in the surge of patients with suspected COVID-19, reduce crowding in the emergency department, and provide continued community health center support, our institution created respiratory infection clinics (RICs), ambulatory triage clinics for patients with suspected COVID-19. These clinics included COVID-19 testing and standardized care J o u r n a l P r e -p r o o f algorithms, implemented to aid in triage, with an efficient referral process for same-day immediate imaging [8] . As a result, there was a precipitous increase in demand for chest radiography as well as interpreter services in some communities. Institutional experience demonstrated the need for interpreter services in the radiography suite was driven by three factors: ensuring the patient can safely undergo the exam; minimizing close contact to ensure patient and staff safety; and ensuring high quality image acquisition, as COVID radiographic findings can be subtle. To enhance multilingual patient care in this rapidly evolving setting and to ensure highquality imaging for all patients during COVID-19, including those with LEP, a web app was developed and piloted which, using novel AI-created natural sounding audio clips, provides exam instructions to patients in their preferred language. Its utility was evaluated though an implementation science framework utilizing technologist surveys and tracking key operational indicators. This project was developed through diversityxMESH, a collaboration between the Diversity, Equity, and Inclusion Committee of a large academic medical center's radiology department and the MESH Incubator, a technology and entrepreneurship center [9, 10] . The mission of diversityxMESH is to improve health equity though the development, implementation, and dissemination of practical technologies, relying heavily on user-centered design strategies piloted by MESH ( Figure 1 ) as well as implementation science. J o u r n a l P r e -p r o o f This intervention was guided by the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework, a methodology to comprehensively plan and evaluate public health interventions [11] . While other metrics focus solely on efficacy, RE-AIM focuses on robustness, translatability, and impact. The RE-AIM framework is applied by evaluating how a project satisfies the 5 domains: 1. Reach-How do I reach the targeted population? 2. Effectiveness-How do I know my intervention is effective? 3. Adoption-How do I develop organizational support to deliver my intervention? 4. Implementation-How do I ensure the intervention is delivered properly? 5. Maintenance-How do I incorporate the intervention so it is delivered over the long term? Using the standard lexicon for implementation science put forth by Proctor et al, defined in Table 1 , these five domains can be further characterized as eight outcomes: acceptability, adoption, appropriateness, feasibility, fidelity, cost, penetration, and sustainability [12] . These outcomes were considered throughout the design of the application, during its implementation/release, iterative refinement, and data collection, except for fidelity and sustainability which were outside the scope of this pilot study. and penetration, an iterative design process was utilized with continual refinement of the interface by end-user feedback including navigation tree, button size and style, and consensus phrases. Standardized exam instructions were created by a consensus group of front-line technologists, radiologists, and clinical operations managers. These standardized scripts were translated into Spanish by certified bilingual staff who were native speakers of the languages they were translating. Translators were instructed to use colloquial language in their translations (e.g. "picture" instead of "radiograph") whenever possible to facilitate comprehension by LEP patients. Once written translations were completed, audio instructions were synthesized using a neural text-to-speech program (NTTS), capable of creating high quality human-like audio files using synthetic voices [13] . The audio files were integrated with the secure web app and cached to datacenters regionally and globally, allowing audio-clips to load instantly. Quality control was performed on the audio files by certified bilingual staff members to ensure accuracy and clarity. To solicit continued input from the end-users, an integrated feedback system was included, which allows users to suggest new phrases, languages, and any additional updates suggestions. This quality improvement study was approved with exemption by our institutional review board. The study population included all patients over a two-month period following launch of the app who required Spanish translation during RIC chest radiograph evaluation for which RadTranslate was used. The control group consisted of all patients who required Spanish J o u r n a l P r e -p r o o f translation over the 29-day pre-implementation period and were conducted by a technologist who subsequently used RadTranslate at least once following its implementation. This pilot study was restricted to Spanish language instructions for CXR. As the importance of language concordance was understood by staff, the issue became minimizing the barriers to use of RadTranslate. While web-based tools offering real-time J o u r n a l P r e -p r o o f translation, and in some cases text-to-speech, are available, staff were concerned there was no way to ensure the accuracy and quality of the translation (acceptability). To address this, RadTranslate relies on standardized scripts, agreed upon by an interdisciplinary consensus conference, translated by certified medical translators. Using the accelerated prototyping model employed in MESH, translations of CXR instructions were completed within 24 hours of the first standardized language consensus conference and a beta version of the web app was built within 24 hours following interviews with end-user staff. The piloted design of the web app can be found in Figure 2 . To ensure the tool was appropriate for a high-turnover secure environment, navigation was made simple by requiring no user log-in, securing the web app via industry standard secure sockets layer protocols, and not requiring app-store access to download (app-store access on institutional devices is prohibited by many hospital networks). At the pilot site there was 100% penetration among non-Spanish speaking technologists (6 of the 8 technologists at the outpatient site did not speak Spanish). The device agnostic nature of the tool ensured it was feasible to implement in a variety of environments with varying levels of technology, including older or legacy devices in developing environments. Additionally, the cost to the user is none as RadTranslate is free and it can be used on any device with a speaker. Alternatively, the design used here (tablet, mobile stand, Bluetooth speaker) can be replicated using equivalent equipment for $238. RadTranslate was launched on 4/30/2020. The pilot study period lasted from 4/30/2020 to 6/24/2020 while the pre-implementation (control) period was 4/1/2020 to 4/29/2020. During the pilot period there were 1267 uses of the web app. Of the specific CXR instructions, the most J o u r n a l P r e -p r o o f commonly used Spanish phrases were: a general explanation of the chest radiograph exam (30% of total), "Remove everything from the waist up including any necklaces and put the gown on with the opening in the back." (12%), and "Take a deep breath and hold it." (9%). A full list of phrases can be found in Table A1 (see Supplementary online Appendix) . There were no outliers in the RadTranslate group. The mean IAD for the control group was 11±7 min (mean ± standard deviation), and 12±3 min for RadTranslate. The IADs of patients who received RadTranslate was well modeled by a normal distribution. The control group was not well-modeled by a normal distribution (p=6E-8) and statistical evaluation with nonparametric tests were subsequently used. Differences in IAD were not significant between the control and RadTranslate groups using a Mann-Whitney U test (p=0.11); however, there was a statistically significant difference in the exam time variance as measured by the Ansari-Bradley test (p=0.003). A power analysis for differences in IAD was 64.9%, assuming a two tailed alpha level of 0.05. A user survey was sent to 8 technologists, 5 of whom (33%) noted they had a Spanishspeaking patient for which they used RadTranslate. Of note, 80% reported traditional interpreter services hindered their workflow and 60% agreed that interpretation services need to be better integrated into their workflow. Technologist opinions of RadTranslate can be found in Figure 5 . As a small scale proof-of-principle study, there are limitations to the generalizability of the results, mainly due to the small sample size and relative variability in the environment which in turn impact its ability to measure efficacy of the intervention. The sample size was based on convenience, derived from the available number of patients who underwent the intervention. With a calculated power of 64.9% this study was underpowered to measure changes in IAD between groups. Additionally, as conserving patient throughput was placed above creating an ideal measurement environment, no requirements were made on operating procedure. Unlike technologists were only asked to record the exam accession when they used RadTranslate, and data was mined from the EMR [6] . Due to workflow differences among technologists, this limited the time variable to IAD which included the time after the patient had arrived to the imaging waiting room, but before they were received by the technologist. The accuracy of some J o u r n a l P r e -p r o o f of these measurements are unclear. For instance, an IAD in the control group measured 2 minutes, meaning within two minutes after the patient had been checked in by the front desk, their identity had been confirmed, they changed into a hospital gown, and had chest radiographs acquired. Also, the use of an interpreter in the imaging suite could not be definitively verified, and thus the control population may contain a mixture of in-person interpreters, telephone interpreters, and cases were no interpreter was used -although we believe this approximates real-world experience. To evaluate the impact RadTranslate has on workflow, future work could randomize patients to RadTranslate versus standard-of-care. This work underscores the potential for technology driven interventions to address practical challenges in clinical workflow through an equity lens. The impact of this intervention is significant as its free access allows it to be easily scaled and distributed to regions where interpreter services may be limited, can reduce strain on a limited resource of medical interpreters, and improve operations efficiency which may reduce unused equipment times, patient wait times, and potential overcrowding of wait-rooms. Furthermore, as RadTranslate was designed for technologists and with technologist input, it addresses a common problem technologists face and therefore has broad interest. More broadly, this work demonstrates the utility of collaboration between innovation centers and Diversity, Equity, and Inclusion (DE&I) initiatives. Employing the previously published MESH Incubator healthcare design-thinking framework: soliciting end-users for problem identification, rapid iteration, continued end-user feedback, and ultimately production deployment led to the development of a user-friendly tool to improve the patient experience and operational efficacy. From an institutional DE&I standpoint, recruitment and engagement of chest radiograph user interface with Spanish selected as the language; more chest radiograph phrases. Using a device agnostic infrastructure, the tool will appropriately display on a desktop, tablet, phone, etc. The distributed database design allows for the site and audio clips to load instantaneously. Feasibility Similar to appropriateness, but focused on whether or not there are sufficient resources for the intervention to be successfully used. Can we create a tool which is device agnostic? Fidelity How does the adopted use of an intervention differ from its intended use? How else will staff try and use this tool? How much does it cost to implement the intervention? Can we use readily available tools or tools which are relatively low cost? Penetration Integration of a practice within a service setting. Mathematically it is the number of people who routinely use the intervention divided by the people who were trained in the intervention. Can we ensure the maximum staff draw utility from this product? Sustainability Maintenance or continued use or institutionalization of a tool. Can we create something which is dynamic and continues to meet the needs of the end-user? J o u r n a l P r e -p r o o f We're going to take a chest radiograph ordered by your doctor. We're going to take 2 images and we need you to follow the breathing instructions to get the best images. 72 30.1 Are you able to stand today for your x-ray? 19 7.9 Remove everything from the waist up including any necklaces and put the gown on with opening in the back. 31 13 Step up and place chest close to the board. 16 6.7 Now take a deep breath and hold it for a couple of seconds while we take the image. Language of interview: relevance for research of southwest Hispanics Increased Access to Professional Interpreters in the Hospital Improves Informed Consent for Patients with Limited English Proficiency Spanish Interpreter Services for the Hospitalized Pediatric Patient: Provider and Interpreter Perceptions Deploying Clinical Process Improvement Strategies to Reduce Motion Artifacts and Expiratory Phase Scanning in Retrospective analysis of the effect of limited english proficiency on abdominal MRI image quality Workflow Efficiency Comparison of a New CR System with Traditional CR and DR Systems in an Orthopedic Setting Racial/Ethnic Disparities in Disease Severity on Admission Chest Radiographs among Patients Admitted with Confirmed COVID-19: A Retrospective Cohort Study Practice And Policy Reset Post-COVID-19: Reversion, Transition, Or Transformation? Commentary examines possible policy and practice changes for health professionals, regulators, and payers after the COVID-19 pandemic Initial Experience Integrating a Hands-On Innovation Curriculum Into a Radiology Residency Program and Department Medically Engineered Solutions in Health Care: A Technology Incubator and Design-Thinking Curriculum for Radiology Trainees Evaluating the public health impact of health promotion interventions: the RE-AIM framework Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Administration and policy in mental health A Systematic Study of Open Source and Commercial Text-to The author(s) declare(s) that they had full access to all of the data in this study and the author(s) take(s) complete responsibility for the integrity of the data and the accuracy of the data analysis.The author declares no conflict of interest. No sources of funding were obtained for the preparation of this manuscript Institutional Review Board: This study was approved with exemption from the Institutional Review BoardThe authors would like to acknowledge the assistance of Patricia A Daunais and the technologists of Radiology @ Chelsea Community Health Center for their assistance in implementing RadTranslate. The authors would also like to acknowledge the assistance of Seifu J. Chonde, PhD with his assistance in determining the appropriate statistical analysis.