key: cord-0700323-0glunisw authors: Adeoye-Olatunde, Omolola A.; Curran, Geoffrey M.; Jaynes, Heather A.; Hillman, Lisa A.; Sangasubana, Nisaratana; Chewning, Betty A.; Kreling, David H.; Schommer, Jon C.; Murawski, Matthew M.; Perkins, Susan M.; Snyder, Margie E. title: Preparing for the spread of patient-reported outcome (PRO) data collection from primary care to community pharmacy: a mixed-methods study date: 2022-03-14 journal: Implement Sci Commun DOI: 10.1186/s43058-022-00277-3 sha: cdb253a48a93a1f6852ce66c5443005a4fbc83f4 doc_id: 700323 cord_uid: 0glunisw BACKGROUND: Medication non-adherence is a significant public health problem. Patient-reported outcomes (PROs) offer a rich data source to facilitate resolution of medication non-adherence. PatientToc™ is an electronic PRO data collection software originally implemented at primary care practices in California, United States (US). Currently, the use of standardized PRO data collection systems in US community pharmacies is limited. Thus, we are conducting a two-phase evaluation of the spread and scale of PatientToc™ to US Midwestern community pharmacies. This report focuses on the first phase of the evaluation. The objective of this phase was to prepare for implementation of PatientToc™ in community pharmacies by conducting a pre-implementation developmental formative evaluation to (1) identify potential barriers, facilitators, and actionable recommendations to PatientToc™ implementation and (2) create a draft implementation toolkit. METHODS: Data collection consisted of demographics, observations, audio-recorded contextual inquiries, and semi-structured interviews with staff (e.g., primary care providers, pharmacists, pharmacy technicians) and patients during 1-day site visits to a purposive sample of (1) primary care practices currently using PatientToc™ and (2) community pharmacies in Indiana, Wisconsin, and Minnesota interested in the future use of PatientToc™. Post-visit site observation debriefs were also audio-recorded. Verbatim transcripts of all recordings were coded using deductive/inductive approaches and intra-/inter-site summaries were produced identifying potential barriers, facilitators, and actionable recommendations mapped to the Consolidated Framework for Implementation Research constructs. A stakeholder advisory panel engaged in an Evidence-Based Quality Improvement (EBQI) implementation process. This included “member checking” and prioritizing findings, and feedback on the adapted PatientToc™ application, implementation strategies, and accompanying toolkit for community pharmacy implementation. RESULTS: Two primary care practices, nine pharmacies, and 89 individuals participated. Eight major themes (four barriers and four facilitators) and 14 recommendations were identified. Throughout the four EBQI sessions, the panel (1) confirmed findings; (2) designated high priority recommendations: (a) explain PatientToc™ and its benefits clearly and simply to patients, (b) ensure patients can complete questionnaires within 10 min, and (c) provide hands-on training/resources for pharmacy teams; and (3) provided feedback on the adapted PatientToc™ application and finalized toolkit items for initial community pharmacy implementation. CONCLUSIONS: Adoption of electronically captured PROs in community pharmacies is warranted. The implementation strategies systematically developed in this study can serve as a model for implementation of technology-driven health information patient care services, in the understudied context of community pharmacies. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s43058-022-00277-3. Contributions to the literature • Patient-reported outcomes (PROs) offer a rich data source for pharmacists and prescribers to use to resolve medication non-adherence; however, no examples of widespread electronic collection and use of PRO data in community pharmacy settings exist. • We conducted a pre-implementation developmental formative evaluation to inform implementation of an electronic PRO data collection software in community pharmacies. • With the goal of optimizing identification and resolution of medication non-adherence, our findings provide actionable recommendations and an implementation toolkit for spreading an electronic PRO data collection software to community pharmacies. • The resulting implementation toolkit could inform the implementation of other community pharmacy-based patient care services. Medication non-adherence is a significant public health problem. Specifically, in older adults, medication nonadherence has been associated with all-cause hospitalization and mortality [1] . Community pharmacists are well-positioned to identify and intervene on medication non-adherence during medication counseling and other routine pharmacy patient care services [2] . Medication non-adherence may be influenced by negative medication-related outcomes that patients experience [3] . Patient-reported outcome (PRO) measures have become a standard in assessing patients' behaviors towards treatment (e.g., medication adherence) [4] . Many PRO measures (e.g., Adherence Starts with Knowledge 20, General Medication Adherence Scale, Medication Adherence Reasons Scale) for medication non-adherence exist [5] . However, systematically collecting, documenting, tracking, and analyzing PROs can be problematic, particularly in the community pharmacy setting due to implementation challenges (e.g., meeting patient language needs, data privacy and security concerns, timely accessibility of data for pharmacist interventions) and limited use of standardized PRO data collection systems. Likewise, electronic patient-reported outcome data software addressing these challenges are more commonly used in other healthcare settings [4] ; however, use in community pharmacy settings remains limited [6] . PatientToc ™ is one type of PRO data collection software. It was initially developed by L.A. Net Community Health Resources Network investigators in California to meet the unique needs of diverse patients served by primary care practices [7] . The software facilitates PRO data collection using Android devices, with audio assist available in more than 200 languages [7] . A more detailed description and screenshots of PatientToc ™ can be found in our study protocol paper [8] . The spread of this PRO data collection software from use in primary care to community pharmacies is being evaluated using a twophase (pre-implementation followed by implementation) approach. The intended medication adherence-related PROs (i.e., responses to Brief Medication Questionnaire and Merck Medication Adherence Estimator) to be collected via PatientToc ™ and plans for their use in phase two of the evaluation are provided in our study protocol paper [8] . The objectives of the study's first phase, presented here, were to prepare for implementing PatientToc ™ in community pharmacies by conducting a pre-implementation developmental formative evaluation to (1) identify potential barriers and facilitators, and actionable recommendations, to PatientToc ™ implementation, and (2) create a draft implementation toolkit. The research team applied a convergent parallel, qualitatively driven mixed-methods [9] study design. This design equipped researchers to investigate expected barriers, facilitators, and actionable recommendations for PatientToc ™ implementation in community pharmacies. Qualitative methods were the primary methods used for data collection and analysis. We intentionally provide extensive details of methods used to establish qualitative rigor (i.e., "trustworthiness") in terms of credibility, transferability, dependability, and confirmability as defined simply to patients, (b) ensure patients can complete questionnaires within 10 min, and (c) provide hands-on training/resources for pharmacy teams; and (3) provided feedback on the adapted PatientToc ™ application and finalized toolkit items for initial community pharmacy implementation. Conclusions: Adoption of electronically captured PROs in community pharmacies is warranted. The implementation strategies systematically developed in this study can serve as a model for implementation of technology-driven health information patient care services, in the understudied context of community pharmacies. Keywords: Community pharmacy, Patient-reported outcomes, Health information technology Page 3 of 17 Adeoye-Olatunde et al. Implementation Science Communications (2022) 3:29 with supporting accessible examples by Thomas and colleagues [10] . Quantitative methods were used to contextualize the study population and qualitative findings. Qualitative and quantitative data were analyzed concurrently and subsequently triangulated [11] via synthesis documents and group discussions (described in detail in the "Data synthesis" section). Reporting is in accordance with the Good Reporting of a Mixed Methods Study (GRAMMS) criteria (see Additional file 1) [12] . The study design was guided by three conceptual frameworks: EBQI is a multilevel process to systematically incorporate scientific findings into healthcare settings [16, 17] . This process is motivated and facilitated by researcher and local stakeholder partnerships [16, 17] . Specific to this study, we adopted Curran's two-step approach to EBQI: 1. Diagnosis of site-specific implementation needs, barriers, and facilitators (i.e., formative evaluation) 2. The use of multi-disciplinary teams of local staff, implementation experts, and clinical experts to interpret diagnostic data and develop/adapt site-specific interventions EBQI also operates as an implementation strategy by enabling contextual adaptation of interventions and creating buy-in among stakeholders. We applied the CFIR to guide qualitative data collection/ analysis and data synthesis. The CFIR is a well-established determinant implementation framework that is comprehensive and well suited for complex, multilevel interventions. CFIR categorizes implementation constructs across five domains: intervention characteristics, outer setting, inner setting, characteristics of the individuals involved, and the process of implementation [14] . To further classify our actionable recommendations for PatientToc ™ implementation, we applied Waltz et al. 's ERIC taxonomy of implementation strategies. The taxonomy suggests 73 discrete implementation strategies grouped into nine categories or "types" of strategies: use evaluative and iterative strategies, provide interactive assistance, adapt and tailor to context, develop stakeholder interrelationships, train and educate stakeholders, support clinicians, engage consumers, utilize financial strategies, and change infrastructure [15] . To address study objectives, the study setting consisted of Western primary care practices and Midwestern community pharmacies in the United States (US). The American Academy of Family Physicians defines primary care practice in the US as the patient's entry point into the health care system and as the continuing focal point for all needed health care services. Primary care practices are generally located in the community they serve, thereby facilitating access to health care while maintaining a wide variety of specialty and institutional consultative and referral relationships for specific care needs. The primary care practice structure often includes a team of physicians and other health professionals [18] . In addition to dispensing prescriptions (medications prescribed by an authorized provider), community pharmacists in the US are the most accessible healthcare professional to the public. Furthermore, US community pharmacies offer other pharmacy services including immunizations, disease state, and medication therapy management services. The community pharmacy structure typically includes licensed pharmacists and pharmacy technicians. A purposive sample (n = 11) of study sites was recruited primarily through practice-based research networks (PBRN) across four states in the US: (1) , and (4) select community pharmacies in Wisconsin. A description of each PBRN can be found in our protocol paper [8] . Typical recruitment practices (e.g., emails, phone calls) of each PBRN (mirrored for Wisconsin) were followed. To better understand how PatientToc ™ has been implemented in primary care, we recruited two L.A. Net primary care practices in California, with varied approaches to PatientToc ™ implementation. Three community pharmacies, consisting of a wide range of practice types and settings (e.g., urban vs. rural) from each of the remaining three states (nine total), were recruited to better understand potential barriers, facilitators, and recommendations for spreading PatientToc ™ to community pharmacies. A core team of investigators (OAAO, GMC, HAJ, MES) with qualitative research expertise participated in qualitative data collection activities across all study sites. Additional trained investigators (LAH, NS) participated in qualitative data collection at their respective state study sites. All researchers engaged in qualitative data collection participated in pre-and onsite studyspecific training. Qualitative data collection consisted of audio-recorded semi-structured interviews, investigator observations of staff and patients at participating primary care and pharmacy sites, and contextual inquiries. Observations were conducted to provide insights into how PatientToc ™ is currently used (primary care) and could be used (pharmacies). Contextual inquiries consisted of informal interviews with participants demonstrating and describing elements of their work duties (see Additional file 2). Lastly, to be reflexive of the research process, summary recorded debriefs following each site visit and resulting data were included as part of our formal data collection. We also debriefed (not part of data collection) after all site visits for each state were conducted. Our protocol paper provides additional details of each qualitative data collection method [8] . As mentioned previously, CFIR informed qualitative data collection. Specifically, semi-structured interview and observation guides were designed to explore participant experiences and probe for expected barriers, facilitators, and recommendations within the CFIR constructs. Contextual inquiry probes were designed to further explore specific routine tasks associated with PatientToc ™ use by primary care stakeholders as well as pharmacy tasks expected to be influenced by future PatientToc ™ use. By engaging with stakeholders as they completed routine tasks, the probes further elucidated expected barriers, facilitators, and recommendations. Data collection forms were pilot tested and refined prior to use. Specifically, primary care data collection forms were pilot tested for content and process flow by three volunteers (one provider, one staff member, and one patient) familiar with PatientToc ™ . Pharmacy data collection forms were pilot tested for content and process flow by two volunteers (one community pharmacist and one community pharmacy patient) not familiar with PatientToc ™ . Minor modifications were made to all data collection forms for clarification purposes. A purposive sample, targeting five clinicians/staff and five patients (18 years of age or older having at least one chronic condition for which they routinely take medication) from participating primary care and pharmacy practice sites, was invited to engage with researchers during 1-day site visits. The qualitative analysis team included two trained student research assistants and an investigator (LAH). Audio-recorded semi-structured interviews, investigator observation summary debriefs, and contextual inquiries were transcribed verbatim (InfraWare Inc, Terre Haute, IN) and checked for accuracy. The qualitative analysis team coded transcripts using NVivo 12 Pro (QSR International). Both deductive (using CFIR) and inductive (emergent from the data) approaches were used in developing the codebook. The CFIR Codebook Template [22] was adapted to fit the context of our study and frame code definitions and coding inclusion/exclusion criteria. The final codebook is available in the supplemental files (see Additional file 3). We detail our qualitative data coding steps below: 1. Deductive "level 1" broad codes consisted of the 41 CFIR constructs included in the CFIR Codebook Template [22] . a. Of note, several publications report 39 CFIR constructs [23] ; however, we included all 41 listed in the CFIR Guide Codebook Template [22] in our codebook. Additional details regarding this approach to code development are provided in the supplemental files (see Additional file 3). A total of 123 deductive sub-codes, termed "level 2" sub-codes, were included to further delineate each "level 1" broad code into potential barriers, facilitators, and actionable recommendations for Patient-Toc ™ implementation in community pharmacies. 3. Subsequently, after the qualitative analysis team "level 2" coded transcripts, 234 inductive "level 3" codes were created to further delineate "level 2" codes in response to the actual data that were coded. To illustrate this coding process, we will use the CFIR construct "Relative Advantage" as an example. Our "level 1" broad code was RelAdvantage. "Level 2" sub-codes were RelAdvantage_B, RelAdvantage_F, RelAdvantage_R, with B, F, and R indicating barrier, facilitator, and recommendation, respectively. Lastly "level 3" inductive sub-codes were created in response to the actual "level 2" coded data and included inductive codes such as Prefer paper_B, Easier_alternative_F, Increase_efficiency_R. Codes were modified, created, or collapsed as necessary [24] . Rotating pairs of analysts independently coded and reconciled the same transcripts for two study sites, until all transcripts for these sites were reconciled. Subsequently, analysts independently coded an approximately equal number of different transcripts for the remaining study sites. No double coding (i.e., using more than one code to code the same pieces of text) was permitted. Three other investigators (OAAO, HAJ, MES) with expertise in qualitative research reviewed approximately one-third of all coded transcripts and met with the qualitative analysis team regularly to provide feedback and ensure the codebook was applied consistently. Quantitative data collection consisted of the study site and participant demographics (see Additional file 2). Study site demographics, e.g., type of practice, number and type of staff members, prescription volume, were self-reported from the primary site contact and collected by telephone in advance of the 1-day site visits. Participant demographics, e.g., age, role, race, years employed at practice site, were self-reported and collected at the end of each participant's interview. All quantitative data were collected and managed using Research Electronic Data Capture (REDCap ™ ) [25] electronic tools hosted at the Indiana Clinical and Translational Sciences Institute. REDCap ™ is a secure, Web-based application designed to support data capture for research studies, providing (1) an intuitive interface for validated data entry; (2) audit trails for tracking data manipulation and export procedures; (3) automated export procedures for seamless data downloads to common statistical packages; and (4) procedures for importing data from external sources [25] . Site and participant demographics were summarized by computing descriptive statistics (counts and percentages, means, and standard deviations) at the site level, and then per site descriptive statistics were used to create a mean across all sites using SPSS v. 25 (IBM Corp). Intra-site summary documents were created to summarize qualitative and quantitative findings for each participating primary care practice and pharmacy. Mixing and triangulation [11] consisted of the qualitative analysis team noting convergence or salient differences in qualitative findings by qualitative method used (e.g., if investigator observation summary debrief data confirmed or differed from interview data) and select quantitative variables, site, and participant type. Informed by intrasite summary documents, an inter-site summary document was created to facilitate investigator synthesis and identification of final themes during planned research team discussions. During the course of 2 days (13 h total), research team members met virtually (due to COVID-19 travel restrictions) via videoconferencing software to review intra-and inter-site synthesis documents and identify themes by CFIR construct. The principal investigator (MES) compiled and summarized notes from the research team discussions to identify overarching major themes, categorized as barriers or facilitators and recommendations mapped to applicable CFIR constructs. Research team members who participated in the 2-day group discussions had opportunities to review the major themes/recommendations and accuracy of mapping to CFIR constructs. Minor modifications (e.g., re-mapping of a few CFIR constructs) were made through the team review process. Mixing and triangulation [11] consisted of noting convergence or salient differences in final major themes by type of practice and participant role. Resulting major themes informed the EBQI process for this study. This process consisted of assembling a multistakeholder advisory panel. In addition to a subset of study team members, the target number of Advisory Panel members was nine stakeholders-a pharmacist, a technician, and a patient from each state (i.e., one participant of each stakeholder type representing each pharmacy study site). In selecting stakeholders to invite, we attempted to balance demographics including gender, race, number of medications taken, and frequency of pharmacy visits (patients), as well as expected contributions/willingness to share ideas (based on interview responses), and perceived engagement with the site/ project (for pharmacists/staff, based on site visits). For this phase of the study, we held a total of four, 120-min virtual EBQI sessions via videoconferencing software. To gather the unique perspectives of pharmacy staff and patients, "breakout" groups were held during the sessions to discuss priority questions relevant to each stakeholder group. Their insights were then shared with the whole group. The first three sessions consisted of member checking and prioritizing findings. Panelists' initial recommendations from these sessions informed (1) mockup of the adapted PatientToc ™ application and (2) draft toolkit resources for initial spread of PatientToc ™ in community pharmacies. These items were reviewed during the fourth EBQI session and panelist recommendations for PatientToc ™ adaptations and implementation were finalized. Keeping the study objective at the forefront and for confidentiality purposes (given small sample sizes), we first present a brief description of the current PatientToc ™ implementation context at participating primary care practices (n=2) followed by a summary of quantitative, qualitative, and synthesis (barriers and facilitators) findings from participating pharmacies (n=9). Finally, we provide the actionable recommendations and the implementation toolkit items drafted in response and created during the EBQI process. PatientToc ™ implementation at participating primary care practices Across the two primary care practices, a total of two providers (e.g., doctor, nurse practitioner), six support staff (e.g., registered nurses, medical assistants), and two patients participated. Both primary care practices serve a diverse patient population including Hispanic/Latino/ Latinx/Spanish, Black/African-Americans, socioeconomically disadvantaged, medically underserved, Medicaid or dual Medicaid/Medicare beneficiaries, adult and pediatric patients, and pregnant women receiving pre-natal care. PatientToc ™ implementation varied across the participating practices. One practice used it to collect survey data from Consumer Assessment of Healthcare Providers and Systems (CAHPS ® ), a federal program that administers surveys to capture and evaluate patient experiences, and use of the tablets has become well integrated into the facility's workflow. Currently, the PatientToc ™ tablets are mounted on rolling carts and patients' complete questionnaires in private after being "roomed" and waiting for their provider. Many of the primary care providers and support staff are familiar with the tablet. The practice also implemented competitive incentives associated with the number of CAHPS ® completed on PatientToc ™ . The other practice currently stations PatientToc ™ tablets in the waiting area at dedicated tables but signs indicate that tablets should not be used by patients unless instructed by staff. We did not observe tablet use by staff or patients during our visit. At this site, PatientToc ™ is being used to capture "Staying Healthy Assessments, " which are part of the required California Medicaid (Medi-Cal) Initial Health Assessment. Questions focus on topics such as pain, depression, and fall risk. One L.A. Net staff member is embedded in the practice to assist with PatientToc ™ implementation. While some of the primary care providers were familiar with PatientToc ™ , support staff appeared to have minimal familiarity. Quantitative, qualitative, and synthesis findings from participating pharmacies Table 1 summarizes pharmacy and participant characteristics at the site level and by participant type. A total of nine pharmacies participated in this pre-implementation phase of the study. The sites consisted of independent (n=6, 67%) and health system (n=3, 33%) pharmacies. In the US, independent pharmacies are privately held retail pharmacies not owned or operated by a publicly traded company and have no affiliation with any chain of pharmacies. For the purposes of this study, we utilized the National Council for Prescription Drug Programs (NCPDP) definition of independent pharmacy, namely one to three pharmacy locations under common ownership [26] . Likewise, a health system pharmacy is a retail pharmacy that is affiliated with a health system, which is defined as an organization that includes at least one group of providers who provide primary and/or specialty care that is integrated with each other and the hospital through joint management or common ownership. Across all sites, the mean (standard deviation (SD)) weekly prescription volume was 1266 (605) and had a mean (SD) of 3 (1) full-time equivalent pharmacists and 6 (5) full-time equivalent staff. All pharmacies offered 90-day prescription fills as a service to facilitate patients' adherence to prescription medications. Across the nine community pharmacies, a total of 22 pharmacists and 28 pharmacy staff (e.g., pharmacy technicians, service clerks) participated and reported the mean (SD) weekly percent of hours spent working with patients as 80% (15) and 84% (19) , respectively. A total of 34 pharmacy patients participated in the study. Pharmacy patients participants primarily identified as non-Hispanic White (mean (SD) 88% (22)), half (50% (30)) identified as male, 31% (37) visited their pharmacy at least once a week, and regularly used eight medications on average. Table 2 lists major themes (and associated theme description, representative quotations, and CFIR constructs) categorized as expected barriers and facilitators for PatientToc ™ implementation in community pharmacies. We identified a total of 8 major themes: four barriers and four facilitators, for PatientToc ™ implementation in community pharmacies. Convergence of qualitative results was evident across all qualitative data collection methods (semi-structured interviews and contextual inquiries with participant and investigator observation debriefs); thus, all illustrative quotations are from qualitative semi-structured interview data and included nuances by select quantitative variables including site type [(1) independent pharmacies, (2) health system pharmacies, (3) both pharmacy types, (4) primary care, (5) all primary care and pharmacy types] and participant type [(1) pharmacy staff (including staff and pharmacists), (2) pharmacy patients, (3) primary care staff (including providers and staff ), (4) primary care patients, (5) primary care and pharmacy staff, (6) primary care and pharmacy staff and patients]. Of the four barrier-related themes, two included potential PatientToc ™ integration challenges with existing pharmacy technology and within workflow. The other two barrier-related themes included potential challenges with uptake by certain patients due to concerns with data security and preferences (e.g., those who prefer providing information on paper compared to using technology). Table 1 Characteristics of participating community pharmacies (n=9), pharmacists (n=22), pharmacy staff (n=28), and patients (n=34) a Type of pharmacy, n (%) Independent (< 4 locations) 6 (67%) Health Drive-through window 2 (22%) Per site, number of pharmacist participants in the study, mean (SD) 2 (1) Per site, pharmacist age, mean (SD) 38 (10) Per site, pharmacist percent male, mean (SD) 39 (24) Per site, pharmacist percent not Hispanic, mean (SD) 96 (11) Per site, pharmacist percent white, mean (SD) 94 (17) Per site, pharmacist percent PharmD, mean (SD) 78 (36) Per site, pharmacist, percent with no residency training, mean (SD) 83 (25) Per site, pharmacist, years employed at this pharmacy (any position), mean (SD) 6 (4) Per site, percent of pharmacist participants, current position at pharmacy, mean (SD) Owner, mean (SD) 28 (36) Manager, mean (SD) 15 (23) Staff pharmacist, mean (SD) 34 (29) Per site, pharmacist, percent of working hours working with patients within a week, mean (SD) 80 (15) Per site, number of pharmacy staff participants in the study, mean (SD) 3 (1) Per site, pharmacy staff, age, mean (SD) 41 (8) Per site, pharmacy staff percent male, mean (SD) 9 (19) Per site, pharmacy staff, percent not Hispanic, mean (SD) 97 (8) Per site, pharmacy staff, percent white, mean (SD) 89 (19) Per site, pharmacy staff, percent pharmacy technicians, mean (SD) 89 (14) Per site, pharmacy staff, percent highest degree completed, mean (SD) High school (GED) 33 (41) Per site, pharmacy staff, years employed at this pharmacy (any position), mean (SD) 6 (6) Per site, pharmacy staff, years employed as a pharmacy staff member at this pharmacy, mean (SD) 7 (7) Of the four facilitator-related themes, three included pharmacy staff and leadership factors including pharmacy staff members' willingness to try PatientToc ™ if it can potentially improve patient care, respected leadership and strong communication across pharmacy team members (supporting PatientToc ™ implementation), and pharmacy-related measures aligning with those expected to be impacted by PatientToc ™ . The last facilitatorrelated theme was the perceived ease of PatientToc ™ use by both pharmacy staff and patients. A total of 14 actionable recommendations were identified ( Table 3 ). The multi-stakeholder advisory panel was comprised of community pharmacists (n=3), pharmacy staff (n=2), and patient (n=3) participants representing seven (77.8%) of the participating pharmacy sites. Throughout three EBQI sessions, the multi-stakeholder panel confirmed the thematic findings and indicated the highest priority recommendations: (a) provide clear and simple instructions to patients that emphasize the expected benefits of PatientToc ™ , (b) ensure patients can complete questionnaires within 10 min, and (c) provide hands-on training/resources for pharmacy teams. Toolkit items were drafted in response. In the fourth EBQI session, panelists reviewed a mock-up of the PatientToc ™ adapted for use in community pharmacies. Toolkit items for initial community pharmacy implementation were finalized. Four recommendations pertaining to intervention adaptations will be addressed in the final Patient-Toc ™ build during phase two. Table 3 lists all of the 14 recommendations made by type of participant (pharmacy staff, patient, both) and associated ERIC implementation strategies [15] and describes specific strategy/ toolkit resources. Across the 14 actionable recommendations, there was alignment with one or more of the 9 ERIC implementation strategy categories. Of the 14 recommendations, four aligned with Engage consumers: -Provide clear and simple messaging to patients that emphasize the expected benefits of PatientToc ™ to patients -Implement PatientToc ™ first with specific patient sub-groups (e.g., complex patients) Per site, number of patient participants in the study, mean (SD) 4 (1) Per site, patient age, mean (SD) 63 (8) Per site, patient percent male, mean (SD) 50 (30) Per site, patient percent not Hispanic, mean (SD) 100 (0) Per site, patient percent white, mean (SD) 88 (22) Per site, patient percent, frequency of pharmacy visit, mean (SD) At least once a week 31 (37) Less than every week but more than once a month 11 (21) About once a month 26 (22) About once every 3 months 16 (21) Per site, patient number chronic conditions that require a routine prescription, mean (SD) 3 (1) Per site, patient number of prescription medications regularly used, mean (SD) 6 (2) Per site, patient number of over the counter medications regularly used, mean (SD) 1 (1) Per site, patient number of supplements regularly used, mean (SD) 2 (1) Per site, patient number of total medications regularly used, mean (SD) 8 (2) Abbreviations: SD Standard deviation, FTE Full-time equivalent, EQuiPP Electronic Quality Improvement Platform of Plans and Pharmacies, MTM Medication therapy management a Site demographic data were self-reported from the research team's primary contact person at the site; subject-level demographic data were self-reported by the subject. No attempts were made to verify data from other sources (e.g., the EMR/dispensing system for patient/prescription volume or number of reported prescriptions taken by patients) b Responses to "other" included website available for refills, refill reminder calls, targeted disease state adherence checks/ calls, mail prescriptions, texting service for refill reminders, and "Flip the Pharmacy" initiative c Responses to "other" included vaccines, blood pressure monitoring cholesterol monitoring, point of care testing, medical equipment fitting/ supply, travel health consulting, weight management, home care services, and naloxone consulting "Firewalls are a big concern with integrating [PT]... with multiple software … dispensing software, outpatient clinic software and inpatient software. " -Health system pharmacy (staff ) 2. Some sub-groups of patients (e.g., older adults, those with arthritis, those who do not physically come in to the pharmacy, those who prefer paper over technology) might be challenged to use PatientToc ™ . Some pharmacy patients expressed concerns over technological complexity, managing PT, and seeing how it meets a need for them while pharmacy staff described concerns due to patient age and co-morbidities and needing to walk patients through it (both pharmacy types). Relative Advantage, Complexity, Design Quality & Packaging, Patient Needs & Resources, Structural Characteristics, Culture, Compatibility, Engaging, Intervention Participants "There would be some people who would be... you know, we have a pretty elderly clientele. They would look at something like this and say, no, thank you. " -Independent pharmacy (staff ) 3. PatientToc ™ could be difficult to incorporate into pharmacy workflow due to space (e.g., small waiting areas, shared space) or staffing (e.g., time required, possible need for additional staff, competing demands) constraints. Some pharmacy staff expressed concern with available space for pharmacy patients to use PT while many pharmacy staff noted a potential need for additional staff time due to constraints and competing demands (both pharmacy types). "Some of your elderly population come in with their cell phones and they are using them for things, or they are calling in or using the internet to call in the refills, but then there are other ones that absolutely won't use our automated system and want to talk to somebody all the time, so I think those people would probably need more help and from a workflow standpoint, I don't know how much extra time we would have to walk them all the way through... so that would be a concern. " -Health system pharmacy (staff ) 4 . Data security concerns (e.g., privacy of information provided in PatientToc ™ by patients, mistrust of technology, uncertainty regarding where the information is sent) could limit uptake of PatientToc ™ by pharmacy patients. There are some concerns that patients might resist or not trust technology due to concerns with privacy and/or a general mistrust in technology (all primary care practices and pharmacy types) (primary care and pharmacy staff and patients). "I like to talk face-to-face. I don't trust these things… Everybody can get your information on them. " -Independent pharmacy (patient) "There is always some reluctance to everything, because everyone... now they have everyone afraid about data... so everyone is like, what are you going to do with my data? What are you going to do with my data? You know, so that is going to be your biggest challenge... everyone is like, oh you are going to get rich off my data, you know, I mean... just the fact that the media has sort of undermined the credibility… everybody is a suspect now. " -Primary Care (staff ) There is general optimism for PT and expected buy-in from pharmacy staff in trying new things that will benefit and be perceived to be of value to patient care (both pharmacy types) (pharmacy staff ). There is a strong sense that leadership is supportive and respected and motivation to do well was noted with many sites. Teamwork is reflected in regularly scheduled meeting times (all primary care practices and pharmacy types) (primary care and pharmacy staff ). In specific, pharmacy staff generally view technicians as equals and have a good understanding of roles (both pharmacy types), and there is evidence of strong pharmacy-patient relationships in which patients feel motivated to help the pharmacy (both pharmacy types) (pharmacy patients). "I think that helps because I am here on a daily basis and not only do I help in, you know, improving numbers, but I also help in other areas of the workflow… and then just building the rapport with the team, knowing that, you know, I hear them on a daily basis and they can come to me with anything, open door policy, and just... yeah, I have a good rapport with my team. " -Primary Care (staff ) "Well, we all do our huddles. The pharmacist is very encouraging and very helpful, and so, like after the huddle, everyone just feels pumped, and so, ... it feels... because we all get together and we talk about how we could all improve within the pharmacy... " -Health system pharmacy (staff ) Most pharmacy participants felt PT flowed and worked well and generally felt confident in their ability to use it with any improvements for suggestion likely feasible (both pharmacy types) (pharmacy staff and patients). Knowledge & Information, Self-Efficacy "I think, honestly, you've overcome all the barriers as far as navigating the tablet. I don't think you could make it any more simple. " -Independent pharmacy (staff ) "…it's just the print was just right, the lettering was easy to read, the font was okay and the colors were right, the display was easy at a glance you could understand it, so...Even if you are not computer literate, it might take you a minute, but you could figure it out eventually. " -Independent pharmacy (patient) Abbreviations: CFIR Consolidated Framework for Implementation Research, CPESN Community Pharmacy Enhanced Services Network, PT PatientToc ™ a Sample includes all 11 participating pharmacy (n=9) and primary care practice (n=2) sites b Convergence of qualitative results was evident across all qualitative data collection methods (semi-structured interviews and contextual inquiries with participants and investigator observation debriefs); thus, theme descriptions and supporting quotations were informed by semi-structured interview data only c Site type was categorized as follows: (1) independent pharmacies, (2) health system pharmacies, (3) both pharmacy types, (4) primary care, and (5) all primary care and pharmacy types d Participant type was categorized as follows: (1) pharmacy staff (including staff and pharmacists), (2) pharmacy patients, (3) primary care staff (including providers and staff ), (4) primary care patients, (5) primary care and pharmacy staff, and (6) primary care and pharmacy staff and patients -Ensure patients can complete PatientToc ™ questionnaires within 2 to 10 min (e.g., pre-populate information when possible, reduce need for typing) -Use PatientToc ™ to optimize patient prescription wait times as well as other appointment-based services (e.g., MTM, medication synchronization program) Accompanying implementation toolkit items included patient-facing print materials (large and small posters, pamphlets, bag stuffers), scripted language for pharmacies' use, sample workflows, and workflow "cheat sheets. " Three recommendations aligned with Adapt and tailor to context: -Work with pharmacy teams and vendors to ensure PatientToc ™ is well integrated with the pharmacies' dispensing systems -Consider adapting PatientToc ™ for more languages (mockup/demo was provided in English only) -Consider including patient education and/or information/referrals to pharmacy services on Patient-Toc ™ These will primarily be addressed as part of the intervention development/finalized build of the adapted PatientToc ™ application; however, a "Referrals Cheat Sheet" implementation toolkit item was created to provide a modifiable template of resources for pharmacy teams to consider when reviewing PatientToc ™ results and addressing medication non-adherence. Of note, Work with pharmacy teams and vendors to ensure PatientToc ™ is well integrated with the pharmacies' dispensing systems was one of the most extensive adaptations to the Patient-Toc ™ application and we plan to include this capability in the adapted finalized build during phase two. Engaging key stakeholders from multiple perspectives provided invaluable insight into resources and strategies needed for initial implementation of PatientToc ™ in community pharmacies. These collaborative efforts positioned us to prioritize actionable recommendations, refine and tailor an adapted PatientToc ™ design, and develop an initial implementation toolkit unique to the community pharmacy context, which ultimately is expected to result in more effective spreading and scaling efforts. To our knowledge, this is the first study to evaluate electronic collection and use of electronic PRO data in community pharmacy settings using implementation science approaches. As stakeholders' insights were the foundation of this pre-implementation evaluation, we focus our discussion on stakeholders' highest priority recommendations. As well, we situate the findings in the implementation science and pharmacy practice literatures and discuss policy-related implications. The first two high priority recommendations, (1) provide clear and simple instructions to patients that emphasize the expected benefits of PatientToc ™ and (2) ensure that patients can complete PatientToc ™ questionnaires within 2 to 10 min, align with the ERIC implementation strategy "Engage Consumers: Prepare Patients/Consumers to be Active Participants" [15] . The first recommendation regarding the emphasis of expected benefits to patients has practice and policyrelated implications. For example, in the US, medication therapy management (MTM) under Medicare Prescription Drug policy is a service often provided by community pharmacists [27] ; however, the literature indicates lack of MTM uptake by some beneficiaries may be due to gaps in patient-friendly communication and marketing on MTM and its benefits [28] . Our toolkit items and specific strategies for this recommendation, including patient-facing print materials and scripted language for pharmacy team members, align with suggestions cited in the pharmacy practice literature [28] and could serve as a model for promoting adoption of other pharmacy services and related policies. Similar to the second high priority recommendation of using more brief questionnaires, Rolstad et al. [29] review and meta-analysis findings suggested questionnaire response rates were statistically lower for longer questionnaires; however, the researchers argue that the impact of content should not be overlooked, further emphasizing the importance of buy-in of patients, regardless of questionnaire length. Our toolkit items and marketing and messaging strategies include benefits, specifically to patients, expected from completing PatientToc ™ questionnaires. Time to complete the questionnaires will be examined during phase two (implementation) of this study. The third and final high priority recommendation, provide hands-on training and resources for pharmacy teams, possibly for continuing education credit, to support PatientToc ™ implementation, aligns with several ERIC implementation strategies in the "Train and Educate Stakeholders" category, including develop and distribute education materials and work with educational institutions. Our training toolkit items align with these strategies, particularly our Social Determinants of Health Continuing Education modules (per request of advisory panel members) offered in collaboration with a college of pharmacy continuing education office. This recommendation was not surprising, as implementation strategies focusing on training and educating stakeholders are ubiquitous. For example, Thoele et al. describe their intervention implementation process in an acute care hospital setting. Researchers developed 54 toolkit items over a three-phase process. Of these 54 toolkit items, 25 (46%) were related to training medical staff during the first two phases. Training and education strategies are often described as necessary for implementation efforts, but they are usually not sufficient on their own [30] ; hence, our implementation toolkit is composed of a variety of implementation strategies spanning the nine ERIC implementation strategy categories. Implementation science is an iterative process; thus, we are taking a similar approach and aim to do two plan-dostudy-act cycles in the next phase of this study to refine our implementation toolkit for spreading and scaling PatientToc ™ to community pharmacies. We anticipate revisions to the pharmacy team's training modules may be required after initial implementation and prior to further scaling of PatientToc ™ . While this study has several strengths, findings should be considered in the context of its limitations. First, only Midwestern and no chain community pharmacies were recruited in this phase of the study. However, we made great efforts to account for nuances in findings by pharmacy and participant type and we provide participating pharmacies' demographic data, which can aid in transferring findings to different community pharmacy settings with similar characteristics. Nevertheless, we will attempt to recruit chain community pharmacies in the scale-up phase of evaluation and will revise implementation strategies as needed. Although comprehensive, we applied a complex approach to data analysis by including all 41 constructs of CFIR, which ultimately resulted in over 200 inductive sub-codes. This approach was time consuming and required a change in our analysis approach from analysts independently coding and reconciling the same transcripts for all study sites to two sites with the remaining transcripts being divided across analysts to be independently coded with frequent research team member quality checks. Furthermore, results could have varied if a different implementation framework was used. However, CFIR proved to be useful as it addressed our study objectives, facilitated the identification of key implementation barriers and facilitators, and helped elicit recommendations, which corresponded to several widely recognized implementation strategies [15] . Lastly, the developmental formative evaluation approach used in this study prioritized qualitative methods to explore potential barriers, facilitators, and actionable recommendations for implementing PatientToc ™ in community pharmacies. A more quantitative approach, such as a survey, could better capture frequencies of potential barriers, facilitators, and recommendations. However, using qualitative approaches, such as semi-structured interviews, was preferred for our goal of gaining in-depth perspectives from participants rather than a generalized understanding [31] . Thus, using both methods allowed access to data and interpretations of findings that each method alone could not provide. Adoption of health information technology, specifically to electronically capture PROs in the community pharmacy setting, is warranted. Applying implementation science methods and strategies can aid in adopting such interventions. If shown to be effective, the implementation strategies systematically developed in this study can serve as models for implementing other health information technology-driven, patient care services in the understudied context of community pharmacies. The association between medication non-adherence and adverse health outcomes in ageing populations: a systematic review and metaanalysis CDC: Tailored pharmacy-based interventions to improve medication adherence A systematic review of barriers to medication adherence in the elderly: looking beyond cost and regimen complexity Benefits and disadvantages of electronic patient-reported outcome measures: systematic review Measurement properties of existing patient-reported outcome measures on medication adherence: systematic review Studying the impact of a medication use evaluation by the community pharmacist (Simenon): patient-reported outcome measures An evaluation of the spread and scale of PatientToc ™ from primary care to community pharmacy practice for the collection of patient-reported outcomes: a study protocol Qualitative approaches to mixed methods practice Qualitative rigor or research validity in qualitative research Toward a conceptual framework for mixed-method evaluation designs The quality of mixed methods studies in health services research A process for developing an implementation intervention: QUERI series Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the Expert Recommendations for Implementing Change (ERIC) study Practice-based research networks add value to evidence-based quality improvement Impacts of evidence-based quality improvement on depression in primary care: a randomized experiment Community health resource network CFIR Tools and Templates -Data analysis tools: codebook template A systematic review of the use of the Consolidated Framework for Implementation Research Qualitative data analysis for health services research: developing taxonomy, themes, and theory Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support Medication therapy management delivery by community pharmacists: insights from a national sample of Medicare part D beneficiaries Medication therapy management: current challenges Response burden and questionnaire length: is shorter better? A review and meta-analysis. Value Health Improving patient care: the implementation of change in health care Series: practical guidance to qualitative research. Part 3: sampling, data collection and analysis Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations We are grateful to the study sites and individual participants as well as the many students and colleagues who assisted with data collection, analysis, and preparation of the adapted PatientToc ™ and implementation toolkit items. We thank Lyndee Knox, Chief Executive Officer of PatientToc ™ , for assisting with site recruitment from L.A. Net primary care practices and participating in the Evidence-Based Quality Improvement process. The online version contains supplementary material available at https:// doi. org/ 10. 1186/ s43058-022-00277-3. Additional file 2. Data collection forms. Authors' contributions OAAO methodology, validation, formal analysis, investigation, resources, data curation, writing the original draft, writing, review, editing, visualization, supervision, and project administration. GMC conceptualization, methodology, validation, formal analysis, investigation, resources, writing, review, editing, supervision, project administration, and funding acquisition. HAJ conceptualization, methodology, software, validation, formal analysis, investigation, resources, data curation, writing, review, editing, visualization, supervision, project administration, and funding acquisition. LAH validation, formal analysis, investigation, resources, data curation, writing, review, editing, and visualization. NS investigation, resources, data curation, writing, review, and editing. BAC conceptualization, methodology, validation, formal analysis, writing, review, editing, supervision, and funding acquisition. DK conceptualization, formal analysis, writing, review, editing, and funding acquisition. JCS conceptualization, formal analysis, writing, review, editing, supervision, and funding acquisition. MMM conceptualization, writing, review, editing, and funding acquisition. SMP conceptualization, methodology, writing, review, editing, and funding acquisition. MES conceptualization, methodology, validation, formal analysis, investigation, resources, data curation, writing, review, editing, visualization, supervision, project administration, and funding acquisition. All authors read and approved the final manuscript and have agreed both to be personally accountable for the author's own contributions and to ensure that questions related to the accuracy or integrity of any part of the work, even ones in which the author was not personally involved, are appropriately investigated, resolved, and the resolution documented in the literature. The datasets generated and/or analyzed during the current study are not publicly available because individuals' privacy could be compromised. However, reasonable requests for deidentified datasets can be made to the corresponding author for consideration. This study (#1901010204) was granted exemption by the Indiana University Human Subjects Office, Office of Research Compliance. Not applicable. MES served as a consultant to Westat, Inc. on an evaluation of the CMS Enhanced MTM program from 2016 to 2020. All other authors declare that they have no competing interests.