key: cord-0737173-npvjr83y
authors: Harry, Melissa L.; Asche, Stephen E.; Freitag, Laura A.; Sperl-Hillen, JoAnn M.; Saman, Daniel M.; Ekstrom, Heidi L.; Chrenka, Ella A.; Truitt, Anjali R.; Allen, Clayton I.; O’Connor, Patrick J.; Dehmer, Steven P.; Bianco, Joseph A.; Elliott, Thomas E.
title: Human Papillomavirus vaccination clinical decision support for young adults in an upper midwestern healthcare system: a clinic cluster-randomized control trial
date: 2022-03-18
journal: Human vaccines & immunotherapeutics
DOI: 10.1080/21645515.2022.2040933
sha: b1906c2e4115613cd57b64fbbd9d4b4b52e73691
doc_id: 737173
cord_uid: npvjr83y

INTRODUCTION: Human papillomavirus (HPV) vaccination rates are low in young adults. Clinical decision support (CDS) in primary care may increase HPV vaccination. We tested the treatment effect of algorithm-driven, web-based, and electronic health record-linked CDS with or without shared decision-making tools (SDMT) on HPV vaccination rates compared to usual care (UC). METHODS: In a clinic cluster-randomized control trial conducted in a healthcare system serving a largely rural population, we randomized 34 primary care clinic clusters (with three clinics sharing clinicians randomized together) to: CDS; CDS+SDMT; UC. The sample included young adults aged 18–26 due for HPV vaccination with a study index visit from 08/01/2018–03/15/2019 in a study clinic. Generalized linear mixed models tested differences in HPV vaccination status 12 months after index visits by study arm. RESULTS: Among 10,253 patients, 6,876 (65.2%) were due for HPV vaccination, and 5,054 met study eligibility criteria. In adjusted analyses, the HPV vaccination series was completed by 12 months in 2.3% (95% CI: 1.6%–3.2%) of CDS, 1.6% (95% CI: 1.1%–2.3%) of CDS+SDMT, and 2.2% (95% CI: 1.6%–3.0%) of UC patients, and at least one HPV vaccine was received by 12 months in 13.1% (95% CI: 10.6%–16.1%) of CDS, 9.2% (95% CI: 7.3%–11.6%) of CDS+SDMT, and 11.2% (95% CI: 9.1%–13.7%) of UC patients. Differences were not significant between arms. Females, those with prior HPV vaccinations, and those seen at urban clinics had significantly higher odds of HPV vaccination in adjusted models. DISCUSSION: CDS may require optimization for young adults to significantly impact HPV vaccination. TRIAL REGISTRATION: clinicaltrials.gov NCT02986230, 12/6/2016.

Human papillomaviruses (HPV) contribute to 91% of cervical cancers and are the leading cause of anal, oropharynx, penile, vulvar, and vaginal cancers. 1, 2 In the U.S., HPV-associated cancers cost an estimated $2-8 billion annually, 3, 4 costs that have likely risen substantially. 5 A 9-valent HPV vaccine is available in the U.S. for the strains that most frequently cause cancer and genital warts and is recommended by the Advisory Committee on Immunization Practices (ACIP) for all individuals aged 11-26, with initial doses available at age 9. 2 Two doses (0, 6-12 month schedule) of the HPV vaccine are recommended for those aged 9-14, and three doses (0, 1-2, 6 month schedule) for those aged 15-26 or for those with immunocompromising conditions. 2 Early vaccination against HPV is recommended because HPV infections are commonly acquired after adolescents and young adults become sexually active. 6 While cases of HPV can clear naturally, 6 failed attempts at vaccinating adolescents still leads to a large group of unvaccinated and unprotected young adults. Adults aged 27-45 are eligible for the three dose schedule and are encouraged to have a shared decision-making conversation with their provider about taking the HPV vaccine series. 2 HPV vaccines are safe and adverse events are rare. [7] [8] [9] [10] [11] Yet only 49% of adolescents have completed an HPV vaccine series, 12 with lower rates in rural areas. 12, 13 Males make up a small portion of the vaccinated population, [14] [15] [16] [17] [18] although HPV coverage rates have increased in men aged 19-26 in recent years from 8.2% in 2014 to 26 .3% in 2018. 17, 18 Still, young men aged 22-26 have lower rates (21.8%) than those aged 19-21 (34.4%), 18 all of which are lower than the rates for women of comparable ages (19-26 = 52 .8%, 19-21 = 53.3%, 22-26 = 52.5%). 18 Vaccination rates are also lower in young adults than in adolescents. 16 HPV vaccine uptake has been hampered by a number of barriers for both adolescents and young adults, 19 such as knowledge gaps, [19] [20] [21] cost concerns, 19 and negative perceptions of the HPV vaccine. 19 Some individuals may still believe in common myths about HPV vaccines, such as: they are unsafe; evidence is lacking on their ability to prevent cancer; they have been inadequately tested or are too new; are not needed when Pap smears are available; or that the age range for vaccination starts too young. 22 Young adults may experience additional barriers, such as recent findings that married men, and married or partnered young adults in general, are less likely to be vaccinated. 14, 21 Education levels have also been shown to impact HPV vaccination series initiation in young adults, where those with college educations have significantly higher odds of starting the series than those with a high school diploma or less. 16 Young adults aged 18-26 that visited the office of a doctor in the last year have been shown to have significantly greater odds of starting the HPV vaccination series compared to those without any visits. 16 Yet others have reported that a major barrier to HPV vaccination uptake may be a lack of providers recommending HPV vaccination to eligible patients. 23 Furthermore, rural communities have been shown to lack resources and rural-focused interventions. 24 Interventions are needed, 24, 25 like clinical decision support (CDS) for patient and provider shared decision-making, as well as shared decision-making tools (SDMT), also referred to as decision aids, that may educate young adults and counter misinformation.

In primary care clinics in the U.S., primary care clinicians (PCC) provide access to cancer preventive services, such as HPV vaccination. They do so within time-limited visits, navigating what may be conflicting prevention guidelines and competing demands. [26] [27] [28] [29] [30] [31] [32] [33] [34] [35] [36] As we described in a recent paper, 36 CDS systems may take some of the burden off PCCs by utilizing algorithms and electronic health record (EHR) data to identify patients that are due or overdue for HPV vaccination, then providing patient-tailored and intervention specific CDS that patients and PCCs can use during shared decisionmaking discussions regarding how to proceed. [36] [37] [38] [39] [40] Randomized control trials (RCT) on HPV vaccination CDS or EHR prompts for adolescents aged 11-17 present mixed results. [41] [42] [43] [44] One study found no significant impact on vaccination rates in adjusted models. 41 Others found that compared to usual care (UC), CDS modestly increased vaccination rates for those ages 11-13 and age 14, 42, 43 and that clinician-focused CDS impacted HPV vaccination initiation and a familyfocused intervention promoted series completion. 44 Research suggests that decision aids, like SDMT geared toward patients, may also assist with vaccine uptake. Scalia and colleagues recently conducted a systematic review of RCTs testing shared decision-making skills training and/or decision aid interventions on vaccination uptake and other patient outcomes. 45 This review included three studies related to HPV vaccination. 45 However, only two of these three studies used a decision aid, one aimed at mothers of adolescent females and another at parents of adolescents aged 9-18, 46,47 the latter of which included a supplemental file with screenshots of the decision aid that noted it was also for young adults up to age 26, although results for young adults were not included in the paper. 47 Research is needed on the impact of HPV vaccination CDS and SDMT for young adults aged 18-26. We designed webbased, algorithm-driven (following ACIP guidelines), and EHR-linked CDS for young adults due for HPV vaccination seen in primary care, a CDS that included other primary and secondary cancer prevention and screening items and cardiovascular risk reduction areas for eligible patients. 35 In addition, we developed a multi-page and HPV-specific SDMT for young adults aged 18-26. 31,35 The primary objective of this study was to test the treatment effect of HPV CDS with or without HPV SDMT on the percentage of young adult primary care patients aged 18-26 who completed the HPV vaccination series within 12 months after a study index visit in the CDS intervention arm clinics (one arm with and one arm without SDMT) compared to UC clinics. A secondary objective was to test the treatment effect of CDS with or without SDMT on additional HPV vaccinations received during the 12-month follow-up period for young adults without a complete course of HPV vaccinations at the study index visit. We hypothesized that compared to UC, by 12 months intervention arm patients would have significantly higher rates of: 1) completing the HPV vaccine series; and 2) having additional HPV vaccinations. We also tested the exploratory hypothesis that rates differed between intervention arms (CDS versus CDS +SDMT).

We conducted a three-arm, parallel group, primary care clinic cluster-RCT at Essentia Health, an integrated healthcare system with clinics in Minnesota, North Dakota, and Wisconsin that serve a large rural Upper Midwestern population. Thirtyfour clinics (clusters) (with three clinics sharing PCCs randomized together) were randomly allocated in a 1:1:1 ratio to one of three arms: CDS; CDS+SDMT; and UC ( Figure 1 ). 30, 32, [34] [35] [36] The original 18-month patient accrual period planned for this study was shortened to 7.5 months due to disruptions in healthcare, and vaccinations, from the SARS-CoV-2 pandemic. Shortening the accrual period to 7.5 months (8/01/2018-3/15/ 2019) allowed us to maintain the planned 12-month follow-up period for each index visit.

The Essentia Health Institutional Review Board reviewed, approved, and monitored this study, and granted a waiver of informed consent. An Independent Project Safety Officer monitored this study.

The study population included patients receiving care from PCCs (physicians, nurse practitioners, and physician assistants) practicing in randomized primary care clinics. Eligibility criteria included being ages 18-26 and lacking a completed HPV vaccination series at the time of the index visit (the first visit at which a patient was eligible for HPV vaccination during the study accrual period) at a study clinic. The CDS algorithms determined patient eligibility for HPV vaccination from 8/01/2018-3/ 15/2019, and these patients served as the denominator for the analyses. Exclusion criteria included: not meeting eligibility criteria; pregnancy either at the time of the index visit or in the 12 months prior to the index visit; and having hospice, Alzheimer's/cognitive impairment, or non-skin cancer codes in the EHR in the past 12 months. 35 

The CDS archived and stored EHR data for each visit in a protected data repository on or after the index visit. Stored data included patient age, sex, race, tobacco use, current medications, diagnosis and procedure codes, lab results, number of office visits, exclusion criteria, vaccinations completed, and vaccinations due. A separate EHR data pull provided patient ethnicity and insurance type. Regarding payment of HPV vaccinations in the study population, the Patient Protection and Affordable Care Act (ACA) requires that health insurance plans and issuers cover the entire cost of the HPV vaccination series for populations recommended for vaccination by the ACIP in the U.S. 2, 48 However, some insurance types, like workers compensation and automobile accident insurance, may only cover costs associated with the covered injury and not preventive care. Patients lacking insurance would not have had coverage for preventive care like HPV vaccinations. The HPV vaccine was not provided for free as part of the RCT. Clinic descriptors used as control variables in adjusted analyses were obtained from healthcare system administrative records and clinic-level EHR summary data for patient visits occurring prior to randomization (10/2014-3/2016). We captured followup data for 12 months after each patient's index visit between 8/1/2018-3/15/2020, ending the day before regional SARS-CoV -2 restrictions were announced.

Binary dependent endpoints (Yes/No) included: 1) HPV vaccination series completion; and 2) receipt of one or more HPV vaccine doses during the 12-month follow-up period. Receipt of three doses was considered vaccine series completion. 2 Outcomes were derived from manually triggered programmatic EHR data extractions capturing all EHR care recorded from 8/1/2018-3/15/2020. Because EHR data may be out of date, additional binary composite endpoints defined success as: HPV vaccination series completion (or receipt of one or more HPV vaccine doses) during the 12-month followup and/or new documentation in the EHR during follow-up that the HPV vaccination series was completed prior to the index visit, but was not documented at the time of the index visit. These binary composite endpoints were exploratory, rather than planned.

The CDS was programed to provide primary (HPV vaccination, tobacco cessation, weight management) and secondary (breast, cervical, colorectal, and lung) cancer prevention recommendations for patients at average risk meeting eligibility criteria (e.g., not up to date on HPV vaccination or breast, cervical, colorectal, or lung cancer screening; tobacco use; obesity). 35 The cancer prevention and screening CDS areas, which followed United States Preventive Service Task Force (USPSTF) [49] [50] [51] [52] and ACIP guidelines, 2 were added to a cardiovascular risk reduction and glycemic control CDS that was already available in study intervention clinics as part of two other National Institutes of Health-funded studies. 30, 31, 35 For the patients in the present study, who were aged 18-26, cancer prevention and screening CDS components were limited to: HPV vaccination; tobacco cessation and weight management if either applied and patients were overdue for HPV vaccination; and cervical cancer screening CDS Figure 1 . Study CONSORT diagram. † Ineligible for analysis includes patients with delayed documentation of screening (n = 460) and those whose age at follow-up was higher than ages included in algorithms (n = 694).

recommendations for women aged 18 to 26 who were due or overdue for cervical cancer screening. 35 However, there was no SDMT related to cervical cancer screening in this study. 35 The CDS included patient and PCC printed handouts and an electronic EHR interface for PCCs. 30, 31, [34] [35] [36] Rooming staff triggered the CDS by entering patients' vital signs into the EHR. CDS algorithms used EHR data to identify and display an alert for overdue patients. We instructed rooming staff to print the patient and PCC handout(s), give the patient handout to the patient, and place the PCC handout on the exam room door or hand it to the PCC. CDS materials were to be printed regardless of the reason for the visit. For patients eligible for HPV vaccination, patient versions of the CDS printout stated: "Talk to your doctor about whether you are due for an HPV vaccine," and the PCC printout stated: "HPV vaccine may be due," both of which were under a "Cancer Prevention" heading. Examples of the CDS printouts are included in a recent paper by Elliott et al. 35 The CDS included a two-week suppression window after each eligible visit to prevent alert fatigue. 31, 35 In the CDS+SDMT intervention arm, patients also received abbreviated printed information on HPV vaccination for use in shared decision-making (see Appendix 1). 31, 35 The electronic EHR interface gave PCCs access to a multipage HPV vaccination SDMT in the CDS+SDMT intervention arm that PCCs could print for patients (see Appendix 2) . The multi-page SDMT described basic information on HPV and HPV vaccines, HPV-associated genital warts and cancers, vaccine risks and benefits, and statements that patients could consider when deciding whether to take the vaccine. The multi-page SDMT was originally to be printed with the CDS for eligible patients; however, we found during pilot testing of the CDS and SDMT (including breast, colorectal, and lung cancer SDMT for patients eligible for those screenings) in two non-RCT healthcare system clinics that the amount of paper printed was a burden on clinic workflow, resulting in the CDS+SDMT intervention arm instead automatically printing the abbreviated HPV vaccination information along with the CDS patient and PCC printouts for eligible patients. UC clinics did not receive the CDS or SDMT but already had a PCC EHR alert for patients eligible for HPV vaccination. Unlike the current EHR alert, which only alerted the PCC that a patient was due for an HPV vaccination, the CDS alert provided printed material for both eligible patients and PCCs. Intervention clinic PCCs were also encouraged during training to use the CDS during a shared decision-making discussion with the patient about taking the vaccine series. In the CDS+SDMT arm, patients also received the abbreviated SDMT on HPV vaccination, and PCCs could choose to print the multi-page SDMT through the electronic CDS interface that they could then give to patients.

Rooming staff, PCC, and clinic management training was multi-pronged and ongoing. 31, 35 We developed an e-learning module that was assigned to intervention clinic staff using the e-learning platform utilized by the healthcare system for disseminating system-wide training. 31 We also conducted baseline and booster in person training sessions at clinics, 35 where members of the research team demonstrated the CDS (and the SDMT for the CDS+SDMT intervention arm clinics) and talked about how the CDS aligned with institutional aspirational aims, quality measures, team model of care, and was based on the latest recommendations from respected groups like the USPSTF and ACIP. 31 Furthermore, we conducted two training webinars that were recorded and made available to intervention clinics. 31 Research team project managers regularly communicated with clinic management and provided individualized monthly update reports to each intervention clinic showing CDS print rates for the clinic and individual PCCs. 31 The team also initiated a lunch and learn incentive program for clinics that met print rate goals for sustained periods. A "suggested improvements" button was built into the CDS and users were encouraged to reach out to research team project managers when any issues were encountered with the CDS. 30

We based power calculations on preliminary data from 11,970 patients ages 18-26 eligible for HPV vaccination with visits to study clinics over 18 months. Using original power calculation assumptions, but with an updated sample size of 5,054 reflecting patient accrual over 7.5 months, this study was powered at 80% (2-sided test p < .025) to detect a difference in the percentage of patients up-to-date for the composite screening endpoint of 20% in either intervention arm compared to 30% in UC. Power calculations assumed 10 clinics per study arm with equal patient counts (n = 168/clinic), ICC = .02 for the primary endpoint, and a generalized linear mixed model for analysis.

Analyses included descriptive statistics and unadjusted and adjusted generalized linear mixed models with a logit link and binomial error distribution in testing primary and secondary hypotheses. Adjusted models included two terms specifying the planned primary contrasts of CDS versus UC and CDS+SDMT versus UC, pre-specified covariates, and a random intercept for clinic. Clinic-level covariates included the percentage of patients up-to-date on breast cancer screening prior to intervention start (representing a measure of the attentiveness of the clinic to screening) and clinic rurality based on primary Rural-Urban Commuting Area codes (urban 1-6 [metropolitanmicropolitan], rural 7-10 [small town-rural]). 53 These two covariates were chosen due to their inclusion in the clinic randomization process. 54 Patient-level covariates included age at index visit, sex (male/female), race, number of HPV vaccinations as of the index date, and insurance type (private insurance or Medicare for those eligible for Medicare; public payors Medicaid, Federal, or Indian Health Services; unknown or other insurance types [workers compensation, government other, automobile accident insurance, program, other]). Exploratory analyses contrasted the CDS and CDS+SDMT study arms. Tests of the two planned contrasts were two-sided with p-values less than .025 considered statistically significant. Due to multiple comparisons, we report 97.5% confidence intervals (CIs) for odds ratio estimates for key study arm contrasts, and 95% CIs for all other estimates. Heterogeneity of treatment effects analyses tested differences in the magnitude of the intervention effect by patient sex with the inclusion of a study arm by patient sex interaction term in the main analytic models.

In total, 10,543 patients aged 18-26 had index visits at study clinics over the 7.5-month accrual period and were assessed for eligibility for HPV vaccination by the CDS algorithms ( Figure 1 ). Of these patients, 6,876 (65%) were due for HPV vaccination (e.g., were eligible for the HPV vaccination per ACIP guidelines for adults aged 18-26 and had either no prior HPV vaccination doses on record in the EHR or were due for another dose in the series). After exclusions, the analytic sample for the primary analysis concerning completion of an HPV vaccination series within the 12-month follow-up period consisted of 5,054 patients, of whom 5.6% completed the vaccine series (6.0% CDS, 4.7% CDS+SDMT, 6.1% UC). Of those 5,054 patients, 3,910 (77.4%) had no HPV vaccinations at the time of the index visit, while 604 (12.0%) had one HPV vaccination dose, and 540 (10.7%) had two HPV vaccination doses (Table 1) . Those without any HPV vaccination doses at the index visit had the lowest rate (1%) of completing the vaccine series by 12 months, while those with one dose and two doses at the index visit had completion rates of 8% and 34%, respectively.

Patients included in the analysis had a mean age of 21.5 (standard deviation [SD] = 2.4) years, 43.0% were women, 91.5% were White, 3.4% were African American, 1.8% were American Indian/Alaska Native, .9% were Asian, .3% were Native Hawaiian/Other Pacific Islander, 2.4% were Hispanic, 15.3% were current smokers, and 22.1% had an outpatient visit in the year prior to the index visit ( Table 1 ). The majority of patients (90%) had insurance that would have covered HPV vaccination costs under the ACA, and the other 10% may or may not have had HPV vaccination coverage under the ACA. The mean time from the index visit to the 12-month follow-up data pull was 382.2 days (SD = 8.1). Table 1 also presents clinic-level summary information for the randomized clinics. The clinics' aggregated attributes were similar across the three study arms. More clinics (74%) were designated rural than urban (26%).

We documented CDS activation and printing by rooming staff and PCCs. They opened the electronic interface for 45.4% of eligible patients in the CDS arm and 58.6% in the CDS+SDMT arm, and printed materials for 43.8% of eligible patients in the CDS arm and for 58.2% in the CDS +SDMT arm.

Among 5,054 patients due for HPV vaccination, 285 (5.6%) completed their HPV vaccination series by 12 months. The adjusted model-derived percentage of patients with a completed vaccination series was 2.3% (95% CI = 1.6%-3.2%) in the CDS arm, 1.6% (95% CI = 1.1%-2.3%) in the CDS+SDMT arm, and 2.2% (95% CI = 1.6%-3.0%) in the UC arm ( Table 2 ). The adjusted odds ratio was 1.04 (97.5% CI = .68-1.59, p = .827) for the planned contrast comparing the CDS versus UC arms and was .71 (97.5% CI = .46-1.12, p = .086) for the CDS+SDMT versus UC arms. Neither met the pre-specified alpha of .025 in unadjusted or adjusted analyses. In the adjusted analysis of the composite endpoint examining a completed HPV vaccination series and/or new documentation of a completed HPV vaccination series prior to the index date, the model-derived percentage for the endpoint ranged from 9.2% in the CDS+SDMT arm to 11.3% in the CDS arm, with no statistically significant differences found. The adjusted odds ratio for the exploratory contrast comparing vaccination completion in the CDS versus CDS+SDMT arms was 1.46 (95% CI = .96-2.21, p = .077) (not shown in Table 2 ).

Patients in urban clinics had significantly higher odds of completing the HPV vaccine series than those in more rural clinics (aOR = 1.75, 95% CI = 1.28-2.39, p = .001), as did females versus males (aOR = 2.55, 95% CI = 1.92-3.40, p <.001) ( Table 3 ). Compared to those with private insurance or Medicare, those with unknown or other insurance types had significantly lower odds of completing the series (aOR = .30, 95% CI = .15-.62, p = .001). Compared to patients with no vaccinations at the time of the index date, those with one vaccination dose were significantly more likely to complete the series (aOR = 5.70, 95% CI = 3.80-8.55, p < .001) as were those with two vaccination doses (aOR = 34.56, 95% CI = 24.74-48.29, p <.001).

A heterogeneity of treatment effects analysis assessing differences in the magnitude of the intervention effect on completion of the HPV vaccine series by patient sex indicated that the CDS arm versus UC arm comparison did not differ by patient sex (women: aOR = 1.02, 95% CI = .67-1.57; men: aOR = 1.07, 95% CI = .61-1.88; interaction p = .893) and that the CDS+SDMT arm versus UC arm comparison did not differ by patient sex (women: aOR = .71, 95% CI = .45-1.11; men: aOR = .73, 95% CI = .41-1.31; interaction p = .925).

Among 5,054 patients due for HPV vaccination, 754 (14.9%) had received at least one HPV vaccination during the 12month follow-up. As shown in Table 2 , the adjusted modelderived percentage of patients with at least one HPV vaccination was 13.1% (95% CI = 10.6%-16.1%) in the CDS arm, 9.2% (95% CI = 7.3%-11.6%) in the CDS+SDMT arm, and 11.2% (95% CI = 9.1%-13.7%) in the UC arm. The adjusted odds ratio was 1.19 (97.5% CI = .82-1.74, p = .282) for the planned contrast comparing the CDS versus UC arms and was .80 (97.5% CI = .55-1.18, p = .191) for CDS+SDMT versus UC. Neither unadjusted nor adjusted analyses met the prespecified alpha of .025. The adjusted odds ratio for the exploratory contrast comparing at least one HPV vaccination in the CDS versus CDS+SDMT arms was 1.48 (95% CI = 1.05-2.10, p = .028) (not shown in Table 2 ).

In the adjusted analysis of the composite endpoint examining any HPV vaccination and/or new documentation of a completed HPV vaccination series prior to the index date, Table 1 . The effect of clinical decision support with or without shared decision-making tools on HPV vaccination compared to usual care in young adults ages 18-26 in the 12 months following a study index visit from 08/01/2018-03/15/2019 in randomized primary care study clinics: patient characteristics (N = 5,054) and study clinic characteristics (N = 34) by study arm. SDMT = Shared decision-making tool. UC = Usual care. aPredicted percentages from a generalized linear mixed model. bUnadjusted models include fixed effects for study arm and a random intercept for clinic. cAdjusted models include fixed effects for study arm, clinic-level (urbanicity, percent of patients up-to-date on breast cancer screening at baseline), patient-level (age, sex, race, insurance type, number of completed vaccinations as of index date), and a random intercept for clinic. dHPV vaccination series complete and/or new documentation of HPV vaccination series completed prior to the index date. eAny HPV vaccination and/or new documentation of HPV vaccination series completed prior to the index date. fAdjusted models include fixed effects for study arm, clinic-level (urbanicity, percent of patients up-to-date on breast cancer screening at baseline), patient-level (age, sex, race, insurance type), and a random intercept for clinic. the adjusted, model-derived percentages for the endpoint were as follows: 21.1% (95% CI = 18.2%-24.4%) in the CDS arm; 16.3% (95% CI = 13.7%-19.1%) in the CDS+SDMT arm; and 18.6% (95% CI = 16.1%-21.4%) in the UC arm (Table 2 ). There were no statistically significant differences seen by study arm. Patients in urban clinics had significantly higher odds of at least one HPV vaccination during the 12-month follow-up period than those in rural clinics (aOR = 1.48, 95% CI = 1.12-1.96, p = .008) ( Table 3) . Females also had significantly higher odds than males (aOR = 2.27, 95% CI = 1.91-2.69, p <.001), and a one-year increase in age was associated with a 14% decrease in the odds that patients would have at least one new HPV vaccination (aOR = .86, 95% CI = .83-.89, p < .001). Compared to white patients, those with unknown race were more likely to have at least one HPV vaccination (aOR = 1.81, 95% CI = 1.09-3.00, p = .023). Patients with unknown or other insurance types had lower odds compared to those with private insurance or Medicare (aOR = .44, 95% CI = .30-.64, p < .001). Compared to patients with no vaccination doses at the time of the index date, those with one (aOR = 4.36, 95% CI = 3.53-5.37, p < .001) or two (aOR = 3.97, 95% CI = 3.20-4.94, p < .001) vaccination doses had higher odds of at least one additional vaccination during follow-up.

A heterogeneity of treatment effects analysis assessing differences in the magnitude of the intervention effect on receipt of at least one HPV vaccination by patient sex indicated that the CDS arm versus UC arm comparison did not differ by patient sex (women: aOR = 1.23, 95% CI = .86-1.75; men: aOR = 1.15, 95% CI = .77-1.70; interaction p = .739) and that the CDS+SDMT arm versus UC arm comparison did not differ by patient sex (women: aOR = .77, 95% CI = .53-1.11; men: aOR = .86, 95% CI = .57-1.29; interaction p = .595).

In a clinic cluster-RCT of a patient-tailored, point-of-care CDS intervention for HPV vaccination in young adults aged 18-26, we found no statistically significant impact on HPV vaccination in either the CDS or the CDS+SDMT intervention arms compared to UC in the 12 months following a study index visit. HPV vaccination rates were low across all study arms. Patients had Table 3 . The effect of clinical decision support with or without shared decision-making tools on HPV vaccination compared to usual care in young adults ages 18-26 in the 12 months following a study index visit from 08/01/2018-03/15/2019 in randomized primary care study clinics: independent variables in the adjusted models presented in Table 2 slightly higher predicted percentages of having at least one HPV vaccination compared to completing the full series. In adjusted models, we found that having one and two doses already completed at the study index visit significantly increased the odds that patients would complete the vaccine series during follow-up compared to those without any initial vaccine doses. We also found that females were more likely than males to complete the full series and have at least one HPV vaccination during followup. This is similar to other research comparing HPV vaccination uptake by sex. [14] [15] [16] [17] [18] 55, 56 However, we found no difference in the magnitude of the intervention effect on HPV vaccination uptake after adding an interaction term between study arm and patient sex to adjusted models. As previously reported, 12, 13 patients seen in rural clinics had lower likelihoods of vaccination than those in urban clinics. Younger patients in the study were also more likely to receive an additional vaccination dose, similar to prior research. 16, 18 Our finding that patients with other types of health insurance had lower HPV vaccination odds is unsurprising, as some of these insurance types (e.g., workers compensation and automobile accident insurance) would not be required to cover unassociated visit costs by the ACA. Of note, only 1.2% of participants had other types of insurance in the present study.

We observed that the arm receiving the CDS+SDMT intervention had directionally lower rates of HPV vaccination than those receiving the CDS alone, and significantly lower rates of receipt of at least one additional dose, although intervention print rates were 14.8% higher in the CDS+SDMT arm than in the CDS arm. PCCs reported higher satisfaction with the CDS +SDMT intervention than the CDS intervention alone (CDS arm = 75% very/somewhat satisfied; CDS+SDMT arm = 83.6% very/somewhat satisfied) in a recently published study survey. 36 This suggests the need for a better understanding of how SDMT affect HPV vaccination decisions in patientcentered care.

Unfortunately, RCT on CDS and SDMT for HPV vaccination amongst young adults seen in a general primary care population are rare. While some prior research supported CDS and reminder prompts for HPV vaccination with adolescents, [42] [43] [44] other research showed similar results as the present study. One study presented conflicting results, with generally low HPV vaccination rates and lower rates in a CDS intervention group for children aged 13 compared to historical UC. 42 Yet in the same study, the last year of the CDS intervention period included "4 of the 5 highest quarterly rates" (p. 2) of HPV vaccination series completion for children aged 14. 42 Also, a recent RCT of an automated EHR reminder prompting clinicians when patients were due for a second or third HPV vaccination dose reported the intervention was associated with significantly higher vaccination rates than UC in unadjusted models for adolescents aged 11-17. 41 Nevertheless, adjusted models showed no significant differences in vaccination rates. 41 Factors other than CDS or EHR reminders likely impacted HPV vaccination uptake. For example, recent research suggests that renewed efforts at public health campaigns on HPV vaccination in the U.S. may be needed, as the awareness of HPV and HPV vaccination has dropped in recent years both overall and to a greater extent within some groups, such as adult males, individuals living rurally, and racial minority groups. 57

Our sample, and the healthcare system population, was over 90% white, limiting generalizability to more diverse populations. We excluded patients aged 11-17, who received a separate EHR alert. We shortened the accrual period to remove the impact of the SARS-CoV-2 pandemic adversely impacting vaccinations, although power was maintained. Consequently, we were unable to examine how the CDS and the CDS with SDMT may have impacted vaccination rates over a longer accrual period (e.g., 18 months) or following the resumption of in person healthcare visits during the SARS-CoV-2 pandemic. We did not restart accrual after the end of regional healthcare restrictions, as telemedicine visits became common within the healthcare system and vaccination rates declined institutionally. We also lacked data on HPV vaccination completion outside of the healthcare system either before or after an index visit that was not updated in the healthcare system's EHR in the 12-month follow-up period after an index visit.

Pragmatic clinical care trials provide evidence of how interventions function in real world practice, but have weaknesses that may affect robustness, like noncompliance. 58, 59 In a recent survey of PCCs practicing in intervention arm clinics in this study, majorities of PPCs agreed that the CDS alerted them to patients they did not know were overdue for a cancer screening (57% in the CDS arm, 68% in the CDS+SDMT arm); however, PCCs gave low self-reported CDS use rates, with only 18% of CDS arm and 19% of CDS+SDMT arm PCCs responding that they always or usually used the CDS paper printouts, with even fewer (12% and 20%, respectively) agreeing that they always or usually used the electronic version of the CDS. 36 In addition, intervention clinics experienced a variety of printing issues during the study, primarily related to printer driver, firmware, and network connection errors, which may have adversely affected CDS use and fidelity to the intervention. 30, 31, 34 However, research team members and healthcare system information services troubleshot these issues. 30, 31, 34 The CDS alone may also have lacked the strength to alter vaccination noncompliance in young adults, as it only notified patients and PCCs that patients were due. Furthermore, preimplementation interviews with PCCs suggested some PCCs had doubts about the HPV vaccine. 30 Yet qualitative interviews with 37 patients in the present RCT's CDS intervention arms showed that patients found many benefits in the CDS for cancer prevention and screening generally. 60 The qualitative results from those interviews also suggested that patients were more likely to make a cancer prevention and screening decision when their PCC reviewed the CDS with them than when the PCC did not use the CDS in the appointment, presenting an area for future research. 60 Previous research has highlighted how providers recommending HPV vaccination can significantly increase HPV vaccination uptake. 61 Lack of use of the CDS within intervention arm clinics may be the main factor impacting our non-significant findings, as suggested by our recently published PCC survey paper. 36 The results of this survey suggested that relying on print rates alone does not provide enough information on if and how the CDS and SDMT were used in practice. As we reported in a previous paper, of the four patients aged 18-26 due for an HPV vaccine and seen in intervention arm clinics who took part in qualitative interviews, one received an HPV vaccination, one did not make a choice, and two expressed opinions against HPV vaccination and chose not to receive the vaccine. 60 Three of the four also responded that they discussed HPV vaccination with their PCC during the visit. 60 However, capturing how PCCs and patients used the CDS and SDMT for HPV vaccination within intervention clinics was beyond the scope of the present study. Future research is needed to understand how often and in what ways PCCs and patients utilize these sorts of tools during clinic visits and how that use may impact HPV vaccination rates for young adults.

Another limitation to this study is that not all clinics met the target print rate, and one intervention clinic stopped printing the CDS at times during the accrual period. Workflows could also differ by clinic. 31 For example, while a few clinics had printers within exam rooms, most healthcare system clinics in this study had centralized printers, and printing the CDS required rooming staff to leave the room in these cases, retrieve the printed materials, and bring them back to the patient's exam room, a process that may not have been followed consistently in practice.

When reviewing internal CDS use tracking in our RCT, we found that the multi-page HPV vaccine SDMT available for PCCs to print electronically in the CDS+SDMT intervention arm was rarely accessed and printed. If patients had had direct access to the multi-page SDMT, the SDMT may have assisted with addressing previously identified knowledge gap barriers to HPV vaccination. 19, 20 The SDMT may also have helped dispel myths regarding the HPV vaccine, 22 as was suggested by healthcare system PCCs in key informant interviews conducted prior to HPV CDS and SDMT implementation. 30 Ensuring education on HPV and HPV vaccination is received by young adults, for example through brief educational interventions, may help improve vaccination uptake in this group. 62 As more clinical care moves online, embedding CDS and SDMT within electronic patient portals would give patients more control of the receipt of these tools and the ability to review materials outside of appointments. Mobile health (mHealth) also has the potential to increase HPV vaccination in both adolescents and young adults. 63 However, more replication research is needed in broader populations to adapt and maximize mHealth and other electronic direct-to-patient approaches for disseminating HPV vaccination information to young adults. 63 Research could also include young adults and clinic staff in the development of electronic tools and decision aids aimed at increasing HPV vaccination uptake.

HPV vaccination, a clinically effective strategy for reducing cancer burden, ranks above average in terms of population health impact and cost-effectiveness among other preventive services. 64 We identified large gaps in the percentages of young adults aged 18-26 not up-to-date on HPV vaccination in the 12 months following a study index visit in primary clinics within one Upper Midwestern healthcare system. Unfortunately, a point-of-care CDS with or without SDMT did not significantly improve HPV vaccine uptake in young adults compared to UC. Future research could assess whether alternative CDS content, a more engaging format, or a different workflow strategy would be more effective.

US Department of Health and Human Services; 2020 Sept

Human papillomavirus vaccination for adults: updated recommendations of the Advisory Committee on Immunization Practices

Estimates of the annual direct medical costs of the prevention and treatment of disease associated with human papillomavirus in the United States

The estimated direct medical cost of selected sexually transmitted infections in the United States

Updated medical care cost estimates for HPV-associated cancers: implications for cost-effectiveness analyses of HPV vaccination in the United States

Human papillomavirus vaccination: recommendations of the Advisory Committee on Immunization Practices (ACIP)

Autoimmune, neurological, and venous thromboembolic adverse events after immunisation of adolescent girls with quadrivalent human papillomavirus vaccine in Denmark and Sweden: cohort study

Quadrivalent HPV vaccine safety review and safety monitoring plans for nine-valent HPV vaccine in the United States

Association between human papillomavirus vaccination and serious adverse events in South Korean adolescent girls: nationwide cohort study

HPV vaccination and risk of chronic fatigue syndrome/myalgic encephalomyelitis: a nationwide register-based study from Norway

Rural-Urban differences in human papillomavirus vaccination among young adults in 8 U.S. states

Human papillomavirus (HPV) vaccination initiation and completion among adult males in the United States

Prevalence of genital human papillomavirus infection and human papillomavirus vaccination rates among US adult men

Correlates of human papillomavirus (HPV) vaccination initiation and completion among 18-26 year olds in the United States

Surveillance of vaccination coverage among adult populations -United States

Surveillance of vaccination coverage among adult populations -United States

Barriers to and facilitators of human papillomavirus vaccination among people aged 9 to 26 years: a systematic review

A significant portion of college students are not aware of HPV disease and HPV vaccine recommendations

HPV literacy and its link to initiation and completion of HPV vaccine among young adults in Minnesota

Addressing HPV vaccine myths: practical information for healthcare providers

National HPV Vaccination Roundtable. Rural disparities in HPV vaccination coverage

A narrative review of HPV vaccination interventions in rural U.S. communities

Primary care: is thereenough time for prevention?

It's about time: physicians' perceptions of time constraints in primary care medical practice in three national healthcare systems

Primary care physicians' cancer screening recommendation practices and perceptions of cancer risk of Asian Americans

Understanding primary care provider attitudes and behaviors regarding cardiovascular disease risk and diabetes prevention in the Northern Midwest

Barriers and facilitators to implementing cancer prevention clinical decision support in primary care: a qualitative study

Pre-Implementation adaptation of primary care cancer prevention clinical decision support in a predominantly rural healthcare system

Understanding primary care providers' perceptions of cancer prevention and screening in a predominantly rural healthcare system in the upper Midwest

Cancer prevention in primary care: perception of importance, recognition of risk factors and prescribing behaviors

The impact of personalized clinical decision support on primary care patients' views of cancer prevention and screening: a cross-sectional survey

Design and rationale of an intervention to improve cancer prevention using clinical decision support and shared decision making: a clinic-randomized trial

Primary care clinicians' opinions before and after implementation of cancer screening and prevention clinical decision support in a clinic clusterrandomized control trial: a survey research study

Effect of clinical decision-support systems: a systematic review

An overview of clinical decision support systems: benefits, risks, and strategies for success

Improving outcomes with clinical decision support: an implementer's guide. 2 nd . Chicago: IL: HIMSS publishing

Priorities wizard: multisite web-based primary care clinical decision support improved chronic care outcomes with high use rates and high clinician satisfaction rates

Physician clinical decision support system prompts and administration of subsequent doses of HPV vaccine: a randomized clinical trial

Improving immunization delivery using an electronic health record: the ImmProve project

Simple and elaborated clinician reminder prompts for human papillomavirus vaccination: a randomized control trial

Effectiveness of decision support for families, clinicians, or both on HPV vaccine receipt

Shared decision-making interventions: an overview and a meta-analysis of their impact on vaccine uptake

Effectiveness of a web-based tailored intervention with virtual assistants promoting the acceptability of HPV vaccination among mothers of invited girls: randomized controlled trial

Effect of a health care professional communication training intervention on adolescent human papillomavirus vaccination: a cluster randomized clinical trial

Patient Protection and Affordable Care Act, Pub. Law 111-148, as amended by the Health Care and Education Reconciliation Act (HCERA). Pub. Law 111-52

Preventive Services Task Force. Final recommendation statement: Breast cancer: Screening

Screening for colorectal cancer: US Preventive Services Task Force recommendation statement

US Preventive Services Task Force. Screening for lung cancer: U.S. Preventive Services Task Force recommendation statement

Rural-Urban Commuting Area Codes

A review of the use of covariates in cluster randomized trials uncovers marked discrepancies between guidance and practice

Gender and sexual orientation differences in human papillomavirus (HPV) vaccine uptake among Chinese young adults

MF 2021 Race, nativity, and sex disparities in human papillomavirus vaccination among young adults in the USA

S 2021 Declining awareness of HPV and HPV vaccine within the general US population Hum Vaccin Immunother 17

A pragmatic view on pragmatic trials

The cohort multiple randomized controlled trial design: a valid and efficient alternative to pragmatic trials?

Patient perceptions of using clinical decision support for cancer screening and prevention: "I wouldn't have thought about getting screened without it

Predictors of HPV vaccine uptake among women aged 19-26: importance of a physician's recommendation

The effects of a brief educational intervention on human papillomavirus knowledge and intention to initiate HPV vaccination in 18-26 year old young adults

Can mHealth interventions contribute to increased HPV vaccination uptake? A systematic review

Updated priorities among effective clinical preventive services

The authors thank Essentia Health and HealthPartners Institute, and Austin Land, Essentia Health Research Informatics Analyst II, for assistance with data collection and transfer.The research presented in this paper is that of the authors and does not reflect the official policy of the NIH.

No potential competing interest was reported by the authors.

This work was supported by the National Cancer Institute under Grant R01CA193396.

Melissa L. Harry http://orcid.org/0000-0003-2016-4398