key: cord-0768734-pp1inpmx
authors: Gruenberg, Katherine; Brock, Tina; Garcia, Joshua; MacDougall, Conan
title: A Randomized, Crossover Pilot Study of a Novel Web-Based/Mobile Platform for Collaborative Small Group Practice in Therapeutic Reasoning
date: 2020-11-26
journal: J Med Educ Curric Dev
DOI: 10.1177/2382120520977189
sha: 14799d65ef7d235c049593b409d2afe0a477f8eb
doc_id: 768734
cord_uid: pp1inpmx

BACKGROUND AND PURPOSE: Therapeutic reasoning—the mental process of making judgments and decisions about treatment—is developed through acquisition of knowledge and application in actual or simulated experiences. Health professions education frequently uses collaborative small group work to practice therapeutic reasoning. This pilot study compared the impact of a web-based/mobile tool for collaborative case work and discussion to usual practice on student perceptions and performance on questions designed to test therapeutic knowledge and reasoning. METHODS: In a therapeutics course that includes case-based workshops, student teams of 3 to 4 were randomly assigned to usual workshop preparation (group SOAP sheet) or preparation using the Practice Improvement using Virtual Online Training (PIVOT) platform. PIVOT was also used in the workshop to review the case and student responses. The next week, groups crossed over to the other condition. Students rated favorability with the preparatory and in-workshop experiences and provided comments about the PIVOT platform via a survey. Student performance on examination items related to the 2 workshop topics was compared. RESULTS: One hundred and eleven students (94%) completed post-workshop surveys after both workshops. The majority of students (57%) preferred using the PIVOT platform for workshop collaboration. Favorability ratings for the in-workshop experience did not change significantly from first to second study week, regardless of sequence of exposure. There was no relationship between examination item scores and the workshop platform the students were exposed to for that content (P = .29). Student responses highlighted the efficiency of working independently before collaborating as a group and the ability to see other students’ thought processes as valuable aspects of PIVOT. Students expressed frustration with the PIVOT user interface and the lack of anonymity when discussing their answers in the workshop. CONCLUSION: A web-based/mobile platform for student team collaboration on therapeutic reasoning cases discussed in small group settings yielded favorable ratings, examination performance comparable to standard approaches, and was preferred by a majority of students. During the rapid shift to substantial online learning for the COVID-19 pandemic, virtual collaboration tools like PIVOT may help health professions teachers to better support groups working virtually on scaffolded therapeutic reasoning tasks.

Journal of Medical Education and Curricular Development distance learning, and are of especial interest given the physical distancing measures enacted due to the COVID-19 pandemic. 13, 14 One such platform is Practice Improvement using Virtual Online Training (PIVOT), a web-based and mobile application that provides case information in a simulated clinical learning environment. This program allows students to first gather, assess, and respond to patient information independently. After completing this step, students can see and comment on other students' responses using the PIVOT discussion feature. In previous work, this study demonstrated that PIVOT was an effective class preparation tool that increases accountability and thought transparency in pharmacy student groups. 15 When studied as a collaborative learning tool among medical and pharmacy student pairs, PIVOT showed improved student self-reported knowledge, clinical reasoning, and communication skills. 16 To date, the value of PIVOT has been determined based primarily on student perceptions. The effect, if any, of PIVOT on student performance is unknown. Therefore, this study sought to determine if the use of PIVOT to prepare for and use within small group workshops results in improved student performance on examination questions designed to test therapeutic reasoning.

This study used a mixed methods approach to explore the student experience and the role of PIVOT within pharmacy education. This study was certified as exempt from review by the University of California, San Francisco (UCSF) Institutional Review Board (IRB # 16-21171).

The PIVOT program provides patient case information in a manner similar to an Electronic Medical Record. In this way, some data gathering is required. After reviewing the information presented in the case, students respond to questions on the web portal or in the mobile application. Once students have submitted their individual responses to the questions, they can view the responses of their groupmates. PIVOT has the capability of "liking" the responses of other students, similar to common social media platforms, to facilitate group consensus.

This pilot study was conducted at the UCSF School of Pharmacy during the 2016 to 2017 academic year. The general demographic characteristics of students at our pharmacy school can be found at https://pharmd.ucsf.edu/about/facts-figures. Third year pharmacy students, naïve to the intervention, participated during their required infectious diseases therapeutics course. Weekly, 90-minute, small group (10-12 students) clinical case-based therapeutic reasoning workshops were a required component of the course. Prior to each workshop, student subgroups of 3 to 4 individuals were assigned a practice case to complete together. The expected output was a collaborativelydesigned therapeutic plan using the standard Subjective, Objective, Assessment, Plan (SOAP) format. The logistics of this process were left to the sub-groups, who could hold inperson meetings and/or use electronic systems such as Google Docs (Google, Inc, Mountain View, California) to collaborate. The sub-group plans were submitted at the start of each workshop and then discussed in the wider group.

This pilot study took place during the workshops held in course weeks 2 and 3 (study weeks 1 and 2). The topics of these workshops were therapeutic approaches to pneumonia and gastrointestinal tract (GI) infections, respectively. One investigator (CM) randomly assigned each of the 10 workshop sections to either the typical workshop preparation (SOAP arm) or to use PIVOT for workshop preparation (PIVOT arm) during the first week of the study ( Figure 1 ). The following week, the sections were crossed over to the alternative workshop preparation method. During the first 45 minutes of the workshops studied, the workshop facilitator led a discussion of the prepared subgroup case responses via SOAP or PIVOT. In the SOAP arm, the facilitator selected 1 team to lead the group in a discussion of the preparatory plans, scribing answers on a whiteboard. In the PIVOT arm, the facilitator projected a webpage that displayed the different responses each team had submitted to the questions embedded in the PIVOT program, along with the number of likes each response received. In this way, PIVOT was used to facilitate both preparation and in-workshop discussion. In both conditions, the facilitator helped to guide the group in discussing key points for the case. After resolution of the pilot study activities during the workshop, students were introduced to and collaborated on another case during second half of the workshop, without using technology tools.

Student perspectives and performance were investigated to evaluate the impact of PIVOT. To collect perspectives, in the last 5 minutes of the 2 studied workshops, students completed an online survey that explored favorability with that week's preparatory method and in-workshop experience using a 5-point Likert-type scale: very unfavorable (1), unfavorable (2), neither unfavorable nor favorable (3), favorable (4), very favorable (5) . At the conclusion of study week two, students were asked to rate which platform they preferred using a 5-item Likert-type scale: strongly prefer PIVOT, somewhat prefer PIVOT, no preference, somewhat prefer SOAP, and strongly prefer SOAP. Students also selected how they accessed information using a 3-item scale: mostly or exclusively mobile app, both about the same, mostly or exclusively web portal, and were asked to select how they collaborated using the SOAP approach using a 3-item scale: exclusively online, both online and in person, or Gruenberg et al 3 exclusively in person. Students could also provide comments within a free-response question that asked their thoughts on the program for case-based learning in small groups.

To characterize performance, students completed a 2-hour examination during the fourth week of the course using a secure computer-based platform (ExamSoft). Five items on the exam were related to the workshop content: 1 multiple choice and 2 short-answer questions on pneumonia, and 2 multiplechoice questions on GI infections. Grading of short-answer questions was performed according to a standard rubric provided by the course director (CM) and performed by 2 of the authors (KG and JG) who were blinded to student identity. Scores for these items were summed by content area (pneumonia and GI infections) and analyzed to determine whether there was an association between workshop format (SOAP arm or PIVOT arm) and score on the corresponding content.

Descriptive statistics were used to summarize survey response information. Stata SE version 15 (Statacorp, College Station, TX) was used to perform quantitative analysis. Mixed-effects regression, clustered by workshop section, student subgroup within the workshop section, and student, were used to estimate the effects of workshop format (SOAP or PIVOT) on student favorability for that week and on examination performance for the items related to the workshop content. The Wilcoxon rank-sum test was used to analyse student ratings of teaching methods by platform. A significance level of .05 was established a priori for all statistical tests.

Free response comments were analyzed in Dedoose Version 7.0.23 (SocioCultural Research Consultants, Los Angeles, CA). Two of the investigators (KG & TB) used grounded theory as a framework to generate an understanding of student preferences for case-based, small group learning using technology. 17 One investigator (KG) served as a workshop facilitator within this course. The second investigator (TB) did not teach in the course, but understood the goals and activities due to her administrative role at UCSF School of Pharmacy. Both were familiar with the PIVOT platform. These investigators drew upon their personal experiences as pharmacy educators to guide interpretation of the data. These investigators first coded all the data independently and then met to discuss codes and identify overarching themes.

All 119 students enrolled in the course participated in the learning activities. Of these, 111 (94%) completed post-workshop surveys for study weeks 1 and 2. Mean scores for favorability of preparatory and in-workshop experience ranged from 3.29 to 3.98 (Table 1) . For both SOAP and PIVOT arms, ratings numerically decreased from the first to the second study week. The mixed-effects regression models for student perception of the preparatory experience found that the decrease in favorability ratings between study weeks 1 and 2 was statistically significant for both the PIVOT-first group (−0.68, 95% CI −1.05 to −0.32, P = .003) and the SOAP-first group (−0.41, 95% CI −0.83 to −0.01, P = .04), with the magnitude of the decrease greater in the PIVOT-first group (P = .024 for interaction). This was not seen for the ratings of the in-workshop experience, where the change in favorability ratings from first to second study week was not statistically significant (−0.09, 95% CI −0.44-0.27, P = .61 for PIVOT to SOAP, −0.11, 95% CI −0.48-0.26, P = .67 for SOAP to PIVOT) and was similar regardless of sequence of exposure (P = .54 for interaction).

The vast majority (95%) of students reported mostly or exclusively using PIVOT via the web portal (95%) and most students (78%) reported collaborating exclusively online (eg, through Google Docs) when preparing their SOAP note. When asked to rate their preference of PIVOT versus SOAP, a majority (57%) of students indicated a preference for using PIVOT to collaborate on cases in a small group (Table 2) . There was no statistically significant difference in preference according to the sequence of exposure to PIVOT or SOAP (P = .43).

With regards to performance, mean total examination scores on questions related to the study workshops topics were similar between those students who were exposed to SOAP and PIVOT for pneumonia (32.6 vs 33.6, mean difference −0.94, 95% confidence interval −2.70-0.82, P = .18) and for GI infections (17.1 vs 17.2, mean difference 0.06, 95% confidence interval −0.90-1.01, P = .93) (Figure 2 ).

Four themes were identified from the student comments: thought transparency, ability to work independently before collaborating as a group, frustration with the user interface, and lack of anonymity (Table 3) . These themes represented both positive and negative aspects of PIVOT compared to the SOAP preparation method.

While this study did not detect a significant difference in student performance from using PIVOT as a preparatory tool, a majority of students indicated a preference for using PIVOT to collaborate on cases in small groups. This pilot study adds a Online and in person 21 (19) Exclusively in person 3 (3) Figure 2 . Examination item scores for related content by instructional type.

Plot is the mean of sum of scores for related examination items and 95% confidence intervals. Abbreviations: SOAP, subjective, objective, assessment, plan based collaboration; PIVOT, Practice Improvement using Virtual Online Training based collaboration. Gruenberg et al 5 large scale (N = 119 pharmacy students) evaluation of student perspectives after using PIVOT to prepare for and within small group therapeutics workshops. These findings augment previous results describing student perspectives on using PIVOT to prepare for large group discussions on cardiovascular cases (N = 110 pharmacy students) and interprofessional student experiences with receiving high/low guidance case prompts via PIVOT (N = 16 medical/pharmacy student pairs) studying an immunology topic. 15, 16 With regard to student perspectives, our results are aligned with those from prior study 15 -students generally describe positive experiences using PIVOT. Students highlighted the benefit of working independently before collaborating as a group, which promoted both independent student learning and accountability. Students also described seeing the therapeutic reasoning processes of their group members through the PIVOT program as valuable. Because inexperienced students often rely on instructors to cue their learning, this thought transparency may serve as an important scaffold. 18 Enabling visualization of the thought processes of peers and instructors may be especially important during distance learning, when remote interactions may be less rich than those that would occur in-person.

One of the aspects of PIVOT that students viewed as a negative was its perceived lack of efficiency. For example, some students commented that obtaining information through PIVOT took additional time compared to when this information was provided to them directly in the SOAP format. Previous studies of pharmacy students also report similar frustrations with health simulation technologies. 15, [19] [20] [21] For example, one study reported <30% of pharmacy students felt a computer simulation program was helpful for applying knowledge in a timely manner or developing a therapy plan, despite an improvement in student performance with more frequent use of the program. 20 Because this information seeking skill is a realistic feature of using electronic health records and because using electronic health records optimally is increasingly a skill needed by pharmacy graduates, 4,22 this is likely a frustration that should be discussed openly in case preparation and debriefing. Students should also be made aware that learning therapeutic reasoning within a realistic information gathering setting can promote stronger contextlearning and therefore may better prepare students for pharmacy practice. 9, 23 Several students also described displeasure with the nonanonymous postings within PIVOT where students could see and "upvote" the responses of their colleagues. This is consistent with findings of a systematic review of peer feedback in medical education which suggested that many students feel uncomfortable providing identifiable feedback to their peers. 24 This discomfort may be common in human nature but it can be addressed by teaching feedback techniques purposefully prior to using the program. 25 Because feedback is a key component of context-learning, 9 discomfort with the peer feedback structure may have limited the extent of learning with PIVOT.

Finally, this paper adds an evaluation of performance on short-term assessments of knowledge and problem solving with the PIVOT platform. This study did not detect a significant difference between PIVOT and SOAP on examination items measuring therapeutic reasoning related examination items. Context-learning posits that repetition is a key factor influencing knowledge organization and retrieval, 9 but students were only exposed to the PIVOT platform once before the examination. With more opportunities to practice therapeutic reasoning within PIVOT, student performance may have improved. Context-learning also requires students to acknowledge and respond to feedback. 9 While students in this course were responsible for completing preparatory assignments via PIVOT, their responses to feedback within the Ability to work independently before collaborating as a group "I think PIVOT is a better method because the work is distributed equally and everyone gets the chance to look at the SOAP on their own." "The PIVOT application is useful because everyone gets the chance to work on the SOAP on their own at first. This makes everyone think about every section before teammates say what they think."

Frustration with user interface ". . .currently the interface is not very user-friendly. The app version is very buggy and crashes all the time. I also think the box to enter the responses are too small." "PIVOT exercises did not have a user-friendly interface. It was difficult to toggle back and forth to obtain information needed to answer questions and ended up taking me 2 hours to do when it would have been a lot faster with the SOAP format." lack of anonymity "I didn't like how PIVOT showed your name to your response. That was a little stressful-especially if you answered something completely different from your team." "I would have preferred the names to be blanked out. That removes the 'popularity contest' because getting 'likes' is exactly that."

workshop were largely self-directed and occurred outside of PIVOT. To address this, future investigations of PIVOT may include student repairs of their initial responses following instructor feedback within the PIVOT platform to reinforce student responsibility in the learning process. During the rapid shift to substantial online learning related, for example, to the COVID-19 pandemic, virtual collaboration tools like PIVOT may help health professions teachers to better support groups working virtually on scaffolded therapeutic reasoning tasks. Still, these results have some limitations. Although the students in this project had not used PIVOT previously, they had completed courses in other therapeutic areas, such as cardiovascular and respiratory, which also incorporated small group case-based workshops. The composition of these small groups was not consistent across the courses, so it is unlikely, but possible, that previous group norms influenced behaviors. For example, introducing PIVOT later in the curriculum may have led to more frustrations because students had already established their workshop preparation approach using the SOAP format. This study did not collect any specific demographics from the cohort, so it is also possible that other factors not analysed in this study influenced perceptions and performance. Finally, several students commented that the user interface of PIVOT needed to be improved. Specifically, the way information was displayed made it difficult for students to read their classmates responses and the platform would occasionally malfunction. The investigators plan to address these user interface issues with the PIVOT developer to improve the student experience.

Students preferred the PIVOT platform over typical workshop preparation, and demonstrated similar performance on exam items covering the respective therapeutic content areas in the treatment of pneumonia and GI infections. The thought transparency, accountability, and authenticity encouraged by this platform may provide value for instructors and students beyond short-term assessments.

The second round of the PHAR-QA survey of competences for pharmacy practice

International Pharmaceutical Federation. Global vision for education and workforce

General Pharmaceutical Council. Future pharmacists: standards for the initial education and training of pharmacists

Accreditation standards and key elements for the professional program in pharmacy leading to the Doctor of Pharmacy degree

Harmonized Curriculum for Doctor of Pharmacy Degree (PharmD.) Training in the ECOWAS region

Therapeutic reasoning: from hiatus to hypothetical model

Therapeutic decision making of physicians

Clinical reasoning model for pharmacy students

Teaching clinical pharmacology and therapeutics with an emphasis on the therapeutic reasoning of undergraduate medical students

Scripts and medical diagnostic knowledge: theory and applications for clinical reasoning instruction and research

A cognitive perspective on medical expertise: theory and implication

The development of diagnostic competence: comparison of a problem-based, an integrated, and a conventional medical curriculum

The pedagogical shift during COVID 19 pandemic: online medical education, barriers and perceptions in central Kerala

Medical student education in the time of COVID-19

A novel online platform promotes asynchronous class preparation and thought transparency

A mobile learning module to support interprofessional knowledge construction in the health professions

Qualitative Inquiry and Research Design: Choosing Among Five Approaches

Do medical students copy the drug treatment choices of their teachers or do they think for themselves?

Implementation of a virtual dispensing simulator to support US pharmacy education

Integration of a community pharmacy simulation program into a therapeutics course

Pharmacy student performance in a capstone course utilizing the pharmacists' patient care process

Emerging innovations and professional skills needed within pharmacy curricula

Simulated electronic health record (Sim-EHR) curriculum: teaching EHR skills and use of the EHR for disease management and prevention

The utilization of peer feedback during collaborative learning in undergraduate medical education: a systematic review

Improving student feedback quality: a simple model using peer review and feedback rubrics

The authors acknowledge the contributions of Mr. Sebastian Andreatta from Kiana Analytics for development of the PIVOT program.

KG-conceptualization, formal analysis, investigation, project administration, original draft, review and editing. TB-conceptualization, funding acquisition, software, methodology, formal analysis, review and editing. JG-investigation, project administration, review and editing. CM-conceptualization, funding acquisition, methodology, software, formal analysis, visualization, review and editing.

Katherine Gruenberg https://orcid.org/0000-0003-2552-353X Conan MacDougall https://orcid.org/0000-0002-8690-5796