key: cord-0812784-467nrx6d authors: O’Brien, Bridget C.; Teherani, Arianne; Boscardin, Christy K.; O’Sullivan, Patricia S. title: Pause, Persist, Pivot: Key Decisions Health Professions Education Researchers Must Make About Conducting Studies During Extreme Events date: 2020-06-09 journal: Acad Med DOI: 10.1097/acm.0000000000003535 sha: 8d783f7eb6e0d54e98326658b481ad54a5d1e7a9 doc_id: 812784 cord_uid: 467nrx6d When extreme events occur, some research becomes a clear priority, but what becomes of all other research? Does it stop indefinitely, or can it be paused with plans to resume, persist with modifications, or pivot to address new priorities? Facing this dilemma and witnessing it among their fellow health professions education researchers, the authors recognized a need for guidance. This Invited Commentary presents a framework, organized as key questions related to the research stage and process, to assist health professions education researchers in making decisions about how to proceed with research that was planned or in progress when an extreme event occurred. Although at the time of this writing, the COVID-19 pandemic was the extreme event at hand, the authors intentionally created questions and discussed considerations that can be helpful for thinking through decisions in a variety of disruptions in health professions education research—many of which require similar difficult decisions and creative solutions to carry important research forward and maintain high quality. During this pandemic, many faculty members have found that their mental energy is redirected toward efforts to adapt and adjust their clinical, educational, and research work to a rapidly shifting reality of uncertain duration. 1, 2 Clinicians are delivering care in new ways. Educators are teaching in virtual worlds. Researchers have stopped studies or are creating innovative workarounds. The literature is replete with suggestions for these various audiences. 3, 4 For researchers, national and local institutions focus much of their guidance on policies, principles, and technical details that pertain primarily to clinical and lab-based research. [5] [6] [7] Meanwhile, health professions education (HPE) researchers need guidance about whether and how to proceed with education studies they have either begun or were about to begin. Should such research simply stop or at least wait for the "new normal"? We believe there are viable options for education research to continue. Extreme events such as earthquakes, hurricanes, political conflicts, and pandemics are "occurrences that, relative to some class of related occurrences, are either notable, rare, unique, profound, or otherwise significant in terms of their impacts, effects, or outcomes." 8(p407) While disruptions to researchers' plans occur frequently and often can be addressed through contingency plans for typical occurrences, 9 extreme events are so unanticipated and thoroughly disruptive that they necessitate review and reconsideration of the research project. When these events arise, the researcher's dilemma is "what to do now?" Some researchers will abandon or reduce their studies to data already collected, perhaps to pursue new studies. Others will continue their studies. [10] [11] [12] [13] [14] [15] For those who continue, options include pausing the research for a period of time with criteria for when to resume, persisting with the research through the event-often with modifications to the study procedures, or pivoting the focus of their research. These choices are not mutually exclusive, as some research may continue with the original question (pause or persist) and add new questions or modify existing questions (pivot). In this Invited Commentary, we offer a framework, presented as guiding questions, to assist HPE researchers in deciding how to proceed with their research. The questions address the stage of research and the main components of the research process (study purpose, design or approach, procedures, and findings). These are summarized in Chart 1. Chart 2 demonstrates using the framework with an example. We acknowledge that extreme events will spawn their own set of novel education studies, and researchers will follow traditional guidance for quality research. If a study is still in the design stage, researchers can adjust their plans depending on how intertwined the topic is with the extreme event and whether they can study the topic in other ways or at another time. For example, if the event occurs in a limited region, can the research be conducted elsewhere? Is it worthwhile to adjust the design to compare affected and unaffected regions or populations? If the study is already underway, researchers will need to decide how to manage the event's impact on the study. Some researchers may decide to pivot, particularly if new funding sources support research in their topic areas with a focus on how the extreme event affects education. Funding may also provide access to new data sources related to the extreme event and encourage collaboration across institutions, disciplines, and fields. Which stage the study is in, along with considerations of Abstract When extreme events occur, some research becomes a clear priority, but what becomes of all other research? Does it stop indefinitely, or can it be paused with plans to resume, persist with modifications, or pivot to address new priorities? Facing this dilemma and witnessing it among their fellow health professions education researchers, the authors recognized a need for guidance. This Invited Commentary presents a framework, organized as key questions related to the research stage and process, to assist health professions education researchers in making decisions about how to proceed with research that was planned or in progress when an extreme event occurred. Although at the time of this writing, the COVID-19 pandemic was the extreme event at hand, the authors intentionally created questions and discussed considerations that can be helpful for thinking through decisions in a variety of disruptions in health professions education research-many of which require similar difficult decisions and creative solutions to carry important research forward and maintain high quality. the research process, will help with pause, persist, or pivot decisions. How will the extreme event affect the purpose of the study? Studies serve a variety of purposes, including description of a phenomenon (e.g., prevalence of burnout in a population), exploration for deeper understanding of a phenomenon (e.g., how do learners perceive and experience inequity in clinical learning environments), testing hypotheses or interventions (e.g., is team-based learning as effective for teaching diversity, equity, inclusion topics as it is for biochemistry topics), and translating theory or knowledge from research to educational practices (e.g., how best to apply retrieval practice in clinical education). 16 Extreme events may affect these purposes in different ways. Consider, for example, a descriptive research study examining how students use online modules to study for exams. When the study started, modules were optional or supplemental to in-person lectures. Now, as a result of the pandemic, online modules are the primary option for learning course material. Does this new situation still yield a valid description of the phenomenon? Probably not. Is it still useful to know what patterns looked like before the pandemic? Perhaps-particularly if the purpose of the study could be revised to compare use pre-post or to document trends over time. What is the study's stage? • Design (including pre-or postethics submission and approval) • Data collection • Data analysis • Reporting findings • Design: Persist or pivot? • Data collection: Pause, persist (potentially with adjustments), or pivot • Data analysis: Persist (check for event influence on data, then decide if more data are needed) • Reporting findings: Persist Do I need to • consult with and/or submit a modification to the ethics review board? • meet with and receive approval from funders? • discuss with those who will help to identify participants or implement a change in intervention? • address limitations and implications of findings in light of the event? How will the extreme event affect the purpose of my study? • Study purpose/questions remain relevant: Pause or persist • Study purpose/questions can provide helpful baseline or benchmark data: Persist • Study purpose/questions can easily incorporate new circumstances: Persist • Study purpose/questions need some adjusting to be relevant and useful: Persist and/or pivot • Do I still believe the work has merit and something to offer and is worth continuing (motivation)? • Does pivoting offer an option to preserve some aspects and add to the purpose? How is my study design or approach affected? • Not affected-as long as I describe the context and incorporate it into findings: Persist with possible modifications • Affected-data and findings will likely be viewed as compromised: Pause (if there is a possible return to preevent state), persist (if additional data collection can help make the data more generalizable, less biased), pivot (if possible to change design) • Will editors, reviewers, readers raise concerns about validity, trustworthiness, generalizability, bias? How should my study procedures change? • Recruitment • Data collection • Data analysis • No changes-I will use the data I have and not collect additional data: Persist • No changes-I will keep procedures the same and continue data collection when it is safe and appropriate to do so: Pause • May need to change-I will collect some data and see if patterns are different and adjustments are needed: Persist, possibly pause or pivot • Will need to change-I will adjust recruitment, data collection procedures, survey/interview/focus group questions, analysis: Persist or pivot • Do I have the resources I need to collect additional data or data in a different format; do I need to alter the intervention? • What is the projected timeline, and will it still work? • Do I need to submit a modification to the ethics review board? Check with funder? • Do I need to pursue additional training for myself and research assistants? • Do I need new members for the research team? • What are the consequences of any changes made? How should my findings address the extreme event? • Study findings can provide useful benchmark or comparison for subsequent studies. • Discussion will consider findings in context of present circumstances. • Study findings will incorporate findings from the extreme event and discuss changes observed in the data (if any). • Do my findings make sense in the context of the extreme event and associated changes? • How much should I mention the extreme event and where? • Are my findings still relevant and useful? • How does the extreme event influence the limitations of my study? Considerations for the pause, persist, or pivot decision relate to the usefulness and credibility of a study. First, is the purpose of the study still relevant and useful? In many cases, the answer is yes, and the study should persist on this basis but may require modifications. Then, the decision will depend on whether these modifications are possible (e.g., ethically, feasibly The team approached the study from a constructivist orientation, beginning analysis of data as soon as data collection began. The team completed interviews with 30 physicians from multiple specialties and career stages by January 2020 and had completed the first phase of data analysis in mid-March when concerns about the COVID-19 pandemic escalated. The team had just begun preparing for the second phase of data collection in which they planned to interview trainees about their perceptions and observations of faculty members. Initially, the team shifted attention to clinical and personal responsibilities and concerns related to the pandemic, but eventually they returned to regular research meetings (virtually) and discussed how to proceed with the study. The discussion posed concerns aligned with research components as detailed below. Stage in the research process Phase 1 of data collection was complete, and phase 2 had not begun. Should the team stop plans to proceed with phase 2 and just focus on writing up findings from phase 1, even though the study seemed incomplete? Persist-but with a pause: Proceed with phase 2 of the study but not during the height of the pandemic for trainees with clinical responsibilities. To explore a phenomenon that has not been well described in the literature. Persist: Still an important phenomenon to study, perhaps even more so now given vulnerability associated with the pandemic Study design/approach Constructivist orientation would require researchers to take the context into account in trainee interviews. Since faculty data were collected before the pandemic, the trainee data may not align. The team would consider whether the data from faculty and trainees should be written as separate studies or as one study. • Third-and fourth-year medical students would likely have time to participate in interviews because many clinical experiences were curtailed. These students would not be able to speak to observations and perceptions of faculty vulnerability during the pandemic in clinical settings, so their responses might be more comparable to prepandemic responses from faculty. However, given many concerns on students' minds, would this be the best time to interview them about this topic? Would it be better to wait until students return to clinical settings to capture the full range of perceptions-potentially including COVID learnings? • Residents and fellows might be less available, and their responses would likely reflect a combination of responses before and during the COVID pandemic. • The team also considered follow-up interviews with faculty to see if they might offer a different perspective and have new examples from the pandemic to share. • Persist-but with a pause: Interview students, residents, and fellows. Start the student interviews earlier; add a question about changes during the pandemic and after the pandemic; make sure to capture the timing of all examples. • Persist-but with partial pivot: Conduct follow-up interviews with some of the faculty participants in approximately 6 months. These would be shorter, more focused interviews asking if and how they experienced and shared vulnerability with trainees during the pandemic and whether this influenced their willingness to do so now. Given how vulnerability emerged during the pandemic, faculty may be much more willing to show and discuss vulnerability than they were before the pandemic. This may be a lasting effect that would make pre-COVID findings less valid, useful, and relevant. The presentation of findings would, at minimum, need to acknowledge this if the team decided to stick with the data they had. Analysis of the interview data would pay close attention to the examples to check for similarities and differences before, during, and after the pandemic and include these perspectives in the study findings. Study reporting Study context will be explained along with modifications to the interview guide and any adjustments to study procedures that occurred in response to the pandemic. The team would consider whether the findings from faculty and trainee interviews should be written as separate studies or one study. The context, timing, and decisions about study procedures that require explanation in relation to the extreme event would be reported in the method section of the manuscript. Study design relies on the researcher's philosophical orientation. 17 Qualitative research conducted from a constructivist orientation needs to reflect contextual changes. Correspondingly, this research may incorporate changing circumstances into the original research question. Quantitative descriptive studies aim to represent a phenomenon or state of affairs in a designated population and time frame, often from a positivist or postpositivist stance that strives for statistically generalizable claims. How events threaten the external validity of the study outcomes will be important to determine. Extreme events can affect many aspects of study procedures, including recruitment, intervention, data collection, and data analysis. There are numerous procedural considerations, so we mention a few general points here and acknowledge that many require exploration in the context of a particular study. Recruitment. Recruitment may be difficult or impossible during or after the extreme event. Certain groups may be disproportionately affected by the event and harder to access than others. Researchers may use statistical techniques to address this issue or recruit in an unaffected area. Recruitment strategies could change. For example, trying to recruit residents may be challenging using email, but announcements at pandemicgenerated, virtual social events may work effectively. Interventions may need to be paused either until delivery of the original intervention can be resumed or, if that is not possible, until the intervention has been reconceptualized. A key deciding factor is how much the intent, fidelity, or integrity of the intervention would change if altered. A complete redesign may be necessary as educational systems change or content is deemed more or less important during or after an extreme event. Data collection. Data collection methods could be scrutinized. New questions or scales might be added to surveys as the study design either pivots or additional information is required for context. Interviews may include new questions to collect information about participants' current circumstances and experiences of the extreme event or to reflect on changes in their thinking, perspectives, and beliefs. Researchers should examine data collection procedures such as format (e.g., shifting from in-person to video interviews or focus groups or from paper to electronic surveys), consent procedures (e.g., if signature cannot be obtained in person, if additional risks or benefits need to be identified), and researcher training (e.g., to collect data in a new way, to ensure safety, to recognize signs of distress). Any changes made partway through data collection require careful review for changes in the quality and integrity of the data. If initial data sources are limited or no longer available, researchers could consider entirely different data sources and techniques that require flexible or no recruitment, such as the use of text mining methods to study social media content to answer research questions. Data analysis. Data analysis may require checks for differences in responses, ratings, patterns, or themes before and after the event. Additional focus groups or interviews could check resonance and nonresonance of findings. The analysis in qualitative studies should be attentive to interpretations related to the extreme event. Researchers might turn to realist framing, which invokes the primacy of context. 18 Studies involving data collection before and after the event can use techniques such as interrupted time series and the unexpected event during survey design to determine the significance of the event on the outcome. 19 Missing data analysis techniques, such as intent to treat or multiple imputations, should be considered when the data collection has been disrupted or impeded by the event. 20 Changes or modifications in the analytic techniques may also require additional collaborators or team members to appropriately address these new challenges. When researchers report their findings, they need to consider how design and process decisions may affect their findings. We suggest paying careful attention in 2 areas. Representativeness/validity/ trustworthiness. The researchers will need to ponder whether the data they report on still represent the phenomenon of interest, the population intended, and the perspectives of that population. The researchers may want to describe their participants in phases and clarify any changes in representation in the sample based on the extreme events. Discrepancies may need to be reported as findings or limitations. Framing/interpretation. The experience and effects of the extreme event may bring new insights to include in the findings. Researchers should consider how to clarify the additional interpretations that come to light in the aftermath of the extreme event. For example, education studies that examine learning during extreme events such as war and forced migration will need to be interpreted carefully, underscoring the context, politics, and policies that circumscribe the findings from the study. How should the impact of the extreme event on the study be acknowledged in the manuscript? The researchers' decisions have implications for the writing of the manuscript. The researchers will have to decide where to best position the extreme event-introduction, method, discussion, or all 3? If the study pivoted, the extreme event may be best positioned up front, as part of the framing in the introduction. The method section likely will include details explaining the decision to pause, persist, or pivot and associated changes. In qualitative studies, researcher reflexivity will require careful attention to how members of the research team experienced the extreme event and how their experiences influence their interpretations. To enrich the discussion, researchers might think through the relevance and implications of their findings in light of the extreme event. Some questions include the following: Can we still learn something from the findings, especially if all data were collected before the extreme event? Are data useful for comparison or benchmarking in future studies? Are the research questions still meaningful and important despite changes that have occurred and will likely continue to occur after the event (e.g., if a technology studied before the event becomes obsolete or a policy changes)? The limitations section may elaborate on context and explain modifications, particularly for studies that paused or persisted. Extreme events are likely to disrupt all aspects of education 1,21 and, correspondingly, education research. Researchers may question the relevance of topics and studies chosen before an extreme event. Such reservations stem from concern about pressing social issues, guilt for using limited resources or burdening participants, and observed shifts in funding and publication priorities. 22 For example, several funding agencies, journals, and academic outlets have shifted attention to pandemic issues (the "covidisation" of research) 23 and may distract researchers from focus on important educational issues that are fundamental to education even in a time of crisis. While some of the challenges faced in education research are similar to those of clinical and other social and behavioral science research, the lack of funding for and small-scale and local focus of most HPE research can bring these studies to a halt, especially during times of scarcity and heightened competition for limited resources. We provided questions as reminders of issues that we encourage those who continue their work to consider when determining how to proceed and how to do so in a scholarly manner. These questions should provide a way to think through an educational research study during any unexpected event. They should provide ways forward when such movement initially feels overwhelming or impossible. One academic health system's early (and ongoing) experience responding to COVID-19: Recommendations from the initial epicenter of the pandemic in the United States Medical student education in the time of COVID-19 Action at a distance: Geriatric research during a pandemic Conducting clinical research during the COVID-19 pandemic: Investigator and participant perspectives NOT-OD-20-087 Guidance for NIH-funded clinical trials and human subjects studies affected by COVID-19 COVID-19 Current research operations and continuity planning Revised interim UCSF policy on human subjects-related research Extreme events: A research and policy framework for disasters in context Introduction: Sanitized research descriptions and messy realities Conducting surveys when disasters strike The impact of the Great East Japan earthquake on research activity in a disaster-stricken area From the flight deck to the operating room: An initial pilot study of the feasibility and potential impact of true interdisciplinary team training using high-fidelity simulation The mobile mock operating room: Bringing team training to the point of care Notes from Case Zero: Anthropology in the time of Ebola Two steps forward, one step back: Obstacles and progress in conducting research in elementary classrooms The research compass': An introduction to research in medical education: AMEE guide no Philosophy of science series: Harnessing the multidisciplinary edge effect by exploring paradigms, ontologies, epistemologies, axiologies, and methodologies Realist methods in medical education research: What are they and what can they contribute? Unexpected event during survey design: Promise and pitfalls for causal inference The analysis of social science data with missing values Responding to disasters: Lessons for academic health science centers Strange days Covidisation" of academic research: Opportunities and risks