key: cord-1040757-nslp897x
authors: Farmer, Russell W.; Saner, Staci; Weingartner, Laura A.; Rabalais, Gerard
title: Questioning Aid for Rich, Real-Time Discussion (QARRD): A Tool to Improve Critical Thinking in Clinical Settings
date: 2021-04-01
journal: MedEdPORTAL : the journal of teaching and learning resources
DOI: 10.15766/mep_2374-8265.11132
sha: 8c857c1ac532f86d0634fa23063b5f7df5627548
doc_id: 1040757
cord_uid: nslp897x

INTRODUCTION: Critical thinking skills are crucial for health professionals, especially in clinical settings. However, most health professions educators engage learners with only lower-level concepts such as definitions, fact recall, or basic explanations. Employing strategic questioning methods that require learners to use higher-order thinking can help develop clinical reasoning skills. METHODS: The Questioning Aid for Rich, Real-time Discussion (QARRD) was created for health professions educators to purposefully implement concepts from Bloom's taxonomy and hierarchical questioning in clinical settings. The tool was introduced to faculty in a 1-hour, interprofessional workshop that described learning science and evidence-based questioning methods. Participants practiced QARRD questioning strategies and completed a pre/post case-based evaluation in which they developed discussion prompts for learners. RESULTS: Thirty-seven educators participated in two separate workshops. The majority (71%) of preworkshop prompts were lower-order thinking skills (remembering/understanding). After the workshop, the complexity of participants’ discussion prompts increased significantly. Most postworkshop prompts (69%) reflected higher-level thinking skills (apply/analyze/evaluate/create). Many participants reported that, despite previously knowing about Bloom's taxonomy, they had not known how to implement this learning framework in clinical instruction until completing the QARRD training. DISCUSSION: The QARRD is a versatile, practical tool for health professions educators to practice promoting higher-level thinking in clinical settings. QARRD strategies allow educators to make small, purposeful adjustments to instructional methods that meaningfully engage learners to help facilitate clinical reasoning. This workshop can be delivered at other institutions and adapted as a virtual grand rounds to broadly enhance strategic questioning in clinical education.

Encouraging trainees to think critically in the clinic helps learners develop skills to provide effective care. Instruction that fosters critical thinking is thus a responsibility for health professions educators, 1 and it is also a mandate from our accrediting bodies. 2, 3 Common challenges in clinical instruction, however, include a lack of time, unclear objectives, and a lack of applied problem-solving skills, 4 all of which can make critical thinking exercises seem difficult to implement.

The instructional gap around engaging learners with higherorder thinking has been documented consistently across both time and health professions. [5] [6] [7] [8] [9] [10] [11] Faculty members often resort to teaching in the clinic largely by telling students information rather than asking questions, and when faculty do ask questions, these typically require fact-based answers. [12] [13] [14] While remembering and understanding facts are foundational for higher-level learning, asking for the learner only to remember facts does not assist them in developing critical thinking or clinical reasoning skills to help learners develop expertise. 15 Higher-order thinking skills, such as evaluating or elaborating on ideas, can drive the development of critical thinking skills by helping learners retrieve information more effectively, 16 strengthen connections between concepts, and extrapolate concepts to novel problems. [17] [18] [19] The hierarchy of instructional tasks is a foundational concept taught in pedagogical and andragogical training. Learning domains provide educators with a framework to organize and thus purposefully implement different levels of difficulty in an educational task or strategic questioning. 20, 21 The revised Bloom's taxonomy, 22 for example, is a popular model that is ordered from lower-level skills like remembering, to higher-level skills like creating ( Figure 1 ). Previous work in MedEdPORTAL has emphasized the importance of Bloom's hierarchical model to promote critical thinking. One MedEdPORTAL module teaches community-based clinical faculty about Bloom's taxonomy as a model to ask higher-order questions, 23 and another uses Bloom's taxonomy as a framework to develop learning objectives. 24 These tools were intended for faculty to use in the planning stages of curriculum development outside of an active educational setting. However, this previous work does not show faculty how these strategies can also be purposefully implemented with students in an ad-hoc manner.

We thus sought to create a faculty development pocket tool for clinicians that could facilitate the construction of higher-order questions for learners specifically in busy clinical settings. Our Questioning Aid for Rich, Real-time Discussion (QARRD) tool extended the previously reported work by delineating clinical questioning strategies using the Bloom's taxonomy hierarchy and additional strategies using elaboration lines of questioning. 25 Our tool facilitated the application of verbs at different levels of Bloom's for each clinical questioning strategy while the faculty was interacting with the learner. The novelty of this tool was that it organized various active learning questioning strategies for clinical settings where it can be difficult to translate active learning strategies intended for didactic settings, because clinical learning is often brief, spontaneous, and discontinuous. The tool gives faculty explicit examples of how to apply the strategies in the moment with learners. This approach is broadly applicable across different health professions, so the target audience for this strategic questioning tool includes faculty from schools of medicine, nursing, dentistry, and community-based faculty who train students and residents. This MedEdPORTAL module also contributed evidence that faculty develop higherlevel questioning prompts with the tool after the training, which suggests that this resource could be used by clinicians to actively promote critical thinking in clinical settings.

We developed the QARRD (pronounced "card") tool as a strategic questioning guide for clinical instructors to implement with health professions students (Appendix A). This tool used the revised Bloom's taxonomy framework 22 and elaboration questioning methods (how/why/what if prompts) 25 to help instructors move away from "What is …?" style questions. We designed the QARRD to be a portable tool that instructors could actively use in clinical settings while developing these questioning skills. The QARRD can be printed front to back on 6 × 15-inch foldable cardstock to create a durable tool that can be carried as a teaching reference card in an instructor's pocket (Appendix B).

The QARRD provides a framework for clinical instructors to practice asking higher-level clinical reasoning questions to students by introducing the revised Bloom's taxonomy and giving instructors examples of action verbs that reflected each level. The QARRD is organized using 11 different clinical questioning strategies, each of which is briefly described within the tool. Under each of these strategies, instructors are also given examples of different types of prompts that could be used with learners. These example prompts were developed by two clinically active physicians and a health professions education expert (Russell W. Farmer, Gerard Rabalais, and Staci Saner) to adapt common active learning strategies. 25, 26 The authors developed various discussion prompts for clinical settings, standardized the prompt language, and practiced using the prompts until reaching a consensus on the final set of examples that would best represent each strategy for clinical learners. Each of these example prompts in the QARRD is color coded to its corresponding level of Bloom's taxonomy included in the tool. Example prompts are also ordered so that higher-level questions are found at the top of the block and lower-level prompts are toward the bottom. This ordering allows instructors to easily tailor their questioning level to learners: instructors may use lowerlevel strategies for topics that students were just introduced to and higher-level strategies to move beyond recall for topics or learners who have more experience.

Training on how to use the QARRD was completed in a 1-hour, interprofessional workshop (Appendix C) for health professions faculty. This workshop oriented participants to critical thinking and taught them how to ask higher-order clinical questions with the Bloom's and elaboration questioning strategies. The workshop facilitators (Russell W. Farmer, Staci Saner, and Gerard Rabalais) included faculty development educators and physicians. Workshop content was developed to be appropriate for all types of health professions educators who interact with learners in clinical settings.

The workshop used a combination of didactic and practicebased instructional activities detailed in the facilitator guide (Appendix D). First, participants were introduced to learning theory and learning domains including Bloom's taxonomy. Participants were then asked to describe typical discussions that they have with students in clinical settings and barriers they face to clinical instruction. Next, the workshop facilitators described methods to strengthen question and discussion strategies to promote higher-order thinking from learners in clinical settings. For example, rather than asking a learner to list the symptoms of a disease presentation (a lower-level recall question), an instructor could instead ask the learner to contrast the disease presentation of the present patient with a hypothetical patient having different risk factors (a higher-level analysis question). We next gave each participant a copy of the QARRD (Appendix A), and participants observed a role-play demonstration of how to implement the QARRD for clinical instruction. Finally, participants practiced using the tool in pairs/small groups with various roleplay activities that required constructing discussion prompts on their own. To adapt to the COVID-19 crisis, many of our more recent versions of this workshop have been given over teleconference and suggested modifications for this format are also included in the facilitator guide.

In 2020, we evaluated the workshop at the University of Louisville Health Sciences Center Campus twice: once with faculty enrolled in a health professions education training program and once as a session at a university-wide annual teaching conference. Participants at both workshops were asked to fill out a pre-and postevaluation (Appendix E) before the workshop began and then in the last 5 minutes of the workshop, respectively. The preevaluation asked participants to use a brief patient scenario to develop three questions or discussion prompts that the participant would typically ask learners during rounds or after a patient presentation. We also asked participants about their incoming knowledge of learning domains and perceptions of clinical teaching. For the postevaluation, we asked participants to use the QARRD to develop new or revised prompts for learners with the same patient scenario. We then asked participants if and how they envisioned using the tool and to provide feedback on the tool and the workshop.

Individual responses to the pre-and postevaluations were paired through a unique identifier. Prompts from attendees of both workshops were pooled, deidentified, sorted randomly, and then categorized by each author into the most appropriate Bloom's level. Classification discrepancies were reviewed by all authors until the group reached a consensus for each prompt. We then compared the Bloom's level between the pre-and postevaluations using a Wilcoxon Rank Sum test to determine whether prompts were developed at a significantly higher Bloom's level after the workshop. We also reviewed openresponse evaluation questions and summarized common themes from this feedback. This study was approved by the University of Louisville Institutional Review Board.

Thirty-seven educators participated in, and completed evaluations for, the two workshops, which drew an interprofessional audience representing dental, medical, nursing, and public health educators (Table 1 ). Most participants reported that they had previously known of learning domains like Bloom's taxonomy (32 of 37, 86%). However, less than half of the participants (35%) reported that they had previously applied these learning theories to their own teaching (Table 1) .

On the pre-evaluation, participants described gaps in their current clinical instruction efforts. Many faculty described learner participation and engagement as a difficulty such as, "The silence after asking a question." Many educators also described students' anxiety about answering questions in the clinic as a barrier, and faculty did not want to make learners feel intimidated by their questions. Educators ultimately desired to learn how to "ask better questions" during the QARRD workshop so that their students would be able to apply and retain more information in a nonthreatening environment.

Outcomes and Impact Given the brief patient scenario, most participants developed preevaluation discussion prompts that required lower-order thinking, such as fact-based recall (49 of 69, 71% at Bloom's levels 1 or 2). After practicing high-order questioning with the QARRD, participants constructed significantly higher-level prompts in response to the patient scenario on the postevaluation (W = 5387.5, p < .001; Figure 2 ). Most postworkshop prompts (80 of 98, 69%) reflected higher-level thinking skills (apply/ analyze/evaluate/create).

Twenty-eight (76%) participants completed both pre/postevaluation prompts that could be compared. Examples of paired prompts from six participants ( Table 2) showed how participants used the questioning strategies to develop discussion prompts that required higher-order thinking after the QARRD training. The median prompt level was similar across participants who reported no previous knowledge and application of Bloom's taxonomy for both the pre-evaluations (all at level 2, understanding) and postevaluations (all at level 4, analyze), which suggests that previous familiarity or use of Bloom's taxonomy did not drive differences in prompt development among participants.

Participants reported positive perceptions of the QARRD and the strategic questioning workshop. They described being eager to implement these strategies in clinical settings and appreciated how the QARRD methods did not require an overhaul in their current clinical teaching strategies: the techniques could be incorporated easily into existing discussion formats with learners on rounds. One participant described how the training made them realize that they could "utilize specific terms to ensure fundamental knowledge as well as move up the taxonomy chain."

Participants also reported that they planned to use methods from the QARRD to purposefully engage students to think critically. When asked what they would implement or take away from the workshop, one participant described that, "I'll certainly challenge myself to ask higher-level questions and not just fact-based questions [and] keep the QARRD readily available for reference." Another participant realized how the faculty instructor must "initiate discussions that require the student to do the work of thinking rather than just tell them what they need to know."

Suggestions for improvements of the tool and workshop focused on additional practice using the QARRD and expanding applications of the questioning methods to other instructional settings, such as for non-health professions graduate students.

Critical thinking skills drive the maturation of our learners into qualified health professionals. These skills are nurtured by the quality of instruction they receive and are framed by type of questions they are asked. 27 We found that a brief faculty development intervention using the QARRD as an instructional tool tailored for clinical settings could help health professions educators develop higher-order questioning skills.

We have also identified a gap between knowing about learning domain frameworks like Bloom's taxonomy and applying these techniques. The prevalence of faculty who knew about learning domains like Bloom's taxonomy was likely inflated in our sample compared to the general population of health professions educators because our workshops were aimed at existing faculty development education programs. However, even among these participants with a preexisting interest in education, many had not applied the learning theories to their own teaching, and most of their pre-evaluation prompts required only lower-order thinking from learners. This suggested that a clearer application of the framework-such as the guided examples given by the QARRD training-may help health professions educators better translate learning science like Bloom's taxonomy to their own instruction.

The QARRD and its associated workshop provided a novel mechanism for health professionals to improve both instructional quality and questioning practices. The questioning methods can facilitate engagement with learners around higherorder thinking skills, thereby stimulating critical thinking. 28 Additionally, the QARRD specifically adapted tested educational instructional strategies 17 into common clinical questions familiar to our audience. Busy clinicians can use the tool to engage learners with only cursory knowledge of Bloom's taxonomy as an educational adjunct. The QARRD was thus a practical and effective tool that can be implemented within existing instructional strategies, and the methods employed with the QARRD and explained in our workshop could allow clinicians with little formal educational training to facilitate higher-level discussion like more seasoned educators.

The questioning strategies implemented with the QARRD also highlighted learning through difficulty. Higher-order questions enhance learners' clinical reasoning and critical thinking. We acknowledge that these methods could feel foreign to both learners and educators alike, and educators face two major adjustments when trying to implement the QARRD and its methods. First, educators must learn to use Bloom's as a guide to achieve better questions. Second, educators may find it a challenge not to recede back into the world of lower-order questions by default. Both sides of the educational equationeducator and learner-may end up taxed as a result of these educational prompts, despite our attempts with the QARRD to minimize the cortical processing power needed on behalf of the educator. Even while demonstrating examples for the QARRD workshop, we found it is easy to revert into the old habit of lowerlevel questioning when giving examples of discussion prompts ("What is this structure? Which viral exanthem is most common? How many teeth should the average human have?"). This typically resulted in a cofacilitator making a real-time reminder to revisit our own proffered methods, which allowed us to realize that breaking the habit of lower-level questioning may require purposeful practice and reminders to regularly implement the techniques. This ultimately suggested that the reliance on lowerlevel questioning is ingrained, and we need to actively work to aim for higher-level discussions, which the QARRD aimed to do.

The QARRD training ultimately made educators aware of the various components of critical thinking development through clinical questioning and provided a template to apply in the clinic. Our uptake has led the QARRD training to be incorporated into the framework of our undergraduate and graduate medical education environments as part of the orientation to learning medicine at our university. Learners are being asked to expect difficult, higher-order thinking questions as part of their training and to request these levels of questions if they are not provided.

The primary limitation of the QARRD was that our evaluation took place in a faculty training session. Participants demonstrated higher-order questioning, but our assessment did not observe the faculty in an actual instructional setting. Another limitation was that the example question prompts within the QARRD focused largely on the teaching of medicine. Although the prompts were purposefully general to clinical settings and faculty from other health professions reported that the QARRD was useful, less-relevant prompts could be an implementation barrier. The limitations of the QARRD may also be better identified through direct observation in the clinical settings or discussed in focus group testing.

Currently, we are planning observational studies to determine if and how the QARRD training and methodology are actually implemented with learners. In addition to the planned observational studies with the QARRD and questioning methodologies, future directions for this work include expanding the settings and format in which the QARRD is implemented. We are currently working to develop a more widely applicable version with the potential to reach out to nonhealthcare disciplines, such as basic science graduate programs.

In addition, learners could benefit from using the QARRD in instructional settings like team-based learning or problem-based learning. Individual use of the QARRD in these settings could not only prompt more critical thinking, but its use could purposefully introduce users to the science of learning that drives these strategies. These are concepts that many health professions students do not learn although many go on to become clinical educators.

We are also in development of other forms of content delivery for the QARRD itself. We are currently developing a digital, appbased version of the QARRD that could be more accessible to instructors and learners in a clinical setting. It is our hope that the higher levels of questions asked by our faculty as prompted by the QARRD can serve as a nidus for critical thinking and ultimately lead to improved cognition for learners. 

The educational attributes and responsibilities of effective medical educators

Functions and Structure of a Medical School: Standards for Accreditation of Medical Education Programs Leading to the MD Degree. Association of American Medical Colleges and American Medical Association

ACGME common program requirements (residency)

Accreditation Council for Graduate Medical Education

Learning and teaching in the clinical environment

Questioning skills demonstrated by approved clinical instructors during clinical field experiences

The questioning skills of nursing instructors

Questioning as a teaching tool

The use of questioning strategies by clinical teachers

Best practice strategies for effective use of questions as a teaching tool

Application of Bloom's taxonomy of verbs to evaluate the cognitive domain in undergraduate medical physiology question papers: a critique

Milestones of critical thinking: a developmental model for medicine and nursing

The level of questions posed by nursing educators

Effect of a program to increase the cognitive level of questions asked in clinical postconferences

It's worth the wait: optimizing questioning methods for effective intraoperative teaching

How Learning Works: Seven Research-Based Principles for Smart Teaching

Powerful Teaching: Unleash the Science of Learning

The importance of questioning in developing critical thinking skills

A multi-site study of strategies to teach critical thinking: 'why do you think that?

Critical thinking in critical care: five strategies to improve teaching and learning in the intensive care unit

Strategic Questioning

Questioning: a teaching strategy to foster clinical thinking and reasoning

A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives

Questioning as a tool in teaching

Faculty Learning Committee. Bloom's taxonomy in action

Make it Stick: The Science of Successful Learning

Small Teaching: Everyday Lessons From the Science of Learning

Paper presented at: Annual Conference of the National Council of States on Inservice Education

How does questioning influence nursing students' clinical reasoning in problem-based learning? A scoping review