key: cord-0960750-qlhy1fqr authors: Stringer, J. K.; Gruppen, Larry D.; Ryan, Michael S.; Ginzburg, Samara B.; Cutrer, William B.; Wolff, Margaret; Santen, Sally A. title: Measuring the Master Adaptive Learner: Development and Internal Structure Validity Evidence for a New Instrument date: 2022-01-04 journal: Med Sci Educ DOI: 10.1007/s40670-021-01491-9 sha: d5f6eaabcba6ed0faf10666b3ac5c29facb1a31b doc_id: 960750 cord_uid: qlhy1fqr BACKGROUND: The master adaptive learner (MAL) uses self-regulated learning skills to develop adaptive, efficient, and accurate skills in practice. Given rapid changes in healthcare, it is essential that medical students develop into MALs. There is a need for an instrument that can capture MAL behaviors and characteristics. The objective of this study was to develop an instrument for measuring the MAL process in medical students and evaluate its psychometric properties. METHODS: As part of curriculum evaluation, 818 students completed previously developed instruments with validity evidence including the Self-Regulated Learning Perception Scale, Brief Resilience Scale, Goal Orientation Scale, and Jefferson Scale of Physician Lifelong Learning. The authors performed exploratory factor analysis to examine underlying relationships between items. Items with high factor loadings were retained. Cronbach’s alpha was computed. In parallel, the multi-institutional research team rated the same items to provide content validity evidence of the items to MAL model. RESULTS: The original 67 items were reduced to 28 items loading onto four factors: Planning, Learning, Resilience, and Motivation. Each subscale included the following number of items and Cronbach’s alpha: Planning (10 items, alpha = 0.88), Learning (6 items, alpha = 0.81), Resilience (6 items, alpha = 0.89), and Motivation (6 items, alpha = 0.81). The findings from the factor analyses aligned with the research team ratings of linkage to the components of MAL. CONCLUSION: These findings serve as a starting point for future work measuring master adaptive learning to identify and support learners. To fully measure the MAL construct, additional items may need to be developed. knowledge and skills required to achieve competence, physicians must refine their practice amidst rapid changes in science, technology, and public health crises [1, 2] . The COVID-19 pandemic underscores the importance of adaptability now more than ever. Front-line providers care for increasingly complex patients and must rapidly incorporate new evidence into their daily practice and address novel challenges, thus demonstrating the application of adaptive expertise. In order to ensure adaptive physicians, medical education must start with medical students and develop and promote lifelong learners ready to engage in self-regulated learning, continuous improvement, and the ability to adapt [3, 4] . This set of skills characterizes the master adaptive learner. The master adaptive learner [5] (MAL) model was created to be a unifying framework to guide efforts to study and instill the skills and attributes of lifelong learning. The MAL model was developed to represent the daily problem-solving work of students and physicians and includes the need to regularly engage in and apply new learning, and to encompass the relationship between routine and adaptive expertise [3] . Incorporated in the MAL model are four stages of self-regulated learning (SRL) [6, 7] : planning, learning, assessment, and adjustment [8] . The MAL model extends this SRL model to describe specific behaviors within each of the four phases as well as cognitive skills and internal characteristics that support SRL [9] . Curiosity, motivation, mindset, and resilience are the internal characteristics necessary for MALs [9] . Through the medical education continuum, learners must develop and refine self-regulated learning skills. Supporting training across the medical education continuum, the Liaison Committee on Medical Education requires that self-directed learning is part of the undergraduate medical education curriculum [10] . In the MAL context, we see the behavioral elements of selfregulated learning as necessary for success with selfdirected learning. The preclinical phase is a time in which many medical schools focus on engaging students in self-directed learning, but the Accreditation Council for Graduate Medical Education mandate to develop these foundational learning skills suggests that we cannot ignore SRL after medical school [11] . While undergraduate and graduate medical education programs provide learning experiences to support this development, determining the best way to assess learners' acquisition of these skills has been proven difficult [10] [11] [12] . Contributing to this challenge are the various, overlapping frameworks applicable to lifelong learning and practicebased assessment [13] . As we are directed by our accrediting bodies to support the lifelong development of our students and physicians, it is necessary to identify elements that may help us predict learners' orientations toward this kind of learning. Given the wide range of instruments available to capture cognitive, affective, and behavioral elements of the broad SRL process [14] , decisions must be made about what is most salient toward leaners' growth in this area. Social cognitive theory and reciprocal determinism [15] suggest that learners' environments, behaviors, and cognitions all impact each other. Elements that fit into this framework would include supports in the learning environment for lifelong learning [16] , individual selfregulated learning behaviors [17] , and personal characteristics such as resilience [18] and motivation [19] . By looking at these components not as individual factors, but instead as part of the overall tapestry of a learner's career, we may be better able to understand and support high-quality learning. The MAL model, then, represents an appealing conceptual framework for unifying some of these varied components, yet application to medical school training is underdeveloped. While the theoretical framework of the MAL model has been described, specific measures for its key constructs do not yet exist. An important next step involves the development of an instrument that may be used to identify MAL skills present in learners. Such an instrument would allow for baseline determination of MAL skills, provide the ability to test relevant hypotheses, and may set the stage for measuring the outcome of curricular interventions designed to impact MAL development. In addition, a MAL instrument may serve to provide feedback for learning and development. The purpose of this study was to develop a shortened instrument for measuring the MAL process in medical students and evaluate its psychometric properties. To achieve this aim, we performed an exploratory factor analysis (EFA) with a pool of existing items that had the potential to measure aspects of the MAL model. The MAL framework incorporates several constructs, many of which have previously been measured using various tools and instruments. We elected to build the MAL instrument on this foundation. We initially searched for published instruments measuring the constructs and subcomponents associated with the MAL framework. We reviewed existing instruments focusing on whether the items and constructs aligned with the MAL model. Using a consensus process, we attempted to identify the best fit with the MAL model. Based on our collective experiences and research, we identified four instruments including the Jefferson Scale of Physician Lifelong Learning -Medical Student version [16] , the Self-Regulated Learning Perception Scale [17] , the Brief Resilience Scale [18] , and the Goal Orientation Scale [19] , with 67 total items from the potential pool of instruments to be aligned with the MAL model. For this study, the decision was made to use these instruments both for their theoretical linkages and to capitalize on existing data (see "Sample" section for more details). These selected instruments and their connection to MAL phases are listed in Table 1 . After the initial identification of instruments, we asked the ten members of our multi-institutional research team, as subject matter experts (SMEs) in medical education, educational research, the MAL model, and clinical care to review these instruments (2 MD/PhD, 3 MD/MHPE/ MED, 2 MD, 2 PhD, 1 Master all with expertise in medical education and over 80 years in medical education). After reviewing, each SME indicated their a priori judgment to map each instrument item onto the MAL behaviors (gap identification, selects learning opportunity, searchers for resources, engages in learning, tests learning, incorporates learning, curiosity, motivation, mindset, and resilience). The SMEs rated items as zero if there was no association between the item and a behavior, one if there was weak or tangential association, or two if there was a strong association. Ratings were aggregated by summing the responses for each item from all SMEs. The SMEs discussed the similarities and differences in scoring and overall results to provide the opportunity to think deeply about these items, align them with the MAL model, and analyze the evidence for content validity. *Instrument includes a subscale not used in the current study. The proposed 67 MAL items were included in existing curriculum evaluation and learning environment surveys at the Virginia Commonwealth University School of Medicine (VCU-SOM). The purpose of these surveys to provide data to the medical school about trends in cognition and behavior over the course of students' careers, and as such, they were administered at several time points for each cohort of medical students. Instrument choice was aligned with strategic priorities including self-regulated learning, but also covering topics such as professional identity and tolerance for ambiguity. Data for this study came from surveys administered to 1040 students across the medical school curricula, from matriculation to graduation during the 2018-2019 and 2019-2020 academic years. Complete responses of 818 students were used for analysis, for an overall response rate of 78%. Surveys were administered for each class as follows: matriculation (C2022), 8/2018; end of M1 (C2022), 6/2019; end of M2 (C2021), 4/2019; end of M3 (C2020), 3/2019; and end of M4 (C2019), 6/2019. We intentionally surveyed students in different training years for a cross-sectional analysis because the MAL model was not intended to be specific to preclinical or clinical students. To assess validity evidence related to the internal structure of the instrument, we conducted an EFA using IBM SPSS 26 to identify potential factor structures that could summarize the data and form a basis for subscales within a new instrument and compared these findings with expert review. We followed an iterative process that involved reflecting on the quantitative findings from the EFA alongside the content of the items and our shared theoretical understandings of MAL. Exploratory factor analysis is a statistical method that allows for latent elements in a dataset to emerge by comparing the contribution of common and unique variance. An initial factor structure was produced using principal axis factoring, extracting all factors with an eigenvalue greater than one. However, this produced a 15-factor model which was uninterpretable. Using a scree plot, the elbow was identified at three, four, and five factors, so we evaluated factor solutions ranging from each [15] 6 N/A Resilience Goal Orientation Scale (GO) [16] 13 Motivation by forcing the extraction of those factors, respectively. The four-factor model was chosen. Follow-up analyses were conducted using a promax rotation to allow factors within the model to correlate with each other. Allowing correlations was permitted due to the interplay between MAL facets. To reduce data to create a model, items with loadings less than 0.6 were suppressed. The findings were confirmed by alignment with the SME scoring. The internal consistency of each of these factors was assessed with Cronbach's alpha. This study was reviewed and approved by the VCU Institutional Review Board and classified as exempt. Response rates from our students were as follows: 178 students at matriculation (Class of 2023, 96%), 132 at the completion of the first year (Class of 2022, 65%), 199 preclerkship (Class of 2021, 91%), 156 post-clerkship (Class of 2020, 70%), and 153 prior to graduation (Class of 2019, 73%). Individual items were allocated to the factors extracted in EFA based on their pattern matrix loadings. The four-factor model retaining 65 items with factor loadings above 0.6 was selected as the best fit based on a lack of cross loadings and conceptually distinct factors. Total variance explained by the model was 39.08% (factor 1: 22.37%, factor 2: 7.12%, factor 3: 5.16%, factor 4: 4.43%). These factors resulted in factor 1 having 15 items, factor 2 having six, factor 3 having six, and factor 4 having six. To limit the number of items on the overall scale, the ten most highly loading items were retained for factor 1. All told, this process reduced the total item load from 67 to 28 with Cronbach's alpha values ranging from 0.81 to 0.89. These factors were discussed and labeled Planning, Learning, Resilience, and Motivation (Tables 2 and 3 ). This labeling was achieved by comparing the items maintained in each factor with the relative weights assigned by the SMEs. For example, items remaining in the Planning factor were highly rated by SMEs as having strong associations to the planning element of the original MAL model. Appendix Table 4 presents all items and their factor loadings to detail the structure of the items that were included and excluded from the final model. The MAL framework was developed by Cutrer and colleagues [5] based on existing frameworks, such as selfregulated learning and motivation theory, to be a more comprehensive framework to study and promote complex learning for future competence. It describes a metacognitive approach to learning based on self-regulation that can foster the development and use of adaptive expertise. These results represent our first step in developing a shortened instrument to measure the MAL model in medical students. This preliminary instrument successfully assesses several components of MAL in a reduced fashion. Specifically, the EFA model identified four categories related to MAL including Planning, Learning, Resilience, and Motivation. Items came from the Jefferson Scale of Physician Lifelong Learning -Medical Student version [16] , the Self-Regulated Learning Perception Scale [17] , the Brief Resilience Scale [18] , and the Goal Orientation Scale [19] . The resultant instrument is 28 items (a 58% reduction in item load) with four subscales and reasonable internal consistency. In this work, we intentionally included students across the continuum of undergraduate medical education. While the context for each cohort of students is different from pre-clerkship, clerkship, and post-clerkship, the instrument demonstrated a shared structure across these phases of the curriculum. The EFA identified four categories of the MAL model. The Planning phase incorporates three stages (identifying a gap, selecting an opportunity for learning, and searching for resources for learning). Items in this factor explore gap identification, goal setting, and resource identification. The items in this factor were from the Self-Regulated Learning Perception Scale (SRLPS). In the Learning phase, the student begins to engage in the learning process through challenges and opportunities, and items in this factor were from Goal Orientation and SRLPS. In addition to the process of MAL, there are "batteries" or internal characteristics that facilitate the MAL process. Of these, the EFA identified items associated with Motivation and Resilience. Not surprisingly, the Brief Resilience Scale factored to Resilience. Similarly, Goal Orientation items represented Motivation. The EFA did not identify factors associated with Assessing and Adjusting. Similarly, the a priori content alignment content alignment by the subject matter expert panel did not map these domains. It is interesting to note that despite the fact that the Jefferson Scale of Physician Lifelong Learning [16] and SRLPS [17] are commonly used to assess learning, neither instrument robustly represented assessment or adjustment, which are very important parts of any self-regulated or lifelong learning process. Next, future work will include questions that delve into the specific aspects of Assessing and Adjusting. If we find that existing tools do not contain these questions, one explanation could be that our understanding of self-regulated learning through the development of the MAL model has matured to recognize these specific pieces as important, and if that is the case, we will work with SMEs to create the needed questions. As we explored motivation through the SME, the items, and the MAL model, it is likely that curiosity, motivation, and mindset are interconnected, and it may not be possible to separate into subscales. The researchers on this team believe the theory of MAL is reasonably sound, and we are interested in exploring this further through a wider search for questions from new instruments, specifically focused on these areas, for inclusion in a future EFA study. We hypothesize that by grounding future item development in the lived experiences of learners, a novel instrument can be developed that will account for a greater degree of variance in scores than this first draft shortened version. There are several important implications to these results. First, an instrument that measures the MAL construct may provide valuable diagnostic information. If a learner struggles with clinical reasoning or has context-specific challenges, a MAL instrument may help identify the domain of these issues and may streamline efforts for remediation. Second, via further exploration of master adaptive learning through the lens of this instrument, we hope to better understand the development of students' learning processes. The ability to assess learners utilizing a MAL framework will be a tangible way to codify areas for individual growth and facilitate research into the best ways to fill those gaps. The COVID-19 pandemic has exemplified how adaptive our practitioners can be when faced with the unknown. Finally, these results may offer opportunity for answering existing or future research questions related to the framework. For example, is master adaptive learning a fixed or variable characteristic? Do some learners approach situations in a method more in-line with this framework? If so, what are the short-and long-term outcomes? Are there multiple ways to accomplish each part of the framework and what are they? The MAL instrument may help explore these questions while acknowledging the limitations of utilizing a self-reported instrument. Finally, a MAL instrument may provide feedback to students about their approach to adaptive learning. The current instrument covers four aspects of master adaptive learning, but additional domains are needed. We will gather additional instruments and/or develop novel items to capture a more comprehensive picture of master adaptive learning. In addition, we will explore the administration of the instrument at other institutions along with additional factor analysis. Finally, it may be useful to analyze separately cohorts of preclinical and clinical students. Through this work, we also recognize that master adaptive learning is complex in nature, and to adequately assess it, we will need to include behavioral measures which may not lend themselves to a traditional self-report format. How can the information gathered through student self-report be combined with workplace-based assessment? How do narrative assessments from faculty coaches align with students' perceptions? Furthermore, there is much value in testing this framework on learners in other contexts. The overall MAL process is important as well as the relationships to all the steps and the batteries. It may be important to consider instrument development in terms of a programmatic assessment of this complex, multifaceted construct [20, 21] . The development of an overall assessment instrument to measure master adaptive learning and the subscales may contribute to a portfolio effort to measure components of MAL and assemble those assessments into an overall assessment of the master adaptive learner. EFA has an inherent limitation in that data analysis can only show results based on the existing items and respondents and an assumption is made that the resulting factor structure is a best fit of those data. Furthermore, the addition of novel items or modifications of item wording may change the internal structure of the aggregate instrument and alter this solution. There are also a wide range of other surveys that could be used to capture the constructs of interest in slight or significantly different ways. In terms of selected instruments, the SRLPS has four subscales, and we included three of the subscales (the "lack of self-directedness" subscale was excluded). Interestingly, the 3 we used did not load together into those three subscales which may be illustrative of the fact that context is important. It is also worth to consider the four areas of the MAL model that are not part of this preliminary tool (Assessing, Adjusting, Curiosity, Mindset) and considering why this may have happened. There are several possibilities: • We did not have the right collection of instruments in this study. • The questions from the instruments we used may not be sensitive enough to discriminate the various categories of the MAL model. • The phases of the MAL model are not as distinct as described, and there is some overlap among them which would suggest the MAL construct needs some adjustment. • The MAL model needs further development to align with authentic learning. We will need to collect additional validity evidence on a new sample of student responses to judge the stability and generalizability of the four-factor solution. We made the decision to group the cohort data together based on academic phase because there were no significant curricular changes, but other factors could result in bias by merging this data. It is also important that future studies include the perspectives of students in special populations and use instrumentation that has been validated for those populations such as the Metacognitive Awareness Inventory [22] . We are also limited by the instruments that we selected to begin with. This is a single institution study which limits generalizability. While the goal was to create an instrument that applies across all years of medical school, it may be that the items do not adequately reflect learner skills and development during every phase. Master adaptive learning is multidimensional, and the current instrument identifies four important domains. The MAL model provides a practical framework to guide learning across the continuum in medicine. By measuring individual components within the framework, we can support the development of these internal characteristics and behaviors throughout a learner's career. Supporting physicians at all stages of their career, whether during times of crisis or within the context of daily learning and practice, with individualized recommendations/ coaching in what areas they should focus on and with specific suggestions for how they can improve, is our professional responsibility as medical educators. A tool to assess master adaptive learners will be a key to accomplishing this. Reflections from the rearview mirror: internal medicine physicians' reactions to clinical feedback after transitions of responsibility Presenting characteristics, comorbidities, and outcomes among 5700 patients hospitalized with COVID-19 in the New York City area Adaptive expertise in medical education: accelerating learning trajectories by fostering self-regulated learning Preparing future adaptive experts: why it matters and how it can be done Fostering the development of master adaptive learners: a conceptual model to guide skill acquisition in medical education Self-directed learning: a new look at an old concept Supporting self-directed learning: a national needs analysis Self-regulated learning in medical education Exploring the characteristics and context that allow master adaptive learners to thrive Functions and structure of a medical school: standards for accreditation of medical education programs leading to the M.D Accreditation Council for Graduate Medical Education. ACGME common program requirements Coaching to improve self-directed learning Understanding assessment systems for Clinical Competency Committee decisions: evidence from a multisite study of psychiatry residency training programs Instruments for measuring self-directed learning and self-regulated learning in health professions education: a systematic review Social cognitive theory in cultural context Measuring medical students' orientation toward lifelong learning: a psychometric evaluation Metacognitive awareness and selfregulated learning skills of medical students in different medical curricula The Brief Resilience Scale: assessing the ability to bounce back Development and validation of a work domain goal orientation instrument A model for programmatic assessment fit for purpose Programmatic assessment of competency-based workplace learning: when theory meets practice Assessing metacognitive awareness Ethics Approval and Consent to Participate This study was reviewed and approved by the VCU Institutional Review Board and classified as exempt. Informed consent is not applicable. The authors declare no competing interests.