key: cord-0711907-0vp9mbkq authors: Clark, Renee M.; Kaw, Autar K.; Braga Gomes, Rafael title: Adaptive learning: Helpful to the flipped classroom in the online environment of COVID? date: 2021-11-03 journal: Computer Applications in Engineering Education DOI: 10.1002/cae.22470 sha: 70ba3cd457bed16cec8d91ea89d569f575306e51 doc_id: 711907 cord_uid: 0vp9mbkq Flipped instruction in an undergraduate numerical methods course in the online, remote environment during the COVID‐19 pandemic was conducted with and without the use of adaptive‐learning lessons for pre‐class preparation. This comparison was made to explore potential differences with and without adaptive software relative to exam and concept inventory performance and student perceptions of the classroom environment, learning and motivation, and benefits and drawbacks. Student perceptions were gathered via the College and University Classroom Environment Inventory (CUCEI) and a survey designed to capture feedback specific to flipped instruction. The analysis was made possible by a current NSF grant to study adaptive learning in the flipped classroom at three universities and extensive prior research with the flipped classroom and adaptive learning by the authors. Results gathered in the online flipped classroom with adaptive learning suggested positive changes in the following: classroom environmental perceptions, preference for flipped instruction, perceived responsibility imposed, motivation for independent learning, and perceived learning. Furthermore, based on an open‐ended question, there was a significant decrease in the proportion of students who experienced load, burden, or stressors in the online flipped classroom when adaptive learning was available versus not. Multiple‐choice exam and concept‐inventory results were slightly higher with adaptive lessons (although not significantly so), with the most promising results occurring for Pell grant recipients. The emerging medical education literature has suggested that adaptive learning and flipped instruction will be key to post‐pandemic education. The present article begins advocacy for adaptive learning with flipped instruction in engineering education. studies [6, 8] . Adaptive learning software is meant to personalize pre-class instruction in the flipped classroom. Adaptive learning software automatically monitors student progress throughout an online lesson and adjusts the instruction to the student's needs based on his/her performance within the software. Each student thus encounters an individualized learning pace and path, which includes videos, textual instruction, graphics, simulations, and assessments with immediate feedback [13] . Adaptive lessons were previously developed for the author's engineering numerical methods course to individualize the pre-class learning portion of the flipped classroom format. This was accomplished through an exploratory grant where adaptive lessons for approximately half of the course topics were developed [1, 6] . The authors have since begun a collaborative NSF study on the use of adaptive software in a numerical methods course at two other schools for all course topics (NSF Award No. 2013271). The theoretical framework surrounding adaptive learning focuses on the belief that long-term educational success and development starts with adapting instruction to individual student needs [10] . However, the longerterm goal is for students to learn to adapt to instruction and become self-regulating [10] . Emerging literature in medical education has been vocal about the role that adaptive learning may play in the future of medical training, as discussed further in the literature review. Several recent meta-analyses and literature reviews have characterized the flipped classroom as promising for higher education, as elaborated upon in the literature review section. However, the flipped classroom has received mixed reviews from students [5] . In particular, the independent learning aspect has been associated with disadvantages, such as distractions at home, difficulty learning from a video, feelings of having to "teach oneself," and lack of motivation to watch the videos [8] . However, with the elevated need to operate electronically in the COVID environment, student acceptance of learning with electronic resources may have increased, with the pandemic possibly having a positive impact on the flipped classroom and its reliance on electronic resources (Clark et al., submitted 2021) . In fact, the idea that the pandemic may have "flipped" the flipped classroom in terms of popularity and desirability has begun to emerge in the literature, as will be discussed further. In the present article, remote (i.e., online) flipped instruction in the author's numerical methods course during the fall of 2020 was compared to the same approach, but with the addition of adaptive learning for all course topics in the spring of 2021. The goal was to identify potential differences, perhaps benefits, associated with adaptive learning in an online flipped classroom. Thus far, the results suggest a positive impact of adaptive learning software on the online flipped classroom, including decreased perceptions of the "load" and "responsibility" imposed by flipped instruction. Other desirable perceptions when the adaptive lessons were used included significantly increased preference and motivation for flipped instruction and greater learning gains. These and other findings from conducting an online flipped course using adaptive learning software are presented, including enhanced classroom environment perceptions and student requests for more support when the lessons were used. The medical education community has begun to advocate for adaptive learning given the challenges of COVID. The present article is the first-known article to study adaptive learning in a flipped-classroom setting in engineering during the pandemic. As such, it begins advocacy for adaptive learning in engineering education under these circumstances as well. The following research questions served as the framework for this study and article: 1) What impact has adaptive learning had on the classroom environment in an online flipped engineering course? 2) What impact has adaptive learning had on student perceptions in an online flipped engineering course? 3) What impact has adaptive learning had on exam performance in an online flipped engineering course? 2 | LITERATURE REVIEW Over the past 3 years, several meta-analyses and systematic literature reviews have been published on student perceptions and achievement in the flipped classroom. These have been conducted in engineering and the health professions as well as across educational disciplines. They have shown the flipped classroom as preferable to the lecture-based classroom. For example, a 2019 meta-analysis involving 29 studies in engineering education concluded that the flipped classroom promoted student achievement relative to the traditional classroom, with an average Hedge's g effect size of 0.29 and qualitative evidence suggesting that self-paced learning before class and increased problem-solving during class were the predominant reasons for the advantage [28] . Another systematic review on the flipped classroom in engineering highlighted the following as benefits of the flipped classroom: flexibility (i.e., 24/7 access to materials, video pause/rewind/re-watch), enhanced interaction with peers and instructors, professional skills development, and student engagement (i.e., better class preparation) [23] . A review of 71 research articles from across educational disciplines found the most frequently reported benefit was improved learning performance, with 52% of articles qualitatively citing this advantage [1] . Relative to challenges, the majority related to outside-of-class activities, including inadequate student preparation [1] . In the health professions, a meta-analysis involving 28 studies found a significant effect size of 0.33 in favor of flipped over traditional classrooms [21] . This review also found student preference for flipped versus traditional instruction, with 71% of student respondents across five studies preferring flipped to traditional learning [21] . A second metaanalysis that focused on health-professions education found effect sizes that were significantly different from zero in favor of the flipped versus traditional lecture-based classroom with respect to the outcomes of exam scores, pre-to-post exam score change, and course grade [3] . Various distinguished organizations have lent direct support to personalized learning. A call to "Advance Personalized Learning" was made by the National Academy of Engineering, and it aligned with one of its 14 Grand Challenges for Engineering in the 21st Century [30] . An MIT report on "The Global State of the Art in Engineering Education" identified"…student-centered learning to large student cohorts through a blend of off-campus personalized online learning and on-campus hands-on experiential learning" as a top trend in the future of engineering education [19] . The Educause 2018 Horizon Report (i.e., Higher Education Edition), identified adaptive learning and AI as among those top developments likely to impact higher education within a 5-year timeframe [32] . Two studies of adaptive tutorials for engineering mechanics by the same group of researchers identified student satisfaction and enhanced outcomes [37, 38] . One study found reduced failure rates, increased student satisfaction, and highly positive student comments [37] . A second study of a particular adaptive tutorial reported a 2:1 ratio of student comments identifying the tutorial as effective versus ineffective, with the top reasons for effectiveness being engagement, immediate feedback, and understanding of concepts [38] . In the authors' exploratory grant, they concluded that the flipped classroom with adaptive learning might be the preferred instructional method for STEM courses based on several direct and indirect measures [6] . Two other articles by the authors provide additional findings related to the use of adaptive learning in the engineering classroom, with each article providing and discussing additional sources from the literature related to adaptive learning [7, 25] . Georgia State University, whose student body consists of 56% Pell Grant recipients and 60% non-White students, utilized adaptive learning in an introductory algebra course taken by 7,500 students annually [40] . Since beginning the use of adaptive learning, the percentage of students earning a D or F or withdrawing from the algebra course dropped from 43% to 21% [40] . Additional recent literature on adaptive learning includes a literature review of articles from 2007 to 2017 on adaptive learning as well as a book on flipped instruction and adaptive learning entitled Innovative Trends in Flipped Teaching and Adaptive Learning [41, 47] . This book states that the latest and most innovative advances in educational research are those incorporating personalized learning to the flipped classroom and specifically to the pre-class learning aspect [41] . Personalized learning was identified as an innovative advancement for flipped instruction, with the editors stating that knowledge of personalized learning is the key to enhancing the flipped classroom [41] . The literature review involved an analysis of 70 articles and reported that 25% of the articles involved engineering and computer courses or content, with another 28% involving mathematics and science content. Interestingly, no articles involved health sciences education. This represents an opportunity for medical education and also aligns with ideas advanced about the future of medical education vis-à-vis emerging technologies, to be discussed next [47] . The global medical education community has been vocal about the challenges and disruptions to training and education during the pandemic as well as potential approaches. In a letter to the editor, the pandemic was described as having a negative impact on cardiovascular training in India because of the redeployment of residents to emergency COVID duty and a reduction in elective procedures [48] . However, adaptive learning and flipped instruction were proposed as strategies to enhance and supplement the training of residents during pandemic times [48] . In three additional articles about challenges with medical education in India and the Caribbean during the pandemic, the authors stated that the potential adoption of emergent technology such as adaptive learning for future, post-COVID medical education is a consideration to reckon with [17, 34, 35] . Goh and Sandars believe that regardless of the extent of change in medical education after the pandemic, there will be greater acceptance of the pedagogical potential of emergent technology such as adaptive learning [18] . They believe that transformative change is likely to occur in medical education after COVID through innovative and trending technology, including adaptive software [18] . Likewise, the flipped classroom has been advocated by medical professionals in the United States to maintain rigorous educational training during COVID-19, including for surgical residency programs [11] . Other US medical professionals have suggested that the online infrastructure, videos, and curricular flexibility resulting from COVID should be leveraged to "modernize" medical education via the implementation of the flipped classroom on a wider scale [4] . Given the severe challenges that medical education faced during the pandemic, engineering education can benefit from the perspectives and experiences of medical education. In addition, engineering and medical education are similar vis-à-vis their science, practice-based, and hands-on educational focus. Social scientists Brewer and Hunter state that different disciplines should learn from one another. "…Cross-overs of theories, or applications of a theory developed in one subfield to another may provoke new questions, and provide useful insights, and suggest new ways of looking at phenomena" ( [2] , p. 74). Various articles have "advocated for" flipped instruction during the pandemic and beyond. In an article in an Asian educational journal, the flipped classroom was recommended for online university education in the wake of COVID-19, including the use of instructor incentives [49] . The flipped classroom was also advocated for developing countries during COVID-19, as prerecorded videos relieve some of the internet burden imposed by synchronous platforms [42] . Other articles have suggested that flipped instruction was strengthened during the pandemic. A summary at one Peruvian and two Spanish universities led to the conclusion that prerecorded videos will remain, given the positive impact on teaching practices via the flipped classroom in engineering and construction courses during the pandemic [29] . At another Spanish university, there were a significant increase in the number of flipped classroom sessions and video and audio files during the lockdown, based on a faculty survey [9] . They stated that the "flipped classroom is boosted by the circumstances." ( [9] , pp. 10). Multiple articles have emerged globally, describing positive outcomes from the flipped classroom during the pandemic. In a college of medicine in the UAE, an "e-flipped classroom" model was used with success during the pandemic, citing benefits of reinforcement of topics with multiple coverage opportunities (i.e., pre, during, and post-class) and high student engagement [20] . At a university in China, the flipped classroom was associated with improved learning, attention, and concentration of engineering students in the online environment [44] . Due to the COVID-19 pandemic, flipped teaching and learning in an undergraduate numerical methods course at a US public research university was employed in the online environment during the fall 2020 and spring 2021 semesters. Additionally, during the spring of 2021, adaptive learning lessons were used for all course topics in the flipped classroom to enhance the pre-class, independent learning component. The numerical methods course covered the topics of differentiation, nonlinear equations, simultaneous linear equations, interpolation, regression, integration, and ordinary differential equations. The participants consisted of undergraduate mechanical engineering majors. The number of students for whom we had demographic, final exam, and concept inventory data that could be "joined" for analysis was as follows: 86 (fall 2020) and 78 (spring 2021). This number represented 88% and 79% of enrolled students in the fall and spring semesters, respectively, for a combined percentage of 83% of enrolled students. To prepare for the online class without the adaptive learning software (fall 2020), students watched You-Tube video lectures or read the textbook, attempted a quiz via the course learning management system (LMS) (i.e., Canvas), and responded to an open-ended question (also via the LMS) on challenging or interesting topics. With the addition of adaptive learning, students prepared for the class using adaptive lessons implemented in the Realize IT platform (https://realizeitlearning. com/). These lessons similarly consisted of video, text, simulations, and multiple-choice and algorithmic quiz questions for all course topics. Students continued to submit their open-ended "challenges and interesting items" responses via the LMS. Students could retake a lesson as many times as desired, with each try getting them closer to mastery-level achievement (i.e., 90%) and full points on the lesson. The adaptive-learning preclass work was worth 18% of the final course grade. In the fall of 2020 when adaptive lessons were not used, the pre-class preparation was worth 15% of the final course grade. Relative to class time in the online flipped environment (both with and without adaptive learning), the objective was to maintain active and interactive learning, as in the onsite classroom. Thus, synchronous sessions using Blackboard Collaborate Ultra (BBCU) and its breakout rooms, clicker questions via Microsoft Forms, and mini-lectures to address challenges were utilized. The breakout rooms each contained approximately four students for group problem solving and were formed randomly. The Proctorio software was used during the open-notes, open-book exams for proctoring purposes (https://proctorio.com). The concept inventory was a closed-notes, closed-book assessment for which proctoring software was not used. Students' post-class responsibilities were the same with and without the use of adaptive lessons. Here, students completed a quiz after class and worked on problem sets and programming projects outside of class. A visual summary of these two classroom implementations is provided in Figure 1 , with the only differences being the pre-class activities. A final exam consisting of 14 multiple-choice and four freeresponse questions was administered. The electronically administered multiple-choice questions remained exactly the same with and without the use of adaptive software (i.e., across the fall and spring semesters). The free-response questions differed between the two semesters due to having to administer the exam remotely and not being able to "collect" the exam afterwards. However, the instructor created the free-response questions to be of similar difficulty and topic areas between the two semesters. The freeresponse questions assessed the higher-levels of Bloom's taxonomy, while the multiple-choice questions tested the lower-level skills [46] . Each free-response question was graded by the instructor (i.e., second author) on a 0-4 scale using an existing rubric. The rubric levels corresponded to no, little, partial, considerable, and complete understanding, respectively. A numerical methods concept inventory that had been developed under previous NSF funding was also administered to assess conceptual understanding. This test did not allow any notes to be used. It centered on critical thinking, logic, and the identification of misconceptions [24] . The inventory consists of six key concepts associated with the largest number of misconceptions as identified by an expert team using a Delphi process [24] . The averages for the multiple-choice, free-response, and concept inventory questions were each adjusted using the student's pre-requisite GPA, which was calculated based upon six math, science, and programming courses and served as a control variable. The adjustment and statistical comparison of the two cohorts were made via the analysis of covariance (ANCOVA) procedure in SPSS, a parametric test that used pre-requisite GPA as a covariate to account for historical academic performance [33] . As some of the demographic segments had smaller sample sizes, p values from the parametric ANCOVA procedure were corroborated by p-values from the analogous non-parametric procedure known as Quade's test, which uses ranked data and was run using SPSS [27, 39] . Conversely, to assess practical significance, Cohen's d effect sizes were calculated [26, 43] . The flipped classroom without adaptive learning (i.e., fall 2020) was considered the reference category in calculating the effect size. A confidence interval for each d was also determined [12] . Based on traditional ranges, the calculated effect sizes were generally small to medium. However, they were in line with effect sizes reported in metaanalyses of the flipped classroom, some of which were discussed in the literature review. A demographic survey was administered to collect the pre-requisite GPA and other demographic data, including gender, race/ethnicity, and community-collegetransfer and Pell-grant statuses. Pell grant status was used as a proxy for socioeconomic background, as these grants are usually awarded only to students with high F I G U R E 1 Online flipped instruction with and without adaptive learning financial need, as determined in part by income [14] . On the demographics survey, students entered a study code, which was matched to the code provided by the student on the final exam and concept inventory. This code enabled us to conduct GPA-adjusted exam and conceptinventory averages as well as stratified analysis of average scores by demographic group. Two instruments were administered anonymously to gather student perspectives on the classroom environment, flipped classroom, and the use of adaptive learning. This was done for both semesters, enabling a direct comparison between them on the classroom environment and flipped instruction in the remote, online environment. The College and University Classroom Environment Inventory (CUCEI) was used to measure the classroom environment [16] . This validated instrument measures multiple psychosocial dimensions of the classroom, such as innovation (i.e., unique class activities or teaching approaches), personalization (i.e., interaction with the instructor & concern for student welfare), individualization (i.e., individual or differential treatment of students who are empowered to make decisions), and satisfaction (i.e., enjoyment of classes). The dimensions of the CUCEI align with the objectives of both the flipped classroom and adaptive learning. The CUCEI data for the two semesters were compared using MANOVA, or multivariate analysis of variance, given the seven dependent variables (i.e., seven CUCEI dimensions) [33, 15] . The CUCEI has a 1 to 5 scale, with 5 being the most desirable. As seven dependent variables (i.e., dimensions) were tested, each of the seven univariate p-values was adjusted using the Bonferroni correction [36] . Specifically, each univariate p-value was multiplied by seven, and this adjusted p-value was compared to α = .05 to determine statistical significance. Practical significance was assessed using Cohen's d effect size. For the online environment, one question from the CUCEI was removed because it pertained to classroom seating. A median substitution was done for this item using the median item value from the onsite implementation of the flipped classroom. For the spring 2021 CUCEI data, the median value from the onsite flipped classroom with adaptive learning was used, as this data had been collected during initial research with adaptive learning [7] . For the fall 2020 CUCEI data, the median from the onsite implementation of the flipped classroom without adaptive learning was used, also collected during previous research [8] . A second survey, which consisted of a mixture of closed and open-ended questions, was administered to collect student perspectives relative to their motivation, learning, perceived responsibility, and preferences with flipped instruction during both semesters. Open-ended questions were used to gather student beliefs about the benefits and drawbacks of flipped instruction as well. Additionally, during the semester with adaptive learning (spring 2021), questions were posed to gather perspectives on the impact of adaptive lessons on learning and engagement. The open-ended responses underwent a content analysis (i.e., double-coding) by two analysts [31] . Using an established coding scheme, each analyst coded the responses independently, subsequently discussing their codes to reach a consensus on all student responses [8] . The inter-rater reliability was measured using Cohen's κ [33] . The following Kappa values were achieved: κ = 0.80 for the benefits responses and κ = 0.76 for the drawbacks/suggestions, suggesting strong agreement beyond chance. Proportions for the various assigned codes and survey categories were compared using a z-test of proportions to assess statistical significance and the odds ratio, an effect size measure, to assess practical significance [43, 45] . The study was reviewed by the institutional review board (IRB) of both universities, and both determined that the protocol met the criteria for exemption from review. The results from the classroom environment inventory (CUCEI) comparing the online flipped classroom with and without adaptive learning software are presented next. The response rate for the CUCEI was 85% of students enrolled in the course during the online flipped semester (fall 2020) and 75% enrolled when adaptive learning was additionally employed (spring 2021), for a strong overall response rate of 80%. Results are presented as dimension means on the 1-5 CUCEI scale, with 5 being the most desirable. The innovation dimension was associated with the largest increase in the use of adaptive software. This increase was significant after correction with the Bonferroni adjustment (p = .007) and had an effect size of d = 0.54 (Table 1 ). The innovation dimension measures novel teaching practices or classroom activities. The students likely viewed the adaptive lessons as innovative or unique. Two other CUCEI dimensions with noteworthy increases with the addition of adaptive software were the satisfaction and personalization dimensions. After correction using the (highly conservative) Bonferroni adjustment, the p value for the test of differences in the satisfaction means was .063, with an effect size of d = 0.42. Thus, students had greater satisfaction, including enjoyment of classes, with the addition of the adaptive lessons. The personalization dimension was similar in that, upon adjustment, the p-value for the difference in the means was 0.070, and the effect size was d = 0.41. The personalization dimension captures perceived interaction with the instructor and concern for student welfare. Thus, three of the seven CUCEI dimensions had statistically significant or otherwise noteworthy increases with the use of adaptive software. In fact, all seven dimensions increased with the addition of the adaptive lessons to the online flipped classroom, although some increases were very small. Thus, the use of adaptive lessons may be beneficial for the classroom environment in a flipped online course. Results from both closed and open-ended questions on the flipped-classroom evaluation survey are presented next. Results from two open-ended questions on perceived benefits and drawbacks of the flipped classroom are presented first, followed by a selection of closedended questions. The response rate for the evaluation survey was 84% of enrolled students during the fall 2020 semester and 77% during the spring 2021 semester, for an overall response rate of 80%. Relative to benefits, as discussed in an open-ended fashion by students, there was an increase in the perception of enhanced learning or learning processes when the adaptive lessons were available in the online flipped classroom. The increase was from 34% to 42% of total student responses (Table 2) . Thus, 42% of the responses mentioned the benefit of enhanced learning or learning processes when adaptive software was used in the online flipped classroom, versus just 34% who mentioned this in the online flipped classroom without adaptive lessons. However, these proportions were not significantly different based on a z-test of proportions (p = .32), and the effect size of the odds ratio (OR) = 1.42 was small. Conversely, there was a decrease in the proportion of students who perceived "Video/Online Learning" as a benefit. This category pertains to re-watching videos and recordings, being able to work at one's own pace and time, and the associated flexibility and conveniences. The percentage of responses decreased from 49% to 32% in going from the nonadaptive to the adaptive online semester (i.e., fall 2020 to spring 2021). These two proportions (49% vs. 32%) were significantly different based on a z-test of proportions (p = .041). The OR (i.e., effect size with 49% in the numerator) was medium at 2.03. In interpreting this finding, the expectation of recorded content in the online environment (e.g., Zoom recordings, Panopto videos, etc.) may have increased as the remote instructional period due to COVID went on, leading to the belief that recorded content was no longer a benefit per se, but rather a basic feature of any course. Abbreviations: Coh, cohesiveness (students know & help one another); Indiv, individualization (students treated individually/differentially & can make decisions); Inn, innovation (Novel class activities or teaching techniques); Invol, involvement (active student participation in class activities); Pers, personalization (Interaction w/instructor & concern for student welfare); Satis, satisfaction (Enjoyment of classes); Task Or, Task orientation (Organization and clarity of class activities). On the other hand, however, there may be greater acceptance of learning with recorded content postpandemic, potentially "helping" the flipped classroom. For, during the pandemic, it became mainstream to learn independently using prerecorded content. This recognition starkly contrasts with beliefs about the flipped classroom as a method by which "instructors don't have to teach" [22] . Changes in the other benefits categories in Table 2 showed minor changes or no change. Two notable decreases were observed in the open-ended drawbacks and suggestions mentioned by students when adaptive learning was available in the spring 2021 semester versus when it was not in the fall 2020 term. First, as shown in Table 3 , there was a decrease in the percentage of students who experienced load, burden, or stressors in the online flipped classroom when adaptive learning was available. Specifically, the percentage decreased from 31% to 16%. This category pertains to items such as the required amount of time or work, grade concerns, or feelings of having to "teach oneself" or "learn on one's own." These percentages were significantly different based on a z-test of proportions (p = .036), with a medium effect size OR = 2.4. It is possible the helpfulness of the adaptive lessons contributed to the perception of lessened burden or stressors. In a separate closed-ended question, 78% of students responded to the statement The adaptive lessons were helpful to me using the categories "mostly true" or "always true." This was opposed to indicating only "sometimes true" or "not true at all." It is also possible that as time went on in the remote instructional period, students became more accustomed to learning from recorded or online content, lessening feelings of any stress or burden imposed by a nontraditional learning environment. The second notable decrease in open-ended drawbacks and suggestions was related to the use of class time in the flipped classroom, including the amount of content review, amount and type of active learning, and quality of group work. It is possible the individualized learning paths created by the adaptive software may have contributed to a lessened need for in-class review of content by the instructor. As shown in Table 3 , the percentage for the "Class Time Usage" category decreased from 29% to 11%, and these proportions were significantly different (p = .011), with a large effect size (OR = 3.2). A notable increase in perceived drawbacks and suggestions occurred with the "Prepare, Equip, & Incentivize Students" category. The total proportion of responses increased from 25% to 40%. These two proportions were not significantly different (p = .057), although the effect size was medium at OR = 2.0. Thus, students requested substantially more support and resources during the semester in which adaptive learning was used. This category included mentions of the adaptive software not providing a sufficient amount of feedback or guidance when incorrect answers were submitted. However, the somewhat limited feedback was partly "by design" to drive enhanced critical thinking during the lessons and less reliance on a "solutions manual" approach to learning. In addition, the students had multiple avenues to pose their questions, including a discussion board, daily office hours, and email. Based on this student feedback, however, the instructor was inclined to add small hints to particular questions within the lessons. Results related to student preferences for the flipped classroom, perceived learning gains, imposed responsibility, and motivation are presented next, with a direct comparison of the online versus "online with adaptive learning" environments. Preference for flipped classroom Survey question. Do you prefer a flipped classroom over the usual method of instruction? The percentages associated with this question in both the online and online + adaptive learning environments are shown in Table 4 . There was a statistically significant increase from 23% to 42% in the proportion who preferred the online flipped classroom when adaptive learning software was used (p = .011). The OR, a measure of effect size, was mediumto-large at 2.41. Thus, the use of adaptive learning software may be associated with a positive, significant effect on student preference for the flipped classroom in the online environment. Learning gains Survey question. I had greater learning gains with the flipped classroom versus the usual method of instruction. The results of this statement also potentially point favorably to the use of adaptive lessons in the online flipped classroom. There was a significant increase from 19% to 38% in student perceptions of greater learning gains in the online flipped classroom that contained the adaptive lessons (i.e., agree or strongly agree in Table 5 ). These proportions were significantly different (p = .009), and the OR was medium-to-large at 2.55. Responsibility imposed Survey question. With the flipped classroom, how would you rate the responsibility placed on you compared to the usual method of instruction? There was a decrease in the belief that more or much more responsibility was placed on the student in the online flipped classroom containing the adaptive learning software. As shown in Table 6 , this proportion significantly dropped from 78% to 63% (p = .040) with a medium effect size of OR = 2.07. Thus, the use of adaptive learning software may assist the online flipped classroom in offering students pre-class support and scaffolding through personalized and diversified instruction with immediate feedback. Survey question. With the flipped classroom, I had the motivation to engage in the necessary learning outside the classroom. Academic motivation in higher education has been a highly-concerning issue in the remote learning environment (Clark et al., submitted 2021). However, when adaptive lessons in the online flipped classroom were available, there was a significant increase in student motivation for the independent learning component, jumping from 32% to 47% upon combining the "Agree" and "Strongly Agree" categories in Table 7 . These proportions (32% vs. 47%) were significantly different (p = .044), with a medium OR = 1.94. Thus, adaptive learning software may boost student motivation for the online flipped classroom, particularly the student-initiated learning that must precede active class time. Adaptive learning gains Table 8 provides results on student perceptions of learning gains when flipped instruction is combined with adaptive lessons. This is in comparison to usual teaching approaches as well as other flipped courses without adaptive lessons. Thirty-eight percent agreed or strongly agreed with the first statement, while 41% did so with the second statement. This latter result was a good outcome for the use of adaptive learning lessons in the flipped classroom. Finally, using a scale of not true, sometimes true, mostly true, and always true, one-third of respondents chose "mostly or always true" to the statement, The adaptive lessons were what I liked most about this class. This was also a good finding for adaptive learning. With the multiple-choice questions (Table 9) , there was a small, nonsignificant increase in the adjusted average score with the use of adaptive learning software, from 56.1% to 57.6%, and an effect size of d = 0.08. More favorable results were found with the community-college transfers and Pell grant recipients, each associated with an effect size of d ≈ 0.40 with the use of adaptive learning. Their averages were not significantly different, however, as shown by the various p values in Table 9 . The p values are provided for both the ANCOVA (i.e., parametric) and Quade's test (i.e., non-parametric) procedures and are presented vertically, respectively, within the various cells containing p-values. Each unadjusted p-value was adjusted using the Bonferroni correction, and these adjusted p values are shown in the column to the right of the unadjusted p values in Table 9 . Unfortunately, females experienced a decline in multiplechoice performance when adaptive learning was used, with an effect size similar in magnitude to that of the Pell grant and CC transfer students (d = −0.38). Recall that averages were adjusted using the pre-requisite GPA as a control variable to take the student's academic background into account. For the free-response exam questions, the adjusted average for all students was significantly lower when adaptive learning was used (p = .030), based on the adjusted ANCOVA (i.e., parametric) p value in Table 10 . The effect size was d = −0.44. This result was opposite to the result from the multiple-choice questions. With the free-response questions, despite efforts by the instructor to equalize the difficulty levels, it may have been difficult in reality to achieve true equivalency. The multiple-choice questions remained exactly the same from semester to semester. The only demographic segment that showed an increase in free-response averages with adaptive learning was the Pell grant recipients, further suggesting these students may benefit from personalized learning and other benefits of adaptive lessons. Females again experienced a decrease in As with the multiple-choice averages, the concept inventory averages showed a small, nonsignificant increase when adaptive learning was used, with d = 0.14. Again, the Pell grant recipients experienced the largest positive effect with adaptive learning, with d = 0.30 (Table 11 ). This effect suggests again that Pell grant recipients may benefit from the use of adaptive learning software in the online flipped environment. Flipped instruction in a numerical methods course during the remote instructional period of the COVID pandemic was conducted with and without the use of adaptive learning lessons. This was done to investigate potential differences in student perceptions and performance with and without personalized learning for class preparation in the remote setting. This study was done at a public research university, where previous extensive research had been conducted with flipped instruction as well as adaptive learning in the flipped classroom since 2013. Results with the use of adaptive learning software (vs. without) in the flipped online classroom were promising, with an increase in all seven of the CUCEI classroom environment dimensions. The innovation dimension (i.e., novel teaching) significantly increased (p = .007), with d = 0.54. The personalization and satisfaction dimensions had noteworthy increases also. Student preference for flipped instruction, motivation for outside-ofclass learning, and perceived greater learning gains each showed a statistically and practically significant increase when adaptive learning software was used versus not. This is noteworthy because student motivation and perceived learning have declined in higher education with the COVID-induced remote instruction (Clark et al., submitted 2021). Also, there was a significant decline in the proportion of students who perceived more responsibility with flipped instruction when the adaptive lessons were available versus not. Thus, adaptive lessons may assist the online flipped classroom by virtue of outside-of-class support, individualized instruction, and immediate feedback to students. Similarly, based on a content analysis of an open-ended question, there was a significant decrease in Note: The online flip method is the reference category for computing the effect size. the proportion of students who experienced load, burden, or stressors in the online flipped classroom when adaptive learning was used versus not, serving to triangulate these results. Relative to direct assessment of student learning, multiple-choice exam and concept-inventory results were higher when the adaptive lessons were available versus when they were not, although not significantly so. The most promising results with the direct assessments occurred with Pell grant recipients. As a whole, our results for adaptive learning in the online, flipped classroom begin an advocacy for the use of this technology in these types of circumstances. This aligns with the recent and emerging literature, including in medical education, that suggests the pivotal role that adaptive learning will play in higher education in the future, including for flipped instruction. As the free-response questions had to be altered from fall 2020 to spring 2021 due to remote test-taking conditions during COVID-19, an exact comparison of higher-order skills via these questions was not possible, potentially impacting the results and findings of this study. Given that the multiple-choice and concept inventory averages were both higher with the use of adaptive learning (albeit non-significantly so), it seems reasonable that the differences in the free-response questions between the semesters may have been a limiting factor. Our study was quasi-experimental as students could not be randomly assigned to the various types of classrooms. The sample sizes for some of the demographic segments were small, reducing the power and robustness of the parametric statistical tests with the exam and concept inventory scores. However, the parametric results were corroborated using the equivalent non-parametric procedure Quade's test. Emerging literature has suggested that the remote, online learning environment created by the COVID-19 pandemic may be beneficial for flipped instruction going forward. Furthermore, emerging literature is suggesting the pivotal role that adaptive learning software can and will play with the flipped classroom and higher education in general after the pandemic. The present study also suggests the desirability of adaptive learning, particularly enhanced student perceptions across multiple items when adaptive lessons were used in an online flipped classroom. These items included perceived learning, outside-of-class motivation, preference for flipped instruction, classroom environmental dimensions, and lessened feelings of burden, stress, and personal responsibility. Unfortunately, the direct assessment results were not highly conclusive at this point in favor of adaptive learning in the flipped classroom, but that is often the case when evaluating interventional successes via traditional exams. Noteworthy and/or significant direct-assessment outcomes may result for particular demographic groups, however. This observation should become more evident, however, upon our continued research with adaptive learning in the flipped classroom across multiple schools over the next several years. The authors recommend adaptive learning for enhanced pre-class preparation for flipped as well as blended instruction. Adaptive learning can be used by all students to self-pace their learning, including with pre-requisite content needed for their advanced courses. In the instructor's experience, adaptive learning addresses notable challenges and gaps in required, pre-class learning with flipped instruction. Specifically, he noted that students did not arrive as prepared for active learning when the "one size fits all" preparation approach was used. Thus, the authors suggest an adaptive learning platform to accommodate students individually and quickly inform the instructor when students are struggling based on its data collection and dashboard capabilities. The flipped classroom: A review of its advantages and challenges Multimethod research: A synthesis of styles Academic outcomes of flipped classroom learning: a meta-analysis COVID-19 can catalyze the modernization of medical education Impact of COVID-19 on engineering students at a U.S. University in the middle phases of the pandemic: Academic motivation, perceived stress, and valued university experiences Benefits of adaptive lessons for preclass preparation in a flipped numerical methods course Adaptive learning in a numerical methods course for engineers: evaluation in blended and flipped classrooms Evaluating blended and flipped instruction in numerical methods at multiple engineering schools Flipped classroom: Active methodology for sustainable learning in higher education during social distancing due to COVID-19 On teaching adaptively Using technology to maintain the education of residents during the COVID-19 pandemic A primer on the understanding, use and calculation of confidence intervals based on central and noncentral distributions Educause Learning Initiative. 7 Things you should know about… Adaptive learning Discovering statistics using SPSS Validity and use of an instrument for assessing classroom psychosocial environment in higher education Challenges and opportunities of preclinical medical education: COVID-19 crisis and beyond, SN Compr A vision of the use of technology in medical education after the COVID-19 pandemic The global state of the art in engineering education Combating the COVID-19 outbreak with a technology-driven e-flipped classroom model of educational transformation Flipped classroom improves student learning in health professions education: A meta-analysis The flip side of flipped classrooms A systematic review of research on the flipped learning method in engineering education Building a concept inventory for numerical methods: A chronology Analyzing the use of adaptive learning in a flipped classroom for preclass learning Reporting and interpreting effect size in quantitative agricultural education research Rank analysis of covariance: Alternative approaches The impact of flipped classrooms on student achievement in engineering education: A meta-analysis of 10 years of research Some web-based experiences from flipped classroom techniques in AEC modules during the COVID-19 lockdown The content analysis guidebook, Sage Publications COVID-19 and education system: Impact of current pandemic on adaptive learning strategies in medical education system The impact of COVID-19 on medical education in India What's wrong with Bonferroni adjustments Engaging students in learning threshold concepts in engineering mechanics: Adaptive eLearning tutorials Adaptive tutorials to target threshold concepts in mechanics-A community of practice approach Rank analysis of covariance Georgia State improved its graduation rate by 22 points in 10 years, The Atlantic Innovative trends in flipped teaching and adaptive learning A hybrid flipped-classroom approach for online teaching of biochemistry in developing countries during COVID-19 crisis Using effect size-Or why the p value is not enough Efficiency of flipped classroom with onlinebased teaching under COVID-19 Probability & statistics for engineers & scientists Understanding by design Trends and development in technology-enhanced adaptive/personalized learning: A systematic review of journal publications from Cardiology training in times of COVID-19: Beyond the present The performance of online teaching for flipped classroom based on COVID-19 aspect This material is based upon work supported partially by the National Science Foundation under Grant Number 2013271, and the Research for Undergraduates Program in the College of Engineering at the University of South Florida (USF). Any opinions, findings, and conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. We wish to thank the following undergraduates: Kayla Edwards of the University of Pittsburgh, and Luis Serrano, Jonas Vinol Fernandes and Nicole Figueroa of the University of South Florida who provided invaluable assistance. The authors declare that there are no conflicts of interest. http://orcid.org/0000-0002-1782-073X