key: cord-0046908-yom1uum6
authors: Huang, Yun; Aleven, Vincent; McLaughlin, Elizabeth; Koedinger, Kenneth
title: A General Multi-method Approach to Design-Loop Adaptivity in Intelligent Tutoring Systems
date: 2020-06-10
journal: Artificial Intelligence in Education
DOI: 10.1007/978-3-030-52240-7_23
sha: 63b57a608054b44ed9422b3bd9b10d36078126c3
doc_id: 46908
cord_uid: yom1uum6

Design-loop adaptivity, which involves data-driven redesign of an instructional system based on student learning data, has shown promise in improving student learning. We present a general, systematic approach that combines new and existing data mining and instructional design methods to redesign intelligent tutors. Our approach is driven by the main goal of identifying knowledge components that are demonstrably difficult for students to learn and to optimize effective and efficient practice of them. We applied this approach to redesigning an algebraic symbolization tutor. Our classroom study with 76 high school freshmen shows that, compared to the original tutor, the redesigned tutor led to higher learning efficiency on more difficult skills, higher learning gain on unscaffolded whole tasks, and more robust transfer to less practiced tasks. Our work provides general guidance for performing design-loop adaptations for continuous improvement of intelligent tutors.

Design-loop adaptivity [1] uses student learning data to drive instructional decision making for design and iterative improvement of a course or system. It is part of a broader set of endeavors of data-driven instruction and learning designs for continuous improvement in classrooms and schools [2] [3] [4] [5] . This paper focuses on the context of intelligent tutoring systems (ITSs), a widely adopted and proven technology, where empirical research on data-driven redesign and optimization is still lacking. Numerous data mining methods have been demonstrated to improve prediction accuracy using data from fielded ITSs [6] [7] [8] , but most stop at better predictions without demonstrating whether and how these methods can improve student learning. One reason for a shortage of such "close the loop" experiments may be that there is no good general guidance for how to convert data-mining outcomes into better tutor design. Prior closethe-loop studies [9] [10] [11] were often driven by a limited set of methods or narrow redesign features. This paper demonstrates a general, systematic approach that combines new and existing data mining and instructional design methods to redesign ITSs.

We applied this approach to redesign an algebraic symbolization tutor, and provide empirical evidence of its effectiveness through a classroom study comparing the redesigned tutor to the original tutor.

2 Method: How to Use Data to Improve Intelligent Tutors Our approach focuses on the continuous improvement of ITSs by mining tutor log data collected from previous iterations. It starts with knowledge component (KC) refinement, followed by content and task selection revisions aiming at more effective and efficient practice of KCs. Our process ( Table 1 ) is generalizable to other domains and other ITSs grounded in a KC approach [12] to instructional design. [17] We applied this process to redesigning the Algebraic Expressions unit in Mathtutor [18] , a free online tutor based on prior instructional design research [19] (Fig. 1) . We utilized prior log data from 356 students with 50,279 student steps. We describe our new methods below, and refer readers to prior work for existing methods.

Difficulty Factor Effect Analysis. A difficulty factor (DF) refers to a property that makes some tasks more difficult than other comparable tasks. We first identified a broad set of potential DFs by coding task features hypothesized to impact difficulty (e.g., requiring parentheses or not). Then we ran a regression for each targeted KC to examine the main and interaction effects of potential DFs on performance, controlling for student proficiencies and learning from prior opportunities. These regressions might be viewed as an efficient simplification of LFA [20] . A KC was split by a set of DFs when there was an interaction or by a DF when there was a main effect.

Probability-Propagation Practice Estimation. We estimated the number of opportunities needed for mastery by fitting parameters of a student model (e.g., BKT [21] ) to the data and used the parameters to estimate knowledge for each step. We then compared the estimates to actual opportunities to get the over-or under-practice. Instead of simulating many sequences by propagating simulated outcomes [22] , we simulate one sequence by propagating the probability of succeeding, and use it as weights to update knowledge (i.e., P(L) new = P(C)P(L|C) new + P(W)P(L|W) new ). The extrapolation of a KC-student sequence stops when P(L) new ! .95, or the extrapolated opportunities reach a threshold (e.g., 20). Our offline experiments showed that our method reached similar estimations as the method in [22] with higher efficiency.

Focused Practice Task Design. We created new focused problems for hard KCs which eliminate interface steps of easier KCs (Fig. 2) . These problems aim to reduce under-practicing hard KCs and over-practicing easier KCs, which were prevalent in the original tutor according to our practice estimation. We introduced composition scaffolding (inspired by prior work [14, 23, 24] ) to break down problems (Fig. 2a) , because our method estimated that many opportunities ( ! 58) would be needed to master each difficult KC (two-operator KCs) in the original tutor. Fig. 2 . Focused tasks target a KC that data reveals is particularly hard. a) A focused whole task practices this KC in more realistic problems that require mental steps of easier KCs along with the hard KC. If students fail on the whole task, dynamic composition scaffolding isolates the individual KCs (step 1-3) including the hard one (step 3). b) A focused part task practices the hard KC in isolation without the busy work (mental and interface steps) of the easier KCs.

We conducted a classroom study to investigate whether the data-driven redesigned tutor (treatment condition) yields better learning than the original tutor (control condition). We ran the study in two high school freshman Algebra I classes during three 40-min periods for one week in 2019. Students were randomly assigned to conditions within each class, with 38 students per condition. We used linear mixed models to examine learning gains and two-sample t-tests to examine practice time (Fig. 3) .

Overall, both conditions produced significant learning gains (ps < .01). Students in the treatment condition spent 13% less practice time than students in the control condition (p < .1), with no difference in gains (p = .94). In particular, on more difficult skills, two-operator skills, they spent 19% less time (p < .05) with no difference in gains (p = .78). Treatment students had significant gains on algebraic modeling (AM) in both unscaffolded whole tasks (Fig. 2a without scaffolding steps) and table tasks ( Fig. 1 ) (ps < .05) even with little practice on table tasks. Control students had a significant gain on AM table tasks (p < .001) with no difference from treatment students (p = .14), and no improvement on unscaffolded whole tasks (p = .58) with the gain different from that of treatment students (p < .05). These results suggest that treatment students acquired more robust, transferable learning. Control students had marginally higher gains on arithmetic solving (AS) (p = .096); the treatment condition was designed to shift practice away from these (easier) skills to the harder AM skills.

We demonstrate a general multi-method approach to design-loop adaptivity and provide empirical evidence of its effectiveness. The results are encouraging, although they are not as pronounced as anticipated. Students spent much less time on the tutor than planned, but our theoretical predictions were based on longer time. Thus, a more stringent test of our approach requires a future longer span study. After all, design-loop adaptivity is intended as an iterative process. Our work provides general guidance for how to convert data-mining outcomes into better tutor design, an important need in AIED/EDM research and practice. Our work may also help define and enhance datadriven learning engineering processes. 1 

Instruction based on adaptive learning technologies

The new instructional leadership: creating data-driven instructional systems in school

Informing pedagogical action: aligning learning analytics with learning design

The RISE framework: using learning analytics to automatically identify open educational resources for continuous improvement

Data-driven decision making in the K-12 classroom

Advances in Neural Information Processing Systems

Automatic discovery of cognitive skills to improve the prediction of student learning

General features in knowledge tracing to model multiple subskills, temporal item response theory, and expert knowledge

The Open Learning Initiative: Measuring the effectiveness of the OLI statistics course in accelerating student learning

Using data-driven discovery of better student models to improve student learning

Evolution of an intelligent deductive logic tutor using data-driven elements

Knowledge component approaches to learner modeling

Human-machine student model discovery and improvement using datashop

Seeing language learning inside the math: cognitive analysis yields transfer

A data repository for the EDM community: the PSLC DataShop

Is over practice necessary? improving learning efficiency with the cognitive tutor through educational data mining. Front

Robust evaluation matrix: towards a more principled offline exploration of instructional policies

The frequency of tutor behaviors: a case study

Illustrating principled design: the early evolution of a cognitive tutor for algebra symbolization

Learning Factors Analysis -A General Method for Cognitive Model Evaluation and Improvement

Knowledge tracing: modeling the acquisition of procedural knowledge. User Model. User-adapted Interact

The impact on individualizing student models on necessary practice opportunities

The composition effect in symbolizing: the role of symbol production vs. text comprehension

The ASSISTments ecosystem: building a platform that brings scientists and teachers together for minimally invasive research on human learning and teaching