key: cord-0046959-z1fyodmx
authors: Razzaq, Renah; Ostrow, Korinn S.; Heffernan, Neil T.
title: Effect of Immediate Feedback on Math Achievement at the High School Level
date: 2020-06-10
journal: Artificial Intelligence in Education
DOI: 10.1007/978-3-030-52240-7_48
sha: 4a4d8ea9e1162313302a4cb6e71cf6ee49fb81c0
doc_id: 46959
cord_uid: z1fyodmx

We examine the use of computer-based learning in the classroom and the effect of immediate feedback on student performance. Since it is well known in educational research that it is possible to observe a “Matthew Effect” in which the rich get richer, we wanted to see if feedback was useful for low prior knowledge students, as defined by students whose pretest score was at or below the median. In this counterbalanced randomized controlled trial, 243 tenth and eleventh grade mathematics students were exposed to one of two conditions, as we measured their learning from: 1) immediate feedback (where the computer told them correctness and they could also ask for hints) or 2) practice only (where they received feedback only after taking a posttest). Results suggest that immediate feedback from computer-based learning tasks benefit both high and low prior knowledge students, with low prior knowledge students exhibiting greater gains. The implications of these findings support further investigation into the use of computer-based learning tasks that provide immediate feedback.

Despite widespread acceptance of feedback in computer-based instruction, empirical support for varying types of feedback has been inconsistent and contradictory [1] . Recent meta-analyses claim that the type and timing of feedback can have an effect on student learning [2] .

Studies in the field of learning science have posited several questions about feedback. What does productive or effective feedback look like when students are solving a problem or working on a learning task? How does feedback affect student performance? Do the effects of feedback vary from student to student? Ideally, feedback on learning tasks should be constructed for each individual student. Researchers have defined feedback-specifically formative feedbackas information communicated to the student with the intent of modifying their thought process on a learning task and improving their performance. Formative feedback, should be non-evaluative, supportive, timely and targeted [3] . This feedback is commonly offered to students after they started working on a task and is presented in a variety of ways. Examples include informing the student of correctness or errors, providing hints, or providing completed examples of the problem at hand or of a similar problem. Feedback can be presented to the student during the learning task, immediately after its completion or after some interval of time.

A prior experiment with eighth grade students revealed that computer-supported homework can lead to better results over traditional paper-and-pencil homework [4] . The study compared the use of immediate feedback and tutoring (treatment) to a control condition in which students received feedback the next day in math class. The study was conducted using, ASSISTments, the online learning platform also used in the present work. The original experiment was counterbalanced, with each student receiving each condition. The data collected suggested that students learned significantly more (effect size 0.40) with computer-supported homework. This result had practical significance, suggesting an improvement over widely used paper-and-pencil homework [4] .

In other related work, high school students participated in a controlled evaluation of an interactive online tutoring system for math achievement test problem solving [5] . A sample of 202 students completed a pretest, were randomly assigned by their teacher to receive either online tutoring (treatment) or regular classroom instruction without online tutoring support (control). The posttest revealed that students using the online platform performed better than students who received only classroom instruction. The control group showed no improvement on the posttest. The use of multimedia hints predicted pre-to posttest gains and the benefits of tutoring were greatest for students exhibiting the weakest initial math skills [5] .

With this past work as context, the present work examines the effects of immediate feedback in comparison to no feedback. Specifically, a randomized controlled trial is used to answer:

Research Question 1: Are there differences in student learning when comparing a computer feedback condition (or "Tutor Mode") to a business-as-usual condition (or "Test Mode") that simply provides practice? Research Question 2: Is computer feedback effective for both high prior knowledge and low prior knowledge students? 2 Methods and Experimental Design

Student inclusion was based on teachers who volunteered to participate in this study. Content used for the assignment in ASSISTments was identical to the content that students would have received if not participating in the study. Participants included 243 students across 10 classrooms. The demographic breakdown was 15.9% African American, 7.1% Asian, 42.6% Hispanic, .2% Native American, 30.2% White, and 4.2% Multi-Race, Non-Hispanic. Fifty-five percent of students in the district from which the sample was drawn identify their first language as a language other than English, 33.4% are English Learners (EL), 18.8% are students with disabilities, 77.5% are considered high needs, and 59.5% are economically disadvantaged.

This study used ASSISTments, an online learning platform that provides correctness feedback and supplemental tutoring. A counterbalanced randomized controlled experiment was used to assign each student to condition. Specifically, this study leveraged a within-subjects design in which each student experienced both the control and the treatment conditions, allowing for measurement of the effect of each condition on each student. Students were successfully randomized into two groups by ASSISTments, as evidenced by a near equal distribution of students in each condition. In the treatment condition, students received math problems in "Tutor Mode," receiving both immediate feedback on the correctness of their answers and optional hints on demand. In Tutor Mode, students were also given: 1) an unlimited number of attempts, and after each attempt students were given an indicator of correctness (yes/no feedback) 2) a button that depicted the existence of multiple hints. If a student was confused, the last hint would provide the answer before to enter before progressing to the next problem. In the control condition, or "Test Mode," students practiced solving the same math problems as those in the treatment, but received feedback only after completing a posttest. In Test Mode, students also only had one attempt to answer each question. Conditions were counterbalanced after five questions, thereby allowing all students to experience each condition. Randomization and counterbalancing were also meant to address any potential order effects of domain content, with students answering questions on either trigonometry or factoring in each condition. Each student had a class period lasting 42 min to complete the experiment but teachers varied in specific pretest and posttest assignment protocol.

On average, students scored lower on the pretest (M = .50, SD = .35) than on the posttest (M = .63, SD = .31). Students assigned to Test Mode performed approximately the same as students assigned to Tutor Mode at pretest (M = .51, SD = .40 and M = .50, SD = .38, respectively), but slightly underperformed at posttest (M = .61, SD = .36 and M = .65, SD = .34, respectively).

An independent samples t-test revealed that there was no reliable difference in pretest scores among students, p = .46. Subsequently, a repeated measures ANOVA was conducted to assess the difference in posttest scores between the two conditions. The within-subjects factor was test (pretest and posttest) and the between-subjects factor was condition (Tutor Mode (feedback) or Test Mode (no feedback)). Results showed a significant effect of feedback on posttest scores F(1,241) = 78.32, p < .001.

A median split on students' pretest scores was used to differentiate students with high prior knowledge and low prior knowledge. Students above the median (50%) were considered high prior knowledge, while those at or the below the median were considered low prior knowledge. Students with low prior knowledge exhibited lower scores on the posttest (M = .37, SD = .27) than their peers with high prior knowledge (M = .85, SD = .13). A paired-samples t-test revealed a marginally significant effect in learning gains when low prior knowledge students were assigned to Tutor Mode (p = .10).

Looking at overall performance, students performed significantly better at posttest than at pretest. While gains in overall performance were expected, results also suggested that Tutor Mode was more effective than Test Mode, answering Research Question 1. Learning gains were observed in both low and high prior knowledge students assigned to Tutor Mode. However, overall, low prior knowledge students exhibited greater gains on average.

This study had limitations that may have affected experimental findings. Pretests and posttests were administered by several different teachers. The amount of time that students were given to take each test was not controlled across teachers. In addition, the timing of test administration was not controlled across teachers; students may have taken the pretest in the beginning, middle, or near the end of class. Further, the experimental design could have been improved with stronger feedback distribution protocols. Each teacher distributed a handout with all correctness feedback to their students after the posttest, instead of providing feedback before the posttest, thereby making the assignment a stronger learning opportunity.

Overall, the implications of the findings presented herein support further investigation into the use of computer-based learning tasks that provide immediate feedback to students in classroom environments.

A meta-analysis of the effects of feedback in computer-based instruction

Effects of feedback in a computerbased learning environment on students' learning outcomes: a meta-analysis

Focus on formative feedback

Feedback during web-based homework: the role of hints

On-line tutoring for math achievement testing: a controlled evaluation

Acknowledgements. The authors were funded by the NSF (1931523, 1940236, 1917713, 1903304, 1822830