Summer treat or fruitless chore?

The Question

Alta delivers just-in-time remediation to prerequisite skills when students demonstrate a knowledge gap while working on an adaptive assignment—we call these “refreshers.” Refreshers start by giving the student a couple of practice questions on the most immediately relevant, nearby prerequisite to the target learning objective they’re struggling on. If the student has no difficulty with those refresher questions, they’ll immediately head back on target to their assigned objectives. But if the student has difficulty with the initial refresher objective, Alta will deliver instructional material aligned to that prerequisite and further practice—and if that doesn’t close the knowledge gap, Alta can deliver further remediation to more distant prerequisites. If we think of each step “away” from the assigned, target learning objective as a “hop” on the Knowledge Graph, refreshers can include content up to three learning objective “hops” away. If you were to pick a student’s assignment at random from across your class (or all of Alta), it probably doesn’t have any refresher material in it at all; most students don’t need or see refreshers on most of their assignments. However, the majority of students need a refresher on some learning objective at some point during their course. How often are students encountering this refresher material? How many “hops” is it from the instructor’s assigned objectives? What happens to the assignment as a whole when refreshers are present? In short: what’s the impact of an Alta refresher?

The Caveat

We have a huge amount of robust data on student interactions flowing through the adaptive engine powering Alta, and we can and do perform analysis on trends, outcomes, and pain points in the student adaptive experience on a rolling basis. But getting directly at the question of what impact a refresher made for a particular student is a very hard question—to really know for sure, we’d need to do something like find an identical “control student” to take the assignment with no refresher, and to compare their results with the “test student” who engaged with refresher material! The analysis below is not meant to prove that any particular refresher definitely helped any particular student—instead, we’ll show some high-level, aggregate trends in what tends to happen when students engage with refreshers.

The Dataset

This analysis draws on data sampled from the four largest families of titles in Alta for Fall 2021—those happen to be the Alta titles in College Algebra and Precalculus, Chemistry, Foundations of Algebra, and Quantitative and Statistical Reasoning. Student usage of these titles amounted to more than 40 million interactions with questions or instructional materials, and for this dataset, we sampled at random about a quarter of the student-assignment pairs from each subject area.

The Results

What percentage of student-assignment pairs had any refreshers?

In general, about 10-15% of student assignments featured any refreshers at all. A student who enters their class with less subject-matter familiarity might see refreshers on more assignments than this, while a highly prepared student might see them on fewer—but around 80% of students encounter a refresher at least once across their entire engagement with Alta.

When refreshers occurred, how much of the assignment were they?

A typical refresher experience is short, both in terms of the number of questions students answer and as a proportion of the overall assignment in which it occurs.

When refreshers occurred, what was the subsequent assignment completion rate?

While this data does not prove that the refresher is the reason why students were able to reach 100% progress on their assignment, we can see that the vast majority of students were able to close their knowledge gap(s) and perform well enough to reach mastery on the assigned learning objectives after engaging with a refresher. These completion rates are quite close to those of students who start their assignment with a high level of preparedness (and thus correctness) -- that’s an important outcome we look for in Alta data.

When refreshers occurred, how many “hops” away from target were they?

Most frequently, when a student encounters a refresher, they’ll only work on a single-hop prerequisite to the assigned learning objective. Among the already small percentage of assignments featuring refreshers, only about 20% of assignments result in refreshers more than one hop away. This reflects Alta’s attempt to provide just-in-time interventions only as needed: multi-hop remediation is Alta’s most time-intensive intervention, and Alta only provides it to students in cases of persistent knowledge gaps where we think we might be of help.

When remediated many “hops” away, how did students end up doing on the remainder of the assignment?

Completion rates (and final progress scores) for students remain fairly high, even in the presence of multi-hop refreshers, but rates do drop as the population of students is restricted to only those engaging with such refreshers. Our hypothesis is not that 2- or 3-hop refreshers are causing lower completion rates—but that students who demonstrate a need for significant remediation have a higher hill to climb than their more prepared peers, with or without Alta’s help. We think that these “struggling students” are a key group underserved by traditional homework assignments and online courseware, and many new features developed for Alta are designed precisely to help them.

What’s next?

Alta’s refreshers occur infrequently, and usually on the way to student success. But these high-level statistics are small comfort to an individual student who feels “stuck” working on a refresher that they either can’t master or think is irrelevant. That’s why Alta has current and newly emerging functionality designed to enable more student agency in digital learning. First, when students get stuck and exhaust the available refresher learning objectives, Alta will surface a pop-up screen suggesting students reach out to their instructor for some “real human” guidance. If the student’s not ready to do that, Alta can also suggest the student take a quick break to clear their mind and play a little game we call Knerdie. Second, we’re nearing the end of our pilots for new “end refresher” agency in Alta. For institutions enrolled in the pilot, this functionality lets students decide after a couple of refresher questions that they think it’s a better use of their time to go back to assigned, target work and try again, instead of continuing to work on refresher content until Alta thinks they’re ready. Our initial pilot results have indicated that assignment completion rates have not dropped as a result of this new freedom of choice, and we’re hard at work expanding other modes of student agency elsewhere in Alta this summer.