An important lesson I’ve learnt from working at Sparx Learning is that subtle changes can have surprisingly large and sometimes unexpected effects on student behaviour and learning. That’s why it’s crucial to carefully roll out new features; we need to understand the impact and can only do this by monitoring closely and talking to students and teachers.

We are fortunate to have teachers who are passionate about collaborating with Sparx Learning and help us come up with great solutions before rolling out to the wider community.

In this blog, I’ll share some of our key learnings from the development of a new feature that we released at the start of term: Second Chance.

What’s the problem?

In order to complete tasks in homework or lessons, students need to answer each task item correctly. Prior to the release of this new feature, after each incorrect attempt the student was given a new question covering the same learning point at the same difficulty but with different numbers or context.

Students and teachers in the Sparx community gave us feedback that it would be beneficial to give students a chance to correct their mistakes, particularly with longer questions. We also wanted to encourage students to look back at their workings and try to understand where they’ve made a mistake; a key mathematical skill!

Always give the same question

Our first step was to always give students the same question after an incorrect attempt. This has the advantage that it’s very simple and understandable for students. However, we hypothesized that it might lead to more guessing.

Trialling with two schools we had lots of positive feedback from students. But we saw a small but significant proportion of students being able to answer questions without understanding.

Here is an example of a student’s activity on one question in their homework.

The student does seem to have a genuine first attempt but then gives up and works up through the integers and is able to answer the question correctly without genuinely understanding.

Although this type of behaviour was rare, it was increasing over time and also reported by teachers in the trial schools.

Second chance

The next step was to try another approach: give one opportunity to retry the question before switching to a new question.

This gives students the chance to correct typos and learn from their mistakes but limits the opportunity for guessing.

We knew there was additional complexity here for students to understand when they are given a new question. To address this we’ve built in a number of prompts to help support students.

Releasing this to the original trial schools we saw that the guessing behaviour reduced and we were still getting great feedback. At this point we made the decision to rollout to all schools to coincide with the start of term.

Another twist

With changing from the original delivery to Second Chance, we expected to see a higher rate of success on the second attempt as students are able to correct workings instead of starting from scratch.

But although we saw that students are taking less time on that second attempt, we saw a very similar success rate.

Investigating further we found that with Second Chance, students were frequently entering the same incorrect answer twice. Here is an example from another student’s homework:

The student has a Second Chance at the same question. However they enter exactly the same incorrect answer as on the first attempt. On the third attempt, the student is given a different question, at which point they answer correctly.

So what’s going on here?

We can’t know for sure for this individual case but it looks like after the first incorrect attempt, the student incorrectly assumed they had made a typo when entering the answer and tried typing the same answer in again. The fact that this is a trend across thousands of students adds weight to this theory.

To address this, we have planned an additional change to show the entered answer on the incorrect screen so that students can differentiate between when they have mis-typed the answer into Sparx and when they have made a mistake in their workings that needs to be corrected.

Small changes can have a large impact

We’ve made this change in response to feedback but also because we believe it will increase learning by encouraging students to write good bookwork and interrogate their mistakes.

Small changes can have a surprisingly large impact and that’s why it is crucial to test and iterate before releasing new features, particularly when a change relates to student learning. We couldn’t do this without the support of our teaching community who give brilliant ideas and feedback to help us continually improve.

Have feedback about Second Chance? Or other ideas/suggestions for how we could improve? We would love to hear from you. Get in touch and join the conversation at @SparxLearning