The Most Important Question About Multiple Choice: when?

This is the second in a series of blog posts looking at the use of multiple choice questions (MCQs). The first blog looked at some simple tips for writing MCQs, and can be found here.

Consider the MCQ below:

This is not a good MCQ. We learnt why in the first post: there are no plausible distractors. It’s obvious that three of these answers are wrong, so by a process of elimination we can establish the correct one without having to actually retrieve the correct answer from our memories:

What is the word equation for photosynthesis

1. Boris Johnson ❌
2. Carbon dioxide + water → glucose + oxygen ✅
3. Cheese ❌
4. The Magna Carta ❌

Let’s tweak it to look like this:

What is the word equation for photosynthesis?

1. Hydrogen + oxygen → water
2. Hydrogen + nitrogen → ammonia
3. Carbon dioxide + water → glucose + oxygen
4. Magnesium + sulfuric acid → magnesium sulfate + hydrogen

This question is a bit better. The other choices (A, B and D) are a lot more plausible, as they at least relate to a chemical reaction, and they are reactions that students might have come across in school.

However, let’s say I don’t remember the word equation for photosynthesis, but I do remember that it involves carbon dioxide. I can now get the question right without having to actually retrieve the word equation for photosynthesis, because there is only one choice here that has carbon dioxide in it:

1. Hydrogen + oxygen → water ❌
2. Hydrogen + nitrogen → ammonia ❌
3. Carbon dioxide + water → glucose + oxygen ✅
4. Magnesium + sulfuric acid → magnesium sulfate + hydrogen ❌

We could call this partial retrieval — where a student does retrieve something from their memory, but it isn’t the whole thing. They have retrieved:

The word equation for photosynthesis has the word “carbon dioxide” in it

They have not retrieved:

The word equation for photosynthesis

Let’s try and make it a bit better:

1. Methane + oxygen → carbon dioxide + water ❓
2. Magnesium + sulfuric acid → magnesium sulfate + hydrogen ❌
3. Carbon dioxide + water → glucose + oxygen ❓
4. Hydrogen + nitrogen → ammonia ❌

Again, partial retrieval. Better than no retrieval, but we can do better.

We therefore need to change it once more:

1. Carbon dioxide + water → glucose + oxygen
2. Water + glucose → carbon dioxide + oxygen
3. Glucose + oxygen → carbon dioxide + water
4. Oxygen + carbon dioxide → glucose + water

This is now much better. The substances in each choice are plausible, there are two options with carbon dioxide on the reactant side, and I can feel reasonably confident that a student would not get it right first time by a simple process of elimination. I’ve also included a classic misconception. The third option is the word equation for respiration, the direct opposite of photosynthesis. Students commonly confuse these two processes, so my analytics will give me some information about whether that misconception is playing out:

There are, however, still two problems with this question. Let’s say a student gets this question wrong. When marking the whole quiz, they see what the correct answer to this question was, and then try the quiz again. Once more, they are presented with the question they got wrong before:

What is the word equation for photosynthesis?

1. Water + glucose → carbon dioxide + oxygen
2. Oxygen + carbon dioxide → glucose + water
3. Carbon dioxide + water → glucose + oxygen
4. Glucose + oxygen → carbon dioxide + water

They look at the question, and can’t remember the word equation for photosynthesis. Carousel shuffles the answers, so they can’t just click option A as that’s what it was last time. However, they do remember that the correct answer to this question had “carbon dioxide” as the first word:

1. Water + glucose → carbon dioxide + oxygen ❌
2. Oxygen + carbon dioxide → glucose + water ❌
3. Carbon dioxide + water → glucose + oxygen ✅
4. Glucose + oxygen → carbon dioxide + water ❌

They select option C and get it right. They are now pleased because they got it right, but the only thing they’ve really done retrieval on is “what is the first word in the word equation for photosynthesis?” They haven’t done retrieval on the rest of the equation, resulting in partial retrieval as above.

Either way, even if they did get it right, and they haven’t taken the “first word” shortcut, it seems to me that even in the best of cases they’ve still only done a partial retrieval. This is because they haven’t freely recalled the equation for themselves: they’ve looked at the equation and said “yes, that’s the one that matches my memory.” They haven’t had to retrieve that entire equation from memory for themselves.

This kind of retrieval makes me a bit worried, because let’s say I later ask the same student this:

Here, I’ve taken away the MC, and made it short answer. And the students have never had to answer the question like this. Has their multiple-choice-induced partial retrieval until now done enough to help them answer this question?

All of the above — all of this partial retrieval — leaves us in a bit of a bind. I want my students to know the word equation for photosynthesis off by heart. If I use MCQs, however well the question has been written, however well the distractors crafted and sequenced, students still haven’t retrieved the entire equation for themselves. At best, they have thought hard about it and identified the correct one, but they haven’t really retrieved the entire equation for themselves, from memory. Over the long term I think it’s hard to see how these MCQs can help a student move to the stage of “fluent recall.” If they don’t guarantee full retrieval, and rely on partial retrieval, they cannot be sufficient by themselves.

So let’s imagine that in response to the above, immediately following a lesson on photosynthesis, a teacher sets their students a quiz involving short answer questions on photosynthesis. They say to themselves:

I want students to be able to freely and fluently recall the word equation for photosynthesis, so along with other questions I am going to ask them questions like:

This is a logical move because the teacher is aiming for full retrieval, but it still might not be the right play. It could be that this question is just too hard for students right now. When students have just learnt something new, free and uncued retrieval of that idea can be extremely difficult, to the point of impossibility. Rather than make a full retrieval, the student stares at a blank screen with a blank mind and makes no retrieval! This is another example of what we meant in the first blog in the series where we outlined the concept of a desirable difficulty: too easy and the retrieval may only be partial. Too difficult, and there may not be any retrieval at all.

Perhaps we therefore need to appreciate that when student understanding is new and nascent, partial retrieval is a reasonable target:

At the start, students might find free recall of the word equation too challenging, resulting in a failed retrieval attempt and potential demotivation. This isn’t surprising, and we’ve always had lots of tools in Carousel to mitigate this initial difficulty:

It’s at this point that MCQs might also be useful. Well-crafted MCQs can promote some hard thought about which of the plausible options is correct, but they don’t leave students staring at a blank screen with no idea where to start.

Today’s “difficult” is tomorrow’s “easy,” so you won’t want to keep your students on the MCQ forever. You’ll probably want to start them off on MCQs but then as time goes on to “graduate” to short answer response as their knowledge and understanding strengthens and develops.

We can summarise all this as:

When all is said and done, MCQs are a powerful tool for assessment and retrieval. Short answer responses are too, and we need to make sure we are aware of the strengths and limitations of each and where they fit into our teaching.

In the next blog post, we will look at feedback and review of multiple choice questions.

_________________________________________________________________

Addendum number 1: it’s also true that MCQs can be used at the end of teaching as part of a summative assessment, but generally these would be “unseen” or “application” type questions, and not the type of question that teachers mich be uploading to Carousel as day-to-day retrieval practice.

Addendum number 2: there are no firm empirical studies which investigate the distinction I make above between the timing of short answer questions and MCQs. It is mostly theory based on my experience and the literature that deals with MCQs vs short answer questions in a more general sense. Normally that literature shows that they are both similarly effective for retrieval practice, but my concern about those studies is that they use super-straightforward questions in the sense that they tend to have the question/answer format of definition/term. E.g.:

These questions are taken from this experiment, arguing that MCQ and short answer are comparable for retrieval. Because each answer is just a couple of words, my worry is that for anything longer like a sentence or sequence the retrieval would not be equivalent. Hopefully there will be some more research on this in due course (who knows, we could even do this kind of research with Carousel data..!)

_________________________________________________________________

☞ TO ENJOY A FREE TRIAL OF CAROUSEL GOLD OR TO PURCHASE A CAROUSEL GOLD SUBSCRIPTION (just £60 p.a.), LOG IN TO YOUR ACCOUNT HERE AND GO TO THE ‘MY ACCOUNT’ SECTION.

➢

☞ TO FIND OUT ABOUT PRICES FOR CAROUSEL PLATINUM, CLICK HERE.

➢

☞ TO LEARN MORE ABOUT CAROUSEL’S GOLD AND PLATINUM TIERS, CLICK HERE.