What do you do when you have a plate full of food? Do you eat the tastiest things first and save what you don’t like for last or vice versa? Or do you eat everything together: alternating between a tasty bite and what you ‘hate’? And what does all this have to do with education?
Standardized tests are commonly used to assess what people know and how skilled they are. These tests have different design elements that can impact how well people perform on them. We need to think about these design choices in relation to what we want to achieve with the tests.
For example, in multiple-choice tests, should we give different scores for wrong answers compared to questions that students didn’t even attempt? And when comparing essay-type questions to multiple-choice questions, should we consider how much time students have to answer them, as time limits can affect performance?
One aspect that hasn’t been studied much in education and economics is whether the order of questions in a test affects how well students do. A perfectly logical test-taker should do equally well no matter the order of questions, but research shows that humans are influenced by the order. For instance, as a test goes on, people tend to get tired, and their performance can decline. So, it might be a good idea to put harder questions at the beginning of a test because they require more thinking. However, students might feel discouraged if they see a tough question early, thinking the whole test will be hard.
Lastly, in some tests, you can’t move on to the next question until you answer the previous one. This means the order of difficult questions can really impact a person’s experience.
In short, if we want to accurately measure what people know without the influence of psychological factors, we need to consider how we order questions by difficulty. Should we start with the easiest, hardest, or mix them up? And does the order of difficulty even make a difference?
‘Understanding Performance in Test-Taking: Does Question Difficulty Order Matter?‘ investigates whether the order of difficult questions in a test can influence how well people perform and potentially lead to biased results in assessing knowledge and skills. We conducted two studies to explore this idea, which complement each other and provide reliable results.
The first study was a real-world experiment conducted in partnership with an online teaching platform. They advertised a math challenge with ten questions to about 19,000 volunteers. They divided the participants into four groups, all given the same ten questions but with different arrangements. In two main groups, they placed the three most challenging questions either at the beginning or the end, while the middle four questions were the same for all groups. Additionally, they had two more groups where we mixed difficult and easy questions throughout the test to ensure our findings were consistent.
They collected data on the number of questions completed by participants, how many they answered correctly, and also gathered information about their demographics and how they perceived their own performance.
In a second study, they looked at data from three years of the PISA exam (2009, 2012, and 2015), which included more than 600,000 students from all over the world. They took advantage of the fact that the order of questions in the test booklets varies, and these booklets are randomly assigned to students. This allowed them to estimate how the order of questions and their difficulty level affected the number of questions left unanswered and overall performance in the exam.
In the field experiment, they discovered that when questions are arranged from easiest to most difficult, it reduces the likelihood of students giving up on the test and leads to more correct answers. For example, when they presented difficult questions first, 44% of participants didn’t complete the test, whereas only 30% of those who started with easy questions dropped out. On average, students answered more correct questions when they began with easy questions compared to when they started with difficult ones (3.53 vs. 2.42 out of 10 questions).
Their findings from the PISA exam support this. Students who faced a cluster of questions that was about 10 percentage points more difficult at the beginning left more questions unanswered in the middle and towards the end of the test. They also got fewer correct answers in those sections.
In conclusion, the order of question difficulty in tests has important implications for test design and performance. It can also affect how we rank students, evaluate educational institutions, and compare educational systems. The reason for this could be that starting with an easy question helps students overcome initial hesitation and boosts their confidence, which encourages them to keep trying.
Why? Probably because a simple first question helps students overcome their initial doubts, and the feeling of success they experience motivates them to continue for longer. So first eat what you like best and then eat the bullet!
Anaya, L., Iriberri, N., Rey-Biel, P., & Zamarro, G. (2022). Understanding performance in test taking: The role of question difficulty order. Economics of Education Review, 90, 102293. https://doi.org/10.1016/j.econedurev.2022.102293
3-Star learning experiences 