By Paul A. Kirschner & Mirjam Neelen
Consider this title of an item on an Internet news site: “Digital practice improves learning outcomes in secondary education”. Quite an eye-catching and noteworthy message. This item was posted by Emerce, a Dutch e-commerce magazine and weblog. The original article said:
Digital practice improves learning outcomes and motivation of students in secondary education. This is shown by research from slimleren.nl[1]. The study included 600 active users of an online platform that offers practice exercises for secondary education students. With the program, students can practice Math, Dutch, English and Arithmetic interactively. The students’ learning results improved by 1.5 point on average. In addition, 73% said that they now had more confidence that they would do well on a test and 77% said to be enjoying the learning better.
Sounds impressive, although on further inspection the message seems to say: practise improves learning outcomes. That, in itself is not quite earth shattering, but maybe there was good research involved and is our scepticism ungrounded. However, the sentence “This is shown by research from slimleren.nl” made us wonder. What’s the situation? Slimleren conducted a study in which students practised on the ‘Slimleren’ platform. Hmm, such a study sounds a bit seedy and we become even more suspicious because of the fact that the so-called research didn’t appear to include a control group in which the students practised with the same materials for the same amount of time but in a different way, for example, offline.
Skimming through the website of slimleren.nl, it becomes clear that the researchers are not only testing their ‘own stuff’ on the platform, they are also the sellers of the platform! Now our eyebrows are raised to the max.
Coca Cola eat your heart out. Slimleren takes it a step further than Coca-Cola paying others to conduct research to show that people don’t gain any weight by consuming sugar-saturated fizzy drinks. In Slimleren’s case it’s like Coca Cola conducting the research themselves.
But hold the boat here. What do we see? The online article on the Emerce website is also posted by Slimleren.nl. So… their news on the outstanding results on a Slimleren product is based on research conducted by Slimleren and is published on Emerce by Slimleren. Are you still with us?
The saddest thing in this case is that media, in this case Emerce along with other online tech and education related sites unthinkingly spread the word. Commerce fooled by commerce or?
Now let’s discuss the study itself. Or should we put “study”, between quotation marks, as this cannot be considered serious research, at least not a solid, scientific, independent research. After all, just thinking up a question, measuring some results and analysing them does NOT constitute research.
The Slimleren website explains that in March 2016 slimleren.nl conducted surveys amongst 600 active Slimleren platform users; secondary education students. These surveys which were filled in by the students intended to determine the effects of digital practise using the Slimleren online platform on students’ school results and motivation.
First things first. Surveys, or self-reports are not the most effective instruments – and that’s put very mildly – to get valid and reliable data. At least, not in this context. Surveys are OK if you are looking for, for example, students’ perception of or feelings about something. Furthermore, all kinds of relevant information such as gender, age, group, school type, types of assessment that determine final grades are not mentioned. Hence, we don’t know to what extent results are generalizable if at all. But hey, just a critic, grumbler, or maybe a bona fide researcher pays attention to these types of ‘details’.
Students included in the study practised through Slimleren independently (at home) and were active on the Slimleren platform between 1 and 12 months so that, in their words, the relation between practise and grades and motivation could be demonstrated. This sounds like the answer was already decided upon before the study was carried out, or is it us?
Ok, so there’s no control group with students who practise with the same materials in a different way. There’s also no randomisation applied which is without any doubt the golden standard for most types of research. Is it the case that these students hardly practised before they started using Slimleren and now they suddenly DO practise with Slimleren? We don’t know but if that’s the case, you can probably guess what the outcome would be…
Although not mentioned in the article on Emerce, the slimleren.nl website implies that they have divided groups in ‘more and less’ practice. This isn’t the case at all. What they did was to leave the learners to their own devices and ask them afterwards how much they studied. Again self-report, and as we know from lots of research, this is not really reliable[2]. And, by the way, as the practice was in a computer programme and online, how hard could it have been to just log the study time (and along with that the number of practice minutes/hours per session, and so forth)? The website states that 0-1 hour of practice per week is good for a 0.9 point grade increase. 1-3 hours of practice promises 1 full point extra and 3-5 hours of practice 1.9 point.
In other words, if you practise independently for five hours a week you achieve better results than if you practise on your own for, let’s say, one hour per week. Congrats to Slimleren who has now proved that more practise produces better results than little practise. Wow, very impressive indeed. Excuse the snarky sarcasm.
But the ‘researchers’ go further. Slimleren also explains that they suspect that direct feedback during practise, insight in what the student has and hasn’t yet understood, game elements, and high quality learning and practice materials are the drivers behind the improved learning results. So, they take it a shameless step further by not just drawing the conclusion that ‘practise makes perfect’ but they go on by speculating on ‘cause and effect’ without having even considered, let alone studied, these factors in the research itself!
Now, just for fun, let’s look at the (sequence of) questions that the students had to answer in the survey.
- What was your grade in Maths/Dutch before you started practising with Slimleren?
- What was your grade in Maths/Dutch after you had practised with Slimleren?
- Do you feel more confident that you will achieve a good grade now that you have practised with Slimleren?
- Are you enjoying learning better now that you have practised with Slimleren?
First of all, the questions are quite suggestive. Second, as we can see by looking at the sequence of the questions the students had already been able to compare their ‘before and after Slimleren grades’ before they answered the questions on so-called motivation. How can it be a surprise that students feel more confident that they will receive a higher grade after they have just received a higher grade? Wow!! Nobel prize committee take note.
We are calling this blog a day before we drown in our own sarcasm. Don’t let yourself be fooled by this type of commercially driven so-called research! Monique Marreveld and Bea Ros recently published an article in Didactief, titled “Science and Commerce? Beware!” Now that hits the nail on the head.
[1] Translated from Dutch: smart learning
[2] Self-report studies suffer from specific disadvantages. [They] may be exaggerated; respondents may be too embarrassed to reveal private details; various biases may affect the results, like social desirability bias. Subjects may also forget pertinent details. Self-report studies are inherently biased by the person’s feelings at the time they filled out the questionnaire. If a person feels bad at the time they fill out the questionnaire, for example, their answers will be more negative. If the person feels good at the time, then the answers will be more positive. (https://en.wikipedia.org/wiki/Self-report_study#Reliability)
3-Star learning experiences 