Paul A. Kirschner & Mirjam Neelen
Some of the most glamorous, popular claims in the field are nothing but tabloid fodder. The weakest work with the boldest claims often attracts the most publicity, helped by promotion from newspapers, television, websites, and best-selling books. And members of the educated public typically only get one side of the story. (Andrew Gelman & Kaiser Fung writing about The Power of the “Power Pose”).
Of course, if you’ve studied something as a scientist/researcher, you want others to know about your work and your results. A press release is often the way to do this. Sometimes the press release is written by you and sometimes by your university or funding agency. No matter who creates the release, it goes without saying that its goal is to catch the reader’s attention and one way to do that is to give it a sexy title. Nothing wrong with that. However, the title needs to go beyond sexiness; it also needs to be true. In Dutch we say: De vlag moet de lading dekken; the ship should fly under its true colours (many Dutch sayings find their origin in the maritime). Ionica Smeets, science journalist and professor of science communication at Leiden University in the Netherlands explains in her (Dutch) inaugural address that science journalism is critical for the scientist, university, funding agent, and society. However, she also warns us that it can go painfully wrong. She even states that journalists “don’t get it”. And when it goes wrong the consequences can be quite nasty. Notorious are the articles we read in the newspaper almost daily on diets and dietary advice. These articles are usually written based only on a press release which journalists unfortunately don’t really seem to understand. What makes it even worse, is that these journalists don’t even seem interested in what the actual facts are. They rarely if ever go to the original source (i.e., the scientific article) and if they do, they don’t have either the knowledge or the will to understand the article and evaluate the methods used and the results. These things combined cause journalists to – basically – manipulate the actual results. Our American friends tell us that the saying here is: If it bleeds, it leads.
An interesting example was published by the NRO (Netherlands Initiative for Education Research). They’ve funded an excellent piece of research from Judith ter Vrugte[1] from University of Twente which resulted in her PhD dissertation “Serious Support for Serious Gaming: Enhancing Knowledge Acquisition in a Game for Prevocational Mathematics Education”. The title of the press release was “Educational Maths Game is Effective for Vocational High School Students[2]”. The title suggests that there was a study (or perhaps a sequence of studies) where instruction including math games (i.e., the intervention / experimental condition) was compared with instruction without this type of games (i.e., the control condition) and that instruction with/through math games led to significantly better results than instruction without those games. Right?
However, when we check Ter Vrugte’s PhD defence, we can see that that’s not what happened. The University of Twente writes (some noteworthy phrases are placed by us in bold):
…studies in the current dissertation adopted a value-added approach: a standard version of the game (i.e., an educational math game ‘Zeldenrust’) was compared to versions that were enhanced with different forms of instructional support (i.e. self-explanation prompts, collaboration, and faded worked examples). Three consecutive empirical studies investigated the effect of these forms of instructional support on the in-game performance and knowledge acquisition of prevocational students.
It’s important to highlight the outcome that learning was significantly enhanced in all three studies…These findings show that the intervention involving the educational game ‘Zeldenrust’ formed a solid foundation for learning about the domain of proportional reasoning. The results of the three studies show that further improvement of this environment was difficult, but possible. The investigated implementations of self-explanation prompts and collaboration did not result in improved effectiveness, but the addition of faded worked examples did. This seems to suggest that prevocational students benefit from continuous support that contains expert advice. Though game designers show a tendency to hide the instructional component and educational content in educational games, results of the current line of study suggest that a formal representation of the educational content in the game-environment could support students gameplay and learning.
So, while Judith conducted excellent research, she has reported something completely different than the sexy NRO heading suggests. Judith’s research showed that one specific math game with a certain form of instructional support resulted in better learning than the same game without the instructional support. This is even stated in the summary of her PhD, where it says: “This dissertation addressed how to optimise game-based learning with approaches that, in theory,…are likely to stimulate knowledge acquisition”. In other words, she studied something completely different from what NRO wrote in their press release.
She also writes that, for a certain part of her research where both three versions of the interventions (the game with instructional support) as well as the “plain game” (the control version) were involved, all students improved and didn’t differ significantly from each other. She says that her research suggests that playing the game has contributed to a knowledge increase, but that most of the added instructional support methods didn’t have any added value for learning. All these details were neatly reported by her, as it should by the good scientist that she is.
Noteworthy is that (1) in her third study the addition of worked examples to the game did have a significant effect on learning compared to the game without the worked examples and (2) the game without the additional instructional support always was the control context and, thus, there was no comparison with “traditional” education. Therefore, we don’t have a clue as to what the knowledge increase would have been, if any, if the students would have just received “regular maths instruction”.
Why are we sharing all this? Because the heading as used by the NRO would have been closer to the truth if it had stated: “Math games supported by worked examples lead to better learning than games without them”. Of course not quite as sexy as the one NRO opted for, but correct.
In all fairness the NRO message did state (although they keep focussing on the positive effects of computer games which is NOT what Ter Vrugte’s study is about):
Instructional support in the form of worked examples can optimise the effectiveness of such a game… Educational games usually use active and discovery learning by students as a starting point. The educational content is generally embedded in the game and the storyline. Judith ter Vrugte: “The knowledge that students acquire by playing the game is usually implicit and context-dependent. Although this type of knowledge is valuable, education usually strives for explicit knowledge. The addition of instructional support can potentially support this transfer from implicit to explicit and in that way the effectiveness of educational games can be optimised. However, we need to keep in mind that we still know very little about what this should look like exactly; the same goes for implementation of instructional support in educational games.
By saying this, Judith steals the thunder of the evangelists of “innovative (self-) discovery learning”, but we fear that this subtle message won’t diminish the impact of the sexy heading. It’s very likely that people in general, but also journalists, politicians, and educational policy makers will overlook this nuance. In this way, the title as composed and spread by the NRO can cause readers who only read the titles of newspaper articles, or who are perhaps a bit lazy, or who haven’t really read Judith’s work will get the wrong message. Teachers, politicians, education policy makers, parents, and so forth might think that using math games in general leads to better math learning and to better math learning achievements. In the end, it’s the student who will suffer and pay the consequences, especially the more/most vulnerable ones. And, as you will probably understand, the consequences can be terrible!
The main takeaway of all this? If the title sounds sexy, let the reader beware!
References
Gelman, A., & Fung, K., (2016, January 16). The power of the “Power Pose”: Amy Cuddy’s famous finding is the latest example of scientific overreach. Slate. Retrieved from http://www.slate.com/articles/health_and_science/science/2016/01/amy_cuddy_s_power_pose_research_is_the_latest_example_of_scientific_overreach.html
NRO (2016). Educatief rekenspel is effectief voor vmbo’ers [Educational maths game is effective for vocational high school students] [Press release]. Retrieved from https://www.nro.nl/educatief-rekenspel-is-effectief-voor-vmboers/
Smeets, I., (2016). Enige beschouwingen over de waarde der wetenschapscommunicatie [Some thoughts on the value of scientific communication] [Inaugural address]. Retrieved from https://openaccess.leidenuniv.nl/bitstream/handle/1887/40075/Oratie%20Smeets.pdf?sequence=1https://openaccess.leidenuniv.nl/bitstream/handle/1887/40075/Oratie%20Smeets.pdf?sequence=1
Ter Vrugte, J., (2016). Serious Support for Serious Gaming: Enhancing Knowledge Acquisition in a Game for Prevocational Mathematics Education (Unpublished doctoral dissertation). University of Twente, Enschede, The Netherlands.Retrieved from http://doc.utwente.nl/100489/1/thesis_J_ter_Vrugte.pdf
[1] The Dutch version of this blog was approved by Judith.
[2] Somewhat freely translated from “Educatief rekenspel is effectief for vmbo’ers”
3-Star learning experiences 