Paul A. Kirschner & Mirjam Neelen
Kripa Sundar and Pooja Agarwal have published a guide to understanding meta-analyses and meta-meta-analyses.
Wikipedia defines a meta-analysis as:
a statistical analysis that combines the results of multiple scientific studies. Meta-analyses can be performed when there are multiple scientific studies addressing the same question, with each individual study reporting measurements that are expected to have some degree of error. The aim then is to use approaches from statistics to derive a pooled estimate closest to the unknown common truth based on how this error is perceived. Meta-analytic results are considered the most trustworthy source of evidence by the evidence-based medicine literature.
Not only can meta-analyses provide an estimate of the unknown effect size, it also has the capacity to contrast results from different studies and identify patterns among study results, sources of disagreement among those results, or other interesting relationships that may come to light with multiple studies.
And a meta-meta-analysis is a meta-analysis of meta-analyses! A prime example of this is John Hattie’s Visible Learning in which he ranked 138 influences that are related to learning outcomes from very positive effects to very negative effects. He followed this up with Visible Learning for Teachers (2011) which discussed 150 effects and finally to The Applicability of Visible Learning to Higher Education (2015) with 195 effects. For more information on Hattie’s works see this.
It’s also the case that more and more meta-analytical articles are being written about many different educational interventions. But meta-analysis is tricky and its interpretation is sometimes even trickier. Why? Because:
- Not all meta-analyses are trustworthy. When it comes to learning outcomes measured in a meta-analysis, researchers may be comparing apples to oranges.
- The learning strategies being researched in a meta-analysis are not consistent both with respect to their definition and the research methods used to study them.
- Often, the meta-analysis does not include recent studies. A meta-analysis published today was probably submitted at least a year ago (that’s how long it takes to get an article published). The analysis of the studies found and the writing of the article in umpteen versions probably took about two years. That means the most recent articles studied might be at least four years old. Results can change as more studies are conducted on the topic, and research will continue to be published but are not included in the meta-analysis published today!
The guide consists of three parts, namely:
- an overview of meta-analyses,
- an introduction to meta-meta-analyses, and
- effect size statistics, tables, and more
With this guide, Sundar and Agarwal hope, in their words, “to empower you to question the “evidence” in the evidence-based practices you encounter. Specifically, we want to equip you with the tools to assess whether summative evidence that’s presented based on effect sizes is trustworthy [in] meta-analyses and meta-meta-analyses.
3-Star learning experiences 