What It Takes for Employees to Own and Drive Their Own Learning (A Lot!) – Part 2

Mirjam Neelen & Paul A. Kirschner

Last week’s blog discussed what SDL and SRL are and how it can be supported it in the workplace. This week, we’ll explore how to measure the quality of this SDL and SRL and how we can help employees become better self-directed and self-regulated learners.

How to measure SDL and SRL

The table below gives an overview of subjective and objective ways to measure SDL and SRL (see Saks & Leijen, 2014 and Endedijk, Brekelmans, Sleegers, & Vermunt, 2016).

Measures of SDL – after the fact	Measures of SRL – after the fact
Questionnaires (self-report)	Stimulated recall interviews
Interviews	Portfolios and diaries/logs
	Task-based questionnaires or interviews
	Hypothetical task interviews
Measures of SDL – real-time	Measures of SRL – real-time
Observation	Eye tracking
	Think aloud protocols
	Observation and video -registration of behaviour
	Performance assessment through concrete tasks, situational manipulations, or error detection tasks
	Trace analysis

We’ll discuss questionnaires and self-reporting tools first (subjective measurement of SDL and SRL after the fact), as they’re the dominant measurement instruments for both SDL and SRL in the workplace (Saks & Leijen, 2014 and Fontana, Milligan, Littlejohn, & Margaryan, 2015). Note that for SRL specifically, structured interviews and teacher ratings are also widely used.

Questionnaires and self-reporting tools

While these methods have their advantages, primarily that they make it easy to reach many respondents (facilitating statistical analysis) and respondents can answer questions around more implicit things like moods, plans, and beliefs. The problem with them is that they’re in no way objective (what one person sees as important can be seen as completely irrelevant by someone else) and have also been found to be extremely unreliable (i.e., they don’t measure what they purport to measure). Another disadvantage is that they only provide insights after the ‘learning fact’ and not in real time. Therefore, the context of the learner responses is missing. For example, they don’t give any insight in which situations learners have in mind when answering the questions. Also, when thinking about their behaviour, they might already start to interpret or rationalize it so that it doesn’t necessarily give a ‘real image’ of the learners’ SRL behaviours. Finally, people have ‘selective’ memories and thus they don’t remember or report their steps completely or objectively.

Especially for measuring SRL, the focus has shifted to find more suitable and objective ways of measuring it either offline or online. To paint the (limited) picture, we’ll discuss think aloud protocols, unstructured interviews, and tracing.

Think aloud protocols and unstructured interviews

Think aloud protocols and unstructured interviews, although still subjective, provide the opportunity to capture the learning processes more precisely and follow their contextualised and dynamic nature (Saks & Leijen, 2014). Winne (2010) also explains that, first, these methods remain closer to cognitive events and mental states that learners describe (compared to interviews or questionnaires after the fact). Second, it’s the learner’s decision which topic to think aloud about rather than a researcher’s question about a particular topic. Prompting learners to “Remember to think aloud” is, according to Winne, not a good idea because data that’s initiated by the learners can’t be treated as equivalent to data that came out as a consequence of the researcher prompting.

Tracing

Another way of capturing SRL ‘instances’ is through tracing (objective and during the fact). Winne (2010) defines a trace as “a datum generated by a learner that is approximately simultaneous with the cognitive operations the learner applies to information in working memory” (p. 272). An example of a very simple trace is a learner’s notes made in a text. There are online applications (for example Winne and Hadwin’s nStudy (2013)) for personal learning. To put it very simply, such an application generates time stamps for each software event such as completing the selection of text in the browser, choosing an option from a menu, opening a window, and so forth. Such traces (as is also the case for eye tracking and videoed behaviour) can be the basis of stimulated recall. Here, the trace (eye movements, observation video) serves as a stimulus for the learner to recall and reflect on / explain what (s)he has done and why.

The advantage of this method is that trace data operationalise what learners do as they do it. You don’t have to ask asking learners what they believe they do. When traces are strong operational definitions of theoretical cognitive and metacognitive operations, they provide a robust opportunity to test theories about when, whether, and how SRL processes affect learning.

However, there are practical downsides to tracing. Winne (2010) points out that “gathering traces requires intervening in learning experiences to generate the data. If the interventions are too unnatural, their capacity to support valid inferences is undermined (p. 275)”.

Pros, cons… so now what?

How to move forward in today’s workplace?

There are a number of different aspects that are important to take into account when selecting an appropriate method for assessing SDL and SRL, such as the goal of the assessment, type of data to be collected, way of data processing, financial aspects of the data collection, content of the assessment (which skills are assessed), participants and context, assessment procedure, and psychometric quality of the instrument (for example, Endedijk and colleagues, 2016). From this also flows the decision on what type of instrument is the best fit.

It seems that, for a workplace context, we can choose between practical, yet subjective and after the fact, or unpractical, more objective, and during an event. Endedijk and colleagues (2016) point out that, for contexts of workplace learning one needs to consider that learning is often unplanned and therefore, using online instruments for measuring SRL seems to be less relevant and useful.

Let’s park the ‘best way to measure’ for now and explore how we can help employees to become better self-directed and self-regulated learners.

How to learn to be a better self-directed and self-regulated learner

First off, the good news is that with adequate training, everyone can improve their degree of control over learning and performance (e.g., Brand-Gruwel et al., 2014). What’s important here, is that although both SDL and SRL function at different levels, both are needed to optimise an individual’s learning process. According to Bjork, Dunlosky, and Kornell (2013) and Zimmerman (2002), to become strong self-directed and self-regulated learners we need to:

1. have a basic understanding of how human memory works. For example, that we don’t just store literal recordings of information in our long-term memory but instead relate new information to what we already know.
2. encourage learners to establish specific goals for themselves.
3. know which learning strategies enhance knowledge storage and retrieval of information (e.g., spaced and variable practice) and then teach these strategies and include them as support tools in any learning experience.
4. teach learners how to monitor their own learning and control their learning activities effectively (e.g., through teaching them how to continuously assess themselves and how to keep the balance between monitoring and control). The image below (from Bjork, Dunlosky, & Kornell, 2013) gives an idea of all the activity that is involved in SRL. Not something everyone and their mother would be able to do easily!
5. assess learners’ beliefs, and especially their self-efficacy perceptions. Also, people need to overcome certain intuitions and avoid being fooled by current performance and feelings about ‘what works’ as well as societal attitudes and assumptions that are often counterproductive for learning, such as the misunderstanding that errors or mistakes must be avoided. We’re talking about a learning process and making mistakes is a part of learning!
6. integrate support and guidance in acquiring SDL and SRL skills into a flexible learning environment (Brand-Gruwel et al.; Van Merriënboer & Kirschner, 2017 because these skills do not develop spontaneously! Instructional strategies to foster SRL skills things like process worksheets, modelling and prompting. The figure below (see Kicken and colleagues, 2009) shows an example for a hairdresser[1] (overview of skills in the left column, standards for performance in the Likert scale and possibility to formulate learning needs in the textbox).
7. Improve the learner’s judgments of their own learning. This is a difficult one because often learners are suffering from the ‘Dunning-Kruger effect’ (see our blog here).

Long story short, training and ongoing structured support to remind people of these points and guide them in the right direction is critical! It’s not realistic nor fair to expect that employees are able to do all these things, let alone that they can do this effectively and efficiently. It takes a LOT to own and drive one’s own learning!

References

Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-regulated learning: Beliefs, techniques, and illusions. Annual review of psychology, 64, 417-444. Retrieved from https://bjorklab.psych.ucla.edu/wp-content/uploads/sites/13/2016/07/RBjork_Dunlosky_Kornell_2013.pdf

Brand-Gruwel, S., Kester, L., Kicken, W., & Kirschner, P. A. (2014). Learning ability development in flexible learning environments. In Handbook of research on educational communications and technology (pp. 363-372). New York, NY: Springer.

Endedijk, M. D., Brekelmans, M., Sleegers, P., & Vermunt, J. D. (2016). Measuring students’ self-regulated learning in professional education: bridging the gap between event and aptitude measurements. Quality & quantity, 50, 2141-2164. Retrieved from https://link.springer.com/article/10.1007/s11135-015-0255-4

Fontana, R. P., Milligan, C., Littlejohn, A., & Margaryan, A. (2015). Measuring self‐regulated learning in the workplace. International Journal of Training and Development, 19(1), 32-52.

Kicken, W., Brand-Gruwel, S., van Merriënboer, J. J. G., & Slot, W. (2009a). Design and evaluation of a development portfolio: How to improve students’ self-directed learning skills. Instructional Science, 37, 453-473.

Kicken, W., Brand-Gruwel, S., van Merriënboer, J. J. G., & Slot, W. (2009b). The effects of portfolio-based advice on the development of self-directed learning skills in secondary vocational education. Educational Technology Research and Development, 57, 439-460.

Saks, K., & Leijen, Ä. (2014). Distinguishing Self-directed and Self-regulated Learning and Measuring them in the E-learning Context. Procedia-Social and Behavioral Sciences, 112, 190-198. Retrieved from http://www.sciencedirect.com/science/article/pii/S1877042814011720

Van Merriënboer, J. J., & Kirschner, P. A. (2017). Ten steps to complex learning: A systematic approach to four-component instructional design. Routledge: New York.

Winne, P. H., & Hadwin, A. F. (2013). nStudy: Tracing and supporting self-regulated learning in the Internet. In International handbook of metacognition and learning technologies (pp. 293-308). Springer New York.

Winne, P. H. (2010). Improving measurements of self-regulated learning. Educational Psychologist, 45(4), 267-276. Retrieved from https://www.researchgate.net/publication/233214307_Improving_Measurements_of_Self-Regulated_Learning

Zimmerman, B. J. (2002). Becoming a self-regulated learner: An overview. Theory into practice, 41(2), 64-70. Retrieved from http://mathedseminar.pbworks.com/w/file/fetch/94760840/Zimmerman%20-%202002%20-%20Becoming%20a%20Self-Regulated%20Learner%20An%20Overview.pdf

 

[1] Guidance and support must be embedded in domain specific content! In other words, it doesn’t make any sense to teach a generic course on ‘learning how to learn’!

 

Advertisement