I was defending Kirkpatrick’s levels the other day, and after being excoriated by my ITA colleagues, I realized there was not only a discrepancy between principle and practice, but between my interpretation and as it’s espoused. Perhaps I’ve been too generous.
The general idea is that there are several levels at which you can evaluate interventions:
- whether the recipient considered the intervention appropriate or not
- whether the recipient can demonstrate new ability after the intervention
- whether the intervention is being applied in the workplace, and
- whether the intervention is impacting desired outcomes.
That this is my interpretation became abundantly clear. But let’s start with what’s wrong in practice.
In practice, first, folks seem to think that just doing level 1 (‘smile sheets’) is enough. Far fewer people take the next logical step and assess level 2. When they do, it’s too often a knowledge test. Both of these fail to understand the intention: Kirkpatrick (rightly) said you have to start at level 4. You have to care about a business outcome you’re trying to achieve, and then work backwards: what performance change in the workplace would lead to the desired outcome. Then, you can design a program to equip people to perform appropriately and determine whether they can, and finally see if they like it. And, frankly, level 1 is useless until you finally have had the desired impact, and then care to ensure a desirable user experience. As a standalone metric, it ranks right up there with measuring learning effectiveness by the pound of learners served.
Now, one of the things my colleagues pointed out to me, beyond the failure in implementation, is that Kirkpatrick assumes that it has to be a course. If it’s just misused, I can’t lay blame, but my colleagues proceeded to quote chapter and verse from the Kirkpatrick site to document that the Kirkpatricks do think courses are the solution. Consequently, any mention of Kirkpatrick only reinforces the notion that courses are the salve to all ills.
Which I agree is a mindset all too prevalent, and so we have to be careful of any support that could lead a regression to the status quo. Courses are fine when you’ve determined that a skill gap is the problem. And then, applying Kirkpatrick starting with Level 4 is appropriate. However, that’s more like 15% of the time, not 100%.
So where did I go wrong? As usual, when I look at models, I abstract to a useful level (my PhD focused on this, and Felice Ohrlich did an interesting study that pointed out how the right level of abstraction is critical). So, I didn’t see it tied to courses, but that it could in principal be used for performance support as well (at least, levels 3 and 4). I think also at least for some social learning interventions.
Moreover, I was hoping that by starting at level 4, you’d look to the outcome you need, and be more likely to look at other solutions as well as courses. But I had neglected to note the pragmatic issue that the Kirkpatrick’s imply courses are the only workplace intervention to move the needles, and that’s not good. So, from now on I’ll have to be careful in my reference to Kirkpatrick.
The model of assessing the change needed and working backward is worthwhile, as is doing so systematically. Consequently, at an appropriate level of abstraction, the model’s useful. However, in it’s current incarnation it carries too much baggage to be recommended without a large amount of qualification.
So I’ll stick to talking about impacting the business, and determining how we might accomplish that, rather than talk about levels, unless I fully qualify it.
