Evaluation

Bad value

Don’t waste your time energy and money working on products and services nobody wants.

I was asked if I had any thoughts on evaluation this evening, well I guess they wanted to know if I had thoughts on evaluation at any time, it’s just that they asked me this evening, anyway it turns out I didn’t. It’s not that I don’t have any thoughts at all on the subject, it’s just that I can’t pull anything insightful straight off the top of my head at 8pm on a Monday evening.

I decided to give that twitter place a try and see if I could find some inspiration there and you know what, it certainly got me thinking.

First challenge was to define what was I actually asking about – I clarified slightly that I was talking about evaluation of programmes and interventions like leadership/management development programmes or OD interventions.

Top bloke Ian Pettigrew pointed me towards Kirkpatrick’s model. My simple understanding of it is:

  • Reaction
  • Learning
  • Behaviour
  • Results

He also pointed to one of his own blogs about mapping ROI in this case for social media, but applicable widely – good quick read, thx Ian.

His final offering really got me thinking – how could I use one of my favourite frameworks, Alex Osterwalder’s Value Proposition thinking to support evaluation.

This approach would mean evaluation was driven by the customer (learner) and the value they have derived (or perceived to derive) from the learning experience.

At the other end of the spectrum Julie Drybrough reminded me that even big brains struggle with this stuff, pointing out that someone has produced a 104 page evaluation of a Google Leadership Programme that doesn’t mention the learner experience at all.

There are some folk who take evaluation, and it’s cousin Evidence Based Practice very seriously. Thanks to Siobhan for pointing me towards the Center for Evidence-Based Management and the amazing (and slightly intimidating) amount of resource they have available.

Con Sotidis went a stage further and found me a free webinar of the actual Kitkpatricks, presumably talking about the Kirkpatrick model. – Loads of other great stuff there too – Makes you wonder why we bother trying to put anything together ourselves?

Con also mentioned the #OzLearn twitter chat 10 am GMT Tuesday 10th Nov which is in Evaluation. I’m busy unfortunately but Con assures me a Storify will appear on their LinkedIn page

And as the twitter folk headed to bed Siobhan found me a great article on the evidence for Leadership from Karen Lynas

I did do a bit of searching for myself (not much to be fair – will add more later)

I also had a look at The LnDConnect LinkedIn page to see what was there, and found a couple of interesting storifies of #LDInsight chats that were related.

Ken Fee also came up trumps and shared some great resources from Training Zone, which has introduced me to the Success Case Method and Robert O Brinkerhoff. He notes that we often make the mistake of evaluating learning in isolation as one off events, which is like testing the success of a marriage by evaluating the wedding.

My thinking

No deep thinking yet, just a couple of reflections.

  • Evaluation at the end is too late, it has to be an ongoing and iteritive process. What’s the use of a happy sheet at the end of a one day workshop? Really!
  • Evaluation should be based on what the learner thinks is valuable and how the learning experience helped them. They are grown ups for gods sakes, who generally know more about their job than you. We should trust them to judge if the learning they have experienced has helped them.
  • There is a huge amount of resource out there if you know where to look. This can make evaluation seem big and scary…I think when evaluation is big and scary it is probably taking up too much space in the world.

2 Comments

  1. Meg Peppin · 10 November, 2015 Reply

    My offering; using the principles of Action Research the research question becomes the intervention in a way. Starting with all the questions creates the foundation and the evaluation becomes integrated with the development/growth/change process. I Love evaluation, but it ain’t no form baby.

    • Kev Wyke · 10 November, 2015 Reply

      Thanks for the comment Meg, I will add action research to my list of stuff I need to know more about. I think starting with the questions in mind is really important, evaluation seems most useful to me when it helps to inform what you are doing as you are doing it rather than after you have finished. I also found an ace quote to illustrate the difference between formative and summative “When a cook tastes the soup, it is formative evaluation; when the dinner guest tastes the soup it is summative evaluation.”. It might be important for the dinner guest to like the soup, but it’s to damn late to do anything about it by the time they taste it!

Leave a Reply

%d bloggers like this: