Kirpatrick’s four levels of evaluation are often cited and regularly used as a standard for training evaluation. His approach is handy because it's one-size-fits-all and is useful for Instructor Led Training evaluation. However, it's sometimes faulted for not being detailed enough. When choosing evaluation methods for your training – be it electronic or instructor led- there are other training evaluation models to choose from. Below I describe Kirkpatrick's four levels, and go on to outline some of the more popular, alternative training evaluation models.
Kirkpatrick
Kirkpatrick is without a doubt the most frequently used technique. It suggests measurement of training at four levels:
- Reaction: Measures the learner’s reaction to the training immediately after the session. Seeks to gain information on their perception of the quality of the training content, and the trainer. (information often gained through reaction sheets)
- Learning: Measures actual learning achieved in the training.
- Behaviour: Measures transfer of learning, including any changes in behaviour back on the job.
- Results: Measures the business impact of the training programme.
Phillip's ROI Model
This model adds a fifth level to Kirkpatrick’s four levels: Return-On-Investment (ROI). This level is purely concerned with the measure of financial benefit (or lack thereof) to an organisation after training. The Training ROI formula is as follows: ROI (%) = NET Program Benefits / Program Costs x 100.
Villanova University offers certification in Phillips ROI methodology, and they outline it as a ten-step process to gather and analyze evaluation data (as opposed to four steps). In their case, however, they are applying the model more generally to any programme evaluation, not just training evaluation.
Hodges’s Components for HRD Evaluation
Hodges sees training evaluation as a selection of the following components:
- Needs Assessment = what do they need?
- Formative Evaluation = will it work?
In addition, there are four components to Summative Evaluation as follows:
- Reaction Evaluation = were they satisfied?
- Learning Evaluation = did they learn?
- Performance Evaluation = were they able to use it, and where they successful in using it?
- Impact Evaluation = what impact has it had and was there a financial return?
Holton: HRD Evaluation and Research Model
Holton includes three evaluation targets (learning, transfer, and results). He excludes reaction sheets collected at the end of a training course (also known as “Happy Sheets” or “Smiley Sheets”), which he doesn’t consider valid as a primary outcome of training. He argues for an integration of effectiveness and evaluation, determining that certain effectiveness criteria be included in evaluation.
Kraiger: Decision-Based Evaluation
Kraiger’s model involves three multi-dimensional areas for evaluation.
- Training Content and Design: design, deliver, validate your training courses using expert judgment and advisory panels; establish a methodology.
- Changes in Learners: measure cognitive, behavioral, and affective outcomes with written tests, interviews, surveys and work samples.
- Organizational Payoffs: these should include results, performance and training transfer. Evaluate these with cost-benefit analyses, surveys and ratings.
Kraiger emphasizes evidence, or “decision-based” evaluation. He considers the purpose of training evaluation to be to present evidence for decision-making, feedback to designers, and trainers and marketing. Kraiger is also dubious of the value of reaction sheet measures, as he doesn’t believe they’re related to either changes in learner behaviour or to payoff. In addition, he deems that novice learners don’t reply satisfactorily.
Evaluation Summary
Evaluation of training programmes is often neglected, with possible reasons for this including lack of time, lack of budget, or lack of expertise and knowledge of evaluation tools. There is evidence that trainers themselves may also be resistant to evaluation, as they may have additional work put back on them in the form of re-writing training materials, or in fact being held accountable for failures in training transfer and poor learning outcomes. However, the re-working of the training material is necessary as organizations have ever-changing goals, often in reaction to changing market-place requirements. Organizations also need to know if they are getting the change they need, including financial benefits. The tools above can help you to be clear about training achievements in your company.
Sources:
- Alvarez, K., Salas, E., and Garofano, C. M. (2004). An Integrated Model of Training Evaluation and Effectiveness. Human Resource Development Review, Vol. 3, No. 4, December 2004, pp. 385-416
- Garavan, Thomas N., Hogan, Carole, and Cahir-O’Donnell, Amanda. (2003) Making Training and Development Work: A Best Practice Guide. Cork, Ireland: Oak Tree Press.
- Harris, Frank. (2006). Evaluation Models and Methods, in The Role of Capacity Building in Police Reform, OSCE Mission in Kosovo: Department of Police Education and Reform (pp. 153-172).
- Holton, Elwood, F. (2005). Holton's Evaluation Model: New Evidence and Construct Elaborations. Advances in Developing Human Resources , Vol. 7, No. 1 February 2005, pp. 37-54
Lory Manrique-Hyland - Lory Manrique-Hyland, Novelist ("Revolutions" Lilliput, 2004), trainer & contributor on parenting for a national Irish newspaper.
Join the Conversation