ASTD Benchmarking Forum Spring 2004 meeting Managing with Metrics: Data-Driven Workplace Learning and Performance.

After a hefty investment of financial and human capital, your new training program has finally gotten off the ground, and now your boss wants to know: Where's the return on our investment?

If you've done your homework, you can respond with credible and meaningful data that can pinpoint how that investment in training and performance is paying off.

At the ASTD Benchmarking Forum "Managing with Metrics: Data-Driven Workplace Learning and Performance," held at Defense Acquisition University April 28-29 and hosted by Boeing and DAU, learning and performance professionals gathered with this focus in mind. Providing the big picture presentation was Reza Sisakhti. Director, Learning & Performance Practice, Productivity Dynamics, who enumerated steps for capturing the impact of training in a work environment. Titled "Managing with Metrics at a Macro and Micro Level: Experiences and Lessons Learned in Multiple Organizations," the presentation outlined frameworks for measuring the overall benefits of training initiatives (macro-level perspective) and capturing the bottom-line business impact of particular, individual strategic initiatives (micro-level perspective).

"I don't have a silver bullet," Sisakhti admitted, "just a lot of experience in measuring these things." He demonstrated this experience with an overview of how to prepare the appropriate metrics that allow various types of training and education to be evaluated. "You need to really do your homework," Sisakhti said, adding that anticipating the measurement criteria before deploying any training is the key to creating successful metrics.

Methods of measuring the impact of training must be tailored to fit specific circumstances. From a micro perspective, these measurements vary between easily established, fixed criteria to more subjective evaluations. For example, evaluating skill-building or technical training is straightforward. Such training is an easy sell: it provides a new procedure, tool, or technique, such as teaching a technician to install a cable, that results in the establishment of a solid skill set. If the training works, the results are easily quantifiable; management can count the number of technicians now trained to install cables, or measure how much faster cables can be installed after new training is conducted.

Measuring context-dependent initiatives, such as management of training or leadership skills, is more subjective. Such training cannot be applied uniformly; not everyone will apply the new skills and styles in the same way. Establishing uniform measurements of effectiveness results in ambiguous, meaningless goals such as "increased production" or "increased profits" that are too far removed from the actual training to be of value. To meaningfully capture the results of such training, measurements must shadow how the learned skills are being used; measurements must be contextual, and applied on a case-by-case basis. After teaching management skills, trainers must investigate how the skills were then employed. Did managers focus the new skills on teamwork? On mentoring? On improving communication? Where and how did different groups receiving the training use the new knowledge? Individualized followup may be necessary to track how people used new skills. This kind of tracking suggests that context-dependent initiatives need a menu of measurement criteria, not pre-determined specific outcomes.

If such tailored tracking seems unwieldy, Sisakhti provided an action plan. After being exposed to education and new ideas, members of the group each developed a three-month action plan that outlined how the new skills could be leveraged in their jobs. After 45 days, interviews are conducted to solicit feedback on the implementation and success of these action plans. Interviews are conducted again after three months. Such feedback can then be collected and reviewed, and the impact of the training can be measured.

No matter the type of training, it is essential to establish what the criteria are for evaluating success. Sisakhti added that setting up a measurement system based on the client's own language and technical vocabulary is also critical in effectively capturing the right measurements. Again, adequate preparation before deploying the training is key.

Infrastructure investment initiatives require yet another approach. For example, if an independent Web-based learning system has been established, the content and skill sets being taught must be evaluated, but the portal itself needs also be evaluated. If the infrastructure is not supporting the training, that must be captured by the appropriate metrics.

Moving to a macro view adds yet another layer of complexity. After considering all the micro initiatives, a company may question the overall impact on the bottom line. All the seminars, ongoing training, new infrastructures to improve the learning environment--what does it all add up to?

A macro perspective must provide a means of measuring the aggregate impact of multiple initiatives and the cumulative impact of total investment. One solution to the challenge of measuring the overall impact of isolated learning initiatives is a time series measurement design. Such a framework selects a "unit of analysis"--managers, sales people, service professionals, departments--who have all completed multiple initiatives, and measures their progress over time to assess the overall impact of training. Metrics to consider for such a design include employee loyalty, retention, innovation, customer satisfaction, and financial and industry benchmarks.

Sisakhti added a further level for consideration: the organizational perspective. The learning function itself is an organization in its own right; metrics are often necessary to create a balanced scorecard for the learning organization.

Too often, lower-level metrics aimed at micro initiatives focus on the whole, creating criteria that are too broad or far removed to effectively measure the results of training. Micro initiatives demand specifically tailored metrics. A focus on the big picture, however, remains equally important; a macro perspective is necessary to provide answers when a CEO asks, "What are we finally getting for all our investment?" Planning must be done in advance of deploying training to establish a soundly structured system to capture information for both the micro and macro perspectives.

Cavoli is a freelance writer/editor providing contract support to Defense AT & L.