Measurement Demystified: Creating Your L&D Measurement, Analytics, and Reporting Strategy was the culmination of 10 years of work to bring standard measurement principles and reporting to human capital processes. We formed the Center for Talent Reporting (CTR) in the fall of 2012 with the mission to promote the adoption of those standards, known as Talent Development Reporting principals (TDRp), across talent functions. In 2013, we launched a two-day workshop to build measurement skills for talent practitioners and, over the next five years, conducted 22 workshops and trained more than 300 students.
Before each workshop, we asked participants to assess the measurement maturity of their organization. Here is what we found:
- 68 percent agreed that talent reporting was critical to senior business leaders.
- 54 percent indicated that they had a defined process to align the learning organization’s priorities to business requirements.
So far, so good.
As we interacted with our students, however, we observed some concerning patterns. First, our students used the same words and terms that we did but meant different things. Business alignment at one firm was a robust process to identify business needs and translate them into strategic learning solutions. At another firm, business alignment was a process of mapping existing L&D solutions to business goals and declaring that L&D was aligned. We realized that L&D functions defined alignment to mean whatever they wanted it to.
The second pattern was that despite L&D expressing a desire to “sit at the grown-ups table,” L&D was, in most cases, still operating with minimal accountability. Goal setting for those key performance measures was not the norm. (In our survey, just 34 percent said they set goals for efficiency, effectiveness, and outcome measures.)
A third pattern was that L&D organizations did not appear to value standard and repeatable practices, measures, or methods to manage operations (only 26 percent indicated that they had such practices). Rather, each organization reinvented the wheel, using variations of the Kirkpatrick or Phillips methodologies for effectiveness measures, but they struggled to demonstrate business impact or how to integrate activity measures into the mix.
Finally, despite low scores on several dimensions of our assessment, we later realized that survey respondents had unwittingly inflated their organization’s capability. During this five-year period, CTR certified a handful of practitioners in TDRp. These dedicated professionals committed to apply the practices and demonstrate adoption in their own organization. In our concluding meeting for certification, we asked candidates to assess their organization’s current measurement maturity. In all cases, their scores were lower than they had originally reported. Their organization had not lost ground in the intervening period. Rather, candidates admitted that their original scores suffered from the “I didn’t know what I didn’t know” syndrome. For example, until they learned what an actual business alignment process looked like, how were they to know that they weren’t doing it optimally?
Our experiences over those five years reinforced the need, not only for continued training of L&D professionals, but also for a resource that they could use to guide them in their measurement journey. We felt that Measurement Demystified would meet an important, unfilled need in the L&D profession: a comprehensive and holistic introduction to the topics of measurement, analytics, and reporting. We authored the book to provide the foundation, framework, and language for L&D professionals to gain a basic understanding of these areas and then apply that knowledge to create their own measurement and reporting strategy. We also developed the book as a reference and performance support tool, including 120 learning measures, definitions, and formulas, as well as usage recommendations.
After Measurement Demystified hit the bookstores, we reinvented our workshop as a six-module virtual program following the outline of the book. We modified our case study, incorporated knowledge checks, discussions, and Q&A. Homework encouraged students to apply the concepts to their own organization and share their insights with the group. In teaching the refreshed program, we observed that even the most committed practitioners still struggled with some key concepts.
However, strategic programs should also include measures to assess the business outcome the program is designed to impact. While this seemed simple enough, students were uncertain which programs were strategic. If their organization lacked an upfront business alignment process, there were no signals to indicate which programs were strategic. While students understood the concept in theory and tried to apply it, when the L&D function lacked a disciplined alignment process, practitioners found it difficult to implement TDRp as designed. We asked ourselves, How can we enable L&D professionals to build skills—particularly in L&D functions that are new to being managed like a business?
When we developed the Measurement Demystified Field Guide, we committed to addressing this problem by:
1. Acknowledging that advancing measurement maturity is a change effort requiring commitment and discipline.
As with any change effort, leaders first must understand and describe the current state of measurement and reporting in their organization. Do they mostly monitor results with little oversight, goal setting, and action planning? Or is the organization more sophisticated in its measurement practices, and instead needs to expand what it measures and enhance the quality of and approach to reporting? Gaining an unvarnished view of the current capabilities and gaps is a critical first step in the journey.
2. Recognizing that every organization has a different capacity and timeline for change. Once organizations know where they are, then must decide how far they want to go and at what speed.
We concluded that a maturity model would help L&D professionals gauge the capacity of their organization to advance its measurement maturity. If the organization was immature, we recommended that L&D focus on getting its own house in order and gradually engage the business in its planning, prioritization, and reporting processes. A more mature organization could leap ahead, standardize its current practices, and then move to more advanced processes such as collaborating with business-goal owners to agree on the isolated impact of training on a specific business initiative.
The Measurement Demystified Field Guide helps readers assess their organization’s maturity and serves as the bridge between the current state and where the organization aspires to go. It walks readers through the process step by step with opportunities to examine the practices of their own organization through inventories, interviews, and reflection exercises. While each volume of Measurement Demystified stands alone, together they address many of the challenges we have encountered over the past ten years and should enable organizations to embrace measurement and its ability to improve the operations of the L&D function.
