Advertisement
Advertisement
ai, analysis, artificial intelligence, automation, big data, brain, business, cg, cloud computing, communication, computer graphics, concept, creative, cyber, deep learning, digital transformation.png
ATD Blog

Measurement Demystified Field Guide: Q&A With Peggy Parskey and David Vance

Thursday, January 6, 2022
Advertisement

Few organizations have a well-thought-out measurement and reporting strategy. Often there are scant resources, limited time, and imperfect data to work with when organizations attempt to create one. Dave Vance and Peggy Parskey have written two books to demystify the process and provide practical, down-to-earth guidance for L&D practitioners.

Measurement Demystified: Creating Your L&D Measurement, Analytics, and Reporting Strategy (2021) offers a framework to simplify the discussion of measurement and help talent development professionals present their organizations’ data and analytics clearly. Measurement Demystified Field Guide (2022) builds on the first book to deepen understanding of the content and principles. The Field Guide contains more than 100 exercises guiding practitioners in applying the key concepts to a case study and then to the reader’s organization.

1. What prompted you to write the first book?

Vance: We launched an initiative called Talent Development Reporting Principles (TDRp) 11 years ago. Our goal was to bring measurement and reporting standards, best practices, and management principles to the field of L&D. We wanted the same kind of practical guidance for our field that accountants find in their Generally Accepted Accounting Principles (GAAP). During the last 11 years, we have refined and expanded our approach to the point where we wanted to share it broadly. We wanted to write the definitive book on TDRp and on measurement and reporting in general—a book that would provide detailed guidance for anyone to create their own measurement and reporting strategy.

Parskey: As we consulted with myriad talent organizations over the years, we discovered that leaders and practitioners struggled with measurement. While they understood the broad strokes of measurement, most of our clients did not have measurement frameworks or standard practices. We felt that one way to drive standards in our profession was to provide talent leaders and their staff with the nuts and bolts of talent measurement through this book.

2. Why is the Field Guide needed? Can readers use it as a stand-alone book?

Parskey: The first book shares the framework and all the important concepts but doesn’t provide an opportunity for the reader to apply them. The Field Guide meets this need by providing more than 100 exercises to help the reader gain a deeper understanding of the concepts and ensure they can apply the concepts in their own organization. Completing the exercises in the Field Guide should also significantly increase the reader’s confidence in their ability to create and execute their measurement and reporting strategy.

Vance: While each book is written to stand alone, the reader will benefit most from reading them together, gaining an in-depth understanding of a topic from the first book and then being able to immediately apply that understanding in the Field Guide. We have mapped Measurement Demystified to the Field Guide using seven lessons or topics. For example, Lesson 2 (Measures) recommends reading chapters 3–5 in the first book to gain a complete understanding of the three types of measures and then reading Chapter 2 in the Field Guide to test your understanding and apply it to your organization.

3. Why did you choose the title Measurement Demystified?

Vance: In our experience collaborating with clients and conducting workshops, most L&D professionals are put off by measurement, reporting, and the general notion of running learning like a business. Most would say they entered the profession out of a desire to help others learn and develop—not to measure or report. Few have experience with measurement or reporting, particularly from a business point of view. For most, measurement is a mystery. They understand why it is important, but they don’t have a framework to organize their thinking or a language to use to talk about measures. They don’t know where to begin and the steps to follow. We wanted a title that would let them know that we understand their lack of comfort with the topic and that help is here.

Parskey: When I first started working in learning and development within a large high-tech company, the measurement methods seemed arcane. I would hear people talking about Level 1 and Level 2, and I had no idea what that meant. I kept hoping that a secret decoder ring would fall into my lap so I could join the conversation. When we were noodling ideas for a title, this one seemed apt. This book would provide the step-by-nitty-gritty-step about measurement and remove the need for the decoder ring.

4. There are already several books on measurement. Why two more?

Vance: Jack and Patti Phillips have authored or co-authored more than 100 books on learning program evaluation, and others have also made important contributions. So, the field has excellent guidance for determining whether a learning program is effective and impactful. However, Measurement Demystified is the first book to present the TDRp framework for selecting measures, including a description of more than 100 efficiency measures that are generally not the subject of other works. This is also the first book to integrate the recent important work on human capital reporting by the International Organization for Standardization (ISO). It is also the first book to categorize the types of reports, create a framework for reporting, and directly tie the selection of a report to the reason for measuring. Similarly, the Field Guide is the first book to provide practice applying the TDRp concepts. Last, unlike other books, we provide detailed step-by-step guidance in both books to enable readers to create their own measurement and reporting strategy, including how to create the all-important plan, year-to-date results, and forecast values in advanced reports.

Advertisement

5. What are the right steps to create a measurement and reporting strategy?

Vance: The first step is to be clear on the reasons to measure. Talking with the user to understand their need may simply be an answer to a question (to inform), or to ensure the value of a measure meets a threshold (to monitor), or to determine if a program is effective (to evaluate), or to ensure a program delivers promised results over the course of the year (to manage). With the purpose clearly in mind, the next step is to select the appropriate and balanced set of measures. The TDRp framework groups measures into three buckets: efficiency, effectiveness, and outcomes. Generally, leaders select both efficiency and effectiveness measures and sometimes an outcome measure as well. The book describes more than 120 measures in detail so there are many from which to choose. Lastly, practitioners need to include the selected measures in the right type of report, ranging from a simple scorecard to a more complex dashboard to a program evaluation report to a detailed management report like your colleagues in sales and manufacturing employ.

Parskey: The process of creating a measurement and reporting strategy needs to be a collaborative effort. The L&D measurement team (or person) should not go off into a corner and create a brilliant plan on their own. Instead, L&D practitioners should engage with the CLO, directors, and program managers to ensure that the measurement and reporting strategy reflects both the stated and unstated needs of both learning and business leaders. Without engagement and approval of the plan by these leaders, the organization will struggle to successfully implement the strategy.

Analytics has gotten considerable attention over the last several years and is a trendy topic in the HR field. What role do you think analytics should play in a measurement strategy?

Vance: Analytics is not the highest-level purpose of measurement, although it is a very important part of measurement. The profession needs to move away from merely reporting historical results in scorecards and dashboards—which is all backward looking—and use measures to evaluate the effectiveness of a program. We are also excited by the opportunity to use analytical methods like regression and other statistical techniques to explore the relationships among measures. However, we firmly believe that an even more important purpose exists for measurement, a purpose that employs all the reporting and analysis that must precede it. We believe the highest-level purpose for measurement is the active management of key programs to deliver significant business value. This requires high-level analytics to properly understand and plan the program, and a continuing high level of analysis to understand what corrective action to take each month to maximize the probability of delivering promised value by year end.

Parskey: Advanced analytics tools, methods, and technology enable organizations to get deep insights into relationships among HR variables and can often predict outcomes such as the likelihood that high performers will stay with the organization or the degree of organizational fit for prospective hires. Analytics is critically important to enable HR to “compete on analytics” as Tom Davenport wrote in 2007. It is also a hot area, and who doesn’t want to be working on hot stuff? However, as Dave indicates, analytics is not enough. When we go beyond analytics to managing the function, we become accountable for our results and start behaving like our colleagues in every other function. While not nearly as hot as analytics, the practice of managing (versus monitoring) enables us to make significant contributions to business success.

Why do L&D professionals find it hard to get started?

Vance: The lack of a framework and knowledge our book addresses has been holding back L&D professionals in the measurement and reporting area. In addition, even when L&D practitioners know the steps, many lack the confidence to create a measurement and reporting strategy, which is one reason we wrote the Field Guide. For some, the task simply seems too big. For others, there are resource constraints. We believe it is important to do something, even with a few measures contained in a few basic reports. It is too easy to delay if you are waiting for perfection, and perfection is an illusion in any case. Plus, the only way to build confidence is to practice. Better to jump in and gradually grow your sophistication, adding measures and reports each year as you go. Over the years of plugging away, you will have a respectable strategy.

Parskey: Another factor holding L&D back is that we don’t fully emulate business practices of our colleagues in other functions, in particular, in goal setting and assuming accountability for concrete goals for activity, effectiveness, and also impact. Being accountable can be daunting, particularly when the practitioner doesn’t control the variables that impact the outcome. But that’s the nature of managing. Consider sales. Sales managers have a target sales level they must meet. They don’t control the buying habits of their accounts, they don’t control purchases practices within their client base, and they certainly don’t control the economic landscape that can make or break success. But salespeople find it perfectly natural to be assigned a goal and be held accountable to meet it. L&D needs to view the world from the same lens. We can’t control everything, but we can influence the results we are accountable to achieve. L&D organizations simply need to start setting goals, reviewing progress, taking action to achieve planned results, and reforecasting year-end performance. With practice and experience, L&D leaders and their teams will become comfortable with the process and will reap the benefits of managing their function.

What does “running learning like a business” mean, and why is it important?

Vance: The concept of running learning like a business means that leaders create specific, measurable plans for their key initiatives and programs at the start of the year. They execute those plans with discipline to ensure they achieve the planned results. For example, you may have committed to reaching 1,000 participants through five new programs by the end of the third quarter. Once the year is underway, you use monthly management reports to compare your progress against plan. If you are falling behind plan, you identify the reasons why and options to get back on track and then you enact those options. If that doesn’t work, next month you are at it again. You keep at it until you get back on plan or the year ends. This is how your colleagues in other departments have managed for years. Running learning with this same business discipline is key to earning senior leaders’ trust and resources. The concept also underpins the management reports and the highest-level use of measures for L&D.

Advertisement

Why is a balanced set of measures important?

Parskey: Eliyahu Goldratt, in one of his management books, says, “Tell me how you measure me, and I will tell you how I will behave.” Given that measurement is such a powerful trigger for behavior change, we need to ensure that our measures will incent the behaviors we desire. When we develop a balanced set of measures, we mitigate the unintended consequences that result from focusing on just one. For example, in the book, we provide a real-life example of a CLO who only focused on efficiency measures. He cut staff, reduced spending, and provided training that while efficient, was not even remotely effective. The people in the business filled the gap by developing their own training, which resulted in further lost time, incremental expense, and a delayed time to competency for new hires. Had the CLO focused on both efficiency and effectiveness, he would have identified how to provide the most effective training at the lowest cost, therefore optimizing both measures of success.

What are the next steps for the profession with regards to measurement and reporting?
Parskey: Because of the new SEC rules about disclosure of human capital metrics, HR, talent, and L&D in publicly owned organizations must now disclose people measures material to the business. As such, talent functions will need to get their internal reporting house in order to ensure their public disclosures are accurate and valid reflections of their internal processes. For many organizations, the SEC rules will be the catalyst to develop or adopt standards (such as TDRp), streamline their processes, improve the quality of their data, and implement technology that simplifies data collection and reporting. We believe these efforts will enhance not only the quality of reporting but also how L&D and HR manage the function.

What feedback have you received for Measurement Demystified?

Vance: Readers really appreciate our straightforward writing style and our ability to present complex topics in a simple and easy-to-understand way. They appreciate the framework which ties reasons to measure, measures, and reports together. And they like the step-by-step guidance we share to create a measurement and reporting strategy. They also find our approach very practical. Lastly, they like the comprehensive approach, which means we include all types of measures and define more than 120, all in one book.

Parskey: I’ve been delighted at the response to the book. I have found that L&D practitioners are using the book in different ways. One colleague told me that he uses it as a comprehensive reference guide. He said, “I can go directly to the relevant chapter in the book and use that content to guide my immediate work.” In other cases, L&D organizations are using it with their teams to build internal capability. A few contacts reached out on LinkedIn to let me know that the book is now a must read in their department. They assign a chapter to their team and then discuss its implication for their own measurement practices. The book’s logical organization and comprehensive contents enable it to serve both uses.

About the Authors

David Vance is the executive director of the Center for Talent Reporting, a nonprofit dedicated to the creation and implementation of standards for human capital measurement, reporting, and management. He is the former president of Caterpillar University, which he founded in 2001. Prior to that position, Dave was chief economist and head of the business intelligence group at Caterpillar Inc., with responsibility for economic outlooks, sales forecasts, market research, and competitive analysis.

He is the author of The Business of Learning, now in its second edition, and co-author with Peggy Parskey of Measurement Demystified and Measurement Demystified Field Guide, both recently published by ATD Press. He teaches in the PhD programs at Bellevue University and the University of Southern Mississippi and in the executive education program at George Mason University.

Peggy Parskey is the assistant director of the Center for Talent Reporting. Peggy owns her own consulting firm, Parskey Consulting, providing measurement and organizational change consulting services. Peggy leverages more than 25 years of expertise in performance measurement, management of change, human performance improvement, and knowledge management to improve clients’ organizational, team, and individual performance.

Peggy is also a principal consultant in measurement with Explorance, a firm focused on talent measurement across the entire employee lifecycle. She is the co-author of Learning Analytics, second edition, with John Mattox and Cristina Hall, and is the co-author with David Vance of Measurement Demystified and Measurement Demystified Field Guide, recently published by ATD Press.

About ATD and ATD Press

The Association for Talent Development (ATD) is the world’s largest association dedicated to those who develop talent in organizations. ATD’s members come from more than 120 countries and work in public and private organizations in every industry sector. ATD Press publications are written by industry thought leaders and offer anyone who works with adult learners the best practices, academic theory, and guidance necessary to move the profession forward. For more information, visit td.org/books.

About the Author

The Association for Talent Development (ATD) is a professional membership organization supporting those who develop the knowledge and skills of employees in organizations around the world. The ATD Staff, along with a worldwide network of volunteers work to empower professionals to develop talent in the workplace.