Thanks to all who could attend Tuesday’s webcast on survey basics with Patti Phillips. You can view the archive here in case you missed it. The topic of creating quality effective surveys inspired some terrific questions from attendees. Here are two Patti didn’t address due to time.
Q: If there are more neutral answers [to a survey], is that good or bad? What do you do?
A: If you do have a high number of neutral answers, first look at the frequency with which people select the neutral response. If more people are neutral than otherwise, it may be that whatever they are rating truly is “middle of the road.” On the other hand, you might also look at the specific question to see if the question is well written. Oftentimes we see the majority of people score the neutral response because the question is so vague they can't answer it objectively, but they want to give an answer, so they select the middle.
Also, consider the audience. Is there alignment? Sometimes we ask questions that are not relevant to everyone in the audience. When this is the case, then the N/A response choice may be applicable (But don't add N/A to all of your questions. N/A is only useful when the audience is so large and diverse that a question really may not be applicable).
If after you change the question you are still seeing a majority of your respondents selecting the middle response choice, consider going with a six-point scale.
Keep in mind: So much of an argument about scales is based on preference versus science. Use the response choice that gives you the best information given the question you need to ask. For some questions, choices will be a simple yes or no; for others, you may need four response choices, some five, and some six. Six to seven response choices is about as many as you need for most self-administered questionnaires.
Q: The organization I joined regularly surveys its leaders (approximately 200). We're lucky if 50 people respond. I suspect that the problem is two-fold: 1) survey fatigue, and 2) more importantly, a fear that the survey responses are not truly anonymous. The feedback is important and needed to guide future program development. How would you handle this issue given that it needs to be corrected in a short period of time? (That is, it can't involve total culture change.)
A: First, don't claim the survey to be anonymous. In today's culture, most people don't believe much in anonymity. But do explain that the results will be held in confidence and explain how.
Also, explain why the information is important and what you plan to do with it. Then report it out in such a way that respondents see that you have done what you said you would do. It is likely that the leaders view the survey as trivial and one that will add no value because they are not seeing the value. So set it up so that they see the potential value, then demonstrate the value by sharing results and the actions taken as a consequence of those results.
Finally, consider having a higher level executive support the administration of the survey. You may be doing this already. If that is the case, take a look at the process and see how you can improve it. These actions do not involve a total culture change—just a new way a communicating.
Special thanks to Dr. Phillips for taking time to answer these questions. Of course, if you want to dive deeper into this topic, attend the all-day workshop with Patti at ASTD 2014!
