organizational development and change tools and techniques for consultants and others

Tips for building a valid survey

by David Zatz, Ph.D. (Toolpack Consulting)

First, some warnings for employee surveys in particular:

Some key validity tips

  1. Questions must be clearly understood by everyone, so the language should be as simple as possible to avoid literacy issues (which are not pointed out by most respondents).
  2. Unless there are social expectations to fight, we should be direct.
  3. The shorter the sentence, the more likely people will read it rather than scanning it.
  4. All statements should be phrased in a positive way, or the phrasing effect will drown out the content.
  5. Doing statistics is far easier when you have a single scale.
  6. Rank orders are hard for the respondent and even harder for reporting and statistics.
  7. Each concept gets its own question.
  8. Questions should be behavioral and concrete rather than conceptual, wherever possible.
  9. As a general rule to remember, people do not read instructions.
  10. Large print and frequent paragraph breaks increases the likelihood that adults will read the full text.
  11. Follow survey conventions so people don't get confused. For employee surveys, this means go from left to right, negative to positive, with the most positive items having the highest numbers (e.g. strongly agree = 5 and strongly disagree = 1). On Web surveys, they normally aren't numbered. Help respondents to fill out the survey using the right scale. One person using the wrong scale can wreak havoc — and it’s very common.
  12. Work hard to get as many people in the sample as possible to complete it, to avoid nonresponse bias.
  13. Avoid "binary" questions that "lose" information ("Are you satisfied?" should be "How satisfied are you?" and "Do you want this service?" should be "How much would you pay for this service?")
  14. When necessary, define the anchors completely. While this creates a statistical violation (you can no longer simply assume the distance between each number is identical in size), the effects may be minimal, and you may be able to avoid a great deal of bias and guessing whether respondents are interpreting the scales the same way. (Rather than simply asking "How well does the organization's mission guide your actions? -- Completely to Not at all," define each step, e.g. "I refer to it each time I make a decision," to "I never use the mission to make real decisions," with intermediate steps also filled in.)

A basic process for reliability and validity testing

This is a relatively fast process for basic reliability and validity testing. For the “more proper” method you can see the APA, AMA, or other research-organization Web sites.

Reliability is whether the survey gives you the same answers at different times, and whether the questions within it measure the same thing (only applicable if you're doing a set of questions to measure a single issue, e.g. engagement, involvement, satisfaction, depression, etc.)

Validity is whether the survey measures what it’s supposed to measure. If a survey is not reliable over time, it cannot be valid, because it will vary depending on when it’s taken.

Ideally you'd check a new instrument against an older one that measures the same thing and has been validity tested already. However, most often people are developing something new because nothing exists already.

  1. Develop the survey after doing a literature search and gathering needed information.
  2. Use extra items where possible partly to deal with items that are struck out, and partly to provide some degree of internal validity testing via interitem correlations (that is, by seeing if any items within the survey tend not to change with the others.)
  3. Circulate to local experts for their opinion, for face validity.
  4. Pilot test.
    1. Ask 5-10 people to take it
    2. Ask them to tell you immediately if anything is confusing or hard to answer
    3. Watch where they "get stuck"
    4. Ask where things could be easier to understand, or better in general
    5. Ask for any criticisms.
  5. Give patents the survey, wait as long as you can, and give it to them again with the questions in a different order, and see if the link between the same people is greater than the link between different people.
  6. Find another method to compare the survey to, e.g. face to face interviews, and compare results of both; or by give the survey in a different form, e.g. open-ended questions / fill in the blanks.
  7. Repeat as needed.
  8. Ideally, when the survey is adminstered the first few times, have extra open-ended questions so you can do a “quality check” on the numerical data.

Consumer Market Research 


For your further perusal:

This site provides organizational development and change information. All materials are copyright © 2001-2014, Toolpack Consulting, LLC, except where noted. All rights reserved. Do not reproduce without written permission. Thank you.

Effective mergers

Surveys for change

Useful employee surveys

Executive credibility

Solving problems quickly

Why consultants fail

Survey norms

Engagement and informal contracts

How cultural change can fail

Process mapping guide

Building valid surveys

Change in education

Career advice

Helpful books

Mac statistics software

Terms of service