Oversurveying, overpolling, and diminishing returns

For a pleasant 17 years, I worked with a consumer goods maker on annual and, later, monthly surveys. The goal was to provide a steady stream of market research, tapping a set of motivated, dedicated, and often enthusiastic customers.

We originally had enviable response rates from our Web surveys, coming close to matching in-person and phone surveys — an incredible 60% response in some cases, more like an employee survey than market research. In some cases we had giveaways, and in other cases we did not, but we rarely tapped less than 40% of the people we sent requests to.

survey-in-computer

Over the years, that response rate kept dropping. To a degree, the mailing list was diluted, which was responsible for part of it; but even among people we knew to be current, valid customers, we started finding it hard to get a response rate over 25%. The returns dwindled and it became harder to justify taking action on the results.

What happened?

I recently came across a 2010 article in the Harvard Business Review lamenting the same issue, some eight years ago, when it hadn’t yet hit my little part of the world. The writer briefly talked about some of the reasons for the drop in response rate, but his main concern was its impact (making survey data invalid). I’d like to talk a little about the reasons for it, though.

survey

If you are like me, your phone is constantly being spammed with telemarketing calls. When you do get asked to join a survey, half the time it’s a political survey where they’re fishing for a particular set of answers or setting you up for a donation. Your physical mailbox, for a time, was probably full of surveys from political and issues groups, designed for the same reasons — talking points and donations. At the same time, all our contact points — phone, physical mail, email, Facebook, LinkedIn — are prone to constant phishing and spam attacks. Even legit-looking emails, half the time, turn out to either be phishing or trying to direct you to a malware site that has a URL similar to a real one.

When you do get real surveys, they often seem to have no effect, or worse. Your car company sends you a survey about your experience at the dealership; if you’re brutally honest, your service advisor or the manager may show you a copy of your survey when you get there, which is a betrayal of the trust you had in the company. Anything less than a rating of “10 of 10” can result in a dealership or an individual losing their bonus, which prevents some people from giving honest answers. Pointing out problems rarely seems to result in any change, and when companies or governments do change things as a result of the surveys, they rarely announce that link.

It’s no surprise to me that response rates have dropped!

There are still ways to get responses, and I’ll talk about those later this month.

Leave a Reply