Surveys are probably the most used but least useful research tool. It is ever so tempting to say, “lets run a quick survey” when you find yourself wondering about your customers. Surveys result in “hard numbers” to look at, and modern web-based survey tools have made surveys cheap to produce. But as anyone who has ever tried running a “quick survey” can attest, they rarely, if ever, provide the insight you are looking for.
In the words of Erika Hall, survey’s are “too easy”. They are too easy to create, too easy to disseminate, and too easy to tally. This inherent ease of creating surveys masks their biggest flaw as a research method: It is far far too easy to create biased, useless survey questions. And when you run a survey littered with biased, useless questions, you either (1) realize that your results are not reliable and start all over again, or (2) proceed with the analysis and make decisions based on biased results. If you aren’t careful, a survey can be a complete waste of time, or worse, can lead you in the wrong direction entirely.
However, that being said, I have found surveys to be useful in exactly two situations:
- When I need to gather demographic data that I can’t obtain otherwise
- When I work for a client who will send an email blast with a survey link, but otherwise wont help me recruit research participants.
If you ever find that launching a survey is the only way to move the research forward, keep these tips in mind.
Tip 1: Try to stick to questions about facts, not opinions
If you were building a website for ordering dog food and supplies, it would be worthwhile to ask a question like, “how many dogs do you own?” in a survey. It is a demographic question that cannot be gotten through your usual analytics tool. It would also be super easy, quick, and convenient to get the answer through a short survey. However, if what you really want is an answer to, “why did you decide to adopt a dog in the first place?” then you are better off asking that in a user interview.
If you try asking any kind of “why” question in a survey, you usually end up with a lot of “I don’t know” and otherwise blank responses. This is because people are, in general, to busy to write out long responses. For example, no one wants to write an essay on why they’ve chosen to adopt a dog when they are in the middle of ordering pet food.. However, when people schedule time for a phone-call, they are more than willing to talk (at length!) about the “whys” behind pet ownership. In short, people like to talk about their opinions, but are generally too lazy or busy to write about their opinions. Save the why questions for later (and see Tip #5).
Tip 2: Avoid asking about the future.
People live in the present, and only dream about the future. There are a lot of things outside of our control that affect what we will buy, eat, wear, and do in the future. Also, sometimes the future selves we imagine are more aspirational than factual. For example, if you were to ask a random group of people how many times they plan to go to gym next month you might be (not so) surprised to see that their prediction is significantly different from the actual number. It is much better to ask “how many times did you go to the gym this week?” as an indicator of general gym attendance than to ask about any future plans.
Tip 3: Know how you are going to analyze responses before you launch the survey
A lot of times, people will create and send-out a survey without thinking through what they are going to do once they have a bunch of results. Depending on the length and type of survey, the analysis could take a significant amount of time. Also, if you were hoping to answer some specific questions, you’ll want to make sure you know how you’ll arrive at those answers. I recommend that after you draft a survey, go through each question and draft an analysis plan.
In your analysis plan, think about what you are ultimately trying to learn from each survey question. How will you know when you’ve arrived at the answer? If you are doing an A/B test, what statistical analysis should you run ? You should also think about what the numbers will look like. What kind of graphs or tables you will need to build? Ultimately, you should try to visualize what the data will look like before you gather it and plan accordingly.
Tip 4: You should never rely on a survey, by itself, to make important decisions
Surveys are hard to get right, and even when they are well made, the results are often approximations to what you really want to measure. However, if you pair a survey with a series of interviews or contextual inquiries, you will have a richer data set. In the social sciences, this is called triangulation. You use multiple methods to triangulate and study the same phenomenon. This provides a richer, more complete picture. This leads me to my final tip…
Tip 5: End every survey with an opportunity to participate in future research
There have been many times when I have launched surveys with only one objective in mind: to gather contact information for potential study participants. The survey questions themselves are superfluous. The fact that they took the time to fill out a few questions tells me they are interested in participating in research. All I need then is their email address. Shortly after completing the survey, they will find an email from me inviting them to participate in more research.
More resources on writing surveys:
- On Surveys, by Erika Hall
- Writing Usable Survey Questions, by Tanya Feinstein, UXPA