- This is the Survey Design chapter from The Survey System's
Tutorial, revised July, 2000. It is reproduced here as a service to the
research community. Copyright 2000, Creative Research Systems.
- Constructing a Questionnaire
- The Steps in a Survey Project
- 1.Establish the goals of the project - What you want to learn?
- 2.Determine your sample - Who you will ask?
- 3.Choose interviewing methodology - How you will ask?
- 4.Create your questionnaire - What you will ask?
- 5.Pre-test the questionnaire, if practical - Test the questions.
- 6.Analyze the data - Produce the reports.
- Selecting Your Sample
- There are two main components in determining whom you will interview.
The first is deciding what kind of people to interview. Researchers often
call this group the target population. If you conduct an employee attitude
survey or an association membership survey, the population is obvious.
If you are trying to determine the likely success of a product, the target
population may be less obvious. Correctly determining the target population
is critical. If you do not interview the right kinds of people, you will
not successfully meet your goals.
- The next thing to decide is how many people you need to interview.
Statisticians know that a small, representative sample will reflect the
group from which it is drawn. The larger the sample, the more precisely
it reflects the target group. However, the rate of improvement in the precision
decreases as your sample size increases. For example, to increase a sample
from 250 to 1,000 only doubles the precision. You must make a decision
about your sample size based on factors such as: time available, budget
and necessary degree of precision.
- This Web site includes a sample size calculator that can help you decide
on the sample size (jump to the calculator page for a general discussion
of sample size considerations).
- Questionnaire Design
- KISS - keep it short and simple.If you present a 20-page questionnaire
most potential respondents will give up in horror before even starting.
Ask yourself what you will do with the information from each question.
If you cannot give yourself a satisfactory answer, leave it out. Avoid
the temptation to add a few more questions just because you are doing a
questionnaire anyway. If necessary, place your questions into three groups:
must know, useful to know and nice to know. Discard the last group, unless
the previous two groups are very short.
- Start with an introduction or welcome message.State who you are and
why you want the information in the survey. A good introduction or welcome
message will encourage people to complete your questionnaire.
- Allow a "Don't Know" or "Not Applicable" response
to all questions, except to those in which you are certain that all respondents
will have a clear answer. In most cases, these are wasted answers as far
as the researcher is concerned, but are necessary alternatives to avoid
frustrated respondents. Sometimes "Don't Know" or "Not Applicable"
will really represent some respondents' most honest answers to some of
your questions. Respondents who feel they are being coerced into giving
an answer they do not want to give often do not complete the questionnaire.
For the same reason, include "Other" or "None" whenever
either of these are a logically possible answer. When the answer choices
are a list of possible opinions, preferences or behaviors you should usually
allow these answers.
- Question Types
- Researchers use three basic types of questions: multiple choice, numeric
open end and text open end (sometimes called "verbatims"). Examples
of each kind of question follow:
- Multiple choice
- 1. Where do you live? ___ north
- ___ south
- ___ east
- ___ west
- Numeric open end
- 2. How much did you spend on food this week?
- Text open end
- 3. How can our company improve working conditions?
- Rating Scales and Agreement Scales are two common types of questions
that some researchers treat as multiple choice questions and others treat
as numeric open end questions. Examples of these kinds of questions are:
- Rating scales
- 4. How would you rate this product?
- ___ excellent
- ___ good
- ___ fair
- ___ poor
- 5. On a scale where 10 means you have a great amount of interest and
1 means you have none at all, how would you rate your interest in each
of the following topics?
- ___ domestic politics
- ___ foreign affairs
- ___ science
- ___ business
- Agreement scale
- 6. How much do you agree with each of the following statements?
- strongly agree disagree strongly
- agree disagree
- My manager provides constructive criticism ___ _____
- Our medical plan provides adequate coverage ___ _____
- I would prefer to work longer hours on ___ _____ ____
- fewer days
- Question and Answer Choice Order
- There are two broad issues to keep in mind when considering question
and answer choice order. One is how the question and answer choice order
can encourage people to complete your survey. The other issue is how the
order of questions or the order of answer choices could affect the results
of your survey. Ideally, the early questions in a survey should be easy
and pleasant to answer. These kinds of questions encourage people to continue
the survey. Grouping together questions on the same topic also makes the
questionnaire easier to answer. Whenever possible leave difficult or sensitive
questions until near the end of your survey. Any rapport that has been
built up will make it more likely people will answer these questions. If
people quit at that point anyway, at least they will have answered most
of your questions. Answer choice order can make individual questions easier
or more difficult to answer. Whenever there is a logical or natural order
to answer choices, use it. Always present agree-disagree choices in that
order. Presenting them in disagree-agree order will seem odd. For the same
reason, positive to negative and excellent to poor scales should be presented
in those orders. When using numeric rating scales higher numbers should
mean a more positive or more agreeing answer.
- Question order can affect the results in two ways.One is that mentioning
something (an idea, an issue, a brand) in one question can make people
think of it while they answer a later question, when they might not have
thought of it if it had not been previously mentioned. The other way question
order can affect results is habituation. This problem applies to a series
of questions that all have the same answer choices. It means that some
people will usually start giving the same answer, without really considering
it, after being asked a series of similar questions. People tend to think
more when asked the earlier questions in the series and so give more accurate
answers to them.
- The order in which the answer choices are presented can also affect
the answers given. People tend to pick the choices nearest the start of
a list when they read the list themselves on paper or a computer screen.
People tend to pick the most recent answer when they hear a list of choices
read to them. As mentioned previously, sometimes answer choices have a
natural order (e.g., Yes, followed by No; or Excellent - Good - Fair -
Poor). If so, you should use that order. At other times, questions have
answers that are obvious to the person that is answering them (e.g., "What
brand(s) of car do you own?"). In these cases, the order in which
the answer choices are presented is not likely to affect the answers given.
However, there are kinds of questions, particularly questions about preference
or recall or questions with relatively long answer choices that express
an idea or opinion, in which the answer choice order is more likely to
affect which choice is picked.
- Other Tips
- 1. Keep the questionnaire as short as possible. We mentioned this principle
before, but it is so important it is worth repeating. More people will
complete a shorter questionnaire, regardless of the interviewing method.
If a question is not necessary, do not include it.
- 2. Start with a title (e.g., Leisure Activities Survey). Always include
a short introduction - who you are and why you are doing the survey. You
may want to leave a space for the respondent to add their name and title.
Some people will put in their names, making it possible for you to recontact
them for clarification or follow-up questions. Indicate that filling in
their name is optional. Do not have a space for a name, if the questions
are sensitive in nature. Some people would become suspicious and not complete
- 3. Start with general questions. If you want to limit the survey to
users of a particular product, you may want to disguise the qualifying
product. As a rule, start from general attitudes to the class of products,
through brand awareness, purchase patterns, specific product usage to questions
on specific problems (i.e., work from "What types of coffee have you
bought in the last three months" to "Do you recall seeing a special
offer on your last purchase of Brand X coffee?"). If possible put
the most important questions into the first half of the survey. If a person
gives up half way through, at least you have the most important information.
- 4. Make sure you include all the relevant alternatives as answer choices.
Leaving out a choice can give misleading results. For example, a number
of recent polls that ask Americans if they support the death penalty yes
or no have found 70-75% of the respondents choosing "yes." But
polls that offer the choice between the death penalty and life in prison
without the possibility of parole show support for the death penalty at
about 50-60%. While polls that offer the alternatives of the death penalty
or life in prison without the possibility of parole, with the inmates working
in prison to pay restitution to their victims' families have found support
of the death penalty closer to 30%.
- 5. So what is the true level of support for the death penalty? The
lowest figure is probably best, since it represents the percentage that
favor that penalty regardless of the alternative offered. The need to include
all relevant alternatives is not limited to political polls. You can get
misleading data anytime you leave out alternatives.
- 6. Do not put two questions into one. Avoid questions such as "Do
you buy frozen meat and frozen fish?" A "Yes" answer can
mean the respondent buys meat or fish or both. Similarly with a question
such as "Have you ever bought Product X and, if so, did you like it?"
A "No" answer can mean "never bought" or "bought
and disliked." Be as specific as possible. "Do you ever buy pasta?"
can include someone who once bought some in 1990. It does not tell you
whether the pasta was dried, frozen or canned and may include someone who
had pasta in a restaurant. It is better to say "Have you bought pasta
(other than in a restaurant) in the last three months?" "If yes,
was it frozen, canned or dried?" Few people can remember what they
bought more than three months ago unless it was a major purchase such as
an automobile or appliance.
- 7. The overriding consideration in questionnaire design is to make
sure your questions can accurately tell you what you want to learn. The
way you phrase a question can change the answers you get. Try to make sure
the wording does not favor one answer choice over another.
- 8. Avoid emotionally charged words or leading questions that point
towards a certain answer. You will get different answers from asking "What
do you think of the XYZ proposal?" than from "What do you think
of the Republican XYZ proposal?" The word "Republican" in
the second question would cause some people to favor or oppose the proposal
based on their feelings about Republicans, rather than about the proposal
itself. It is very easy to create bias in a questionnaire. This is another
good reason to test it before going ahead.
- 9. If you are comparing different products to find preferences, give
each one a neutral name or reference. Do not call one "A" and
the second one "B." This immediately brings images of A grades
and B grades to mind, with the former being seen as superior to the latter.
It is better to give each a "neutral" reference such "M"
or "N" that do not have as strong a quality difference image.
If possible, just refer to the "first" product and the "second"
- 10. Avoid technical terms and acronyms, unless you are absolutely sure
that respondents know they mean. LAUTRO, AGI, GPA, EIEIO (Life Assurance
and Unit Trust Regulatory Organization, Adjusted Gross Income, Grade Point
Average and Engineering Information External Inquiries Officer) are all
well-known acronyms to people in those particular fields, but very few
people would unde rstand all of them. If you must use an acronym, spell
it out the first time it is used.
- 11. Make sure your questions accept all the possible answers. A question
like "Do you use regular or premium gas in your car?" does not
cover all possible answers. The owner may alternate between both types.
The question also ignores the possibility of diesel or electric-powered
cars. A better way of asking this question would be "Which type(s)
of fuel do you use in your cars?" The responses allowed might be:
- Regular gasoline
- Premium gasoline
- Do not have
- 12. If you want only one answer from each person, ensure that the options
are mutually exclusive. For example:
- n which of the following do you live?
- a house
- an apartment
- the suburbs
- This question ignores the possibility of someone living in a house
or an apartment in the suburbs.
- 13. Score or Scale questions (e.g., "If "5" means very
good and "1" means very poor how would rate this product?")
are a particular problem. Researchers are very divided on this issue. Many
surveys use a ten-point scale, but there is considerable evidence to suggest
that anything over a five point scale is irrelevant. This depends partially
on education. Among university graduates a ten point scale will work well.
Among people with less than a high school education five points is sufficient.
In third world countries, a three- point scale (good/acceptable/bad) is
often all a respondent can understand. Another problem is that you are
assuming that the difference in the factors is within the scale limits
- you may have a five-point scale but in a respondent's mind one factor
may rate 10 points in comparison to the others.
- If you do use a rating scale be sure the labels are meaningful. For
- What do you think about product X?
- It's the best
on the market
- It's about average
- It's the worst
on the market
- A question phrased like the one above will force most answers into
the middle category, resulting in very little usable information.
- If you have used a particular scale before and need to compare results,
use the same scale. Four on a five-point scale is not equivalent to eight
on a ten-point scale. Someone who rates an item "4" on a five-point
scale might rate that item anywhere between "6" and "9"
on a ten-point scale.
- 14. Be aware of cultural factors. In the third world, respondents have
a strong tendency to exaggerate answers. Researchers are often perceived
as being government agents, with the power to punish or reward according
to the answer given (and this is sometimes true). Accordingly they often
give "correct" answers rather than what they really believe.
Even when the questions are not overtly political and deal purely with
commercial products or services, the desire not to disappoint important
visitors with answers that may be considered negative may lead to exaggerated
scores. Always discount "favorable" answers by a significant
factor in all cases. The desire to please is not limited to the third world.
- 15. Leave your demographic questions (age, sex, income, education,
etc.) until the end of the questionnaire. By then the Interviewer should
have built a rapport with the Interviewee that will allow honest responses
to such personal questions. Mail questionnaires should do the same, although
the rapport must be built by good question design, rather than personality.
Exceptions are any demographic questions that qualify someone to be included
in the survey. For example, many researchers limit some surveys to people
in certain age groups. These questions must come near the beginning.
- 16. Paper questionnaires requiring text answers, should always leave
sufficient space for handwritten answers. Lines should be about half-an-inch
(one cm.) apart. The number of lines you should have depends on the question.
Three to five lines are average. Leave a space at the end of a questionnaire
entitled "Other Comments." Sometimes respondents offer casual
remarks that are worth their weight in gold and cover some area you did
not think of, but which respondents consider critical. Many products have
a wide range of secondary uses that the manufacturer knows nothing about
but which could provide a valuable source of extra sales if approached
properly. In one third world market, a major factor in the sale of candles
was the ability to use the spent wax as floor polish - but the manufacturer
only discovered this by a chance remark.
- 17. Always consider the layout of your questionnaire.This is especially
important on paper, computer direct and Internet surveys.You want to make
it attractive, easy to understand and easy to complete. If you are creating
a paper survey, you also want to make it easy for your data entry personnel.
Try to keep your answer spaces in a straight line, either horizontal or
vertical. A single answer choice on each line is best. Eye tracking studies
show the best place to use for answer spaces is the right hand edge of
the page.It is much easier for a field worker or respondent to follow a
logical flow across or down a page. Using the right edge is also easiest
for data entry.
- 18. Questions and answer choice grids, as in the second of the following
examples, are popular with many researchers. They can look attractive and
save paper, or computer screen space. They also can avoid a long series
of very repetitive question and answer choice lists. Unfortunately, they
also are a bit harder than the repeated lists for some people to understand.
As always, consider whom you are studying when you create your questionnaire.
- Look at the following layouts and decide which you would prefer to
- Do you agree, disagree or have no opinion that this company has:
- A good vacation policy -
- Good management feedback
- agree/not sure/disagree.
- Good medical insurance -
- High wages - agree/not sure/disagree.
- An alternative layout is:
- Do you agree, disagree or have no opinion that this company has:
- agree not
- good management ___ ___ ___
- good medical coverage ___ ___ ___
- high wages ___ ___ ___
- The second example shows the answer choices in neat columns and has
more space between the lines. It is easier to read. The numbers in the
second example will also speed data entry. Surveys are a mixture of science
and art, and a good researcher will save their cost many times over by
knowing how to ask the correct questions.
- Pre-test the Questionnaire
- The last step in questionnaire design is to test a questionnaire with
a small number of interviews before conducting your main interviews. Ideally,
you should test the survey on the same kinds of people you will include
in the main study. If that is not possible, at least have a few people,
other than the question writer, try the questionnaire. This kind of test
run can reveal unanticipated problems with question wording, instructions
to skip questions, etc. It can help see if the interviewees are understanding
your questions and giving useful answers. If you change any questions after
a pre-test, you should not combine the results from the pre-test with the
results of post-test interviews.
- Further Reference
- Designing Survey Questions. (Online). Available: http://www.tgsa.edu/online/cybrary/lwedekin.html
[October 10, 2001].