From Chapter 2, Science Watch
Baloney Buster 12: Questioning Questionable Questions
Beside wording question carefully, good researchers avoid asking poor questions through pretesting -- trying out the questions on a sample of people similar to the test subjects. After a pretest, the researchers discuss the questions with the people who answered them. The object is to learn which questions were too hard, too confusing, or too limiting. That information helps the researchers pinpoint questions that should be reworded, dropped, or added.
Here are some examples of questions before pretesting and the changes that might result. Unfortunately, poor researchers seldom pretest their questions, so they don't catch problems before they carry out their studies.
Before Pretest: Do you ever watch hockey and soccer on TV?
Problem: Combining two questions. After Pretest: Do you ever watch hockey on TV?
Do you ever watch soccer on TV?
Before Pretest: Like most people your age, do you watch TV documentaries only rarely? Problem: Biased After Pretest: How often do you watch TV documentaries?
- frequently? - occasionally? - rarely? - never?
Before Pretest: About how much time do you spend watching TV in a normal week? Problem: No problem After Pretest: No change is needed
Before Pretest: What impact has TV had on your
family? Problem: Unanswerable. Respondent can't say how TV affects each individual After Pretest: Drop the question.
ALWAYS ASK: Were the questions pretested? YOUR TURN: Try your own question pretest. Ask some of your friends the unanswerable question and the vague question in Baloney Buster #11, and see what kinds of responses you get. Then ask your friends why they had a tough time answering.