nunosempere.github.io/ea/Surveys/Checklist.md
2019-12-13 11:12:02 +01:00

3.6 KiB
Executable File
Raw Blame History

Pareto Principle Checklist for Question Design

Problems to watch out for before testing the questionnaire.

  1. PROBLEMS WITH READING: Determine whether it is difficult for the interviewers to read the question uniformly to all respondents.
    1a WHAT TO READ: Interviewers may have difficulty determining what parts of the question are to be read.
    1b MISSING INFORMATION: Information the interviewer needs to administer the question is not contained in the question.
    1c HOW TO READ: Question is not fully scripted and therefore difficult to read.

  2. PROBLEMS WITH INSTRUCTIONS: Look for problems with any introductions, instructions, or explanations from the respondents point of view.
    2a CONFLICTING OR INACCURATE INSTRUCTIONS, introductions, or explanations.
    2b COMPLICATED INSTRUCTIONS, introductions, or explanations.

  3. PROBLEMS WITH ITEM CLARITY: Identify problems related to communicating the intent or meaning of the question to the respondent.
    3a WORDING: The question is lengthy, awkward, ungrammatical, or contains complicated syntax.
    3b TECHNICAL TERMS are undefined, unclear, or complex.
    3c VAGUE: The question is vague because there are multiple ways in which to interpret it or to determine what is to be included and excluded.
    3d REFERENCE PERIODS are missing, not well specified, or are in conflict.

  4. PROBLEMS WITH ASSUMPTIONS: Determine whether there are problems with assumptions made or the underlying logic.
    4a INAPPROPRIATE ASSUMPTIONS are made about the respondent or his/her living situation.
    4b ASSUMES CONSTANT behavior: The question inappropriately assumes a constant pattern of behavior or experience for situations that in fact vary.
    4c DOUBLE-BARRELED question that contains multiple implicit questions.

  5. PROBLEMS WITH KNOWLEDGE/MEMORY: Check whether respondents are likely to not know
    or have trouble remembering information.
    5a KNOWLEDGE: The respondent is unlikely to know the answer.
    5b An ATTITUDE that is asked about may not exist.
    5c RECALL failure.
    5d COMPUTATION or calculation problem.

  6. PROBLEMS WITH SENSITIVITY/BIAS: Assess questions for sensitive nature or wording, and for bias.
    6a SENSITIVE CONTENT: The question is on a topic that people will generally be uncomfortable talking about.
    6b A SOCIALLY ACCEPTABLE response is implied.

  7. PROBLEMS WITH RESPONSE CATEGORIES: Assess the adequacy of the range of responses to be recorded.
    7a OPEN-ENDED QUESTIONS that are inappropriate or difficult.
    7b MISMATCH between question and answer categories.
    7c TECHNICAL TERMS are undefined, unclear, or complex
    7d VAGUE response categories.
    7e OVERLAPPING response categories.
    7f MISSING response categories.
    7g ILLOGICAL ORDER of response categories.

Things to pay attention to when testing the questionnaire.

  • Which items require respondents to think hard before they answer?
  • What cognitive processes do they adopt to answer difficult questions?
  • Which items seem to produce irritation, embarrassment, or confusion?
  • Are there any items that respondents consider comical?
  • Does the style of the question generate bias?
  • Are the answers we get what we really want for the purpose of the study?
  • Is there enough variability in the answers received?
  • Are there local expressions that should be incorporated into the items to avoid ambiguity?
  • Is the questionnaire too long?
  • In the eye of the respondent, have any other important issues been overlooked in the questionnaire?