|
Questionnaires are popular method for obtaining information from a large number of people. Most likely, you have filled out a survey questionnaire. And, you probably came across a question that didn't quite make sense. Or, maybe you weren't sure exactly what information the question was trying to obtain from you. If you don't understand the question, chances are that you won't answer it correctly. In order to make questionnaires more understandable, a tool has been developed to help make those confusing questions more understandable. QUAID, (Question Understanding Aid) is a software program designed to assist survey methodologists, social scientists, and designers of questionnaires in improving the wording, syntax, and semantics of questions. Specifically, QUAID analyzes questions by flagging potential problems in the questions. This system works similar to that of a spell check facility. QUAID flags potential problems similar to how a spell checker flags misspelled words. However, a spell checker gives you the option of changing the word in question while QUAID does not give the option of revising the question, therefore, it is ultimately the user's decision on whether or not to revise the question. The tool identifies 5 potential problems that respondents might have in comprehending the meaning of questions on survey questionnaires. The 5 problems found by QUAID can be scrutinized by researchers when they revise questions to improve question comprehension and, thereby, enhance the reliability and validity of answers. The 5 problems that QUAID finds are unfamiliar technical terms, vague or imprecise relative terms, vague or ambiguous noun phrases, complex syntax and working memory overload. To test QUAID's validity, several experiments were conducted. First an experiment was conducted with language experts. Questions that QUAID revised were compared with ratings of language experts who evaluated a corpus of questions on the five classes of problems. The corpus consisted of 550 questions on 11 surveys contributed by the US Census Bureau. Analyses of this study showed that QUAID critiques questions similar to the way an expert performs. In order to test the "real-world" value of QUAID, an eye tracking experiment was conducted with the questions by the the Census Bureau. This study showed that, when QUAID considered particular words to be a problem, participants tended to fixate on those words longer. When QUAID considered the question to be syntactically complicated or pose a high load on short-term memory, participants tended to not completely read the question, demonstrating an early exit strategy. Finally, to prove QUAID's effectiveness to the specific population that it was directed towards, a study was conducted with experts in language. This study showed that experts tended to pick questions that were revised by QUAID over questions that were revised without QUAID and questions that were not revised.
|
