Probing in Cognitive Interviews Can Promote Acquiescence

Abstract

Cognitive interviewing is widely used to pretest survey questionnaires and is considered a best practice (e.g., Willis, 2005, 2018; Beatty & Willis, 2007). However, the method has been controversial because, among other concerns, it requires interviewers to probe respondents for more detail or clarity about their experience answering draft survey questions which may lead them to report “problems’’ they have not actually experienced (e.g., Conrad & Blair, 2009). The present study investigates this possibility from the perspective of Acquiescent Response Style (ARS) – the tendency for survey respondents to select positive responses such as “yes” or “strongly agree,” irrespective of the question’s content (e.g., Baumgartner & Steenkamp, 2001). For example, respondents in a cognitive interview might affirm experiencing a problem mentioned in or implied by an interviewer’s probe even if they have not actually experienced it. We embedded a probing experiment in a pretest of a health survey in which respondents participated in cognitive interviews that used either directive probes (n=41) or non-directive probes (n=26). Directive probes explicitly queried respondents about a specific, intentionally unlikely interpretation of each question in a draft questionnaire; non-directive probes were open-ended. Directive probe (DP) respondents affirmed the interpretation queried in the probes over five times more often than respondents in the non-directive probe (NP) group volunteered these interpretations. This pattern was reversed for interpretations of the questions that were volunteered, i.e., about which DP respondents were not asked: NP respondents volunteered alternative interpretations over four times more than DP respondents. These effects were particularly pronounced for respondents with lower levels of education and who were younger. The findings suggest that directive probing in cognitive interviewing can promote responding that is reminiscent of ARS – an affirmation bias – and likely harmful for the quality of evidence produced in cognitive interviews.