Assessing the Use of Mode Preference as a Covariate for the Estimation of Measurement Effects between Modes. A Sequential Mixed Mode Experiment

Abstract

Mixed mode surveys are presented as a solution to increasing survey costs and decreasing response rates. The disadvantage of such designs is the lack of control over mode effects and the interaction between selection and measurement effects. In a mixed mode survey, measurement effects can put into doubt data comparability between subgroups, or similarly between waves or rounds of a survey conducted using different modes. To understand the extent of measurement effects, selection and measurement effects between modes have to be disentangled. Almost all techniques to separate these effects depend on covariates that are assumed to be mode-insensitive and to fully explain selection effects. Most of the time, these covariates are sociodemographic variables that might be mode-insensitive, but fail to sufficiently explain selection effects. The aim of this research is to assess the performance of mode preference variables as covariates to evaluate selection and measurement effects between modes.

In 2012, a mixed mode survey – a web questionnaire followed by face-to-face interviews– was conducted alongside the face-to-face European Social Survey in Estonia (Ainsaar et al., 2013). The questionnaire included mode preference items. In this paper, the effects of the trade-offs between the two assumptions on the precision of estimated selection and measurement effects are compared. The results show that while adding the mode preference to the propensity score model seems to increase the explanatory power of web participation, it decreases the correlation between propensity scores and target variables. In addition, the estimated selection and measurement effects do not always fit the expectation that more selection effects are explained and more measurement effects are detected.