From Clicks to Quality: Assessing Advertisement Design’s Impact on Social Media Survey Response Quality

Abstract

Researchers are increasingly using social media platforms for survey recruitment. However, empirical evidence remains sparse on how the content and design characteristics of advertisements used for recruitment affect response quality in surveys. Building on leverage-salience and self-determination theory, we assess the effects of advertisement design on response quality. We argue that different advertisement designs may resonate with specific social groups who vary in their commitment to the survey, resulting in differences in the observed response quality. We use data from a study conducted via ads placed on Facebook in Germany and the United States in June 2023. The survey, focusing on attitudes toward climate change and immigration, featured images with varying thematic associations with the topics (strong, loose, neutral). The sample consisted of 4,170 respondents in Germany and 5,469 respondents in the United States. We compare several data quality indicators, including break-off rate, completion time, non-differentiation, item non-response, passing an attention check question, and follow-up availability, across different advertisement features. Regression analyses indicate differences in response quality across advertisement designs, with a strong thematic design generally being associated with poorer response quality. Strongly themed ad designs are generally associated with higher attrition, non-differentiation, and item non-response, and with a lower probability of passing an attention check and providing an e-mail address for future survey inquiries. Our study advances the literature by highlighting the substantial impact of advertisement design on survey data quality, and emphasizing the importance of tailored decision-making in recruitment design for social media-based survey research.