Measuring Twitter Use: Validating Survey-Based Measures

Abstract

An important challenge for research on social media use is to relate users’ activity on these platforms to user characteristics such as demographics. Surveys allow researchers to measure these characteristics but may be subject to measurement error in self-reported social media use. We compare survey responses to observed behavior in order to assess the validity of self-reported frequency of posting to Twitter, retweeting content, sharing photos, sharing videos, and sending direct messages. Additionally, we examine correlations between self-reported and observed behavior across a range of time frames, from 1 month to 114 months before the survey. We find variation in the quality of self-reports across types of Twitter activity. We also find that self-reports about posting and retweeting tend to reflect recent activity, while self-reports about other activities tend to reflect behavior over a longer span. Furthermore, we find that two characteristics of experience with the platform—the length of time that a person has been active on Twitter and how much their activity on the platform changes over time—predict individual-level discrepancies between survey response and observed behavior, but these discrepancies cancel out when averaged across individuals. Nevertheless, other sources of bias remain. Taken together, our results indicate that while surveys are quite useful for collecting characteristics of social media users, relying on self-reported social media behavior distorts inferential results from what is found when relying on observed social media behavior.