An Empirical Test of the Rapid Message Testing at Scale (RMTS) Approach (Preprint)

Abstract

Message testing can provide vital information to make data-informed decisions for effective communication, but traditional methods tend to be time and resource intensive. During an infodemic, the need for timely, reliable, and accessible information is even more crucial to combat the proliferation of health misinformation. Recognizing this need, we developed the Rapid Message Testing at Scale (RMTS) approach to allow communicators to repurpose existing social media advertising tools to help them understand the full spectrum of audience engagement.

We had two main objectives: a) to demonstrate the use of the RMTS approach for message testing, especially when resources are limited and there is a need to act in a timely manner, such as during an infodemic, and (b) to propose and test the efficacy of an outcome variable that goes beyond just views but, instead, incorporates a continuum of viewing experience.

We developed twelve versions of a single vaccine promotion video, manipulating video length (original/shortened), aspect ratio (rectangular/square/portrait), and use of subtitles (presence/absence). We created and launched 48 unique ads on Facebook that tested the 12 variations across four demographic groups (women/men, younger/older). Using Meta’s audience segmentation tools, we assessed user engagement along a continuum of viewing experience: from getting attention, sustaining attention, conveying the message, and inspiring action.

The 48 video ads were placed on Facebook over four consecutive days, at the cost of $450 (U.S.) and garnered a total of 3.34 million impressions. Overall, our results indicate that, regardless of aspect ratio manipulations and the presence or absence of subtitles, greater engagement occurred for shorter videos, those presented in portrait aspect ratio, and those without subtitles (up to 25% watch time, after which subtitles scored higher). We report differential effects for time retention and click-throughs.

This test served as a proof of concept for the RMTS approach. We obtained rapid feedback on formal message attributes from a very large sample in a very short timeframe. Additionally, the results of this test reinforce the need for platform-specific tailoring of communications. While our data showed a general preference for the short video shown in portrait orientation and without subtitles, RMTS testing highlights nuances that communication professionals can address instead of being limited to a ‘one size fits all’ approach. To effectively compete in a crowded information environment characteristic of an infodemic, it is vital that online communication efforts use all the tools and data available.