Moderating manipulation: Demystifying extremist tactics for gaming the (regulatory) system

Abstract

AbstractDue to its ease of scalability and broad applicability, the use of artificial intelligence (AI) and machine learning in platform management has gained prominence. This has led to widespread debates about the use of deplatforming as the default tool for repeated or severe violations of terms or service. But technologically deterministic approaches are not infallible and can be predictable based on their actions. This opens the door for manipulation of media content and technological affordances to become tactical options for actors seeking to subvert regulation. Existing discussions often neglect the topic of manipulation of content, algorithms, or platform affordances as a primary aspect of the strategies used by extremists in relation to the difficulties of moderation from a policy perspective. This study argues that it is essential to understand how extremists and conspiracy theorists use manipulation tactics to ‘game’ the current policy, regulatory and legislative systems of content moderation. Developing approaches that attend to manipulation as a strategy and focus on platform and context‐specific tactics will generate more effective policies, platform rules, AI developments and moderation procedures. This study analyses and demystifies three primary tactics, which the authors categorize as numerology, borderlands and merchandising, regularly used by extremists online in their strategies to ‘game’ content moderation. We provide case examples from a variety of ideologies including far‐right, QAnon and male supremacism to highlight the tactics rather than ideological nature of such manipulation. We conclude with a discussion of how demystification processes could be incorporated into content moderation settings. This study contributes new insights about evasion tactics to the content moderation discussion and expands current understanding of how platforms can develop sociotechnical remedial measures.