Autonomous military systems beyond human control: putting an empirical perspective on value trade-offs for autonomous systems design in the military

Abstract

AbstractThe question of human control is a key concern in autonomous military systems debates. Our research qualitatively and quantitatively investigates values and concerns of the general public, as they relate to autonomous military systems, with particular attention to the value of human control. Using participatory value evaluation (PVE), we consulted 1980 Australians about which values matter in relation to two specific technologies: an autonomous minesweeping submarine and an autonomous drone that can drop bombs. Based on value sensitive design, participants were tasked to enhance the systems with design features that can realize values. A restriction (limited budget) in each design task forced participants to make trade-offs between design options and the values that these options realize. Our results suggest that the ‘general public’ has diverse and nuanced stances on the question of human control over autonomous military systems. A third of participants that is opposed to autonomous military systems when asked directly, selected different combinations of design features realizing varying degrees of human control. Several contextual factors, technology-specific concerns, and certain values seemed to explain these different choices. Our research shows that a focus on human control might overlook other important values that the general public is concerned about, such as system reliability, verifiability, and retrievability.