Assessing a Decision Support Tool for SOC Analysts

Abstract

It is difficult to discern real-world consequences of attacks on an enterprise when investigating network-centric data alone. In recent years, many tools have been developed to help understand attacks using visualisation, but few aim to predict real-world consequences. We have developed a visualisation tool that aims to improve decision support during attacks in Security Operation Centres (SOCs). Our tool visualises propagation of risks from sensor alert data to Business Process (BP) tasks. This is an important capability gap present in many SOCs today, as most threat detection tools are technology-centric. In this article, we present a user study that assesses our tool’s usability and ability to support the analyst. Ten analysts from seven SOCs performed carefully designed tasks related to understanding risks and recovery decision-making. The study was conducted in laboratory conditions with simulated attacks and used a mixed-method approach to collect data from questionnaires, eye tracking, and semi-structured interviews. Our findings suggest that relating business tasks to network asset in visualisations can help analysts prioritise response strategies. Finally, our article also provides an in-depth discussion on user studies conducted with SOC analysts more generally, including lessons learned, recommendations and a critique of our own study.