A robot-based surveillance system for recognising distress hand signal

Abstract

Abstract Unfortunately, there are still cases of domestic violence or situations where it is necessary to call for help without arousing the suspicion of the aggressor. In these situations, the help signal devised by the Canadian Women’s Foundation has proven to be effective in reporting a risky situation. By displaying a sequence of hand signals, it is possible to report that help is needed. This work presents a vision-based system that detects this sequence and implements it in a social robot, so that it can automatically identify unwanted situations and alert the authorities. The gesture recognition pipeline presented in this work is integrated into a cognitive architecture used to generate behaviours in robots. In this way, the robot interacts with humans and is able to detect if a person is calling for help. In that case, the robot will act accordingly without alerting the aggressor. The proposed vision system uses the MediaPipe library to detect people in an image and locate the hands, from which it extracts a set of hand landmarks that identify which gesture is being made. By analysing the sequence of detected gestures, it can identify whether a person is performing the distress hand signal with an accuracy of 96.43%.