UTILIZATION OF VOICE EMBEDDINGS IN INTEGRATED SYSTEMS FOR SPEAKER DIARIZATION AND MALICIOUS ACTOR DETECTION

Abstract

This paper explores the use of diarization systems which employ advanced machine learning algorithms for the precise detection and separation of different speakers in audio recordings for the implementation of an intruder detection system. Several state-of-the-art diarization models including Nvidia’s NeMo Pyannote and SpeechBrain are compared. The performance of these models is evaluated using typical metrics used for the diarization systems such as diarization error rate (DER) and Jaccard error rate (JER). The diarization system was tested on various audio conditions including noisy environment clean environment small number of speakers and large number of speakers. The findings reveal that Pyannote delivers superior performance in terms of diarization accuracy and thus was used for implementation of the intruder detection system. This system was further evaluated on a custom dataset based on Ukrainian podcasts and it was found that the system performed with 100% recall and 93.75% precision meaning that the system has not missed any criminal from the dataset but could sometimes falsely detect a non-criminal as a criminal. This system proves to be effective and flexible in intruder detection tasks in audio files with different file sizes and different numbers of speakers which are present in these audio files. Keywords: deep learning diarization speaker embeddings speaker recognition cyber security.