Computer System for Converting Gestures to Text and Audio Messages

Abstract

Today, there are quite a large number of deaf- mute and hard-of-hearing people which communicate using gestures. Therefore, it is simply necessary to provide them with modern means of communication with the surrounding world. This paper creates a holistic computer system architecture for converting gestures into text and audio messages. The principles of construction and basic design solutions of a computer system based on a modern element base with increased productivity and minimization of hardware costs and energy consumption have been developed. The most popular existing solutions for gesture recognition are considered and analyzed. The operation of the main components has been described, the principle of functioning of the entire system has been analyzed, and their advantages and disadvantages have been compared. The latest structural components for building a computer system (both physical and software) have been selected and investigated. Physical features include: the state-of-the-art Arduino Nano computing platform, the HC-05 Bluetooth module, the ADXL335 accelerometer, and the latest ZD10-100 Information sensor (flexibility sensor). Software features include: firmware for the Arduino Nano hardware platform, Python-based software for splitting the flow of letters into words, displaying them, and voicing them. The methods of Google Media Translation API and Google Text-to-speech (gTTS) have been analyzed. The expediency of conducting research has improved performance through the use of a new information sensor, which is a flexibility sensor ZD10-100 500 g. The general structural scheme of all systems has been designed.