Models and Algorithms for Multimodal Data Processing

Abstract

Information technologies and computer equipment are used in almost all areas of activity, which is why new areas of their use are emerging, and the level of ICT implementation is deepening, with more and more functions that were the prerogative of humans being assigned to computers. As science and technology develop, new technologies and technical means are emerging that enable a human-centered approach to software development, better adaptation of human-machine interfaces to user needs, and an increase in the ergonomics of software products, etc. These measures contribute to the formation of fundamentally new opportunities for presenting and processing information about real-world objects with which an individual interacts in production, educational and everyday activities in computer systems. The article aims to identify current models and algorithms for processing multimodal data in computer systems based on a survey of company employees and to analyze these models and algorithms to determine the benefits of using models and algorithms for processing multimodal data. Research methods: comparative analysis; systematization; generalization; survey. Results. It has been established that the recommended multimodal data representation models (the mixed model, the spatiotemporal linked model, and the multilevel ontological model) allow for representing the digital twin of the object under study at differentiated levels of abstraction, and these multimodal data processing models can be combined to obtain the most informative way to describe the physical twin. As a result of the study, it was found that the "general judgment of the experience of using models and algorithms for multimodal data processing" was noted by the respondents in the item "Personally, I would say that models and algorithms for multimodal data processing are practical" with an average value of 8.16 (SD = 0 1.70), in the item "Personally, I would say that models and algorithms for multimodal data processing are understandable (not confusing)" with an average value of 7.52. It has been determined that respondents positively evaluate (with scores above 5.0) models and algorithms for processing multimodal data in work environments as practical, understandable, manageable, and original. columns finish at the same distance from the top of the page.