In recent years, with the increase of work intensity and life pressure, many people are suffering from unhealthy conditions. In this case, it is very necessary for those who care about health to maintain and enjoy an active and healthy life. Therefore, many people tend to improve their health status through nationwide fitness and exercise. With the development of electronic technology, more and more people use smart devices to monitor their movement. In view of the difficulties and challenges people are facing, this paper will propose a multi-user behavior monitoring algorithm based on machine learning, which can improve the frequency and enthusiasm of physical exercise in people's daily life, thus improving the overall health of society. The research on action recognition responds to the call of the World Health Organization and adapts to social development, which has great practical significance.