Convolutional neural networks to classify human stress that occurs during in‐field sugarcane harvesting: A case study

Abstract

AbstractAssessing human stress in agriculture proves to be a complex and time‐intensive endeavor within the field of ergonomics, particularly for the development of agricultural systems. This methodology involves the utilization of instrumentation and the establishment of a dedicated laboratory setup. The complexity arises from the need to capture and analyze various physiological and psychological indicators, such as heart rate (HR), muscle activity, and subjective feedback to comprehensively assess the impact of farm operations on subjects. The instrumentation typically includes wearable devices, sensors, and monitoring equipment to gather real‐time data of subject during the performance of farm operations. Deep learning (DL) models currently achieve human performance levels on real‐world face recognition tasks. In this study, we went beyond face recognition and experimented with the recognition of human stress based on facial features during the drudgery‐prone agricultural operation of sugarcane harvesting. This is the first research study for deploying artificial intelligence‐driven DL techniques to identify human stress in agriculture instead of monitoring several ergonomic characteristics. A total of 20 (10 each for male and female) subjects comprising 4300 augmented RGB images (215 per subject) were acquired during sugarcane harvesting seasons and then these images were deployed for training (80%) and validation (20%). Human stress and nonstress states were determined based on four ergonomic physiological parameters: heart rate (ΔHR), oxygen consumption rate (OCR), energy expenditure rate (EER), and acceptable workload (AWL). Stress was defined when ΔHR, OCR, EER, and AWL reached or exceeded certain standard threshold values. Four convolutional neural network‐based DL models (1) DarkNet53, (2) InceptionV3, (3) MobileNetV2 and (4) ResNet50 were selected due to their remarkable feature extraction abilities, simple and effective implementation to edge computation devices. In all four DL models, training performance results delivered training accuracy ranging from 73.8% to 99.1% at combinations of two mini‐batch sizes and four levels of epochs. The maximum training accuracies were 99.1%, 99.0%, 97.7%, and 95.4% at the combination of mini‐batch size 16 and 25 epochs for DarkNet53, InceptionV3, ResNet50, and MobileNetV2, respectively. Due to the best performance, DarkNet53 was tested further on an independent data set of 100 images and found 89.8%–93.3% confident to classify stressed images for female subjects while 92.2%–94.5% for male subjects, though it was trained on the integrated data set. The comparative classification of the developed model and ergonomic measurements for stress classification was carried out with a net accuracy of 88% where there were few instances of wrong classifications.