3D Camera and Single-Point Laser Sensor Integration for Apple Localization in Spindle-Type Orchard Systems

Abstract

Accurate localization of apples is the key factor that determines a successful harvesting cycle in the automation of apple harvesting for unmanned operations. In this regard, accurate depth sensing or positional information of apples is required for harvesting apples based on robotic systems, which is challenging in outdoor environments because of uneven light variations when using 3D cameras for the localization of apples. Therefore, this research attempted to overcome the effect of light variations for the 3D cameras during outdoor apple harvesting operations. Thus, integrated single-point laser sensors for the localization of apples using a state-of-the-art model, the EfficientDet object detection algorithm with an mAP@0.5 of 0.775 were used in this study. In the experiments, a RealSense D455f RGB-D camera was integrated with a single-point laser ranging sensor utilized to obtain precise apple localization coordinates for implementation in a harvesting robot. The single-point laser range sensor was attached to two servo motors capable of moving the center position of the detected apples based on the detection ID generated by the DeepSORT (online real-time tracking) algorithm. The experiments were conducted under indoor and outdoor conditions in a spindle-type apple orchard artificial architecture by mounting the combined sensor system behind a four-wheel tractor. The localization coordinates were compared between the RGB-D camera depth values and the combined sensor system under different light conditions. The results show that the root-mean-square error (RMSE) values of the RGB-D camera depth and integrated sensor mechanism varied from 3.91 to 8.36 cm and from 1.62 to 2.13 cm under 476~600 lx to 1023~1100 × 100 lx light conditions, respectively. The integrated sensor system can be used for an apple harvesting robotic manipulator with a positional accuracy of ±2 cm, except for some apples that were occluded due to leaves and branches. Further research will be carried out using changes in the position of the integrated system for recognition of the affected apples for harvesting operations.