Fruit Detection and Segmentation for Apple Harvesting Using Visual Sensor in Orchards

Abstract

Autonomous harvesting shows a promising prospect in the future development of theagriculture industry, while the vision system is one of the most challenging components in theautonomous harvesting technologies. This work proposes a multi-function network to perform thereal-time detection and semantic segmentation of apples and branches in orchard environments byusing the visual sensor. The developed detection and segmentation network utilises the atrous spatialpyramid pooling and the gate feature pyramid network to enhance feature extraction ability of thenetwork. To improve the real-time computation performance of the network model, a lightweightbackbone network based on the residual network architecture is developed. From the experimentalresults, the detection and segmentation network with ResNet-101 backbone outperformed on thedetection and segmentation tasks, achieving an F1 score of 0.832 on the detection of apples and 87.6%and 77.2% on the semantic segmentation of apples and branches, respectively. The network modelwith lightweight backbone showed the best computation efficiency in the results. It achieved an F1score of 0.827 on the detection of apples and 86.5% and 75.7% on the segmentation of apples andbranches, respectively. The weights size and computation time of the network model with lightweightbackbone were 12.8 M and 32 ms, respectively. The experimental results show that the detection andsegmentation network can effectively perform the real-time detection and segmentation of applesand branches in orchards.