Neighborhood Weighted Voting-Based Noise Correction for Crowdsourcing

Abstract

In crowdsourcing scenarios, we can obtain each instance’s multiple noisy labels set from different crowd workers and then use a ground truth inference algorithm to infer its integrated label. Despite the effectiveness of ground truth inference algorithms, a certain level of noise still remains in the integrated labels. To reduce the impact of noise, many noise correction algorithms have been proposed in recent years. To the best of our knowledge, however, nearly all existing noise correction algorithms only exploit each instance’s own multiple noisy label sets but ignore the multiple noisy label sets of its neighbors. Here neighbors refer to the nearest instances found in the feature space based on the distance metric learning. In this article, we propose neighborhood weighted voting-based noise correction (NWVNC). In NWVNC, we at first take advantage of the multiple noisy label sets of each instance’s neighbors (including itself) to estimate the probability that it belongs to its integrated label. Then, we use the estimated probability to identify and filter noise instances and thus obtain a clean set and a noise set. Finally, we train three heterogeneous classifiers on the clean set and correct the noise instances by the consensus voting of three trained classifiers. The experimental results on 34 simulated and two real-world crowdsourced datasets show that NWVNC significantly outperforms all the other state-of-the-art noise correction algorithms used for comparison.