Affiliation:
1. Faculty of Information Technology, Brno University of Technology, Brno, Czech Republic
2. Kempelen Institute of Intelligent Technologies, Bratislava Slovakia
3. Slovak.AI, Bratislava Slovakia
Abstract
Learning with limited labelled data, such as prompting, in-context learning, fine-tuning, meta-learning or few-shot learning, aims to effectively train a model using only a small amount of labelled samples. However, these approaches have been observed to be excessively sensitive to the effects of uncontrolled randomness caused by non-determinism in the training process. The randomness negatively affects the stability of the models, leading to large variances in results across training runs. When such sensitivity is disregarded, it can unintentionally, but unfortunately also intentionally, create an imaginary perception of research progress. Recently, this area started to attract research attention and the number of relevant studies is continuously growing. In this survey, we provide a comprehensive overview of 415 papers addressing the effects of randomness on the stability of learning with limited labelled data. We distinguish between four main tasks addressed in the papers (investigate/evaluate; determine; mitigate; benchmark/compare/report randomness effects), providing findings for each one. Furthermore, we identify and discuss seven challenges and open problems together with possible directions to facilitate further research. The ultimate goal of this survey is to emphasise the importance of this growing research area, which so far has not received an appropriate level of attention, and reveal impactful directions for future research.
Publisher
Association for Computing Machinery (ACM)
Reference160 articles.
1. Rishabh Adiga Lakshminarayanan Subramanian and Varun Chandrasekaran. 2024. Designing Informative Metrics for Few-Shot Example Selection. arXiv preprint arXiv:2403.03861(2024).
2. Mayank Agarwal, Mikhail Yurochkin, and Yuekai Sun. 2021. On sensitivity of meta-learning to support data. In Advances in Neural Information Processing Systems, Vol. 34. Curran Associates, Inc., 20447–20460.
3. Anirudh Ajith Mengzhou Xia Ameet Deshpande and Karthik R Narasimhan. 2023. InstructEval: Systematic Evaluation of Instruction Selection Methods. In R0-FoMo:Robustness of Few-shot and Zero-shot Learning in Large Foundation Models. https://openreview.net/forum?id=6FwaSOEeKD
4. Riccardo Albertoni Sara Colantonio Piotr Skrzypczyński and Jerzy Stefanowski. 2023. Reproducibility of Machine Learning: Terminology Recommendations and Open Issues. arXiv preprint arXiv:2302.12691(2023).