Affiliation:
1. Department of Data Science, National Physical Laboratory (NPL), Hampton Road, Teddington TW11 0LW, UK
Abstract
We investigate the utility of side information in the context of machine learning and, in particular, in supervised neural networks. Side information can be viewed as expert knowledge, additional to the input, that may come from a knowledge base. Unlike other approaches, our formalism can be used by a machine learning algorithm not only during training but also during testing. Moreover, the proposed approach is flexible as it caters for different formats of side information, and we do not constrain the side information to be fed into the input layer of the network. A formalism is presented based on the difference between the neural network loss without and with side information, stating that it is useful when adding side information reduces the loss during the test phase. As a proof of concept we provide experimental results for two datasets, the MNIST dataset of handwritten digits and the House Price prediction dataset. For the experiments we used feedforward neural networks containing two hidden layers, as well as a softmax output layer. For both datasets, side information is shown to be useful in that it improves the classification accuracy significantly.
Funder
UK Government’s Department for Science, Innovation and Technology
Subject
Computational Mathematics,Computational Theory and Mathematics,Numerical Analysis,Theoretical Computer Science
Reference28 articles.
1. A new learning paradigm: Learning using privileged information;Vapnik;Neural Netw.,2009
2. Shekhar, S., and Akoglu, L. (2019, January 10–14). Incorporating privileged information to unsupervised anomaly detection. Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD), Dublin, Ireland.
3. Jonschkowski, R., Hoefer, S., and Brock, O. (2015). Patterns for learning with side information. arXiv.
4. Adel, T., Ghahramani, Z., and Weller, A. (2018, January 10–15). Discovering interpretable representations for both deep generative and discriminative models. Proceedings of the 35th International Conference on Machine Learning (ICML), Stockholm, Sweden.
5. Hasan, A., Levene, M., and Weston, D. (2020). Learning structured medical information from social media. J. Biomed. Inform., 110.