A Study on Affine Transformations and a Novel Universal Prediction Theory

Abstract

Artificial neural networks (NNs) with various architectures are widely used for practical problems, including multilayer perceptron (MLP), the Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), among others. The TrumpetNets and TubeNets were also recently proposed for creating two-way deepnets for both forward and inverse problems. All these studies have demonstrated the power of NNs, but the theoretical foundation needs a more thorough study. This paper first conducts an intensive study on affine transformations and its role as a prediction function. This reveals a data-to-parameter (or [Formula: see text]) encoding mechanism and its uniqueness with proof. Then, effects of the architecture and the roles of nonlinear activation functions wrapping the affine transformations are investigated. The effects of stacking of affine transformations and chaining of the wrapped affine transformations in an NN are examined, leading finally to a novel Universal Prediction Theory (UPT).