Transformer Encoder-Based Deep Learning Model for Time Series Prediction

Abstract

Historical weather data shows time-sensitive patterns and exhibits seasonality and trends. Machine learning (ML) models can learn to identify these patterns and predict future events. Weather forecasting is a popular time series problem. Historically, RNNs like long short-term memory and gated recurrent unit addressed weather prediction. Yet, the vanishing gradient (VG) problem hindered long-term context connections in extended input sequences. The transformer addresses the VG problem and was initially designed for the natural language processing tasks, where it has proven highly successful. However, there have been few works to unfold the effectiveness of the transformer in regression-based problems. This study introduces a transformer encoder-based ML framework for weather prediction. Analysing historical data and augmenting the dataset with essential temporal information, the model enhances accuracy in forecasting air temperature one step ahead. The results of the experiments are compared to modern RNN-based models and with other similar works previously done in the same field.