Metastable Decomposition of High-Dimensional Meteorological Data with Gaps

Abstract

Abstract This paper presents an extension of the recently developed method for simultaneous dimension reduction and metastability analysis of high-dimensional time series. The modified approach is based on a combination of ensembles of hidden Markov models (HMMs) with state-specific principal component analysis (PCA) in extended space (guaranteeing that the overall dynamics will be Markovian). The main advantage of the modified method is its ability to deal with the gaps in the high-dimensional observation data. The proposed method allows for (i) the separation of the data according to the metastable states, (ii) a hierarchical decomposition of these sets into metastable substates, and (iii) calculation of the state-specific extended empirical orthogonal functions simultaneously with identification of the underlying Markovian dynamics switching between those metastable substates. The authors discuss the introduced model assumptions, explain how the quality of the resulting reduced representation can be assessed, and show what kind of additional insight into the underlying dynamics such a reduced Markovian representation can give (e.g., in the form of transition probabilities, statistical weights, mean first exit times, and mean first passage times). The performance of the new method analyzing 500-hPa geopotential height fields [daily mean values from the 40-yr ECMWF Re-Analysis (ERA-40) dataset for a period of 44 winters] is demonstrated and the results are compared with information gained from a numerically expensive but assumption-free method (Wavelets–PCA), and the identified metastable states are interpreted w.r.t. the blocking events in the atmosphere.