Statistical time-series forecast error and bias assessment using hold-out samples and a single benchmark (data-reduction) scoring metric

Abstract

In Time-Series (TS) forecast modeling, we utilize a Hold-Out (HO) sample to assess a candidate model forecast error and bias, with the Mean Absolute Percentage Error (MAPE) as a single benchmark (data-reduction) for a scoring metric. The purpose of such HO sampling is to assess error rates and accuracy (unbiasedness) level whether the forecasted values of a candidate TS model is within a pre-specified (targeted) margin-of-error (MOE) rate of α-percent. For instance, for an accuracy expectation of 80% level, the MOE rate of α is tolerated to be, 20%. If the MAPE of a candidate TS model is less than the MOE rate of α, then the model is considered reasonably accurate enough. Otherwise, we select the next available candidate TS model with the smallest-possible MAPE. For an illustration, we apply such method to a TS sales-data example.