Statistical Error Propagation Affecting the Quality of Experience Evaluation in Video on Demand Applications

Abstract

In addition to the traditional Quality of Service (QoS) metrics of latency, jitter and Packet Loss Ratio (PLR), Quality of Experience (QoE) is now widely accepted as a numerical proxy for the actual user experience. The literature has reported many mathematical mappings between QoE and QoS, where the QoS parameters are measured by the network providers using sampling. Previous research has focussed on sampling errors in QoS measurements. However, the propagation of these sampling errors in QoS through to the QoE values has not been evaluated before. This is important: without knowing how sampling errors propagate through to QoE estimates there is no understanding of the precision of the estimates of QoE, only of the average QoE value. In this paper, we used industrially acquired measurements of PLR and jitter to evaluate the sampling errors. Additionally, we evaluated the correlation between these QoS measurements, as this correlation affects errors propagating to the estimated QoE. Focusing on Video-on-Demand (VoD) applications, we use subjective testing and regression to map QoE metrics onto PLR and jitter. The resulting mathematical functions, and the theory of error propagation, were used to evaluate the error propagated to QoE. This error in estimated QoE was represented as confidence interval width. Using the guidelines of UK government for sampling in a busy hour, our results indicate that confidence intervals around estimated the Mean Opinion Score (MOS) rating of QoE can be between MOS = 1 to MOS = 4 at targeted operating points of the QoS parameters. These results are a new perspective on QoE evaluation and are of potentially great significance to all organisations that need to estimate the QoE of VoD applications precisely.