A New Method for the Evaluation and Visualization of Air Pollutant Level Predictions

Abstract

Accurately predicting air pollutant levels is very important for mitigating their effects. Prediction models usually fail to predict sudden large increases or decreases in pollutant levels. Conventional measures for the assessment of the performance of air pollutant prediction models provide an overall assessment of model behavior, but do not explicitly address model behavior when large changes are observed. In our work, we propose a method to automatically label the observed large changes. We also propose two visualization methods and two measures that can help assess model performance when sudden large changes in pollutant levels occur. The developed measures enable the assessment of model performance only for large changes (MAE of large changes), or weigh the model residuals by the rate of change (WErr), making the evaluation measures “cost-sensitive”. To show the value of the novel evaluation and visualization methods, we employ them in the evaluation of three empirical examples—different statistical models used in real-life settings and a popular atmospheric dispersion model. The proposed visualizations and measures can be a valuable complement to conventional model assessment measures when the prediction of large changes is as important as (even if they are rare) or more important than predictions of other levels.