Scoring Probability Forecasts by a User’s Bets Against a Market Consensus

Abstract

The purpose of our paper is to describe a probability scoring rule that reflects the economic performance of a hypothetical investor who acts upon the probability forecasts emanating from a given model or human expert by trading against a market-clearing consensus of competing models and forecasts. The probability forecasts being compared are aggregated by an equilibrium condition into a market consensus reflecting the wisdom of the crowd. A good forecaster (model or human expert) is measured as one who allows the user to bet profitably against the market consensus. By asking forecasts to beat the market, forecasters are discouraged from herding and motivated to obtain better information than rival forecasters. We illustrate and prove that each trader’s personal incentive to hedge or fudge disappears when the number of forecasts in the market is sufficiently large. Our score exhibits the forecaster’s ability to assist economically profitable action and reveals how the user’s profits depend strongly on the accuracy of the forecasts and the decision rule (boldness) with which they are acted upon.