Statistical inference for unreliable grading using the maximum entropy principle

Abstract

Quantitatively assessing the level of confidence on a test score can be a challenging problem, especially when the available information is based on multiple criteria. A concrete example beyond the usual grading of tests occurs with recommendation letters, where a recommender assigns a score to a candidate, but the reliability of the recommender must be assessed as well. Here, we present a statistical procedure, based on Bayesian inference and Jaynes’ maximum entropy principle, that can be used to estimate the most probable and expected score given the available information in the form of a credible interval. Our results may provide insights on how to properly state and analyze problems related to the uncertain evaluation of performance in learning applied to several contexts, beyond the case study of the recommendation letters presented here.