MEASURES OF LANGUAGE EFFECTIVENESS AND THE SWETSIAN HYPOTHESES

Abstract

‘Language measures’ such as Swets's E or Brookes's S, which measure the separation of the PMFs defined by a weighting formula applied to the sets of relevant and non‐relevant documents, are different in kind to probabilistic ‘system measures’ such as Precision (P) and Recall (R). For a given query and collection the subset of {P} × {R} defined by varying the threshold is unaffected by monotonic transformations of the weighting formula used. If S is redefined so as to relate only to ranked probabilities it will retain its value, and so reflect this constancy in the graph, under such transformations. S, as redefined, is also an (approximate) indicator of a rule to retrieve documents with weights greater [less] than the threshold when S is positive [negative]. Language measures can also be used to determine the retrieval algorithm when multivariate weights are used.