The REVELANCE OF RELEVANCE TO The TESTING AND EVALUATION OF DOCUMENT RETRIEVAL SYSTEMS

Abstract

There is some evidence of a slackening of interest in the testing and evaluation of retrieval systems. The estimated significance of this problem also appears to have decreased markedly. A literature search reveals the existence of less than fifty papers published to date in 1966, most of which do not report experimental activity. This dearth of significant effort may be due to a number of factors: investigators engaged in testing and evaluation have exhausted their repertoire; difficulty in undertaking further effort and at the same time satisfying more stringent experimental design requirements; realization that there never was any theoretical basis to testing and evaluation in the first place and that some basic‐type research is necessary before beginning any new testing activity. It is disappointing to consider how little the work on evaluation has contributed to the improved design and optimization of information retrieval systems. Little agreement exists either with respect to the nature of the phenomena which we wish to subject to quantification (what we are testing) or as to the requisite methodology (bow we are to test).