Employing log metrics to evaluate search behaviour and success: case study BBC search engine

Abstract

This paper argues that metrics can be generated from search transactional web logs that can help evaluate search engine effectiveness. Search logs from the BBC website were analysed and metrics extracted. Two search metrics — the time lapse between searches and the number of searches in a session — were developed to see whether they could measure search success or satisfaction. In all, 4 million search statements by 900,000 users were evaluated. The BBC search engine possessed a number of functional attributes which sought to improve retrieval and these were subjected to the two metrics to help determine how successful they were in practice. There was some evidence to support the proposition that the search outcome metrics did indeed indicate the effectiveness of engine functionality. The authors argue that this result is significant in that the identification of search outcome metrics will pave the way for assessing the effectiveness of site specific search engines and a greater understanding of the effectiveness of search engine functionality.