A statistical approach to designing search filters to find systematic reviews: objectivity enhances accuracy

Abstract

Search filters are increasingly used to search medical databases to identify specific topics or study designs. In particular, search filters have been designed to help health-care professionals identify systematic reviews of the effectiveness of health interventions. Identifying systematic reviews in databases such as MEDLINE is problematic and research has previously been undertaken into methods to design search filters that retrieve systematic reviews effectively. The aim of this study was to improve previously developed methods to derive a more objective search strategy to identify systematic reviews in MEDLINE. A ‘quasi-gold standard’ collection of known systematic reviews was identified. A frequency analysis of words within a subset of the ‘quasi-gold standard’ was undertaken followed by a statistical analysis of the most frequently occurring words. This analysis determined which terms would best distinguish between systematic reviews, non-systematic reviews and non-reviews. The performance of the best models was tested on the remaining subset of ‘quasi-gold standard’ records and then using the OVID interface to MEDLINE. The best model had a sensitivity of 73.4% for systematic reviews in the test set and 84.2% when used with the validation set. The best model had a specificity of 98.3% in the test set and 93% in the validation set. When tested on the same ‘quasi-gold standard’ using OVID MED-LINE the model showed 100% sensitivity and 4.4% precision. The number of times a term occurs in a record adds discriminatory power to the search strategy. Apparently highly relevant terms chosen subjectively do not perform as well as those derived by a statistical approach. Some search terms may not immediately seem useful in identifying systematic reviews, but when used in combination with other terms they prove to be highly discriminating. The best performing filters were tested on the OVID interface, but without frequency and term weightings. Their performance was also compared to previously published filters. One of the strategies was found to perform better with respect to sensitivity than previously published filters, although with lower precision.