Efficiency and comparability of using new evidence platforms for updating recommendations: experience with a diabetes guideline in Colombia

Abstract

Abstract Background: Updating recommendations of guidelines requires both comprehensive and efficient literature searches. While new information platforms are available for developing groups, their relative contribution to this purpose remains uncertain. Methods: As part of a review/update of 8 selected evidence-based recommendations (EBR) on type 2 diabetes, we evaluated the following five literature search approaches (targeting systematic reviews, using predetermined criteria): PubMed for MEDLINE; Epistemonikos database basic search; Epistemonikos database, using a structured search strategy; Living overview of evidence (L.OVE) platform, and TRIP database. Three reviewers independently classified retrieved references as definitely or probably eligible/not eligible. Those falling in the same “definitely” categories for all reviewers were labelled as “true” positives/negatives. The rest went to re-assessment and if found eligible/not eligible by consensus became “false” negatives/positives, respectively. We described the yield for each approach, computed their “diagnostic accuracy” measures and agreement statistics. Results: Altogether, the five approaches identified 318-505 references for the 8 recommendations, from which reviewers considered 4.2-9.4% eligible after the two rounds. While Pubmed outperformed the other approaches (diagnostic odds ratio 12.5 versus 2.6-5.3), no single search approach returned eligible references for all recommendations. Individually, searches identified up to 40% of all eligible references (n=71), and no combination of any three approaches could identify over 80% of them. Kappa statistics for retrieval between searches were very poor (9 out of 10 paired comparisons did not surpass chance-expected agreement). Conclusion: Among the information platforms assessed, Pubmed seemed the more efficient to update this set of recommendations. However, the very poor agreement in the reference yield demands developing groups to add information from several (probably over 3) sources for this purpose. Further research is needed to replicate our findings and enhance our understanding of how to update recommendations efficiently.