Introducing re-weighted range voting in clinical practice guideline prioritization: Development and testing of the re-weighted priority-setting (REPS) tool

Abstract

We aimed to develop and test a tool based on the re-weighted range voting mechanism to prioritize items (i.e. key questions) in a priority-setting assessment for clinical practice guidelines. The secondary aim was to provide methodological context of the tool. We iteratively developed the tool and used qualitative methods (i.e. think-aloud and semi-structured interviews) to test the tool’s usability and make adjustments accordingly. An observational approach was used to test the tool’s outcome satisfaction in a real-world priority-setting assessment within a rare-disease guideline of a European Reference Network and under four different conditions in the tool. Four guideline methodologists tested the usability of the tool. The real-world testing was performed with a guideline panel consisting of a core working group, five expertise working groups, and a working group with patient representatives. Thirty-one panel members assigned scores in the priority-setting assessment. Seventeen panel members rated the priority-setting outcome, and sixteen panel members rated the outputs generated under the four conditions. Upon initial use, guideline methodologists found the tool to be quite overwhelming. However, with some initial effort they were able to easily identify the tool’s structure. Based on observations and feedback, the tool was further refined and user guidance was developed. Guideline panel members expressed (high) satisfaction with the priority-setting outcome. They particularly preferred the condition when using mean subgroup scores as input or employing aggressive penalties in the weighting method to determine the outputs. The tool generates a ranked list of items and offers flexibility for different choices in priority-setting assessments as long as its input format requirements are met. Although it is not a consensus method, the tool assists in narrowing down a set of priority items. Additional steps in the priority-setting assessment can lead to a consensus being reached regarding the final outcome.