Assessing the Inter-rater Reliability of the Usability Model for Software Development Processes and Practices

Abstract

Abstract Software processes and practices have a leading role in software development and in the last few decades a wide variety of processes and practices have emerged to face the challenges arising in the software industry. The success of process and practice adoption initiatives depends at least partially on the experience and satisfaction of the people who use them, thus making usability an interesting process quality attribute. This paper describes the inter-rater reliability evaluations performed on the UMP to assess consistency among metric values produced by different evaluators. It presents two inter-rater reliability assessment studies, the Scrum study and the TDD-BDD study. The paper presents four inter-rater reliability statistics for the process and practices under study, a comparative analysis of their strengths and weaknesses, and an analysis of the study results and their interpretation. The results show varying reliability results among the different UMP metrics, which seem to depend mostly on object of evaluation (Scrum, TDD, BDD), metric subjectivity and context sensitivity. The UMP was refined from the data gathered in the studies and a metric categorization is proposed to facilitate metric usage.