STATISTICAL COMPARISON OF MODELLING METHODS FOR SOFTWARE MAINTAINABILITY PREDICTION

Abstract

The objective of this paper is statistical comparison of modelling methods for software maintainability prediction. The statistical comparison is performed by building software maintainability prediction models using 27 different regression and machine learning based algorithms. For this purpose, software metrics datasets of two different commercial object-oriented systems are used. These systems were developed using an object oriented programming language Ada. These systems are User Interface Management System (UIMS) and Quality Evaluation System (QUES). It is shown that different measures like MMRE, RMSE, Pred(0.25) and Pred(0.30) calculated on predicted values obtained from leave one out (LOO) cross validation produce very divergent results regarding accuracy of modelling methods. Therefore the 27 modelling methods are evaluated on the basis of statistical significance tests. The Friedman test is used to rank various modelling methods in terms of absolute residual error. Six out of the ten top ranked modelling methods are common to both UIMS and QUES. This indicates that modelling methods for software maintainability predicton are solid and scalable. After obtaining ranks, pair wise Wilcoxon Signed rank test is performed. Wilcoxon Sign rank test indicates that the top ranking method in UIMS data set is significantly superior to only four other modelling methods whereas the top tanking method in QUES data set is significantly superior to 11 other modelling methods. The performance of instance based learning algorithms — IBk and Kstar is comparable to modelling methods used in earlier studies.