Predicting software reliability

Abstract

This paper surveys some aspects of the state of the art of software reliability modelling. By far the greatest effort to date has been expended on the problem of assessing and predicting the reliability growth which takes place as faults are found and fixed, so the greater part of the paper addresses this problem. We begin with a simple conceptual model of the software failure process in order to set the scene and motivate the detailed stochastic models which follow. This conceptual model suggests certain minimal characteristics which all growth models for software should possess. There are now several detailed models which aim to represent software reliability growth, but their accuracy of prediction seems to vary greatly from one application to another. As it is not possible to decide a priori which will give the most accurate answers for a particular context, the potential user is faced with a dilemma. There seems to be no alternative to analysing the predictive accuracy on the data source under examination and selecting for the current prediction that model which has demonstrated greatest accuracy on earlier predictions for that data. Some ways in which this selection can be effected are described in the paper. It turns out that examination of accuracy of past predictions can be used to improve future predictions by a simple recalibration procedure. Sometimes this technique works dramatically well, and results are shown for some real software failure data. Finally, there is a brief discussion of some wider issues which are not covered by a simple reliability growth study. These include cost modelling, the evaluation of software engineering methodologies, the relationship between testing and reliability, and the important issues of ultra-high reliability and safety-critical systems. On the last point, a warning note is sounded on the wisdom of building systems which depend on software having a very high reliability; this will be very hard to achieve and even harder to demonstrate.