Online Testing of Nondeterministic Systems with the Reactive Planning Tester

Abstract

We describe a model-based construction of an online tester for black-box testing. Contemporary model-based online test generators focusing mainly on computationally cheap but far from optimal planning strategies cover just a fraction of the wide spectrum of test control strategies. Typical examples of those used are simple random choice and anti-ant. Exhaustive planning during online testing of nondeterministic systems looks out of reach because of the low scalability of the methods in regard to the model size. The reactive planning tester (RPT) studied in this chapter is targeted to fill the gap between these two extremes. The key idea of RPT lies in offline static analysis of the IUT (implementation under test) model to prepare the data and constraints for efficient online reactive planning. The external behavior of the IUT is modelled as an output observable nondeterministic EFSM (extended finite state machine) with the assumption that all the transition paths are feasible. A test purpose is attributed to the transitions of the IUT model by a set of Boolean variables called traps that are used to measure the progress of the test run. We present a way to construct a tester that at runtime selects a suboptimal test path from trap to trap by finding the shortest path that covers unvisited traps within planning horizon. The principles of reactive planning are implemented in the form of the decision rules of selecting the shortest paths at runtime. Based on an industrial scale case study, namely the city lighting system controller, we demonstrate the practical use of the RPT for systems with high degree of nondeterminism, deep nested control loops, and requiring strictly bounded tester response time. Tuning the planning horizon of the RPT allows a trade-off to be found between close to optimal test length and scalability of tester behavior with computationally feasible expenses.