Mutta: a novel tool for E2E web mutation testing

Abstract

AbstractMutation testing is an important technique able to evaluate the bug-detection effectiveness of existing software test suites. Mutation testing tools exist for several languages, e.g., Java and JavaScript, but no solutions are available for managing the mutation testing process for entire web applications, in the context of end-to-end (E2E) web testing. In this paper, we propose Mutta, a novel tool able to automate the entire mutation testing process. Mutta mutates the various server source files of the target web application, runs the E2E test suite against the mutated web applications, and finally collects the test outcomes. To evaluate Mutta, we designed a case study using the mutated versions of the target web application with the aim of comparing the effectiveness of two different approaches to E2E web testing: (1) test cases based on classical assertions and (2) test cases relying on differential testing. In detail, Mutta has been executed on two web applications, each equipped with different test suites to compare assertions with differential testing. In this scenario, Mutta generated a large number of mutants (more than 15k overall), took into account the coverage information to consider only the mutants actually executed, deployed the mutated web app, ran the entire E2E test suites (about 87k tests runs overall), and finally, it correctly saved the test suite results. Thus, results of the case study show that Mutta can be successfully employed to automate the entire mutation testing process of E2E web test suites and, therefore, can be used in practice to evaluate the effectiveness of different test suites (e.g., based on different techniques, E2E frameworks, or composed by a different number of test scripts).