Augmenting Field Data for Testing Systems Subject to Incremental Requirements Changes

Abstract

When testing data processing systems, software engineers often use real-world data to perform system-level testing. However, in the presence of new data requirements, software engineers may no longer benefit from having real-world data with which to perform testing. Typically, new test inputs complying with the new requirements have to be manually written. We propose an automated model-based approach that combines data modelling and constraint solving to modify existing field data to generate test inputs for testing new data requirements. The approach scales in the presence of complex and structured data, thanks to both the reuse of existing field data and the adoption of an innovative input generation algorithm based on slicing the model into parts. We validated the scalability and effectiveness of the proposed approach using an industrial case study. The empirical study shows that the approach scales in the presence of large amounts of structured and complex data. The approach can produce, within a reasonable time, test input data that is over ten times larger in size than the data generated with constraint solving only. We also demonstrate that the generated test inputs achieve more code coverage than the test cases implemented by experienced software engineers.