A Systematic Mapping of the Proposition of Benchmarks in the Software Testing and Debugging Domain
Author:
Silva-Junior Deuslirio da1ORCID, Graciano-Neto Valdemar V.1ORCID, de-Freitas Diogo M.1, Leitão-Junior Plinio de Sá1, Kassab Mohamad2ORCID
Affiliation:
1. Instituto de Informática, Universidade Federal de Goiás, Goiânia 74690-900, Goiás, Brazil 2. Engineering Division, The Pennsylvania State University, Malvern, PA 16801, USA
Abstract
Software testing and debugging are standard practices of software quality assurance since they enable the identification and correction of failures. Benchmarks have been used in that context as a group of programs to support the comparison of different techniques according to pre-established parameters. However, the reasons that inspire researchers to propose novel benchmarks are not fully understood. This article reports the investigation, identification, classification, and externalization of the state of the art about the proposition of benchmarks on software testing and debugging domains. The study was carried out using systematic mapping procedures according to the guidelines widely followed by software engineering literature. The search identified 1674 studies, from which, 25 were selected for analysis. A list of benchmarks is provided and descriptively mapped according to their characteristics, motivations, and scope of use for their creation. The lack of data to support the comparison between available and novel software testing and debugging techniques is the main motivation for the proposition of benchmarks. Advancements in the standardization and prescription of benchmark structure and composition are still required. Establishing such a standard could foster benchmark reuse, thereby saving time and effort in the engineering of benchmarks for software testing and debugging.
Funder
Coordenação de Aperfeiçoamento de Pessoal de Nível Superior-Brasil (CAPES)-Finance
Reference67 articles.
1. Pelliccione, P., Kobetski, A., Larsson, T., Aramrattana, M., Aderum, T., Agren, S.M., Jonsson, G., Heldal, R., Bergenhem, C., and Thorsén, A. (December, January 28). Architecting cars as constituents of a system of systems. Proceedings of the International Colloquium on Software-Intensive Systems-of-Systems at 10th European Conference on Software Architecture (SiSoSECSA’16), Copenhagen, Denmark. 2. A city-wide real-time traffic management system: Enabling crowdsensing in social internet of vehicles;Wang;IEEE Commun. Mag.,2018 3. Horita, F.E.A., Rhodes, D.H., Inocêncio, T.J., and Gonzales, G.R. (2019, January 20–24). Building a conceptual architecture and stakeholder map of a system-of-systems for disaster monitoring and early-warning: A case study in brazil. Proceedings of the XV Brazilian Symposium on Information Systems (SBSI’19), Aracaju, Brazil. 4. Cha, S., Taylor, R.N., and Kang, K. (2019). Software Testing, Springer International Publishing. 5. The real cost of software errors;Zhivich;IEEE Secur. Priv.,2009
|
|