Affiliation:
1. University of Electronic Science and Technology of China, Chengdu China and Loughborough University, United Kingdom
2. University of Birmingham, Birmingham, United Kingdom
Abstract
In presence of multiple objectives to be optimized in Search-Based Software Engineering (SBSE), Pareto search has been commonly adopted. It searches for a good approximation of the problem’s Pareto-optimal solutions, from which the stakeholders choose the most preferred solution according to their preferences. However, when clear preferences of the stakeholders (e.g., a set of weights that reflect relative importance between objectives) are available prior to the search, weighted search is believed to be the first choice, since it simplifies the search via converting the original multi-objective problem into a single-objective one and enables the search to focus on what only the stakeholders are interested in.
This article questions such a “
weighted search first
” belief. We show that the weights can, in fact, be harmful to the search process even in the presence of clear preferences. Specifically, we conduct a large-scale empirical study that consists of 38 systems/projects from three representative SBSE problems, together with two types of search budget and nine sets of weights, leading to 604 cases of comparisons. Our key finding is that weighted search reaches a certain level of solution quality by consuming relatively less resources at the early stage of the search; however, Pareto search is significantly better than its weighted counterpart the majority of the time (up to 77% of the cases), as long as we allow a sufficient, but not unrealistic search budget. This is a beneficial result, as it discovers a potentially new “rule-of-thumb” for the SBSE community: Even when clear preferences are available, it is recommended to always consider Pareto search by default for multi-objective SBSE problems, provided that solution quality is more important. Weighted search, in contrast, should only be preferred when the resource/search budget is limited, especially for expensive SBSE problems. This, together with other findings and actionable suggestions in the article, allows us to codify pragmatic and comprehensive guidance on choosing weighted and Pareto search for SBSE under the circumstance that clear preferences are available. All code and data can be accessed at
https://github.com/ideas-labo/pareto-vs-weight-for-sbse
.
Publisher
Association for Computing Machinery (ACM)
Reference109 articles.
1. Vahid Alizadeh, Houcem Fehri, and Marouane Kessentini. 2019. Less is more: From multi-objective to mono-objective refactoring via developer’s knowledge extraction. In Proceedings of the 19th International Working Conference on Source Code Analysis and Manipulation. IEEE, 181–192.
2. Search-based techniques applied to optimization of project planning for a massive maintenance project
3. Allysson Allex Araújo and Matheus Paixão. 2014. Machine learning for user modeling in an interactive genetic algorithm for the next release problem. In Proceedings of the 6th International Symposium on Search-based Software Engineering. Springer, 228–233. DOI:10.1007/978-3-319-09940-8_17
4. Andrea Arcuri and Lionel C. Briand. 2011. A practical guide for using statistical tests to assess randomized algorithms in software engineering. In Proceedings of the 33rd International Conference on Software Engineering. 1–10.
5. The next release problem
Cited by
10 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献