Search Budget in Multi-Objective Refactoring Optimization: a Model-Based Empirical Study

Software model optimization is the task of automatically generate design alternatives, usually to improve quality aspects of software that are quantifiable, like performance and reliability. In this context, multi-objective optimization techniques have been applied to help the designer find suitable trade-offs among conflicting non-functional properties. In this process, design alternatives can be generated through automated model refactoring, and evaluated on non-functional models. Due to their complexity, this type of optimization tasks require considerable time and resources, often limiting their application in software engineering processes.
In this paper, we investigate the effects of using a search budget, specifically a time limit, to the search for new solutions. We performed experiments to quantify the impact that a change in the search budget may have on the quality of solutions. Furthermore, we analyzed how different genetic algorithms (i.e., NSGA-II, SPEA2, and PESA2) perform when imposing different budgets. We experimented on two case studies of different size, complexity, and domain.
We observed that imposing a search budget considerably deteriorates the quality of the generated solutions, but the specific algorithm we choose seems to play a crucial role. From our experiments, NSGA-II is the fastest algorithm, while PESA2 generates solutions with the highest quality. Differently, SPEA2 is the slowest algorithm, and produces the solutions with the lowest quality.

Transcript

1. Search Budget in Multi-Objective Refactoring Optimization: a Model-Based Empirical Study Daniele Di Pompeoα Michele Tucciβ daniele.dipompeo@univaq.itα, tucci@d3s.mff.cuni.czβ α University of L’Aquila, Italy β Charles University, Czech Republic MDEML @ SEAA 2022, 1st September 2022 D. Di Pompeo Search Budget in Multi-Objective Refactoring Optimization: a Model-Based Empirical Study 1 / 22
2. Motivation Where Software model optimization is the task of automatically generate design alternatives [1]a, usually to improve quality aspects of software that are quantifiable, like performance and reliability. a[1] Cortellessa and Di Pompeo. “Analyzing the sensitivity of multi-objective software architecture refactoring to configuration characteristics”. 2021. When M-Obj optimization techniques have been applied to help the designer find suitable trade-offs among several non-functional properties (e.g., performance and reliability) [2]a. a[2] Cortellessa et al. “On the impact of Performance Antipatterns in multi-objective software model refactoring optimization”. 2021. Why Due to their complexity, M-Obj optimization requires considerable time and resources D. Di Pompeo Search Budget in Multi-Objective Refactoring Optimization: a Model-Based Empirical Study 2 / 22
3. Motivation Where Software model optimization is the task of automatically generate design alternatives [1]a, usually to improve quality aspects of software that are quantifiable, like performance and reliability. a[1] Cortellessa and Di Pompeo. “Analyzing the sensitivity of multi-objective software architecture refactoring to configuration characteristics”. 2021. When M-Obj optimization techniques have been applied to help the designer find suitable trade-offs among several non-functional properties (e.g., performance and reliability) [2]a. a[2] Cortellessa et al. “On the impact of Performance Antipatterns in multi-objective software model refactoring optimization”. 2021. Why Due to their complexity, M-Obj optimization requires considerable time and resources D. Di Pompeo Search Budget in Multi-Objective Refactoring Optimization: a Model-Based Empirical Study 2 / 22
4. Motivation Where Software model optimization is the task of automatically generate design alternatives [1]a, usually to improve quality aspects of software that are quantifiable, like performance and reliability. a[1] Cortellessa and Di Pompeo. “Analyzing the sensitivity of multi-objective software architecture refactoring to configuration characteristics”. 2021. When M-Obj optimization techniques have been applied to help the designer find suitable trade-offs among several non-functional properties (e.g., performance and reliability) [2]a. a[2] Cortellessa et al. “On the impact of Performance Antipatterns in multi-objective software model refactoring optimization”. 2021. Why Due to their complexity, M-Obj optimization requires considerable time and resources D. Di Pompeo Search Budget in Multi-Objective Refactoring Optimization: a Model-Based Empirical Study 2 / 22
5. Related Work In the last decade, software model multi-objective optimization studies have been introduced to optimize various quality attributes [3, 4, 5, 6, 1]1 with different degrees of freedom in the modification of models (e.g., service selection [7]2). Recent work compares the ability of two different multi-objective optimization approaches to improve non-functional attributes [8, 9]3 within a specific modeling notation (i.e., Palladio Component Model [10]). Our work relies on UML models and considers more complex refactoring actions, as well as different target attributes for the fitness function. 1[3] Martens et al. “Automatically improve software architecture models for performance, reliability, and cost using evolutionary algorithms”. 2010. 2[7] Cardellini et al. “QoS-driven Runtime Adaptation of Service Oriented Architectures”. 2009. 3[8] Ni et al. “Multi-objective software performance optimisation at the architecture level using randomised search rules”. 2021. D. Di Pompeo Search Budget in Multi-Objective Refactoring Optimization: a Model-Based Empirical Study 3 / 22
6. Approach at a glance Genetic Algorithm PerfQ Reliability #changes #pas Model Alternatives Refactoring Initial UML Model Refactoring Actions Pareto frontier D. Di Pompeo Search Budget in Multi-Objective Refactoring Optimization: a Model-Based Empirical Study 4 / 22
7. Approach at a glance Initial UML Model UML + DAM-MARTE Software Models Objectives #pas: min, #changes: min, reliability: max, perfQ: max Genetic Algorithms NSGA-II/ SPEA2/ PESA2 selected on the basis of their different searching policies. Genetic Algorithm PerfQ Reliability #changes #pas Model Alternatives Refactoring Initial UML Model Refactoring Actions Pareto frontier D. Di Pompeo Search Budget in Multi-Objective Refactoring Optimization: a Model-Based Empirical Study 5 / 22
8. Study Design Goal Establish how much the imposition of a time-based search budget can hamper the quality of computed Pareto fronts (PFc) Target We run the optimization 31 times for each search budget (15, 30, and 60 minutes). We compute the HyperVolume for each PFc [11]. We apply the Mann–Whitney U non-parametric statistical test [12] with the null hypothesis that the experiments do not show a statistically significant difference [13]. D. Di Pompeo Search Budget in Multi-Objective Refactoring Optimization: a Model-Based Empirical Study 6 / 22
9. Study Design Goal Establish how much the imposition of a time-based search budget can hamper the quality of computed Pareto fronts (PFc) Target We run the optimization 31 times for each search budget (15, 30, and 60 minutes). We compute the HyperVolume for each PFc [11]. We apply the Mann–Whitney U non-parametric statistical test [12] with the null hypothesis that the experiments do not show a statistically significant difference [13]. D. Di Pompeo Search Budget in Multi-Objective Refactoring Optimization: a Model-Based Empirical Study 6 / 22
10. Case Study We applied our optimization approach to two case studies from the literature:4 Train Ticket Booking Service [14]: is a web-based booking application, whose architecture is based on the microservice paradigm. The system is made up of 40 microservices, and it provides different scenarios through users that can perform realistic operations, e.g., book a ticket or watch trip information. CoCoME [15]: describes a trading system containing several stores. A store can have one or more cash desks for processing goods. A cash desk is equipped with all the tools needed to serve a customer (e.g., a Cash Box, Printer, Bar Code Scanner). Covers possible scenarios performed at a cash desk (e.g., scanning products, paying by credit card, or ordering new goodies). 4https://github.com/SEALABQualityGroup/uml2lqn-casestudies D. Di Pompeo Search Budget in Multi-Objective Refactoring Optimization: a Model-Based Empirical Study 7 / 22
11. Research Questions RQ1 : To what extent does the time budget penalize the quality of Pareto fronts? RQ2 : Which algorithm performs better when limited by a time budget? D. Di Pompeo Search Budget in Multi-Objective Refactoring Optimization: a Model-Based Empirical Study 8 / 22
12. HyperVolume (HV) Establishing the quality of a computed Pareto front is a NP-hard problem [16]5. HV (0 ≤ HV ≤ 1) measures the amount of the volume of the solution space that a Pareto front (PFc) covers with respect to a reference Pareto front (PFref ) If HV = 0, the PFc is fully dominated by the PFref If HV = 1 each point within the PFc is non-dominated by any points within the PFref . We use the HV, since it has been proved to be a valid estimator for Pareto fronts comparison [17]6. 5[16] Ali et al. “Quality Indicators in Search-based Software Engineering: An Empirical Evaluation”. 2020. 6[17] Falc´ on-Cardona and Coello. “Indicator-based Multi-objective Evolutionary Algorithms: A Comprehensive Survey”. 2021. D. Di Pompeo Search Budget in Multi-Objective Refactoring Optimization: a Model-Based Empirical Study 9 / 22
13. RQ1 : To what extent does the time budget penalize the quality of Pareto fronts? Algor. Budget HV avg HV stdev TTBS NSGA-II 15 min 0.3060 0.0915 NSGA-II 30 min 0.3469 0.1071 NSGA-II 60 min 0.3437 0.0980 PESA2 15 min 0.3532 0.0794 PESA2 30 min 0.4084 0.0757 PESA2 60 min 0.4182 0.0819 SPEA2 15 min 0.3041 0.0794 SPEA2 30 min 0.2917 0.0920 SPEA2 60 min 0.2868 0.0769 Algor. Budget HV avg HV stdev CoCoME NSGA-II 15 min 0.0931 0.0335 NSGA-II 30 min 0.1199 0.0523 NSGA-II 60 min 0.1125 0.0604 PESA2 15 min 0.1363 0.0277 PESA2 30 min 0.1460 0.0300 PESA2 60 min 0.1514 0.0366 SPEA2 15 min 0.1189 0.0336 SPEA2 30 min 0.1098 0.0309 SPEA2 60 min 0.1023 0.0384 D. Di Pompeo Search Budget in Multi-Objective Refactoring Optimization: a Model-Based Empirical Study 10 / 22
14. RQ1 : To what extent does the time budget penalize the quality of Pareto fronts? Mann–Whitney U non-parametric statistical test Algor. Budget 1 Budget 2 MWU p ˆ A12 CoCoME NSGA-II 15 min 30 min 0.0574 (S) 0.359 NSGA-II 60 min 15 min 0.1054 (S) 0.6202 NSGA-II 60 min 30 min 0.8769 (N) 0.488 PESA2 15 min 30 min <0.0001 (L) 0.2092 PESA2 60 min 15 min <0.0001 (L) 0.7992 PESA2 60 min 30 min 0.6024 (N) 0.539 SPEA2 30 min 15 min 0.2483 (S) 0.4142 SPEA2 60 min 15 min 0.1249 (S) 0.3861 SPEA2 60 min 30 min 0.6123 (N) 0.462 D. Di Pompeo Search Budget in Multi-Objective Refactoring Optimization: a Model-Based Empirical Study 11 / 22
15. RQ2 : Which algorithm performs better when limited by a time budget? Mann–Whitney U non-parametric statistical test Budget Algor. 1 Algor. 2 MWU p ˆ A12 TTBS 15 min PESA2 NSGA-II 0.0487 (S) 0.6462 15 min SPEA2 NSGA-II 0.8548 (N) 0.486 15 min SPEA2 PESA2 0.0234 (M) 0.3319 30 min NSGA-II PESA2 0.0167 (M) 0.3226 30 min SPEA2 NSGA-II 0.0385 (S) 0.3465 30 min SPEA2 PESA2 <0.0001 (L) 0.1582 60 min NSGA-II PESA2 0.0037 (M) 0.2851 60 min SPEA2 NSGA-II 0.0202 (M) 0.3278 60 min SPEA2 PESA2 <0.0001 (L) 0.1301 D. Di Pompeo Search Budget in Multi-Objective Refactoring Optimization: a Model-Based Empirical Study 12 / 22
16. RQ2 : Which algorithm performs better when limited by a time budget? Mann–Whitney U non-parametric statistical test Budget Algor. 1 Algor. 2 MWU p ˆ A12 CoCoME 15 min NSGA-II SPEA2 0.0085 (M) 0.3049 15 min PESA2 NSGA-II 0.0072 (M) 0.6993 15 min PESA2 SPEA2 0.7999 (N) 0.4807 30 min PESA2 NSGA-II 0.0066 (M) 0.7014 30 min SPEA2 NSGA-II 0.5543 (N) 0.4558 30 min SPEA2 PESA2 <0.0001 (L) 0.1738 60 min PESA2 NSGA-II 0.0127 (M) 0.6847 60 min SPEA2 NSGA-II 0.3789 (N) 0.4344 60 min SPEA2 PESA2 <0.0001 (L) 0.1686 D. Di Pompeo Search Budget in Multi-Objective Refactoring Optimization: a Model-Based Empirical Study 12 / 22
17. RQ2 : Which algorithm performs better when limited by a time budget? Number of evolution performed and the HV TTBS CoCoME 4 6 8 10 12 14 16 18 20 Average number of performed evolutions 0.0 0.1 0.2 0.3 0.4 0.5 Average Hypervolume NSGAII 15 min NSGAII 30 min NSGAII 60 min PESA2 15 min PESA2 30 min PESA2 60 min SPEA2 15 min SPEA2 30 min SPEA2 60 min 4 6 8 10 12 14 16 18 Average number of performed evolutions 0.000 0.025 0.050 0.075 0.100 0.125 0.150 0.175 Average Hypervolume NSGAII 15 min NSGAII 30 min NSGAII 60 min PESA2 15 min PESA2 30 min PESA2 60 min SPEA2 15 min SPEA2 30 min SPEA2 60 min D. Di Pompeo Search Budget in Multi-Objective Refactoring Optimization: a Model-Based Empirical Study 13 / 22
18. RQ2 : Which algorithm performs better when limited by a time budget? Number of evolution performed and the HV TTBS CoCoME 4 6 8 10 12 14 16 18 20 Average number of performed evolutions 0.0 0.1 0.2 0.3 0.4 0.5 Average Hypervolume NSGAII 15 min NSGAII 30 min NSGAII 60 min PESA2 15 min PESA2 30 min PESA2 60 min SPEA2 15 min SPEA2 30 min SPEA2 60 min 4 6 8 10 12 14 16 18 Average number of performed evolutions 0.000 0.025 0.050 0.075 0.100 0.125 0.150 0.175 Average Hypervolume NSGAII 15 min NSGAII 30 min NSGAII 60 min PESA2 15 min PESA2 30 min PESA2 60 min SPEA2 15 min SPEA2 30 min SPEA2 60 min D. Di Pompeo Search Budget in Multi-Objective Refactoring Optimization: a Model-Based Empirical Study 13 / 22
19. RQ2 : Which algorithm performs better when limited by a time budget? Actual quality of PFc with 60 min of budget TTBS 0.00 0.05 0.10 0.15 0.20 0.25 PerfQ 0.0 0.2 0.4 0.6 0.8 1.0 Reliability initial Algorithm NSGAII PESA2 SPEA2 CoCoME 0.1 0.0 0.1 0.2 0.3 0.4 PerfQ 0.0 0.2 0.4 0.6 0.8 1.0 Reliability initial Algorithm NSGAII PESA2 SPEA2 D. Di Pompeo Search Budget in Multi-Objective Refactoring Optimization: a Model-Based Empirical Study 14 / 22
20. Result at a glance We establish a clear difference among the algorithms when comparing them on the basis of the HV We observe different speed in completing evolutions There is not much difference in the shape of the PFc and in the explored design space when considering non-functional properties only (performance and reliability) Our results showed: NSGA-II is the fastest algorithm because it generated the highest number of evolution genetic within the search budget. PESA2 is the algorithm that generated the highest quality results in terms of HV. SPEA2 is the slowest algorithm, and it generated the worst quality results. D. Di Pompeo Search Budget in Multi-Objective Refactoring Optimization: a Model-Based Empirical Study 15 / 22
21. Conclusion In this study we presented an investigation on the impact of the search budget for model-based multi-objective refactoring optimization. We validated the study on two model benchmarks, Train Ticket Booking Service, and CoCoME, and on three genetic algorithms, NSGA-II, SPEA2, and PESA2. We assessed the overall quality of each algorithm through the Hypervolume indicator. D. Di Pompeo Search Budget in Multi-Objective Refactoring Optimization: a Model-Based Empirical Study 16 / 22
22. Any Questions? Search Budget in Multi-Objective Refactoring Optimization: a Model-Based Empirical Study Daniele Di Pompeoα Michele Tucciβ daniele.dipompeo@univaq.itα, tucci@d3s.mff.cuni.czβ α University of L’Aquila, Italy β Charles University, Czech Republic MDEML @ SEAA 2022, 1st September 2022 D. Di Pompeo Search Budget in Multi-Objective Refactoring Optimization: a Model-Based Empirical Study 17 / 22
23. References I [1] Vittorio Cortellessa and Daniele Di Pompeo. “Analyzing the sensitivity of multi-objective software architecture refactoring to configuration characteristics”. In: JIST 135 (2021), p. 106568. issn: 0950-5849. doi: https://doi.org/https://doi.org/10.1016/j.infsof.2021.106568https: //doi.org/10.1016/j.infsof.2021.106568. [2] Vittorio Cortellessa et al. “On the impact of Performance Antipatterns in multi-objective software model refactoring optimization”. In: SEAA. 2021, pp. 224–233. isbn: 9781665427050. url: https://ieeexplore.ieee.org/document/9582578/. [3] Anne Martens et al. “Automatically improve software architecture models for performance, reliability, and cost using evolutionary algorithms”. In: ICPE 2010 - Proceedings of the 1st ACM/SPEC International Conference on Performance Engineering. New York, New York, USA: ACM Press, 2010, pp. 105–116. D. Di Pompeo Search Budget in Multi-Objective Refactoring Optimization: a Model-Based Empirical Study 18 / 22
24. References II [4] Rui Li et al. “An evolutionary multiobjective optimization approach to component-based software architecture design”. In: CEC. IEEE, 2011, pp. 432–439. [5] Indika Meedeniya et al. “Architecture-Driven Reliability and Energy Optimization for Complex Embedded Systems”. In: QoSA. Springer, 2010, pp. 52–67. [6] Anne Martens et al. “A Hybrid Approach for Multi-attribute QoS Optimisation in Component Based Software Systems”. In: Research into Practice – Reality and Gaps. 2010, pp. 84–101. [7] Valeria Cardellini et al. “QoS-driven Runtime Adaptation of Service Oriented Architectures”. In: ESEC/FSE. 2009, pp. 131–140. [8] Youcong Ni et al. “Multi-objective software performance optimisation at the architecture level using randomised search rules”. In: JIST 135 (2021), p. 106565. issn: 0950-5849. doi: https://doi.org/https://doi.org/10.1016/j.infsof.2021.106565https: //doi.org/10.1016/j.infsof.2021.106565. D. Di Pompeo Search Budget in Multi-Objective Refactoring Optimization: a Model-Based Empirical Study 19 / 22
25. References III [9] Alejandro Rago et al. “Distributed quality-attribute optimization of software architectures”. In: SBCARS. 2017, 7:1–7:10. doi: https://doi.org/10.1145/3132498.313250910.1145/3132498.3132509. [10] Steffen Becker, Heiko Koziolek, and Ralf H Reussner. “The Palladio component model for model-driven performance prediction”. In: Systems and Software 82.1 (Jan. 2009), pp. 3–22. [11] Eckart Zitzler, Kalyanmoy Deb, and Lothar Thiele. “Comparison of Multiobjective Evolutionary Algorithms: Empirical Results”. In: Evolutionary Computation 8.2 (2000), pp. 173–195. doi: https://doi.org/10.1162/10636560056820210.1162/106365600568202. url: https://direct.mit.edu/evco/article/8/2/173-195/868. [12] H. B. Mann and D. R. Whitney. “On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other”. In: The Annals of Mathematical Statistics 18.1 (1947), pp. 50–60. doi: https://doi.org/10.1214/aoms/117773049110.1214/aoms/1177730491. url: https://doi.org/10.1214/aoms/1177730491. D. Di Pompeo Search Budget in Multi-Objective Refactoring Optimization: a Model-Based Empirical Study 20 / 22
26. References IV [13] Andrea Arcuri and Lionel C. Briand. “A Hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering”. In: Softw. Test. Verification Reliab. 24.3 (2014), pp. 219–250. doi: https://doi.org/10.1002/stvr.148610.1002/stvr.1486. url: https://doi.org/10.1002/stvr.1486. [14] Daniele Di Pompeo et al. “A Microservice Reference Case Study for Design-Runtime Interaction in MDE”. In: STAF MDE@DeRun Workshop. Vol. 2405. 2019, pp. 23–32. [15] Sebastian Herold et al. “CoCoME - The Common Component Modeling Example”. In: The Common Component Modeling Example. Ed. by Andreas Rausch et al. Vol. 5153. lncs. Springer Berlin Heidelberg, 2008, pp. 16–53. isbn: 9783540852889 9783540852896. doi: https://doi.org/10.1007/978-3-540-85289-6_310.1007/978-3-540-85289-6_3. url: http://link.springer.com/10.1007/978-3-540-85289-6%5C_3. D. Di Pompeo Search Budget in Multi-Objective Refactoring Optimization: a Model-Based Empirical Study 21 / 22
27. References V [16] Shaukat Ali et al. “Quality Indicators in Search-based Software Engineering: An Empirical Evaluation”. In: ACM Transactions on Software Engineering and Methodology 29.2 (2020), pp. 1–29. doi: https://doi.org/10.1145/337563610.1145/3375636. url: https://dl.acm.org/doi/10.1145/3375636. [17] Jes´ us Guillermo Falc´ on-Cardona and Carlos A. Coello Coello. “Indicator-based Multi-objective Evolutionary Algorithms: A Comprehensive Survey”. en. In: ACM Computing Surveys 53.2 (Mar. 2021), pp. 1–35. doi: https://doi.org/10.1145/337691610.1145/3376916. url: https://dl.acm.org/doi/10.1145/3376916. D. Di Pompeo Search Budget in Multi-Objective Refactoring Optimization: a Model-Based Empirical Study 22 / 22