TACO: Toolkit for Automatic Comparison Optimizers (for LSGO)

Global optimization is a research subject that has attracted significant attention, including both theoretical and practical studies, in recent years. In this sense, some of the main conferences in the field of Evolutionary Computation have been organizing special sessions on this topic for more than a decade. Those special sessions normally propose a well defined benchmark of functions to allow a fair comparison of participating algorithms. Being able to manually tackle all this information has become a difficult task for many researchers. Which algorithm obtained the best results on a particular benchmark? How does the new method that I am developing compare to that algorithm? To answer this question, we propose TACO, a web application that stores all this information and makes it possible to seamlessly analyze it and generate detailed reports with the results of these analyses. The application has been designed in such a flexible way that it is extremely easy to add new benchmarks and their associated (possibly specific) analyses. Of course, these benchmarks are not limited to large-scale global optimization, but potentially any type of optimization problems.

Transcript

1. TACO: Toolkit for Automatic Comparison Optimisers (for LSGO) Daniel Molina1 Antonio LaTorre2 1 University of Granada, Spain 2 Universidad Politécnica de Madrid, Spain
2. Good comparisons are mandatory Evolutionary Algorithms 1 Design good algorithm for a particular problem (or category of problems). 2 Experimental section with optimiser's results. Require comparisons To show that your algorithm is competitive enough. To show the inuence of each component of the algorithm.
3. Diculties comparing optimizers Compare is not easy 1 Good functions benchmark. 2 Search competitive algorithms as reference algorithms. 3 Compare against (at least) that previous algorithms. 4 Obtain several measures: mean, maximum, ... 5 Create comparative tables. 6 Plots: convergence, ... 7 Statistical testing (parametric and non-parametric tests). 8 ...
4. Time consuming It is even worse New reference algorithms by ourself or suggestions. New ideas studing changes in our algorithm. Errors by code or experimental conditions. Very time consuming Imply a lot of researcher time (the important one). Can be automatically done?
5. The story of Large-Scale Global Optimization Specially important for organizing competitions High number of proposals (not yet in LSGO). Particular comparison tests. Similar each year. Phases 1 Manually (Excel les). 2 Several automatic small programs. 3 More general toolkit. Aim: Toolkit for comparison Useful for competition organizers (non-only LSGO). Useful for researcher, during design of algorithms.
6. Several alternatives Dierent software options Specic tools KEEL, R, .... ⇒ not integrated with data. Frameworks ECJ, ... ⇒ strongly depends on framework. In Large-Scale Global Optimization http://www.cercia.ac.uk/projects/lsgo/ references of paper about Large-Scale Global Optimization. Project MIDAS http://vps128.cesvima.upm.es/lab/ website comparing previously-saved proposals.
7. MIDAS Problems Only compare algorithms previous-saved in database Not useful during the work of the researcher. All algorithm visible for all. Manual Management Authors cannot submit its own algorithms. Technical debt Wordpress module, not secure. Dicult to integrate in an existing website.
8. TACO Toolkit Toolkit for Automatic Comparison Optimisers (TACO) Online Not installation. Completely responsible (mobile, ...). Database Extensive database of previous algorithms. Authors can submit its own results. Private/public algorithms. Direct usage No login-required. Compare from Excel les (and the database).
9. TACO Toolkit; Modern design Easy to integrate existing website static and dynamic websites. Python combine with other libraries and languages as R.
10. TACO Toolkit Extensive New algorithms. New benchmarks. New reports (including specic ones).
11. TACO Toolkit Benchmark oriented Benchmark Benchmark. Dimension Dimension of benchmark. Algorithm Algorithm with name and results. Functions Functions. Group of functions Analysis based on features. Milestone Dierent measures. Author Owner of algorithms (private). Report Selected report by the user.
12. TACO Demo http://tsgo.herokuapp.com http://tacolab.org/ (soon) (in migration)
13. TACO frontend
14. TACO frontend
15. TACO frontend
16. TACO Report Two types Tabular result. Graphic results. Table results Remark visually the best one. Calculate automatically (all results can be storage). Graphical results Generic API. Export in several formats (png, pdf, ...). Several libraries support: Bokeh Open-source library, slow. Highcharts Quickly, limited license.
17. Current TACO Reports Mean results Mean for each function and algorithm. Highlight best result for function. Convergence curve Show results for each milestone as plot. Not only for one run, accumulate by all run. Large-Scale Global Optimization Results (F1) 1 Rank algorithms by position for each function. 2 Give points to each algorithm by ranking. 3 Sum points for algorithm. 4 Show grouped by separability degree.
18. Example of mean reports
19. Example of convergence plot Evaluations Mean Error Function: 04 CC-CMA-ES DEEPSO MOS VMODE 1.20e+5 3.00e+6 6.00e+5 1.00e+8 1.00e+9 1.00e+10 1.00e+11 1.00e+12 Highcharts.com Figure: Few milestoness Evaluations (%) Mean Error Function: 07 DYYPO LSHADE_SPACMA MM_OED MOS PPSO RB-IPOP-CMA-ES TLBO-FL jSO 1% 10% 10… 20% 30% 40% 5% 50% 60% 70% 80% 90% 4.00e+0 1.00e+1 2.00e+1 4.00e+1 1.00e+2 2.00e+2 4.00e+2 Highcharts.com Figure: More milestones ⇒ soft.
20. Large-Scale Global Optimization report Algorithm Values Accuracy: 1.200e+05 CC-CMA-ES DECC-G MOS VMODE 0 5 10 15 20 25 30 Highcharts.com Algorithm Values Accuracy: 6.000e+05 CC-CMA-ES DECC-G MOS VMODE 0 5 10 15 20 25 30 Highcharts.com Algorithm Values Accuracy: 3.000e+06 CC-CMA-ES DECC-G MOS VMODE 0 5 10 15 20 25 30 Highcharts.com For a particular group of functions
21. Large-Scale Global Optimization report Algorithm Values Accuracy: 1.200e+05 CC-CMA-ES DECC-G MOS VMODE 0 100 200 300 400 Highcharts.com
22. Large-Scale Global Optimization report Algorithm Values Accuracy: 6.000e+05 CC-CMA-ES DECC-G MOS VMODE 0 100 200 300 400 Highcharts.com
23. Large-Scale Global Optimization report Algorithm Values Accuracy: 3.000e+06 CC-CMA-ES DECC-G MOS VMODE 0 100 200 300 400 Highcharts.com
24. Future works It is a work in progress Future Concept of competition Dierent visibility for same competitions. Statistical tests Dierent statistical tests. Save complete reports zip with gures/tables. Create Latex tables Dierent formats. Working together? It is currently an only-one person work. Working together we can improve it a lot. Simplify life for us and others.
25. Thanks for your attention Thanks you for your attention!! Together, we can make a dierence interested? Daniel Molina <dmolina@decsai.ugr.es>