TACO: Toolkit for Automatic Comparison Optimisers (for LSGO) Daniel Molina1 Antonio LaTorre2 1 University of Granada, Spain 2 Universidad Politécnica de Madrid, Spain 

Good comparisons are mandatory Evolutionary Algorithms 1 Design good algorithm for a particular problem (or category of problems). 2 Experimental section with optimiser's results. Require comparisons To show that your algorithm is competitive enough. To show the inuence of each component of the algorithm. 

Diculties comparing optimizers Compare is not easy 1 Good functions benchmark. 2 Search competitive algorithms as reference algorithms. 3 Compare against (at least) that previous algorithms. 4 Obtain several measures: mean, maximum, ... 5 Create comparative tables. 6 Plots: convergence, ... 7 Statistical testing (parametric and non-parametric tests). 8 ... 

Time consuming It is even worse New reference algorithms by ourself or suggestions. New ideas studing changes in our algorithm. Errors by code or experimental conditions. Very time consuming Imply a lot of researcher time (the important one). Can be automatically done? 

The story of Large-Scale Global Optimization Specially important for organizing competitions High number of proposals (not yet in LSGO). Particular comparison tests. Similar each year. Phases 1 Manually (Excel les). 2 Several automatic small programs. 3 More general toolkit. Aim: Toolkit for comparison Useful for competition organizers (non-only LSGO). Useful for researcher, during design of algorithms. 

Several alternatives Dierent software options Specic tools KEEL, R, .... ⇒ not integrated with data. Frameworks ECJ, ... ⇒ strongly depends on framework. In Large-Scale Global Optimization http://www.cercia.ac.uk/projects/lsgo/ references of paper about Large-Scale Global Optimization. Project MIDAS http://vps128.cesvima.upm.es/lab/ website comparing previously-saved proposals. 

MIDAS Problems Only compare algorithms previous-saved in database Not useful during the work of the researcher. All algorithm visible for all. Manual Management Authors cannot submit its own algorithms. Technical debt Wordpress module, not secure. Dicult to integrate in an existing website. 

TACO Toolkit Toolkit for Automatic Comparison Optimisers (TACO) Online Not installation. Completely responsible (mobile, ...). Database Extensive database of previous algorithms. Authors can submit its own results. Private/public algorithms. Direct usage No login-required. Compare from Excel les (and the database). 

TACO Toolkit; Modern design Easy to integrate existing website static and dynamic websites. Python combine with other libraries and languages as R. 

TACO Toolkit Extensive New algorithms. New benchmarks. New reports (including specic ones). 

TACO Toolkit Benchmark oriented Benchmark Benchmark. Dimension Dimension of benchmark. Algorithm Algorithm with name and results. Functions Functions. Group of functions Analysis based on features. Milestone Dierent measures. Author Owner of algorithms (private). Report Selected report by the user. 

TACO Demo http://tsgo.herokuapp.com http://tacolab.org/ (soon) (in migration) 

TACO frontend 

TACO frontend 

TACO frontend 

TACO Report Two types Tabular result. Graphic results. Table results Remark visually the best one. Calculate automatically (all results can be storage). Graphical results Generic API. Export in several formats (png, pdf, ...). Several libraries support: Bokeh Open-source library, slow. Highcharts Quickly, limited license. 

Current TACO Reports Mean results Mean for each function and algorithm. Highlight best result for function. Convergence curve Show results for each milestone as plot. Not only for one run, accumulate by all run. Large-Scale Global Optimization Results (F1) 1 Rank algorithms by position for each function. 2 Give points to each algorithm by ranking. 3 Sum points for algorithm. 4 Show grouped by separability degree. 

Example of mean reports 

Example of convergence plot Evaluations Mean Error Function: 04 CC-CMA-ES DEEPSO MOS VMODE 1.20e+5 3.00e+6 6.00e+5 1.00e+8 1.00e+9 1.00e+10 1.00e+11 1.00e+12 Highcharts.com Figure: Few milestoness Evaluations (%) Mean Error Function: 07 DYYPO LSHADE_SPACMA MM_OED MOS PPSO RB-IPOP-CMA-ES TLBO-FL jSO 1% 10% 10… 20% 30% 40% 5% 50% 60% 70% 80% 90% 4.00e+0 1.00e+1 2.00e+1 4.00e+1 1.00e+2 2.00e+2 4.00e+2 Highcharts.com Figure: More milestones ⇒ soft. 

Large-Scale Global Optimization report Algorithm Values Accuracy: 1.200e+05 CC-CMA-ES DECC-G MOS VMODE 0 5 10 15 20 25 30 Highcharts.com Algorithm Values Accuracy: 6.000e+05 CC-CMA-ES DECC-G MOS VMODE 0 5 10 15 20 25 30 Highcharts.com Algorithm Values Accuracy: 3.000e+06 CC-CMA-ES DECC-G MOS VMODE 0 5 10 15 20 25 30 Highcharts.com For a particular group of functions 

Large-Scale Global Optimization report Algorithm Values Accuracy: 1.200e+05 CC-CMA-ES DECC-G MOS VMODE 0 100 200 300 400 Highcharts.com 

Large-Scale Global Optimization report Algorithm Values Accuracy: 6.000e+05 CC-CMA-ES DECC-G MOS VMODE 0 100 200 300 400 Highcharts.com 

Large-Scale Global Optimization report Algorithm Values Accuracy: 3.000e+06 CC-CMA-ES DECC-G MOS VMODE 0 100 200 300 400 Highcharts.com 

Future works It is a work in progress Future Concept of competition Dierent visibility for same competitions. Statistical tests Dierent statistical tests. Save complete reports zip with gures/tables. Create Latex tables Dierent formats. Working together? It is currently an only-one person work. Working together we can improve it a lot. Simplify life for us and others. 

Thanks for your attention Thanks you for your attention!! Together, we can make a dierence interested? Daniel Molina