A Parallel General Purpose Multi-Objective Optimization Framework, Applied to Beam Dynamic Studies Yves Ineichen1,2,3, Andreas Adelmann2, Costas Bekas3, Alessandro Curioni3, Peter Arbenz1 1ETH Zürich Department of Computer Science CH-8092 Zürich, Switzerland 2Paul Scherrer Institut Accelerator Modelling and Advanced Simulations CH-5234 Villigen, Switzerland 3IBM Research-Zurich CH-8803 Rüschlikon, Switzerland 21st August 2012 1 / 37 

. . Complex Decision Problem . Trade-off . Effect 2 / 37 

. . Complex Decision Problem . Trade-off . Effect . Design and oper- ation of particle accelerators . Multi-Objective Optimization Problem . MOO Algorithms . Simulation Codes . High performance computing . Applications . Visualization 2 / 37 

. Multi-Objective Optimization [a short introduction] 3 / 37 

Multi-Objective Optimization Problem min . fm(x. ), m = 1 . . . M s.t. gj(x) ≥ 0, j = 0 . . . J xL i ≤ x = xi ≤ xU i , i = 0 . . . n . . Objectives . Design variables . Constraints 4 / 37 

Mapping design to objective space . . x1 . x2 Design space . . f1 . f2 Objective space . . min . min 5 / 37 

Mapping design to objective space . . x1 . x2 Design space . . f1 . f2 Objective space . . min . min 5 / 37 

Optimality? . . f2 . f1 . price . performance . low . low . high . high . x∗ 0 . x∗ 1 . x∗ 2 . x∗ 3 . x4 • conflicting objectives: minimize price maximize performance • red points are “equally optimal”: cannot improve one point without hurting at least one other solution → Pareto optimality • blue curve is called Pareto front • x4 is dominated by x∗ 1 and x∗ 2 6 / 37 

Preference Specification . . f2 . f1 . price . performance . low . low . high . high . x∗ a-priori a-priori preference: e.g. performance ≫ price → x∗ a-priori a-posteriori preference: → Pareto front • provides deeper understanding of solution space • visualizes how choice affects design space 7 / 37 

. Evolutionary Algorithms [how can we solve multi-objective optimization problems?] 8 / 37 

. . Multiobjective Optimization . reduce to single- objective . . a priori ar- ticulation of preference . . weighted sum . weighted global criterion . Chankong, Haimes 83 . Zeleny 82 . Yu, Leit- mann 74 . lexico- graphic . Osyczka 84 . Waltz 67 . weighted min-max . Miettin- en 99 . exponen-tial weighted . Athan, Pa- palam- bros 96 . weighted product . Gerasimov, Repko 78 . Bridg- man 22 . goal pro- gramming . Charnes et al. 67/55 . Ijiri 65 . Charnes, Cooper 61 . bounded objective function . Hwang, Md. Masud 79 . Haimes et al. 71 . physical pro- gramming . Chen et al. 00 . Messac 96 . a posteriori articula- tion of preference . . physical pro- gramming . Messac, Mattson 02 . Martinez et al. 01 . NBI . Das 99 . Das, Dennis 98 . NC . Messac et al. 03 . no artic- ulation of preference . . global criterion . TOPSIS . Hwang et al. 93 . Yoon 80 . object- ive sum . Chankong, Haimes 83 . Zeleny 82 . Yu, Leit- mann 74 . min-max . Li 92 . Osyczka 78 . Yu 73 . Nash arbitration . Straffin 93 . Davis 83 . object- ive product . Cheng, Li 96 . Rao . Rao 87 . Rao and Freiheit 91 . multi- objective . . simulated annealing . . SMOSA . Suppap- itnarm et al 00 . UMOSA . Ulunga et al 98,99 . PSA . Czyzak et al 96-98 . WMOSA . Susman 02-04 . PDMOSA . Susman 03-05 . particle swarm . . Aggrega- tion . Parso- poulos et al . Baum- gartner et al . Lexico- graphic . Hu and Eberhart . Sub- Popula-tion . Parso- poulos et al . Chow and Tsui . Comb-ined . Mah- fouf et al. . Xiao-hua et al. . Other . Li . Zhang et al. . Pareto- based . Moore and Chap- man . Ray and Liew . Field- send and Singh . ... . evolution- ary algorithms . . ranking . Gold- berg 89 . Fonseca, Fleming 93 . Srinivas, Deb 95 . Cheng, Li 95 . VEGA . Schaf-fer 85 . Pareto-set filter . Cheng, Li 97 . tourna-ment selection . Horn et al. 94 . niche techniques . fitness sharing . additional techniques . eucli- dean distance . Osyc- zka, Kundu 96 . weigh- ted sum . Ishi- buchi, Murata 96 . zero- one- weigh- ted sum . Kurs- awe 91 . constr. preemp. goal prog. . Gen, Liu 95 . Pareto fitness func. . Schau- mann et al. 98 . • only one Pareto solution can be found in one run • preference-based (specify preference for trade-off solution) • not all can be found in non-convex MOOPS • all algorithms require a prior knowledge (weights, ε, targets) . • multiple Pareto solutions can be found in one run • a posteriori articulation of preference • “easier”: diversity in decision and objective space (non-linear mapping) 9 / 37 

Evolutionary Algorithms . . . Population . I1 . Ik . I2 . I3 . I4 . . Selector . 1. I4 , 2. Ik , 3. I2 , 4. I3 . 5. I1 , . . ., n. In . . Variator . I4 · Ik : . = 10 / 37 

Ranking individuals Non-dominated sorting genetic algorithm (Nsga-II) initialization: • count how many solutions np dominate solution p • store all solutions p dominates in set Sp • set k ← 0 Repeat while there exists solutions with np > 0: • for all solutions p with np = 0: • store solution in k-th non-dominated front • visit all members i of Sp and reduce ni by one • k ← k + 1 Order relation corresponds to index in set of non-dominated fronts A fast and elitist multiobjective genetic algorithm: NSGA-II, K. Deb et. al., IEEE Transactions on Evolutionary Computation, 6(2):182–197, Apr. 2002. 11 / 37 

Evolutionary Algorithms . . NSGA-II Selector . dispatch individuals . Variator . 2 individuals ready • PISA • Finite state machine • Nsga-II selector • access to many other selectors • “Continuous generations” • Independent bit mutations • Various crossover polices A Platform and Programming Language Independent Interface for Search Algorithms: http://www.tik.ee.ethz.ch/pisa/ 12 / 37 

First Population . . dE [MeV] . 0.20 . 0.22 . 0.24 . 0.26 . 0.28 . 0.30 . emitx [mm mrad] . 0 . 5 . 10 . 15 . 20 13 / 37 

649th Population . . dE [MeV] . 0.20 . 0.22 . 0.24 . 0.26 . 0.28 . 0.30 . emitx [mm mrad] . 0 . 5 . 10 . 15 . 20 Now scientist/operator can specify preference 14 / 37 

. The Framework [how can we facilitate solving multi-objective problems?] 15 / 37 

Multi-Objective Optimization Framework . . Optimizer . Pilot . - - input . - - obj . - - constr . - - sims . OPAL . Convex Optimization Algorithms . Heuristic Algorithms 16 / 37 

Master/Worker Model . . . Optimizers . O1 . Oi . . Pilot . job . queue . j2 . j1 . j3 . j4 . r1 . . Workers . W1 . Wj 17 / 37 

Master/Worker Model Comp. Domain Optimizeri [coarse] multiple starting points, multiple opt. problems Corei Forward Solverj [fine] parallel E-T Workers [coarse] eval forward problem Optimizer [coarse] parallel optimizer 18 / 37 

. Application [how can we use the framework?] 19 / 37 

Ingredients 1× optimization algorithm, 1× forward solver and 1× specification of optimization problem, e.g., annotating the simulation input file: //d1: DVAR, VARIABLE="D_LAG_RGUN", LOWERBOUND="-0.1", UPPERBOUND="0.1"; //d2: DVAR, VARIABLE="D_LAG_B01", LOWERBOUND="-0.1", UPPERBOUND="0.1"; //obj1: OBJECTIVE, EXPR="energy*-1"; //obj2: OBJECTIVE, EXPR="dE * meas_error("file", "rms_x")"; //objs: OBJECTIVES = (obj1, obj2); //dvars: DVARS = (d1, d2); //constrs: CONSTRAINTS = (); //opt: OPTIMIZE, OBJECTIVES=objs, DVARS=dvars, CONSTRAINTS=constrs; 20 / 37 

Forward solver typically is expensive to run.. 21 / 37 

Maxwell’s Equation in the Electrostatic approximation . . Electro Magneto Optics . N-Body Dynamics . 1,2 or 3D Field Maps & Analytic Models (E, B)ext . Poisson Problem s.t. BC’s ∆ϕ′ sc = − ρ′ ϵ0 → (E, B)SC . H = Hext + Hsc . If E(x, t) and B(x) are known, the equation of motion can be integrated: • Boris-pusher (adaptive version soon!) • Leap-Frog • RK-4 22 / 37 

and we require a massive number of forward solves per optimization 23 / 37 

Object Oriented Parallel Accelerator Library (OPAL) OPAL is a tool for precise charged-particle optics in large accelerator structures and beam lines including 3D space charge. • built from the ground up as a parallel application • runs on your laptop as well as on the largest HPC clusters • uses the mad language with extensions • written in C++ using OO-techniques, hence OPAL is easy to extend • nightly regression tests track the code quality OPAL: https://amas.psi.ch/OPAL 24 / 37 

3D Tracker . . e− . e− repulsive force of charged particles • Huge # of macro particles (100’000 – 100’000’000) • Computing space-charge is expensive • Load balancing difficult • Lots of synchronization points . . Slow but “high resolution” forward solver The Object Oriented Parallel Accelerator Library (OPAL), Design, Implementation and Application, A. Adelmann et. al. 25 / 37 

Envelope Tracker . . z . x . y • # slices ≪ # macro particles • Analytical space-charge • Slices distributed in contiguous blocks • Load imbalance of at most 1 slice • Low number of synchronization points Fast but “low resolution” forward solver A fast and scalable low dimensional solver for charged particle dynamics in large particle accelerators, Y. Ineichen et. al. 26 / 37 

. Usage [Ferrario Matching Point] 27 / 37 

Optimization Problem min [εx, ∆rmsx,peak, ∆εx,peak] //min_ex: OBJECTIVE, EXPR="emit_x"; //peak_rms_x: FROMFILE, FILE="rms_x-err.dat"; //peak_e_x: FROMFILE, FILE="emit_x-err.dat"; //sig_x: DVAR, VARIABLE="SIGX", LOWERBOUND="0.000250", UPPERBOUND="0.000250"; //sol_ks: DVAR, VARIABLE="MSOL10_i", LOWERBOUND="110", UPPERBOUND="120"; //lag_gun: DVAR, VARIABLE="D_LAG_GUN", LOWERBOUND="0.0", UPPERBOUND="0.05"; //volt_gun: DVAR, VARIABLE="RACC_v", LOWERBOUND="25", UPPERBOUND="40"; 28 / 37 

. . ∆εx, peak [m] . 0.05 . 0.10 . 0.15 . 0.20 . 0.25 . 0.30 . 0.35 . ∆rmsx, peak [m] . 0.2 . 0.4 . 0.6 . 0.8 . 1.0 . 1.2 . 1.4 Pareto front after 1’000 generations (approx. 20 minutes on 16 cores) 29 / 37 

. . ∆εx, peak [m] . 0.05 . 0.10 . 0.15 . 0.20 . 0.25 . 0.30 . 0.35 . ∆rmsx, peak [m] . 0.2 . 0.4 . 0.6 . 0.8 . 1.0 . 1.2 . 1.4 Pareto front after 1’000 generations (approx. 20 minutes on 16 cores) 29 / 37 

Simulation results for individual 38 30 / 37 

Conclusions Multi-Objective Optimization Problems • omnipresent in many fields in research and design • important in understanding problem and trade-off solutions • expensive to solve Framework • closes the gap between theory and user • combining OPAL and EA results in a viable MOOP solver for beam dynamics • HPC necessary to compute Pareto front in meaningful timeframe 31 / 37 

Outlook Investigate other multi-objective optimization algorithms Run real optimization problem on massive number of cores Visualization of results 32 / 37 

This project is funded by: IBM Research – Zurich Paul Scherrer Institut Acknowledgements • OPAL developer team • SwissFel team • Sumin Wei 33 / 37 

Backup 34 / 37 

Optimization Problem min [energy spread, emittance] s.t. ∂t f(x, v, t) + v · ∇x f(x, v, t) + q m0 (Etot + v × Btot) · ∇v f(x, v, t) = 0 ∇ × B = µ0 J + µ0 ε0 ∂E ∂t , ∇ × E = − ∂B ∂t ∇ · E = ρ ε0 , ∇ · B = 0 ρ = −e ∫ f(x, v, t) d3p, J = −e ∫ f(x, v, t)v d3p E = Eext + Eself, B = Bext + Bself . . . 35 / 37 

Envelope Equations d2 d2t Ri + βiγ2 i d dt (βi Ri) + Ri ∑ j Kj i = 2c2kp Riβi × ( G(∆i, Ar) γ3 i − (1 − β2 i ) G(δi, Ar) γi ) + 4εth n c γi 1 R3 i d dt βi = e0 m0 cγ3 i ( Eext z (zi, t) + Esc z (zi, t) ) d dt zi = cβi 36 / 37 

SwissFel 1: Switzerland’s X-ray free-electron laser Project at PSI • big project: > 100 people, expensive, 700 m long • 1 Ångström • to reach target it is of crucial importance to attain “good” beam properties (e.g. narrow beam/small phase space volume) . 1http://www.psi.ch/swissfel/ 37 / 37 

SwissFel 1: Switzerland’s X-ray free-electron laser Project at PSI • big project: > 100 people, expensive, 700 m long • 1 Ångström • to reach target it is of crucial importance to attain “good” beam properties (e.g. narrow beam/small phase space volume) . . Calls for optimization of Injector • Several conflicting objectives • Key technology: multi-objective optimization 1http://www.psi.ch/swissfel/ 37 / 37