Introduction PaGMO Astropy A Snake in Space The rise of scientific Python in Astrodynamics and Astronomy Francesco Biscani & Dario Izzo Max Planck Institute for Astronomy, European Space Agency September 13, 2014 Francesco Biscani & Dario Izzo A Snake in Space

Introduction PaGMO Astropy Outline 1 Introduction About us Projects: PaGMO and Astropy Background and context 2 PaGMO History Interlude Architectural overview 3 Astropy Francesco Biscani & Dario Izzo A Snake in Space

Introduction PaGMO Astropy About us Projects: PaGMO and Astropy Background and context About us Francesco: PhD in Astronomy Research fellow at ESA (2008–2011) HPC software engineer (2012–2014) Postdoc at MPIA (since 2014) Dario: PhD in Mathematical Statistics Research fellow at ESA (2005–2007) Scientific Director of the Advanced Concepts Team (ACT) (since 2007) Francesco Biscani & Dario Izzo A Snake in Space

Introduction PaGMO Astropy About us Projects: PaGMO and Astropy Background and context PaGMO Parallel Global Multiobjective Optimiser Started in 2009 building on existing in-house research code Initially thought as a trajectory optimisation tool, evolved as a general-purpose optimiser Mixed C++/Python codebase (PaGMO vs PyGMO) Focused on parallel and distributed computing Around 64000 SLOC Multiple GSoC and SOCIS participations http://esa.github.io/pygmo/ Francesco Biscani & Dario Izzo A Snake in Space

Introduction PaGMO Astropy About us Projects: PaGMO and Astropy Background and context Astropy Started in 2011 A community effort to develop a single core package for Astronomy An archipelago of “affiliated packages” Developed by an international team of researchers from different institutions Predominantly Python (with interfacing to C and Fortran libraries) Circa 90000 SLOC Multiple GSoC participations http://www.astropy.org/ Francesco Biscani & Dario Izzo A Snake in Space

Introduction PaGMO Astropy About us Projects: PaGMO and Astropy Background and context Optimisation A large area of applied mathematics Wikipedia: “The selection of a best element (with regard to some criteria) from some set of available alternatives” Can often be formulated as a function minimisation problem Examples: travelling salesman (TSP), supply-chain management, network routing, interplanetary spacecraft trajectories. Many types of algorithms: gradient-based methods, evolutionary algorithms, dynamic programming, stochastic algorithms, . . . Francesco Biscani & Dario Izzo A Snake in Space

Introduction PaGMO Astropy About us Projects: PaGMO and Astropy Background and context Interplanetary trajectories Space mission trajectories are defined by sets of parameters: launch date, initial velocity vector, sequence of flybys, sequence of deep-space maneuvers (DSM), . . . Usually, we want to minimise fuel consumption The resulting optimisation problem is hard: multimodal objective function, highly nonlinear, highly dimensional. Traditionally tackled by teams of human experts Francesco Biscani & Dario Izzo A Snake in Space

Introduction PaGMO Astropy About us Projects: PaGMO and Astropy Background and context Example: Messenger’s path to Mercury Francesco Biscani & Dario Izzo A Snake in Space

Introduction PaGMO Astropy About us Projects: PaGMO and Astropy Background and context Bio-inspired algorithms and Island Model A class of optimisation algorithms inspired by biological processes: genetic algorithms, differential evolution, ant-colony optimisation, artificial bee-colony optimisation, . . . They can deliver good performance on hard problems Island model: different algorithms run in parallel on the same optimisation problem, periodically exchanging information Francesco Biscani & Dario Izzo A Snake in Space

Introduction PaGMO Astropy History Interlude Architectural overview History of PaGMO Origin: C/C++ research code (∼2005–2008) Initial framework: C++ OO hierarchy + Python bindings (2008–2009) parallelisation capabilities limited to local threads non-extensible Python bindings (PyGMO) open-sourced, GIT repository extensive use for research purposes Architectural overhaul and first GSoC (2010–2011) parallelisation capabilities extended to clusters fully-extensible Python bindings Accretion of features and maturity (2012–2014) bindings for many open-source optimisation libraries (SciPy, NLopt, GSL, Ipopt, etc.) implementation of many state-of-the-art algorithms second GSoC (2013) website, documentation, tutorials, etc. Francesco Biscani & Dario Izzo A Snake in Space

Introduction PaGMO Astropy History Interlude Architectural overview Interlude: the Pros and Cons of Scientific Computing The Good: emphasis on correctness and reproducibility powerful driver for innovations in high-performance computing The Ugly: most scientists are not trained in software engineering code is often written with a one-paper-horizon mindset code often abandoned after use, wheel re-invention legacy code/tools What scientists want: performance interactivity and exploratory computing reproducibility and the ability to inspect and modify Francesco Biscani & Dario Izzo A Snake in Space

Introduction PaGMO Astropy History Interlude Architectural overview The PaGMO framework Clear separation of concepts: optimisation algorithm and problem abstract classes the abstract island class includes a problem, an algorithm and a population of candidate solutions, and a virtual evolve() method that dispatches the optimisation (to a thread, or a process on another machine) an archipelago is a collection of islands linked by a topology, across which inter-island communication happens All the base classes are extensible: define your own problems, algorithms, islands and topologies the classes can be extended from C++ or Python large collection of algorithms (∼50) and problems already implemented Francesco Biscani & Dario Izzo A Snake in Space

Introduction PaGMO Astropy History Interlude Architectural overview The C++ ⇐⇒ Python bridge Implemented via Boost.Python Challenging, non-trivial issues: serialisation across language boundaries involving virtual classes, base pointers, etc. extension from Python of C++ base classes working around some of Python’s limitation wrt parallel programming (GIL) Example: local parallelisation of custom problems/algorithms defined from Python Problem: cannot call the Python interpreter from multiple threads at the same time Solution: offload the computation to a separate process (multiprocessing), use the threads only for communication Francesco Biscani & Dario Izzo A Snake in Space

Introduction PaGMO Astropy History Interlude Architectural overview Leveraging Python’s strengths Reliance on the scientific Python stack (NumPy, SciPy, matplotlib, IPython, etc.) IPython notebooks for interactivity and exploratory computing matplotlib for trajectory visualisation NumPy for crunching through the results SciPy optimisation algorithms usable from the framework High-performance and parallel computing multiprocessing frees us from the GIL shackles IPython cluster processing Francesco Biscani & Dario Izzo A Snake in Space

Introduction PaGMO Astropy History Interlude Architectural overview Some highlights Selection of NEAs for future human missions (in collaboration with NASA JPL) Evolution of neural networks for autonomous Martian rovers (University of Plymouth) Widely used in GTOC by the ACT Francesco Biscani & Dario Izzo A Snake in Space

Introduction PaGMO Astropy Astropy: overview NumPy as only dependency Works on Linux, OSX, Windows Core functionality: Units, quantities, constants Celestial coordinates representation and transformation Handling of FITS files Convolution/filtering utilities Cosmological calculations Model fitting Astrostatistics . . . Community consensus for inclusion of new functionality Wide selection of affiliated packages Francesco Biscani & Dario Izzo A Snake in Space

Introduction PaGMO Astropy Astropy: code Overwhelmingly Python Heavy reliance on NumPy (e.g., the Table class) Bits of Fortran, C and Cython for performance (possibly Numba in the future) Interoperability with existing Astronomical software (e.g., montage-wrapper) Development with the GitHub workflow Well-tested and documented (e.g., Travis) Francesco Biscani & Dario Izzo A Snake in Space

Introduction PaGMO Astropy Pretty picture! Francesco Biscani & Dario Izzo A Snake in Space