Francesco Biscani: A Snake in Space - The rise of scientific Python in Astrodynamics and Astronomy (Keynote)

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
Francesco Biscani:
A Snake in Space - The rise of scientific Python in Astrodynamics and Astronomy
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
@ Kiwi PyCon 2014 - Sunday, 14 Sep 2014 - Keynote
http://kiwi.pycon.org/

**YouTube**

https://www.youtube.com/watch?v=9-PxGDp6R1Q

Transcript

1. Introduction
   PaGMO
   Astropy
   A Snake in Space
   The rise of scientific Python in Astrodynamics and Astronomy
   Francesco Biscani & Dario Izzo
   Max Planck Institute for Astronomy,
   European Space Agency
   September 13, 2014
   Francesco Biscani & Dario Izzo A Snake in Space
   

   View Slide

2. Introduction
   PaGMO
   Astropy
   Outline
   1 Introduction
   About us
   Projects: PaGMO and Astropy
   Background and context
   2 PaGMO
   History
   Interlude
   Architectural overview
   3 Astropy
   Francesco Biscani & Dario Izzo A Snake in Space
   

   View Slide

3. Introduction
   PaGMO
   Astropy
   About us
   Projects: PaGMO and Astropy
   Background and context
   About us
   Francesco:
   PhD in Astronomy
   Research fellow at ESA (2008–2011)
   HPC software engineer (2012–2014)
   Postdoc at MPIA (since 2014)
   Dario:
   PhD in Mathematical Statistics
   Research fellow at ESA (2005–2007)
   Scientific Director of the Advanced Concepts Team (ACT)
   (since 2007)
   Francesco Biscani & Dario Izzo A Snake in Space
   

   View Slide

4. Introduction
   PaGMO
   Astropy
   About us
   Projects: PaGMO and Astropy
   Background and context
   PaGMO
   Parallel Global Multiobjective Optimiser
   Started in 2009 building on existing in-house research code
   Initially thought as a trajectory optimisation tool, evolved as a
   general-purpose optimiser
   Mixed C++/Python codebase (PaGMO vs PyGMO)
   Focused on parallel and distributed computing
   Around 64000 SLOC
   Multiple GSoC and SOCIS participations
   http://esa.github.io/pygmo/
   Francesco Biscani & Dario Izzo A Snake in Space
   

   View Slide

5. Introduction
   PaGMO
   Astropy
   About us
   Projects: PaGMO and Astropy
   Background and context
   Astropy
   Started in 2011
   A community effort to develop a single core package for
   Astronomy
   An archipelago of “affiliated packages”
   Developed by an international team of researchers from
   different institutions
   Predominantly Python (with interfacing to C and Fortran
   libraries)
   Circa 90000 SLOC
   Multiple GSoC participations
   http://www.astropy.org/
   Francesco Biscani & Dario Izzo A Snake in Space
   

   View Slide

6. Introduction
   PaGMO
   Astropy
   About us
   Projects: PaGMO and Astropy
   Background and context
   Optimisation
   A large area of applied mathematics
   Wikipedia: “The selection of a best element (with regard to
   some criteria) from some set of available alternatives”
   Can often be formulated as a function minimisation problem
   Examples:
   travelling salesman (TSP),
   supply-chain management,
   network routing,
   interplanetary spacecraft trajectories.
   Many types of algorithms:
   gradient-based methods,
   evolutionary algorithms,
   dynamic programming,
   stochastic algorithms,
   . . .
   Francesco Biscani & Dario Izzo A Snake in Space
   

   View Slide

7. Introduction
   PaGMO
   Astropy
   About us
   Projects: PaGMO and Astropy
   Background and context
   Interplanetary trajectories
   Space mission trajectories are defined by sets of parameters:
   launch date,
   initial velocity vector,
   sequence of flybys,
   sequence of deep-space maneuvers (DSM),
   . . .
   Usually, we want to minimise fuel consumption
   The resulting optimisation problem is hard:
   multimodal objective function,
   highly nonlinear,
   highly dimensional.
   Traditionally tackled by teams of human experts
   Francesco Biscani & Dario Izzo A Snake in Space
   

   View Slide

8. Introduction
   PaGMO
   Astropy
   About us
   Projects: PaGMO and Astropy
   Background and context
   Example: Messenger’s path to Mercury
   Francesco Biscani & Dario Izzo A Snake in Space
   

   View Slide

9. Introduction
   PaGMO
   Astropy
   About us
   Projects: PaGMO and Astropy
   Background and context
   Bio-inspired algorithms and Island Model
   A class of optimisation algorithms inspired by biological
   processes:
   genetic algorithms,
   differential evolution,
   ant-colony optimisation,
   artificial bee-colony optimisation,
   . . .
   They can deliver good performance on hard problems
   Island model: different algorithms run in parallel on the same
   optimisation problem, periodically exchanging information
   Francesco Biscani & Dario Izzo A Snake in Space
   

   View Slide

10. Introduction
    PaGMO
    Astropy
    History
    Interlude
    Architectural overview
    History of PaGMO
    Origin: C/C++ research code (∼2005–2008)
    Initial framework: C++ OO hierarchy + Python bindings
    (2008–2009)
    parallelisation capabilities limited to local threads
    non-extensible Python bindings (PyGMO)
    open-sourced, GIT repository
    extensive use for research purposes
    Architectural overhaul and first GSoC (2010–2011)
    parallelisation capabilities extended to clusters
    fully-extensible Python bindings
    Accretion of features and maturity (2012–2014)
    bindings for many open-source optimisation libraries (SciPy,
    NLopt, GSL, Ipopt, etc.)
    implementation of many state-of-the-art algorithms
    second GSoC (2013)
    website, documentation, tutorials, etc.
    Francesco Biscani & Dario Izzo A Snake in Space
    

    View Slide

11. Introduction
    PaGMO
    Astropy
    History
    Interlude
    Architectural overview
    Interlude: the Pros and Cons of Scientific Computing
    The Good:
    emphasis on correctness and reproducibility
    powerful driver for innovations in high-performance computing
    The Ugly:
    most scientists are not trained in software engineering
    code is often written with a one-paper-horizon mindset
    code often abandoned after use, wheel re-invention
    legacy code/tools
    What scientists want:
    performance
    interactivity and exploratory computing
    reproducibility and the ability to inspect and modify
    Francesco Biscani & Dario Izzo A Snake in Space
    

    View Slide

12. Introduction
    PaGMO
    Astropy
    History
    Interlude
    Architectural overview
    The PaGMO framework
    Clear separation of concepts:
    optimisation algorithm and problem abstract classes
    the abstract island class includes a problem, an algorithm and
    a population of candidate solutions, and a virtual evolve()
    method that dispatches the optimisation (to a thread, or a
    process on another machine)
    an archipelago is a collection of islands linked by a topology,
    across which inter-island communication happens
    All the base classes are extensible:
    define your own problems, algorithms, islands and topologies
    the classes can be extended from C++ or Python
    large collection of algorithms (∼50) and problems already
    implemented
    Francesco Biscani & Dario Izzo A Snake in Space
    

    View Slide

13. Introduction
    PaGMO
    Astropy
    History
    Interlude
    Architectural overview
    The C++ ⇐⇒ Python bridge
    Implemented via Boost.Python
    Challenging, non-trivial issues:
    serialisation across language boundaries involving virtual
    classes, base pointers, etc.
    extension from Python of C++ base classes
    working around some of Python’s limitation wrt parallel
    programming (GIL)
    Example: local parallelisation of custom problems/algorithms
    defined from Python
    Problem: cannot call the Python interpreter from multiple
    threads at the same time
    Solution: offload the computation to a separate process
    (multiprocessing), use the threads only for communication
    Francesco Biscani & Dario Izzo A Snake in Space
    

    View Slide

14. Introduction
    PaGMO
    Astropy
    History
    Interlude
    Architectural overview
    Leveraging Python’s strengths
    Reliance on the scientific Python stack (NumPy, SciPy,
    matplotlib, IPython, etc.)
    IPython notebooks for interactivity and exploratory computing
    matplotlib for trajectory visualisation
    NumPy for crunching through the results
    SciPy optimisation algorithms usable from the framework
    High-performance and parallel computing
    multiprocessing frees us from the GIL shackles
    IPython cluster processing
    Francesco Biscani & Dario Izzo A Snake in Space
    

    View Slide

15. Introduction
    PaGMO
    Astropy
    History
    Interlude
    Architectural overview
    Some highlights
    Selection of NEAs for future human missions (in collaboration
    with NASA JPL)
    Evolution of neural networks for autonomous Martian rovers
    (University of Plymouth)
    Widely used in GTOC by the ACT
    Francesco Biscani & Dario Izzo A Snake in Space
    

    View Slide

16. Introduction
    PaGMO
    Astropy
    Astropy: overview
    NumPy as only dependency
    Works on Linux, OSX, Windows
    Core functionality:
    Units, quantities, constants
    Celestial coordinates representation and transformation
    Handling of FITS files
    Convolution/filtering utilities
    Cosmological calculations
    Model fitting
    Astrostatistics
    . . .
    Community consensus for inclusion of new functionality
    Wide selection of affiliated packages
    Francesco Biscani & Dario Izzo A Snake in Space
    

    View Slide

17. Introduction
    PaGMO
    Astropy
    Astropy: code
    Overwhelmingly Python
    Heavy reliance on NumPy (e.g., the Table class)
    Bits of Fortran, C and Cython for performance (possibly
    Numba in the future)
    Interoperability with existing Astronomical software (e.g.,
    montage-wrapper)
    Development with the GitHub workflow
    Well-tested and documented (e.g., Travis)
    Francesco Biscani & Dario Izzo A Snake in Space
    

    View Slide

18. Introduction
    PaGMO
    Astropy
    Pretty picture!
    Francesco Biscani & Dario Izzo A Snake in Space
    

    View Slide