Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Francesco Biscani: A Snake in Space - The rise of scientific Python in Astrodynamics and Astronomy (Keynote)

Francesco Biscani: A Snake in Space - The rise of scientific Python in Astrodynamics and Astronomy (Keynote)

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
Francesco Biscani:
A Snake in Space - The rise of scientific Python in Astrodynamics and Astronomy
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
@ Kiwi PyCon 2014 - Sunday, 14 Sep 2014 - Keynote
http://kiwi.pycon.org/

**YouTube**

https://www.youtube.com/watch?v=9-PxGDp6R1Q

New Zealand Python User Group

September 14, 2014
Tweet

More Decks by New Zealand Python User Group

Other Decks in Programming

Transcript

  1. Introduction
    PaGMO
    Astropy
    A Snake in Space
    The rise of scientific Python in Astrodynamics and Astronomy
    Francesco Biscani & Dario Izzo
    Max Planck Institute for Astronomy,
    European Space Agency
    September 13, 2014
    Francesco Biscani & Dario Izzo A Snake in Space

    View full-size slide

  2. Introduction
    PaGMO
    Astropy
    Outline
    1 Introduction
    About us
    Projects: PaGMO and Astropy
    Background and context
    2 PaGMO
    History
    Interlude
    Architectural overview
    3 Astropy
    Francesco Biscani & Dario Izzo A Snake in Space

    View full-size slide

  3. Introduction
    PaGMO
    Astropy
    About us
    Projects: PaGMO and Astropy
    Background and context
    About us
    Francesco:
    PhD in Astronomy
    Research fellow at ESA (2008–2011)
    HPC software engineer (2012–2014)
    Postdoc at MPIA (since 2014)
    Dario:
    PhD in Mathematical Statistics
    Research fellow at ESA (2005–2007)
    Scientific Director of the Advanced Concepts Team (ACT)
    (since 2007)
    Francesco Biscani & Dario Izzo A Snake in Space

    View full-size slide

  4. Introduction
    PaGMO
    Astropy
    About us
    Projects: PaGMO and Astropy
    Background and context
    PaGMO
    Parallel Global Multiobjective Optimiser
    Started in 2009 building on existing in-house research code
    Initially thought as a trajectory optimisation tool, evolved as a
    general-purpose optimiser
    Mixed C++/Python codebase (PaGMO vs PyGMO)
    Focused on parallel and distributed computing
    Around 64000 SLOC
    Multiple GSoC and SOCIS participations
    http://esa.github.io/pygmo/
    Francesco Biscani & Dario Izzo A Snake in Space

    View full-size slide

  5. Introduction
    PaGMO
    Astropy
    About us
    Projects: PaGMO and Astropy
    Background and context
    Astropy
    Started in 2011
    A community effort to develop a single core package for
    Astronomy
    An archipelago of “affiliated packages”
    Developed by an international team of researchers from
    different institutions
    Predominantly Python (with interfacing to C and Fortran
    libraries)
    Circa 90000 SLOC
    Multiple GSoC participations
    http://www.astropy.org/
    Francesco Biscani & Dario Izzo A Snake in Space

    View full-size slide

  6. Introduction
    PaGMO
    Astropy
    About us
    Projects: PaGMO and Astropy
    Background and context
    Optimisation
    A large area of applied mathematics
    Wikipedia: “The selection of a best element (with regard to
    some criteria) from some set of available alternatives”
    Can often be formulated as a function minimisation problem
    Examples:
    travelling salesman (TSP),
    supply-chain management,
    network routing,
    interplanetary spacecraft trajectories.
    Many types of algorithms:
    gradient-based methods,
    evolutionary algorithms,
    dynamic programming,
    stochastic algorithms,
    . . .
    Francesco Biscani & Dario Izzo A Snake in Space

    View full-size slide

  7. Introduction
    PaGMO
    Astropy
    About us
    Projects: PaGMO and Astropy
    Background and context
    Interplanetary trajectories
    Space mission trajectories are defined by sets of parameters:
    launch date,
    initial velocity vector,
    sequence of flybys,
    sequence of deep-space maneuvers (DSM),
    . . .
    Usually, we want to minimise fuel consumption
    The resulting optimisation problem is hard:
    multimodal objective function,
    highly nonlinear,
    highly dimensional.
    Traditionally tackled by teams of human experts
    Francesco Biscani & Dario Izzo A Snake in Space

    View full-size slide

  8. Introduction
    PaGMO
    Astropy
    About us
    Projects: PaGMO and Astropy
    Background and context
    Example: Messenger’s path to Mercury
    Francesco Biscani & Dario Izzo A Snake in Space

    View full-size slide

  9. Introduction
    PaGMO
    Astropy
    About us
    Projects: PaGMO and Astropy
    Background and context
    Bio-inspired algorithms and Island Model
    A class of optimisation algorithms inspired by biological
    processes:
    genetic algorithms,
    differential evolution,
    ant-colony optimisation,
    artificial bee-colony optimisation,
    . . .
    They can deliver good performance on hard problems
    Island model: different algorithms run in parallel on the same
    optimisation problem, periodically exchanging information
    Francesco Biscani & Dario Izzo A Snake in Space

    View full-size slide

  10. Introduction
    PaGMO
    Astropy
    History
    Interlude
    Architectural overview
    History of PaGMO
    Origin: C/C++ research code (∼2005–2008)
    Initial framework: C++ OO hierarchy + Python bindings
    (2008–2009)
    parallelisation capabilities limited to local threads
    non-extensible Python bindings (PyGMO)
    open-sourced, GIT repository
    extensive use for research purposes
    Architectural overhaul and first GSoC (2010–2011)
    parallelisation capabilities extended to clusters
    fully-extensible Python bindings
    Accretion of features and maturity (2012–2014)
    bindings for many open-source optimisation libraries (SciPy,
    NLopt, GSL, Ipopt, etc.)
    implementation of many state-of-the-art algorithms
    second GSoC (2013)
    website, documentation, tutorials, etc.
    Francesco Biscani & Dario Izzo A Snake in Space

    View full-size slide

  11. Introduction
    PaGMO
    Astropy
    History
    Interlude
    Architectural overview
    Interlude: the Pros and Cons of Scientific Computing
    The Good:
    emphasis on correctness and reproducibility
    powerful driver for innovations in high-performance computing
    The Ugly:
    most scientists are not trained in software engineering
    code is often written with a one-paper-horizon mindset
    code often abandoned after use, wheel re-invention
    legacy code/tools
    What scientists want:
    performance
    interactivity and exploratory computing
    reproducibility and the ability to inspect and modify
    Francesco Biscani & Dario Izzo A Snake in Space

    View full-size slide

  12. Introduction
    PaGMO
    Astropy
    History
    Interlude
    Architectural overview
    The PaGMO framework
    Clear separation of concepts:
    optimisation algorithm and problem abstract classes
    the abstract island class includes a problem, an algorithm and
    a population of candidate solutions, and a virtual evolve()
    method that dispatches the optimisation (to a thread, or a
    process on another machine)
    an archipelago is a collection of islands linked by a topology,
    across which inter-island communication happens
    All the base classes are extensible:
    define your own problems, algorithms, islands and topologies
    the classes can be extended from C++ or Python
    large collection of algorithms (∼50) and problems already
    implemented
    Francesco Biscani & Dario Izzo A Snake in Space

    View full-size slide

  13. Introduction
    PaGMO
    Astropy
    History
    Interlude
    Architectural overview
    The C++ ⇐⇒ Python bridge
    Implemented via Boost.Python
    Challenging, non-trivial issues:
    serialisation across language boundaries involving virtual
    classes, base pointers, etc.
    extension from Python of C++ base classes
    working around some of Python’s limitation wrt parallel
    programming (GIL)
    Example: local parallelisation of custom problems/algorithms
    defined from Python
    Problem: cannot call the Python interpreter from multiple
    threads at the same time
    Solution: offload the computation to a separate process
    (multiprocessing), use the threads only for communication
    Francesco Biscani & Dario Izzo A Snake in Space

    View full-size slide

  14. Introduction
    PaGMO
    Astropy
    History
    Interlude
    Architectural overview
    Leveraging Python’s strengths
    Reliance on the scientific Python stack (NumPy, SciPy,
    matplotlib, IPython, etc.)
    IPython notebooks for interactivity and exploratory computing
    matplotlib for trajectory visualisation
    NumPy for crunching through the results
    SciPy optimisation algorithms usable from the framework
    High-performance and parallel computing
    multiprocessing frees us from the GIL shackles
    IPython cluster processing
    Francesco Biscani & Dario Izzo A Snake in Space

    View full-size slide

  15. Introduction
    PaGMO
    Astropy
    History
    Interlude
    Architectural overview
    Some highlights
    Selection of NEAs for future human missions (in collaboration
    with NASA JPL)
    Evolution of neural networks for autonomous Martian rovers
    (University of Plymouth)
    Widely used in GTOC by the ACT
    Francesco Biscani & Dario Izzo A Snake in Space

    View full-size slide

  16. Introduction
    PaGMO
    Astropy
    Astropy: overview
    NumPy as only dependency
    Works on Linux, OSX, Windows
    Core functionality:
    Units, quantities, constants
    Celestial coordinates representation and transformation
    Handling of FITS files
    Convolution/filtering utilities
    Cosmological calculations
    Model fitting
    Astrostatistics
    . . .
    Community consensus for inclusion of new functionality
    Wide selection of affiliated packages
    Francesco Biscani & Dario Izzo A Snake in Space

    View full-size slide

  17. Introduction
    PaGMO
    Astropy
    Astropy: code
    Overwhelmingly Python
    Heavy reliance on NumPy (e.g., the Table class)
    Bits of Fortran, C and Cython for performance (possibly
    Numba in the future)
    Interoperability with existing Astronomical software (e.g.,
    montage-wrapper)
    Development with the GitHub workflow
    Well-tested and documented (e.g., Travis)
    Francesco Biscani & Dario Izzo A Snake in Space

    View full-size slide

  18. Introduction
    PaGMO
    Astropy
    Pretty picture!
    Francesco Biscani & Dario Izzo A Snake in Space

    View full-size slide