IPython: tools for the lifecycle of research computing

Transcript

1. IPython: tools for the lifecycle of research computing Fernando Pérez

   @fperez_org http://ipython.org SciPy 2012 July 19, 2012

2. The lifecycle of a scientific idea (the toy version) 1

   Individual exploratory work 2 Collaborative development 3 Production work (HPC, cloud, parallel) 4 Publication (with reproducible results!) 5 Education 6 Goto 1. The Problem with most tools Barriers and discontinuities in worfklow in between all the steps FP (UC Berkeley) IPython 7/19/2012 2 / 24

3. The lifecycle of a scientific idea (the toy version) 1

   Individual exploratory work 2 Collaborative development 3 Production work (HPC, cloud, parallel) 4 Publication (with reproducible results!) 5 Education 6 Goto 1. The Problem with most tools Barriers and discontinuities in worfklow in between all the steps FP (UC Berkeley) IPython 7/19/2012 2 / 24

4. IPython’s goal: Fluid transitions in all these steps

5. Demos

6. Important: The notebook format JSON but version control-friendly Easy for

   machine processing, fixable by hand if need be. Lots of hooks for metadata Produce Markdown, reST, L A TEX, HTML, etc... An open format for sharing, publishing and archiving executable computational work FP (UC Berkeley) IPython 7/19/2012 5 / 24

7. The Notebook: the tip of the spear FP (UC Berkeley)

   IPython 7/19/2012 6 / 24

8. The rest of the spear FP (UC Berkeley) IPython 7/19/2012

   7 / 24

9. An architecture for interactive computing FP (UC Berkeley) IPython 7/19/2012

   8 / 24

10. Text console with visualization

11. Qt console: inline plots, html, multiline editing, ... Evan Patterson

    (Enthought)

12. Browser-based notebook Brian Granger, James Gao (Berkeley), rest of the

    team

13. High-level parallel computing Min Ragan-Kelley, Brian Granger

14. A growing ecosystem of applications

15. Microsoft Visual Studio 2010 integrated console Dino Viehland and Shahrokh

    Mortazavi; http://pytools.codeplex.com

16. A vim client to control an IPython kernel/console Paul Ivanov,

    https://github.com/ivanov/vim-ipython

17. Notebooks on Windows Azure Cloud Shahrokh Mortazavi, B.G., F.P.: http://bit.ly/JQeojD.

18. Star Cluster: IPython parallel+Notebook on Amazon EC2 Justin Riley: http://web.mit.edu/star/cluster

19. One-click single notebook on Amazon EC2 Carl Smith: https://notebookcloud.appspot.com.

20. Canopy: Enthought’s new Python environment (EPD 8)

21. An Emacs Notebook Client! Takafumi Arakaki: http://tkf.github.com/emacs-ipython-notebook.

22. vIPer: a dedicated notebook client Damián Avila: https://github.com/damianavila/vIPer

23. (Incomplete) Cast of Characters Brian Granger - Physics, Cal State

    San Luis Obispo Min Ragan-Kelley - Nuclear Engineering, UC Berkeley Matthias Bussonnier - Physics, Institut Curie, Paris Jonathan March- Enthought Thomas Kluyver - Biology, U. Sheffield Jörgen Stenarson - Elect. Engineering, Sweden. Paul Ivanov - Neuroscience, UC Berkeley. Robert Kern - Enthought Evan Patterson - Physics, Caltech/Enthought Brad Froehle - Mathematics, UC Berkeley Stefan van der Walt - UC Berkeley John Hunter - TradeLink Securities, Chicago. Prabhu Ramachandran - Aerospace Engineering, IIT Bombay. Satra Ghosh- MIT Neuroscience Gaël Varoquaux - Neurospin (Orsay, France) Ville Vainio - CS, Tampere University of Technology, Finland Barry Wark - Neuroscience, U. Washington. Ondrej Certik - Physics, U Nevada Reno Darren Dale - Cornell Justin Riley - MIT Mark Voorhies - UC San Francisco Nicholas Rougier - INRIA Nancy Grand Est Thomas Spura - Fedora project Many more! (~150 commit authors)

24. Support Thank you! Enthought, Austin, TX: Lots! Microsoft: WinHPC support,

    Visual Studio integration, Azure (thanks to Shahrokh Mortazavi). DoD/DRC Inc: funding through Sept. 2012 (thanks to Jose Unpingco and Chris Keees). NIH: via NiPy grant NSF: via Sage compmath grant Google: summer of code 2005, 2010. Tech-X Corporation, Boulder, CO: Parallel/notebook (previous versions) We REALLY need more stable funding...

25. Support Thank you! Enthought, Austin, TX: Lots! Microsoft: WinHPC support,

    Visual Studio integration, Azure (thanks to Shahrokh Mortazavi). DoD/DRC Inc: funding through Sept. 2012 (thanks to Jose Unpingco and Chris Keees). NIH: via NiPy grant NSF: via Sage compmath grant Google: summer of code 2005, 2010. Tech-X Corporation, Boulder, CO: Parallel/notebook (previous versions) We REALLY need more stable funding...

26. Plug: Reproducibility in Computational and Experimental Mathematics December 10-14 2012,

    Brown, Rhode Island. http://icerm.brown.edu/tw12-5-rcem