Upgrade to Pro — share decks privately, control downloads, hide ads and more …

IPython: tools for the lifecycle of research computing

IPython: tools for the lifecycle of research computing

Talk delivered at SciPy 2012 in Austin, TX.
Video: http://www.youtube.com/watch?v=iwVvqwLDsJo.

Note: the middle section of the talk consisted of interactive demos with the IPython notebook.

95198572b00e5fbcd97fb5315215bf7a?s=128

Fernando Perez

July 19, 2012
Tweet

Transcript

  1. IPython: tools for the lifecycle of research computing Fernando Pérez

    @fperez_org http://ipython.org SciPy 2012 July 19, 2012
  2. The lifecycle of a scientific idea (the toy version) 1

    Individual exploratory work 2 Collaborative development 3 Production work (HPC, cloud, parallel) 4 Publication (with reproducible results!) 5 Education 6 Goto 1. The Problem with most tools Barriers and discontinuities in worfklow in between all the steps FP (UC Berkeley) IPython 7/19/2012 2 / 24
  3. The lifecycle of a scientific idea (the toy version) 1

    Individual exploratory work 2 Collaborative development 3 Production work (HPC, cloud, parallel) 4 Publication (with reproducible results!) 5 Education 6 Goto 1. The Problem with most tools Barriers and discontinuities in worfklow in between all the steps FP (UC Berkeley) IPython 7/19/2012 2 / 24
  4. IPython’s goal: Fluid transitions in all these steps

  5. Demos

  6. Important: The notebook format JSON but version control-friendly Easy for

    machine processing, fixable by hand if need be. Lots of hooks for metadata Produce Markdown, reST, L A TEX, HTML, etc... An open format for sharing, publishing and archiving executable computational work FP (UC Berkeley) IPython 7/19/2012 5 / 24
  7. The Notebook: the tip of the spear FP (UC Berkeley)

    IPython 7/19/2012 6 / 24
  8. The rest of the spear FP (UC Berkeley) IPython 7/19/2012

    7 / 24
  9. An architecture for interactive computing FP (UC Berkeley) IPython 7/19/2012

    8 / 24
  10. Text console with visualization

  11. Qt console: inline plots, html, multiline editing, ... Evan Patterson

    (Enthought)
  12. Browser-based notebook Brian Granger, James Gao (Berkeley), rest of the

    team
  13. High-level parallel computing Min Ragan-Kelley, Brian Granger

  14. A growing ecosystem of applications

  15. Microsoft Visual Studio 2010 integrated console Dino Viehland and Shahrokh

    Mortazavi; http://pytools.codeplex.com
  16. A vim client to control an IPython kernel/console Paul Ivanov,

    https://github.com/ivanov/vim-ipython
  17. Notebooks on Windows Azure Cloud Shahrokh Mortazavi, B.G., F.P.: http://bit.ly/JQeojD.

  18. Star Cluster: IPython parallel+Notebook on Amazon EC2 Justin Riley: http://web.mit.edu/star/cluster

  19. One-click single notebook on Amazon EC2 Carl Smith: https://notebookcloud.appspot.com.

  20. Canopy: Enthought’s new Python environment (EPD 8)

  21. An Emacs Notebook Client! Takafumi Arakaki: http://tkf.github.com/emacs-ipython-notebook.

  22. vIPer: a dedicated notebook client Damián Avila: https://github.com/damianavila/vIPer

  23. (Incomplete) Cast of Characters Brian Granger - Physics, Cal State

    San Luis Obispo Min Ragan-Kelley - Nuclear Engineering, UC Berkeley Matthias Bussonnier - Physics, Institut Curie, Paris Jonathan March- Enthought Thomas Kluyver - Biology, U. Sheffield Jörgen Stenarson - Elect. Engineering, Sweden. Paul Ivanov - Neuroscience, UC Berkeley. Robert Kern - Enthought Evan Patterson - Physics, Caltech/Enthought Brad Froehle - Mathematics, UC Berkeley Stefan van der Walt - UC Berkeley John Hunter - TradeLink Securities, Chicago. Prabhu Ramachandran - Aerospace Engineering, IIT Bombay. Satra Ghosh- MIT Neuroscience Gaël Varoquaux - Neurospin (Orsay, France) Ville Vainio - CS, Tampere University of Technology, Finland Barry Wark - Neuroscience, U. Washington. Ondrej Certik - Physics, U Nevada Reno Darren Dale - Cornell Justin Riley - MIT Mark Voorhies - UC San Francisco Nicholas Rougier - INRIA Nancy Grand Est Thomas Spura - Fedora project Many more! (~150 commit authors)
  24. Support Thank you! Enthought, Austin, TX: Lots! Microsoft: WinHPC support,

    Visual Studio integration, Azure (thanks to Shahrokh Mortazavi). DoD/DRC Inc: funding through Sept. 2012 (thanks to Jose Unpingco and Chris Keees). NIH: via NiPy grant NSF: via Sage compmath grant Google: summer of code 2005, 2010. Tech-X Corporation, Boulder, CO: Parallel/notebook (previous versions) We REALLY need more stable funding...
  25. Support Thank you! Enthought, Austin, TX: Lots! Microsoft: WinHPC support,

    Visual Studio integration, Azure (thanks to Shahrokh Mortazavi). DoD/DRC Inc: funding through Sept. 2012 (thanks to Jose Unpingco and Chris Keees). NIH: via NiPy grant NSF: via Sage compmath grant Google: summer of code 2005, 2010. Tech-X Corporation, Boulder, CO: Parallel/notebook (previous versions) We REALLY need more stable funding...
  26. Plug: Reproducibility in Computational and Experimental Mathematics December 10-14 2012,

    Brown, Rhode Island. http://icerm.brown.edu/tw12-5-rcem