IPython: tools for the lifecycle of research computing

Slide 1

Slide 1 text

IPython: tools for the lifecycle of research computing Fernando Pérez @fperez_org http://ipython.org SciPy 2012 July 19, 2012

Slide 2

Slide 2 text

The lifecycle of a scientiﬁc idea (the toy version) 1 Individual exploratory work 2 Collaborative development 3 Production work (HPC, cloud, parallel) 4 Publication (with reproducible results!) 5 Education 6 Goto 1. The Problem with most tools Barriers and discontinuities in worfklow in between all the steps FP (UC Berkeley) IPython 7/19/2012 2 / 24

Slide 3

Slide 3 text

Slide 4

Slide 4 text

IPython’s goal: Fluid transitions in all these steps

Slide 5

Slide 5 text

Demos

Slide 6

Slide 6 text

Important: The notebook format JSON but version control-friendly Easy for machine processing, ﬁxable by hand if need be. Lots of hooks for metadata Produce Markdown, reST, L A TEX, HTML, etc... An open format for sharing, publishing and archiving executable computational work FP (UC Berkeley) IPython 7/19/2012 5 / 24

Slide 7

Slide 7 text

The Notebook: the tip of the spear FP (UC Berkeley) IPython 7/19/2012 6 / 24

Slide 8

Slide 8 text

The rest of the spear FP (UC Berkeley) IPython 7/19/2012 7 / 24

Slide 9

Slide 9 text

An architecture for interactive computing FP (UC Berkeley) IPython 7/19/2012 8 / 24

Slide 10

Slide 10 text

Text console with visualization

Slide 11

Slide 11 text

Qt console: inline plots, html, multiline editing, ... Evan Patterson (Enthought)

Slide 12

Slide 12 text

Browser-based notebook Brian Granger, James Gao (Berkeley), rest of the team

Slide 13

Slide 13 text

High-level parallel computing Min Ragan-Kelley, Brian Granger

Slide 14

Slide 14 text

A growing ecosystem of applications

Slide 15

Slide 15 text

Microsoft Visual Studio 2010 integrated console Dino Viehland and Shahrokh Mortazavi; http://pytools.codeplex.com

Slide 16

Slide 16 text

A vim client to control an IPython kernel/console Paul Ivanov, https://github.com/ivanov/vim-ipython

Slide 17

Slide 17 text

Notebooks on Windows Azure Cloud Shahrokh Mortazavi, B.G., F.P.: http://bit.ly/JQeojD.

Slide 18

Slide 18 text

Star Cluster: IPython parallel+Notebook on Amazon EC2 Justin Riley: http://web.mit.edu/star/cluster

Slide 19

Slide 19 text

One-click single notebook on Amazon EC2 Carl Smith: https://notebookcloud.appspot.com.

Slide 20

Slide 20 text

Canopy: Enthought’s new Python environment (EPD 8)

Slide 21

Slide 21 text

An Emacs Notebook Client! Takafumi Arakaki: http://tkf.github.com/emacs-ipython-notebook.

Slide 22

Slide 22 text

vIPer: a dedicated notebook client Damián Avila: https://github.com/damianavila/vIPer

Slide 23

Slide 23 text

(Incomplete) Cast of Characters Brian Granger - Physics, Cal State San Luis Obispo Min Ragan-Kelley - Nuclear Engineering, UC Berkeley Matthias Bussonnier - Physics, Institut Curie, Paris Jonathan March- Enthought Thomas Kluyver - Biology, U. Sheﬃeld Jörgen Stenarson - Elect. Engineering, Sweden. Paul Ivanov - Neuroscience, UC Berkeley. Robert Kern - Enthought Evan Patterson - Physics, Caltech/Enthought Brad Froehle - Mathematics, UC Berkeley Stefan van der Walt - UC Berkeley John Hunter - TradeLink Securities, Chicago. Prabhu Ramachandran - Aerospace Engineering, IIT Bombay. Satra Ghosh- MIT Neuroscience Gaël Varoquaux - Neurospin (Orsay, France) Ville Vainio - CS, Tampere University of Technology, Finland Barry Wark - Neuroscience, U. Washington. Ondrej Certik - Physics, U Nevada Reno Darren Dale - Cornell Justin Riley - MIT Mark Voorhies - UC San Francisco Nicholas Rougier - INRIA Nancy Grand Est Thomas Spura - Fedora project Many more! (~150 commit authors)

Slide 24

Slide 24 text

Support Thank you! Enthought, Austin, TX: Lots! Microsoft: WinHPC support, Visual Studio integration, Azure (thanks to Shahrokh Mortazavi). DoD/DRC Inc: funding through Sept. 2012 (thanks to Jose Unpingco and Chris Keees). NIH: via NiPy grant NSF: via Sage compmath grant Google: summer of code 2005, 2010. Tech-X Corporation, Boulder, CO: Parallel/notebook (previous versions) We REALLY need more stable funding...

Slide 25

Slide 25 text

Slide 26

Slide 26 text

Plug: Reproducibility in Computational and Experimental Mathematics December 10-14 2012, Brown, Rhode Island. http://icerm.brown.edu/tw12-5-rcem