Slide 1

Slide 1 text

IPython: Interact, Compute, Understand Fernando Pérez @fperez_org http://ipython.org PyData San Francisco Sept 13, 2012

Slide 2

Slide 2 text

Briefly, who am I? CU Boulder, Particle Physics (Lattice QCD), started IPython. CU Applied Mathematics (fast algorithms for PDEs in integral formulations). All code in Python UC Berkeley Neuroscience: analysis tools. NiPy, IPython. Good scientific computing needs good tools.

Slide 3

Slide 3 text

Why IPython? (something other than “I’d rather not finish my dissertation”)

Slide 4

Slide 4 text

Why IPython? (something other than “I’d rather not finish my dissertation”)

Slide 5

Slide 5 text

I is for interactive... In scientific computing, we typically don’t know what we’re doing. Scientific computing ⇔ Exploratory computing

Slide 6

Slide 6 text

I is for interactive... In scientific computing, we typically don’t know what we’re doing. Scientific computing ⇔ Exploratory computing

Slide 7

Slide 7 text

I is for interactive... In scientific computing, we typically don’t know what we’re doing. Scientific computing ⇔ Exploratory computing

Slide 8

Slide 8 text

Interactive systems Features Execute/explore cycle instead of edit/compile/run Rich Libraries Plotting and data visualization Examples Mathematica/Maple: symbolic, now numerics... IDL/Matlab: numerics, image processing, ... Unix system shell: file management/text processing.

Slide 9

Slide 9 text

Interactive systems Features Execute/explore cycle instead of edit/compile/run Rich Libraries Plotting and data visualization Examples Mathematica/Maple: symbolic, now numerics... IDL/Matlab: numerics, image processing, ... Unix system shell: file management/text processing.

Slide 10

Slide 10 text

The lifecycle of a scientific idea (the toy version) 1 Individual exploratory work 2 Collaborative development 3 Production work (HPC, cloud, parallel) 4 Publication (with reproducible results!) 5 Education 6 Goto 1. The Problem with most tools Barriers and discontinuities in worfklow in between all the steps FP (UC Berkeley) IPython: Interact, Compute, Understand 93/13/2012 6 / 29

Slide 11

Slide 11 text

The lifecycle of a scientific idea (the toy version) 1 Individual exploratory work 2 Collaborative development 3 Production work (HPC, cloud, parallel) 4 Publication (with reproducible results!) 5 Education 6 Goto 1. The Problem with most tools Barriers and discontinuities in worfklow in between all the steps FP (UC Berkeley) IPython: Interact, Compute, Understand 93/13/2012 6 / 29

Slide 12

Slide 12 text

IPython’s goal: Fluid transitions in all these steps

Slide 13

Slide 13 text

Core IPython: what makes the “IPython VM” Interactive Python on steroids Tab completion, object introspection, better tracebacks, syntax shortcuts, autoindent, history management, output caching, ... The %magic command system Orthogonal namespace to Python’s Command-line semantics: no parens, no quotes, no commas, –dash options, ... Fully extensible and customizable, clean API. OS Integration Your filesystem is yours !echo “to the shell” x = !command_output x = f.lower(); !mv $f $x # interpolate Python vars Software development %run, %timeit, %debug, %prun GUI support: %pylab, mayavi, most Qt/Wx/GTK apps: %gui Easy embedding: “from IPython import embed; embed()”

Slide 14

Slide 14 text

An architecture for interactive computing FP (UC Berkeley) IPython: Interact, Compute, Understand 93/13/2012 9 / 29

Slide 15

Slide 15 text

Text console with visualization

Slide 16

Slide 16 text

Qt console: inline plots, html, multiline editing, ... Evan Patterson (Enthought)

Slide 17

Slide 17 text

Browser-based notebook Brian Granger, James Gao (Berkeley), rest of the team

Slide 18

Slide 18 text

High-level parallel computing Min Ragan-Kelley, Brian Granger

Slide 19

Slide 19 text

The mental model in IPython.parallel Client Results gathering, exploration, analysis, insight. You, and your colleagues. Transient connections to cluster. Cluster Persistent parallel resource N IPython engines working for you Dynamic and fault-tolerant Flexible execution semantics: Direct (multiplexed) execution Dynamic load balancing

Slide 20

Slide 20 text

Demos

Slide 21

Slide 21 text

The big-picture view of IPython Core IPython: a powerful REPL (Read-Eval Print Loop, aka interactive shell). Classic Python shell + OS access + GUI support + steroids A protocol Abstract the REPL over the network, use ZeroMQ The Qt console A terminal emulator for modern analysis workflows The Notebook Non-linear workflow beyond the terminal Capture interactive work with code, results, rich text and media Work in the browser. Parallel computing One IPython, many IPythons... Parallel resources supporting insight. FP (UC Berkeley) IPython: Interact, Compute, Understand 93/13/2012 16 / 29

Slide 22

Slide 22 text

Important: The notebook format JSON but version control-friendly Easy for machine processing, fixable by hand if need be. Lots of hooks for metadata Produce Markdown, reST, L A TEX, HTML, etc... An open format for sharing, publishing and archiving executable computational work FP (UC Berkeley) IPython: Interact, Compute, Understand 93/13/2012 17 / 29

Slide 23

Slide 23 text

A growing ecosystem of applications

Slide 24

Slide 24 text

Microsoft Visual Studio 2010 integrated console Dino Viehland and Shahrokh Mortazavi; http://pytools.codeplex.com

Slide 25

Slide 25 text

A vim client to control an IPython kernel/console Paul Ivanov, https://github.com/ivanov/vim-ipython

Slide 26

Slide 26 text

Notebooks on Windows Azure Cloud Shahrokh Mortazavi, B.G., F.P.: http://bit.ly/JQeojD.

Slide 27

Slide 27 text

Star Cluster: IPython parallel+Notebook on Amazon EC2 Justin Riley: http://web.mit.edu/star/cluster

Slide 28

Slide 28 text

One-click single notebook on Amazon EC2 Carl Smith: https://notebookcloud.appspot.com.

Slide 29

Slide 29 text

Canopy: Enthought’s new Python environment (EPD 8)

Slide 30

Slide 30 text

An Emacs Notebook Client! Takafumi Arakaki: http://tkf.github.com/emacs-ipython-notebook.

Slide 31

Slide 31 text

vIPer: a dedicated notebook client Damián Avila: https://github.com/damianavila/vIPer

Slide 32

Slide 32 text

(Incomplete) Cast of Characters Brian Granger - Physics, Cal State San Luis Obispo Min Ragan-Kelley - Nuclear Engineering, UC Berkeley Matthias Bussonnier - Physics, Institut Curie, Paris Jonathan March- Enthought Thomas Kluyver - Biology, U. Sheffield Jörgen Stenarson - Elect. Engineering, Sweden. Paul Ivanov - Neuroscience, UC Berkeley. Robert Kern - Enthought Evan Patterson - Physics, Caltech/Enthought Brad Froehle - Mathematics, UC Berkeley Stefan van der Walt - UC Berkeley John Hunter - TradeLink Securities, Chicago. Prabhu Ramachandran - Aerospace Engineering, IIT Bombay. Satra Ghosh- MIT Neuroscience Gaël Varoquaux - Neurospin (Orsay, France) Ville Vainio - CS, Tampere University of Technology, Finland Barry Wark - Neuroscience, U. Washington. Ondrej Certik - Physics, U Nevada Reno Darren Dale - Cornell Justin Riley - MIT Mark Voorhies - UC San Francisco Nicholas Rougier - INRIA Nancy Grand Est Thomas Spura - Fedora project Many more! (~150 commit authors)

Slide 33

Slide 33 text

Support Thank you! Enthought, Austin, TX: Lots! Microsoft: WinHPC support, Visual Studio integration, Azure (thanks to Shahrokh Mortazavi). DoD/DRC Inc: funding through Sept. 2012 (thanks to Jose Unpingco and Chris Keees). NIH: via NiPy grant NSF: via Sage compmath grant Google: summer of code 2005, 2010. Tech-X Corporation, Boulder, CO: Parallel/notebook (previous versions) We REALLY need more stable funding...

Slide 34

Slide 34 text

Support Thank you! Enthought, Austin, TX: Lots! Microsoft: WinHPC support, Visual Studio integration, Azure (thanks to Shahrokh Mortazavi). DoD/DRC Inc: funding through Sept. 2012 (thanks to Jose Unpingco and Chris Keees). NIH: via NiPy grant NSF: via Sage compmath grant Google: summer of code 2005, 2010. Tech-X Corporation, Boulder, CO: Parallel/notebook (previous versions) We REALLY need more stable funding...

Slide 35

Slide 35 text

John D. Hunter, 1968-2012: matplotlib.org Memorial fund: numfocus.org/johnhunter