Upgrade to Pro — share decks privately, control downloads, hide ads and more …

IPython: Interact, Compute, Understand

Fernando Perez
September 14, 2012

IPython: Interact, Compute, Understand

Slides for my presentation at the first PyData San Francisco meetup, held at the Hearsay Social HQ.

Fernando Perez

September 14, 2012
Tweet

More Decks by Fernando Perez

Other Decks in Technology

Transcript

  1. Briefly, who am I? CU Boulder, Particle Physics (Lattice QCD),

    started IPython. CU Applied Mathematics (fast algorithms for PDEs in integral formulations). All code in Python UC Berkeley Neuroscience: analysis tools. NiPy, IPython. Good scientific computing needs good tools.
  2. I is for interactive... In scientific computing, we typically don’t

    know what we’re doing. Scientific computing ⇔ Exploratory computing
  3. I is for interactive... In scientific computing, we typically don’t

    know what we’re doing. Scientific computing ⇔ Exploratory computing
  4. I is for interactive... In scientific computing, we typically don’t

    know what we’re doing. Scientific computing ⇔ Exploratory computing
  5. Interactive systems Features Execute/explore cycle instead of edit/compile/run Rich Libraries

    Plotting and data visualization Examples Mathematica/Maple: symbolic, now numerics... IDL/Matlab: numerics, image processing, ... Unix system shell: file management/text processing.
  6. Interactive systems Features Execute/explore cycle instead of edit/compile/run Rich Libraries

    Plotting and data visualization Examples Mathematica/Maple: symbolic, now numerics... IDL/Matlab: numerics, image processing, ... Unix system shell: file management/text processing.
  7. The lifecycle of a scientific idea (the toy version) 1

    Individual exploratory work 2 Collaborative development 3 Production work (HPC, cloud, parallel) 4 Publication (with reproducible results!) 5 Education 6 Goto 1. The Problem with most tools Barriers and discontinuities in worfklow in between all the steps FP (UC Berkeley) IPython: Interact, Compute, Understand 93/13/2012 6 / 29
  8. The lifecycle of a scientific idea (the toy version) 1

    Individual exploratory work 2 Collaborative development 3 Production work (HPC, cloud, parallel) 4 Publication (with reproducible results!) 5 Education 6 Goto 1. The Problem with most tools Barriers and discontinuities in worfklow in between all the steps FP (UC Berkeley) IPython: Interact, Compute, Understand 93/13/2012 6 / 29
  9. Core IPython: what makes the “IPython VM” Interactive Python on

    steroids Tab completion, object introspection, better tracebacks, syntax shortcuts, autoindent, history management, output caching, ... The %magic command system Orthogonal namespace to Python’s Command-line semantics: no parens, no quotes, no commas, –dash options, ... Fully extensible and customizable, clean API. OS Integration Your filesystem is yours !echo “to the shell” x = !command_output x = f.lower(); !mv $f $x # interpolate Python vars Software development %run, %timeit, %debug, %prun GUI support: %pylab, mayavi, most Qt/Wx/GTK apps: %gui Easy embedding: “from IPython import embed; embed()”
  10. The mental model in IPython.parallel Client Results gathering, exploration, analysis,

    insight. You, and your colleagues. Transient connections to cluster. Cluster Persistent parallel resource N IPython engines working for you Dynamic and fault-tolerant Flexible execution semantics: Direct (multiplexed) execution Dynamic load balancing
  11. The big-picture view of IPython Core IPython: a powerful REPL

    (Read-Eval Print Loop, aka interactive shell). Classic Python shell + OS access + GUI support + steroids A protocol Abstract the REPL over the network, use ZeroMQ The Qt console A terminal emulator for modern analysis workflows The Notebook Non-linear workflow beyond the terminal Capture interactive work with code, results, rich text and media Work in the browser. Parallel computing One IPython, many IPythons... Parallel resources supporting insight. FP (UC Berkeley) IPython: Interact, Compute, Understand 93/13/2012 16 / 29
  12. Important: The notebook format JSON but version control-friendly Easy for

    machine processing, fixable by hand if need be. Lots of hooks for metadata Produce Markdown, reST, L A TEX, HTML, etc... An open format for sharing, publishing and archiving executable computational work FP (UC Berkeley) IPython: Interact, Compute, Understand 93/13/2012 17 / 29
  13. (Incomplete) Cast of Characters Brian Granger - Physics, Cal State

    San Luis Obispo Min Ragan-Kelley - Nuclear Engineering, UC Berkeley Matthias Bussonnier - Physics, Institut Curie, Paris Jonathan March- Enthought Thomas Kluyver - Biology, U. Sheffield Jörgen Stenarson - Elect. Engineering, Sweden. Paul Ivanov - Neuroscience, UC Berkeley. Robert Kern - Enthought Evan Patterson - Physics, Caltech/Enthought Brad Froehle - Mathematics, UC Berkeley Stefan van der Walt - UC Berkeley John Hunter - TradeLink Securities, Chicago. Prabhu Ramachandran - Aerospace Engineering, IIT Bombay. Satra Ghosh- MIT Neuroscience Gaël Varoquaux - Neurospin (Orsay, France) Ville Vainio - CS, Tampere University of Technology, Finland Barry Wark - Neuroscience, U. Washington. Ondrej Certik - Physics, U Nevada Reno Darren Dale - Cornell Justin Riley - MIT Mark Voorhies - UC San Francisco Nicholas Rougier - INRIA Nancy Grand Est Thomas Spura - Fedora project Many more! (~150 commit authors)
  14. Support Thank you! Enthought, Austin, TX: Lots! Microsoft: WinHPC support,

    Visual Studio integration, Azure (thanks to Shahrokh Mortazavi). DoD/DRC Inc: funding through Sept. 2012 (thanks to Jose Unpingco and Chris Keees). NIH: via NiPy grant NSF: via Sage compmath grant Google: summer of code 2005, 2010. Tech-X Corporation, Boulder, CO: Parallel/notebook (previous versions) We REALLY need more stable funding...
  15. Support Thank you! Enthought, Austin, TX: Lots! Microsoft: WinHPC support,

    Visual Studio integration, Azure (thanks to Shahrokh Mortazavi). DoD/DRC Inc: funding through Sept. 2012 (thanks to Jose Unpingco and Chris Keees). NIH: via NiPy grant NSF: via Sage compmath grant Google: summer of code 2005, 2010. Tech-X Corporation, Boulder, CO: Parallel/notebook (previous versions) We REALLY need more stable funding...