%%async_run: an IPython notebook extension for asynchronous cell execution

# `%%async_run`: an IPython notebook magic for asynchronous cell execution

The IPython `notebook` project is one the preferred tools of data scientists,
and it is nowadays the bastion for *Reproducible Research*.

In fact, notebooks are now used as in-browser IDE (*Integrated Development Environment*) to implement the whole data analysis process, along with the corresponding documentation.

However, since this kind of processes usually include heavy-weight computations,
it may likely happen that execution results get lost if something wrong happens, e.g. the connection to the
notebook server hangs or an accidental page refresh is issued.

To this end, `[%]%async_run` notebook line/cell magic to the rescue.

In this talk, I would like to talk about some of the technologies I played with since I decided to develop
this extension.
These technologies include **asynchronous I/O** libraries (e.g. `asyncio`, `tornado.websocket`),
**`multiprocessing`**, along with IPython `kernels` and `notebooks`.

During the talk, I would like to discuss pitfalls, failures, and adopted solutions (e.g. *namespace management
among processes*) , aiming at getting as many feedbacks as possible from the community.

A general introduction to the actual state-of-the-art of the **Jupyter** projects (an libraries) will be
presented as well, in order to help those who are willing to know some more details about the internals of
IPython.

Transcript

1. [%]%async_run an IPython notebook* magic for asynchronous (code) cell execution Fondazione Bruno Kessler (FBK) - Trento, IT vmaggio@fbk.eu Valerio Maggio @leriomaggio
2. Jupyter Notebook
3. Reproducible Research Open Science and Open Code
4. Motivations • Sometimes it may be required to apply for heavy computations in IPython notebooks • computationally intensive code cells • Moreover, sometimes may be required that this computation is actually executed on a remote server machine • reminder: Jupyter Notebook Server • In the general case, this could work… but since…
5. –Murphy’s Law, 1952 “Anything that can possibly go wrong, does.”
6. Main Goal (for this one-weekend hack project) Try to define a strategy to cope with this kind of situation keeping the following requirements in mind: • Allow the execution on a remote machine (also) • Avoid the client machine to busy waiting • Keep the interactivity of the notebook as much as possible
8. [%]%async_run an IPython notebook* magic for asynchronous (code) cell execution What I learned during my adventures in the 
 world of Jupyter, Multiprocessing and Asynchronous I/O
9. IPython Magics (since IPython 3.x) • IPython has a system of commands we call magics • provide effectively a mini command language that is orthogonal to the syntax of Python • easily extensible by the user with new commands. • Magics are meant to be typed interactively • i.e. command-line conventions • e.g. whitespace for separating arguments, dashes for options. • Magics come in two kinds: • Line magics: prepended by one % character • Cell magics: two percent characters as a marker (%%)
10. Activate matplotlib inline-backend to have charts displayed inline with notebook cells
11. [%]%timeit Line Magic Cell Magic
12. Custom Magics: how to
16. Notebook Data Format
17. Notebook Data Format
21. First Idea (very early stage) Drawbacks: • No interactivity • No way to auto-refresh the content run the heavy computation (in some way) and use the write API to add a new cell to the notebook and that’s it.
22. Try to see if there’s any existing solution to this! Take away: avoid reinventing the wheel!
23. %run to the rescue (?) ipython.org/ipython-doc/3/interactive/magics.html#magic-run
24. %run
26. Test in the notebook
27. A bit more complicated
28. test it! Blocking Call No interactivity
29. runipy to the rescue (?) https://github.com/paulgb/runipy
30. runipy to the rescue (?) https://github.com/paulgb/runipy
31. A closer look
32. Notebook Runner
33. runipy features • (+) Notebook APIs • (+) Kernel Protocol Messaging • (+) Support for multiple document formats • nbformat.versions • (-) No interactivity • (-) No support for online/non-blocking execution • (~) No support for multi-processing
34. Idea: try to borrow some code from runipy and re-implement it as an IPython Magic (w/ steroids)
36. • But if you : Hangs on protocol communication and it has no link with the current shell
38. [%]%async_run an IPython notebook* magic for asynchronous (code) cell execution What I learned during my adventures in the 
 world of Jupyter, Multiprocessing and Asynchronous I/O
39. IPython is based on Tornado!
40. Reference
41. AsyncRun Magic
42. AsyncRun Server
43. AsyncRun Handler
44. Javascript (Browser) Side
45. WSConnector
47. AsyncRun Handler
48. Cell Execution
49. After Execution
50. WSConnector
52. Limitations and 
 concurrent.futures (work) • Pickle Serialisation Dependency • Major flaw of Python Multiprocessing Module • Try to use dill & multiprocess* • Improve the infrastructure to handle errors • not really handled yet • Allow the async execution of multiple cells at a time • revise multiprocessing architecture and dependencies
53. Demo Time even considering the aforementioned Murphy’s Laws :P https://github.com/leriomaggio/async-ipython-magic
54. Thanks a lot for your kind attention +ValerioMaggio vmaggio@fbk.eu it.linkedin.com/in/valeriomaggio @leriomaggio https://github.com/leriomaggio/async-ipython-magic