How a Notebook changed the world of Science!

Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

Who am I? -> Software Engineer @ Semut.io -> Electrical Engineer from IIT Mandi -> Lead Developer @ EinsteinPy -> Managing Member @ PSF -> Love being in Mountains of Himachal Pradesh (India)

Slide 4

Slide 4 text

Let’s start with what went wrong?

Slide 5

Slide 5 text

What is it? -> Published in Non Peer Reviewed Issue of American Economic Review. -> Cited by politicians worldwide in debates to prove effectiveness of austerity in ﬁscal policies for debt burdened economies. -> When “Gross external debt reaches 60 percent of GDP", a country's annual growth declined by two percent, and "for levels of external debt in excess of 90 percent" GDP growth was "roughly cut in half.” -> Proved Wrong!

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

Catch? Science needs explanation. Papers can be hard to understand. Excel is not made for this. You can’t document code, logic and results in excel!

Slide 8

Slide 8 text

More Issues with using Excel for Science

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

Is there a better way? -> Is there a better way to have the text, analysis, results, code, plots, comments in one place? -> Is it possible to write it sequentially such that anyone going through the code/paper/book could not just understand it, but run it, reproduce results, ﬁnd ﬂaws and suggest enhancements 1000X faster? -> Is it possible to distribute the results properly?

Slide 11

Slide 11 text

What would be a better way? -> Excel users are people avoiding hardcore programming languages. -> Easy interface to write logic -> Low Cognitive Complexity -> Easy Syntax (Preferably like English) [Think BQL (Bloomberg Query Language)]

Slide 12

Slide 12 text

Python stands out! -> Easy to grab! -> Easy interface to write logic -> Low Cognitive Complexity -> Mostly like writing english. -> Out of the box support, extremely friendly community. -> Democratic

Slide 13

Slide 13 text

Results?

Slide 14

Slide 14 text

Rise of iPython

Slide 15

Slide 15 text

iPython? -> First version in 2001 (Started with just 259 lines) -> Inbuilt support for parallelization came in 2004 -> Support for running the code in remote cluster from the shell. -> Inspired by Mathematica -> Boon for scientiﬁc computing

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

Rise to fame!

Slide 18

Slide 18 text

What are notebooks? -> Notebook is a rather old concept. -> Sequentially written logic -> Interactive -> Saves State (like a paper notebook does)

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

Benefits of using iPython -> obj.[tab] , obj? , obj?? -> files = !ls , !wget $url -> iPython Magic: - %run script.py (-p -> profile, -t -> time) - %debug (jump in after an exception) - %lsmagic (See the rest of magics)

Slide 21

Slide 21 text

Some myths -> There’s no support -> Won’t be free forever -> Free software has bugs

Slide 22

Slide 22 text

Some myths -> There’s no support Enthought, Continuum Analytics -> Won’t be free forever -> Free software has bugs

Slide 23

Slide 23 text

Some myths -> There’s no support Enthought, Continuum Analytics -> Won’t be free forever Free software Belongs to the community -> Free software has bugs

Slide 24

Slide 24 text

Some myths -> There’s no support Enthought, Continuum Analytics -> Won’t be free forever Free software Belongs to the community -> Free software has bugs Naturally!

Slide 25

Slide 25 text

And it happened

Slide 26

Slide 26 text

Jupyter or Jupiter?

Slide 27

Slide 27 text

What is Jupyter?

Slide 28

Slide 28 text

Jupyter : An Ecosystem -> JupyterLab -> Slides/Documents -> Write Books (O’Reilly) - jupyterbook -> JupyterHub -> Collab Notebooks, binder

Slide 29

Slide 29 text

Jupyter : A way to teach in classrooms! -> Easy to create interactive tutorials which lets students play around with the code. -> Perform Live Coding, share lecture notes and materials -> Grade homeworks -> No need to run every script supplied by students

Slide 30

Slide 30 text

Project Jupyter -> Separation of the language agnostic components - Jupyter : protocol, format, multi-user server - iPython : Jupyter Kernel, interactive python -> Jupyter Kernels: Languages which can be used in notebook: ~ 100 programming languages.

Slide 31

Slide 31 text

Notebook Extensions -> Add ons to extend functionality. Much like VS Code extensions. -> VS Code supports Jupyter -> Written in JavaScript, send browser notiﬁcations, autoformat code. There are immense possibilities.

Slide 32

Slide 32 text

Widgets -> Interact with the code output! -> Have slide bars, text boxes, inputs -> Like mini-GUIs -> Very helpful when working with hyper parameters. -> You can write one for yourself!

Slide 33

Slide 33 text

Widgets

Slide 34

Slide 34 text

Widgets

Slide 35

Slide 35 text

Version Control in Notebooks -> The notebooks have extension .ipynb but are plain text ﬁles and are represented in JSON. -> The diffs used to be pretty large earlier even when nothing changed. Storing output is another issue. -> It’s getting better. With nbdime, it’s much easier to merge and compare now.

Slide 36

Slide 36 text

Using Notebooks in Production!

Slide 37

Slide 37 text

Issues with Jupyter Notebooks -> Hidden states in the Notebooks

Slide 38

Slide 38 text

Issues with Jupyter Notebooks -> Hidden states in the Notebooks

Slide 39

Slide 39 text

Issues with Jupyter Notebooks Notebooks are great for iterative development BUT Notebooks are *very* dangerous unless you run each cell only ONCE in CORRECT ORDER.

Slide 40

Slide 40 text

A way to solve this... The %history magic! But you shouldn’t have to run a magic to ﬁnd the state!

Slide 41

Slide 41 text

Now take this!

Slide 42

Slide 42 text

Problem number 2 The ability to run a code in a non sequential arbitrary order is counter intuitive to most programmers out there. Can be daunting for beginners!

Slide 43

Slide 43 text

Notebooks in Cloud -> Notebooks as a service is a cool new thing! -> Azure Notebooks, Collab Notebooks and what not. -> You can create your own notebook service!

Slide 44

Slide 44 text

How does a Jupyter Notebook Work? Credits: Carol Willing

Slide 45

Slide 45 text

What does JupyterHub do? -> Manages Authentication -> Spawns single-user notebook servers on demand -> Gives a user their complete notebook server! -> Hub and server are different entities.

Slide 46

Slide 46 text

JupyterHub

Slide 47

Slide 47 text

Parts of JupyterHub -> The Hub: User Database, Auth, and Spawner -> Users and their individual notebook servers -> Conﬁgurable HTTP Proxy The auth supports OAuth, pem etc. Deploy: https://github.com/jupyterhub/jupyterhub-deploy-docker

Slide 48

Slide 48 text

Real Time Collaboration Jupyter RTC: https://jupyter-rtc.readthedocs.io/ This works directly with JupyterLab

Slide 49

Slide 49 text

Summary -> Jupyter / iPython is a useful tool, not only for coding but also for teaching, sharing, documenting, publishing! -> We don’t have to throw away previous work in different languages, now we can integrate them. -> Jupyter is gaining relevance in Open Science, Finance, Music, Teaching. We must go further!