Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How a Notebook changed the world of Science!

How a Notebook changed the world of Science!

Tech talk on Semut.io

Notebooks are disrupting science, finance and every other sector. Learn more about Jupyter Notebooks on this tech talk with Shreyas Bapat, B.Tech. IIT Mandi, who is spearheading efficiency initiatives at Semut.io.
Watch this talk here:

Follow us on our social media channels to stay updated.
Website: https://semut.io
Twitter: https://bit.ly/semut_twitter
LinkedIn: https://bit.ly/semut_linkedin
Youtube: https://bit.ly/semut_youtube
Twitch: https://bit.ly/semut_twitch

Be209241ca741792b3071d0dd5c55771?s=128

Shreyas Bapat

March 12, 2021
Tweet

Transcript

  1. None
  2. None
  3. Who am I? -> Software Engineer @ Semut.io -> Electrical

    Engineer from IIT Mandi -> Lead Developer @ EinsteinPy -> Managing Member @ PSF -> Love being in Mountains of Himachal Pradesh (India)
  4. Let’s start with what went wrong?

  5. What is it? -> Published in Non Peer Reviewed Issue

    of American Economic Review. -> Cited by politicians worldwide in debates to prove effectiveness of austerity in fiscal policies for debt burdened economies. -> When “Gross external debt reaches 60 percent of GDP", a country's annual growth declined by two percent, and "for levels of external debt in excess of 90 percent" GDP growth was "roughly cut in half.” -> Proved Wrong!
  6. None
  7. Catch? Science needs explanation. Papers can be hard to understand.

    Excel is not made for this. You can’t document code, logic and results in excel!
  8. More Issues with using Excel for Science

  9. None
  10. Is there a better way? -> Is there a better

    way to have the text, analysis, results, code, plots, comments in one place? -> Is it possible to write it sequentially such that anyone going through the code/paper/book could not just understand it, but run it, reproduce results, find flaws and suggest enhancements 1000X faster? -> Is it possible to distribute the results properly?
  11. What would be a better way? -> Excel users are

    people avoiding hardcore programming languages. -> Easy interface to write logic -> Low Cognitive Complexity -> Easy Syntax (Preferably like English) [Think BQL (Bloomberg Query Language)]
  12. Python stands out! -> Easy to grab! -> Easy interface

    to write logic -> Low Cognitive Complexity -> Mostly like writing english. -> Out of the box support, extremely friendly community. -> Democratic
  13. Results?

  14. Rise of iPython

  15. iPython? -> First version in 2001 (Started with just 259

    lines) -> Inbuilt support for parallelization came in 2004 -> Support for running the code in remote cluster from the shell. -> Inspired by Mathematica -> Boon for scientific computing
  16. None
  17. Rise to fame!

  18. What are notebooks? -> Notebook is a rather old concept.

    -> Sequentially written logic -> Interactive -> Saves State (like a paper notebook does)
  19. None
  20. Benefits of using iPython -> obj.[tab] , obj? , obj??

    -> files = !ls , !wget $url -> iPython Magic: - %run script.py (-p -> profile, -t -> time) - %debug (jump in after an exception) - %lsmagic (See the rest of magics)
  21. Some myths -> There’s no support -> Won’t be free

    forever -> Free software has bugs
  22. Some myths -> There’s no support Enthought, Continuum Analytics ->

    Won’t be free forever -> Free software has bugs
  23. Some myths -> There’s no support Enthought, Continuum Analytics ->

    Won’t be free forever Free software Belongs to the community -> Free software has bugs
  24. Some myths -> There’s no support Enthought, Continuum Analytics ->

    Won’t be free forever Free software Belongs to the community -> Free software has bugs Naturally!
  25. And it happened

  26. Jupyter or Jupiter?

  27. What is Jupyter?

  28. Jupyter : An Ecosystem -> JupyterLab -> Slides/Documents -> Write

    Books (O’Reilly) - jupyterbook -> JupyterHub -> Collab Notebooks, binder
  29. Jupyter : A way to teach in classrooms! -> Easy

    to create interactive tutorials which lets students play around with the code. -> Perform Live Coding, share lecture notes and materials -> Grade homeworks -> No need to run every script supplied by students
  30. Project Jupyter -> Separation of the language agnostic components -

    Jupyter : protocol, format, multi-user server - iPython : Jupyter Kernel, interactive python -> Jupyter Kernels: Languages which can be used in notebook: ~ 100 programming languages.
  31. Notebook Extensions -> Add ons to extend functionality. Much like

    VS Code extensions. -> VS Code supports Jupyter -> Written in JavaScript, send browser notifications, autoformat code. There are immense possibilities.
  32. Widgets -> Interact with the code output! -> Have slide

    bars, text boxes, inputs -> Like mini-GUIs -> Very helpful when working with hyper parameters. -> You can write one for yourself!
  33. Widgets

  34. Widgets

  35. Version Control in Notebooks -> The notebooks have extension .ipynb

    but are plain text files and are represented in JSON. -> The diffs used to be pretty large earlier even when nothing changed. Storing output is another issue. -> It’s getting better. With nbdime, it’s much easier to merge and compare now.
  36. Using Notebooks in Production!

  37. Issues with Jupyter Notebooks -> Hidden states in the Notebooks

  38. Issues with Jupyter Notebooks -> Hidden states in the Notebooks

  39. Issues with Jupyter Notebooks Notebooks are great for iterative development

    BUT Notebooks are *very* dangerous unless you run each cell only ONCE in CORRECT ORDER.
  40. A way to solve this... The %history magic! But you

    shouldn’t have to run a magic to find the state!
  41. Now take this!

  42. Problem number 2 The ability to run a code in

    a non sequential arbitrary order is counter intuitive to most programmers out there. Can be daunting for beginners!
  43. Notebooks in Cloud -> Notebooks as a service is a

    cool new thing! -> Azure Notebooks, Collab Notebooks and what not. -> You can create your own notebook service!
  44. How does a Jupyter Notebook Work? Credits: Carol Willing

  45. What does JupyterHub do? -> Manages Authentication -> Spawns single-user

    notebook servers on demand -> Gives a user their complete notebook server! -> Hub and server are different entities.
  46. JupyterHub

  47. Parts of JupyterHub -> The Hub: User Database, Auth, and

    Spawner -> Users and their individual notebook servers -> Configurable HTTP Proxy The auth supports OAuth, pem etc. Deploy: https://github.com/jupyterhub/jupyterhub-deploy-docker
  48. Real Time Collaboration Jupyter RTC: https://jupyter-rtc.readthedocs.io/ This works directly with

    JupyterLab
  49. Summary -> Jupyter / iPython is a useful tool, not

    only for coding but also for teaching, sharing, documenting, publishing! -> We don’t have to throw away previous work in different languages, now we can integrate them. -> Jupyter is gaining relevance in Open Science, Finance, Music, Teaching. We must go further!
  50. Questions? Semut Twitter: @semut_io GitHub: @shreyasbapat Twitter: @shreyasb94

  51. None