Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Start Small and Scale: Big Data and Jupyter's Ecosystem

Start Small and Scale: Big Data and Jupyter's Ecosystem

Keynote Presentation at PyData LA conference held at Cal State Los Angeles.

Jupyter notebooks have become the de-facto standard as a scientific and data science tool for producing computational narratives. Over five million Jupyter notebooks exist on GitHub today. Beyond the classic Jupyter notebook, Project Jupyter's tools have evolved to provide end to end workflows for research that enable scientists to prototype, collaborate, and scale with ease. JupyterLab, a web-based, extensible, next generation interactive development environment enables researchers to combine Jupyter notebooks, code and data to form computational narratives. JupyterHub brings the power of notebooks to groups of users. It gives users access to computational environments and resources without burdening the users with installation and maintenance tasks. Binder builds upon JupyterHub and provides free, sharable, interactive computing environments to people all around the world.


Carol Willing

December 05, 2019


  1. @WillingCarol Start Small and Scale Carol Willing PyData LA December

    5, 2019 https://speakerdeck.com/willingc Big Data and Jupyter's Ecosystem
  2. @WillingCarol Hi! I'm Carol. • Python Steering Council • Jupyter

    Steering Council • Core Developer, Python, Jupyter, nteract • PSF Fellow and Former Director • Frank Willison Award 2019 • Open Source Directions Podcast Co- host 2
  3. @WillingCarol 3 Core maintainer Papermill, Scrapbook, Bookstore, Commuter Steering Council,

    Core Developer JupyterHub, BinderHub, mybinder.org I love creating tools which educate and empower people.
  4. @WillingCarol What is Data Science 4 Start Here

  5. @WillingCarol 5 Using data responsibly to solve real world issues

    and improve human lives
  6. @WillingCarol Predictions at Scale 6 A real world tale

  7. @WillingCarol 7 San Diego, CA

  8. @WillingCarol 8 Tokyo

  9. @WillingCarol 9 Sunday Oct 6 Source: ECMWF

  10. Copyright: 2019 European Union, contains modified Copernicus Sentinel data 2019,

    processed by EUMETSAT Super Typhoon Hagibis View of Super Typhoon Hagibis south-west of Japan, as captured by the Copernicus Sentinel-3 satellite on 08 October at 00:16 UTC.
  11. Title Typhoon Hagibis Released 10/10/2019 4:45 pm Copyright contains modified

    Copernicus Sentinel data (2019), processed by ESA, CC BY-SA 3.0 IGO
  12. Source:Twitter

  13. @WillingCarol 13

  14. @WillingCarol 14

  15. A sign is partially submerged as the Tama River floods

    during Typhoon Hagibis. Source:Getty Images Source:Japan Times
  16. @WillingCarol Preparation Evacuation Safety 16

  17. @WillingCarol With scale comes responsibility 17 Reality Check

  18. @WillingCarol 18 "Diabetes awareness: character design" by Carlos Hernandez is

    licensed under CC BY-NC-ND 4.0
  19. @WillingCarol –Kevin Sayer, DexCom CEO This whole integration of health

    care data is really going to be the next frontier. 19 https://www.cnbc.com/2019/11/13/big-data-is-the-next-frontier-for-medicine-says-dexcom-ceo.html https://www.businesswire.com/news/home/20191106005764/en/Dexcom-Reports-Quarter-2019-Financial-Results
  20. @WillingCarol Outage Midnight Friday: mysterious outage Dexcom did not announce

    there was an outage until about 8 a.m. Pacific time Saturday, which is 11 a.m. on the East Coast, when it posted a brief notice on its Facebook page. Monday morning: Dexcom Follow partly restored 20 https://www.nytimes.com/2019/12/02/well/live/Dexcom-G6-diabetes-monitor-outage.html Source: https://www.dexcom.com/
  21. @WillingCarol Getting from Start to Scale 21 The Challenge

  22. @WillingCarol 22 jupyter.org

  23. @WillingCarol 23 2014 Now, 5 years later... 23

 Millions of Notebooks https://github.com/trending/jupyter-notebook Over 5 million on GitHub

  25. @WillingCarol 25 ‣ Growth ‣ ACM Award ‣ Industry adoption

    ‣ Creative uses ‣ Open Source Book https://www.youtube.com/watch?v=qbtDVdEr8SY
  26. jupyter.org

  27. @WillingCarol 27 Start Small Deploy and Sustain Explore Paths to

    Scale The Roadmap
  28. @WillingCarol Start Small 28 Step 1

  29. @WillingCarol 29 Source: xkcd

  30. @WillingCarol 30 small

  31. @WillingCarol 31 Binder 2.0 blog post elifesciences: Share your interactive

    research environment Nature article about Binder 31 mybinder.org Try it. No install needed.
  32. @WillingCarol 32 Scale in Production

  33. @WillingCarol Choose your Tools 33

  34. @WillingCarol JupyterLab 34

  35. 35 jupyter.org demo

  36. 36 jupyter.org demo

  37. @WillingCarol 37 ReactJS front end nteract nteract.io

  38. @WillingCarol VS Code 38 PyCharm

  39. @WillingCarol Avoid reinventing the wheel 39

  40. @WillingCarol ecosystem 40

  41. @WillingCarol Install Promising Libraries 41 Use Anaconda Use pip Use

    miniconda, conda, and conda-forge
  42. @WillingCarol Start 42 Try it in the browser Install Libraries

    Choose your tools Avoid reinventing the wheel Step 1
  43. @WillingCarol Explore Paths to Scale 43 Step 2

  44. @WillingCarol Turn no way into it's possible 44

  45. @WillingCarol Community 45 Conferences Meetups PyLadies Carpentries Photo: Python Sul

  46. @WillingCarol 46 Ten Simple Rules for Reproducible Research in Jupyter

    Notebooks Adam Rule et al. https://github.com/jupyter-guide/ ten-rules-jupyter https://github.com/jupyter-guide/ jupyter-guide
  47. @WillingCarol 47 build, try, change, repeat

  48. @WillingCarol 48 ipyvolume https://towardsdatascience.com/multivolume- rendering-in-jupyter-with-ipyvolume-cross- language-3d-visualization-64389047634a

  49. @WillingCarol 49

  50. @WillingCarol 50 github.com/napari/napari napari https://ilovesymposia.com/2019/10/24/introducing-napari- a-fast-n-dimensional-image-viewer-in-python/

  51. A pictorial representation of the different tools constituting BinderHub. This

    image was created by Scriberia for The Turing Way community and is used under a CC-BY licence. Zenodo record. https://blog.jupyter.org/diving-into- leadership-to-build-push-button-code- df2a075c9914
  52. @WillingCarol What's new 52 Talk Python to Me Tracking Jupyter

    Newsletter https://tinyletter.com/TrackingJupyter/archive Open Source Directions GitHub Trending Follow projects on Social Media
  53. @WillingCarol Explore 53 Use the ecosystem to learn Best practices

    Infrastructure/Analysis What's new Step 2
  54. @WillingCarol Deploy and Sustain 54 Step 3

  55. @WillingCarol 55 "Digital World" by NBroekzitter86 is licensed under CC

    BY 2.0
  56. @WillingCarol Notebooks to web 56 https://blog.jupyter.org/and- voil%C3%A0-f6a2c08a4a93

  57. @WillingCarol 57 nteract Papermill Scrapbook Bookstore Commuter Production data at

    scale 57 https://medium.com/netflix-techblog/notebook-innovation-591ee3221233 Bookstore
  58. @WillingCarol 58 Papermill - parameterize / run Scrapbook - recording

    / reading Bookstore - store notebooks Commuter - share notebooks Production data at scale 58
  59. @WillingCarol 59 Enterprise data workflows 59

  60. zero-to-jupyterhub.readthedocs.io

  61. @WillingCarol 61 61 Deploy your own BinderHub

  62. 62 Juliette Taka

  63. @WillingCarol 63 From a phone in the park! 63

  64. @WillingCarol Pangeo 64 https://pangeo.io

  65. @WillingCarol 65

  66. @WillingCarol Deploy and Sustain 66 Workflows Document Monitor Involvement Step

  67. @WillingCarol 67 From Small to Scale Keys for Success

  68. @WillingCarol Choose to Start 68

  69. @WillingCarol Why > how 69

  70. @WillingCarol Automate the Boring Stuff 70

  71. @WillingCarol Plan Execute Change 71 https://jupyterhub-team-compass.readthedocs.io https://github.com/jupyterhub/team-compass

  72. @WillingCarol Consider complexity and observability 72

  73. @WillingCarol People = Responsibility 73

  74. @WillingCarol 74 Call to Action

  75. @WillingCarol ecosystem 75

  76. @WillingCarol 76 Using data responsibly to solve real world issues

    and improve human lives
  77. @WillingCarol 77 Justine Dupont surfs the greatest wave of her

    life in Nazaré, Portugal © Rafael G. Riancho / Red Bull Content Pool
  78. @WillingCarol 78 Thank you https://speakerdeck.com/willingc @WillingCarol

  79. @WillingCarol 79 Questions https://speakerdeck.com/willingc

  80. @WillingCarol 80 Thank you PyData LA Project Jupyter Team Core

    Python Team PSF NumFOCUS
  81. @WillingCarol Attributions 81 Attributions on slides. Photos [7-8] Carol Willing

    and Linnea Willing [14] The Carpentries, Tracy Teal, Bérénice Batut [14] Godzilla By Toho Company Ltd. (東宝株式会社, Tōhō Kabushiki-kaisha) © 1954 - movie poster made by Toho Company Ltd. (東宝株式会社, Tōhō Kabushiki-kaisha), Public Domain, https://commons.wikimedia.org/w/index.php?curid=3648684