Open Infrastructure in the Cloud with JupyterHub

In this talk I cover the principles, design, goals, and general structure of JupyterHub, a tool for providing interactive computing environments to others on shared infrastructure. I also cover several cool deployments of JupyterHub in the open community - using JupyterHub for education (data8.org), for large-scale geoscience in the cloud (pangeo.io) and for open community collaboration (openhumans.org).

Transcript

1. Open Infrastructure in the Cloud with JupyterHub @choldgraf Chris Holdgraf, UC Berkeley and Project Jupyter
2. bit.ly/jupyterhub-sdss-2019
3. you???
5. @choldgraf A bit about me then... Cognitive Neuroscience Open Source
6. @choldgraf A bit about me now... Research and Open Source Education and Open Source Jupyter @ Berkeley
7. a community of people and an ecosystem of open tools and standards for interactive computing
8. create things that are language-agnostic and modular. Empower people to use other open tools.
9. For example: the Jupyter Notebook
10. You Your awesome report The Jupyter Notebook is a stack of modular, open tools server .ipynb package ecosystem Notebook document specification Jupyter server protocol Interactive Kernels Notebook interfaces
11. How does the ☁ fit into this?
12. (some) data science should be taught to everyone (no, really)
13. Here’s what this means at Berkeley...
14. How can Jupyter connect people with computation?
15. Build infrastructure tools that are workflow- and platform-agnostic. Give people control over resources, freedom to deploy what and where they wish. (in the ☁)
16. What is JupyterHub? Host pre-configured data science environments on shared infrastructure jupyter.org/hub
17. myhub.org My fancy machine in the cloud
18. myhub.org
19. myhub.org environments
20. myhub.org interfaces environments
21. AUTHENTICATION myhub.org interfaces environments
22. JupyterHub distributions The Littlest JupyterHub tljh.jupyter.org JupyterHub on Kubernetes z2jh.jupyter.org A pre-configured JupyterHub setup with sensible defaults and lots of documentation, fit for many use-cases ☁
23. Scalable in both users and in resources Uses Docker for environment management Agnostic to the provider and hardware configuration Zero to JupyterHub for Kubernetes z2jh.jupyter.org
25. The littlest JupyterHub Deploy JupyterHub on a single virtual machine Faster, lightweight setup and administration More easily created and destroyed tljh.jupyter.org
27. JupyterHub in the wild ✨ ✨
28. education and training
29. datahub.berkeley.edu
30. inferentialthinking.com
31. nbgitpuller - one-click interactive content jupyterhub.github.io/nbgitpuller
32. Chris Is Trying A Live Demo Hopefully he doesn’t embarrass himself too badly. data8.org/sp19/
33. Data 8 is... jupyter book github gofer grader jupyter notebook nbgitpuller scipy stack
36. large-scale science ☁
37. pangeo.io
38. • A JupyterHub + BinderHub managing an open stack of tools for geospatial analysis • Utilize and improve pre-existing tools, rather than build new ones. Push improvements upstream. • Add value with customization and configuration • Provide access to high-performance hardware The Pangeo pattern pangeo.io
39. pangeo.io
40. Pangeo is... zarr jupyter widgets dask jupyter lab xarray scipy stack big data
41. Open and Interactive collaboration
42. openhumans.org https://zenodo.org/record/2697185#.XNV2a_DJe5Y
44. Chris Is Trying A Live Demo Hopefully he doesn’t embarrass himself too badly. exploratory.openhumans.org
45. OpenHumans is... open data R stack exploratory jupyter notebook community scipy stack
46. • JupyterHub connects users with interactive environments on shared infrastructure using open tools • JupyterHub distributions are opinionated deployments of a JupyterHub for a specific scale or purpose. ◦ The Littlest JupyterHub - Deploy on a single VM ◦ Zero to JupyterHub for Kubernetes - Deploy in the ☁ • JupyterHub has been used for ◦ Large-scale education (data8.org) ◦ High-performance analysis in the ☁(pangeo.io) ◦ Collaborative community analytics (openhumans.org) In summary
47. Get involved with Jupyter @choldgraf jupyterhub-team-compass.readthedocs.io discourse.jupyter.org • All of these projects are open source, run by open communities • Jupyter is a place where *anybody* can participate • If you’d like to get involved: