The current state of JupyterHub

presented at JupyterCon 2018

Transcript

1. 
 The current state of JupyterHub and what’s in store for the future Carol Willing Min Ragan-Kelley Yuvi Panda JupyterCon August 23, 2018
2. JupyterHub • Fundamentals • What’s New • What’s Coming • JupyterHub with Kubernetes • Next Steps
3. Where we are github.com/jupyterhub Previous 12 months Since JC17 repos 26 37 commit authors 47 101 commits 4520 5920 contributors 448 1032 comments 7k 20k issues open/closed 2193/1952 6019/5294 JupyterCon 2017-JupyterCon 2018
4. Where we are github.com/jupyterhub Previous 12 months Since JC17 repos 26 37 commit authors 47 101 commits 4520 5920 contributors 448 1032 comments 7k 20k issues open/closed 2193/1952 6019/5294 JupyterCon 2017-JupyterCon 2018
5. Where we are Pre 0.7
6. Where we are Key pieces of JupyterHub • Notebook is what people want to use • Proxy in front (configurable-http-proxy) • Authenticator handles authentication • Spawner launches single-user servers • Hub connects everything
7. Where we are Key pieces of JupyterHub • Notebook is what people want to use • Proxy in front (configurable-http-proxy) • Authenticator handles authentication • Spawner launches single-user servers • Hub connects everything
8. Where we are Pre 0.7
9. Authenticators class MyAuthenticator(Authenticator): def authenticate(self, handler, data): username = data['username'] password = data['password'] if library.check_password(username, password): return username else: self.log.warning("Failed to authenticate user: %r", username)
10. Authenticators • OAuthenticator: Several OAuth providers (Google, GitHub, GitLab, Bitbucket, Wikimedia, CILogon, Auth0, Globus, KeyCloak, Generic) • REMOTE_USER for Shibboleth • LDAP • LTIAuthenticator • First-use authenticator (set password on first login)
11. Spawners class MySpawner(Spawner): async def start(self): env = self.get_env() cmd = self.cmd + self.get_args() await start_the_server(cmd, env) return (ip, port) start(): starts a notebook server poll(): check if a server is still running stop(): stop a running server
12. Spawners • DockerSpawner • KubeSpawner (Kubernetes) • SwarmSpawner • BatchSpawner (PBS, etc.) • MarathonSpawner • SystemdSpawner • ProfileSpawner (multiplexes spawner configurations)
13. JupyterHub • Fundamentals • What’s New • What’s Coming • JupyterHub with Kubernetes • Next Steps
14. Auth state • Auth state can be encrypted and stored, for passing credentials such as API tokens or certificates to user environments. • Now implemented in many authenticators, including oauthenticator. 0.8
15. Auth state • Auth state can be encrypted and stored, for passing credentials such as API tokens or certificates to user environments. • Now implemented in many authenticators, including oauthenticator. 0.8
16. Auth state • Auth state can be encrypted and stored, for passing credentials such as API tokens or certificates to user environments. • Now implemented in many authenticators, including oauthenticator. 0.8
17. 0.9 - 06/2018 • Refine REST API • admins can request auth state • token management • async/await syntax everywhere: require Python 3.5, tornado 5 • Applications outside JupyterHub can use Hub as OAuth provider 0.9
18. Improved Colorblind Accessibility 0.9 http://www.visualisingdata.com/2015/11/ colour-swatch-alternatives-to-green-and-red/
19. Improved Colorblind Accessibility 0.9 http://www.visualisingdata.com/2015/11/ colour-swatch-alternatives-to-green-and-red/
20. Improved Colorblind Accessibility 0.9 http://www.visualisingdata.com/2015/11/ colour-swatch-alternatives-to-green-and-red/
21. Token management 0.9
22. Progress! 0.9
23. Progress! 0.9
24. await all the things! 0.9 @async_generator async def progress(self): async for event in self.get_events(): await yield_({ 'progress': 50, 'message': "%s [%s] %s" % ( event.last_timestamp, event.type, event.message, ) })
25. proxying non-Jupyter applications nbserverproxy nbrsessionproxy
26. proxying non-Jupyter applications nbserverproxy nbrsessionproxy
27. proxying non-Jupyter applications nbserverproxy nbrsessionproxy
28. • Fundamentals • What’s New • JupyterHub with Kubernetes • What’s Coming • Next Steps JupyterHub
29. Zero to JupyterHub Scalable JupyterHub deployments for Education, Research, Business Analytics & Unicorns https://z2jh.jupyter.org
30. Architecture Google Cloud AWS Azure Your private cloud Your own hardware Kubernetes JupyterHub #1 Other software JupyterHub #2 Jupyter Notebook Servers, one per user
31. What does it look like? helm upgrade --install myhub \ jupyterhub/jupyterhub \ --version=0.7.0-beta.1 \ -f config.yaml
32. What does it look like? helm upgrade --install myhub \ jupyterhub/jupyterhub \ --version=0.7.0-beta.1 \ -f config.yaml proxy: secretToken: “59883596346…” https: hosts: - myhost.jovyan.org auth: type: github singleuser: image: name: jupyter/r-notebook tag: 8f56e3c47fec
33. zero-to-jupyterhub 0.7 is in beta!
34. chartpress https://github.com/jupyterhub/chartpress automates building and tagging images for helm charts pip install chartpress
35. chartpress https://github.com/jupyterhub/chartpress automates building and tagging images for helm charts pip install chartpress charts: - name: jupyterhub imagePrefix: jupyterhub/k8s- repo: git: jupyterhub/helm-chart published: https://jupyterhub.github.io/helm-chart images: hub: valuesPath: hub.image buildArgs: JUPYTERHUB_VERSION: 0.9.2 network-tools: valuesPath: singleuser.networkTools.image image-awaiter: valuesPath: prePuller.hook.image
36. hubtraf https://github.com/yuvipanda/hubtraf automated jupyterhub load testing run against any jupyterhub deployment testing is good!
37. • Fundamentals • What’s New • JupyterHub with Kubernetes • What’s Coming • Next Steps JupyterHub
38. Projects • Users are members of projects • Each project gets a notebook server, files, etc. • All members have access to the project JupyterHub
39. UI for named servers JupyterHub
40. Improving autoscaling • Problem: user who triggers scale-up takes a long time • Solution: Kubernetes 1.11 feature PodPriority enables “headroom” for improved scale-up • Problem: scale-down requires manual cordoning, draining • Solution: custom scheduler and node- taints to enable packing users and aid scale-down Z2JH
41. • Fundamentals • What’s New • JupyterHub with Kubernetes • What’s Coming • Next Steps JupyterHub
42. • Share your knowledge and experiences with the Jupyter community • Spawner authors: implement Spawner.progress! • Try out Zero to JupyterHub 0.7 beta What’s Next?
43. Over to Yuvi