Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The current state of JupyterHub

Min RK
August 23, 2018

The current state of JupyterHub

presented at JupyterCon 2018

Min RK

August 23, 2018
Tweet

Other Decks in Programming

Transcript

  1. 
 The current state of JupyterHub and what’s in store

    for the future Carol Willing Min Ragan-Kelley Yuvi Panda JupyterCon August 23, 2018
  2. Where we are github.com/jupyterhub Previous 12 months Since JC17 repos

    26 37 commit authors 47 101 commits 4520 5920 contributors 448 1032 comments 7k 20k issues open/closed 2193/1952 6019/5294 JupyterCon 2017-JupyterCon 2018
  3. Where we are github.com/jupyterhub Previous 12 months Since JC17 repos

    26 37 commit authors 47 101 commits 4520 5920 contributors 448 1032 comments 7k 20k issues open/closed 2193/1952 6019/5294 JupyterCon 2017-JupyterCon 2018
  4. Where we are Key pieces of JupyterHub • Notebook is

    what people want to use • Proxy in front (configurable-http-proxy) • Authenticator handles authentication • Spawner launches single-user servers • Hub connects everything
  5. Where we are Key pieces of JupyterHub • Notebook is

    what people want to use • Proxy in front (configurable-http-proxy) • Authenticator handles authentication • Spawner launches single-user servers • Hub connects everything
  6. Authenticators class MyAuthenticator(Authenticator): def authenticate(self, handler, data): username = data['username']

    password = data['password'] if library.check_password(username, password): return username else: self.log.warning("Failed to authenticate user: %r", username)
  7. Authenticators • OAuthenticator: Several OAuth providers (Google, GitHub, GitLab, Bitbucket,

    Wikimedia, CILogon, Auth0, Globus, KeyCloak, Generic) • REMOTE_USER for Shibboleth • LDAP • LTIAuthenticator • First-use authenticator (set password on first login)
  8. Spawners class MySpawner(Spawner): async def start(self): env = self.get_env() cmd

    = self.cmd + self.get_args() await start_the_server(cmd, env) return (ip, port) start(): starts a notebook server poll(): check if a server is still running stop(): stop a running server
  9. Spawners • DockerSpawner • KubeSpawner (Kubernetes) • SwarmSpawner • BatchSpawner

    (PBS, etc.) • MarathonSpawner • SystemdSpawner • ProfileSpawner (multiplexes spawner configurations)
  10. Auth state • Auth state can be encrypted and stored,

    for passing credentials such as API tokens or certificates to user environments. • Now implemented in many authenticators, including oauthenticator. 0.8
  11. Auth state • Auth state can be encrypted and stored,

    for passing credentials such as API tokens or certificates to user environments. • Now implemented in many authenticators, including oauthenticator. 0.8
  12. Auth state • Auth state can be encrypted and stored,

    for passing credentials such as API tokens or certificates to user environments. • Now implemented in many authenticators, including oauthenticator. 0.8
  13. 0.9 - 06/2018 • Refine REST API • admins can

    request auth state • token management • async/await syntax everywhere: require Python 3.5, tornado 5 • Applications outside JupyterHub can use Hub as OAuth provider 0.9
  14. await all the things! 0.9 @async_generator async def progress(self): async

    for event in self.get_events(): await yield_({ 'progress': 50, 'message': "%s [%s] %s" % ( event.last_timestamp, event.type, event.message, ) })
  15. Architecture Google Cloud AWS Azure Your private cloud Your own

    hardware Kubernetes JupyterHub #1 Other software JupyterHub #2 Jupyter Notebook Servers, one per user
  16. What does it look like? helm upgrade --install myhub \

    jupyterhub/jupyterhub \ --version=0.7.0-beta.1 \ -f config.yaml
  17. What does it look like? helm upgrade --install myhub \

    jupyterhub/jupyterhub \ --version=0.7.0-beta.1 \ -f config.yaml proxy: secretToken: “59883596346…” https: hosts: - myhost.jovyan.org auth: type: github singleuser: image: name: jupyter/r-notebook tag: 8f56e3c47fec
  18. chartpress https://github.com/jupyterhub/chartpress automates building and tagging images for helm charts

    pip install chartpress charts: - name: jupyterhub imagePrefix: jupyterhub/k8s- repo: git: jupyterhub/helm-chart published: https://jupyterhub.github.io/helm-chart images: hub: valuesPath: hub.image buildArgs: JUPYTERHUB_VERSION: 0.9.2 network-tools: valuesPath: singleuser.networkTools.image image-awaiter: valuesPath: prePuller.hook.image
  19. Projects • Users are members of projects • Each project

    gets a notebook server, files, etc. • All members have access to the project JupyterHub
  20. Improving autoscaling • Problem: user who triggers scale-up takes a

    long time • Solution: Kubernetes 1.11 feature PodPriority enables “headroom” for improved scale-up • Problem: scale-down requires manual cordoning, draining • Solution: custom scheduler and node- taints to enable packing users and aid scale-down Z2JH
  21. • Share your knowledge and experiences with the Jupyter community

    • Spawner authors: implement Spawner.progress! • Try out Zero to JupyterHub 0.7 beta What’s Next?