JupyterHub

Transcript

1. Machine Learning for Large Scale Code Analysis JupyterHub Infra Session
2. What is JupyterHub? AS SEEN BY THE USER • Multi-user • No Installation • No Maintenance • Scalable • Cloud Native (this came out of the presentation template and happened to fit right in) 2
3. 3 Login Data Science!
4. 4 2 official “distributions” “The Littlest JupyterHub” • 1 server • No containers • Quick & Cheap “JupyterHub with Kubernetes” • Scalable • Containers!
5. Behind the scenes 5
6. 6 The architecture HTTP Reverse Proxy Hub User Pod OAuth Spawner Creates + Redirects PostgreSQL
7. 7 HTTP Proxy • Node.js based Proxy • Configurable via REST • Entrypoint for all traffic jupyterhub/configurable-http-proxy
8. 8 Hub • Main component • Contains the spawner • Manages Auth • Manages Sessions • Stores info in DB (psql/sqlite) • Configured via a configmap jupyterhub/k8s-hub
9. 9 Authenticators • User login • Plug-in to the Hub Hub Components • GitHub/BitBucket/MediaWiki/CILogon/Google OAuth • Dummy Authenticator • LDAP Authenticator • Kerberos Authenticator • KDC Authenticator • REMOTE_USER Authenticator • JSONWebToken Authenticator • HashAuthenticator • CASAuthenticator • Shibboleth Authenticator • SAML Authenticator • LTI Authenticator
10. 10 Spawner • Creates the “singleuser” instance • Plug-in to the Hub Hub Components • BatchSpawner • DockerSpawner • FargateSpawner • ImageSpawner • KubeSpawner • MarathonSpawner • UCRSpawner • RemoteSpawner • SimpleSpawner • SystemdSpawner • SudoSpawner • WrapSpawner • YarnSpawner
11. 11 “Singleuser” pod • Runs Jupyter notebook • Created from a template • Manages pod config: resources/storage/labels/… • Contains network config utility (useful in Cloud only) • Contains multiple kernels/dependencies from the Docker image Created by the KubeSpawner
12. Deployment 12
13. 13 JupyterHub Deployment Using The Official Helm Chart • Proxy (+ ingress) • Hub (+ KubeSpawner + Auth + RBAC) • Singleuser configuration • “prePuller”
14. 14 PrePuller • Helm Hook • DaemonSet + Job • Job checks DaemonSet • Meant to start pods fast on upgrade Hook Continuous • DaemonSet • Pulls image in init container • Runs pause • Meant to start pods fast on new node
15. 15 Let’s get practical
16. 16 Setting up JupyterHub in minikube • https://zero-to-jupyterhub.readthedocs.io/e n/latest/ • Keep it easy use dummy auth • Tip: read values.yaml, sometimes the documentation is not the best
17. Machine Learning for Large Scale Code Analysis Thank you!