Binder 2.0 - The Next Generation of Reproducible Scientific Environments with repo2docker and BinderHub

This talk covers recent developments in the Binder Project, and describes the architecture and design of BinderHub, an open-source tool that runs on Kubernetes and lets your users create sharable, interactive, reproducible data analytics environments. It was originally given at SciPy 2018 (video here: https://www.youtube.com/watch?v=KcC0W5LP9GM)

Transcript

6. • • • • • •
7. • •
8. Analysis + Visualization Interaction + Authoring Sharing + Discovery Bundling + Reproducing
10. <3 <3
13. • • • •
17. • • • • •
19. $ jupyter repo2docker \ > https://github.com/minrk/ligo-binder Cloning into '/var/folders/.../T/repo2dockermu6z66sd'... Using CondaBuildPack builder Step 1/31 : FROM buildpack-deps:bionic ---> 29f4eef41002 Step 2/31 : ENV DEBIAN_FRONTEND=noninteractive ---> Using cache ---> ee1ba7c4f5f4 Step 3/31 : RUN apt-get update && apt-get install --yes --no-install-recommends locales && apt-get purge && apt-get clean && rm -rf /var/lib/apt/lists/*
20. git clone https://github.com/me/myproject
24. ... COPY conda/install-miniconda.bash /tmp/install-miniconda.bash COPY conda/environment.py-3.6.frozen.yml /tmp/environment.yml RUN bash /tmp/install-miniconda.bash && \ rm /tmp/install-miniconda.bash /tmp/environment.yml ...
25. ... # Copy and chown stuff. COPY src/ ${HOME} RUN chown -R ${NB_USER}:${NB_USER} ${HOME} # Run assemble scripts! These will actually build the spec # in the repository into the image. USER ${NB_USER} RUN ${KERNEL_PYTHON_PREFIX}/bin/pip install --no-cache-dir \ -r "requirements.txt" ...
26. docker build -t myimage docker push myimage
27. $ jupyter repo2docker \ > https://github.com/minrk/ligo-binder Cloning into '/var/folders/.../T/repo2dockermu6z66sd'... Using CondaBuildPack builder Step 1/31 : FROM buildpack-deps:bionic ---> 29f4eef41002 Step 2/31 : ENV DEBIAN_FRONTEND=noninteractive ---> Using cache ---> ee1ba7c4f5f4 Step 3/31 : RUN apt-get update && apt-get install --yes --no-install-recommends locales && apt-get purge && apt-get clean && rm -rf /var/lib/apt/lists/*
28. • Dockerfile • environment.yml • requirements.txt • REQUIRE • install.R • apt.txt • setup.py • postBuild • runtime.txt
29. • • • • pip install jupyter-repo2docker
34. binder-<hash> build-<hash> Data and I/O User flow Trigger action
35. Data and I/O User flow Trigger action hub-<hash> proxy-<hash> jupyter-<username>-<hash> binder-<hash>
36. Data and I/O User flow Trigger action binder-<hash> jupyter-<username>-<hash>
38. 12,000 6,000 December, 2017 July, 2018 Daily unique sessions
39. grafana.mybinder.org
40. github.com/jupyterhub/binder-billing
42. • • • • ◦ ◦ binder.youruniversity.edu
43. • • ◦ ◦ ◦ ◦ ◦ •
45. # URL to build a given version of a repo url = f"https://mybinder.org/build/gh/{repo}/{ref}" r = requests.get(url, stream =True) for line in r.iter_lines(): line = line.decode("utf8", "replace") # EventStream messages if line.startswith("data:"): evt = json.loads(line.split(":", 1)[1]) # echo each message if "message" in evt: sys .stdout.write(evt["message"]) if evt.get("phase") == "ready": # open in a browser when it’s ready url = evt["url"] token = evt["token"] webbrowser .open(f"{url}?token={token}" )