Open infrastructure for Open Science - How Binder powers an open stack in the cloud

This talk covers several recent open projects from the Jupyter and Binder communities that make it easier to create reproducible, sharable, interactive computing environments. It covers tools for shared infrastructure (JupyterHub), creating reproducible Docker images (repo2docker) and bundling these as a web application for reproducible environments that you can deploy for your community (BinderHub).

Transcript

1. Open Infrastructure for open science How Binder Powers an Open Stack in the Cloud Chris Holdgraf, UC Berkeley and Project Jupyter @choldgraf
2. you???
4. A bit about me then...
5. A bit about me now...
6. a community of people and an ecosystem of open tools and standards for interactive computing
7. create things that are language-agnostic and modular. Empower people to use other open tools.
8. Aside: Jupyter and the last mile problem
9. You San Jose Coffee!
10. You San Jose Coffee!
11. You San Jose Coffee!
12. BART Bus You San Jose Coffee!
13. BART Bus The “last mile” You San Jose Coffee!
14. Public infrastructure gets us closer to our goal. It makes the last mile shorter.
15. How does Jupyter fit in to this?
16. You Your awesome report
17. You Your Awesome report
18. You Your awesome report Jupyter shortens the last mile by creating and leveraging public infrastructure server .ipynb package ecosystem Notebook document specification Jupyter server protocol Interactive Kernels Notebook interfaces
19. Back to our talk...
21. Our mission for this talk Us .ipynb Open, reproducible, sharable environments
22. Part 1: from your laptop to the cloud with JupyterHub
23. (some) data science should be taught to everyone (no, really)
25. How can we connect people with computation?
26. What is JupyterHub? Host pre-configured data science environments on shared infrastructure jupyter.org/hub
27. myhub.org My fancy machine in the cloud
28. myhub.org
29. myhub.org environments
30. myhub.org interfaces environments
31. myhub.org interfaces environments documents
32. compute data
33. AUTHENTICATION
34. Chris Is Trying A Live Demo Hopefully he doesn’t embarrass himself too badly. Textbook link Interact link
35. EXAMPLE + shout-out to open humans notebooks.openhumans.org
36. Us .ipynb Open, reproducible, sharable environments
37. How can users package+share their work?
38. Part 2: packaging and sharing your environment with repo2docker
39. What is repo2docker? Convert a repository into a Docker image that runs the code inside. repo2docker.readthedocs.io
40. github.com/minrk/ligo-binder
42. repo2docker what does it do? $ jupyter repo2docker \ > https://github.com/minrk/ligo-binder
43. repo2docker what does it do? $ jupyter repo2docker \ > https://github.com/minrk/ligo-binder Cloning into '/var/folders/.../T/repo2dockermu6z66sd'... Using CondaBuildPack builder Step 1/31 : FROM buildpack-deps:bionic ---> 29f4eef41002 Step 2/31 : ENV DEBIAN_FRONTEND=noninteractive ---> Using cache ---> ee1ba7c4f5f4 Step 3/31 : RUN apt-get update && apt-get install --yes --no-install-recommends locales && apt-get purge && apt-get clean && rm -rf /var/lib/apt/lists/*
44. repo2docker what does it do? Step 1: get the repo git clone https://github.com/me/myproject
45. repo2docker what does it do? Step 2: Identify requirements
46. Step 2: Identify requirements repo2docker what does it do?
47. Step 2: Identify requirements repo2docker what does it do?
48. ... COPY conda/install-miniconda.bash /tmp/install-miniconda.bash COPY conda/environment.py-3.6.frozen.yml /tmp/environment.yml RUN bash /tmp/install-miniconda.bash && \ rm /tmp/install-miniconda.bash /tmp/environment.yml ... Step 3: generate Dockerfile repo2docker what does it do?
49. Step 3: generate Dockerfile ... # Copy and chown stuff. COPY src/ ${HOME} RUN chown -R ${NB_USER}:${NB_USER} ${HOME} # Run assemble scripts! These will actually build the spec # in the repository into the image. USER ${NB_USER} RUN ${KERNEL_PYTHON_PREFIX}/bin/pip install --no-cache-dir \ -r "requirements.txt" ... repo2docker what does it do?
50. Step 4: build (& push) image docker build -t myimage docker push myimage repo2docker what does it do?
51. repo2docker what does it do? $ jupyter repo2docker \ > https://github.com/minrk/ligo-binder Cloning into '/var/folders/.../T/repo2dockermu6z66sd'... Using CondaBuildPack builder Step 1/31 : FROM buildpack-deps:bionic ---> 29f4eef41002 Step 2/31 : ENV DEBIAN_FRONTEND=noninteractive ---> Using cache ---> ee1ba7c4f5f4 Step 3/31 : RUN apt-get update && apt-get install --yes --no-install-recommends locales && apt-get purge && apt-get clean && rm -rf /var/lib/apt/lists/*
52. Some supported configuration files • environment.yml • requirements.txt • REQUIRE • install.R • apt.txt • setup.py • postBuild • runtime.txt • Dockerfile • yournewbuildpack.txt
53. • Repos should be human and machine readable • Use existing specifications and standards • Support many languages and interfaces • Be lightweight and tightly-scoped, but extendable Guiding principles of repo2docker
54. Us .ipynb Open, reproducible, sharable environments
55. Part 3: tying this together with BinderHub
56. What is BinderHub? One-click sharable, interactive, reproducible environments from your public git repository mybinder.org binderhub.readthedocs.io
58. • • • • BinderHub is open tech...
59. One example: mybinder.org
60. Chris Is Trying A Live Demo Hopefully he doesn’t embarrass himself too badly. mybinder.org binder-examples/requirements
62. Mybinder.org is an open service...
63. mybinder.org weekly sessions, last ~year
64. mybinder.org sessions, last month
66. Some cool new projects (or, stuff you can help out with)
67. Interactive books with Jupyter-Book github.com/jupyter/jupyter-book
68. https://github.com/simpeg-research/heagy-2018-aem Publishable documents (with pandoc?)
69. HTML dashboards with voila github.com/QuantStack/voila
70. In summary jupyter.org Jupyter makes the “last-mile” problem as small as possible by building modular, open tools. Us .ipynb Open, reproducible, sharable environments
71. In summary JupyterHub lets you create a shared, interactive analytics environment Us .ipynb Open, reproducible, sharable environments mybinder.org
72. In summary repo2docker creates reproducible Docker images from a repository Us .ipynb Open, reproducible, sharable environments mybinder.org
73. In summary BinderHub is an open web application to create shareable, reproducible coding environments Us .ipynb Open, reproducible, sharable environments Mybinder.org
74. Get involved with Jupyter @choldgraf jupyterhub-team-compass.readthedocs.io discourse.jupyter.org • All of these projects are open source, run by open communities • Jupyter is a place where *anybody* can participate • If you’d like to get involved: