Talk at PyTennessee on tmpnb, nature, IPython, Jupyter, cloudpipe, and ephemeral workloads with Docker.
Ephemeral DockerServiceswith IPython, Jupyter, andCloudpipe
View Slide
Hi!
Goodmorning!
Kyle Kelley @rgbkrk
♥I
Sharing
Deployment 㱺
FROM python:3-onbuildCMD [“python”, “./app.py”]
from flask import Flaskapp = Flask(__name__)!@app.route('/')def hello_pytn():return 'Hello PyTennessee!'!if __name__ == '__main__':app.run(‘0.0.0.0’)
$ docker build -t pytn .
$ docker build -t pytn .Sending build context to Docker daemon 92.67 kBSending build context to Docker daemonStep 0 : FROM python:3-onbuild# Executing 3 build triggersTrigger 0, COPY requirements.txt /usr/src/app/Step 0 : COPY requirements.txt /usr/src/app/Trigger 1, RUN pip install -r requirements.txtStep 0 : RUN pip install -r requirements.txt---> Running in dea1149d79ba…Successfully installed Flask Werkzeug Jinja2itsdangerous markupsafeCleaning up...
$ docker build -t pytn .…Trigger 2, COPY . /usr/src/appStep 0 : COPY . /usr/src/app---> 6067170ab7feRemoving intermediate container 7ba25a7d8c3bRemoving intermediate container dea1149d79baRemoving intermediate container 98f8579edc94Step 1 : CMD python ./app.py---> Running in 53031fa4ba6d---> a7324c447943Removing intermediate container 53031fa4ba6dSuccessfully built a7324c447943
$ docker run -p 5000:5000 pytn* Running on http://*:5000/(Press CTRL+C to quit)
$ docker run ipython/scipystack
ipython/ipythonipython/scipystackipython/notebookipython/scipyserver
“Could we do a livedemo for Nature?”
“Yes?”
JUPYTERHUB IN ACTIONKyle visits /user/rgbkrk/ to reach hisnotebook server
CONFIGURABLE HTTP PROXY• Based on node-http-proxy • Proxy routes to localhost, external IPs, etc. • /user/123/ → 127.0.0.1:56790 • Provides an administrative API for adding routes • POST /api/routes/[:path]
github.com/jupyter/configurable-http-proxy
“Yes!”
github.com/jupyter/tmpnb
/user/reqz//user/asdf//user/uiop/:59678:59632:59680Configurable HTTP Proxy
tmpnb.org
bit.ly/inature
> 20,000notebookservers
What else isephemeral?
computejobs
>>> import cloud >>> def add(x, y): ... return x+y >>> add(1, 2) 3 >>> jid = cloud.call(add, ... 1, 2)
!>>> cloud.result(jid) 3
>>> import cloud >>> def scrape(x,y): ... # Expensive operation >>> xs = [] # list of x’s >>> ys = [] # list of y’s >>> jids = cloud.map(scrape, ... xs, ys)
>>> cloud.result(jids)
PICLOUD• Chuck a python pickle to the cloud(f, args, kwargs) = pickle.load(sys.stdin)res = f(*args, **kwargs)!cloudpickle.dump(res, open("/tmp/.result", "wb"))
Acquisition
:(
?
github.com/cloudpipe
import multyvac !# Connect to the cluster multyvac.config.set_key( api_key=rackspace_user, api_secret_key=rackspace_api_key, api_url=‘https://api.cloudpi.pe/v1’) !!!!
CLOUDPIPE• Docker Backed • Processes run as non-root • API compatible with multyvac
CLOUDPIPE• Open Source • Want Need Python 3 binding, new cloud pickler • Node library in the works
Where else can we take things?
On Demand IPython Kernels !github.com/rgbkrk/kernels-service
bit.ly/oreillyjupyter
FRONTIER• Put kernels close to data stores, spin out compute • IPython parallel on demand • Compute jobs on demand • Trigger jobs by events
JOIN MY TEAM[email protected]JOIN RACKSPACE[email protected]
KYLE KELLEY@rgbkrk [email protected]
rgbkrk
miyakemito
regonn
fixstars
sue445
sbtechnight
oekazuma
abenben
hirosatogamo
stefafafan
tsubakimoto_s
tarao
6onoda
chrislema
pedronauck
holman
colly
carmenintech
dougneiner
chriscoyier
soudai
trishagee
robhawkes
bkeepers
malarkey