Ephemeral workloads with Docker.

Talk at PyTennessee on tmpnb, IPython, Jupyter, cloudpipe, and ephemeral workloads with Docker.

Transcript

1. DOCKER FOR EPHEMERAL WORKLOADS Kyle Kelley @ PyTennessee
2. Hi!
3. Good morning!
4. Kyle Kelley @rgbkrk
13. ♥ I
15. Sharing
17. Deployment 㱺
19. FROM python:3-onbuild CMD [“python”, “./app.py”]
20. from flask import Flask app = Flask(__name__) ! @app.route('/') def hello_pytn(): return 'Hello PyTennessee!' ! if __name__ == '__main__': app.run(‘0.0.0.0’)
21. $ docker build -t pytn .
22. $ docker build -t pytn . Sending build context to Docker daemon 92.67 kB Sending build context to Docker daemon Step 0 : FROM python:3-onbuild # Executing 3 build triggers Trigger 0, COPY requirements.txt /usr/src/app/ Step 0 : COPY requirements.txt /usr/src/app/ Trigger 1, RUN pip install -r requirements.txt Step 0 : RUN pip install -r requirements.txt ---> Running in dea1149d79ba … Successfully installed Flask Werkzeug Jinja2 itsdangerous markupsafe Cleaning up...
23. $ docker build -t pytn . … Trigger 2, COPY . /usr/src/app Step 0 : COPY . /usr/src/app ---> 6067170ab7fe Removing intermediate container 7ba25a7d8c3b Removing intermediate container dea1149d79ba Removing intermediate container 98f8579edc94 Step 1 : CMD python ./app.py ---> Running in 53031fa4ba6d ---> a7324c447943 Removing intermediate container 53031fa4ba6d Successfully built a7324c447943
24. $ docker build -t pytn .
25. $ docker run -p 5000:5000 pytn * Running on http://*:5000/ (Press CTRL+C to quit)
27. $ docker run ipython/scipystack
28. ipython/ipython ipython/scipystack ipython/notebook ipython/scipyserver
33. “Could we do a live demo for Nature?”
34. “Yes?”
36. JUPYTERHUB IN ACTION Kyle visits /user/rgbkrk/   to reach his notebook server
37. CONFIGURABLE HTTP PROXY • Based on node-http-proxy • Proxy routes to localhost, external IPs, etc. • /user/123/  →  127.0.0.1:56790   • Provides an administrative API for adding routes • POST  /api/routes/[:path]
38. github.com/jupyter/ configurable-http-proxy
42. “Yes!”
43. github.com/jupyter/tmpnb
44. /user/reqz/ /user/asdf/ /user/uiop/ :59678 :59632 :59680 Configurable HTTP Proxy
46. tmpnb.org
47. bit.ly/inature
48. > 20,000 notebook servers
54. What else is ephemeral?
55. compute jobs
57. >>>  import  cloud   >>>  def  add(x,  y):   ...      return  x+y   >>>  add(1,  2)   3   >>>  jid  =  cloud.call(add,   ...                                    1,  2)
58. ! >>>  cloud.result(jid)   3
59. >>>  import  cloud   >>>  def  scrape(x,y):   ...      #  Expensive  operation   >>>  xs  =  []  #  list  of  x’s   >>>  ys  =  []  #  list  of  y’s   >>>  jids  =  cloud.map(scrape,   ...                                    xs,  ys)
60. >>>  cloud.result(jids)
61. PICLOUD • Chuck a python pickle to the cloud (f, args, kwargs) = pickle.load(sys.stdin) res = f(*args, **kwargs) ! cloudpickle.dump(res, open("/tmp/.result", "wb"))
62. Acquisition
64. :(
66. ?
67. github.com/cloudpipe
69. import  multyvac   ! #  Connect  to  the  cluster   multyvac.config.set_key(      api_key=rackspace_user,      api_secret_key=rackspace_api_key,      api_url=‘https://api.cloudpi.pe/v1’)   ! ! ! !
70. CLOUDPIPE • Docker Backed • Processes run as non-root • API compatible with multyvac
71. CLOUDPIPE • Open Source • Want Need Python 3 binding, new cloud pickler • Node library in the works
72. Where else can we take things?
73. Where else can we take things?
74. On Demand IPython kernels
75. github.com/ rgbkrk/kernels- service
76. bit.ly/oreillyjupyter
77. FRONTIER • Put kernels close to data stores, spin out compute • IPython parallel on demand • Compute jobs on demand • Trigger jobs by events
79. JOIN MY TEAM carla.crull@rackspace.com JOIN RACKSPACE odavid@rackspace.com