ephemeral docker workloads with jupyter at pytn

Talk at PyTennessee on tmpnb, nature, IPython, Jupyter, cloudpipe, and ephemeral workloads with Docker.

Transcript

1. Ephemeral Docker Services with IPython, Jupyter, and Cloudpipe
2. Hi!
3. Good morning!
4. Kyle Kelley @rgbkrk
13. ♥ I
15. Sharing
17. Deployment 㱺
19. FROM python:3-onbuild CMD [“python”, “./app.py”]
20. from flask import Flask app = Flask(__name__) ! @app.route('/') def hello_pytn(): return 'Hello PyTennessee!' ! if __name__ == '__main__': app.run(‘0.0.0.0’)
21. $ docker build -t pytn .
22. $ docker build -t pytn . Sending build context to Docker daemon 92.67 kB Sending build context to Docker daemon Step 0 : FROM python:3-onbuild # Executing 3 build triggers Trigger 0, COPY requirements.txt /usr/src/app/ Step 0 : COPY requirements.txt /usr/src/app/ Trigger 1, RUN pip install -r requirements.txt Step 0 : RUN pip install -r requirements.txt ---> Running in dea1149d79ba … Successfully installed Flask Werkzeug Jinja2 itsdangerous markupsafe Cleaning up...
23. $ docker build -t pytn . … Trigger 2, COPY . /usr/src/app Step 0 : COPY . /usr/src/app ---> 6067170ab7fe Removing intermediate container 7ba25a7d8c3b Removing intermediate container dea1149d79ba Removing intermediate container 98f8579edc94 Step 1 : CMD python ./app.py ---> Running in 53031fa4ba6d ---> a7324c447943 Removing intermediate container 53031fa4ba6d Successfully built a7324c447943
24. $ docker build -t pytn .
25. $ docker run -p 5000:5000 pytn * Running on http://*:5000/ (Press CTRL+C to quit)
27. $ docker run ipython/scipystack
28. ipython/ipython ipython/scipystack ipython/notebook ipython/scipyserver
33. “Could we do a live demo for Nature?”
34. “Yes?”
36. JUPYTERHUB IN ACTION Kyle visits /user/rgbkrk/   to reach his notebook server
37. CONFIGURABLE HTTP PROXY • Based on node-http-proxy • Proxy routes to localhost, external IPs, etc. • /user/123/  →  127.0.0.1:56790   • Provides an administrative API for adding routes • POST  /api/routes/[:path]
38. github.com/jupyter/ configurable-http-proxy
42. “Yes!”
43. github.com/jupyter/tmpnb
44. /user/reqz/ /user/asdf/ /user/uiop/ :59678 :59632 :59680 Configurable HTTP Proxy
46. tmpnb.org
47. bit.ly/inature
48. > 20,000 notebook servers
54. What else is ephemeral?
55. compute jobs
57. >>>  import  cloud   >>>  def  add(x,  y):   ...      return  x+y   >>>  add(1,  2)   3   >>>  jid  =  cloud.call(add,   ...                                    1,  2)
58. ! >>>  cloud.result(jid)   3
59. >>>  import  cloud   >>>  def  scrape(x,y):   ...      #  Expensive  operation   >>>  xs  =  []  #  list  of  x’s   >>>  ys  =  []  #  list  of  y’s   >>>  jids  =  cloud.map(scrape,   ...                                    xs,  ys)
60. >>>  cloud.result(jids)
61. PICLOUD • Chuck a python pickle to the cloud (f, args, kwargs) = pickle.load(sys.stdin) res = f(*args, **kwargs) ! cloudpickle.dump(res, open("/tmp/.result", "wb"))
62. Acquisition
64. :(
66. ?
67. github.com/cloudpipe
69. import  multyvac   ! #  Connect  to  the  cluster   multyvac.config.set_key(      api_key=rackspace_user,      api_secret_key=rackspace_api_key,      api_url=‘https://api.cloudpi.pe/v1’)   ! ! ! !
70. CLOUDPIPE • Docker Backed • Processes run as non-root • API compatible with multyvac
71. CLOUDPIPE • Open Source • Want Need Python 3 binding, new cloud pickler • Node library in the works
72. Where else can we take things?
73. On Demand IPython Kernels ! github.com/rgbkrk/ kernels-service
74. bit.ly/oreillyjupyter
75. FRONTIER • Put kernels close to data stores, spin out compute • IPython parallel on demand • Compute jobs on demand • Trigger jobs by events
77. JOIN MY TEAM carla.crull@rackspace.com JOIN RACKSPACE odavid@rackspace.com
78. KYLE KELLEY @rgbkrk rgbkrk@gmail.com