Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ephemeral docker workloads with jupyter at pytn

Kyle Kelley
February 07, 2015

ephemeral docker workloads with jupyter at pytn

Talk at PyTennessee on tmpnb, nature, IPython, Jupyter, cloudpipe, and ephemeral workloads with Docker.

Kyle Kelley

February 07, 2015
Tweet

More Decks by Kyle Kelley

Other Decks in Technology

Transcript

  1. Hi!

  2. from flask import Flask app = Flask(__name__) ! @app.route('/') def

    hello_pytn(): return 'Hello PyTennessee!' ! if __name__ == '__main__': app.run(‘0.0.0.0’)
  3. $ docker build -t pytn . Sending build context to

    Docker daemon 92.67 kB Sending build context to Docker daemon Step 0 : FROM python:3-onbuild # Executing 3 build triggers Trigger 0, COPY requirements.txt /usr/src/app/ Step 0 : COPY requirements.txt /usr/src/app/ Trigger 1, RUN pip install -r requirements.txt Step 0 : RUN pip install -r requirements.txt ---> Running in dea1149d79ba … Successfully installed Flask Werkzeug Jinja2 itsdangerous markupsafe Cleaning up...
  4. $ docker build -t pytn . … Trigger 2, COPY

    . /usr/src/app Step 0 : COPY . /usr/src/app ---> 6067170ab7fe Removing intermediate container 7ba25a7d8c3b Removing intermediate container dea1149d79ba Removing intermediate container 98f8579edc94 Step 1 : CMD python ./app.py ---> Running in 53031fa4ba6d ---> a7324c447943 Removing intermediate container 53031fa4ba6d Successfully built a7324c447943
  5. CONFIGURABLE HTTP PROXY • Based on node-http-proxy • Proxy routes

    to localhost, external IPs, etc. • /user/123/  →  127.0.0.1:56790   • Provides an administrative API for adding routes • POST  /api/routes/[:path]
  6. >>>  import  cloud   >>>  def  add(x,  y):   ...

         return  x+y   >>>  add(1,  2)   3   >>>  jid  =  cloud.call(add,   ...                                    1,  2)
  7. >>>  import  cloud   >>>  def  scrape(x,y):   ...  

       #  Expensive  operation   >>>  xs  =  []  #  list  of  x’s   >>>  ys  =  []  #  list  of  y’s   >>>  jids  =  cloud.map(scrape,   ...                                    xs,  ys)
  8. PICLOUD • Chuck a python pickle to the cloud (f,

    args, kwargs) = pickle.load(sys.stdin) res = f(*args, **kwargs) ! cloudpickle.dump(res, open("/tmp/.result", "wb"))
  9. :(

  10. ?

  11. import  multyvac   ! #  Connect  to  the  cluster  

    multyvac.config.set_key(      api_key=rackspace_user,      api_secret_key=rackspace_api_key,      api_url=‘https://api.cloudpi.pe/v1’)   ! ! ! !
  12. CLOUDPIPE • Open Source • Want Need Python 3 binding,

    new cloud pickler • Node library in the works
  13. FRONTIER • Put kernels close to data stores, spin out

    compute • IPython parallel on demand • Compute jobs on demand • Trigger jobs by events