$30 off During Our Annual Pro Sale. View Details »

ephemeral docker workloads with jupyter at pytn

Kyle Kelley
February 07, 2015

ephemeral docker workloads with jupyter at pytn

Talk at PyTennessee on tmpnb, nature, IPython, Jupyter, cloudpipe, and ephemeral workloads with Docker.

Kyle Kelley

February 07, 2015
Tweet

More Decks by Kyle Kelley

Other Decks in Technology

Transcript

  1. Ephemeral Docker
    Services
    with

    IPython, Jupyter, and
    Cloudpipe

    View Slide

  2. Hi!

    View Slide

  3. Good
    morning!

    View Slide

  4. Kyle Kelley

    @rgbkrk

    View Slide

  5. View Slide

  6. View Slide

  7. View Slide

  8. View Slide

  9. View Slide

  10. View Slide

  11. View Slide

  12. View Slide


  13. I

    View Slide

  14. View Slide

  15. Sharing

    View Slide

  16. View Slide

  17. Deployment


    View Slide

  18. View Slide

  19. FROM python:3-onbuild
    CMD [“python”, “./app.py”]

    View Slide

  20. from flask import Flask
    app = Flask(__name__)
    !
    @app.route('/')
    def hello_pytn():
    return 'Hello PyTennessee!'
    !
    if __name__ == '__main__':
    app.run(‘0.0.0.0’)

    View Slide

  21. $ docker build -t pytn .

    View Slide

  22. $ docker build -t pytn .
    Sending build context to Docker daemon 92.67 kB
    Sending build context to Docker daemon
    Step 0 : FROM python:3-onbuild
    # Executing 3 build triggers
    Trigger 0, COPY requirements.txt /usr/src/app/
    Step 0 : COPY requirements.txt /usr/src/app/
    Trigger 1, RUN pip install -r requirements.txt
    Step 0 : RUN pip install -r requirements.txt
    ---> Running in dea1149d79ba

    Successfully installed Flask Werkzeug Jinja2
    itsdangerous markupsafe
    Cleaning up...

    View Slide

  23. $ docker build -t pytn .

    Trigger 2, COPY . /usr/src/app
    Step 0 : COPY . /usr/src/app
    ---> 6067170ab7fe
    Removing intermediate container 7ba25a7d8c3b
    Removing intermediate container dea1149d79ba
    Removing intermediate container 98f8579edc94
    Step 1 : CMD python ./app.py
    ---> Running in 53031fa4ba6d
    ---> a7324c447943
    Removing intermediate container 53031fa4ba6d
    Successfully built a7324c447943

    View Slide

  24. $ docker build -t pytn .

    View Slide

  25. $ docker run -p 5000:5000 pytn
    * Running on http://*:5000/
    (Press CTRL+C to quit)

    View Slide

  26. View Slide

  27. $ docker run ipython/scipystack

    View Slide

  28. ipython/ipython
    ipython/scipystack
    ipython/notebook
    ipython/scipyserver

    View Slide

  29. View Slide

  30. View Slide

  31. View Slide

  32. View Slide

  33. “Could we do a live
    demo for Nature?”

    View Slide

  34. “Yes?”

    View Slide

  35. View Slide

  36. JUPYTERHUB IN ACTION
    Kyle visits

    /user/rgbkrk/  
    to reach his
    notebook server

    View Slide

  37. CONFIGURABLE HTTP PROXY
    • Based on node-http-proxy

    • Proxy routes to localhost, external IPs, etc.

    • /user/123/  →  127.0.0.1:56790  
    • Provides an administrative API for adding routes

    • POST  /api/routes/[:path]

    View Slide

  38. github.com/jupyter/
    configurable-http-proxy

    View Slide

  39. View Slide

  40. View Slide

  41. View Slide

  42. “Yes!”

    View Slide

  43. github.com/jupyter/tmpnb

    View Slide

  44. /user/reqz/
    /user/asdf/
    /user/uiop/
    :59678
    :59632
    :59680
    Configurable HTTP Proxy

    View Slide

  45. View Slide

  46. tmpnb.org

    View Slide

  47. bit.ly/inature

    View Slide

  48. > 20,000
    notebook
    servers

    View Slide

  49. View Slide

  50. View Slide

  51. View Slide

  52. View Slide

  53. View Slide

  54. What else is
    ephemeral?

    View Slide

  55. compute
    jobs

    View Slide

  56. View Slide

  57. >>>  import  cloud  
    >>>  def  add(x,  y):  
    ...      return  x+y  
    >>>  add(1,  2)  
    3  
    >>>  jid  =  cloud.call(add,  
    ...                                    1,  2)

    View Slide

  58. !
    >>>  cloud.result(jid)  
    3

    View Slide

  59. >>>  import  cloud  
    >>>  def  scrape(x,y):  
    ...      #  Expensive  operation  
    >>>  xs  =  []  #  list  of  x’s  
    >>>  ys  =  []  #  list  of  y’s  
    >>>  jids  =  cloud.map(scrape,  
    ...                                    xs,  ys)

    View Slide

  60. >>>  cloud.result(jids)

    View Slide

  61. PICLOUD
    • Chuck a python pickle to the cloud
    (f, args, kwargs) = pickle.load(sys.stdin)
    res = f(*args, **kwargs)
    !
    cloudpickle.dump(res, open("/tmp/.result", "wb"))

    View Slide

  62. Acquisition

    View Slide

  63. View Slide

  64. :(

    View Slide

  65. View Slide

  66. ?

    View Slide

  67. github.com/cloudpipe

    View Slide

  68. View Slide

  69. import  multyvac  
    !
    #  Connect  to  the  cluster  
    multyvac.config.set_key(  
       api_key=rackspace_user,  
       api_secret_key=rackspace_api_key,  
       api_url=‘https://api.cloudpi.pe/v1’)  
    !
    !
    !
    !

    View Slide

  70. CLOUDPIPE
    • Docker Backed

    • Processes run as non-root

    • API compatible with multyvac

    View Slide

  71. CLOUDPIPE
    • Open Source

    • Want Need Python 3 binding, new cloud pickler

    • Node library in the works

    View Slide

  72. Where else can

    we take things?

    View Slide

  73. On Demand

    IPython Kernels

    !
    github.com/rgbkrk/
    kernels-service

    View Slide

  74. bit.ly/oreillyjupyter

    View Slide

  75. FRONTIER
    • Put kernels close to data stores, spin out compute

    • IPython parallel on demand

    • Compute jobs on demand

    • Trigger jobs by events

    View Slide

  76. View Slide

  77. JOIN MY TEAM
    [email protected]
    JOIN RACKSPACE
    [email protected]

    View Slide

  78. KYLE KELLEY
    @rgbkrk

    [email protected]

    View Slide