Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ephemeral docker workloads with jupyter at pytn

E76c7ebc9d2e8a4b840f13cd01946437?s=47 Kyle Kelley
February 07, 2015

ephemeral docker workloads with jupyter at pytn

Talk at PyTennessee on tmpnb, nature, IPython, Jupyter, cloudpipe, and ephemeral workloads with Docker.

E76c7ebc9d2e8a4b840f13cd01946437?s=128

Kyle Kelley

February 07, 2015
Tweet

More Decks by Kyle Kelley

Other Decks in Technology

Transcript

  1. Ephemeral Docker Services with IPython, Jupyter, and Cloudpipe

  2. Hi!

  3. Good morning!

  4. Kyle Kelley @rgbkrk

  5. None
  6. None
  7. None
  8. None
  9. None
  10. None
  11. None
  12. None
  13. ♥ I

  14. None
  15. Sharing

  16. None
  17. Deployment 㱺

  18. None
  19. FROM python:3-onbuild CMD [“python”, “./app.py”]

  20. from flask import Flask app = Flask(__name__) ! @app.route('/') def

    hello_pytn(): return 'Hello PyTennessee!' ! if __name__ == '__main__': app.run(‘0.0.0.0’)
  21. $ docker build -t pytn .

  22. $ docker build -t pytn . Sending build context to

    Docker daemon 92.67 kB Sending build context to Docker daemon Step 0 : FROM python:3-onbuild # Executing 3 build triggers Trigger 0, COPY requirements.txt /usr/src/app/ Step 0 : COPY requirements.txt /usr/src/app/ Trigger 1, RUN pip install -r requirements.txt Step 0 : RUN pip install -r requirements.txt ---> Running in dea1149d79ba … Successfully installed Flask Werkzeug Jinja2 itsdangerous markupsafe Cleaning up...
  23. $ docker build -t pytn . … Trigger 2, COPY

    . /usr/src/app Step 0 : COPY . /usr/src/app ---> 6067170ab7fe Removing intermediate container 7ba25a7d8c3b Removing intermediate container dea1149d79ba Removing intermediate container 98f8579edc94 Step 1 : CMD python ./app.py ---> Running in 53031fa4ba6d ---> a7324c447943 Removing intermediate container 53031fa4ba6d Successfully built a7324c447943
  24. $ docker build -t pytn .

  25. $ docker run -p 5000:5000 pytn * Running on http://*:5000/

    (Press CTRL+C to quit)
  26. None
  27. $ docker run ipython/scipystack

  28. ipython/ipython ipython/scipystack ipython/notebook ipython/scipyserver

  29. None
  30. None
  31. None
  32. None
  33. “Could we do a live demo for Nature?”

  34. “Yes?”

  35. None
  36. JUPYTERHUB IN ACTION Kyle visits /user/rgbkrk/   to reach his

    notebook server
  37. CONFIGURABLE HTTP PROXY • Based on node-http-proxy • Proxy routes

    to localhost, external IPs, etc. • /user/123/  →  127.0.0.1:56790   • Provides an administrative API for adding routes • POST  /api/routes/[:path]
  38. github.com/jupyter/ configurable-http-proxy

  39. None
  40. None
  41. None
  42. “Yes!”

  43. github.com/jupyter/tmpnb

  44. /user/reqz/ /user/asdf/ /user/uiop/ :59678 :59632 :59680 Configurable HTTP Proxy

  45. None
  46. tmpnb.org

  47. bit.ly/inature

  48. > 20,000 notebook servers

  49. None
  50. None
  51. None
  52. None
  53. None
  54. What else is ephemeral?

  55. compute jobs

  56. None
  57. >>>  import  cloud   >>>  def  add(x,  y):   ...

         return  x+y   >>>  add(1,  2)   3   >>>  jid  =  cloud.call(add,   ...                                    1,  2)
  58. ! >>>  cloud.result(jid)   3

  59. >>>  import  cloud   >>>  def  scrape(x,y):   ...  

       #  Expensive  operation   >>>  xs  =  []  #  list  of  x’s   >>>  ys  =  []  #  list  of  y’s   >>>  jids  =  cloud.map(scrape,   ...                                    xs,  ys)
  60. >>>  cloud.result(jids)

  61. PICLOUD • Chuck a python pickle to the cloud (f,

    args, kwargs) = pickle.load(sys.stdin) res = f(*args, **kwargs) ! cloudpickle.dump(res, open("/tmp/.result", "wb"))
  62. Acquisition

  63. None
  64. :(

  65. None
  66. ?

  67. github.com/cloudpipe

  68. None
  69. import  multyvac   ! #  Connect  to  the  cluster  

    multyvac.config.set_key(      api_key=rackspace_user,      api_secret_key=rackspace_api_key,      api_url=‘https://api.cloudpi.pe/v1’)   ! ! ! !
  70. CLOUDPIPE • Docker Backed • Processes run as non-root •

    API compatible with multyvac
  71. CLOUDPIPE • Open Source • Want Need Python 3 binding,

    new cloud pickler • Node library in the works
  72. Where else can we take things?

  73. On Demand IPython Kernels ! github.com/rgbkrk/ kernels-service

  74. bit.ly/oreillyjupyter

  75. FRONTIER • Put kernels close to data stores, spin out

    compute • IPython parallel on demand • Compute jobs on demand • Trigger jobs by events
  76. None
  77. JOIN MY TEAM carla.crull@rackspace.com JOIN RACKSPACE odavid@rackspace.com

  78. KYLE KELLEY @rgbkrk rgbkrk@gmail.com