ephemeral docker workloads with jupyter at pytn

E76c7ebc9d2e8a4b840f13cd01946437?s=47 Kyle Kelley
February 07, 2015

ephemeral docker workloads with jupyter at pytn

Talk at PyTennessee on tmpnb, nature, IPython, Jupyter, cloudpipe, and ephemeral workloads with Docker.

E76c7ebc9d2e8a4b840f13cd01946437?s=128

Kyle Kelley

February 07, 2015
Tweet

Transcript

  1. Ephemeral Docker Services with IPython, Jupyter, and Cloudpipe

  2. Hi!

  3. Good morning!

  4. Kyle Kelley @rgbkrk

  5. None
  6. None
  7. None
  8. None
  9. None
  10. None
  11. None
  12. None
  13. ♥ I

  14. None
  15. Sharing

  16. None
  17. Deployment 㱺

  18. None
  19. FROM python:3-onbuild CMD [“python”, “./app.py”]

  20. from flask import Flask app = Flask(__name__) ! @app.route('/') def

    hello_pytn(): return 'Hello PyTennessee!' ! if __name__ == '__main__': app.run(‘0.0.0.0’)
  21. $ docker build -t pytn .

  22. $ docker build -t pytn . Sending build context to

    Docker daemon 92.67 kB Sending build context to Docker daemon Step 0 : FROM python:3-onbuild # Executing 3 build triggers Trigger 0, COPY requirements.txt /usr/src/app/ Step 0 : COPY requirements.txt /usr/src/app/ Trigger 1, RUN pip install -r requirements.txt Step 0 : RUN pip install -r requirements.txt ---> Running in dea1149d79ba … Successfully installed Flask Werkzeug Jinja2 itsdangerous markupsafe Cleaning up...
  23. $ docker build -t pytn . … Trigger 2, COPY

    . /usr/src/app Step 0 : COPY . /usr/src/app ---> 6067170ab7fe Removing intermediate container 7ba25a7d8c3b Removing intermediate container dea1149d79ba Removing intermediate container 98f8579edc94 Step 1 : CMD python ./app.py ---> Running in 53031fa4ba6d ---> a7324c447943 Removing intermediate container 53031fa4ba6d Successfully built a7324c447943
  24. $ docker build -t pytn .

  25. $ docker run -p 5000:5000 pytn * Running on http://*:5000/

    (Press CTRL+C to quit)
  26. None
  27. $ docker run ipython/scipystack

  28. ipython/ipython ipython/scipystack ipython/notebook ipython/scipyserver

  29. None
  30. None
  31. None
  32. None
  33. “Could we do a live demo for Nature?”

  34. “Yes?”

  35. None
  36. JUPYTERHUB IN ACTION Kyle visits /user/rgbkrk/   to reach his

    notebook server
  37. CONFIGURABLE HTTP PROXY • Based on node-http-proxy • Proxy routes

    to localhost, external IPs, etc. • /user/123/  →  127.0.0.1:56790   • Provides an administrative API for adding routes • POST  /api/routes/[:path]
  38. github.com/jupyter/ configurable-http-proxy

  39. None
  40. None
  41. None
  42. “Yes!”

  43. github.com/jupyter/tmpnb

  44. /user/reqz/ /user/asdf/ /user/uiop/ :59678 :59632 :59680 Configurable HTTP Proxy

  45. None
  46. tmpnb.org

  47. bit.ly/inature

  48. > 20,000 notebook servers

  49. None
  50. None
  51. None
  52. None
  53. None
  54. What else is ephemeral?

  55. compute jobs

  56. None
  57. >>>  import  cloud   >>>  def  add(x,  y):   ...

         return  x+y   >>>  add(1,  2)   3   >>>  jid  =  cloud.call(add,   ...                                    1,  2)
  58. ! >>>  cloud.result(jid)   3

  59. >>>  import  cloud   >>>  def  scrape(x,y):   ...  

       #  Expensive  operation   >>>  xs  =  []  #  list  of  x’s   >>>  ys  =  []  #  list  of  y’s   >>>  jids  =  cloud.map(scrape,   ...                                    xs,  ys)
  60. >>>  cloud.result(jids)

  61. PICLOUD • Chuck a python pickle to the cloud (f,

    args, kwargs) = pickle.load(sys.stdin) res = f(*args, **kwargs) ! cloudpickle.dump(res, open("/tmp/.result", "wb"))
  62. Acquisition

  63. None
  64. :(

  65. None
  66. ?

  67. github.com/cloudpipe

  68. None
  69. import  multyvac   ! #  Connect  to  the  cluster  

    multyvac.config.set_key(      api_key=rackspace_user,      api_secret_key=rackspace_api_key,      api_url=‘https://api.cloudpi.pe/v1’)   ! ! ! !
  70. CLOUDPIPE • Docker Backed • Processes run as non-root •

    API compatible with multyvac
  71. CLOUDPIPE • Open Source • Want Need Python 3 binding,

    new cloud pickler • Node library in the works
  72. Where else can we take things?

  73. On Demand IPython Kernels ! github.com/rgbkrk/ kernels-service

  74. bit.ly/oreillyjupyter

  75. FRONTIER • Put kernels close to data stores, spin out

    compute • IPython parallel on demand • Compute jobs on demand • Trigger jobs by events
  76. None
  77. JOIN MY TEAM carla.crull@rackspace.com JOIN RACKSPACE odavid@rackspace.com

  78. KYLE KELLEY @rgbkrk rgbkrk@gmail.com