Reed Wade: Bottle + uWSGI: simple web app configuration and fun hidden features

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
Reed Wade:
Bottle + uWSGI: simple web app configuration and fun hidden features
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
@ Kiwi PyCon 2014 - Sunday, 14 Sep 2014 - Track 2
http://kiwi.pycon.org/

**Audience level**

Intermediate

**Description**

I will introduce the Bottle Python Web Framework and show how simple it is to configure a python web application to run under Nginx using uWSGI. I'll also demonstrate some lesser known features of uWSGI that let your web app work a little smarter.

**Abstract**

The Bottle Python Web Framework makes it easy to create a little web application. Nginx and uWSGI make it easy for your web app to scale and perform well. In this talk I'll describe how to create and configure an API focused web application.

I'll demonstrate some features of uWSGI which allow your web app to share data amongst separate processes and delegate work to non-client facing processes.

I'll show how all this can fit together with a full stack of: bottle+uWSGI+Nginx+Bootstrap+AngularJS

**YouTube**

https://www.youtube.com/watch?v=wGPGRW1wXdk

Transcript

1. Your Little
   Web API+app
   bottle + uWSGI
   + nginx
   + angularJS
   typist.geek.nz
   Reed Wade
   Kiwi Pycon 2014
   

   View Slide

2. Agenda
   - make a web API
   - make a web API+App
   - add some uWSGI trickery
   - try it at home
   

   View Slide

3. Stack
   - bottle
   - uWSGI
   - Nginx
   - AngularJS
   

   View Slide

4. Stack
   - bottle +a little of YOUR python code
   - uWSGI +a little boilerplate config
   - Nginx +a little boilerplate config
   - AngularJS +a little of YOUR html+javascript
   

   View Slide

5. your python code
   Bottle
   uWSGI
   Nginx
   web browser
   AngularJS (and friends)
   your html+javascript
   

   View Slide

6. your python code
   web API
   your html+javascript
   you have 2 different kinds of customer
   

   View Slide

7. bottle (reed ♥s)
   import bottle
   @bottle.route('/hello')
   def hello():
   return "Hi\n"
   @bottle.route('/hello/')
   def hello_name(name):
   return "Hello {name}\n".format(name=name)
   @bottle.route('/api/fun')
   def hello_fun():
   return {
   'fun': 123,
   }
   if __name__ == '__main__':
   bottle.run(host='0.0.0.0', port=8000, debug=True)
   URL routing
   return a string to
   the browser
   return a dict and it’s
   magically converted to
   JSON! sweet
   convenient dev
   web server
   

   View Slide

8. more like real life
   import bottle
   @bottle.route('/api/fun')
   def hello_fun():
   return {
   'fun': 123,
   }
   @bottle.route('/api/slow')
   def slow():
   time.sleep(8)
   return "you didn't plan this well"
   if __name__ == '__main__':
   bottle.run(host='0.0.0.0', port=8000, debug=True)
   I wish we could tell the user not to worry.
   I wish the entire server wasn’t blocked right now.
   

   View Slide

9. you need a proper application
   server
   uWSGI provides an operating
   environment between the web server
   and your WSGI compliant
   application.
   your python code
   Bottle
   uWSGI
   Nginx
   web browser
   AngularJS (and friends)
   your html+javascript
   

   View Slide

10. /etc/uwsgi/apps-enabled/example-uwsgi.ini
    [uwsgi]
    socket = localhost:8002
    chdir = /opt/example
    file = example.py
    master = true
    plugins = python
    uid = www-data
    gid = www-data
    workers = 6
    Your code
    web server will find us here
    (not HTTP)
    

    View Slide

11. you need a fab web server
    Nginx is a simple high performance
    web server which is flexible and
    easy to configure.
    And it knows how to talk to uWSGI.
    your python code
    Bottle
    uWSGI
    Nginx
    web browser
    AngularJS (and friends)
    your html+javascript
    

    View Slide

12. /etc/nginx/sites-enabled/example-nginx.conf
    upstream example {
    server localhost:8002;
    }
    server {
    server_name example-server;
    root /opt/example/www;
    index index.html;
    location / {
    try_files $uri $uri/ =404;
    }
    location /api {
    include uwsgi_params;
    uwsgi_pass example;
    }
    }
    tells Nginx to send any URL
    starting /api to your uWSGI
    applications
    uWSGI protocol socket
    (not HTTP)
    let Nginx serve static files directly
    

    View Slide

13. install the things
    sudo apt-get install \
    python-bottle \
    uwsgi uwsgi-plugin-python \
    nginx
    

    View Slide

14. files
    /opt/example/example.py
    /opt/example/www/index.html
    /etc/uwsgi/apps-enabled/example-uwsgi.ini
    /etc/nginx/sites-enabled/example-nginx.conf
    

    View Slide

15. Let’s play with AngularJS
    an md5 hash calculator
    you type here
    it updates here
    and shows a new md5 value here
    

    View Slide

16. import bottle
    import hashlib
    app = application = bottle.Bottle()
    @app.route('/compute_md5')
    def do_the_trick():
    return {
    'result': hashlib.md5( bottle.request.query.v ).hexdigest()
    }
    if __name__ == '__main__':
    app.run(application=app, host='0.0.0.0', port=8000, debug=True)
    

    View Slide

17. 
    
    
    
    <br/>var app = angular.module('md5', []);<br/>app.controller('c', function($scope,$http){<br/>$scope.update = function() {<br/>$http.get('/compute_md5?v='+$scope.i)<br/>.success(function(data){<br/>$scope.result = data['result'];<br/>});<br/>}<br/>});<br/>
    
    
    {{i}} is {{result}}
    
    
    

    View Slide

18. done?
    It works, go do something else now.
    Too late. I’d started reading the
    uWSGI docs.
    - mules!
    - shared memory!
    - signals!
    - other stuff not covered today!
    J2EE flashbacks
    

    View Slide

19. mules and workers
    - workers service client requests
    - mules don’t
    number of each is set in uWSGI
    config for your app
    - also ‘programmed mule’
    

    View Slide

20. Caches and Queues
    (not the names I’d have used but they’re not insane)
    - caches are a key/value store
    - queues are arrays
    in your uWSGI config file--
    cache2 = name=mycache,items=100
    queue = 100
    

    View Slide

21. Cache Functions
    uwsgi.cache_get(key[,cache])
    uwsgi.cache_set(key,value[,expires,cache])
    uwsgi.cache_update(key,value[,expires,cache])
    uwsgi.cache_exists(key[,cache])
    uwsgi.cache_del(key[,cache])
    uwsgi.cache_clear([cache])
    

    View Slide

22. Queue Functions
    uwsgi.queue_get(numeric_id)
    uwsgi.queue_set(numeric_id, value)
    uwsgi.queue_last(count)
    uwsgi.queue_push(value)
    uwsgi.queue_pull()
    uwsgi.queue_pop()
    uwsgi.queue_slot()
    uwsgi.queue_pull_slot()
    uwsgi.queue_size
    

    View Slide

23. Signals
    1 byte message sent to listeners:
    - all workers
    - all mules
    - worker or mule by id
    - one “available” worker or mule
    sent explicitly or by timer (or
    file modification)
    

    View Slide

24. Sending Signals
    Receive a signal:
    def thanks(signal):
    print "something happened", uwsgi.mule_id()
    uwsgi.register_signal(111, "mule", thanks)
    Triggers:
    uwsgi.add_timer(111, 60)
    uwsgi.signal(111)
    uwsgi.add_file_monitor(111, "/tmp/cheesecake")
    

    View Slide

25. A more complex app
    - accepts bakery orders
    - and bakes you a cake
    

    View Slide

26. View Slide

27. 
    
    
    
    <br/>var app = angular.module('bakery', []);<br/>app.controller('c', function($scope,$http){<br/>$scope.place_order = function(item) {<br/>$http.get('/api/place_order/'+item).success(function(data){<br/>$scope.order_id = data['order_id'];<br/>});<br/>};<br/>$scope.check_inventory = function(item) {<br/>$http.get('/api/check_inventory').success(function(data){<br/>$scope.inventory = data['result'];<br/>});<br/>};<br/>});<br/>
    
    

    View Slide

28. 
    
    
    {{order_id}}
    
    
    
    Pies:
    
    
    {{item.order_id}} :
    {{item.notes}}
    
    
    Cakes:
    
    
    {{item.order_id}} :
    {{item.notes}}
    
    
    
    

    View Slide

29. NEW_ORDER_SIGNAL = 10
    @app.route('/api/place_order/')
    def place_order(item):
    order_id = str(uuid.uuid4())[-6:]
    uwsgi.queue_push("{order_id}\t{item}\tnew\tthanks".format(
    order_id=order_id, item=item))
    uwsgi.signal(NEW_ORDER_SIGNAL)
    return {'order_id': order_id}
    def mule_new_order(signum):
    print "mule {mule_id} sees a new order".format(
    mule_id=uwsgi.mule_id())
    item = uwsgi.queue_pop()
    item = item.split('\t')
    item[2] = 'baking'
    item[3] = 'mule {mule_id} is baking you a {thing}'.format(
    mule_id=uwsgi.mule_id(), thing=item[1])
    uwsgi.queue_push('\t'.join(item))
    uwsgi.register_signal(NEW_ORDER_SIGNAL, "mule", mule_new_order)
    

    View Slide

30. @app.route('/api/check_inventory')
    def check_inventory():
    out = []
    for i in range(0, uwsgi.queue_size):
    item = uwsgi.queue_get(i)
    if item:
    item = item.split('\t')
    out.append({
    'order_id':item[0],
    'item': item[1],
    'state': item[2],
    'notes': item[3]
    })
    return {'result': out}
    @app.route('/api/reload')
    def reload():
    uwsgi.reload()
    return "reloading"
    

    View Slide

31. opportunities for disaster
    Sat Sep 13 10:41:06 2014 - *** Operational MODE: preforking ***
    Sat Sep 13 10:41:06 2014 - *** no app loaded. going in full dynamic mode
    ***
    Sat Sep 13 10:41:06 2014 - *** uWSGI is running in multiple interpreter
    mode ***
    Sat Sep 13 10:41:06 2014 - !!!!!!!!!!!!!! WARNING !!!!!!!!!!!!!!
    Sat Sep 13 10:41:06 2014 - no request plugin is loaded, you will not be
    able to manage requests.
    Sat Sep 13 10:41:06 2014 - you may need to install the package for your
    language of choice, or simply load it with --plugin.
    Sat Sep 13 10:41:06 2014 - !!!!!!!!!!! END OF WARNING !!!!!!!!!!
    Sat Sep 13 10:41:06 2014 - spawned uWSGI master process (pid: 14552)
    Sat Sep 13 10:41:06 2014 - spawned uWSGI worker 1 (pid: 14553, cores: 1)
    Sat Sep 13 10:41:06 2014 - spawned uWSGI worker 2 (pid: 14554, cores: 1)
    (ALSO -- Application Zero means python)
    Sat Sep 13 11:17:08 2014 - WSGI app 0 (mountpoint='') ready in 0 seconds on
    interpreter 0x20c0f80 pid: 15721 (default app)
    

    View Slide

32. opportunities for disaster
    [uwsgi]
    socket = localhost:8005
    chdir = /opt/example
    master = true
    plugins = python
    file = example.py
    uid = www-data
    gid = www-data
    workers = 6
    order matters?
    no request plugin?
    

    View Slide

33. nginx didn’t attempt to route
    to uWSGI -- didn’t think it
    was supposed to
    the nginx config must not be configured to route /booger
    to a uWSGI upstream server
    

    View Slide

34. this means nginx can’t reach
    uWSGI service - but it tried
    2014/09/13 20:59:24 [error] 20743#0: *353 connect() failed
    (111: Connection refused) while connecting to upstream,
    client: 111.69.102.38, server: typist.geek.nz, request:
    "GET /compute_md5?v=jkkjknjk HTTP/1.1", upstream: "uwsgi:
    //127.0.0.1:8002", host: "typist.geek.nz"
    

    View Slide

35. nginx passed request to uWSGI
    but your application hasn’t
    made a route for the URL
    nginx sent /api/booger to a uWSGI upstream server but it
    wasn’t one that had a route for that URL
    

    View Slide

36. - at import time
    * Restarting app server(s) uwsgi
    [ OK ]
    Sat Sep 13 21:23:08 2014 - *** Operational MODE: preforking ***
    Traceback (most recent call last):
    File "compute_md5.py", line 3, in
    assert False
    AssertionError
    Sat Sep 13 21:23:08 2014 - unable to load app 0 (mountpoint='') (callable not found or
    import error)
    Sat Sep 13 21:23:08 2014 - *** no app loaded. going in full dynamic mode ***
    Sat Sep 13 21:23:08 2014 - *** uWSGI is running in multiple interpreter mode ***
    Sat Sep 13 21:23:08 2014 - spawned uWSGI master process (pid: 25219)
    Sat Sep 13 21:23:08 2014 - spawned uWSGI worker 1 (pid: 25226, cores: 1)
    Sat Sep 13 21:23:08 2014 - spawned uWSGI worker 2 (pid: 25229, cores: 1)
    Sat Sep 13 21:23:08 2014 - spawned uWSGI worker 3 (pid: 25230, cores: 1)
    Sat Sep 13 21:23:08 2014 - spawned uWSGI worker 4 (pid: 25231, cores: 1)
    Sat Sep 13 21:23:08 2014 - spawned uWSGI worker 5 (pid: 25232, cores: 1)
    Sat Sep 13 21:23:08 2014 - spawned uWSGI worker 6 (pid: 25233, cores: 1)
    

    View Slide

37. your uWSGI app broke
    - at request time
    Traceback (most recent call last):
    File "/usr/lib/python2.7/dist-packages/bottle.py", line 856, in _handle
    return route.call(**args)
    File "/usr/lib/python2.7/dist-packages/bottle.py", line 1721, in wrapper
    rv = callback(*a, **ka)
    File "compute_md5.py", line 10, in hello
    assert False
    AssertionError
    [pid: 24351|app: 0|req: 1/1] 111.69.102.38 () {42 vars in 783 bytes} [Sat Sep 13 21:07:
    18 2014] GET /compute_md5?v=jkkjknjk => generated 764 bytes in 19 msecs (HTTP/1.1 500) 2
    headers in 99 bytes (1 switches on core 0)
    

    View Slide

38. tips
    - printing to stdout goes to uWSGI
    log for app
    - touch-reload
    - read the uWSGI docs
    

    View Slide

39. References
    - http://bottlepy.org/
    - http://uwsgi-docs.readthedocs.org/
    - http://uwsgi-docs.readthedocs.org/en/latest/PythonModule.html
    - http://uwsgi-docs.readthedocs.org/en/latest/Signals.html
    - http://uwsgi-docs.readthedocs.org/en/latest/Caching.html
    - http://uwsgi-docs.readthedocs.org/en/latest/Queue.html
    - http://michael.lustfield.net/nginx/bottle-uwsgi-nginx-quickstart
    - http://campus.codeschool.com/courses/shaping-up-with-angular-js/intro
    - https://angularjs.org/
    - https://www.youtube.com/user/dwahlin/videos
    More details and examples at:
    - https://typist.geek.nz/
    

    View Slide

40. When you go home you should
    create a little web app.
    It’s a lot of fun.
    

    View Slide