Reed Wade: Bottle + uWSGI: simple web app configuration and fun hidden features

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
Reed Wade:
Bottle + uWSGI: simple web app configuration and fun hidden features
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
@ Kiwi PyCon 2014 - Sunday, 14 Sep 2014 - Track 2
http://kiwi.pycon.org/

**Audience level**

Intermediate

**Description**

I will introduce the Bottle Python Web Framework and show how simple it is to configure a python web application to run under Nginx using uWSGI. I'll also demonstrate some lesser known features of uWSGI that let your web app work a little smarter.

**Abstract**

The Bottle Python Web Framework makes it easy to create a little web application. Nginx and uWSGI make it easy for your web app to scale and perform well. In this talk I'll describe how to create and configure an API focused web application.

I'll demonstrate some features of uWSGI which allow your web app to share data amongst separate processes and delegate work to non-client facing processes.

I'll show how all this can fit together with a full stack of: bottle+uWSGI+Nginx+Bootstrap+AngularJS

**YouTube**

https://www.youtube.com/watch?v=wGPGRW1wXdk

Transcript

1. Your Little Web API+app bottle + uWSGI + nginx + angularJS typist.geek.nz Reed Wade Kiwi Pycon 2014
2. Agenda - make a web API - make a web API+App - add some uWSGI trickery - try it at home
3. Stack - bottle - uWSGI - Nginx - AngularJS
4. Stack - bottle +a little of YOUR python code - uWSGI +a little boilerplate config - Nginx +a little boilerplate config - AngularJS +a little of YOUR html+javascript
5. your python code Bottle uWSGI Nginx web browser AngularJS (and friends) your html+javascript
6. your python code web API your html+javascript you have 2 different kinds of customer
7. bottle (reed ♥s) import bottle @bottle.route('/hello') def hello(): return "Hi\n" @bottle.route('/hello/<name>') def hello_name(name): return "Hello {name}\n".format(name=name) @bottle.route('/api/fun') def hello_fun(): return { 'fun': 123, } if __name__ == '__main__': bottle.run(host='0.0.0.0', port=8000, debug=True) URL routing return a string to the browser return a dict and it’s magically converted to JSON! sweet convenient dev web server
8. more like real life import bottle @bottle.route('/api/fun') def hello_fun(): return { 'fun': 123, } @bottle.route('/api/slow') def slow(): time.sleep(8) return "you didn't plan this well" if __name__ == '__main__': bottle.run(host='0.0.0.0', port=8000, debug=True) I wish we could tell the user not to worry. I wish the entire server wasn’t blocked right now.
9. you need a proper application server uWSGI provides an operating environment between the web server and your WSGI compliant application. your python code Bottle uWSGI Nginx web browser AngularJS (and friends) your html+javascript
10. /etc/uwsgi/apps-enabled/example-uwsgi.ini [uwsgi] socket = localhost:8002 chdir = /opt/example file = example.py master = true plugins = python uid = www-data gid = www-data workers = 6 Your code web server will find us here (not HTTP)
11. you need a fab web server Nginx is a simple high performance web server which is flexible and easy to configure. And it knows how to talk to uWSGI. your python code Bottle uWSGI Nginx web browser AngularJS (and friends) your html+javascript
12. /etc/nginx/sites-enabled/example-nginx.conf upstream example { server localhost:8002; } server { server_name example-server; root /opt/example/www; index index.html; location / { try_files $uri $uri/ =404; } location /api { include uwsgi_params; uwsgi_pass example; } } tells Nginx to send any URL starting /api to your uWSGI applications uWSGI protocol socket (not HTTP) let Nginx serve static files directly
13. install the things sudo apt-get install \ python-bottle \ uwsgi uwsgi-plugin-python \ nginx
14. files /opt/example/example.py /opt/example/www/index.html /etc/uwsgi/apps-enabled/example-uwsgi.ini /etc/nginx/sites-enabled/example-nginx.conf
15. Let’s play with AngularJS an md5 hash calculator you type here it updates here and shows a new md5 value here
16. import bottle import hashlib app = application = bottle.Bottle() @app.route('/compute_md5') def do_the_trick(): return { 'result': hashlib.md5( bottle.request.query.v ).hexdigest() } if __name__ == '__main__': app.run(application=app, host='0.0.0.0', port=8000, debug=True)
17. <!DOCTYPE html> <html lang="en" ng-app="md5"> <head> <script src="http://ajax.googleapis.com/ajax/libs/angularjs/1.2.23/angular.min. js"></script> <script type="text/javascript"> var app = angular.module('md5', []); app.controller('c', function($scope,$http){ $scope.update = function() { $http.get('/compute_md5?v='+$scope.i) .success(function(data){ $scope.result = data['result']; }); } }); </script> </head> <body ng-controller="c"> <input ng-model="i" ng-change="update()" /> {{i}} is {{result}} </body> </html>
18. done? It works, go do something else now. Too late. I’d started reading the uWSGI docs. - mules! - shared memory! - signals! - other stuff not covered today! J2EE flashbacks
19. mules and workers - workers service client requests - mules don’t number of each is set in uWSGI config for your app - also ‘programmed mule’
20. Caches and Queues (not the names I’d have used but they’re not insane) - caches are a key/value store - queues are arrays in your uWSGI config file-- cache2 = name=mycache,items=100 queue = 100
21. Cache Functions uwsgi.cache_get(key[,cache]) uwsgi.cache_set(key,value[,expires,cache]) uwsgi.cache_update(key,value[,expires,cache]) uwsgi.cache_exists(key[,cache]) uwsgi.cache_del(key[,cache]) uwsgi.cache_clear([cache])
22. Queue Functions uwsgi.queue_get(numeric_id) uwsgi.queue_set(numeric_id, value) uwsgi.queue_last(count) uwsgi.queue_push(value) uwsgi.queue_pull() uwsgi.queue_pop() uwsgi.queue_slot() uwsgi.queue_pull_slot() uwsgi.queue_size
23. Signals 1 byte message sent to listeners: - all workers - all mules - worker or mule by id - one “available” worker or mule sent explicitly or by timer (or file modification)
24. Sending Signals Receive a signal: def thanks(signal): print "something happened", uwsgi.mule_id() uwsgi.register_signal(111, "mule", thanks) Triggers: uwsgi.add_timer(111, 60) uwsgi.signal(111) uwsgi.add_file_monitor(111, "/tmp/cheesecake")
25. A more complex app - accepts bakery orders - and bakes you a cake
27. <!DOCTYPE html> <html lang="en" ng-app="bakery"> <head> <script src="http://ajax.googleapis.com/ajax/libs/angularjs/1.2.23/angular.min. js"></script> <script type="text/javascript"> var app = angular.module('bakery', []); app.controller('c', function($scope,$http){ $scope.place_order = function(item) { $http.get('/api/place_order/'+item).success(function(data){ $scope.order_id = data['order_id']; }); }; $scope.check_inventory = function(item) { $http.get('/api/check_inventory').success(function(data){ $scope.inventory = data['result']; }); }; }); </script> </head>
28. <body ng-controller="c"> <input type="button" ng-click="place_order('cake')" value="Order a Cake" /> <input type="button" ng-click="place_order('pie')" value="Order a Pie" /> {{order_id}} <hr/> <input type="button" ng-click="check_inventory()" value="Check Inventory" /> <hr/> Pies: <ul> <li ng-repeat="item in inventory" ng-show="item.item=='pie'"> {{item.order_id}} : {{item.notes}} </li> </ul> Cakes: <ul> <li ng-repeat="item in inventory" ng-show="item.item=='cake'"> {{item.order_id}} : {{item.notes}} </li> </ul> </body> </html>
29. NEW_ORDER_SIGNAL = 10 @app.route('/api/place_order/<item>') def place_order(item): order_id = str(uuid.uuid4())[-6:] uwsgi.queue_push("{order_id}\t{item}\tnew\tthanks".format( order_id=order_id, item=item)) uwsgi.signal(NEW_ORDER_SIGNAL) return {'order_id': order_id} def mule_new_order(signum): print "mule {mule_id} sees a new order".format( mule_id=uwsgi.mule_id()) item = uwsgi.queue_pop() item = item.split('\t') item[2] = 'baking' item[3] = 'mule {mule_id} is baking you a {thing}'.format( mule_id=uwsgi.mule_id(), thing=item[1]) uwsgi.queue_push('\t'.join(item)) uwsgi.register_signal(NEW_ORDER_SIGNAL, "mule", mule_new_order)
30. @app.route('/api/check_inventory') def check_inventory(): out = [] for i in range(0, uwsgi.queue_size): item = uwsgi.queue_get(i) if item: item = item.split('\t') out.append({ 'order_id':item[0], 'item': item[1], 'state': item[2], 'notes': item[3] }) return {'result': out} @app.route('/api/reload') def reload(): uwsgi.reload() return "reloading"
31. opportunities for disaster Sat Sep 13 10:41:06 2014 - *** Operational MODE: preforking *** Sat Sep 13 10:41:06 2014 - *** no app loaded. going in full dynamic mode *** Sat Sep 13 10:41:06 2014 - *** uWSGI is running in multiple interpreter mode *** Sat Sep 13 10:41:06 2014 - !!!!!!!!!!!!!! WARNING !!!!!!!!!!!!!! Sat Sep 13 10:41:06 2014 - no request plugin is loaded, you will not be able to manage requests. Sat Sep 13 10:41:06 2014 - you may need to install the package for your language of choice, or simply load it with --plugin. Sat Sep 13 10:41:06 2014 - !!!!!!!!!!! END OF WARNING !!!!!!!!!! Sat Sep 13 10:41:06 2014 - spawned uWSGI master process (pid: 14552) Sat Sep 13 10:41:06 2014 - spawned uWSGI worker 1 (pid: 14553, cores: 1) Sat Sep 13 10:41:06 2014 - spawned uWSGI worker 2 (pid: 14554, cores: 1) (ALSO -- Application Zero means python) Sat Sep 13 11:17:08 2014 - WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x20c0f80 pid: 15721 (default app)
32. opportunities for disaster [uwsgi] socket = localhost:8005 chdir = /opt/example master = true plugins = python file = example.py uid = www-data gid = www-data workers = 6 order matters? no request plugin?
33. nginx didn’t attempt to route to uWSGI -- didn’t think it was supposed to the nginx config must not be configured to route /booger to a uWSGI upstream server
34. this means nginx can’t reach uWSGI service - but it tried 2014/09/13 20:59:24 [error] 20743#0: *353 connect() failed (111: Connection refused) while connecting to upstream, client: 111.69.102.38, server: typist.geek.nz, request: "GET /compute_md5?v=jkkjknjk HTTP/1.1", upstream: "uwsgi: //127.0.0.1:8002", host: "typist.geek.nz"
35. nginx passed request to uWSGI but your application hasn’t made a route for the URL nginx sent /api/booger to a uWSGI upstream server but it wasn’t one that had a route for that URL
36. - at import time * Restarting app server(s) uwsgi [ OK ] Sat Sep 13 21:23:08 2014 - *** Operational MODE: preforking *** Traceback (most recent call last): File "compute_md5.py", line 3, in <module> assert False AssertionError Sat Sep 13 21:23:08 2014 - unable to load app 0 (mountpoint='') (callable not found or import error) Sat Sep 13 21:23:08 2014 - *** no app loaded. going in full dynamic mode *** Sat Sep 13 21:23:08 2014 - *** uWSGI is running in multiple interpreter mode *** Sat Sep 13 21:23:08 2014 - spawned uWSGI master process (pid: 25219) Sat Sep 13 21:23:08 2014 - spawned uWSGI worker 1 (pid: 25226, cores: 1) Sat Sep 13 21:23:08 2014 - spawned uWSGI worker 2 (pid: 25229, cores: 1) Sat Sep 13 21:23:08 2014 - spawned uWSGI worker 3 (pid: 25230, cores: 1) Sat Sep 13 21:23:08 2014 - spawned uWSGI worker 4 (pid: 25231, cores: 1) Sat Sep 13 21:23:08 2014 - spawned uWSGI worker 5 (pid: 25232, cores: 1) Sat Sep 13 21:23:08 2014 - spawned uWSGI worker 6 (pid: 25233, cores: 1)
37. your uWSGI app broke - at request time Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/bottle.py", line 856, in _handle return route.call(**args) File "/usr/lib/python2.7/dist-packages/bottle.py", line 1721, in wrapper rv = callback(*a, **ka) File "compute_md5.py", line 10, in hello assert False AssertionError [pid: 24351|app: 0|req: 1/1] 111.69.102.38 () {42 vars in 783 bytes} [Sat Sep 13 21:07: 18 2014] GET /compute_md5?v=jkkjknjk => generated 764 bytes in 19 msecs (HTTP/1.1 500) 2 headers in 99 bytes (1 switches on core 0)
38. tips - printing to stdout goes to uWSGI log for app - touch-reload - read the uWSGI docs
39. References - http://bottlepy.org/ - http://uwsgi-docs.readthedocs.org/ - http://uwsgi-docs.readthedocs.org/en/latest/PythonModule.html - http://uwsgi-docs.readthedocs.org/en/latest/Signals.html - http://uwsgi-docs.readthedocs.org/en/latest/Caching.html - http://uwsgi-docs.readthedocs.org/en/latest/Queue.html - http://michael.lustfield.net/nginx/bottle-uwsgi-nginx-quickstart - http://campus.codeschool.com/courses/shaping-up-with-angular-js/intro - https://angularjs.org/ - https://www.youtube.com/user/dwahlin/videos More details and examples at: - https://typist.geek.nz/
40. When you go home you should create a little web app. It’s a lot of fun.