Gunicorn, the thundering herd and other concurrency programming challenges

Gunicorn design has not changed much since the project started. With the increasing number of cores and new concurrency solutions that appeared in python, it is about time to face to new challenges.

This talk will describe the design of Gunicorn, the different solutions we provide to handle more and more concurrent connections. How we fix the thundering herd problem, what solutions are possible and what we finally adopted. It will also show how we improved the support of Python 3 with the addition of the workers gaiohttp and gthreads and how you can use them. While we are here, the talk will recapitulate the different strategies you have to handle the connections coming to your application using the different workers depending on your concurrent programming framework or platform. It will finally highlight the recent changes in the support of different Python web frameworks.

Talk given at Pycon JP 2014

Transcript

1. the thundering herd and other concurrency programming challenges Pycon JP 2014 - Tokyo By Benoit Chesneau
2. GUNICORN DESIGN
3. prefork-worker model share sockets between OS processes workers are monitored allows different kind of workers gunicorn design
4. arbiter worker spawn & share sockets share socket
5. each worker waits (accepts) for a connection the OS balance connections in the backlog sync worker accept only 1 connection at a time async workers accept many connections competition to accept
6. arbiter worker notify monitor workers kill
7. the arbiter workflow Signal check the 
 signal queue Put in signal queue handle the signal not empty empty murder workers sleep manage workers
8. different concurrency strategy different platforms support don’t want to handle WSGI only more… why different workers?
9. THE THUNDERING HERD
 PROBLEM
10. All process can be awoken on accept for the same connection can trigger EAGAIN increase the CPU usage for nothing. not true with modern OSes for 1 socket the thundering herd problem
11. ticket Gunicorn#792 Introduce signaling between workers and the arbiter using a Pipe Use our own internal messaging implementation the arbiter checks if the a socket can read Workers asks for more work or notify they are alive fixing thundering herd: how to solve it
12. arbiter worker new signaling send message
13. Messages Arbiter Worker ACCEPT:SOCKET ALIVE:TIMESTAMP ACCEPT
14. the arbiter has a new job All workers need to be updated new signaling API to be handled by the worker: an helper module will be provided introduce complexity
15. with one socket, let all workers accepting if wanted depending on the load, bypass the signaling possible optimisations
16. ONE COMMAND TO RULE THEM ALL
17. no more gunicorn_django, gunicorn_paster fix preloading issue offer the same experience for all WSGI applications new settings: —paste, —pythonpath, —env Only one command in 20.0 (deprecations are removed) gunicorn - the only command
18. Enhance the user experience
19. ticket Gunicorn#837 plugins everywhere: logging, workers more embeddable: the gunicorn engine A new organization: 
 http:/ /github.com/gunicorn WIP: Deadline 2014/12 Enhance the user experience.
20. ? @benoitc Support me https:/ /gratipay.com/benoitc/