Graham Dumpleton - Secrets of a WSGI master.

The WSGI (Web Server Gateway Interface) specification for hosting Python web applications was created in 2003. Measured in Internet time, it is ancient. The oldest main stream implementation of the WSGI specification is mod_wsgi, for the Apache HTTPD server and it is over 10 years old.

WSGI is starting to be regarded as not up to the job, with technologies such as HTTP/2, web sockets and async dispatching being the way forward. Reality is that WSGI will be around for quite some time yet and for the majority of use cases is more than adequate.

The real problem is not that we need to move to these new technologies, but that we aren't using the current WSGI servers to their best advantage. Moving to a new set of technologies will not necessarily make things better and will only create a new set of problems you have to solve.

As one of the oldest WSGI server implementations, Apache and mod\_wsgi may be regarded as boring and not cool, but it is still the most stable option for hosting WSGI applications available. It also hasn't been sitting still, with a considerable amount of development work being done on mod\_wsgi in the last few years to make it even more robust and easier to use in a development environment as well as production, including in containerised environments.

In this talk you will learn about many features of mod\_wsgi which you probably didn't even know existed, features which can help towards ensuring your Python web application deployment performs to its best, is secure, and has a low maintenance burden.

Topics which will be covered include:

* Easy deployment of Python web applications using mod\_wsgi-express.
* Integration of mod_wsgi-express with a Django web application.
* Using mod\_wsgi-express in a development environment.
* How to make use of mod\_wsgi-express in a production environment.
* Using mod_wsgi-express in a containerised runtime environment.
* Ensuring consistency between development and production environments using warpdrive.
* Using mod\_wsgi-express to bootstrap a system Apache installation for hosting WSGI applications.
* Why you should be using daemon mode of mod\_wsgi and not embedded mode.
* How to properly associate mod\_wsgi with a Python virtual environment.
* Building a robust deployment that can recover from misbehaving application code, backend services, or request overloading.
* Using hooks provided by mod\_wsgi to monitor the performance of your Python web application.

If you are a beginner, come learn why mod\_wsgi is still a good option for deploying your Python web applications. If you are an old time user of mod\_wsgi, find out about all the features you probably didn't know existed, revisit your current Python web application deployment and make it even better.

https://us.pycon.org/2018/schedule/presentation/89/

Transcript

1. Secrets of
 a WSGI master. Graham Dumpleton
 Graham.Dumpleton@gmail.com @GrahamDumpleton
2. What is WSGI? Web Browser Web Browser Web Browser Web Server HTTP HTTP HTTP File System (Static Files) Python Web Application WSGI WSGI == Web Server Gateway Interface (PEP 3333)
3. WSGI is a specification for an Application Programming Interface WSGI is NOT a wire protocol WSGI is NOT an implementation of any anything
4. def application(environ, start_response): status = '200 OK' output = b'Hello World!' response_headers = [ ('Content-Type', 'text/plain'), ('Content-Length', str(len(output)))] start_response(status, response_headers) return [output]
5. Friends don’t let friends use raw WSGI
7. You still need a way to host a WSGI application The development servers builtin to a framework are not good enough
8. Installing mod_wsgi the easy way pip install mod_wsgi https://pypi.python.org/pypi/mod_wsgi
9. Run mod_wsgi from the command line mod_wsgi-express start-server wsgi.py
10. No Apache configuration required $ mod_wsgi-express start-server wsgi.py Server URL : http://localhost:8000/ Server Root : /tmp/mod_wsgi-localhost:8000:502 Server Conf : /tmp/mod_wsgi-localhost:8000:502/httpd.conf Error Log File : /tmp/mod_wsgi-localhost:8000:502/error_log (warn) Request Capacity : 5 (1 process * 5 threads) Request Timeout : 60 (seconds) Startup Timeout : 15 (seconds) Queue Backlog : 100 (connections) Queue Timeout : 45 (seconds) Server Capacity : 20 (event/worker), 20 (prefork) Server Backlog : 500 (connections) Locale Setting : en_AU.UTF-8
11. Django framework integration INSTALLED_APPS = [ 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', 'mod_wsgi.server', ] Add mod_wsgi.server
12. Run with Django management command python manage.py runmodwsgi
13. Automatic code reloading python manage.py runmodwsgi --reload-on-changes
14. Interactive debugger python manage.py runmodwsgi --enable-debugger
15. Production configuration #1 python manage.py runmodwsgi \ --server-root /etc/wsgi-port-80 \ --user www-data --group www-data \ --port 80 --setup-only
16. Running in production /etc/wsgi-port-80/apachectl start /etc/wsgi-port-80/apachectl restart /etc/wsgi-port-80/apachectl stop
17. Production configuration #2 mod_wsgi-express start-server wsgi.py \ --server-root /etc/wsgi-port-80 \ --user www-data --group www-data \ --port 80 --url-alias /static static \ --setup-only
18. Build a container image FROM python:3 RUN apt-get update && \ apt-get install -y --no-install-recommends apache2 apache2-dev locales && \ apt-get clean && rm -r /var/lib/apt/lists/* RUN echo 'en_US.UTF-8 UTF-8' >> /etc/locale.gen && locale-gen ENV LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 RUN pip install --no-cache-dir mod_wsgi WORKDIR /opt/app-root COPY . /opt/app-root EXPOSE 80 CMD [ "mod_wsgi-express", "start-server", "--port", "80", "--user", \ "www-data", "--group", "www-data", "--log-to-terminal", "wsgi.py" ] Install Apache Fix Unicode Problems Run mod_wsgi-express
19. Building the container $ docker build -t mypyapp . Sending build context to Docker daemon 3.584kB Step 1/9 : FROM python:3 ---> 968120d8cbe8 Step 2/9 : WORKDIR /opt/app-root ---> Using cache ---> 003096a40d39 Step 3/9 : .......
20. Starting the container $ docker run --rm -p 80:80 mypyapp
21. Friends don’t let friends run containers as root
22. Friends don’t let friends use Python without a Python virtual environment
23. A better container image #1 FROM python:3 RUN apt-get update && \ apt-get install -y --no-install-recommends apache2 apache2-dev locales && \ apt-get clean && \ rm -r /var/lib/apt/lists/* RUN adduser --disabled-password --gecos "Warp Drive" --uid 1001 \ --gid 0 --home /opt/app-root warpdrive && \ chmod g+w /etc/passwd RUN echo 'en_US.UTF-8 UTF-8' >> /etc/locale.gen && locale-gen ENV LANG=en_US.UTF-8 \ LC_ALL=en_US.UTF-8 \ PATH=/opt/app-root/bin:$PATH \ HOME=/opt/app-root Create non root user
24. A better container image #2 RUN pip install --no-cache-dir virtualenv && \ virtualenv /opt/app-root && \ . /opt/app-root/bin/activate && \ pip install --no-cache-dir warpdrive && \ warpdrive fixup /opt/app-root WORKDIR /opt/app-root COPY . /opt/app-root/src RUN warpdrive fixup /opt/app-root/src USER 1001 RUN warpdrive build && \ warpdrive fixup /opt/app-root EXPOSE 8080 CMD [ "warpdrive", "start" ] Create a Python
 virtual environment Use warpdrive, it's magic Use non root user Non privileged port
25. Running as non root $ docker run mypyapp warpdrive exec id uid=1001(warpdrive) gid=0(root) groups=0(root) $ docker run -u 100001 mypyapp warpdrive exec id uid=100001(warpdrive) gid=0(root) groups=0(root) Default to assigned non root user Can be run as arbitrary high user ID
26. Same tools for development $ warpdrive project mypyapp (warpdrive+mypyapp) $ warpdrive build (warpdrive+mypyapp) $ warpdrive start https://pypi.python.org/pypi/warpdrive
27. Generate image with no Dockerfile $ warpdrive image mypyapp $ docker run --rm -p 80:8080 mypyapp
28. Source-to-Image $ s2i build https://github.com/GrahamDumpleton/warpdrive-django-modwsgi \ getwarped/warp0-debian8-python35 mypyapp https://github.com/openshift/source-to-image
29. OpenShift $ oc new-app --image-stream getwarped/warp0-debian8-python35 \ --code https://github.com/GrahamDumpleton/warpdrive-django-modwsgi \ --name mypyapp https://www.openshift.com
30. Manual Apache configuration $ mod_wsgi-express module-config LoadModule wsgi_module "/.../venv/.../mod_wsgi/server/mod_wsgi-py27.so" WSGIPythonHome "/.../venv"
31. Friends don’t let friends use embedded mode of mod_wsgi
32. Embedded mode
33. Daemon mode
34. Manual daemon mode configuration WSGIRestrictEmbedded On WSGIDaemonProcess mypyapp \ python-home=/.../env python-path=/.../src WSGIScriptAlias / /.../src/wsgi.py \ process-group=mypyapp application-group=%{GLOBAL} <Directory /.../src> <Files wsgi.py> Require all granted </Files> </Directory>
35. The missing options • lang='en_US.UTF-8' • locale='en_US.UTF-8' • display-name='%{GROUP}' • startup-timeout=15 • connect-timeout=15 • socket-timeout=60 • queue-timeout=45 • request-timeout=60 • inactivity-timeout=0 • restart-interval=0 • maximum-requests=0 • shutdown-timeout=5 • deadlock-timeout=60 • graceful-timeout=15 • eviction-timeout=0 http://modwsgi.readthedocs.io/en/develop/configuration-directives/WSGIDaemonProcess.html
36. Failed application loading • startup-timeout=15
 
 Defines the maximum number of seconds allowed to pass waiting to see if a WSGI script file can be loaded successfully by a daemon process. When the timeout is passed, the process will be restarted.
37. Connection timeouts • socket-timeout=60
 
 Defines the timeout on individual reads/writes on the socket connection between the Apache child processes and the mod_wsgi daemon processes. If not specified, the number of seconds specified by the Apache Timeout directive will be used instead. See also response-socket- timeout if need to control this only for writing back content of the response. • queue-timeout=45
 
 Defines the timeout on how long to wait for a mod_wsgi daemon process to accept a request for processing.
38. Long running or stuck requests • request-timeout=60
 
 Defines the maximum number of seconds that a request is allowed to run before the daemon process is restarted.
39. Request Monitoring import mod_wsgi def event_handler(name, **kwargs): if name == 'request_started': ... elif name == 'request_finished': environ = kwargs['request_environ'] application_time = kwargs.get('application_time') ... elif name == 'request_exception': ... mod_wsgi.subscribe_events(event_handler) http://blog.dscpl.com.au/2015/06/implementing-request-monitoring-within.html
40. Performance Monitoring
41. Live Traffic
42. Resources • mod_wsgi - http://www.modwsgi.org
 http://modwsgi.readthedocs.io
 https://pypi.org/project/mod_wsgi/ • warpdrive - http://warpdrive.readthedocs.io • Source-to-Image - https://github.com/openshift/source-to-image
43. Friends don't let friends use Windows for running Python web applications
44. Friends don't let friends use the mod_wsgi which comes packaged with the operating system
45. Friends don't let friends use those other WSGI servers
46. Friends don't let friends make things too complicated, simple is good
47. Graham.Dumpleton@gmail.com @GrahamDumpleton https://www.slideshare.net/GrahamDumpleton/secrets-of-a-wsgi-master