Slide 1

Slide 1 text

Bruno Renié — EuroPython 2013 Deployability of Python web applications

Slide 2

Slide 2 text

Deployability, n The extent to which something is deployable

Slide 3

Slide 3 text

Disclaimer Most of this isn't python-specific or even web-specific Oriented at custom infrastructures Some things still apply if you're on PaaS

Slide 4

Slide 4 text

How easy it is to install, configure and operate your software?

Slide 5

Slide 5 text

Mostly about devs and ops working together

Slide 6

Slide 6 text

12factor.net

Slide 7

Slide 7 text

installation configuration operation

Slide 8

Slide 8 text

Installation

Slide 9

Slide 9 text

Installing postgres sudo apt-get install postgresql

Slide 10

Slide 10 text

Installing a python webapp sudo apt-get install build-essential python-virtualenv git clone https://[email protected]/corp/repo cd repo virtualenv env env/bin/pip install -r requirements.txt # Figure out PYTHONPATH

Slide 11

Slide 11 text

Installing a python webapp sudo apt-get install build-essential python-virtualenv git clone https://[email protected]/corp/repo cd repo virtualenv env env/bin/pip install -r requirements.txt # Figure out PYTHONPATH

Slide 12

Slide 12 text

Installing software is a solved problem Just use packaging Yep, python packaging

Slide 13

Slide 13 text

Why python packaging? Release process Dependency management Trivial rollbacks Easy system packaging

Slide 14

Slide 14 text

Packaging in 30 seconds # setup.py from distutils.core import setup from setuptools import find_packages with open('requirements.txt') as reqs: install_requires = reqs.read().split('\n') setup( name='project', version=__import__('project').__version__, packages=find_packages(), include_package_data=True, zip_safe=False, install_requires=install_requires, ) # MANIFEST.in include requirements.txt recursive-include project *

Slide 15

Slide 15 text

Private package hosting Local filesystem Network-based, ala pypi.python.org python setup.py sdist pip install --download dist -r requirements.txt rsync -avz -e ssh dist/ host.corp.com:/srv/pypi pip install --no-index --find-links=/srv/pypi myproject HTML directory index (apache / nginx / SimpleHTTPServer) pip install --no-index --find-links=http://host myproject

Slide 16

Slide 16 text

System packages fpm -s python -t deb setup.py awk -F= '{printf "fpm -s python -t deb -v %s %s\n", $3, $1}' \ requirements.txt | sh https://github.com/rcrowley/freight https://github.com/jordansissel/fpm Sign, upload to your private repository sudo apt-get install python-myproject sudo apt-get install python-myproject=1.2.3

Slide 17

Slide 17 text

Pin your dependencies Bad Good Django Django>=1.4,<1.5 Django==1.4.5 http://nvie.com/posts/pin-your-packages/ This is for end products, not libraries

Slide 18

Slide 18 text

Configuration

Slide 19

Slide 19 text

Configuring postgres $EDITOR /etc/postgresql/9.2/main/postgresql.conf service postgresql restart

Slide 20

Slide 20 text

Does your app have a config file? settings.py, production_settings.py are not config files Configuration != code

Slide 21

Slide 21 text

Problems with configuration as code Incompatible with packaging Environment-specific code Production-specific code will break production. Code shouldn't be tied to environments Code shouldn't be generated (salt / puppet / fabric)

Slide 22

Slide 22 text

Define your configuration What changes between environments? Database Secret key Host / port Credentials to external services (AWS, Sentry…) Read configuration from your code .ini files yaml environment variables …

Slide 23

Slide 23 text

Code changes ↓ release, deploy app Infrastructure changes ↓ write config, reload app

Slide 24

Slide 24 text

Config as environment variables Trivial to set with $PROCESS_MANAGER Native to every programming language De-facto standard (PaaS). Interoperability! Shared hosting Apache Pros Cons

Slide 25

Slide 25 text

Case study: Django settings Before After DATABASES = {'default': {'HOST': 'localhost', …}} settings_local.py DATABASES = {'default': {'HOST': 'prod', …}} settings_prod.py DATABASES = {'default': {'HOST': 'staging', …}} settings_staging.py DATABASES = {'default': dj_database_url.config()} settings.py DATABASE_URL="postgres://host:5432/db" env

Slide 26

Slide 26 text

Config patterns SECRET_KEY = os.environ['SECRET_KEY'] KeyError: 'SECRET_KEY' PORT = int(os.environ.get('PORT', 8000)) Sane defaults when possible Prevent the app from booting if something critical is missing Use *_URL and parsers to reduce the number of variables EMAIL_URL DATABASE_URL REDIS_URL

Slide 27

Slide 27 text

In development django-dotenv virtualenvwrapper postactivate hooks custom manage.py envdir …

Slide 28

Slide 28 text

Operation

Slide 29

Slide 29 text

WSGI Have a WSGI entry point gunicorn myapp.wsgi -b 0.0.0.0:$PORT

Slide 30

Slide 30 text

Stateless processes Persistence via external services Database Caching Storage …

Slide 31

Slide 31 text

Scale out with processes More traffic? Spawn more processes. Caveat: backend services rarely scale horizontally

Slide 32

Slide 32 text

Maximize dev/prod parity Same software Same people Same versions If you use postgres in production, use postgres in development PostgreSQL 9.1 and 9.2 do not perform equally Developers should know about infrastructure

Slide 33

Slide 33 text

Continuous integration/deployment CI != green badge on your github page CD != always running master in production Having shippable code Deploying it whenever your want CI CD tested packaged installable

Slide 34

Slide 34 text

Example workflow Commit Run tests Package Staging Production

Slide 35

Slide 35 text

Example workflow Staging Production Run tests Commit Package manual or automated Jenkins, SaltStack, IRC bots are your automation friends

Slide 36

Slide 36 text

use packaging to manage software clearly define the configuration contract automate as much as possible to minimize deployment friction

Slide 37

Slide 37 text

@brutasse [email protected] ?