Deploying, at an Unusual Scale

Slide 1

Slide 1 text

Deploying, At An Unusual Scale Andrew Godwin http://www.flickr.com/photos/whiskeytango/1431343034/ @andrewgodwin

Slide 2

Slide 2 text

Hi, I'm Andrew. Serial Python developer Django core committer Co-founder of ep.io

Slide 3

Slide 3 text

Hi, I'm Andrew. Serial Python developer Django core committer Co-founder of ep.io Occasional fast talker

Slide 4

Slide 4 text

""Andrew speaks English like a machine gun speaks bullets."" Reinout van Rees

Slide 5

Slide 5 text

We're ep.io Python Platform-as-a-Service Easy deployment, easy upgrades PostgreSQL, Redis, Celery, and more

Slide 6

Slide 6 text

Why am I here? Our Architecture How we deploy Django How varied Django deployments are

Slide 7

Slide 7 text

Our Architecture

Slide 8

Slide 8 text

Balancer Runner Runner Runner App 1 App 2 App 3 App 2 App 4 App 1 Databases File Storage Balancer

Slide 9

Slide 9 text

Oh My God, It's Full of Pairs Everything is redundant Distributed programming is Hard

Slide 10

Slide 10 text

Hardware Real colo'd machines Linode EC2 (pretty unreliable) (pretty reliable) (pretty reliable) IPv6 (as much as we can)

Slide 11

Slide 11 text

ØMQ We used to use Redis Everything now on ZeroMQ Eliminates SPOF* * Single Point Of Failure. What a pointless acronym.

Slide 12

Slide 12 text

ØMQ Usage Redundant location-resolvers (Nexus) REQ/XREP for control messages PUSH/PULL for stats, logs PUB/SUB for heartbeats, locking

Slide 13

Slide 13 text

Runners Unsurprisingly, these run the code SquashFS filesystem images Virtualenvs per app UID & permission isolation, more coming

Slide 14

Slide 14 text

Logging/Stats All done asynchronously using ØMQ Logs to filesystem (chunked files) Stats to PostgreSQL database, for now

Slide 15

Slide 15 text

Loadbalancers Intercept all incoming HTTP requests Look up hostname (or suffix) HTTP 1.1 compliant

Slide 16

Slide 16 text

Databases Shared (only for PostgreSQL) Dedicated (uses Runner framework) PostgreSQL 9, damnit

Slide 17

Slide 17 text

Django in the backend We use the ORM extensively Annoying settings fiddling in __init__

Slide 18

Slide 18 text

www.ep.io Runs on ep.io, just like any other app* Provides JSON API, web UI * Well not quite - App ID 0 is special - but we're working on it

Slide 19

Slide 19 text

WSGI It's a standard, right?

Slide 20

Slide 20 text

WSGI It's a standard, right? Well, yes, and it works fine, but it's not enough for serving a Python app

Slide 21

Slide 21 text

Static Files CSS, images, JavaScript, etc. Needs a URL and a directory path

Slide 22

Slide 22 text

Python & Dependencies Mostly filled by pip/buildout/etc packaging apparently allows version spec

Slide 23

Slide 23 text

Deploying Django It makes things consistent, right?

Slide 24

Slide 24 text

Settings Layouts Vanilla settings.py local_settings.py configs/HOSTNAME.py Many others...

Slide 25

Slide 25 text

Python Paths Project-level imports App-level imports apps/ directories

Slide 26

Slide 26 text

Databases If it's SQL, it's PostgreSQL Redis for key-value, MongoDB soon Some things assume a safe network

Slide 27

Slide 27 text

HA (High Availability) Not terribly easy with shared DBs PostgreSQL 9's sensible warm standby Redis has SLAVEOF Possibly use DRBD for general solution

Slide 28

Slide 28 text

Backups High Availability is NOT a backup btrfs for consistent snapshotting Archived remote syncs No access to backups from servers

Slide 29

Slide 29 text

Migrations No solution yet for migration/code sync We're working on it...

Slide 30

Slide 30 text

Web serving It's not like it's important or anything

Slide 31

Slide 31 text

gunicorn Small and lightweight Supports long-running requests Pretty stable

Slide 32

Slide 32 text

nginx Even more lightweight Extremely fast Really, really stable

Slide 33

Slide 33 text

The Load Balancer Used to be HAProxy Rewritten to custom Python daemon eventlet used for high throughput Can't use nginx, no HTTP 1.1 for backends

Slide 34

Slide 34 text

Celery See: Yesterday's Talk Slightly tricky to run many We use Redis as the backend

Slide 35

Slide 35 text

Management Commands First off, run as subprocess Then, a custom PTY module Now, run as pty-wrapping subprocesses

Slide 36

Slide 36 text

Some General Advice If you're crazy enough to do this

Slide 37

Slide 37 text

Messaging's Not Enough Having a state to check is handy

Slide 38

Slide 38 text

Why run one, when you can run two for twice the price? Redundancy is good. Double redundancy is better.

Slide 39

Slide 39 text

Always expect the worst Hope you never have to deal with it.

Slide 40

Slide 40 text

The more backups, the better. Make sure you have historical ones, too.

Slide 41

Slide 41 text

Django is very flexible Sometimes a little too flexible...

Slide 42

Slide 42 text

Your real problems will emerge later Don't over-optimise up front for everything

Slide 43

Slide 43 text

Questions? Andrew Godwin [email protected] @andrewgodwin