Deploying, at an Unusual Scale

Transcript

1. 1.

   Deploying, At An Unusual Scale Andrew Godwin http://www.flickr.com/photos/whiskeytango/1431343034/ @andrewgodwin

2. 2.

   Hi, I'm Andrew. Serial Python developer Django core committer Co-founder

   of ep.io
3. 3.

   Hi, I'm Andrew. Serial Python developer Django core committer Co-founder

   of ep.io Occasional fast talker
4. 4.

   ""Andrew speaks English like a machine gun speaks bullets."" Reinout

   van Rees
5. 5.

   We're ep.io Python Platform-as-a-Service Easy deployment, easy upgrades PostgreSQL, Redis,

   Celery, and more
6. 6.

   Why am I here? Our Architecture How we deploy Django

   How varied Django deployments are
7. 7.

   Our Architecture

8. 8.

   Balancer Runner Runner Runner App 1 App 2 App 3

   App 2 App 4 App 1 Databases File Storage Balancer
9. 9.

   Oh My God, It's Full of Pairs Everything is redundant

   Distributed programming is Hard
10. 10.

    Hardware Real colo'd machines Linode EC2 (pretty unreliable) (pretty reliable)

    (pretty reliable) IPv6 (as much as we can)
11. 11.

    ØMQ We used to use Redis Everything now on ZeroMQ

    Eliminates SPOF* * Single Point Of Failure. What a pointless acronym.
12. 12.

    ØMQ Usage Redundant location-resolvers (Nexus) REQ/XREP for control messages PUSH/PULL

    for stats, logs PUB/SUB for heartbeats, locking
13. 13.

    Runners Unsurprisingly, these run the code SquashFS filesystem images Virtualenvs

    per app UID & permission isolation, more coming
14. 14.

    Logging/Stats All done asynchronously using ØMQ Logs to filesystem (chunked

    files) Stats to PostgreSQL database, for now
15. 15.

    Loadbalancers Intercept all incoming HTTP requests Look up hostname (or

    suffix) HTTP 1.1 compliant
16. 16.

    Databases Shared (only for PostgreSQL) Dedicated (uses Runner framework) PostgreSQL

    9, damnit
17. 17.

    Django in the backend We use the ORM extensively Annoying

    settings fiddling in __init__
18. 18.

    www.ep.io Runs on ep.io, just like any other app* Provides

    JSON API, web UI * Well not quite - App ID 0 is special - but we're working on it
19. 19.

    WSGI It's a standard, right?

20. 20.

    WSGI It's a standard, right? Well, yes, and it works

    fine, but it's not enough for serving a Python app
21. 21.

    Static Files CSS, images, JavaScript, etc. Needs a URL and

    a directory path
22. 22.

    Python & Dependencies Mostly filled by pip/buildout/etc packaging apparently allows

    version spec
23. 23.

    Deploying Django It makes things consistent, right?

24. 24.

    Settings Layouts Vanilla settings.py local_settings.py configs/HOSTNAME.py Many others...

25. 25.

    Python Paths Project-level imports App-level imports apps/ directories

26. 26.

    Databases If it's SQL, it's PostgreSQL Redis for key-value, MongoDB

    soon Some things assume a safe network
27. 27.

    HA (High Availability) Not terribly easy with shared DBs PostgreSQL

    9's sensible warm standby Redis has SLAVEOF Possibly use DRBD for general solution
28. 28.

    Backups High Availability is NOT a backup btrfs for consistent

    snapshotting Archived remote syncs No access to backups from servers
29. 29.

    Migrations No solution yet for migration/code sync We're working on

    it...
30. 30.

    Web serving It's not like it's important or anything

31. 31.

    gunicorn Small and lightweight Supports long-running requests Pretty stable

32. 32.

    nginx Even more lightweight Extremely fast Really, really stable

33. 33.

    The Load Balancer Used to be HAProxy Rewritten to custom

    Python daemon eventlet used for high throughput Can't use nginx, no HTTP 1.1 for backends
34. 34.

    Celery See: Yesterday's Talk Slightly tricky to run many We

    use Redis as the backend
35. 35.

    Management Commands First off, run as subprocess Then, a custom

    PTY module Now, run as pty-wrapping subprocesses
36. 36.

    Some General Advice If you're crazy enough to do this

37. 37.

    Messaging's Not Enough Having a state to check is handy

38. 38.

    Why run one, when you can run two for twice

    the price? Redundancy is good. Double redundancy is better.
39. 39.

    Always expect the worst Hope you never have to deal

    with it.
40. 40.

    The more backups, the better. Make sure you have historical

    ones, too.
41. 41.

    Django is very flexible Sometimes a little too flexible...

42. 42.

    Your real problems will emerge later Don't over-optimise up front

    for everything
43. 43.

    Questions? Andrew Godwin andrew@ep.io @andrewgodwin