Upgrade to PRO for Only $50/Yearโ€”Limited-Time Offer! ๐Ÿ”ฅ

Daemons, Deployment and Datacentres

Andrew Godwin
September 26, 2011

Daemons, Deployment andย Datacentres

A talk I gave at DjangoCon US 2011 about Epio's daemons and deployment systems, as well as Django deployment in general.

Andrew Godwin

September 26, 2011
Tweet

More Decks by Andrew Godwin

Other Decks in Technology

Transcript

  1. What's ep.io? ๎€Š Hosts Python sites/daemons ๎€Š Technically language-independent ๎€Š

    Supports multiple kinds of database ๎€Š Mainly hosted in the UK on our own hardware
  2. What I'll Cover ๎€Š Our architecture ๎€Š ZeroMQ and redundancy

    ๎€Š Eventlet everywhere ๎€Š The upload process ๎€Š The joy of networks ๎€Š General Challenges ๎€Š โ€The Stackโ€ ๎€Š Backups and replication ๎€Š Sensible architecture
  3. ZeroMQ ๎€Š Most importantly, not a message queue ๎€Š Advanced

    sockets, with multiple endpoints ๎€Š Has both deliver-to-single-consumer, and deliver-to-all-consumers. ๎€Š Uses TCP (or other things) as a transport.
  4. Redundancy ๎€Š Our internal rule is that there must be

    at least two of everything inside ep.io. ๎€Š Not quite true yet, but getting very close. ๎€Š Even our โ€find the servers running Xโ€ service is doubly redundant.
  5. Example # Make and connect the socket sock = ctx.socket(zmq.REQ)

    for endpoint in self.config.query_addresses(): sock.connect(endpoint) # Construct the message payload = json.dumps({"type": type, "extra": extra}) # Send the message with Timeout(30): sock.send(self.sign_message(payload)) # Recieve the answer return self.decode_message(sock.recv())
  6. Redundancy's Not Easy ๎€Š Several things can only run once

    (cronjobs) ๎€Š We currently have a best-effort distributed locking daemon to help with this
  7. What is Eventlet? ๎€Š Coroutine-based asynchronous concurrency ๎€Š Basically, lightweight

    threads with explicit context switching ๎€Š Reads quite like procedural code
  8. Highly Contrived Example import eventlet from eventlet.green import urllib2 urls

    = ['http://ep.io', 'http://t.co'] results = [] def fetch(url): results.append(urllib2.urlopen(url).read()) for url in urls: eventlet.spawn(fetch, url)
  9. Integration ๎€Š Most of our codebase uses Eventlet (~20,000 lines)

    ๎€Š Used for concurrency in daemons, workers, and batch processing ๎€Š ZeroMQ and Eventlet work together nicely
  10. Why? ๎€Š Far less race conditions than threading ๎€Š Multiprocessing

    can't handle ~2000 threads ๎€Š More readable code than callback-based systems
  11. Background ๎€Š Every time an app is uploaded to ep.io

    it gets a fresh app image to deploy into ๎€Š Each app image has its own virtualenv ๎€Š The typical ep.io app has around 3 or 4 dependencies ๎€Š Some have more than 40
  12. Parellised pip ๎€Š Installing 40 packages in serial takes quite

    a while ๎€Š Our custom pip version installs them in parallel, with caching ๎€Š Not 100% compatable with complex dependency sets yet
  13. Some Rough Numbers ๎€Š 15 requirements, some git, some pypi:

    ๎€Š Traditional: ~300 seconds ๎€Š Parellised, no cache: 30 seconds ๎€Š Parellised, cached: 2 seconds
  14. Compiled Modules ๎€Š ep.io app bundles are technically architecture- independent

    ๎€Š All compiled dependencies currently installed as system packages with dual 2.6/2.7 versions ๎€Š Will probably move to just bundling .so files too
  15. It's not just uploads ๎€Š Upload servers are general SSH

    endpoint ๎€Š Also do rsync, scp, command running ๎€Š Commands have semi-custom terminal emulation transported over ZeroMQ ๎€Š Hope you never have to use pty, ioctl or fcntl
  16. A Little Snippet old = termios.tcgetattr(fd) new = old[:] new[0]

    &= ~(termios.ISTRIP|termios.INLCR| termios.IGNCR|termios.ICRNL|termios.IXON| termios.IXANY|termios.IXOFF) new[2] &= ~(termios.OPOST) new[3] &= ~(termios.ECHO|termios.ISIG|termios.ICANON| termios.ECHOE|termios.ECHOK|termios.ECHONL| termios.IEXTEN) tcsetattr_flags = termios.TCSANOW if hasattr(termios, 'TCSASOFT'): tcsetattr_flags |= termios.TCSASOFT
  17. It's not just the slow ones ๎€Š Any network has

    a significant slowdown compared to local access ๎€Š Locking and concurrent access also an issue ๎€Š You can't run everything on one machine forever
  18. It's also the slow ones ๎€Š Transatlantic latency is around

    100ms ๎€Š Internal latency on EC2 can peak higher than 10s ๎€Š Routing blips can cause very short outages
  19. Heuristics and Optimism ๎€Š Sites and servers get a short

    grace period if they vanish in which to reappear ๎€Š Another site instance gets booted if needed โ€“ if the old one reappears, it gets killed ๎€Š Everything is designed to be run at least twice, so launching more things is not an issue
  20. Security ๎€Š We treat our internal network as public ๎€Š

    All messages signed/encrypted ๎€Š Firewalling of unnecessary ports ๎€Š Separate machines for higher-risk processes
  21. Today ๎€Š Nginx (static files/gzipping) ๎€Š Gunicorn (dynamic pages, unix

    socket best) ๎€Š PostgreSQL 9 ๎€Š Redis ๎€Š virtualenv
  22. Higher loads? ๎€Š Varnish for site caching ๎€Š HAProxy or

    Nginx for loadbalancing ๎€Š Give PostgreSQL more resources
  23. Development and Staging ๎€Š No need to run gunicorn/nginx locally

    ๎€Š PostgreSQL 9 still slightly annoying to install ๎€Š Redis is very easy to set up ๎€Š Staging should be EXACTLY the same as live
  24. Archives != High Availability ๎€Š Your PostgreSQL slave is not

    a backup ๎€Š We back up using multiple formats to diverse locations
  25. It's not just disasters ๎€Š Many other things other than

    theft and failure can lose data ๎€Š Don't back up to the same provider, they can cancel your account...
  26. Keep History ๎€Š You may not realise you need backups

    until the next month ๎€Š Take backups before any major change in database or code
  27. Check your backups restore ๎€Š Just seeing if they're there

    isn't good enough ๎€Š Try restoring your entire site onto a fresh box
  28. Replication is hard ๎€Š PostgreSQL and Redis replication both require

    your code to be modified a bit ๎€Š Django offers some help with database routers ๎€Š It's also not always necessary, and can cause bugs for your users.
  29. An Easy Start ๎€Š Dump your database nightly to a

    SQL file ๎€Š Use rdiff-backup (or similar) to sync that, codebase and uploads to a backup directory ๎€Š Also sync offsite โ€“ get a VPS with a different provider than your main one ๎€Š Make your backup server pull the backups, don't push them to it
  30. Ship long-running tasks off ๎€Š Use celery, or your own

    worker solution ๎€Š Even more critical if you have synchronous worker threads in your web app ๎€Š Email sending can be very slow
  31. Plan for multiple machines ๎€Š That means no SQLite ๎€Š

    Make good use of database transactions ๎€Š How are you going to store uploaded files?
  32. Loose Coupling ๎€Š Simple, loosely-connected components ๎€Š Easier to test

    and easier to debug ๎€Š Enforces some rough interface definitions
  33. Automation ๎€Š Use Puppet or Chef along with Fabric ๎€Š

    If you do something more than three times, automate it ๎€Š Every time you manually SSH in, a kitten gets extremely worried
  34. What happens with a full disk? ๎€Š Redis and MongoDB

    have historically both hated this situation, and lost data ๎€Š We had this with Redis โ€“ there was more than 10% disk free, but that wasn't enough to dump everything into.
  35. Stretching tools ๎€Š Our load balancer was initally HAProxy ๎€Š

    It really doesn't like having 3000 backends reloaded every 10 seconds ๎€Š Custom eventlet-based loadbalancer was simpler and slightly faster
  36. When Usernames Aren't There ๎€Š NFSv4 really, really hates UIDs

    with no corresponding username ๎€Š In fact, git does as well ๎€Š Variety of workarounds for different tools
  37. Even stable libraries have bugs ๎€Š Incompatability between psycopg2 and

    greenlets caused interpreter lockups ๎€Š Fixed in 2.4.2 ๎€Š Almost impossible to debug
  38. Awkward Penultimate Slide ๎€Š You don't have to be mad

    to write a distributed process management system, but it helps ๎€Š ZeroMQ is really, really nice. Really. ๎€Š Eventlet is a very useful concurrency tool ๎€Š Every developer should know a little ops ๎€Š Automation, consistency and preparation are key