Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Deploying, at an Unusual Scale
Search
Andrew Godwin
October 22, 2011
Programming
7
580
Deploying, at an Unusual Scale
A talk I gave at DjangoCon Europe 2011, about Epio's internal architecture at that point.
Andrew Godwin
October 22, 2011
Tweet
Share
More Decks by Andrew Godwin
See All by Andrew Godwin
Reconciling Everything
andrewgodwin
1
310
Django Through The Years
andrewgodwin
0
200
Writing Maintainable Software At Scale
andrewgodwin
0
440
A Newcomer's Guide To Airflow's Architecture
andrewgodwin
0
350
Async, Python, and the Future
andrewgodwin
2
660
How To Break Django: With Async
andrewgodwin
1
730
Taking Django's ORM Async
andrewgodwin
0
720
The Long Road To Asynchrony
andrewgodwin
0
660
The Scientist & The Engineer
andrewgodwin
1
760
Other Decks in Programming
See All in Programming
dbt民主化とLLMによる開発ブースト ~ AI Readyな分析サイクルを目指して ~
yoshyum
3
1.1k
猫と暮らす Google Nest Cam生活🐈 / WebRTC with Google Nest Cam
yutailang0119
0
170
코딩 에이전트 체크리스트: Claude Code ver.
nacyot
0
930
可変変数との向き合い方 $$変数名が踊り出す$$ / php conference Variable variables
gunji
0
180
The Niche of CDK Grant オブジェクトって何者?/the-niche-of-cdk-what-isgrant-object
hassaku63
1
610
ご注文の差分はこちらですか? 〜 AWS CDK のいろいろな差分検出と安全なデプロイ
konokenj
3
580
Azure AI Foundryではじめてのマルチエージェントワークフロー
seosoft
0
200
#QiitaBash MCPのセキュリティ
ryosukedtomita
1
1.5k
顧客の画像データをテラバイト単位で配信する 画像サーバを WebP にした際に起こった課題と その対応策 ~継続的な取り組みを添えて~
takutakahashi
4
1.3k
イベントストーミング図からコードへの変換手順 / Procedure for Converting Event Storming Diagrams to Code
nrslib
2
1.1k
MCPを使ってイベントソーシングのAIコーディングを効率化する / Streamlining Event Sourcing AI Coding with MCP
tomohisa
0
170
オンコール⼊⾨〜ページャーが鳴る前に、あなたが備えられること〜 / Before The Pager Rings
yktakaha4
2
990
Featured
See All Featured
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
656
60k
Measuring & Analyzing Core Web Vitals
bluesmoon
7
520
StorybookのUI Testing Handbookを読んだ
zakiyama
30
5.9k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
Why You Should Never Use an ORM
jnunemaker
PRO
58
9.5k
Building Adaptive Systems
keathley
43
2.7k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
357
30k
Writing Fast Ruby
sferik
628
62k
Practical Orchestrator
shlominoach
189
11k
Done Done
chrislema
184
16k
Typedesign – Prime Four
hannesfritz
42
2.7k
4 Signs Your Business is Dying
shpigford
184
22k
Transcript
Deploying, At An Unusual Scale Andrew Godwin http://www.flickr.com/photos/whiskeytango/1431343034/ @andrewgodwin
Hi, I'm Andrew. Serial Python developer Django core committer Co-founder
of ep.io
Hi, I'm Andrew. Serial Python developer Django core committer Co-founder
of ep.io Occasional fast talker
""Andrew speaks English like a machine gun speaks bullets."" Reinout
van Rees
We're ep.io Python Platform-as-a-Service Easy deployment, easy upgrades PostgreSQL, Redis,
Celery, and more
Why am I here? Our Architecture How we deploy Django
How varied Django deployments are
Our Architecture
Balancer Runner Runner Runner App 1 App 2 App 3
App 2 App 4 App 1 Databases File Storage Balancer
Oh My God, It's Full of Pairs Everything is redundant
Distributed programming is Hard
Hardware Real colo'd machines Linode EC2 (pretty unreliable) (pretty reliable)
(pretty reliable) IPv6 (as much as we can)
ØMQ We used to use Redis Everything now on ZeroMQ
Eliminates SPOF* * Single Point Of Failure. What a pointless acronym.
ØMQ Usage Redundant location-resolvers (Nexus) REQ/XREP for control messages PUSH/PULL
for stats, logs PUB/SUB for heartbeats, locking
Runners Unsurprisingly, these run the code SquashFS filesystem images Virtualenvs
per app UID & permission isolation, more coming
Logging/Stats All done asynchronously using ØMQ Logs to filesystem (chunked
files) Stats to PostgreSQL database, for now
Loadbalancers Intercept all incoming HTTP requests Look up hostname (or
suffix) HTTP 1.1 compliant
Databases Shared (only for PostgreSQL) Dedicated (uses Runner framework) PostgreSQL
9, damnit
Django in the backend We use the ORM extensively Annoying
settings fiddling in __init__
www.ep.io Runs on ep.io, just like any other app* Provides
JSON API, web UI * Well not quite - App ID 0 is special - but we're working on it
WSGI It's a standard, right?
WSGI It's a standard, right? Well, yes, and it works
fine, but it's not enough for serving a Python app
Static Files CSS, images, JavaScript, etc. Needs a URL and
a directory path
Python & Dependencies Mostly filled by pip/buildout/etc packaging apparently allows
version spec
Deploying Django It makes things consistent, right?
Settings Layouts Vanilla settings.py local_settings.py configs/HOSTNAME.py Many others...
Python Paths Project-level imports App-level imports apps/ directories
Databases If it's SQL, it's PostgreSQL Redis for key-value, MongoDB
soon Some things assume a safe network
HA (High Availability) Not terribly easy with shared DBs PostgreSQL
9's sensible warm standby Redis has SLAVEOF Possibly use DRBD for general solution
Backups High Availability is NOT a backup btrfs for consistent
snapshotting Archived remote syncs No access to backups from servers
Migrations No solution yet for migration/code sync We're working on
it...
Web serving It's not like it's important or anything
gunicorn Small and lightweight Supports long-running requests Pretty stable
nginx Even more lightweight Extremely fast Really, really stable
The Load Balancer Used to be HAProxy Rewritten to custom
Python daemon eventlet used for high throughput Can't use nginx, no HTTP 1.1 for backends
Celery See: Yesterday's Talk Slightly tricky to run many We
use Redis as the backend
Management Commands First off, run as subprocess Then, a custom
PTY module Now, run as pty-wrapping subprocesses
Some General Advice If you're crazy enough to do this
Messaging's Not Enough Having a state to check is handy
Why run one, when you can run two for twice
the price? Redundancy is good. Double redundancy is better.
Always expect the worst Hope you never have to deal
with it.
The more backups, the better. Make sure you have historical
ones, too.
Django is very flexible Sometimes a little too flexible...
Your real problems will emerge later Don't over-optimise up front
for everything
Questions? Andrew Godwin
[email protected]
@andrewgodwin