Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
The Peris of Writing a PaaS
Search
Andrew Godwin
May 10, 2011
Programming
0
110
The Peris of Writing a PaaS
A talk I gave at London Devops in May of 2011.
Andrew Godwin
May 10, 2011
Tweet
Share
More Decks by Andrew Godwin
See All by Andrew Godwin
Reconciling Everything
andrewgodwin
1
330
Django Through The Years
andrewgodwin
0
220
Writing Maintainable Software At Scale
andrewgodwin
0
460
A Newcomer's Guide To Airflow's Architecture
andrewgodwin
0
370
Async, Python, and the Future
andrewgodwin
2
680
How To Break Django: With Async
andrewgodwin
1
740
Taking Django's ORM Async
andrewgodwin
0
740
The Long Road To Asynchrony
andrewgodwin
0
680
The Scientist & The Engineer
andrewgodwin
1
790
Other Decks in Programming
See All in Programming
コードとあなたと私の距離 / The Distance Between Code, You, and I
hiro_y
0
170
monorepo の Go テストをはやくした〜い!~最小の依存解決への道のり~ / faster-testing-of-monorepos
convto
2
500
エンジニアインターン「Treasure」とHonoの2年、そして未来へ / Our Journey with Hono Two Years at Treasure and Beyond
carta_engineering
0
230
CSC305 Lecture 04
javiergs
PRO
0
270
Introduce Hono CLI
yusukebe
5
2.1k
Server Side Kotlin Meetup vol.16: 内部動作を理解して ハイパフォーマンスなサーバサイド Kotlin アプリケーションを書こう
ternbusty
3
200
Android16 Migration Stories ~Building a Pattern for Android OS upgrades~
reoandroider
0
110
Cursorハンズオン実践!
eltociear
2
1.1k
The Past, Present, and Future of Enterprise Java
ivargrimstad
0
380
ALL CODE BASE ARE BELONG TO STUDY
uzulla
25
6.1k
技術的負債の正体を知って向き合う / Facing Technical Debt
irof
0
180
kiroとCodexで最高のSpec駆動開発を!!数時間で web3ネイティブなミニゲームを作ってみたよ!
mashharuki
0
570
Featured
See All Featured
How to Ace a Technical Interview
jacobian
280
24k
Balancing Empowerment & Direction
lara
4
690
Typedesign – Prime Four
hannesfritz
42
2.8k
The Cost Of JavaScript in 2023
addyosmani
55
9k
Code Review Best Practice
trishagee
72
19k
Statistics for Hackers
jakevdp
799
220k
Speed Design
sergeychernyshev
32
1.2k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
Navigating Team Friction
lara
190
15k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
20
1.2k
Rebuilding a faster, lazier Slack
samanthasiow
84
9.2k
Designing Experiences People Love
moore
142
24k
Transcript
The Perils of Writing a PaaS Andrew Godwin http://www.flickr.com/photos/jannem/2719976702/
Hi, I'm Andrew. Serial Python developer Django core committer Sysadmin
by night
We're ep.io Python Platform-as-a-Service Utility billing PostgreSQL, Redis, Celery, and
more
We built a… prototype. Me and Ben Firshman Three or
four days' hacking at DjangoCon Ran code, had simple deployment
The last 10%... A month or two of hibernation Went
part-time in December Private beta since February Public launch later this year
Why? Why not?
Why? Why not? Lack of good solutions Strong, technical team
Writing backend code is fun
It's a challenge We're still a closed beta 300+ apps,
on 4 servers Some people just have crazy code Security, security, security
Our Architecture
ep.io Cloud Request Sugar XML Response Code Magic
Balancer Runner Runner Runner App 1 App 2 App 3
App 2 App 4 App 1 Databases File Storage
Load Balancer Started with HaProxy Moved to custom Python loadbalancer
Still needs refinement
Runners Daemon on each machine Nginx + gunicorn for each
app instance Output captured, CPU time measured
Coordinator Analyses whole system Juggles apps between servers Detects dead
servers
PostgreSQL Normal PostgreSQL 9 install Daemon to read query logs,
make users
Redis Custom Redis loadbalancer/manager Starts processes on demand Handles multi-user
security
Upload Receiver SSH endpoint for git, hg, commands Wraps VCSs,
extracts uploaded files Creates filesystem images
Other Services Log aggregation UID assignment Calculate costs
Statistics Queued in Redis Consumed asynchronously Currently stored in Redis,
changing soon Graphed and profiled
Configuration Management Puppet for the simpler stuff Daemons handle complex
stuff Don't try to reinvent the wheel
Monitoring Nagios SaaS monitoring Nagios Emails, texts, pager Several custom
checks
Backups Currently just rdiff-backup Moving to btrfs snapshots + DRBD
HA is not a backup solution
Perils
Initial bad design (To be fair, it was a prototype)
Networks really aren't reliable (Well, EC2's, at least.)
Memory pressure is bad (Prepare to have a fallback. And
another.)
Raw file handles are… fun. (As is the PTY subsystem.
Be very careful.)
Write just enough automation (If a server dies, I now
just go and get a drink)
HaProxy doesn't like 500+ backends (it's not exactly common)
Single redundancy is only so good (and remember, HA is
not backups!)
Future Perils
Payment (Already underway, still hard)
Oversized Sites (we need to get a lot bigger first)
European Servers (people really do want them)
More Databases (how on earth do you measure MongoDB use?)
More Languages (easy to get it working, hard to polish)
The Potential Big Outage (quite useful as a motivational tool)
Thank you. Andrew Godwin @andrewgodwin
[email protected]