Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
The Peris of Writing a PaaS
Andrew Godwin
May 10, 2011
Programming
0
65
The Peris of Writing a PaaS
A talk I gave at London Devops in May of 2011.
Andrew Godwin
May 10, 2011
Tweet
Share
More Decks by Andrew Godwin
See All by Andrew Godwin
A Newcomer's Guide To Airflow's Architecture
andrewgodwin
0
88
Async, Python, and the Future
andrewgodwin
1
360
How To Break Django: With Async
andrewgodwin
1
290
Taking Django's ORM Async
andrewgodwin
0
340
The Long Road To Asynchrony
andrewgodwin
0
380
The Scientist & The Engineer
andrewgodwin
1
370
Pioneering Real-Time
andrewgodwin
0
150
Just Add Await: Retrofitting Async Into Django
andrewgodwin
2
1.1k
Terrain, Art, Python and LiDAR
andrewgodwin
1
230
Other Decks in Programming
See All in Programming
実録mruby組み込み体験
coe401_
0
100
You CANt teach an old dog new tricks
michaelbukachi
0
110
Enterprise Angular: Frontend Moduliths with Nx and Standalone Components @jax2022
manfredsteyer
PRO
0
290
Microsoft Teams の 会議アプリ開発のはじめかた / How to start Microsoft Teams app development
karamem0
0
1.5k
Git Rebase
bkuhlmann
7
1k
Where and how to run UI tests (Droidcon London, 2021)
nonews
0
210
Reinventing the wheel ... as a service
mariofusco
3
250
ebpfとWASMに思いを馳せる2022 / techfeed-conference-2022-ebpf-wasm-amsy810
masayaaoyama
0
570
heyにおけるSREの大切さ~マルチプロダクト運用の「楽しさ」と「難しさ」および今後の展望~
fufuhu
3
1.6k
Kotlin 最新動向2022 #tfcon #techfeed
ntaro
1
970
インフラエンジニアの多様性と評価、またはキャリアへのつなげ方 / Careers as infrastructure engineers
katsuhisa91
0
220
JGS594 Lecture 23
javiergs
PRO
0
400
Featured
See All Featured
Scaling GitHub
holman
451
140k
Side Projects
sachag
449
37k
The Invisible Customer
myddelton
110
11k
Happy Clients
brianwarren
89
5.5k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
3
430
Making Projects Easy
brettharned
98
4.3k
Six Lessons from altMBA
skipperchong
14
1.3k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
7
1k
How to name files
jennybc
39
58k
A Tale of Four Properties
chriscoyier
149
20k
Designing Experiences People Love
moore
130
22k
Large-scale JavaScript Application Architecture
addyosmani
499
110k
Transcript
The Perils of Writing a PaaS Andrew Godwin http://www.flickr.com/photos/jannem/2719976702/
Hi, I'm Andrew. Serial Python developer Django core committer Sysadmin
by night
We're ep.io Python Platform-as-a-Service Utility billing PostgreSQL, Redis, Celery, and
more
We built a… prototype. Me and Ben Firshman Three or
four days' hacking at DjangoCon Ran code, had simple deployment
The last 10%... A month or two of hibernation Went
part-time in December Private beta since February Public launch later this year
Why? Why not?
Why? Why not? Lack of good solutions Strong, technical team
Writing backend code is fun
It's a challenge We're still a closed beta 300+ apps,
on 4 servers Some people just have crazy code Security, security, security
Our Architecture
ep.io Cloud Request Sugar XML Response Code Magic
Balancer Runner Runner Runner App 1 App 2 App 3
App 2 App 4 App 1 Databases File Storage
Load Balancer Started with HaProxy Moved to custom Python loadbalancer
Still needs refinement
Runners Daemon on each machine Nginx + gunicorn for each
app instance Output captured, CPU time measured
Coordinator Analyses whole system Juggles apps between servers Detects dead
servers
PostgreSQL Normal PostgreSQL 9 install Daemon to read query logs,
make users
Redis Custom Redis loadbalancer/manager Starts processes on demand Handles multi-user
security
Upload Receiver SSH endpoint for git, hg, commands Wraps VCSs,
extracts uploaded files Creates filesystem images
Other Services Log aggregation UID assignment Calculate costs
Statistics Queued in Redis Consumed asynchronously Currently stored in Redis,
changing soon Graphed and profiled
Configuration Management Puppet for the simpler stuff Daemons handle complex
stuff Don't try to reinvent the wheel
Monitoring Nagios SaaS monitoring Nagios Emails, texts, pager Several custom
checks
Backups Currently just rdiff-backup Moving to btrfs snapshots + DRBD
HA is not a backup solution
Perils
Initial bad design (To be fair, it was a prototype)
Networks really aren't reliable (Well, EC2's, at least.)
Memory pressure is bad (Prepare to have a fallback. And
another.)
Raw file handles are… fun. (As is the PTY subsystem.
Be very careful.)
Write just enough automation (If a server dies, I now
just go and get a drink)
HaProxy doesn't like 500+ backends (it's not exactly common)
Single redundancy is only so good (and remember, HA is
not backups!)
Future Perils
Payment (Already underway, still hard)
Oversized Sites (we need to get a lot bigger first)
European Servers (people really do want them)
More Databases (how on earth do you measure MongoDB use?)
More Languages (easy to get it working, hard to polish)
The Potential Big Outage (quite useful as a motivational tool)
Thank you. Andrew Godwin @andrewgodwin andrew@ep.io