Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
The Peris of Writing a PaaS
Search
Andrew Godwin
May 10, 2011
Programming
0
110
The Peris of Writing a PaaS
A talk I gave at London Devops in May of 2011.
Andrew Godwin
May 10, 2011
Tweet
Share
More Decks by Andrew Godwin
See All by Andrew Godwin
Reconciling Everything
andrewgodwin
1
330
Django Through The Years
andrewgodwin
0
220
Writing Maintainable Software At Scale
andrewgodwin
0
450
A Newcomer's Guide To Airflow's Architecture
andrewgodwin
0
360
Async, Python, and the Future
andrewgodwin
2
680
How To Break Django: With Async
andrewgodwin
1
730
Taking Django's ORM Async
andrewgodwin
0
730
The Long Road To Asynchrony
andrewgodwin
0
660
The Scientist & The Engineer
andrewgodwin
1
780
Other Decks in Programming
See All in Programming
JSONataを使ってみよう Step Functionsが楽しくなる実践テクニック #devio2025
dafujii
1
590
Processing Gem ベースの、2D レトロゲームエンジンの開発
tokujiros
2
130
Swift Updates - Learn Languages 2025
koher
2
490
スケールする組織の実現に向けた インナーソース育成術 - ISGT2025
teamlab
PRO
1
120
私の後悔をAWS DMSで解決した話
hiramax
4
210
Android端末で実現するオンデバイスLLM 2025
masayukisuda
1
160
Introducing ReActionView: A new ActionView-compatible ERB Engine @ Rails World 2025, Amsterdam
marcoroth
0
700
旅行プランAIエージェント開発の裏側
ippo012
2
920
アセットのコンパイルについて
ojun9
0
130
Oracle Database Technology Night 92 Database Connection control FAN-AC
oracle4engineer
PRO
1
460
GitHubとGitLabとAWS CodePipelineでCI/CDを組み比べてみた
satoshi256kbyte
4
240
もうちょっといいRubyプロファイラを作りたい (2025)
osyoyu
1
450
Featured
See All Featured
Scaling GitHub
holman
463
140k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Facilitating Awesome Meetings
lara
55
6.5k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
8
920
Balancing Empowerment & Direction
lara
3
620
StorybookのUI Testing Handbookを読んだ
zakiyama
31
6.1k
Designing Experiences People Love
moore
142
24k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
113
20k
Writing Fast Ruby
sferik
628
62k
A designer walks into a library…
pauljervisheath
207
24k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
131
19k
Stop Working from a Prison Cell
hatefulcrawdad
271
21k
Transcript
The Perils of Writing a PaaS Andrew Godwin http://www.flickr.com/photos/jannem/2719976702/
Hi, I'm Andrew. Serial Python developer Django core committer Sysadmin
by night
We're ep.io Python Platform-as-a-Service Utility billing PostgreSQL, Redis, Celery, and
more
We built a… prototype. Me and Ben Firshman Three or
four days' hacking at DjangoCon Ran code, had simple deployment
The last 10%... A month or two of hibernation Went
part-time in December Private beta since February Public launch later this year
Why? Why not?
Why? Why not? Lack of good solutions Strong, technical team
Writing backend code is fun
It's a challenge We're still a closed beta 300+ apps,
on 4 servers Some people just have crazy code Security, security, security
Our Architecture
ep.io Cloud Request Sugar XML Response Code Magic
Balancer Runner Runner Runner App 1 App 2 App 3
App 2 App 4 App 1 Databases File Storage
Load Balancer Started with HaProxy Moved to custom Python loadbalancer
Still needs refinement
Runners Daemon on each machine Nginx + gunicorn for each
app instance Output captured, CPU time measured
Coordinator Analyses whole system Juggles apps between servers Detects dead
servers
PostgreSQL Normal PostgreSQL 9 install Daemon to read query logs,
make users
Redis Custom Redis loadbalancer/manager Starts processes on demand Handles multi-user
security
Upload Receiver SSH endpoint for git, hg, commands Wraps VCSs,
extracts uploaded files Creates filesystem images
Other Services Log aggregation UID assignment Calculate costs
Statistics Queued in Redis Consumed asynchronously Currently stored in Redis,
changing soon Graphed and profiled
Configuration Management Puppet for the simpler stuff Daemons handle complex
stuff Don't try to reinvent the wheel
Monitoring Nagios SaaS monitoring Nagios Emails, texts, pager Several custom
checks
Backups Currently just rdiff-backup Moving to btrfs snapshots + DRBD
HA is not a backup solution
Perils
Initial bad design (To be fair, it was a prototype)
Networks really aren't reliable (Well, EC2's, at least.)
Memory pressure is bad (Prepare to have a fallback. And
another.)
Raw file handles are… fun. (As is the PTY subsystem.
Be very careful.)
Write just enough automation (If a server dies, I now
just go and get a drink)
HaProxy doesn't like 500+ backends (it's not exactly common)
Single redundancy is only so good (and remember, HA is
not backups!)
Future Perils
Payment (Already underway, still hard)
Oversized Sites (we need to get a lot bigger first)
European Servers (people really do want them)
More Databases (how on earth do you measure MongoDB use?)
More Languages (easy to get it working, hard to polish)
The Potential Big Outage (quite useful as a motivational tool)
Thank you. Andrew Godwin @andrewgodwin
[email protected]