The Peris of Writing a PaaS

Transcript

1. The Perils of Writing a PaaS Andrew Godwin http://www.flickr.com/photos/jannem/2719976702/

2. Hi, I'm Andrew. Serial Python developer Django core committer Sysadmin

   by night

3. We're ep.io Python Platform-as-a-Service Utility billing PostgreSQL, Redis, Celery, and

   more

4. We built a… prototype. Me and Ben Firshman Three or

   four days' hacking at DjangoCon Ran code, had simple deployment

5. The last 10%... A month or two of hibernation Went

   part-time in December Private beta since February Public launch later this year

6. Why? Why not?

7. Why? Why not? Lack of good solutions Strong, technical team

   Writing backend code is fun

8. It's a challenge We're still a closed beta 300+ apps,

   on 4 servers Some people just have crazy code Security, security, security

9. Our Architecture

10. ep.io Cloud Request Sugar XML Response Code Magic

11. Balancer Runner Runner Runner App 1 App 2 App 3

    App 2 App 4 App 1 Databases File Storage

12. Load Balancer Started with HaProxy Moved to custom Python loadbalancer

    Still needs refinement

13. Runners Daemon on each machine Nginx + gunicorn for each

    app instance Output captured, CPU time measured

14. Coordinator Analyses whole system Juggles apps between servers Detects dead

    servers

15. PostgreSQL Normal PostgreSQL 9 install Daemon to read query logs,

    make users

16. Redis Custom Redis loadbalancer/manager Starts processes on demand Handles multi-user

    security

17. Upload Receiver SSH endpoint for git, hg, commands Wraps VCSs,

    extracts uploaded files Creates filesystem images

18. Other Services Log aggregation UID assignment Calculate costs

19. Statistics Queued in Redis Consumed asynchronously Currently stored in Redis,

    changing soon Graphed and profiled

20. Configuration Management Puppet for the simpler stuff Daemons handle complex

    stuff Don't try to reinvent the wheel

21. Monitoring Nagios SaaS monitoring Nagios Emails, texts, pager Several custom

    checks

22. Backups Currently just rdiff-backup Moving to btrfs snapshots + DRBD

    HA is not a backup solution

23. Perils

24. Initial bad design (To be fair, it was a prototype)

25. Networks really aren't reliable (Well, EC2's, at least.)

26. Memory pressure is bad (Prepare to have a fallback. And

    another.)

27. Raw file handles are… fun. (As is the PTY subsystem.

    Be very careful.)

28. Write just enough automation (If a server dies, I now

    just go and get a drink)

29. HaProxy doesn't like 500+ backends (it's not exactly common)

30. Single redundancy is only so good (and remember, HA is

    not backups!)

31. Future Perils

32. Payment (Already underway, still hard)

33. Oversized Sites (we need to get a lot bigger first)

34. European Servers (people really do want them)

35. More Databases (how on earth do you measure MongoDB use?)

36. More Languages (easy to get it working, hard to polish)

37. The Potential Big Outage (quite useful as a motivational tool)

38. Thank you. Andrew Godwin @andrewgodwin andrew@ep.io