Policy as Code

DevOpsDays Vancouver 2014
Video: http://youtu.be/QEfS0z_iPoo?t=34m1s

Most organizations choose to centralize resource and policy management to make managing it easier. That centralization brings with it some challenges: single points of failure, scaling, etc. Using DevOps practices like configuration management, we have the opportunity to try a different route.

In this talk, we'll discuss why PagerDuty chose to distribute our policy management. We'll look at how we skipped LDAP and distributed user management. We will cover our firewall automation which produces a perfect firewall tailored to each instance. We will profess our love for HAProxy which intelligently balances traffic between our ever changing service layers. We will discuss how we created a point to point IPSec mesh to secure our network traffic. We will discuss how this architecture is working so far and how we plan to improve the experience in the future.

After hearing this talk, I hope to have convinced attendees that implementing policy controls does not require centralized hardware. Treating the policy like code that can be replicated trivially produces a more stable system that is still just as easy to manage, and is architecturally superior.

Transcript

1. 11/16/14 Policy as code DEVOPSDAYS VANCOUVER 2014 @dougbarth
2. 11/16/14 POLICY AS CODE Dev Ops
3. 11/16/14 POLICY AS CODE DevOps Engineer
4. 11/16/14 POLICY AS CODE DevOps Engineer
5. 11/16/14 POLICY AS CODE How is babby PagerDuty formed?
6. 11/16/14 POLICY AS CODE
7. 11/16/14 Reliability is #1 concern POLICY AS CODE
8. 11/16/14 SPOFs POLICY AS CODE
9. 11/16/14 Failover POLICY AS CODE
10. 11/16/14 Scaling POLICY AS CODE
11. 11/16/14 Centralized in code Enforced across all infrastructure POLICY AS CODE
12. 11/16/14 Benefits Examples Tradeoffs POLICY AS CODE
13. 11/16/14 Audit trail for all changes POLICY AS CODE
14. 11/16/14 User management POLICY AS CODE
15. 11/16/14 LDAP POLICY AS CODE
16. 11/16/14 Chef data bags POLICY AS CODE
17. 11/16/14 { “id”: “doug”, “comment”: “Doug Barth”, “github_id”: “dougbarth”, “email”: “doug@pagerduty.com”, “groups”: [“ops”, “wheel”] } POLICY AS CODE
18. 11/16/14 How’s it working? POLICY AS CODE
19. 11/16/14 POLICY AS CODE
20. 11/16/14 POLICY AS CODE
21. 11/16/14 POLICY AS CODE
22. 11/16/14 Automated checks? POLICY AS CODE
23. 11/16/14 Time to convergence POLICY AS CODE
24. 11/16/14 Incremental rollout POLICY AS CODE
25. 11/16/14 Service discovery POLICY AS CODE
26. 11/16/14 HAProxy POLICY AS CODE
27. 11/16/14 POLICY AS CODE Client HAProxy Server Server Server Client HAProxy
28. 11/16/14 How’s it working? POLICY AS CODE
29. 11/16/14 Mostly ok POLICY AS CODE
30. 11/16/14 POLICY AS CODE
31. 11/16/14 POLICY AS CODE
32. 11/16/14 Time to convergence POLICY AS CODE
33. 11/16/14 Zookeeper? Consul? POLICY AS CODE
34. 11/16/14 All boxes enforce policy POLICY AS CODE
35. 11/16/14 Network encryption POLICY AS CODE
36. 11/16/14 POLICY AS CODE us-west-1 us-west-2 Linode
37. 11/16/14 IPSec Transport POLICY AS CODE
38. 11/16/14 POLICY AS CODE
39. 11/16/14 POLICY AS CODE spdadd 50.0.0.70 10.0.0.153 any -P out ipsec esp/transport//require; spdadd 10.0.0.153 50.0.0.70 any -P in ipsec esp/transport//require;
40. 11/16/14 POLICY AS CODE spdadd 10.0.0.121 10.0.0.153 any -P out ipsec esp/transport//require; spdadd 10.0.0.153 10.0.0.121 any -P in ipsec esp/transport//require;
41. 11/16/14 How’s it working? POLICY AS CODE
42. 11/16/14 POLICY AS CODE
43. 11/16/14 Linux & UDP POLICY AS CODE
44. 11/16/14 Time to convergence POLICY AS CODE
45. 11/16/14 No perimeter POLICY AS CODE
46. 11/16/14 Firewalls POLICY AS CODE
47. 11/16/14 POLICY AS CODE Firewall App DB Junk
48. 11/16/14 Define firewall chains by role POLICY AS CODE :app - -A app -s 10.0.0.1 -j ACCEPT -A app -s 50.0.0.1 -j ACCEPT
49. 11/16/14 Use those chains in firewall definitions POLICY AS CODE -A INPUT -p tcp --dport 3306 -j app -A INPUT -p tcp --dport 3306 -j slave
50. 11/16/14 POLICY AS CODE Firewall App DB Junk
51. 11/16/14 How’s it working? POLICY AS CODE
52. 11/16/14 Long chains POLICY AS CODE
53. 11/16/14 O(n) POLICY AS CODE
54. 11/16/14 ipset POLICY AS CODE
55. 11/16/14 O(1) POLICY AS CODE
56. 11/16/14 Time to convergence POLICY AS CODE
57. 11/16/14 POLICY AS CODE Developers welcome
58. 11/16/14 Developers welcome POLICY AS CODE
59. 11/16/14 Developers welcome POLICY AS CODE
60. 11/16/14 Developers welcome POLICY AS CODE
61. 11/16/14 Centralized in code Enforced across all infrastructure POLICY AS CODE
62. 11/16/14 POLICY AS CODE pagerduty.com/jobs SAN FRANCISCO AND TORONTO
63. 11/16/14 pagerduty.com/jobs Thank you.