CFEngine on AWS: a Stateless Infrastructure

How Percolate uses CFEngine on AWS to build a resilient infrastructure, and leverage it to cut its operating cost using EC2 spot instances.

Transcript

1. Laurent Raufaste @_LR_ CFEngine on AWS: a Stateless Infrastructure
2. Hello Ops
3. I work at Percolate
4. Percolate helps brands create content at a social scale We are a tech company
5. We are a SaaS We live in the cloud
6. • 5% serving data • 10% doing chores • 85% working on data We use a bunch of servers
7. • ingest data • digest data • close to RT Those 85% do
8. We need to act smart to keep the business sustainable It’s expensive
9. CFEngine
10. A tool to gently "dictate" what your infrastructure should be #dontgetmadmarkburgess #WTF is #CFE ?
11. • 1993 CFEngine • 2003 Puppet • 2006 EC2 • 2008 CFEngine 3 • 2009 Chef Some history
12. Our Redis policy A simple example
13. Why CFEngine ? Chef, Puppet, Ansible
14. Convergence Keep promises if it can. No need to start from a known state.
15. Portability Same policies on Solaris, GNU/Linux, *BSD, AIX, HP-UX, Windows, OSX, …
16. CFEngine v1 released in 1993, as a “teddy bear”, it’s reassuring: it’s been used for this long without any big problem, cf. OpenBSD’s “2 holes since 1996” It’s old
17. Here come the deal breakers Let’s focus
18. The CFEngine DSL has been tailored for this purpose, no legacy, based on the promise theory Dedicated Language
19. Documented Infrastructure Solves the outdated and useless doc problem
20. • grep the whole cluster • what's in there is what's live • no need to SSH • knowledge is shared • history is kept • company is more valuable Documented Infrastructure
21. We want to build for success, not failure We hope what we build will succeed Scalability
22. • Decentralized by nature • Can scale both ways • Largest cluster is X00,000s • m1.small on AWS Scalability
23. It let us build things that last and can be reused Reusability
24. • DRY • Build service/servers blocks • Reuse them on live, staging, dev • Change them once for all Reusability
25. It’s tailored for the job Footprint
26. • Package to install is < 3MB • Largest binary is 320kB (96% C, 3% C++) • The server is just letting clients download policies • Clients are trying to apply the policies locally Footprint
27. It’s free (libre) and will ever be. It’s in Debian so it passed the DFSG test: Fastest way to check. It’s GPL
28. You can open bug reports and submit Pull Requests on Github, a must nowadays Open & active community
29. Here’s what CFEngine allows us to do
30. We don’t let it pwn us Pwn our infrastructure
31. Minimize redundancy and dependency Normalized Infrastructure
32. As the Netflix Chaos Monkey, I randomly kill instances Being unpredictable, it’s fun
33. 2011-2013: Employees x10, Clients x20, Servers x2, Infrastructure cost x1.2 Maintain costs
34. Don’t let exceptions waste your time Keep your infrastructure homogeneous
35. Ops should not slow things down Not scared of changes
36. Ops at Percolate
37. Ops are sysadmins that do their job well: Build+Automate+Maintain+Monitor+Document Ops are not DevOps
38. Ask your devs for the commands make them a policy Devs are not DevOps
39. Same infrastructure on all environments Live policies are used to build staging, smaller & fewer instances, and it’s always up to date
40. Same infrastructure on all environments It takes a few mins to get a small replica of live on your workstation, and it’s always up to date
41. • Develop in a branch • Test (Vagrant) • Review (Pull Request) • Merge • Deploy GitHub Flow applied to Ops
42. Ops use IaaS+Metal to provide a PaaS to devs Be the Heroku or the GAE of your team
43. Pieces we added around CFEngine It does not solve it all
44. CFEngine is missing the bootstrap process, is it really its job ? We did it in-house, in Python/Bash Bootstrapping
45. • Request an instance • Name it • Install CFEngine • CFEngine handles the rest Bootstrapping
46. We define all our servers in a INI file Bootstrapping
47. Everything can be overridden per instance type Bootstrapping
48. Easy to define, easy to launch Bootstrapping
49. 3 ordered dependencies max, e.g. “Hell” or deploy a Python app with on-demand pip requirements We don’t use CFEngine for complex stuff
50. • [id.][subrole.]role.environment • smtp.live.com • i-1ab345.worker.live.com • i-23f432.api.staging.com • lb.api.staging.com Naming convention to leverage CFEngine classes
51. • Our DNS is our inventory • We leverage it with a coordination service (AWS Tags (does not scale), Zookeeper, …) Naming convention to leverage CFEngine classes
52. • Application layer • CFE: Specialized layer (Role) • CFE: Basic layer (Environment) • Pristine Ubuntu • EC2 Server Structure
53. CFEngine does not take care of it all It takes care of all the basics
54. CFEngine does not take care of it all It makes sure the complex pieces are there and operational
55. syslog, smtp, ... you don’t want to fail big We started with the simple and obvious
56. When we reached the big stuff, it was easy, and we had all the bricks to reuse We finished with the critical
57. Achievements
58. • Documentation • Scalability • Reusability • Easy and fast to change Recap of previous benefits
59. But our huge win is ...
60. What’s the big deal ? Our infrastructure has no state
61. • Policies in git • App code in git • Data in datastores • No backup: Images are cache Our infrastructure has no state
62. 2 exceptions: S3 for cryptic generated config files (Jenkins) EBS for large non-vital changing data (RabbitMQ) No instance backup at all ?
63. No state is left on AWS (No AMI), we migrate away For better prices, stability, features, mood We are independent
64. But tell everyone to shut up (email). When something happens, you'll know. Your goal is silence: 0 email. We know and hear everything
65. It does not scale. We update the live version and every server updates itself. You can do this if your infrastructure is limpid, CFEnginized. We don’t push to deploy
66. Anything can go down, it will go up and rebuild itself automatically - It happens nightly. We are resilient
67. Upgrading a server takes 2 commands: 1. Launch a beefier instance with the same name 2. Kill the weak one We can change our shape
68. We can launch and kill any server anytime. It happens while we sleep. We use spot instances, it’s cheap!
69. For the smaller instance types
70. Some free tips We are almost there
71. It’s pretty dense, e.g. “The Promise of System Configuration” enlightened me Watch Mark’s videos
72. Don’t bother anything else, it will give you the “I understand” feeling we all love Buy Diego’s book
73. We are hiring JS, Mobile, Python, Data, Ops Work with CFEngine at Percolate
74. Laurent Raufaste @_LR_ Thank you