Fearless Ops with Stax

How I write cloudformation templates and do Ops without stress or fear.

Transcript

1. Fearless Ops with Stax Richard Lister
2. Motivation Too much Ops has: • Stress • Sleepless nights • Craziness • Fear! Can we do without them?
3. What are we afraid of? • Instability • Going slow • Mutability: changing things • Pets
4. Cattle vs Pets Don’t do this Do this
5. The Dreaded AWS Degradation Email
6. Pets Why are pets bad? • Live too long • Accumulate magic: config and data • Tempt us to make manual changes • We fear for their health • We don’t trust their replacements
7. Cattle Embracing cattle gives us the following superpowers: • Automation is the norm: autoscaling, deployment • Config must be done as code • Results in repeatability and trust • We can embrace failure and chaos engineering • Automation and code allow multiple copies • Multiple copies means we can move fast
8. Ops Code as First Class Citizen • Infrastructure must be change-controlled and repeatable • Operations source-code is in same git repo as application code • Every release is tracked as a single SHA in Github • Check out a SHA to get a fully self-contained ops+app setup • We use AWS Cloudformation templates to describe all resources
9. Replace prod/stg environments with branches
10. Release deployment
11. Copying the database around • Every stack has a complete database copy • Use SQS and workers to copy data between dynamodb tables • Migrations are performed at the same time as copy • Shoryuken workers for multi-threaded processing github.com/phstc/shoryuken
12. Sync database changes • Track changes since our bulk copy • DynamoDB streams to monitor these changes • New data is continuously migrated • Same migration logic as with bulk copy • No more migrations on release day!
13. Stax Stax is a simple ruby framework for defining and controlling an application using multiple, linked Cloudformation stacks. Makes it easy to create and manipulate a full application for every git branch. Plugins for manipulating specific AWS resources. Plugins for adding non-AWS resources. Pragmatic: Ruby Stack class is designed to be sub-classed and monkey-patched to handle hacks and edge-cases. github.com/rlister/stax
14. Staxfile stack :vpc stack :frontend, import: :vpc stack :waf, import: :frontend stack :build, import: :frontend
15. Create a branch
16. Create a stack
17. Stack commands
18. Stack resources
19. Stack outputs
20. More stacks
21. Plugins are implemented as ruby mixins module Stax class Frontend < Stack include Ecs, Alb end end
22. Using plugin sub-commands
23. Adding custom commands module Stax class Frontend < Stack ## create a new command desc 'dns', 'show DNS entry for this stack' def dns puts stack_output(:DnsRecord) end end end
24. Extending existing commands Monkey-patching is idiomatic in stax. module Stax class Frontend < Stack ## monkey-patch the subclass create method desc 'create', 'create stack' def create super dns end end end
25. Writing templates Stax can use Cloudformation templates written in: • cfer • json • yaml It would be trivial to add any other template generator that emits json or yaml suitable for the Cloudformation API.
26. cfer: templates in ruby Don’t do this Do this github.com/seanedwards/cfer
27. Stax generators Stax has rails-like generators to do the heavy-lifting of Cloudformation boilerplate. See github.com/rlister/stax-examples.
28. Stax extensions • stax-examples: example generators for common uses github.com/rlister/stax-examples • stax-nag: linting for templates github.com/rlister/stax-nag • stax-stacksets: WIP cloudformation stacksets github.com/rlister/stax-stacksets • stax-datadog: Datadog dashboard creation with stax github.com/rlister/stax-datadog
29. Principles • Keep it simple • Immutable: nothing is ever changed once deployed • Automation: infrastructure, config, and deployment is all automated • Embrace failure • Pragmatism: there are always edge-cases, plan for them