Testing in infrastructure

Talk presented at sysadmin days #9

Transcript

1. TESTING IN INFRASTRUCTURE
2. WHO AM I ? • Nico • 37 years old SRE, been using a keyboard for ~25 years • Working at Ledger • Loves automation and Infrastructure As Code
3. TESTING ? SERIOUSLY ?
4. OKAY. LET'S ROLL
5. LET'S GET SERIOUS • Testing comes from software/developers (first mention in 1979) • Many levels : Unit, Integration, End-to-end, ... • Many approaches : Smoke, Regression, Functional, ...
6. WHY DOES IT MATTER ? • We moved from "racking stuff" to "software engineers" (even of we still do rack servers) • We write code to do our job • We also write bugs, and they can have huge effects
7. WHAT CAN WE TEST ? • Fortunately, many things • Unfortunately, way less than we would like to • But this area is quite young in our particular field of application
8. CONFIGURATION MANAGEMENT • Chef : kitchen+inspec is the golden path • Puppet : rspec-puppet, packer+goss, kitchen-puppet
9. A BIT OF FEEDBACK ON CHEF • We focus our efforts on integration tests • All PRs are tested, using CircleCI • Our AWS base images are automatically built upon this code • We have to support multiple platforms (focus is done on Amazon Linux 2 and Ubuntu)
10. A BIT OF FEEDBACK ON CHEF
11. A BIT OF FEEDBACK ON CHEF
12. A BIT OF FEEDBACK ON CHEF • Allows us to iterate quite fast • We know that small changes won't blow up the whole stack • Avoid "refucktoring" and enables us to refactor code with confidence
13. A BIT OF FEEDBACK ON CHEF • Also works for environments, roles
14. INFRASTRUCTURE AS CODE • We will focus on terraform here. YMMV • How to know that the things we wrote to provision via code actually did ? Note : all things here are not real-production feedback, but experiments I've made (in an actual sandbox)
15. INFRASTRUCTURE AS CODE • Kitchen-terraform (yes, I like kitchen) • Quite limited to EC2 testing, but simple to use • Allows to test the "content" of the host, using inspec • Terratest • Many terraform providers supported (weee) • Need Golang knowledge, hence benefit its power
16. INFRASTRUCTURE AS CODE • Kubernetes users in the room ? • Ever got bitten by a bad YAML ? • Unexpected results in resource creation ? • Multiple approaches here • Simple rendering + yamllint : ~1h work • Or Terratest, again. Heavier investment, bigger outcome
17. INFRASTRUCTURE AS CODE • All these require a dedicated testing environment • This has a cost, so do outages and bugs • Testbed must be destroyed (use cloud-nuke) • Components need standalone testing, but also as a whole, complex, assembly
18. CAN IT BE DONE WITHOUT HYPE ? • Of course • Example : patterndb has integrated unit tests (seen at sysadmindays #8 with faxm0dem) • It can be used for single things that are "standalone"
19. CAN IT BE DONE WITHOUT HYPE ? Hold my beer
20. REMEMBER
21. QUESTIONS ?