Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Testing in infrastructure

November 19, 2019

Testing in infrastructure

Talk presented at sysadmin days #9


November 19, 2019

Other Decks in Technology


  1. WHO AM I ? • Nico • 37 years old

    SRE, been using a keyboard for ~25 years • Working at Ledger • Loves automation and Infrastructure As Code
  2. LET'S GET SERIOUS • Testing comes from software/developers (first mention

    in 1979) • Many levels : Unit, Integration, End-to-end, ... • Many approaches : Smoke, Regression, Functional, ...
  3. WHY DOES IT MATTER ? • We moved from "racking

    stuff" to "software engineers" (even of we still do rack servers) • We write code to do our job • We also write bugs, and they can have huge effects
  4. WHAT CAN WE TEST ? • Fortunately, many things •

    Unfortunately, way less than we would like to • But this area is quite young in our particular field of application
  5. CONFIGURATION MANAGEMENT • Chef : kitchen+inspec is the golden path

    • Puppet : rspec-puppet, packer+goss, kitchen-puppet
  6. A BIT OF FEEDBACK ON CHEF • We focus our

    efforts on integration tests • All PRs are tested, using CircleCI • Our AWS base images are automatically built upon this code • We have to support multiple platforms (focus is done on Amazon Linux 2 and Ubuntu)
  7. A BIT OF FEEDBACK ON CHEF • Allows us to

    iterate quite fast • We know that small changes won't blow up the whole stack • Avoid "refucktoring" and enables us to refactor code with confidence
  8. INFRASTRUCTURE AS CODE • We will focus on terraform here.

    YMMV • How to know that the things we wrote to provision via code actually did ? Note : all things here are not real-production feedback, but experiments I've made (in an actual sandbox)
  9. INFRASTRUCTURE AS CODE • Kitchen-terraform (yes, I like kitchen) •

    Quite limited to EC2 testing, but simple to use • Allows to test the "content" of the host, using inspec • Terratest • Many terraform providers supported (weee) • Need Golang knowledge, hence benefit its power
  10. INFRASTRUCTURE AS CODE • Kubernetes users in the room ?

    • Ever got bitten by a bad YAML ? • Unexpected results in resource creation ? • Multiple approaches here • Simple rendering + yamllint : ~1h work • Or Terratest, again. Heavier investment, bigger outcome
  11. INFRASTRUCTURE AS CODE • All these require a dedicated testing

    environment • This has a cost, so do outages and bugs • Testbed must be destroyed (use cloud-nuke) • Components need standalone testing, but also as a whole, complex, assembly
  12. CAN IT BE DONE WITHOUT HYPE ? • Of course

    • Example : patterndb has integrated unit tests (seen at sysadmindays #8 with faxm0dem) • It can be used for single things that are "standalone"