Testing in infrastructure

F37be7bf9f3a70e45167331b60892a08?s=47 nico
November 19, 2019

Testing in infrastructure

Talk presented at sysadmin days #9

F37be7bf9f3a70e45167331b60892a08?s=128

nico

November 19, 2019
Tweet

Transcript

  1. TESTING IN INFRASTRUCTURE

  2. WHO AM I ? • Nico • 37 years old

    SRE, been using a keyboard for ~25 years • Working at Ledger • Loves automation and Infrastructure As Code
  3. TESTING ? SERIOUSLY ?

  4. OKAY. LET'S ROLL

  5. LET'S GET SERIOUS • Testing comes from software/developers (first mention

    in 1979) • Many levels : Unit, Integration, End-to-end, ... • Many approaches : Smoke, Regression, Functional, ...
  6. WHY DOES IT MATTER ? • We moved from "racking

    stuff" to "software engineers" (even of we still do rack servers) • We write code to do our job • We also write bugs, and they can have huge effects
  7. WHAT CAN WE TEST ? • Fortunately, many things •

    Unfortunately, way less than we would like to • But this area is quite young in our particular field of application
  8. CONFIGURATION MANAGEMENT • Chef : kitchen+inspec is the golden path

    • Puppet : rspec-puppet, packer+goss, kitchen-puppet
  9. A BIT OF FEEDBACK ON CHEF • We focus our

    efforts on integration tests • All PRs are tested, using CircleCI • Our AWS base images are automatically built upon this code • We have to support multiple platforms (focus is done on Amazon Linux 2 and Ubuntu)
  10. A BIT OF FEEDBACK ON CHEF

  11. A BIT OF FEEDBACK ON CHEF

  12. A BIT OF FEEDBACK ON CHEF • Allows us to

    iterate quite fast • We know that small changes won't blow up the whole stack • Avoid "refucktoring" and enables us to refactor code with confidence
  13. A BIT OF FEEDBACK ON CHEF • Also works for

    environments, roles
  14. INFRASTRUCTURE AS CODE • We will focus on terraform here.

    YMMV • How to know that the things we wrote to provision via code actually did ? Note : all things here are not real-production feedback, but experiments I've made (in an actual sandbox)
  15. INFRASTRUCTURE AS CODE • Kitchen-terraform (yes, I like kitchen) •

    Quite limited to EC2 testing, but simple to use • Allows to test the "content" of the host, using inspec • Terratest • Many terraform providers supported (weee) • Need Golang knowledge, hence benefit its power
  16. INFRASTRUCTURE AS CODE • Kubernetes users in the room ?

    • Ever got bitten by a bad YAML ? • Unexpected results in resource creation ? • Multiple approaches here • Simple rendering + yamllint : ~1h work • Or Terratest, again. Heavier investment, bigger outcome
  17. INFRASTRUCTURE AS CODE • All these require a dedicated testing

    environment • This has a cost, so do outages and bugs • Testbed must be destroyed (use cloud-nuke) • Components need standalone testing, but also as a whole, complex, assembly
  18. CAN IT BE DONE WITHOUT HYPE ? • Of course

    • Example : patterndb has integrated unit tests (seen at sysadmindays #8 with faxm0dem) • It can be used for single things that are "standalone"
  19. CAN IT BE DONE WITHOUT HYPE ? Hold my beer

  20. REMEMBER

  21. QUESTIONS ?