Testing in infrastructure

Talk presented at sysadmin days #9

Transcript

1. TESTING IN INFRASTRUCTURE
   

   View Slide

2. WHO AM I ?
   • Nico
   • 37 years old SRE, been using a keyboard for ~25 years
   • Working at Ledger
   • Loves automation and Infrastructure As Code
   

   View Slide

3. TESTING ? SERIOUSLY ?
   

   View Slide

4. OKAY. LET'S ROLL
   

   View Slide

5. LET'S GET SERIOUS
   • Testing comes from software/developers (first mention
   in 1979)
   • Many levels : Unit, Integration, End-to-end, ...
   • Many approaches : Smoke, Regression, Functional, ...
   

   View Slide

6. WHY DOES IT MATTER ?
   • We moved from "racking stuff" to "software
   engineers" (even of we still do rack servers)
   • We write code to do our job
   • We also write bugs, and they can have huge effects
   

   View Slide

7. WHAT CAN WE TEST ?
   • Fortunately, many things
   • Unfortunately, way less than we would like to
   • But this area is quite young in our particular field of
   application
   

   View Slide

8. CONFIGURATION MANAGEMENT
   • Chef : kitchen+inspec is the golden path
   • Puppet : rspec-puppet, packer+goss, kitchen-puppet
   

   View Slide

9. A BIT OF FEEDBACK ON CHEF
   • We focus our efforts on integration tests
   • All PRs are tested, using CircleCI
   • Our AWS base images are automatically built upon this
   code
   • We have to support multiple platforms (focus is done
   on Amazon Linux 2 and Ubuntu)
   

   View Slide

10. A BIT OF FEEDBACK ON CHEF
    

    View Slide

11. A BIT OF FEEDBACK ON CHEF
    

    View Slide

12. A BIT OF FEEDBACK ON CHEF
    • Allows us to iterate quite fast
    • We know that small changes won't blow up the whole
    stack
    • Avoid "refucktoring" and enables us to refactor code
    with confidence
    

    View Slide

13. A BIT OF FEEDBACK ON CHEF
    • Also works for environments, roles
    

    View Slide

14. INFRASTRUCTURE AS CODE
    • We will focus on terraform here. YMMV
    • How to know that the things we wrote to provision via
    code actually did ?
    Note : all things here are not real-production feedback, but experiments I've made (in an actual sandbox)
    

    View Slide

15. INFRASTRUCTURE AS CODE
    • Kitchen-terraform (yes, I like kitchen)
    • Quite limited to EC2 testing, but simple to use
    • Allows to test the "content" of the host, using inspec
    • Terratest
    • Many terraform providers supported (weee)
    • Need Golang knowledge, hence benefit its power
    

    View Slide

16. INFRASTRUCTURE AS CODE
    • Kubernetes users in the room ?
    • Ever got bitten by a bad YAML ?
    • Unexpected results in resource creation ?
    • Multiple approaches here
    • Simple rendering + yamllint : ~1h work
    • Or Terratest, again. Heavier investment, bigger outcome
    

    View Slide

17. INFRASTRUCTURE AS CODE
    • All these require a dedicated testing environment
    • This has a cost, so do outages and bugs
    • Testbed must be destroyed (use cloud-nuke)
    • Components need standalone testing, but also as a
    whole, complex, assembly
    

    View Slide

18. CAN IT BE DONE WITHOUT HYPE ?
    • Of course
    • Example : patterndb has integrated unit tests (seen at
    sysadmindays #8 with faxm0dem)
    • It can be used for single things that are "standalone"
    

    View Slide

19. CAN IT BE DONE WITHOUT HYPE ?
    Hold my beer
    

    View Slide

20. REMEMBER
    

    View Slide

21. QUESTIONS ?
    

    View Slide