$30 off During Our Annual Pro Sale. View Details »

Testing in infrastructure

nico
November 19, 2019

Testing in infrastructure

Talk presented at sysadmin days #9

nico

November 19, 2019
Tweet

Other Decks in Technology

Transcript

  1. TESTING IN INFRASTRUCTURE

    View Slide

  2. WHO AM I ?
    • Nico
    • 37 years old SRE, been using a keyboard for ~25 years
    • Working at Ledger
    • Loves automation and Infrastructure As Code

    View Slide

  3. TESTING ? SERIOUSLY ?

    View Slide

  4. OKAY. LET'S ROLL

    View Slide

  5. LET'S GET SERIOUS
    • Testing comes from software/developers (first mention
    in 1979)
    • Many levels : Unit, Integration, End-to-end, ...
    • Many approaches : Smoke, Regression, Functional, ...

    View Slide

  6. WHY DOES IT MATTER ?
    • We moved from "racking stuff" to "software
    engineers" (even of we still do rack servers)
    • We write code to do our job
    • We also write bugs, and they can have huge effects

    View Slide

  7. WHAT CAN WE TEST ?
    • Fortunately, many things
    • Unfortunately, way less than we would like to
    • But this area is quite young in our particular field of
    application

    View Slide

  8. CONFIGURATION MANAGEMENT
    • Chef : kitchen+inspec is the golden path
    • Puppet : rspec-puppet, packer+goss, kitchen-puppet

    View Slide

  9. A BIT OF FEEDBACK ON CHEF
    • We focus our efforts on integration tests
    • All PRs are tested, using CircleCI
    • Our AWS base images are automatically built upon this
    code
    • We have to support multiple platforms (focus is done
    on Amazon Linux 2 and Ubuntu)

    View Slide

  10. A BIT OF FEEDBACK ON CHEF

    View Slide

  11. A BIT OF FEEDBACK ON CHEF

    View Slide

  12. A BIT OF FEEDBACK ON CHEF
    • Allows us to iterate quite fast
    • We know that small changes won't blow up the whole
    stack
    • Avoid "refucktoring" and enables us to refactor code
    with confidence

    View Slide

  13. A BIT OF FEEDBACK ON CHEF
    • Also works for environments, roles

    View Slide

  14. INFRASTRUCTURE AS CODE
    • We will focus on terraform here. YMMV
    • How to know that the things we wrote to provision via
    code actually did ?
    Note : all things here are not real-production feedback, but experiments I've made (in an actual sandbox)

    View Slide

  15. INFRASTRUCTURE AS CODE
    • Kitchen-terraform (yes, I like kitchen)
    • Quite limited to EC2 testing, but simple to use
    • Allows to test the "content" of the host, using inspec
    • Terratest
    • Many terraform providers supported (weee)
    • Need Golang knowledge, hence benefit its power

    View Slide

  16. INFRASTRUCTURE AS CODE
    • Kubernetes users in the room ?
    • Ever got bitten by a bad YAML ?
    • Unexpected results in resource creation ?
    • Multiple approaches here
    • Simple rendering + yamllint : ~1h work
    • Or Terratest, again. Heavier investment, bigger outcome

    View Slide

  17. INFRASTRUCTURE AS CODE
    • All these require a dedicated testing environment
    • This has a cost, so do outages and bugs
    • Testbed must be destroyed (use cloud-nuke)
    • Components need standalone testing, but also as a
    whole, complex, assembly

    View Slide

  18. CAN IT BE DONE WITHOUT HYPE ?
    • Of course
    • Example : patterndb has integrated unit tests (seen at
    sysadmindays #8 with faxm0dem)
    • It can be used for single things that are "standalone"

    View Slide

  19. CAN IT BE DONE WITHOUT HYPE ?
    Hold my beer

    View Slide

  20. REMEMBER

    View Slide

  21. QUESTIONS ?

    View Slide