Testing in infrastructure

Slide 1

Slide 1 text

TESTING IN INFRASTRUCTURE

Slide 2

Slide 2 text

WHO AM I ? • Nico • 37 years old SRE, been using a keyboard for ~25 years • Working at Ledger • Loves automation and Infrastructure As Code

Slide 3

Slide 3 text

TESTING ? SERIOUSLY ?

Slide 4

Slide 4 text

OKAY. LET'S ROLL

Slide 5

Slide 5 text

LET'S GET SERIOUS • Testing comes from software/developers (ﬁrst mention in 1979) • Many levels : Unit, Integration, End-to-end, ... • Many approaches : Smoke, Regression, Functional, ...

Slide 6

Slide 6 text

WHY DOES IT MATTER ? • We moved from "racking stuff" to "software engineers" (even of we still do rack servers) • We write code to do our job • We also write bugs, and they can have huge effects

Slide 7

Slide 7 text

WHAT CAN WE TEST ? • Fortunately, many things • Unfortunately, way less than we would like to • But this area is quite young in our particular ﬁeld of application

Slide 8

Slide 8 text

CONFIGURATION MANAGEMENT • Chef : kitchen+inspec is the golden path • Puppet : rspec-puppet, packer+goss, kitchen-puppet

Slide 9

Slide 9 text

A BIT OF FEEDBACK ON CHEF • We focus our efforts on integration tests • All PRs are tested, using CircleCI • Our AWS base images are automatically built upon this code • We have to support multiple platforms (focus is done on Amazon Linux 2 and Ubuntu)

Slide 10

Slide 10 text

A BIT OF FEEDBACK ON CHEF

Slide 11

Slide 11 text

A BIT OF FEEDBACK ON CHEF

Slide 12

Slide 12 text

A BIT OF FEEDBACK ON CHEF • Allows us to iterate quite fast • We know that small changes won't blow up the whole stack • Avoid "refucktoring" and enables us to refactor code with conﬁdence

Slide 13

Slide 13 text

A BIT OF FEEDBACK ON CHEF • Also works for environments, roles

Slide 14

Slide 14 text

INFRASTRUCTURE AS CODE • We will focus on terraform here. YMMV • How to know that the things we wrote to provision via code actually did ? Note : all things here are not real-production feedback, but experiments I've made (in an actual sandbox)

Slide 15

Slide 15 text

INFRASTRUCTURE AS CODE • Kitchen-terraform (yes, I like kitchen) • Quite limited to EC2 testing, but simple to use • Allows to test the "content" of the host, using inspec • Terratest • Many terraform providers supported (weee) • Need Golang knowledge, hence beneﬁt its power

Slide 16

Slide 16 text

INFRASTRUCTURE AS CODE • Kubernetes users in the room ? • Ever got bitten by a bad YAML ? • Unexpected results in resource creation ? • Multiple approaches here • Simple rendering + yamllint : ~1h work • Or Terratest, again. Heavier investment, bigger outcome

Slide 17

Slide 17 text

INFRASTRUCTURE AS CODE • All these require a dedicated testing environment • This has a cost, so do outages and bugs • Testbed must be destroyed (use cloud-nuke) • Components need standalone testing, but also as a whole, complex, assembly

Slide 18

Slide 18 text

CAN IT BE DONE WITHOUT HYPE ? • Of course • Example : patterndb has integrated unit tests (seen at sysadmindays #8 with faxm0dem) • It can be used for single things that are "standalone"