Testing Infrastructure with Terratest Ladislav Prskavec Ladislav Prskavec - Teststack conference, 4. 6. 2020 1

Who Am I 4 Oracle Cloud Infrastructure - API Services team 4 Twitter: @abtris 4 Website: https://www.prskavec.net/ 4 Golang Prague organizer (@GoMeetupPrague) Ladislav Prskavec - Teststack conference, 4. 6. 2020 2

Infrastructure as code Ladislav Prskavec - Teststack conference, 4. 6. 2020 3

Ladislav Prskavec - Teststack conference, 4. 6. 2020 4

How to Test Infrastructure Code Ladislav Prskavec - Teststack conference, 4. 6. 2020 5

Ladislav Prskavec - Teststack conference, 4. 6. 2020 6

Yevgeniy Brikman 4 co-founder of Gruntwork with Josh Padnick 4 Gruntwork are authors of tools: Terragrunt and Terratest 4 Books Author - Terraform: Up & Running and Hello, Startup 4 Yevgeniy Brikman talk Automated Testing for Terraform, Docker, Packer, Kubernetes, and More, 11 Nov 2019 Ladislav Prskavec - Teststack conference, 4. 6. 2020 7

Test Pyramid 4 https://martinfowler.com/articles/practical-test-pyramid.html Ladislav Prskavec - Teststack conference, 4. 6. 2020 8

How we can test infrastructure code Ladislav Prskavec - Teststack conference, 4. 6. 2020 9

Static Analysis Ladislav Prskavec - Teststack conference, 4. 6. 2020 10

Syntax validation 4 terraform validate 4 kubernetes apply -f --dry-run -- validate=true Ladislav Prskavec - Teststack conference, 4. 6. 2020 11

Dry-run 4 terraform plan 4 kubectl apply -f --server-dry-run Ladislav Prskavec - Teststack conference, 4. 6. 2020 12

Linting & policies checks 4 hadolint 4 kube-score 4 conftest - Open Policy Agent Rego query language Ladislav Prskavec - Teststack conference, 4. 6. 2020 13

Unit Tests Ladislav Prskavec - Teststack conference, 4. 6. 2020 14

No pure unit testing for infrastructure Ladislav Prskavec - Teststack conference, 4. 6. 2020 15

Test strategy (per unit) 1. Deploy real infrastructure 2. Validate it works 3. Undeploy the infrastructure Ladislav Prskavec - Teststack conference, 4. 6. 2020 16

Example Hello World with docker Ladislav Prskavec - Teststack conference, 4. 6. 2020 17

Dockerfile FROM alpine RUN echo 'Hello, World!' > /test.txt Ladislav Prskavec - Teststack conference, 4. 6. 2020 18

Test package test import ( "testing" "github.com/gruntwork-io/terratest/modules/docker" "github.com/magiconair/properties/assert" ) func TestDockerHelloWorldExample(t *testing.T) { tag := "teststack/demo-terratest-docker" buildOptions := &docker.BuildOptions{ Tags: []string{tag}, } docker.Build(t, "../hello-world", buildOptions) opts := &docker.RunOptions{Command: []string{"cat", "/test.txt"}} output := docker.Run(t, tag, opts) assert.Equal(t, "Hello, World!", output) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 19

Local vs CI tests Ladislav Prskavec - Teststack conference, 4. 6. 2020 20

Running tests in CI Ladislav Prskavec - Teststack conference, 4. 6. 2020 21

Integration Tests Ladislav Prskavec - Teststack conference, 4. 6. 2020 22

Test strategy - integration test 1. Deploy all the infrastruture 2. Validate it works 3. Undeploy all the infrastruture Ladislav Prskavec - Teststack conference, 4. 6. 2020 23

Basic example OCI - APIGW and Fn Ladislav Prskavec - Teststack conference, 4. 6. 2020 24

const fdk = require('@fnproject/fdk'); fdk.handle(function(input) { let name = 'World'; if (input.name) { name = input.name; } return {message: 'Hello ' + name}; }); Ladislav Prskavec - Teststack conference, 4. 6. 2020 25

schema_version: 20180708 name: hello-world version: 0.0.2 runtime: node entrypoint: node func.js format: http-stream triggers: - name: hello-world-trigger type: http source: /hello-world-trigger Ladislav Prskavec - Teststack conference, 4. 6. 2020 26

package test import (...) func TestHelloWorldAppUnit(t *testing.T) { uniqueID := random.UniqueId() terraformOptions := &terraform.Options{ TerraformDir: "../", Vars: map[string]interface{}{ "name": fmt.Sprintf("hello-world-%s", uniqueID), "suffix": fmt.Sprintf("%s", uniqueID), "tenancy_ocid": os.Getenv("TF_VAR_tenancy_ocid"), "user_ocid": os.Getenv("TF_VAR_user_ocid"), "private_key_path": os.Getenv("TF_VAR_private_key_path"), "fingerprint": os.Getenv("TF_VAR_fingerprint"), "private_key_password": os.Getenv("TF_VAR_private_key_password"), "region": os.Getenv("TF_VAR_region"), }, } defer terraform.Destroy(t, terraformOptions) terraform.InitAndApply(t, terraformOptions) validateHelloWorldApp(t, terraformOptions) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 27

func validateHelloWorldApp(t *testing.T, terraformOptions *terraform.Options) { url := terraform.Output(t, terraformOptions, "gateway_url") expectedStatus := 200 expectedBody := `{"message":"Hello World"}` maxRetries := 10 timeBetweenRetries := 10 * time.Second http_helper.HttpGetWithRetry(t, url, nil, expectedStatus, expectedBody, maxRetries, timeBetweenRetries ) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 28

Ladislav Prskavec - Teststack conference, 4. 6. 2020 29

Cleanup after tests Ladislav Prskavec - Teststack conference, 4. 6. 2020 30

4 aws-nuke 4 cloud-nuke Ladislav Prskavec - Teststack conference, 4. 6. 2020 31

Validate Ladislav Prskavec - Teststack conference, 4. 6. 2020 32

Validate what make sense for you 4 test terraform output 4 http calls api/application 4 tcp/udp calls 4 ssh commands Ladislav Prskavec - Teststack conference, 4. 6. 2020 33

Test parallelism Ladislav Prskavec - Teststack conference, 4. 6. 2020 34

Parallelism 4 locking resources issue, 4 separations (compartments, accounts, regions) Ladislav Prskavec - Teststack conference, 4. 6. 2020 35

Running tests in parallel func TestKubernetesBasicExample(t *testing.T) { t.Parallel() ... Ladislav Prskavec - Teststack conference, 4. 6. 2020 36

Resource conflicts 4 randomize 4 namespace (k8s) / compartments (OCI) / accounts (AWS) import "github.com/gruntwork-io/terratest/modules/random" randomSuffix := random.UniqueId() // unique namespace namespaceName := fmt.Sprintf("test-%s", strings.ToLower(random.UniqueId())) Ladislav Prskavec - Teststack conference, 4. 6. 2020 37

Test stages Ladislav Prskavec - Teststack conference, 4. 6. 2020 38

import test_structure "github.com/gruntwork-io/terratest/modules/test-structure" func TestDockerComposeWithStagesLocal(t *testing.T) { workingDir := "../hello-world-docker-compose" test_structure.RunTestStage(t, "build_docker_image", func() { buildImage(t, "go-webapp", "../demowebapp") buildImage(t, "local/nginx", "../nginx") }) test_structure.RunTestStage(t, "run_docker_compose", func() { runCompose(t, workingDir) }) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 39

Why skip stages 4 skip cleanup on first run 4 skip deploy/build/cleanup on iterations Ladislav Prskavec - Teststack conference, 4. 6. 2020 40

How skip stages 4 use environment variable SKIP_stage_name=1 4 run with that: SKIP_build_docker_image=1 go test Ladislav Prskavec - Teststack conference, 4. 6. 2020 41

Why and how make retries 4 Exponential Backoff And Jitter 4 Simulator for AWS architecture blog (http://www.awsarchitectureblog.com/ ) about jitter and backoff. Ladislav Prskavec - Teststack conference, 4. 6. 2020 42

How make retries import http_helper "github.com/gruntwork-io/terratest/modules/http-helper" retries := 30 sleepBetweenRetries := 5*time.Second http_helper.HttpGetWithRetry(t, url, nil, 200, "Hello world!", retries, sleepBetweenRetries) Ladislav Prskavec - Teststack conference, 4. 6. 2020 43

How make retries with curl opts := &docker.RunOptions{ Command: []string{"--retry", "5", "--retry-connrefused", "-s", "http://production_nginx:80/hello"}, OtherOptions: []string{"--network", "testdockercomposewithstageslocal_teststack-network"}, } tag := "appropriate/curl" output := docker.Run(t, tag, opts) Ladislav Prskavec - Teststack conference, 4. 6. 2020 44

Example Hello World with K8S Ladislav Prskavec - Teststack conference, 4. 6. 2020 45

apiVersion: apps/v1 kind: Deployment metadata: name: hello-world-deployment spec: selector: matchLabels: app: hello-world replicas: 1 template: metadata: labels: app: hello-world spec: containers: - name: hello-world image: training/webapp:latest ports: - containerPort: 5000 Ladislav Prskavec - Teststack conference, 4. 6. 2020 46

kind: Service apiVersion: v1 metadata: name: hello-world-service spec: selector: app: hello-world ports: - protocol: TCP targetPort: 5000 port: 5000 type: LoadBalancer Ladislav Prskavec - Teststack conference, 4. 6. 2020 47

package test import (...) func TestKubernetesHelloWorldExample(t *testing.T) { kubeResourcePath := "../hello-world-kubernetes/deployment.yaml" // unique namespace namespaceName := fmt.Sprintf("test-%s", strings.ToLower(random.UniqueId())) options := k8s.NewKubectlOptions("", "", namespaceName) k8s.CreateNamespace(t, options, namespaceName) defer k8s.DeleteNamespace(t, options, namespaceName) defer k8s.KubectlDelete(t, options, kubeResourcePath) k8s.KubectlApply(t, options, kubeResourcePath) k8s.WaitUntilServiceAvailable(t, options, "hello-world-service", 10, 3*time.Second) service := k8s.GetService(t, options, "hello-world-service") url := fmt.Sprintf("http://%s", k8s.GetServiceEndpoint(t, options, service, 5000)) http_helper.HttpGetWithRetry(t, url, nil, 200, "Hello world!", 30, 5*time.Second) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 48

Ladislav Prskavec - Teststack conference, 4. 6. 2020 49

Example Hello World with docker-compose and stages Ladislav Prskavec - Teststack conference, 4. 6. 2020 50

1. Build docker images for nginx and webapp 2. Run docker-compose 3. Validate nginx and webapp 4. Destroy docker-compose Ladislav Prskavec - Teststack conference, 4. 6. 2020 51

version: "3" services: nginx: image: local/nginx:latest container_name: "production_nginx" networks: - "teststack-network" ports: - "80:80" go-webapp: image: go-webapp:latest container_name: "production_go-webapp" environment: - SERVER_TEXT=${SERVER_TEXT} expose: - "8080" networks: - "teststack-network" networks: "teststack-network": Ladislav Prskavec - Teststack conference, 4. 6. 2020 52

package main import (...) func helloHandler(res http.ResponseWriter, req *http.Request) { res.Header().Set( "Content-Type", "text/plain", ) io.WriteString( res, fmt.Sprintf("%s", os.Getenv("SERVER_TEXT")), ) } func defaultHandler(w http.ResponseWriter, r *http.Request) { fmt.Fprintf(w, "Go web app powered by Docker") } func main() { http.HandleFunc("/", defaultHandler) http.HandleFunc("/hello", helloHandler) err := http.ListenAndServe(":8080", nil) if err != nil { log.Fatal("ListenAndServe: ", err) return } } Ladislav Prskavec - Teststack conference, 4. 6. 2020 53

func TestDockerComposeLocal(t *testing.T) { tag := "go-webapp" buildargs := "-t local/nginx" buildOptions := &docker.BuildOptions{ Tags: []string{tag}, BuildArgs: []string{buildargs} } docker.Build(t, "../hello-world-docker-compose", buildOptions) tag := "local/nginx" buildargs := "-f Dockerfile.nginx" buildOptions := &docker.BuildOptions{ Tags: []string{tag}, BuildArgs: []string{buildargs} } docker.Build(t, "../hello-world-docker-compose", buildOptions) Ladislav Prskavec - Teststack conference, 4. 6. 2020 54

serverPort := 80 randomSuffix := random.UniqueId() expectedServerText := fmt.Sprintf("Hello, %s!", randomSuffix) dockerOptions := &docker.Options{ WorkingDir: "../hello-world-docker-compose", EnvVars: map[string]string{ "SERVER_TEXT": expectedServerText, "SERVER_PORT": strconv.Itoa(serverPort), "randomSuffix": randomSuffix, }, } defer docker.RunDockerCompose(t, dockerOptions, "down") docker.RunDockerCompose(t, dockerOptions, "up", "-d") maxRetries := 10 timeBetweenRetries := 5 * time.Second url := fmt.Sprintf("http://localhost:%d/hello", serverPort) tlsConfig := tls.Config{} http_helper.HttpGetWithRetry(t, url, &tlsConfig, 200, expectedServerText, maxRetries, timeBetweenRetries) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 55

Ladislav Prskavec - Teststack conference, 4. 6. 2020 56

func TestDockerComposeWithStagesLocal(t *testing.T) { workingDir := "../hello-world-docker-compose" test_structure.RunTestStage(t, "build_docker_image", func() { buildImage(t, "go-webapp", "../demowebapp") buildImage(t, "local/nginx", "../nginx") }) test_structure.RunTestStage(t, "run_docker_compose", func() { runCompose(t, workingDir) }) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 57

func buildImage(t *testing.T, tag string, workingDir string) { buildOptions := &docker.BuildOptions{ Tags: []string{tag}, } docker.Build(t, workingDir, buildOptions) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 58

func runCompose(t *testing.T, workingDir string) { serverPort := 80 randomSuffix := random.UniqueId() expectedServerText := fmt.Sprintf("Hello, %s!", randomSuffix) dockerOptions := &docker.Options{ WorkingDir: workingDir, EnvVars: map[string]string{ "SERVER_TEXT": expectedServerText, "randomSuffix": randomSuffix, "SERVER_PORT": strconv.Itoa(serverPort), }, } defer docker.RunDockerCompose(t, dockerOptions, "down") docker.RunDockerCompose(t, dockerOptions, "up", "-d") opts := &docker.RunOptions{ Command: []string{"--retry", "5", "--retry-connrefused", "-s", "http://production_nginx:80/hello"}, OtherOptions: []string{"--network", "testdockercomposewithstageslocal_teststack-network"}, } tag := "appropriate/curl" output := docker.Run(t, tag, opts) assert.Equal(t, expectedServerText, output) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 59

Ladislav Prskavec - Teststack conference, 4. 6. 2020 60

For iteration skip build stage Ladislav Prskavec - Teststack conference, 4. 6. 2020 61

Ladislav Prskavec - Teststack conference, 4. 6. 2020 62

E2E Tests Ladislav Prskavec - Teststack conference, 4. 6. 2020 63

Can we use this test strategy? 1. Deploy all the infrastruture 2. Validate it works 3. Undeploy all the infrastruture Ladislav Prskavec - Teststack conference, 4. 6. 2020 64

Test type Number of resources Chance of failure Unit test 10 1% Integration test 50 5% E2E test 500+ 40%+ For unit & integration you improve reliability with retries. Ladislav Prskavec - Teststack conference, 4. 6. 2020 65

Please don't write E2E tests in this way. Ladislav Prskavec - Teststack conference, 4. 6. 2020 66

Incremental testing Ladislav Prskavec - Teststack conference, 4. 6. 2020 67

Incremental testing 1. Deploy a persistent test environment and leave it running. 2. Each time you update a module, deploy & validate just that module. 3. Test your deployment process is zero-downtime too. Ladislav Prskavec - Teststack conference, 4. 6. 2020 68

Summary 4 Static Analysis (1-60 seconds) 4 Unit Tests (1-20 minutes) 4 Integration Tests (5-60 minutes) 4 E2E Tests (60-240+ minutes) Ladislav Prskavec - Teststack conference, 4. 6. 2020 69

What Good Worse Static Analysis Very Fast (until 1 min), Stable, Easy to use, No deploy Limited errors for catch, no reals checks with infrastructure Unit tests Fast (1-20 min), very good stability with retries, trust in components Need real deploy, non trivial test code, extra work for CI Integration tests Mostly stable with retries, High level confidence in multiple units Slow (5-60 min), Need real deploy, non trivial test code, extra work for CI E2E tests Build confidence in entire architecture Very slow (60-240+ min), Need real deploy, Still problem 0- deployments, Can be brittle (even with retries) Ladislav Prskavec - Teststack conference, 4. 6. 2020 70

Try make at least one test for your docker environment! Ladislav Prskavec - Teststack conference, 4. 6. 2020 71

Infrastructure code without tests is scary Ladislav Prskavec - Teststack conference, 4. 6. 2020 72

Q & A #test-infrastructure on Teststack Slack Ladislav Prskavec - Teststack conference, 4. 6. 2020 73