Testing Infrastructure with Terratest

Testing Infrastructure with Terratest

How to Test Infrastructure Code using Terraform, Terratest from Gruntwork.io

55d57afc217d360cd3aad3e2a8d4e5a0?s=128

Ladislav Prskavec

April 06, 2020
Tweet

Transcript

  1. Testing Infrastructure with Terratest Ladislav Prskavec Ladislav Prskavec - Teststack

    conference, 4. 6. 2020 1
  2. Who Am I 4 Oracle Cloud Infrastructure - API Services

    team 4 Twitter: @abtris 4 Website: https://www.prskavec.net/ 4 Golang Prague organizer (@GoMeetupPrague) Ladislav Prskavec - Teststack conference, 4. 6. 2020 2
  3. Infrastructure as code Ladislav Prskavec - Teststack conference, 4. 6.

    2020 3
  4. Ladislav Prskavec - Teststack conference, 4. 6. 2020 4

  5. How to Test Infrastructure Code Ladislav Prskavec - Teststack conference,

    4. 6. 2020 5
  6. Ladislav Prskavec - Teststack conference, 4. 6. 2020 6

  7. Yevgeniy Brikman 4 co-founder of Gruntwork with Josh Padnick 4

    Gruntwork are authors of tools: Terragrunt and Terratest 4 Books Author - Terraform: Up & Running and Hello, Startup 4 Yevgeniy Brikman talk Automated Testing for Terraform, Docker, Packer, Kubernetes, and More, 11 Nov 2019 Ladislav Prskavec - Teststack conference, 4. 6. 2020 7
  8. Test Pyramid 4 https://martinfowler.com/articles/practical-test-pyramid.html Ladislav Prskavec - Teststack conference, 4.

    6. 2020 8
  9. How we can test infrastructure code Ladislav Prskavec - Teststack

    conference, 4. 6. 2020 9
  10. Static Analysis Ladislav Prskavec - Teststack conference, 4. 6. 2020

    10
  11. Syntax validation 4 terraform validate 4 kubernetes apply -f <file>

    --dry-run -- validate=true Ladislav Prskavec - Teststack conference, 4. 6. 2020 11
  12. Dry-run 4 terraform plan 4 kubectl apply -f <file> --server-dry-run

    Ladislav Prskavec - Teststack conference, 4. 6. 2020 12
  13. Linting & policies checks 4 hadolint 4 kube-score 4 conftest

    - Open Policy Agent Rego query language Ladislav Prskavec - Teststack conference, 4. 6. 2020 13
  14. Unit Tests Ladislav Prskavec - Teststack conference, 4. 6. 2020

    14
  15. No pure unit testing for infrastructure Ladislav Prskavec - Teststack

    conference, 4. 6. 2020 15
  16. Test strategy (per unit) 1. Deploy real infrastructure 2. Validate

    it works 3. Undeploy the infrastructure Ladislav Prskavec - Teststack conference, 4. 6. 2020 16
  17. Example Hello World with docker Ladislav Prskavec - Teststack conference,

    4. 6. 2020 17
  18. Dockerfile FROM alpine RUN echo 'Hello, World!' > /test.txt Ladislav

    Prskavec - Teststack conference, 4. 6. 2020 18
  19. Test package test import ( "testing" "github.com/gruntwork-io/terratest/modules/docker" "github.com/magiconair/properties/assert" ) func

    TestDockerHelloWorldExample(t *testing.T) { tag := "teststack/demo-terratest-docker" buildOptions := &docker.BuildOptions{ Tags: []string{tag}, } docker.Build(t, "../hello-world", buildOptions) opts := &docker.RunOptions{Command: []string{"cat", "/test.txt"}} output := docker.Run(t, tag, opts) assert.Equal(t, "Hello, World!", output) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 19
  20. Local vs CI tests Ladislav Prskavec - Teststack conference, 4.

    6. 2020 20
  21. Running tests in CI Ladislav Prskavec - Teststack conference, 4.

    6. 2020 21
  22. Integration Tests Ladislav Prskavec - Teststack conference, 4. 6. 2020

    22
  23. Test strategy - integration test 1. Deploy all the infrastruture

    2. Validate it works 3. Undeploy all the infrastruture Ladislav Prskavec - Teststack conference, 4. 6. 2020 23
  24. Basic example OCI - APIGW and Fn Ladislav Prskavec -

    Teststack conference, 4. 6. 2020 24
  25. const fdk = require('@fnproject/fdk'); fdk.handle(function(input) { let name = 'World';

    if (input.name) { name = input.name; } return {message: 'Hello ' + name}; }); Ladislav Prskavec - Teststack conference, 4. 6. 2020 25
  26. schema_version: 20180708 name: hello-world version: 0.0.2 runtime: node entrypoint: node

    func.js format: http-stream triggers: - name: hello-world-trigger type: http source: /hello-world-trigger Ladislav Prskavec - Teststack conference, 4. 6. 2020 26
  27. package test import (...) func TestHelloWorldAppUnit(t *testing.T) { uniqueID :=

    random.UniqueId() terraformOptions := &terraform.Options{ TerraformDir: "../", Vars: map[string]interface{}{ "name": fmt.Sprintf("hello-world-%s", uniqueID), "suffix": fmt.Sprintf("%s", uniqueID), "tenancy_ocid": os.Getenv("TF_VAR_tenancy_ocid"), "user_ocid": os.Getenv("TF_VAR_user_ocid"), "private_key_path": os.Getenv("TF_VAR_private_key_path"), "fingerprint": os.Getenv("TF_VAR_fingerprint"), "private_key_password": os.Getenv("TF_VAR_private_key_password"), "region": os.Getenv("TF_VAR_region"), }, } defer terraform.Destroy(t, terraformOptions) terraform.InitAndApply(t, terraformOptions) validateHelloWorldApp(t, terraformOptions) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 27
  28. func validateHelloWorldApp(t *testing.T, terraformOptions *terraform.Options) { url := terraform.Output(t, terraformOptions,

    "gateway_url") expectedStatus := 200 expectedBody := `{"message":"Hello World"}` maxRetries := 10 timeBetweenRetries := 10 * time.Second http_helper.HttpGetWithRetry(t, url, nil, expectedStatus, expectedBody, maxRetries, timeBetweenRetries ) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 28
  29. Ladislav Prskavec - Teststack conference, 4. 6. 2020 29

  30. Cleanup after tests Ladislav Prskavec - Teststack conference, 4. 6.

    2020 30
  31. 4 aws-nuke 4 cloud-nuke Ladislav Prskavec - Teststack conference, 4.

    6. 2020 31
  32. Validate Ladislav Prskavec - Teststack conference, 4. 6. 2020 32

  33. Validate what make sense for you 4 test terraform output

    4 http calls api/application 4 tcp/udp calls 4 ssh commands Ladislav Prskavec - Teststack conference, 4. 6. 2020 33
  34. Test parallelism Ladislav Prskavec - Teststack conference, 4. 6. 2020

    34
  35. Parallelism 4 locking resources issue, 4 separations (compartments, accounts, regions)

    Ladislav Prskavec - Teststack conference, 4. 6. 2020 35
  36. Running tests in parallel func TestKubernetesBasicExample(t *testing.T) { t.Parallel() ...

    Ladislav Prskavec - Teststack conference, 4. 6. 2020 36
  37. Resource conflicts 4 randomize 4 namespace (k8s) / compartments (OCI)

    / accounts (AWS) import "github.com/gruntwork-io/terratest/modules/random" randomSuffix := random.UniqueId() // unique namespace namespaceName := fmt.Sprintf("test-%s", strings.ToLower(random.UniqueId())) Ladislav Prskavec - Teststack conference, 4. 6. 2020 37
  38. Test stages Ladislav Prskavec - Teststack conference, 4. 6. 2020

    38
  39. import test_structure "github.com/gruntwork-io/terratest/modules/test-structure" func TestDockerComposeWithStagesLocal(t *testing.T) { workingDir := "../hello-world-docker-compose"

    test_structure.RunTestStage(t, "build_docker_image", func() { buildImage(t, "go-webapp", "../demowebapp") buildImage(t, "local/nginx", "../nginx") }) test_structure.RunTestStage(t, "run_docker_compose", func() { runCompose(t, workingDir) }) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 39
  40. Why skip stages 4 skip cleanup on first run 4

    skip deploy/build/cleanup on iterations Ladislav Prskavec - Teststack conference, 4. 6. 2020 40
  41. How skip stages 4 use environment variable SKIP_stage_name=1 4 run

    with that: SKIP_build_docker_image=1 go test Ladislav Prskavec - Teststack conference, 4. 6. 2020 41
  42. Why and how make retries 4 Exponential Backoff And Jitter

    4 Simulator for AWS architecture blog (http://www.awsarchitectureblog.com/ ) about jitter and backoff. Ladislav Prskavec - Teststack conference, 4. 6. 2020 42
  43. How make retries import http_helper "github.com/gruntwork-io/terratest/modules/http-helper" retries := 30 sleepBetweenRetries

    := 5*time.Second http_helper.HttpGetWithRetry(t, url, nil, 200, "Hello world!", retries, sleepBetweenRetries) Ladislav Prskavec - Teststack conference, 4. 6. 2020 43
  44. How make retries with curl opts := &docker.RunOptions{ Command: []string{"--retry",

    "5", "--retry-connrefused", "-s", "http://production_nginx:80/hello"}, OtherOptions: []string{"--network", "testdockercomposewithstageslocal_teststack-network"}, } tag := "appropriate/curl" output := docker.Run(t, tag, opts) Ladislav Prskavec - Teststack conference, 4. 6. 2020 44
  45. Example Hello World with K8S Ladislav Prskavec - Teststack conference,

    4. 6. 2020 45
  46. apiVersion: apps/v1 kind: Deployment metadata: name: hello-world-deployment spec: selector: matchLabels:

    app: hello-world replicas: 1 template: metadata: labels: app: hello-world spec: containers: - name: hello-world image: training/webapp:latest ports: - containerPort: 5000 Ladislav Prskavec - Teststack conference, 4. 6. 2020 46
  47. kind: Service apiVersion: v1 metadata: name: hello-world-service spec: selector: app:

    hello-world ports: - protocol: TCP targetPort: 5000 port: 5000 type: LoadBalancer Ladislav Prskavec - Teststack conference, 4. 6. 2020 47
  48. package test import (...) func TestKubernetesHelloWorldExample(t *testing.T) { kubeResourcePath :=

    "../hello-world-kubernetes/deployment.yaml" // unique namespace namespaceName := fmt.Sprintf("test-%s", strings.ToLower(random.UniqueId())) options := k8s.NewKubectlOptions("", "", namespaceName) k8s.CreateNamespace(t, options, namespaceName) defer k8s.DeleteNamespace(t, options, namespaceName) defer k8s.KubectlDelete(t, options, kubeResourcePath) k8s.KubectlApply(t, options, kubeResourcePath) k8s.WaitUntilServiceAvailable(t, options, "hello-world-service", 10, 3*time.Second) service := k8s.GetService(t, options, "hello-world-service") url := fmt.Sprintf("http://%s", k8s.GetServiceEndpoint(t, options, service, 5000)) http_helper.HttpGetWithRetry(t, url, nil, 200, "Hello world!", 30, 5*time.Second) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 48
  49. Ladislav Prskavec - Teststack conference, 4. 6. 2020 49

  50. Example Hello World with docker-compose and stages Ladislav Prskavec -

    Teststack conference, 4. 6. 2020 50
  51. 1. Build docker images for nginx and webapp 2. Run

    docker-compose 3. Validate nginx and webapp 4. Destroy docker-compose Ladislav Prskavec - Teststack conference, 4. 6. 2020 51
  52. version: "3" services: nginx: image: local/nginx:latest container_name: "production_nginx" networks: -

    "teststack-network" ports: - "80:80" go-webapp: image: go-webapp:latest container_name: "production_go-webapp" environment: - SERVER_TEXT=${SERVER_TEXT} expose: - "8080" networks: - "teststack-network" networks: "teststack-network": Ladislav Prskavec - Teststack conference, 4. 6. 2020 52
  53. package main import (...) func helloHandler(res http.ResponseWriter, req *http.Request) {

    res.Header().Set( "Content-Type", "text/plain", ) io.WriteString( res, fmt.Sprintf("%s", os.Getenv("SERVER_TEXT")), ) } func defaultHandler(w http.ResponseWriter, r *http.Request) { fmt.Fprintf(w, "Go web app powered by Docker") } func main() { http.HandleFunc("/", defaultHandler) http.HandleFunc("/hello", helloHandler) err := http.ListenAndServe(":8080", nil) if err != nil { log.Fatal("ListenAndServe: ", err) return } } Ladislav Prskavec - Teststack conference, 4. 6. 2020 53
  54. func TestDockerComposeLocal(t *testing.T) { tag := "go-webapp" buildargs := "-t

    local/nginx" buildOptions := &docker.BuildOptions{ Tags: []string{tag}, BuildArgs: []string{buildargs} } docker.Build(t, "../hello-world-docker-compose", buildOptions) tag := "local/nginx" buildargs := "-f Dockerfile.nginx" buildOptions := &docker.BuildOptions{ Tags: []string{tag}, BuildArgs: []string{buildargs} } docker.Build(t, "../hello-world-docker-compose", buildOptions) Ladislav Prskavec - Teststack conference, 4. 6. 2020 54
  55. serverPort := 80 randomSuffix := random.UniqueId() expectedServerText := fmt.Sprintf("Hello, %s!",

    randomSuffix) dockerOptions := &docker.Options{ WorkingDir: "../hello-world-docker-compose", EnvVars: map[string]string{ "SERVER_TEXT": expectedServerText, "SERVER_PORT": strconv.Itoa(serverPort), "randomSuffix": randomSuffix, }, } defer docker.RunDockerCompose(t, dockerOptions, "down") docker.RunDockerCompose(t, dockerOptions, "up", "-d") maxRetries := 10 timeBetweenRetries := 5 * time.Second url := fmt.Sprintf("http://localhost:%d/hello", serverPort) tlsConfig := tls.Config{} http_helper.HttpGetWithRetry(t, url, &tlsConfig, 200, expectedServerText, maxRetries, timeBetweenRetries) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 55
  56. Ladislav Prskavec - Teststack conference, 4. 6. 2020 56

  57. func TestDockerComposeWithStagesLocal(t *testing.T) { workingDir := "../hello-world-docker-compose" test_structure.RunTestStage(t, "build_docker_image", func()

    { buildImage(t, "go-webapp", "../demowebapp") buildImage(t, "local/nginx", "../nginx") }) test_structure.RunTestStage(t, "run_docker_compose", func() { runCompose(t, workingDir) }) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 57
  58. func buildImage(t *testing.T, tag string, workingDir string) { buildOptions :=

    &docker.BuildOptions{ Tags: []string{tag}, } docker.Build(t, workingDir, buildOptions) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 58
  59. func runCompose(t *testing.T, workingDir string) { serverPort := 80 randomSuffix

    := random.UniqueId() expectedServerText := fmt.Sprintf("Hello, %s!", randomSuffix) dockerOptions := &docker.Options{ WorkingDir: workingDir, EnvVars: map[string]string{ "SERVER_TEXT": expectedServerText, "randomSuffix": randomSuffix, "SERVER_PORT": strconv.Itoa(serverPort), }, } defer docker.RunDockerCompose(t, dockerOptions, "down") docker.RunDockerCompose(t, dockerOptions, "up", "-d") opts := &docker.RunOptions{ Command: []string{"--retry", "5", "--retry-connrefused", "-s", "http://production_nginx:80/hello"}, OtherOptions: []string{"--network", "testdockercomposewithstageslocal_teststack-network"}, } tag := "appropriate/curl" output := docker.Run(t, tag, opts) assert.Equal(t, expectedServerText, output) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 59
  60. Ladislav Prskavec - Teststack conference, 4. 6. 2020 60

  61. For iteration skip build stage Ladislav Prskavec - Teststack conference,

    4. 6. 2020 61
  62. Ladislav Prskavec - Teststack conference, 4. 6. 2020 62

  63. E2E Tests Ladislav Prskavec - Teststack conference, 4. 6. 2020

    63
  64. Can we use this test strategy? 1. Deploy all the

    infrastruture 2. Validate it works 3. Undeploy all the infrastruture Ladislav Prskavec - Teststack conference, 4. 6. 2020 64
  65. Test type Number of resources Chance of failure Unit test

    10 1% Integration test 50 5% E2E test 500+ 40%+ For unit & integration you improve reliability with retries. Ladislav Prskavec - Teststack conference, 4. 6. 2020 65
  66. Please don't write E2E tests in this way. Ladislav Prskavec

    - Teststack conference, 4. 6. 2020 66
  67. Incremental testing Ladislav Prskavec - Teststack conference, 4. 6. 2020

    67
  68. Incremental testing 1. Deploy a persistent test environment and leave

    it running. 2. Each time you update a module, deploy & validate just that module. 3. Test your deployment process is zero-downtime too. Ladislav Prskavec - Teststack conference, 4. 6. 2020 68
  69. Summary 4 Static Analysis (1-60 seconds) 4 Unit Tests (1-20

    minutes) 4 Integration Tests (5-60 minutes) 4 E2E Tests (60-240+ minutes) Ladislav Prskavec - Teststack conference, 4. 6. 2020 69
  70. What Good Worse Static Analysis Very Fast (until 1 min),

    Stable, Easy to use, No deploy Limited errors for catch, no reals checks with infrastructure Unit tests Fast (1-20 min), very good stability with retries, trust in components Need real deploy, non trivial test code, extra work for CI Integration tests Mostly stable with retries, High level confidence in multiple units Slow (5-60 min), Need real deploy, non trivial test code, extra work for CI E2E tests Build confidence in entire architecture Very slow (60-240+ min), Need real deploy, Still problem 0- deployments, Can be brittle (even with retries) Ladislav Prskavec - Teststack conference, 4. 6. 2020 70
  71. Try make at least one test for your docker environment!

    Ladislav Prskavec - Teststack conference, 4. 6. 2020 71
  72. Infrastructure code without tests is scary Ladislav Prskavec - Teststack

    conference, 4. 6. 2020 72
  73. Q & A #test-infrastructure on Teststack Slack Ladislav Prskavec -

    Teststack conference, 4. 6. 2020 73