$30 off During Our Annual Pro Sale. View Details »

Testing Infrastructure with Terratest

Testing Infrastructure with Terratest

How to Test Infrastructure Code using Terraform, Terratest from Gruntwork.io

Ladislav Prskavec

April 06, 2020
Tweet

More Decks by Ladislav Prskavec

Other Decks in Technology

Transcript

  1. Testing Infrastructure
    with
    Terratest
    Ladislav Prskavec
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 1

    View Slide

  2. Who Am I
    4 Oracle Cloud Infrastructure - API Services team
    4 Twitter: @abtris
    4 Website: https://www.prskavec.net/
    4 Golang Prague organizer (@GoMeetupPrague)
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 2

    View Slide

  3. Infrastructure as code
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 3

    View Slide

  4. Ladislav Prskavec - Teststack conference, 4. 6. 2020 4

    View Slide

  5. How to Test Infrastructure Code
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 5

    View Slide

  6. Ladislav Prskavec - Teststack conference, 4. 6. 2020 6

    View Slide

  7. Yevgeniy Brikman
    4 co-founder of Gruntwork with Josh
    Padnick
    4 Gruntwork are authors of tools:
    Terragrunt and Terratest
    4 Books Author - Terraform: Up &
    Running and Hello, Startup
    4 Yevgeniy Brikman talk Automated
    Testing for Terraform, Docker,
    Packer, Kubernetes, and More, 11
    Nov 2019
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 7

    View Slide

  8. Test Pyramid
    4 https://martinfowler.com/articles/practical-test-pyramid.html
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 8

    View Slide

  9. How we can test infrastructure code
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 9

    View Slide

  10. Static Analysis
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 10

    View Slide

  11. Syntax validation
    4 terraform validate
    4 kubernetes apply -f --dry-run --
    validate=true
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 11

    View Slide

  12. Dry-run
    4 terraform plan
    4 kubectl apply -f --server-dry-run
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 12

    View Slide

  13. Linting & policies checks
    4 hadolint
    4 kube-score
    4 conftest - Open Policy Agent Rego query language
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 13

    View Slide

  14. Unit Tests
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 14

    View Slide

  15. No pure unit testing for
    infrastructure
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 15

    View Slide

  16. Test strategy (per unit)
    1. Deploy real infrastructure
    2. Validate it works
    3. Undeploy the infrastructure
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 16

    View Slide

  17. Example
    Hello World with docker
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 17

    View Slide

  18. Dockerfile
    FROM alpine
    RUN echo 'Hello, World!' > /test.txt
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 18

    View Slide

  19. Test
    package test
    import (
    "testing"
    "github.com/gruntwork-io/terratest/modules/docker"
    "github.com/magiconair/properties/assert"
    )
    func TestDockerHelloWorldExample(t *testing.T) {
    tag := "teststack/demo-terratest-docker"
    buildOptions := &docker.BuildOptions{
    Tags: []string{tag},
    }
    docker.Build(t, "../hello-world", buildOptions)
    opts := &docker.RunOptions{Command: []string{"cat", "/test.txt"}}
    output := docker.Run(t, tag, opts)
    assert.Equal(t, "Hello, World!", output)
    }
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 19

    View Slide

  20. Local vs CI tests
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 20

    View Slide

  21. Running tests in CI
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 21

    View Slide

  22. Integration Tests
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 22

    View Slide

  23. Test strategy - integration test
    1. Deploy all the infrastruture
    2. Validate it works
    3. Undeploy all the infrastruture
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 23

    View Slide

  24. Basic example
    OCI - APIGW and Fn
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 24

    View Slide

  25. const fdk = require('@fnproject/fdk');
    fdk.handle(function(input) {
    let name = 'World';
    if (input.name) {
    name = input.name;
    }
    return {message: 'Hello ' + name};
    });
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 25

    View Slide

  26. schema_version: 20180708
    name: hello-world
    version: 0.0.2
    runtime: node
    entrypoint: node func.js
    format: http-stream
    triggers:
    - name: hello-world-trigger
    type: http
    source: /hello-world-trigger
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 26

    View Slide

  27. package test
    import (...)
    func TestHelloWorldAppUnit(t *testing.T) {
    uniqueID := random.UniqueId()
    terraformOptions := &terraform.Options{
    TerraformDir: "../",
    Vars: map[string]interface{}{
    "name": fmt.Sprintf("hello-world-%s", uniqueID),
    "suffix": fmt.Sprintf("%s", uniqueID),
    "tenancy_ocid": os.Getenv("TF_VAR_tenancy_ocid"),
    "user_ocid": os.Getenv("TF_VAR_user_ocid"),
    "private_key_path": os.Getenv("TF_VAR_private_key_path"),
    "fingerprint": os.Getenv("TF_VAR_fingerprint"),
    "private_key_password": os.Getenv("TF_VAR_private_key_password"),
    "region": os.Getenv("TF_VAR_region"),
    },
    }
    defer terraform.Destroy(t, terraformOptions)
    terraform.InitAndApply(t, terraformOptions)
    validateHelloWorldApp(t, terraformOptions)
    }
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 27

    View Slide

  28. func validateHelloWorldApp(t *testing.T, terraformOptions *terraform.Options) {
    url := terraform.Output(t, terraformOptions, "gateway_url")
    expectedStatus := 200
    expectedBody := `{"message":"Hello World"}`
    maxRetries := 10
    timeBetweenRetries := 10 * time.Second
    http_helper.HttpGetWithRetry(t, url, nil,
    expectedStatus,
    expectedBody,
    maxRetries,
    timeBetweenRetries
    )
    }
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 28

    View Slide

  29. Ladislav Prskavec - Teststack conference, 4. 6. 2020 29

    View Slide

  30. Cleanup after tests
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 30

    View Slide

  31. 4 aws-nuke
    4 cloud-nuke
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 31

    View Slide

  32. Validate
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 32

    View Slide

  33. Validate what make sense for you
    4 test terraform output
    4 http calls api/application
    4 tcp/udp calls
    4 ssh commands
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 33

    View Slide

  34. Test parallelism
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 34

    View Slide

  35. Parallelism
    4 locking resources issue,
    4 separations (compartments, accounts, regions)
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 35

    View Slide

  36. Running tests in parallel
    func TestKubernetesBasicExample(t *testing.T) {
    t.Parallel()
    ...
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 36

    View Slide

  37. Resource conflicts
    4 randomize
    4 namespace (k8s) / compartments (OCI) / accounts
    (AWS)
    import "github.com/gruntwork-io/terratest/modules/random"
    randomSuffix := random.UniqueId()
    // unique namespace
    namespaceName := fmt.Sprintf("test-%s", strings.ToLower(random.UniqueId()))
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 37

    View Slide

  38. Test stages
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 38

    View Slide

  39. import test_structure "github.com/gruntwork-io/terratest/modules/test-structure"
    func TestDockerComposeWithStagesLocal(t *testing.T) {
    workingDir := "../hello-world-docker-compose"
    test_structure.RunTestStage(t, "build_docker_image", func() {
    buildImage(t, "go-webapp", "../demowebapp")
    buildImage(t, "local/nginx", "../nginx")
    })
    test_structure.RunTestStage(t, "run_docker_compose", func() {
    runCompose(t, workingDir)
    })
    }
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 39

    View Slide

  40. Why skip stages
    4 skip cleanup on first run
    4 skip deploy/build/cleanup on iterations
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 40

    View Slide

  41. How skip stages
    4 use environment variable SKIP_stage_name=1
    4 run with that:
    SKIP_build_docker_image=1 go test
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 41

    View Slide

  42. Why and how make retries
    4 Exponential Backoff And Jitter
    4 Simulator for AWS architecture blog (http://www.awsarchitectureblog.com/ ) about jitter and backoff.
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 42

    View Slide

  43. How make retries
    import http_helper "github.com/gruntwork-io/terratest/modules/http-helper"
    retries := 30
    sleepBetweenRetries := 5*time.Second
    http_helper.HttpGetWithRetry(t, url, nil, 200, "Hello world!",
    retries,
    sleepBetweenRetries)
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 43

    View Slide

  44. How make retries with curl
    opts := &docker.RunOptions{
    Command: []string{"--retry", "5",
    "--retry-connrefused", "-s",
    "http://production_nginx:80/hello"},
    OtherOptions: []string{"--network", "testdockercomposewithstageslocal_teststack-network"},
    }
    tag := "appropriate/curl"
    output := docker.Run(t, tag, opts)
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 44

    View Slide

  45. Example
    Hello World with K8S
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 45

    View Slide

  46. apiVersion: apps/v1
    kind: Deployment
    metadata:
    name: hello-world-deployment
    spec:
    selector:
    matchLabels:
    app: hello-world
    replicas: 1
    template:
    metadata:
    labels:
    app: hello-world
    spec:
    containers:
    - name: hello-world
    image: training/webapp:latest
    ports:
    - containerPort: 5000
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 46

    View Slide

  47. kind: Service
    apiVersion: v1
    metadata:
    name: hello-world-service
    spec:
    selector:
    app: hello-world
    ports:
    - protocol: TCP
    targetPort: 5000
    port: 5000
    type: LoadBalancer
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 47

    View Slide

  48. package test
    import (...)
    func TestKubernetesHelloWorldExample(t *testing.T) {
    kubeResourcePath := "../hello-world-kubernetes/deployment.yaml"
    // unique namespace
    namespaceName := fmt.Sprintf("test-%s", strings.ToLower(random.UniqueId()))
    options := k8s.NewKubectlOptions("", "", namespaceName)
    k8s.CreateNamespace(t, options, namespaceName)
    defer k8s.DeleteNamespace(t, options, namespaceName)
    defer k8s.KubectlDelete(t, options, kubeResourcePath)
    k8s.KubectlApply(t, options, kubeResourcePath)
    k8s.WaitUntilServiceAvailable(t, options, "hello-world-service", 10, 3*time.Second)
    service := k8s.GetService(t, options, "hello-world-service")
    url := fmt.Sprintf("http://%s", k8s.GetServiceEndpoint(t, options, service, 5000))
    http_helper.HttpGetWithRetry(t, url, nil, 200, "Hello world!", 30, 5*time.Second)
    }
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 48

    View Slide

  49. Ladislav Prskavec - Teststack conference, 4. 6. 2020 49

    View Slide

  50. Example
    Hello World with
    docker-compose and stages
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 50

    View Slide

  51. 1. Build docker images for nginx and webapp
    2. Run docker-compose
    3. Validate nginx and webapp
    4. Destroy docker-compose
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 51

    View Slide

  52. version: "3"
    services:
    nginx:
    image: local/nginx:latest
    container_name: "production_nginx"
    networks:
    - "teststack-network"
    ports:
    - "80:80"
    go-webapp:
    image: go-webapp:latest
    container_name: "production_go-webapp"
    environment:
    - SERVER_TEXT=${SERVER_TEXT}
    expose:
    - "8080"
    networks:
    - "teststack-network"
    networks:
    "teststack-network":
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 52

    View Slide

  53. package main
    import (...)
    func helloHandler(res http.ResponseWriter, req *http.Request) {
    res.Header().Set(
    "Content-Type",
    "text/plain",
    )
    io.WriteString(
    res,
    fmt.Sprintf("%s", os.Getenv("SERVER_TEXT")),
    )
    }
    func defaultHandler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Go web app powered by Docker")
    }
    func main() {
    http.HandleFunc("/", defaultHandler)
    http.HandleFunc("/hello", helloHandler)
    err := http.ListenAndServe(":8080", nil)
    if err != nil {
    log.Fatal("ListenAndServe: ", err)
    return
    }
    }
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 53

    View Slide

  54. func TestDockerComposeLocal(t *testing.T) {
    tag := "go-webapp"
    buildargs := "-t local/nginx"
    buildOptions := &docker.BuildOptions{
    Tags: []string{tag},
    BuildArgs: []string{buildargs}
    }
    docker.Build(t, "../hello-world-docker-compose", buildOptions)
    tag := "local/nginx"
    buildargs := "-f Dockerfile.nginx"
    buildOptions := &docker.BuildOptions{
    Tags: []string{tag},
    BuildArgs: []string{buildargs}
    }
    docker.Build(t, "../hello-world-docker-compose", buildOptions)
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 54

    View Slide

  55. serverPort := 80
    randomSuffix := random.UniqueId()
    expectedServerText := fmt.Sprintf("Hello, %s!", randomSuffix)
    dockerOptions := &docker.Options{
    WorkingDir: "../hello-world-docker-compose",
    EnvVars: map[string]string{
    "SERVER_TEXT": expectedServerText,
    "SERVER_PORT": strconv.Itoa(serverPort),
    "randomSuffix": randomSuffix,
    },
    }
    defer docker.RunDockerCompose(t, dockerOptions, "down")
    docker.RunDockerCompose(t, dockerOptions, "up", "-d")
    maxRetries := 10
    timeBetweenRetries := 5 * time.Second
    url := fmt.Sprintf("http://localhost:%d/hello", serverPort)
    tlsConfig := tls.Config{}
    http_helper.HttpGetWithRetry(t, url, &tlsConfig, 200, expectedServerText, maxRetries, timeBetweenRetries)
    }
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 55

    View Slide

  56. Ladislav Prskavec - Teststack conference, 4. 6. 2020 56

    View Slide

  57. func TestDockerComposeWithStagesLocal(t *testing.T) {
    workingDir := "../hello-world-docker-compose"
    test_structure.RunTestStage(t, "build_docker_image", func() {
    buildImage(t, "go-webapp", "../demowebapp")
    buildImage(t, "local/nginx", "../nginx")
    })
    test_structure.RunTestStage(t, "run_docker_compose", func() {
    runCompose(t, workingDir)
    })
    }
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 57

    View Slide

  58. func buildImage(t *testing.T, tag string, workingDir string) {
    buildOptions := &docker.BuildOptions{
    Tags: []string{tag},
    }
    docker.Build(t, workingDir, buildOptions)
    }
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 58

    View Slide

  59. func runCompose(t *testing.T, workingDir string) {
    serverPort := 80
    randomSuffix := random.UniqueId()
    expectedServerText := fmt.Sprintf("Hello, %s!", randomSuffix)
    dockerOptions := &docker.Options{
    WorkingDir: workingDir,
    EnvVars: map[string]string{
    "SERVER_TEXT": expectedServerText,
    "randomSuffix": randomSuffix,
    "SERVER_PORT": strconv.Itoa(serverPort),
    },
    }
    defer docker.RunDockerCompose(t, dockerOptions, "down")
    docker.RunDockerCompose(t, dockerOptions, "up", "-d")
    opts := &docker.RunOptions{
    Command: []string{"--retry", "5", "--retry-connrefused", "-s", "http://production_nginx:80/hello"},
    OtherOptions: []string{"--network", "testdockercomposewithstageslocal_teststack-network"},
    }
    tag := "appropriate/curl"
    output := docker.Run(t, tag, opts)
    assert.Equal(t, expectedServerText, output)
    }
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 59

    View Slide

  60. Ladislav Prskavec - Teststack conference, 4. 6. 2020 60

    View Slide

  61. For iteration
    skip build stage
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 61

    View Slide

  62. Ladislav Prskavec - Teststack conference, 4. 6. 2020 62

    View Slide

  63. E2E Tests
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 63

    View Slide

  64. Can we use this test strategy?
    1. Deploy all the infrastruture
    2. Validate it works
    3. Undeploy all the infrastruture
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 64

    View Slide

  65. Test type Number of
    resources
    Chance of failure
    Unit test 10 1%
    Integration test 50 5%
    E2E test 500+ 40%+
    For unit & integration you improve reliability with retries.
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 65

    View Slide

  66. Please don't write E2E
    tests in this way.
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 66

    View Slide

  67. Incremental testing
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 67

    View Slide

  68. Incremental testing
    1. Deploy a persistent test environment and leave it
    running.
    2. Each time you update a module, deploy & validate
    just that module.
    3. Test your deployment process is zero-downtime too.
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 68

    View Slide

  69. Summary
    4 Static Analysis (1-60 seconds)
    4 Unit Tests (1-20 minutes)
    4 Integration Tests (5-60 minutes)
    4 E2E Tests (60-240+ minutes)
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 69

    View Slide

  70. What Good Worse
    Static Analysis Very Fast (until 1 min), Stable, Easy
    to use, No deploy
    Limited errors for catch, no reals
    checks with infrastructure
    Unit tests Fast (1-20 min), very good stability
    with retries, trust in components
    Need real deploy, non trivial test
    code, extra work for CI
    Integration tests Mostly stable with retries, High
    level confidence in multiple units
    Slow (5-60 min), Need real deploy,
    non trivial test code, extra work
    for CI
    E2E tests Build confidence in entire
    architecture
    Very slow (60-240+ min), Need
    real deploy, Still problem 0-
    deployments, Can be brittle (even
    with retries)
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 70

    View Slide

  71. Try make at least one test for your
    docker environment!
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 71

    View Slide

  72. Infrastructure code
    without tests is scary
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 72

    View Slide

  73. Q & A
    #test-infrastructure on Teststack Slack
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 73

    View Slide