Testing Infrastructure with Terratest

How to Test Infrastructure Code using Terraform, Terratest from Gruntwork.io

Transcript

1. Testing Infrastructure
   with
   Terratest
   Ladislav Prskavec
   Ladislav Prskavec - Teststack conference, 4. 6. 2020 1
   

   View Slide

2. Who Am I
   4 Oracle Cloud Infrastructure - API Services team
   4 Twitter: @abtris
   4 Website: https://www.prskavec.net/
   4 Golang Prague organizer (@GoMeetupPrague)
   Ladislav Prskavec - Teststack conference, 4. 6. 2020 2
   

   View Slide

3. Infrastructure as code
   Ladislav Prskavec - Teststack conference, 4. 6. 2020 3
   

   View Slide

4. Ladislav Prskavec - Teststack conference, 4. 6. 2020 4
   

   View Slide

5. How to Test Infrastructure Code
   Ladislav Prskavec - Teststack conference, 4. 6. 2020 5
   

   View Slide

6. Ladislav Prskavec - Teststack conference, 4. 6. 2020 6
   

   View Slide

7. Yevgeniy Brikman
   4 co-founder of Gruntwork with Josh
   Padnick
   4 Gruntwork are authors of tools:
   Terragrunt and Terratest
   4 Books Author - Terraform: Up &
   Running and Hello, Startup
   4 Yevgeniy Brikman talk Automated
   Testing for Terraform, Docker,
   Packer, Kubernetes, and More, 11
   Nov 2019
   Ladislav Prskavec - Teststack conference, 4. 6. 2020 7
   

   View Slide

8. Test Pyramid
   4 https://martinfowler.com/articles/practical-test-pyramid.html
   Ladislav Prskavec - Teststack conference, 4. 6. 2020 8
   

   View Slide

9. How we can test infrastructure code
   Ladislav Prskavec - Teststack conference, 4. 6. 2020 9
   

   View Slide

10. Static Analysis
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 10
    

    View Slide

11. Syntax validation
    4 terraform validate
    4 kubernetes apply -f --dry-run --
    validate=true
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 11
    

    View Slide

12. Dry-run
    4 terraform plan
    4 kubectl apply -f --server-dry-run
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 12
    

    View Slide

13. Linting & policies checks
    4 hadolint
    4 kube-score
    4 conftest - Open Policy Agent Rego query language
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 13
    

    View Slide

14. Unit Tests
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 14
    

    View Slide

15. No pure unit testing for
    infrastructure
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 15
    

    View Slide

16. Test strategy (per unit)
    1. Deploy real infrastructure
    2. Validate it works
    3. Undeploy the infrastructure
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 16
    

    View Slide

17. Example
    Hello World with docker
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 17
    

    View Slide

18. Dockerfile
    FROM alpine
    RUN echo 'Hello, World!' > /test.txt
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 18
    

    View Slide

19. Test
    package test
    import (
    "testing"
    "github.com/gruntwork-io/terratest/modules/docker"
    "github.com/magiconair/properties/assert"
    )
    func TestDockerHelloWorldExample(t *testing.T) {
    tag := "teststack/demo-terratest-docker"
    buildOptions := &docker.BuildOptions{
    Tags: []string{tag},
    }
    docker.Build(t, "../hello-world", buildOptions)
    opts := &docker.RunOptions{Command: []string{"cat", "/test.txt"}}
    output := docker.Run(t, tag, opts)
    assert.Equal(t, "Hello, World!", output)
    }
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 19
    

    View Slide

20. Local vs CI tests
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 20
    

    View Slide

21. Running tests in CI
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 21
    

    View Slide

22. Integration Tests
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 22
    

    View Slide

23. Test strategy - integration test
    1. Deploy all the infrastruture
    2. Validate it works
    3. Undeploy all the infrastruture
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 23
    

    View Slide

24. Basic example
    OCI - APIGW and Fn
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 24
    

    View Slide

25. const fdk = require('@fnproject/fdk');
    fdk.handle(function(input) {
    let name = 'World';
    if (input.name) {
    name = input.name;
    }
    return {message: 'Hello ' + name};
    });
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 25
    

    View Slide

26. schema_version: 20180708
    name: hello-world
    version: 0.0.2
    runtime: node
    entrypoint: node func.js
    format: http-stream
    triggers:
    - name: hello-world-trigger
    type: http
    source: /hello-world-trigger
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 26
    

    View Slide

27. package test
    import (...)
    func TestHelloWorldAppUnit(t *testing.T) {
    uniqueID := random.UniqueId()
    terraformOptions := &terraform.Options{
    TerraformDir: "../",
    Vars: map[string]interface{}{
    "name": fmt.Sprintf("hello-world-%s", uniqueID),
    "suffix": fmt.Sprintf("%s", uniqueID),
    "tenancy_ocid": os.Getenv("TF_VAR_tenancy_ocid"),
    "user_ocid": os.Getenv("TF_VAR_user_ocid"),
    "private_key_path": os.Getenv("TF_VAR_private_key_path"),
    "fingerprint": os.Getenv("TF_VAR_fingerprint"),
    "private_key_password": os.Getenv("TF_VAR_private_key_password"),
    "region": os.Getenv("TF_VAR_region"),
    },
    }
    defer terraform.Destroy(t, terraformOptions)
    terraform.InitAndApply(t, terraformOptions)
    validateHelloWorldApp(t, terraformOptions)
    }
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 27
    

    View Slide

28. func validateHelloWorldApp(t *testing.T, terraformOptions *terraform.Options) {
    url := terraform.Output(t, terraformOptions, "gateway_url")
    expectedStatus := 200
    expectedBody := `{"message":"Hello World"}`
    maxRetries := 10
    timeBetweenRetries := 10 * time.Second
    http_helper.HttpGetWithRetry(t, url, nil,
    expectedStatus,
    expectedBody,
    maxRetries,
    timeBetweenRetries
    )
    }
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 28
    

    View Slide

29. Ladislav Prskavec - Teststack conference, 4. 6. 2020 29
    

    View Slide

30. Cleanup after tests
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 30
    

    View Slide

31. 4 aws-nuke
    4 cloud-nuke
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 31
    

    View Slide

32. Validate
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 32
    

    View Slide

33. Validate what make sense for you
    4 test terraform output
    4 http calls api/application
    4 tcp/udp calls
    4 ssh commands
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 33
    

    View Slide

34. Test parallelism
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 34
    

    View Slide

35. Parallelism
    4 locking resources issue,
    4 separations (compartments, accounts, regions)
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 35
    

    View Slide

36. Running tests in parallel
    func TestKubernetesBasicExample(t *testing.T) {
    t.Parallel()
    ...
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 36
    

    View Slide

37. Resource conflicts
    4 randomize
    4 namespace (k8s) / compartments (OCI) / accounts
    (AWS)
    import "github.com/gruntwork-io/terratest/modules/random"
    randomSuffix := random.UniqueId()
    // unique namespace
    namespaceName := fmt.Sprintf("test-%s", strings.ToLower(random.UniqueId()))
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 37
    

    View Slide

38. Test stages
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 38
    

    View Slide

39. import test_structure "github.com/gruntwork-io/terratest/modules/test-structure"
    func TestDockerComposeWithStagesLocal(t *testing.T) {
    workingDir := "../hello-world-docker-compose"
    test_structure.RunTestStage(t, "build_docker_image", func() {
    buildImage(t, "go-webapp", "../demowebapp")
    buildImage(t, "local/nginx", "../nginx")
    })
    test_structure.RunTestStage(t, "run_docker_compose", func() {
    runCompose(t, workingDir)
    })
    }
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 39
    

    View Slide

40. Why skip stages
    4 skip cleanup on first run
    4 skip deploy/build/cleanup on iterations
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 40
    

    View Slide

41. How skip stages
    4 use environment variable SKIP_stage_name=1
    4 run with that:
    SKIP_build_docker_image=1 go test
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 41
    

    View Slide

42. Why and how make retries
    4 Exponential Backoff And Jitter
    4 Simulator for AWS architecture blog (http://www.awsarchitectureblog.com/ ) about jitter and backoff.
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 42
    

    View Slide

43. How make retries
    import http_helper "github.com/gruntwork-io/terratest/modules/http-helper"
    retries := 30
    sleepBetweenRetries := 5*time.Second
    http_helper.HttpGetWithRetry(t, url, nil, 200, "Hello world!",
    retries,
    sleepBetweenRetries)
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 43
    

    View Slide

44. How make retries with curl
    opts := &docker.RunOptions{
    Command: []string{"--retry", "5",
    "--retry-connrefused", "-s",
    "http://production_nginx:80/hello"},
    OtherOptions: []string{"--network", "testdockercomposewithstageslocal_teststack-network"},
    }
    tag := "appropriate/curl"
    output := docker.Run(t, tag, opts)
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 44
    

    View Slide

45. Example
    Hello World with K8S
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 45
    

    View Slide

46. apiVersion: apps/v1
    kind: Deployment
    metadata:
    name: hello-world-deployment
    spec:
    selector:
    matchLabels:
    app: hello-world
    replicas: 1
    template:
    metadata:
    labels:
    app: hello-world
    spec:
    containers:
    - name: hello-world
    image: training/webapp:latest
    ports:
    - containerPort: 5000
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 46
    

    View Slide

47. kind: Service
    apiVersion: v1
    metadata:
    name: hello-world-service
    spec:
    selector:
    app: hello-world
    ports:
    - protocol: TCP
    targetPort: 5000
    port: 5000
    type: LoadBalancer
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 47
    

    View Slide

48. package test
    import (...)
    func TestKubernetesHelloWorldExample(t *testing.T) {
    kubeResourcePath := "../hello-world-kubernetes/deployment.yaml"
    // unique namespace
    namespaceName := fmt.Sprintf("test-%s", strings.ToLower(random.UniqueId()))
    options := k8s.NewKubectlOptions("", "", namespaceName)
    k8s.CreateNamespace(t, options, namespaceName)
    defer k8s.DeleteNamespace(t, options, namespaceName)
    defer k8s.KubectlDelete(t, options, kubeResourcePath)
    k8s.KubectlApply(t, options, kubeResourcePath)
    k8s.WaitUntilServiceAvailable(t, options, "hello-world-service", 10, 3*time.Second)
    service := k8s.GetService(t, options, "hello-world-service")
    url := fmt.Sprintf("http://%s", k8s.GetServiceEndpoint(t, options, service, 5000))
    http_helper.HttpGetWithRetry(t, url, nil, 200, "Hello world!", 30, 5*time.Second)
    }
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 48
    

    View Slide

49. Ladislav Prskavec - Teststack conference, 4. 6. 2020 49
    

    View Slide

50. Example
    Hello World with
    docker-compose and stages
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 50
    

    View Slide

51. 1. Build docker images for nginx and webapp
    2. Run docker-compose
    3. Validate nginx and webapp
    4. Destroy docker-compose
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 51
    

    View Slide

52. version: "3"
    services:
    nginx:
    image: local/nginx:latest
    container_name: "production_nginx"
    networks:
    - "teststack-network"
    ports:
    - "80:80"
    go-webapp:
    image: go-webapp:latest
    container_name: "production_go-webapp"
    environment:
    - SERVER_TEXT=${SERVER_TEXT}
    expose:
    - "8080"
    networks:
    - "teststack-network"
    networks:
    "teststack-network":
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 52
    

    View Slide

53. package main
    import (...)
    func helloHandler(res http.ResponseWriter, req *http.Request) {
    res.Header().Set(
    "Content-Type",
    "text/plain",
    )
    io.WriteString(
    res,
    fmt.Sprintf("%s", os.Getenv("SERVER_TEXT")),
    )
    }
    func defaultHandler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Go web app powered by Docker")
    }
    func main() {
    http.HandleFunc("/", defaultHandler)
    http.HandleFunc("/hello", helloHandler)
    err := http.ListenAndServe(":8080", nil)
    if err != nil {
    log.Fatal("ListenAndServe: ", err)
    return
    }
    }
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 53
    

    View Slide

54. func TestDockerComposeLocal(t *testing.T) {
    tag := "go-webapp"
    buildargs := "-t local/nginx"
    buildOptions := &docker.BuildOptions{
    Tags: []string{tag},
    BuildArgs: []string{buildargs}
    }
    docker.Build(t, "../hello-world-docker-compose", buildOptions)
    tag := "local/nginx"
    buildargs := "-f Dockerfile.nginx"
    buildOptions := &docker.BuildOptions{
    Tags: []string{tag},
    BuildArgs: []string{buildargs}
    }
    docker.Build(t, "../hello-world-docker-compose", buildOptions)
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 54
    

    View Slide

55. serverPort := 80
    randomSuffix := random.UniqueId()
    expectedServerText := fmt.Sprintf("Hello, %s!", randomSuffix)
    dockerOptions := &docker.Options{
    WorkingDir: "../hello-world-docker-compose",
    EnvVars: map[string]string{
    "SERVER_TEXT": expectedServerText,
    "SERVER_PORT": strconv.Itoa(serverPort),
    "randomSuffix": randomSuffix,
    },
    }
    defer docker.RunDockerCompose(t, dockerOptions, "down")
    docker.RunDockerCompose(t, dockerOptions, "up", "-d")
    maxRetries := 10
    timeBetweenRetries := 5 * time.Second
    url := fmt.Sprintf("http://localhost:%d/hello", serverPort)
    tlsConfig := tls.Config{}
    http_helper.HttpGetWithRetry(t, url, &tlsConfig, 200, expectedServerText, maxRetries, timeBetweenRetries)
    }
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 55
    

    View Slide

56. Ladislav Prskavec - Teststack conference, 4. 6. 2020 56
    

    View Slide

57. func TestDockerComposeWithStagesLocal(t *testing.T) {
    workingDir := "../hello-world-docker-compose"
    test_structure.RunTestStage(t, "build_docker_image", func() {
    buildImage(t, "go-webapp", "../demowebapp")
    buildImage(t, "local/nginx", "../nginx")
    })
    test_structure.RunTestStage(t, "run_docker_compose", func() {
    runCompose(t, workingDir)
    })
    }
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 57
    

    View Slide

58. func buildImage(t *testing.T, tag string, workingDir string) {
    buildOptions := &docker.BuildOptions{
    Tags: []string{tag},
    }
    docker.Build(t, workingDir, buildOptions)
    }
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 58
    

    View Slide

59. func runCompose(t *testing.T, workingDir string) {
    serverPort := 80
    randomSuffix := random.UniqueId()
    expectedServerText := fmt.Sprintf("Hello, %s!", randomSuffix)
    dockerOptions := &docker.Options{
    WorkingDir: workingDir,
    EnvVars: map[string]string{
    "SERVER_TEXT": expectedServerText,
    "randomSuffix": randomSuffix,
    "SERVER_PORT": strconv.Itoa(serverPort),
    },
    }
    defer docker.RunDockerCompose(t, dockerOptions, "down")
    docker.RunDockerCompose(t, dockerOptions, "up", "-d")
    opts := &docker.RunOptions{
    Command: []string{"--retry", "5", "--retry-connrefused", "-s", "http://production_nginx:80/hello"},
    OtherOptions: []string{"--network", "testdockercomposewithstageslocal_teststack-network"},
    }
    tag := "appropriate/curl"
    output := docker.Run(t, tag, opts)
    assert.Equal(t, expectedServerText, output)
    }
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 59
    

    View Slide

60. Ladislav Prskavec - Teststack conference, 4. 6. 2020 60
    

    View Slide

61. For iteration
    skip build stage
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 61
    

    View Slide

62. Ladislav Prskavec - Teststack conference, 4. 6. 2020 62
    

    View Slide

63. E2E Tests
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 63
    

    View Slide

64. Can we use this test strategy?
    1. Deploy all the infrastruture
    2. Validate it works
    3. Undeploy all the infrastruture
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 64
    

    View Slide

65. Test type Number of
    resources
    Chance of failure
    Unit test 10 1%
    Integration test 50 5%
    E2E test 500+ 40%+
    For unit & integration you improve reliability with retries.
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 65
    

    View Slide

66. Please don't write E2E
    tests in this way.
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 66
    

    View Slide

67. Incremental testing
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 67
    

    View Slide

68. Incremental testing
    1. Deploy a persistent test environment and leave it
    running.
    2. Each time you update a module, deploy & validate
    just that module.
    3. Test your deployment process is zero-downtime too.
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 68
    

    View Slide

69. Summary
    4 Static Analysis (1-60 seconds)
    4 Unit Tests (1-20 minutes)
    4 Integration Tests (5-60 minutes)
    4 E2E Tests (60-240+ minutes)
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 69
    

    View Slide

70. What Good Worse
    Static Analysis Very Fast (until 1 min), Stable, Easy
    to use, No deploy
    Limited errors for catch, no reals
    checks with infrastructure
    Unit tests Fast (1-20 min), very good stability
    with retries, trust in components
    Need real deploy, non trivial test
    code, extra work for CI
    Integration tests Mostly stable with retries, High
    level confidence in multiple units
    Slow (5-60 min), Need real deploy,
    non trivial test code, extra work
    for CI
    E2E tests Build confidence in entire
    architecture
    Very slow (60-240+ min), Need
    real deploy, Still problem 0-
    deployments, Can be brittle (even
    with retries)
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 70
    

    View Slide

71. Try make at least one test for your
    docker environment!
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 71
    

    View Slide

72. Infrastructure code
    without tests is scary
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 72
    

    View Slide

73. Q & A
    #test-infrastructure on Teststack Slack
    Ladislav Prskavec - Teststack conference, 4. 6. 2020 73
    

    View Slide