Testing Infrastructure with Terratest

How to Test Infrastructure Code using Terraform, Terratest from Gruntwork.io

1. Testing Infrastructure with Terratest Ladislav Prskavec Ladislav Prskavec - Teststack conference, 4. 6. 2020 1
2. Who Am I 4 Oracle Cloud Infrastructure - API Services team 4 Twitter: @abtris 4 Website: https://www.prskavec.net/ 4 Golang Prague organizer (@GoMeetupPrague) Ladislav Prskavec - Teststack conference, 4. 6. 2020 2
3. Infrastructure as code Ladislav Prskavec - Teststack conference, 4. 6. 2020 3
4. Ladislav Prskavec - Teststack conference, 4. 6. 2020 4
5. How to Test Infrastructure Code Ladislav Prskavec - Teststack conference, 4. 6. 2020 5
6. Ladislav Prskavec - Teststack conference, 4. 6. 2020 6
7. Yevgeniy Brikman 4 co-founder of Gruntwork with Josh Padnick 4 Gruntwork are authors of tools: Terragrunt and Terratest 4 Books Author - Terraform: Up & Running and Hello, Startup 4 Yevgeniy Brikman talk Automated Testing for Terraform, Docker, Packer, Kubernetes, and More, 11 Nov 2019 Ladislav Prskavec - Teststack conference, 4. 6. 2020 7
8. Test Pyramid 4 https://martinfowler.com/articles/practical-test-pyramid.html Ladislav Prskavec - Teststack conference, 4. 6. 2020 8
9. How we can test infrastructure code Ladislav Prskavec - Teststack conference, 4. 6. 2020 9
10. Static Analysis Ladislav Prskavec - Teststack conference, 4. 6. 2020 10
11. Syntax validation 4 terraform validate 4 kubernetes apply -f <file> --dry-run -- validate=true Ladislav Prskavec - Teststack conference, 4. 6. 2020 11
12. Dry-run 4 terraform plan 4 kubectl apply -f <file> --server-dry-run Ladislav Prskavec - Teststack conference, 4. 6. 2020 12
13. Linting & policies checks 4 hadolint 4 kube-score 4 conftest - Open Policy Agent Rego query language Ladislav Prskavec - Teststack conference, 4. 6. 2020 13
14. Unit Tests Ladislav Prskavec - Teststack conference, 4. 6. 2020 14
15. No pure unit testing for infrastructure Ladislav Prskavec - Teststack conference, 4. 6. 2020 15
16. Test strategy (per unit) 1. Deploy real infrastructure 2. Validate it works 3. Undeploy the infrastructure Ladislav Prskavec - Teststack conference, 4. 6. 2020 16
17. Example Hello World with docker Ladislav Prskavec - Teststack conference, 4. 6. 2020 17
18. Dockerfile FROM alpine RUN echo 'Hello, World!' > /test.txt Ladislav Prskavec - Teststack conference, 4. 6. 2020 18
19. Test package test import ( "testing" "github.com/gruntwork-io/terratest/modules/docker" "github.com/magiconair/properties/assert" ) func TestDockerHelloWorldExample(t *testing.T) { tag := "teststack/demo-terratest-docker" buildOptions := &docker.BuildOptions{ Tags: []string{tag}, } docker.Build(t, "../hello-world", buildOptions) opts := &docker.RunOptions{Command: []string{"cat", "/test.txt"}} output := docker.Run(t, tag, opts) assert.Equal(t, "Hello, World!", output) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 19
20. Local vs CI tests Ladislav Prskavec - Teststack conference, 4. 6. 2020 20
21. Running tests in CI Ladislav Prskavec - Teststack conference, 4. 6. 2020 21
22. Integration Tests Ladislav Prskavec - Teststack conference, 4. 6. 2020 22
23. Test strategy - integration test 1. Deploy all the infrastruture 2. Validate it works 3. Undeploy all the infrastruture Ladislav Prskavec - Teststack conference, 4. 6. 2020 23
24. Basic example OCI - APIGW and Fn Ladislav Prskavec - Teststack conference, 4. 6. 2020 24
25. const fdk = require('@fnproject/fdk'); fdk.handle(function(input) { let name = 'World'; if (input.name) { name = input.name; } return {message: 'Hello ' + name}; }); Ladislav Prskavec - Teststack conference, 4. 6. 2020 25
26. schema_version: 20180708 name: hello-world version: 0.0.2 runtime: node entrypoint: node func.js format: http-stream triggers: - name: hello-world-trigger type: http source: /hello-world-trigger Ladislav Prskavec - Teststack conference, 4. 6. 2020 26
27. package test import (...) func TestHelloWorldAppUnit(t *testing.T) { uniqueID := random.UniqueId() terraformOptions := &terraform.Options{ TerraformDir: "../", Vars: map[string]interface{}{ "name": fmt.Sprintf("hello-world-%s", uniqueID), "suffix": fmt.Sprintf("%s", uniqueID), "tenancy_ocid": os.Getenv("TF_VAR_tenancy_ocid"), "user_ocid": os.Getenv("TF_VAR_user_ocid"), "private_key_path": os.Getenv("TF_VAR_private_key_path"), "fingerprint": os.Getenv("TF_VAR_fingerprint"), "private_key_password": os.Getenv("TF_VAR_private_key_password"), "region": os.Getenv("TF_VAR_region"), }, } defer terraform.Destroy(t, terraformOptions) terraform.InitAndApply(t, terraformOptions) validateHelloWorldApp(t, terraformOptions) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 27
28. func validateHelloWorldApp(t *testing.T, terraformOptions *terraform.Options) { url := terraform.Output(t, terraformOptions, "gateway_url") expectedStatus := 200 expectedBody := `{"message":"Hello World"}` maxRetries := 10 timeBetweenRetries := 10 * time.Second http_helper.HttpGetWithRetry(t, url, nil, expectedStatus, expectedBody, maxRetries, timeBetweenRetries ) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 28
29. Ladislav Prskavec - Teststack conference, 4. 6. 2020 29
30. Cleanup after tests Ladislav Prskavec - Teststack conference, 4. 6. 2020 30
31. 4 aws-nuke 4 cloud-nuke Ladislav Prskavec - Teststack conference, 4. 6. 2020 31
32. Validate Ladislav Prskavec - Teststack conference, 4. 6. 2020 32
33. Validate what make sense for you 4 test terraform output 4 http calls api/application 4 tcp/udp calls 4 ssh commands Ladislav Prskavec - Teststack conference, 4. 6. 2020 33
34. Test parallelism Ladislav Prskavec - Teststack conference, 4. 6. 2020 34
35. Parallelism 4 locking resources issue, 4 separations (compartments, accounts, regions) Ladislav Prskavec - Teststack conference, 4. 6. 2020 35
36. Running tests in parallel func TestKubernetesBasicExample(t *testing.T) { t.Parallel() ... Ladislav Prskavec - Teststack conference, 4. 6. 2020 36
37. Resource conflicts 4 randomize 4 namespace (k8s) / compartments (OCI) / accounts (AWS) import "github.com/gruntwork-io/terratest/modules/random" randomSuffix := random.UniqueId() // unique namespace namespaceName := fmt.Sprintf("test-%s", strings.ToLower(random.UniqueId())) Ladislav Prskavec - Teststack conference, 4. 6. 2020 37
38. Test stages Ladislav Prskavec - Teststack conference, 4. 6. 2020 38
39. import test_structure "github.com/gruntwork-io/terratest/modules/test-structure" func TestDockerComposeWithStagesLocal(t *testing.T) { workingDir := "../hello-world-docker-compose" test_structure.RunTestStage(t, "build_docker_image", func() { buildImage(t, "go-webapp", "../demowebapp") buildImage(t, "local/nginx", "../nginx") }) test_structure.RunTestStage(t, "run_docker_compose", func() { runCompose(t, workingDir) }) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 39
40. Why skip stages 4 skip cleanup on first run 4 skip deploy/build/cleanup on iterations Ladislav Prskavec - Teststack conference, 4. 6. 2020 40
41. How skip stages 4 use environment variable SKIP_stage_name=1 4 run with that: SKIP_build_docker_image=1 go test Ladislav Prskavec - Teststack conference, 4. 6. 2020 41
42. Why and how make retries 4 Exponential Backoff And Jitter 4 Simulator for AWS architecture blog (http://www.awsarchitectureblog.com/ ) about jitter and backoff. Ladislav Prskavec - Teststack conference, 4. 6. 2020 42
43. How make retries import http_helper "github.com/gruntwork-io/terratest/modules/http-helper" retries := 30 sleepBetweenRetries := 5*time.Second http_helper.HttpGetWithRetry(t, url, nil, 200, "Hello world!", retries, sleepBetweenRetries) Ladislav Prskavec - Teststack conference, 4. 6. 2020 43
44. How make retries with curl opts := &docker.RunOptions{ Command: []string{"--retry", "5", "--retry-connrefused", "-s", "http://production_nginx:80/hello"}, OtherOptions: []string{"--network", "testdockercomposewithstageslocal_teststack-network"}, } tag := "appropriate/curl" output := docker.Run(t, tag, opts) Ladislav Prskavec - Teststack conference, 4. 6. 2020 44
45. Example Hello World with K8S Ladislav Prskavec - Teststack conference, 4. 6. 2020 45
46. apiVersion: apps/v1 kind: Deployment metadata: name: hello-world-deployment spec: selector: matchLabels: app: hello-world replicas: 1 template: metadata: labels: app: hello-world spec: containers: - name: hello-world image: training/webapp:latest ports: - containerPort: 5000 Ladislav Prskavec - Teststack conference, 4. 6. 2020 46
47. kind: Service apiVersion: v1 metadata: name: hello-world-service spec: selector: app: hello-world ports: - protocol: TCP targetPort: 5000 port: 5000 type: LoadBalancer Ladislav Prskavec - Teststack conference, 4. 6. 2020 47
48. package test import (...) func TestKubernetesHelloWorldExample(t *testing.T) { kubeResourcePath := "../hello-world-kubernetes/deployment.yaml" // unique namespace namespaceName := fmt.Sprintf("test-%s", strings.ToLower(random.UniqueId())) options := k8s.NewKubectlOptions("", "", namespaceName) k8s.CreateNamespace(t, options, namespaceName) defer k8s.DeleteNamespace(t, options, namespaceName) defer k8s.KubectlDelete(t, options, kubeResourcePath) k8s.KubectlApply(t, options, kubeResourcePath) k8s.WaitUntilServiceAvailable(t, options, "hello-world-service", 10, 3*time.Second) service := k8s.GetService(t, options, "hello-world-service") url := fmt.Sprintf("http://%s", k8s.GetServiceEndpoint(t, options, service, 5000)) http_helper.HttpGetWithRetry(t, url, nil, 200, "Hello world!", 30, 5*time.Second) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 48
49. Ladislav Prskavec - Teststack conference, 4. 6. 2020 49
50. Example Hello World with docker-compose and stages Ladislav Prskavec - Teststack conference, 4. 6. 2020 50
51. 1. Build docker images for nginx and webapp 2. Run docker-compose 3. Validate nginx and webapp 4. Destroy docker-compose Ladislav Prskavec - Teststack conference, 4. 6. 2020 51
52. version: "3" services: nginx: image: local/nginx:latest container_name: "production_nginx" networks: - "teststack-network" ports: - "80:80" go-webapp: image: go-webapp:latest container_name: "production_go-webapp" environment: - SERVER_TEXT=${SERVER_TEXT} expose: - "8080" networks: - "teststack-network" networks: "teststack-network": Ladislav Prskavec - Teststack conference, 4. 6. 2020 52
53. package main import (...) func helloHandler(res http.ResponseWriter, req *http.Request) { res.Header().Set( "Content-Type", "text/plain", ) io.WriteString( res, fmt.Sprintf("%s", os.Getenv("SERVER_TEXT")), ) } func defaultHandler(w http.ResponseWriter, r *http.Request) { fmt.Fprintf(w, "Go web app powered by Docker") } func main() { http.HandleFunc("/", defaultHandler) http.HandleFunc("/hello", helloHandler) err := http.ListenAndServe(":8080", nil) if err != nil { log.Fatal("ListenAndServe: ", err) return } } Ladislav Prskavec - Teststack conference, 4. 6. 2020 53
54. func TestDockerComposeLocal(t *testing.T) { tag := "go-webapp" buildargs := "-t local/nginx" buildOptions := &docker.BuildOptions{ Tags: []string{tag}, BuildArgs: []string{buildargs} } docker.Build(t, "../hello-world-docker-compose", buildOptions) tag := "local/nginx" buildargs := "-f Dockerfile.nginx" buildOptions := &docker.BuildOptions{ Tags: []string{tag}, BuildArgs: []string{buildargs} } docker.Build(t, "../hello-world-docker-compose", buildOptions) Ladislav Prskavec - Teststack conference, 4. 6. 2020 54
55. serverPort := 80 randomSuffix := random.UniqueId() expectedServerText := fmt.Sprintf("Hello, %s!", randomSuffix) dockerOptions := &docker.Options{ WorkingDir: "../hello-world-docker-compose", EnvVars: map[string]string{ "SERVER_TEXT": expectedServerText, "SERVER_PORT": strconv.Itoa(serverPort), "randomSuffix": randomSuffix, }, } defer docker.RunDockerCompose(t, dockerOptions, "down") docker.RunDockerCompose(t, dockerOptions, "up", "-d") maxRetries := 10 timeBetweenRetries := 5 * time.Second url := fmt.Sprintf("http://localhost:%d/hello", serverPort) tlsConfig := tls.Config{} http_helper.HttpGetWithRetry(t, url, &tlsConfig, 200, expectedServerText, maxRetries, timeBetweenRetries) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 55
56. Ladislav Prskavec - Teststack conference, 4. 6. 2020 56
57. func TestDockerComposeWithStagesLocal(t *testing.T) { workingDir := "../hello-world-docker-compose" test_structure.RunTestStage(t, "build_docker_image", func() { buildImage(t, "go-webapp", "../demowebapp") buildImage(t, "local/nginx", "../nginx") }) test_structure.RunTestStage(t, "run_docker_compose", func() { runCompose(t, workingDir) }) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 57
58. func buildImage(t *testing.T, tag string, workingDir string) { buildOptions := &docker.BuildOptions{ Tags: []string{tag}, } docker.Build(t, workingDir, buildOptions) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 58
59. func runCompose(t *testing.T, workingDir string) { serverPort := 80 randomSuffix := random.UniqueId() expectedServerText := fmt.Sprintf("Hello, %s!", randomSuffix) dockerOptions := &docker.Options{ WorkingDir: workingDir, EnvVars: map[string]string{ "SERVER_TEXT": expectedServerText, "randomSuffix": randomSuffix, "SERVER_PORT": strconv.Itoa(serverPort), }, } defer docker.RunDockerCompose(t, dockerOptions, "down") docker.RunDockerCompose(t, dockerOptions, "up", "-d") opts := &docker.RunOptions{ Command: []string{"--retry", "5", "--retry-connrefused", "-s", "http://production_nginx:80/hello"}, OtherOptions: []string{"--network", "testdockercomposewithstageslocal_teststack-network"}, } tag := "appropriate/curl" output := docker.Run(t, tag, opts) assert.Equal(t, expectedServerText, output) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 59
60. Ladislav Prskavec - Teststack conference, 4. 6. 2020 60
61. For iteration skip build stage Ladislav Prskavec - Teststack conference, 4. 6. 2020 61
62. Ladislav Prskavec - Teststack conference, 4. 6. 2020 62
63. E2E Tests Ladislav Prskavec - Teststack conference, 4. 6. 2020 63
64. Can we use this test strategy? 1. Deploy all the infrastruture 2. Validate it works 3. Undeploy all the infrastruture Ladislav Prskavec - Teststack conference, 4. 6. 2020 64
65. Test type Number of resources Chance of failure Unit test 10 1% Integration test 50 5% E2E test 500+ 40%+ For unit & integration you improve reliability with retries. Ladislav Prskavec - Teststack conference, 4. 6. 2020 65
66. Please don't write E2E tests in this way. Ladislav Prskavec - Teststack conference, 4. 6. 2020 66
67. Incremental testing Ladislav Prskavec - Teststack conference, 4. 6. 2020 67
68. Incremental testing 1. Deploy a persistent test environment and leave it running. 2. Each time you update a module, deploy & validate just that module. 3. Test your deployment process is zero-downtime too. Ladislav Prskavec - Teststack conference, 4. 6. 2020 68
69. Summary 4 Static Analysis (1-60 seconds) 4 Unit Tests (1-20 minutes) 4 Integration Tests (5-60 minutes) 4 E2E Tests (60-240+ minutes) Ladislav Prskavec - Teststack conference, 4. 6. 2020 69
70. What Good Worse Static Analysis Very Fast (until 1 min), Stable, Easy to use, No deploy Limited errors for catch, no reals checks with infrastructure Unit tests Fast (1-20 min), very good stability with retries, trust in components Need real deploy, non trivial test code, extra work for CI Integration tests Mostly stable with retries, High level confidence in multiple units Slow (5-60 min), Need real deploy, non trivial test code, extra work for CI E2E tests Build confidence in entire architecture Very slow (60-240+ min), Need real deploy, Still problem 0- deployments, Can be brittle (even with retries) Ladislav Prskavec - Teststack conference, 4. 6. 2020 70
71. Try make at least one test for your docker environment! Ladislav Prskavec - Teststack conference, 4. 6. 2020 71
72. Infrastructure code without tests is scary Ladislav Prskavec - Teststack conference, 4. 6. 2020 72
73. Q & A #test-infrastructure on Teststack Slack Ladislav Prskavec - Teststack conference, 4. 6. 2020 73