Best practices for testing AWS infrastructure.

My take on Yevgeniy "Jim" Brikman's talk entitled, "How to test infrastructure code: automated testing for Terraform, Kubernetes, Docker, Packer and more," presented at AWS re:Invent 2019.

See Jim's original slide deck with more content here: https://www.slideshare.net/brikis98/how-to-test-infrastructure-code-automated-testing-for-terraform-kubernetes-docker-packer-and-more

1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Best practices for testing AWS infrastructure Ben Whaley D V C 0 4 Principal Software Engineer Gruntwork.io
2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Slides by Yevgeniy Brikman Co-founder, Gruntwork.io @brikis98
3. The world of software is full of Fear
4. Fear of downtime
5. Fear of security breaches
6. Fear of data loss
7. Fear of change
8. Teams deal with this fear in two ways:
9. 1) Live in a perpetual state of anxiety
10. 2) Deploy code less frequently
11. Sadly, both of these just make the problem worse!
12. There is a better way to deal with the fear:
13. Automated tests
14. Automated tests reduce anxiety and give you confidence
15. We know how to write automated tests for application code
16. resource "aws_lambda_function" "web_app" { function_name = var.name role = aws_iam_role.lambda.arn # ... } resource "aws_api_gateway_integration" "proxy" { type = "AWS_PROXY" uri = aws_lambda_function.web_app.invoke_arn # ... } But how do you test infrastructure provisioned with Terraform?
17. export class DemoCdkStack extends cdk.Stack { constructor(scope: cdk.Construct, id: string) { super(scope, id); new s3.Bucket(this, 'reInventDemoBucket’, { versioned: true, }); } } const app = new cdk.App(); new DemoCdkStack(app, "DemoCdkStack"); How do you test infrastructure provisioned with the AWS CDK?
18. This talk is about how to test infrastructure code. Automated testing techniques: ✓ terraform ✓ cdk ✓ unit tests ✓ integration tests ✓ end-to-end tests Passed: 5. Failed: 0. Skipped: 0. Test run successful.
19. Agenda 1. Static analysis 2. Unit tests 3. Integration tests 4. End-to-end tests 5. Conclusion
20. Static analysis: test your code without deploying it
21. Static analysis 1. Compiler / parser / interpreter 2. Linter 3. Dry run 4. Example: CDK testing
22. Compiler: Check your code for syntactic or structural issues
23. Compiler / parser / interpreter check terraform terraform validate cdk npm run build kubernetes kubectl apply -f <file> --dry-run --validate=true cloudformation aws cloudformation validate-template
24. Static analysis 1. Compiler / parser / interpreter 2. Linter 3. Dry run 4. Example: CDK testing
25. Linter: Validate your code to catch common errors
26. Linters terraform • conftest • terraform_validate • tflint cdk cdk doctor kubernetes • kube-score • kube-lint • yamllint cloudformation • cfn-python-lint • cfn-nag
27. resource "aws_instance" "foo" { ami = "ami-0ff8a91507f77f867" instance_type = “t2.2xlarge” } t2.2xlarge is an invalid type!
28. $ tflint Error: instance_type is not a valid value on main.tf line 3: 3: instance_type = “t2.2xlarge” Linters find this type of problem in advance
29. Static analysis 1. Compiler / parser / interpreter 2. Linter 3. Dry run 4. CDK testing
30. Partially execute to check for errors, but don’t fully deploy
31. Dry run terraform terraform plan cdk cdk diff Snapshots kubernetes kubectl apply -f <file> --server-dry-run cloudformation Change Sets
32. Static analysis 1. Compiler / parser / interpreter 2. Linter 3. Dry run 4. CDK testing
33. Sample code for this talk is at: github.com/gruntwork-io/infrastructure-as-code-testing-talk
34. An example of a CDK app that you might want to test:
35. infrastructure-as-code-testing-talk └ examples └ modules └ cdk-app └ bin └ lib └ cdk-demo-stack.ts └ test cdk-app: Create an S3 bucket
36. export class DemoCdkStack extends cdk.Stack { constructor(scope: cdk.Construct, id: string) { super(scope, id); new s3.Bucket(this, 'reInventDemoBucket’, { versioned: true, }); } } const app = new cdk.App(); new DemoCdkStack(app, "DemoCdkStack"); Create an S3 bucket with versioning enabled
37. $ cdk deploy DemoCdkStack: deploying... DemoCdkStack: creating CloudFormation changeset... 0/3 | 1:04:13 PM | CREATE_IN_PROGRESS … [snip] … 3/3 | 1:04:37 PM | CREATE_COMPLETE ✅ DemoCdkStack Stack ARN: arn:aws:cloudformation:us-west-2:…[snip]… When you run cdk deploy, it creates the CloudFormation stack and outputs the stack ARN
38. Let’s create a test for our CDK app
39. infrastructure-as-code-testing-talk └ examples └ modules └ cdk-app └ bin └ lib └ test └ cdk-demo.test.ts Create cdk-demo.test.ts
40. test('Bucket Stack', () => { const app = new cdk.App(); const stack = new CdkDemo.DemoCdkStack(app, 'DemoTestStack’); expectCDK(stack).to(haveResource('AWS::S3::Bucket’, { VersioningConfiguration: { Status: "Enabled" }, })); }); The basic test structure
41. test('Bucket Stack', () => { const app = new cdk.App(); const stack = new CdkDemo.DemoCdkStack(app, 'DemoTestStack’); expectCDK(stack).to(haveResource('AWS::S3::Bucket’, { VersioningConfiguration: { Status: "Enabled" }, })); }); 1) CDK uses the Jest testing framework
42. test('Bucket Stack', () => { const app = new cdk.App(); const stack = new CdkDemo.DemoCdkStack(app, 'DemoTestStack’); expectCDK(stack).to(haveResource('AWS::S3::Bucket’, { VersioningConfiguration: { Status: "Enabled" }, })); }); 2) Set up the app and stack
43. test('Bucket Stack', () => { const app = new cdk.App(); const stack = new CdkDemo.DemoCdkStack(app, 'DemoTestStack’); expectCDK(stack).to(haveResource('AWS::S3::Bucket’, { VersioningConfiguration: { Status: "Enabled" }, })); }); 3) Assert that the stack has an S3 bucket with versioning enabled
44. $ npm run test PASS test/cdk-demo.test.ts ✓ Bucket Stack (27ms) Test Suites: 1 passed, 1 total Tests: 1 passed, 1 total Snapshots: 0 total Time: 2.391s Ran all test suites. Run npm test. You now have a unit test you can run after every commit!
45. export class DemoCdkStack extends cdk.Stack { constructor(scope: cdk.Construct, id: string) { super(scope, id); //new s3.Bucket(this, 'reInventDemoBucket’, { // versioned: true, //}); new s3.Bucket(this, 'reInventDemoBucket’); } } What if somebody makes a change to the bucket, like removing the versioning?
46. $ npm run test FAIL test/cdk-demo.test.ts ✕ Bucket Stack (29ms) None of 1 resources matches resource 'AWS::S3::Bucket' with properties { "VersioningConfiguration": { "Status": "Enabled" } } All of the properties of the resource must match
47. *CDK says this is a unit test, but from an infra standpoint, it’s actually a static test
48. Agenda 1. Static analysis 2. Unit tests 3. Integration tests 4. End-to-end tests 5. Conclusion
49. Unit tests: test that an individual unit works in isolation
50. Unit tests 1. Unit testing basics 2. Example: Terraform unit tests 3. Cleaning up after tests
51. You can’t “unit test” an entire end-to-end architecture
52. Instead, break your infra code into small modules and unit test those! module module module module module module module module module module module module module module module
53. With app code, you can test units in isolation from the outside world
54. resource "aws_lambda_function" "web_app" { function_name = var.name role = aws_iam_role.lambda.arn # ... } resource "aws_api_gateway_integration" "proxy" { type = "AWS_PROXY" uri = aws_lambda_function.web_app.invoke_arn # ... } But infrastructure code is all about talking to the outside world!
55. You can only truly test infra code by deploying to a real environment
56. Key takeaway: there’s no pure unit testing for infrastructure code.
57. Therefore, the test strategy is: 1. Deploy real infrastructure 2. Validate it works (e.g., via API calls, SSH commands, etc.) 3. Destroy the infrastructure (So it’s really integration testing of a single unit!)
58. Tools that help with infrastructure “unit” testing: Deploy/Destroy Validate Supported Technologies Terratest Yes Yes Terraform, k8s, packer, docker, OS, Cloud APIs kitchen-terraform Yes Yes Terraform Inspec No Yes OS, Cloud APIs awspec No Yes AWS API
59. Unit tests 1. Unit testing basics 2. Example: Terraform unit tests 3. Cleaning up after tests
60. An example of a Terraform module that you might want to test:
61. infrastructure-as-code-testing-talk └ examples └ hello-world-app └ main.tf └ outputs.tf └ variables.tf └ modules └ test └ README.md hello-world-app: deploy a “Hello, World” web service
62. resource "aws_lambda_function" "web_app" { function_name = var.name role = aws_iam_role.lambda.arn # ... } resource "aws_api_gateway_integration" "proxy" { type = "AWS_PROXY" uri = aws_lambda_function.web_app.invoke_arn # ... } Under the hood, this example runs on top of AWS Lambda & API Gateway
63. $ terraform apply Outputs: url = ruvvwv3sh1.execute-api.us-east-2.amazonaws.com $ curl ruvvwv3sh1.execute-api.us-east-2.amazonaws.com Hello, World! When you run terraform apply, it deploys and outputs the URL
64. Let’s write a unit test for hello-world-app with Terratest
65. infrastructure-as-code-testing-talk └ examples └ modules └ test └ hello_world_app_test.go └ README.md Create hello_world_app_test.go
66. func TestHelloWorldAppUnit(t *testing.T) { terraformOptions := &terraform.Options{ TerraformDir: "../examples/hello-world-app", } defer terraform.Destroy(t, terraformOptions) terraform.InitAndApply(t, terraformOptions) validate(t, terraformOptions) } The basic test structure
67. func TestHelloWorldAppUnit(t *testing.T) { terraformOptions := &terraform.Options{ TerraformDir: "../examples/hello-world-app", } defer terraform.Destroy(t, terraformOptions) terraform.InitAndApply(t, terraformOptions) validate(t, terraformOptions) } 1. Tell Terratest where your Terraform code lives
68. func TestHelloWorldAppUnit(t *testing.T) { terraformOptions := &terraform.Options{ TerraformDir: "../examples/hello-world-app", } defer terraform.Destroy(t, terraformOptions) terraform.InitAndApply(t, terraformOptions) validate(t, terraformOptions) } 2. Run terraform init and terraform apply to deploy your module
69. func TestHelloWorldAppUnit(t *testing.T) { terraformOptions := &terraform.Options{ TerraformDir: "../examples/hello-world-app", } defer terraform.Destroy(t, terraformOptions) terraform.InitAndApply(t, terraformOptions) validate(t, terraformOptions) } 3. Validate the infrastructure works. We’ll come back to this shortly.
70. func TestHelloWorldAppUnit(t *testing.T) { terraformOptions := &terraform.Options{ TerraformDir: "../examples/hello-world-app", } defer terraform.Destroy(t, terraformOptions) terraform.InitAndApply(t, terraformOptions) validate(t, terraformOptions) } 4. Run terraform destroy at the end of the test to clean up
71. func validate(t *testing.T, opts *terraform.Options) { url := terraform.Output(t, opts, "url") http_helper.HttpGetWithRetry(t, url, // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3 * time.Second // Time between retries ) } The validate function
72. func validate(t *testing.T, opts *terraform.Options) { url := terraform.Output(t, opts, "url") http_helper.HttpGetWithRetry(t, url, // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3 * time.Second // Time between retries ) } 1. Run terraform output to get the web service URL
73. func validate(t *testing.T, opts *terraform.Options) { url := terraform.Output(t, opts, "url") http_helper.HttpGetWithRetry(t, url, // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3 * time.Second // Time between retries ) } 2. Make HTTP requests to the URL
74. func validate(t *testing.T, opts *terraform.Options) { url := terraform.Output(t, opts, "url") http_helper.HttpGetWithRetry(t, url, // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3 * time.Second // Time between retries ) } 3. Check the response for an expected status and body
75. func validate(t *testing.T, opts *terraform.Options) { url := terraform.Output(t, opts, "url") http_helper.HttpGetWithRetry(t, url, // URL to test 200, // Expected status code "Hello, World!", // Expected body 10, // Max retries 3 * time.Second // Time between retries ) } 4. Retry the request up to 10 times, as deployment is asynchronous
76. $ export AWS_ACCESS_KEY_ID=xxxx $ export AWS_SECRET_ACCESS_KEY=xxxxx To run the test, first authenticate to AWS
77. $ go test -v -timeout 15m -run TestHelloWorldAppUnit … --- PASS: TestHelloWorldAppUnit (31.57s) Then run go test. You now have a unit test you can run after every commit!
78. Since we’re testing a web service, we use HTTP requests to validate it
79. Examples of other ways to validate: Example Validation Helper Web service Containerized or serverless web app HTTP requests Terratest http_helper package Server EC2 instance SSH commands Terratest ssh package Cloud resource SQS queue API calls Terratest aws package Database MySQL SQL queries MySQL driver for Go
80. Unit tests 1. Unit testing basics 2. Example: Terraform unit tests 3. Cleaning up after tests
81. Tests create and destroy many resources!
82. Pro tip: run tests in isolated “sandbox” accounts
83. Features cloud-nuke Delete resources older than a certain date; in a certain region; of a certain type. Janitor Monkey Configurable rules of what to delete. Notify owners of pending deletions. aws-nuke Specify specific AWS accounts and resource types to target. Pro tip #2: run these tools in cron jobs to clean up stale resources
84. Agenda 1. Static analysis 2. Unit tests 3. Integration tests 4. End-to-end tests 5. Conclusion
85. Integration tests: test that multiple “units” work together.
86. Integration tests 1. Example: Terraform integration tests 2. Test parallelism
87. infrastructure-as-code-testing-talk └ examples └ hello-world-app └ docker-kubernetes └ proxy-app └ web-service └ modules └ test └ README.md Let’s say you have two Terraform modules you want to test together:
88. infrastructure-as-code-testing-talk └ examples └ hello-world-app └ docker-kubernetes └ proxy-app └ web-service └ modules └ test └ README.md proxy-app: an app that acts as an HTTP proxy for other web services.
89. infrastructure-as-code-testing-talk └ examples └ hello-world-app └ docker-kubernetes └ proxy-app └ web-service └ modules └ test └ README.md web-service: a web service that you want proxied.
90. variable "url_to_proxy" { description = "The URL to proxy." type = string } proxy-app takes in the URL to proxy via an input variable
91. output "url" { value = module.web_service.url } web-service exposes its URL via an output variable
92. infrastructure-as-code-testing-talk └ examples └ modules └ test └ hello_world_app_test.go └ proxy_app_test.go └ README.md Create proxy_app_test.go
93. func TestProxyApp(t *testing.T) { webServiceOpts := configWebService(t) defer terraform.Destroy(t, webServiceOpts) terraform.InitAndApply(t, webServiceOpts) proxyAppOpts := configProxyApp(t, webServiceOpts) defer terraform.Destroy(t, proxyAppOpts) terraform.InitAndApply(t, proxyAppOpts) validate(t, proxyAppOpts) } The basic test structure
94. func TestProxyApp(t *testing.T) { webServiceOpts := configWebService(t) defer terraform.Destroy(t, webServiceOpts) terraform.InitAndApply(t, webServiceOpts) proxyAppOpts := configProxyApp(t, webServiceOpts) defer terraform.Destroy(t, proxyAppOpts) terraform.InitAndApply(t, proxyAppOpts) validate(t, proxyAppOpts) } 1. Configure options for the web service
95. func TestProxyApp(t *testing.T) { webServiceOpts := configWebService(t) defer terraform.Destroy(t, webServiceOpts) terraform.InitAndApply(t, webServiceOpts) proxyAppOpts := configProxyApp(t, webServiceOpts) defer terraform.Destroy(t, proxyAppOpts) terraform.InitAndApply(t, proxyAppOpts) validate(t, proxyAppOpts) } 2. Deploy the web service
96. func TestProxyApp(t *testing.T) { webServiceOpts := configWebService(t) defer terraform.Destroy(t, webServiceOpts) terraform.InitAndApply(t, webServiceOpts) proxyAppOpts := configProxyApp(t, webServiceOpts) defer terraform.Destroy(t, proxyAppOpts) terraform.InitAndApply(t, proxyAppOpts) validate(t, proxyAppOpts) } 3. Configure options for the proxy app (passing it the web service options)
97. func TestProxyApp(t *testing.T) { webServiceOpts := configWebService(t) defer terraform.Destroy(t, webServiceOpts) terraform.InitAndApply(t, webServiceOpts) proxyAppOpts := configProxyApp(t, webServiceOpts) defer terraform.Destroy(t, proxyAppOpts) terraform.InitAndApply(t, proxyAppOpts) validate(t, proxyAppOpts) } 4. Deploy the proxy app
98. func TestProxyApp(t *testing.T) { webServiceOpts := configWebService(t) defer terraform.Destroy(t, webServiceOpts) terraform.InitAndApply(t, webServiceOpts) proxyAppOpts := configProxyApp(t, webServiceOpts) defer terraform.Destroy(t, proxyAppOpts) terraform.InitAndApply(t, proxyAppOpts) validate(t, proxyAppOpts) } 5. Validate the proxy app works
99. func TestProxyApp(t *testing.T) { webServiceOpts := configWebService(t) defer terraform.Destroy(t, webServiceOpts) terraform.InitAndApply(t, webServiceOpts) proxyAppOpts := configProxyApp(t, webServiceOpts) defer terraform.Destroy(t, proxyAppOpts) terraform.InitAndApply(t, proxyAppOpts) validate(t, proxyAppOpts) } 6. At the end of the test, destroy the proxy app and the web service
100. func configWebService(t *testing.T) *terraform.Options { return &terraform.Options{ TerraformDir: "../examples/web-service", } } The configWebService method
101. func configProxyApp(t *testing.T, webServiceOpts *terraform.Options) *terraform.Options { url := terraform.Output(t, webServiceOpts, "url") return &terraform.Options{ TerraformDir: "../examples/proxy-app", Vars: map[string]interface{}{ "url_to_proxy": url, }, } } The configProxyApp method
102. func configProxyApp(t *testing.T, webServiceOpts *terraform.Options) *terraform.Options { url := terraform.Output(t, webServiceOpts, "url") return &terraform.Options{ TerraformDir: "../examples/proxy-app", Vars: map[string]interface{}{ "url_to_proxy": url, }, } } 1. Read the url output from the web- service module
103. func configProxyApp(t *testing.T, webServiceOpts *terraform.Options) *terraform.Options { url := terraform.Output(t, webServiceOpts, "url") return &terraform.Options{ TerraformDir: "../examples/proxy-app", Vars: map[string]interface{}{ "url_to_proxy": url, }, } } 2. Pass it in as the url_to_proxy input to the proxy-app module
104. func validate(t *testing.T, opts *terraform.Options) { url := terraform.Output(t, opts, "url") http_helper.HttpGetWithRetry(t, url, // URL to test 200, // Expected status code `{"text":"Hello, World!"}`, // Expected body 10, // Max retries 3 * time.Second // Time between retries ) } The validate method
105. $ go test -v -timeout 15m -run TestProxyApp … --- PASS: TestProxyApp (182.44s) Run go test. You’re now testing multiple modules together!
106. $ go test -v -timeout 15m -run TestProxyApp … --- PASS: TestProxyApp (182.44s) But integration tests can take (many) minutes to run…
107. Integration tests 1. Example: Terraform integration tests 2. Test parallelism
108. Infrastructure tests can take a long time to run
109. One way to save time: run tests in parallel
110. func TestProxyApp(t *testing.T) { t.Parallel() // The rest of the test code } func TestHelloWorldAppUnit(t *testing.T) { t.Parallel() // The rest of the test code } Enable test parallelism in Go by adding t.Parallel() as the 1st line of each test.
111. $ go test -v -timeout 15m === RUN TestHelloWorldApp === RUN TestProxyApp Now, if you run go test, all the tests with t.Parallel() will run in parallel
112. But there’s a gotcha: resource conflicts
113. resource "aws_iam_role" "role_example" { name = "example-iam-role" } resource "aws_security_group" "sg_example" { name = "security-group-example" } Example: module with hard-coded IAM Role and Security Group names
114. resource "aws_iam_role" "role_example" { name = "example-iam-role" } resource "aws_security_group" "sg_example" { name = "security-group-example" } If two tests tried to deploy this module in parallel, the names would conflict!
115. Key takeaway: you must namespace all your resources
116. resource "aws_iam_role" "role_example" { name = var.name } resource "aws_security_group" "sg_example" { name = var.name } Example: use variables in all resource names…
117. uniqueId := random.UniqueId() return &terraform.Options{ TerraformDir: "../examples/proxy-app", Vars: map[string]interface{}{ "name": "text-proxy-app-” + uniqueId }, } At test time, set the variables to a randomized value to avoid conflicts
118. Agenda 1. Static analysis 2. Unit tests 3. Integration tests 4. End-to-end tests 5. Conclusion
119. End-to-end tests: test your entire infrastructure works together.
120. How do you test this entire thing?
121. You could use the same strategy… 1. Deploy all the infrastructure 2. Validate it works (e.g., via API calls, SSH commands, etc.) 3. Destroy all the infrastructure
122. But it’s rare to write end-to-end tests this way. Here’s why:
123. e2e Tests Test pyramid Integration Tests Unit Tests Static analysis
124. e2e Tests Integration Tests Unit Tests Static analysis Cost, brittleness, run time
125. e2e Tests Integration Tests Unit Tests Static analysis 60 – 240+ minutes 5 – 60 minutes 1 – 20 minutes 1 – 60 seconds
126. e2e Tests Integration Tests Unit Tests Static analysis E2E tests are too slow to be useful 60 – 240+ minutes 5 – 60 minutes 1 – 20 minutes 1 – 60 seconds
127. Another problem with E2E tests: brittleness
128. Let’s do some math:
129. Assume a single resource (e.g., EC2 instance) has a 1/1000 (0.1%) chance of failure.
130. The more resources your tests deploy, the flakier they will be. Type of test # of resources Chance of failure Unit tests 1-10 ~1% Integration 11-100 ~10% End-to-end 400-500 ~40-50%
131. You can work around the failure rate for unit & integration tests with retries Type of test # of resources Chance of failure Unit tests 1-10 ~1% Integration 11-100 ~10% End-to-end 400-500 ~40-50%
132. Key takeaway: E2E tests from scratch are too slow and too brittle to be useful
133. Instead, you can do incremental E2E testing!
134. module module module module module module module module module module module module module module module 1. Deploy a persistent test environment and leave it running.
135. module module module module module module module module module module module module module module module 2. Each time you update a module, deploy & validate just that module
136. Agenda 1. Static analysis 2. Unit tests 3. Integration tests 4. End-to-end tests 5. Conclusion
137. Testing techniques compared:
138. Technique Strengths Weaknesses Static analysis 1.Fast 2.Stable 3.No need to deploy real resources 4.Easy to use 1.Very limited in errors you can catch 2.You don’t get much confidence in your code solely from static analysis Unit tests 1.Fast enough (1 – 10 min) 2.Mostly stable (with retry logic) 3.High level of confidence in individual units 1.Need to deploy real resources 2.Requires writing non-trivial code Integration tests 1.Mostly stable (with retry logic) 2.High level of confidence in multiple units working together 1.Need to deploy real resources 2.Requires writing non-trivial code 3.Slow (10 – 30 min) End-to-end tests 1.Build confidence in your entire architecture 1.Need to deploy real resources 2.Requires writing non-trivial code 3.Very slow (60 min – 240+ min)* 4.Can be brittle (even with retry logic)*
139. So which should you use?
140. All of them! They all catch different types of bugs.
141. e2e Tests Bear in mind the test pyramid Integration Tests Unit Tests Static analysis
142. e2e Tests Lots of unit tests + static analysis Integration Tests Unit Tests Static analysis
143. e2e Tests Fewer integration tests Integration Tests Unit Tests Static analysis
144. e2e Tests A handful of high-value e2e tests Integration Tests Unit Tests Static analysis
145. Infrastructure code without tests is scary
146. Fight the fear & build confidence in your code with automated tests
147. Questions? info@gruntwork.io