Testing Infrastructure in Production with Terraform

Originally presented as a WWCode Cloud webinar.

Join us to hear about approaches to feature toggling, blue-green deployment, and canary testing production infrastructure. The application of these approaches differs based on the infrastructure resource and its upstream dependencies.

Transcript

1. Copyright © 2019 HashiCorp Test Infrastructure in Production with Terraform WWCode Cloud | Dec. 18, 2019 Rosemary Wang | @joatmon08
2. Network engineers live dangerously.
3. My users = developers ▪ Push and deliver code at any time ▪ Availability of application depends on system
4. How do we change infrastructure without impacting applications?
5. Infrastructure-as-Code* * Not really.
6. Let’s talk software development.
7. How do we work? ▪ Feature release vs. Continuous Delivery ▪ Feature branching vs. Trunk-based ▪ Mono- vs. Multi-Repository
8. Approaches ▪ Shift-left testing (e.g., staging) ▪ Feature Toggles ▪ Canary Testing ▪ A/B Testing
9. Shift-Left Testing ▪ Test before production ▪ Assess change impact ▪ Can apply Test-Driven Development
10. Integration Tests Contract Tests Unit Tests “Ideal” Testing Pyramid Manual Testing Cost (Time, $$$) End-to-End Tests
11. Unit / Contract Testing CODE EDITOR contains_variables(variables) { variables[_].vpc_cidr[0].value = "10.128.0.0/25" variables[_].region[0].value = "eu-central-1" variables[_].owner[0] } deny[msg] { not contains_variables(input.variables) msg = "Variables are not populated with expected values" }
13. Challenges ▪ Cost of lower environments ▪ Imperfect indicator – Net new each time? – Dependencies?
14. Feature Toggles ▪ Preserve state, if possible ▪ Inject with roll forward mindset ▪ Don’t write toggles at the start
15. Feature Toggles CODE EDITOR resource "aws_instance" "example_bionic" { count = var.enable_new_ami ? 1 : 0 instance_type = "t2.micro" ami = data.aws_ami.ubuntu_bionic.id tags = { Terraform = "true" Owner = var.owner Has_Toggle = var.enable_new_ami } }
16. Canary Testing ▪ Smoke test before release ▪ Easier with container architectures – e.g., VM images for Kubernetes worker nodes
17. Canary Testing CODE EDITOR resource "aws_instance" "canary" { count = var.enable_new_network ? 1 : 0 instance_type = "t2.micro" ami = data.aws_ami.ubuntu.id vpc_security_group_ids = [aws_security_group.instances _green.id] subnet_id = aws_subnet.public_green.id tags = { Name = "${var.prefix}-canary" Owner = var.owner } }
18. VPC (blue) 10.128.0.0/24 VPC (green) 10.128.0.0/28 APP APP APP APP KITCHEN INSTANCE APP APP CANARY CAN I CONNECT?
19. Kubernetes Control Plane Kubernetes Node Group (Insecure OS) Kubernetes Node Group (Secure OS) INTERNAL EXTERNAL EXTERNAL EXTERNAL EXTERNAL INTERNAL INTERNAL kubectl taint nodes external=true:NoExecute
20. A/B Testing ▪ Infrastructure that affect upstream Service Level Objectives ▪ Hypotheses: –Does X batch process more quickly than Y? –Does X cost more than Y?
21. Kafka FaaS “Data Lake” versus Kafka Spark “Data Lake” APPLICATION APPLICATION APPLICATION APPLICATION Does Spark + Kafka architecture process faster with lower cost?
22. Regular VM versus Network Optimized VM Does a new instance reduce latency? APPLICATION APPLICATION APPLICATION APPLICATION
23. Conclusions ▪ Test in production organizes infrastructure blast radius ▪ Risk mitigation over risk aversion ▪ “Infrastructure-as-Code” is heuristic
24. Resources ▪ learn.hashicorp.com/terraform ▪ hashicorp.com/blog/terraform-feature-toggles-blue-green- deployments-canary-test ▪ hashicorp.com/resources/test-driven-development-tdd-for- infrastructure ▪ discuss.hashicorp.com ▪ app.terraform.io
25. joatmon08.github.io Rosemary Wang (she/her) Developer Advocate at HashiCorp @joatmon08 joatmon08 linkedin.com/in/rosemarywang/ 25