Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Testing Infrastructure in Production with Terraform

Be8b596c46f4c9a1aec6a7586af33134?s=47 Rosemary Wang
December 18, 2019

Testing Infrastructure in Production with Terraform

Originally presented as a WWCode Cloud webinar.

Join us to hear about approaches to feature toggling, blue-green deployment, and canary testing production infrastructure. The application of these approaches differs based on the infrastructure resource and its upstream dependencies.

Be8b596c46f4c9a1aec6a7586af33134?s=128

Rosemary Wang

December 18, 2019
Tweet

More Decks by Rosemary Wang

Other Decks in Technology

Transcript

  1. Copyright © 2019 HashiCorp Test Infrastructure in Production with Terraform

    WWCode Cloud | Dec. 18, 2019 Rosemary Wang | @joatmon08
  2. Network engineers live dangerously.

  3. My users = developers ▪ Push and deliver code at

    any time ▪ Availability of application depends on system
  4. How do we change infrastructure without impacting applications?

  5. Infrastructure-as-Code* * Not really.

  6. Let’s talk software development.

  7. How do we work? ▪ Feature release vs. Continuous Delivery

    ▪ Feature branching vs. Trunk-based ▪ Mono- vs. Multi-Repository
  8. Approaches ▪ Shift-left testing (e.g., staging) ▪ Feature Toggles ▪

    Canary Testing ▪ A/B Testing
  9. Shift-Left Testing ▪ Test before production ▪ Assess change impact

    ▪ Can apply Test-Driven Development
  10. Integration Tests Contract Tests Unit Tests “Ideal” Testing Pyramid Manual

    Testing Cost (Time, $$$) End-to-End Tests
  11. Unit / Contract Testing CODE EDITOR contains_variables(variables) { variables[_].vpc_cidr[0].value =

    "10.128.0.0/25" variables[_].region[0].value = "eu-central-1" variables[_].owner[0] } deny[msg] { not contains_variables(input.variables) msg = "Variables are not populated with expected values" }
  12. 12

  13. Challenges ▪ Cost of lower environments ▪ Imperfect indicator –

    Net new each time? – Dependencies?
  14. Feature Toggles ▪ Preserve state, if possible ▪ Inject with

    roll forward mindset ▪ Don’t write toggles at the start
  15. Feature Toggles CODE EDITOR resource "aws_instance" "example_bionic" { count =

    var.enable_new_ami ? 1 : 0 instance_type = "t2.micro" ami = data.aws_ami.ubuntu_bionic.id tags = { Terraform = "true" Owner = var.owner Has_Toggle = var.enable_new_ami } }
  16. Canary Testing ▪ Smoke test before release ▪ Easier with

    container architectures – e.g., VM images for Kubernetes worker nodes
  17. Canary Testing CODE EDITOR resource "aws_instance" "canary" { count =

    var.enable_new_network ? 1 : 0 instance_type = "t2.micro" ami = data.aws_ami.ubuntu.id vpc_security_group_ids = [aws_security_group.instances _green.id] subnet_id = aws_subnet.public_green.id tags = { Name = "${var.prefix}-canary" Owner = var.owner } }
  18. VPC (blue) 10.128.0.0/24 VPC (green) 10.128.0.0/28 APP APP APP APP

    KITCHEN INSTANCE APP APP CANARY CAN I CONNECT?
  19. Kubernetes Control Plane Kubernetes Node Group (Insecure OS) Kubernetes Node

    Group (Secure OS) INTERNAL EXTERNAL EXTERNAL EXTERNAL EXTERNAL INTERNAL INTERNAL kubectl taint nodes external=true:NoExecute
  20. A/B Testing ▪ Infrastructure that affect upstream Service Level Objectives

    ▪ Hypotheses: –Does X batch process more quickly than Y? –Does X cost more than Y?
  21. Kafka FaaS “Data Lake” versus Kafka Spark “Data Lake” APPLICATION

    APPLICATION APPLICATION APPLICATION Does Spark + Kafka architecture process faster with lower cost?
  22. Regular VM versus Network Optimized VM Does a new instance

    reduce latency? APPLICATION APPLICATION APPLICATION APPLICATION
  23. Conclusions ▪ Test in production organizes infrastructure blast radius ▪

    Risk mitigation over risk aversion ▪ “Infrastructure-as-Code” is heuristic
  24. Resources ▪ learn.hashicorp.com/terraform ▪ hashicorp.com/blog/terraform-feature-toggles-blue-green- deployments-canary-test ▪ hashicorp.com/resources/test-driven-development-tdd-for- infrastructure ▪

    discuss.hashicorp.com ▪ app.terraform.io
  25. joatmon08.github.io Rosemary Wang (she/her) Developer Advocate at HashiCorp @joatmon08 joatmon08

    linkedin.com/in/rosemarywang/ 25