Copyright © 2020 HashiCorp June 29, 2022 Lessons Learned from Scaling Infrastructure as Code

We started using __________. (Insert infrastructure as code tool.)

Other teams started using it too. Now we have _____________. (Insert problem here.)

Developer Advocate at HashiCorp 
 she/her 
 
 @joatmon08 joatmon08.github.io Rosemary Wang

Delivery Development Security Cost

01 Development Lessons Learned / Challenges

Problem: Changes break infrastructure.

Empower every team to responsibly contribute to infrastructure.

Standardize infrastructure resources. TERMINAL module "boundary" { depends_on = [module.vpc] source = "joatmon08/boundary/aws" version = "0.2.0" vpc_id = module.vpc.vpc_id vpc_cidr_block = module.vpc.vpc_cidr_block public_subnet_ids = module.vpc.public_subnets private_subnet_ids = module.vpc.database_subnets name = var.name key_pair_name = var.key_pair_name allow_cidr_blocks_to_workers = var.client_cidr_block allow_cidr_blocks_to_api = ["0.0.0.0/0"] boundary_db_password = random_password.boundary_database.result }

Infrastructure Modules 1. Set opinionated defaults. 2. Allow collaboration (pull requests). 3. Apply a testing strategy. 
 unit, contract, & integration tests 4. Version modules.

Infrastructure Configuration 1. Minimize blast radius of changes. 2. Use immutability to roll forward. 
 terraform apply -target, terraform taint 3. Use version control when possible.

Note: If you use a monorepo, make sure your build tool can handle it.

Decouple dependencies. TERMINAL data "aws_eks_cluster" "cluster" { name = var.aws_eks_cluster_id == "" ? data.terraform_remote_state.infrastructure.outputs.eks_cluster_id : var.aws_eks_cluster_i d } data "aws_eks_cluster_auth" "cluster" { name = var.aws_eks_cluster_id == "" ? data.terraform_remote_state.infrastructure.outputs.eks_cluster_id : var.aws_eks_cluster_i d } provider "kubernetes" { host = data.aws_eks_cluster.cluster.endpoin t cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data ) token = data.aws_eks_cluster_auth.cluster.toke n experiments { manifest_resource = tru e } }

Dependency Injection 1. Reference outputs stored in state 2. Use the infrastructure API 3. Store and reference from configuration manager

Note: Multiple infrastructure providers or environments will require additional abstraction or automation.

02 Delivery Lessons Learned / Challenges

Problem: It takes too long to deploy changes.

Minimize time from commit to production.

terraform.io/plugin

Download modules and plugins. github.com/hashicorp/go- getter TERMINAL > terraform ini t Initializing the backend.. . Initializing provider plugins.. . - terraform.io/builtin/terraform is built in to Terrafor m - Reusing previous version of hashicorp/aws from the dependency lock fil e - Installing hashicorp/aws v4.15.0.. . - Installed hashicorp/aws v4.15.0 (signed by HashiCorp ) - Installing hashicorp/boundary v1.0.6.. . - Installed hashicorp/boundary v1.0.6 (signed by HashiCorp )

1. Use internal artifact repository. 2. Cache providers & modules on local filesystem. 
 git submodule add terraform.io/language/providers/requirements#in-house-providers

Refresh state. Reads information from infrastructure API. > terraform appl y module.hcp.data.aws_region.current: Reading.. . module.hcp.data.aws_region.current: Read complete after 0s [id=us-west-2 ] module.vpc.aws_eip.nat[0]: Refreshing state... [id=eipalloc-REDACTED ] ## omitted for clarit y Plan: 105 to add, 0 to change, 0 to destroy . CODE EDITOR

Apply changes. Create, read, update, and delete resources with infrastructure API. > terraform appl y module.hcp.data.aws_region.current: Reading.. . module.hcp.data.aws_region.current: Read complete after 0s [id=us-west-2 ] module.vpc.aws_eip.nat[0]: Refreshing state... [id=eipalloc-REDACTED ] ## omitted for clarit y Plan: 105 to add, 0 to change, 0 to destroy . CODE EDITOR

> terraform graph terraform.io/internals/graph

1. Enable concurrent operations. 
 terraform apply -parallelism=n 2. Tune infrastructure API (rate limiting). 
 create, read, update, delete 3. Modularize into fewer resources. 
 faster refresh & apply, fewer state locking conflicts

Note: Infrastructure usually has a manual approval step.

03 Security Lessons Learned / Challenges

Problem: Misconfiguration of infrastructure could compromise security.

Use infrastructure as code to enforce security.

CODE EDITOR import "tfplan/v2" as tfplan database_only_has_non_permissive_firewall_rules = rule { all database_firewall_rules as firewall_rule { firewall_rule.values.start_ip_address is not "0.0.0.0" and firewall_rule.values.end_ip_address is not "255.255.255.255" } } resources_with_tag_field_have_defined_tags = rule { all resources_with_tag_field as resource { resource.values.tags is not null } }

1. Standardize tests for static analysis of IaC. 
 secure standards & defaults 2. Enforce changes through IaC. 
 terraform apply -target, terraform taint 3. Enable dynamic analysis of infrastructure. 
 drift detection, automated reconciliation

Note: Control access to infrastructure. Store secrets outside of IaC.

04 Cost Lessons Learned / Challenges

Problem: We could be more efficient with our infrastructure.

Apply cost management techniques to infrastructure as code.

Commit changes. Run unit tests. Run integration tests. Estimate cost. Deploy changes. ✓ Security ✓ Cost compliance test_cpu_size_less_than_or_equal_to_32()

Cost Compliance 1. Enforce tags. 
 expiration date, standard tagging 2. Implement reboot schedule. 3. Set resource type, size, or reservation. 4. Check autoscaling enabled.

Note: Testing in production can eliminate some development environments.

Delivery Development Security Cost

Thank You Rosemary Wang @joatmon08 joatmon08.github.io