Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Testing Your Cloud Infrastructure (as Code)

Testing Your Cloud Infrastructure (as Code)

Presented at Odyssey 2021.

You update your infrastructure modules or configurations and quickly push production changes. Meanwhile, you’re secretly hoping they all work but dreading the alert that something has gone wrong. Is there a way to alleviate some of the dread and catch errors before production?

In this talk, Rosemary will apply the test pyramid to testing cloud infrastructure and show how some types of tests can flag potential misconfigurations to infrastructure as code before pushing to production. She'll also compare the financial and time cost of running each test type. By the end of the session, you’ll leave with a better understanding of when to write tests for your cloud infrastructure, how to manage the cost of running tests, and which tests can benefit your changes.


Rosemary Wang

March 31, 2021

More Decks by Rosemary Wang

Other Decks in Programming


  1. Copyright © 2020 HashiCorp Testing your Cloud Infrastructure (as Code)

    Rosemary Wang (@joatmon08) OdysseyConf | March 31, 2021 1
  2. Migrate to public cloud infrastructure. @JOATMON08 2

  3. Infrastructure as Code Declare the configuration you want and store

    it in source control. Immutable Create a new resource for every update, delete the old resource. On-Demand Create and delete resources when you need them (or don’t). 3 @JOATMON08
  4. @JOATMON08 4 We wrote some tests for our cloud infrastructure.

    Testing environments cost us 70% of our cloud provider bill. We pushed a change that affected production. We deactivated tests and deleted the testing environments
  5. Rosemary Wang (She/Her) Developer Advocate at HashiCorp @joatmon08 joatmon08.github.io 5

  6. @JOATMON08 6 Infrastructure tests to catch problems before production. Cost

    of environments for running tests.
  7. @JOATMON08 7 Integration Tests Contract Tests Unit Tests Cost* (Time,

    $$$) End-to-End Tests
  8. Unit Tests 8

  9. Unit Tests Test configuration or state metadata ▪ Lint and

    check syntax ▪ Any testing framework or language that parses data formats – Programming languages – Behavior-driven development (BDD) – Other tools (e.g., CloudFormation linter, HashiCorp Sentinel, terraform- compliance, Helm unittest plugin) 9 @JOATMON08
  10. @JOATMON08 10 You write some configuration. INFRASTRUCTURE AS CODE CONFIGURATION

    Tool uses configuration to generate changes to infrastructure state. LIST OF CHANGES TO INFRASTRUCTURE STATE UNIT TESTS PARSE CONFIGURATION OR LIST OF CHANGES.
  11. Unit Tests Cost assessment ▪ Worth writing a few… –

    Communicate expectations, including security – No infrastructure required ▪ Do not… – Work offline (i.e., retrieve remote state) – Prevent system failure 11 @JOATMON08
  12. Contract Tests 12

  13. Contract Tests Test configuration inputs and outputs ▪ Validate inputs

    or outputs – Password requirements – Special values (e.g., port ranges, constants) ▪ Public cloud APIs constantly change ▪ Check infrastructure dependencies ▪ Any testing framework or language for validation 13 @JOATMON08
  14. CODE EDITOR variable "listener_rule_priority" { type = number default =

    1 description = "Priority of listener rule between 1 to 50000" validation { condition = var.listener_rule_priority > 0 && var.listener_rule_priority < 50000 error_message = "The priority of listener rule must between 1 to 50000." } } @JOATMON08 14
  15. Contract Tests Cost assessment ▪ Worth writing a few… –

    (Usually) no infrastructure required – Fail fast for invalid input values ▪ Do not… – Work offline (i.e., retrieve remote state) – Prevent system failure 15 @JOATMON08
  16. Integration Tests 16

  17. Integration Tests Test if the configuration creates infrastructure ▪ Can

    you even apply the changes to infrastructure? ▪ Verify dependencies with real infrastructure ▪ Any testing framework or language with setup, test, teardown – Integration testing frameworks (e.g., TaskCat, kitchen, terratest, goss, Inspec) – Programming language – Add to delivery pipeline 17 @JOATMON08
  18. @JOATMON08 18 You push some configuration to source control. INFRASTRUCTURE

    AS CODE CONFIGURATION Pipeline sets up resources in testing environment. Pipeline tears down resources in testing environment. INTEGRATION TESTS
  19. CODE EDITOR @pytest.fixture(scope='session') def apply_changes(): generate_json(TEST_SERVER_NAME) assert os.path.exists(SERVER_CONFIGURATION_FILE) assert test_utils.initialize()

    == 0 yield test_utils.apply() assert test_utils.destroy() == 0 os.remove(SERVER_CONFIGURATION_FILE) def test_changes_should_add_1_resource(apply_changes): output = apply_changes[1].decode(encoding='utf-8').split('\n') assert 'Apply complete! Resources: 1 added, 0 changed, 0 destroyed' \ in output[-2] def test_server_is_in_running_state(apply_changes): gcp_server = test_utils.get_server(TEST_SERVER_NAME) assert gcp_server.state == NodeState.RUNNING 
  20. Integration Tests Cost assessment ▪ Worth writing… – Test dependencies

    – Build confidence in “successful” changes ▪ Do not… – Eliminate cost of resources – Speed up feedback loop 20 @JOATMON08
  21. End-to-End Tests 21

  22. End-to-End Tests Test if infrastructure supports the workflows ▪ Can

    you create or access resources? ▪ Verify end-to-end functionality ▪ Run after you apply changes in testing / production ▪ Includes smoke tests ▪ Any testing framework or language with enough access to create or read resources 22 @JOATMON08
  23. CODE EDITOR $ kitchen test -----> Starting Kitchen (v2.3.3) …

    Profile: End-to-End Tests for Application (default) ✔ db: Database: check routing from public to private subnet ✔ Host port 27017 proto tcp should be reachable ✔ Host port 27017 proto tcp should be resolvable ✔ Host port 80 proto tcp should not be reachable ✔ outbound: Public Subnet: check routing out to public internet ✔ HTTP GET on https://hashicorp.com status should cmp == 301 Profile Summary: 2 successful controls, 0 control failures, 0 controls skipped Test Summary: 3 successful, 0 failures, 0 skipped @JOATMON08 23
  24. End-to-End Tests Cost assessment ▪ Worth writing… – Test functionality

    – Check system works before “release” ▪ Do not… – Eliminate cost of resources – Speed up feedback loop 24 @JOATMON08
  25. Look at our cloud bill! 25

  26. @JOATMON08 26 Integration Tests Contract Tests Unit Tests Cost* (Time,

  27. Use Cheaper Resources ✅ Smaller size ✅ Shorter lifecycle ❌

    Not always accurate ❌ Drift between testing & production Use Infrastructure API Mocks ✅ No infrastructure ✅ Can run offline ❌ Dependencies? ❌ Drift between mock & actual APIs Delete Long-Lived Environments ✅ Elasticity ✅ Shorter lifecycle ❌ Confidence? ❌ Time spent creating environments 27 @JOATMON08
  28. How much is really from testing? Use resource tagging to

    answer this question. @JOATMON08 28
  29. Resource Tagging Identify why you have them in the first

    place ▪ Environment (testing/production) ▪ Test Type (integration/end-to-end) ▪ Repository (joatmon08/terraform-aws-listenerrule-nia) ▪ Teardown (true/false) 29 @JOATMON08
  30. AWS Listener Rule Integration Tests (02/2021) $15.12 1 Application Load

    Balancer us-east-1, 672 hours $0.00 1 Elastic IP us-east-1 $0.82 1 EC2 Instance us-east-1, t2.micro, 72 hours $15.94 Total us-east-1 @JOATMON08 30
  31. Solution: Use elasticity! Create and delete resources before and after

    testing. 31 @JOATMON08
  32. Problem: Assumes immutability. You might make some changes in-place. 32

  33. Reality: Some long-lived resources. e.g., networking, databases, Kubernetes control planes

    33 @JOATMON08
  34. Time to Change Shorter = Setup & Teardown Number of

    Dependencies Fewer = Setup & Teardown Frequency of Change Less = Setup & Teardown Statefulness Less = Setup & Teardown 34
  35. @JOATMON08 35 We write some tests for our cloud infrastructure.

    We assess the cost based on tagging. We push the change to production. We can replace the long-lived resources for this set of tests.
  36. References Infrastructure testing is a heuristic. ▪ github.com/joatmon08/tdd-infrastructure ▪ puppet.com/blog/hitchhikers-guide-to-testing-infrastructure-as-and-code

    ▪ github.com/joatmon08/terraform-aws-listenerrule-nia ▪ hashicorp.com/resources/testing-your-hcl-modules-in-terraform 36 @JOATMON08
  37. Thank you! Rosemary Wang @joatmon08 joatmon08.github.io 37