Test-Driven Infrastructure 101

Test-Driven Infrastructure 101

Creating better Infrastructure with Terraform, Test Kitchen, AWS & awspec.

374ccc5bcc36b5a07fa84af039e0cb95?s=128

GetCloudnative

August 04, 2018
Tweet

Transcript

  1. MARTIN ETMAJER Founder | GetCloudnative e.U. Version: 20180804 Creating Better

    Infrastructure with Terraform, Test Kitchen, AWS & awspec Test-Driven Infrastructure 101
  2. metmajer@getcloudnative.io @metmajer Innovation Technologist & Strategist. Founder at GetCloudnative. Open

    Source Maintainer. Martin Etmajer
  3. Why Test Infrastructure?

  4. Why Test Infrastructure? Because with Infrastructure as Code… you‘re running

    on code!
  5. MARTIN ETMAJER Founder | GetCloudnative e.U. Slide 5 „Our highest

    priority is to satisfy the customer through early and continuous delivery of valuable software.“ Agile Manifesto Principle No. 1
  6. MARTIN ETMAJER Founder | GetCloudnative e.U. Slide 6 The Continuous

    Delivery Deployment Pipeline Overview Developer Version Control Production Environment Code check-in Application release query check-out CI/CD Server (Pipeline Executor) Release Test Deploy Build working and deployable software a build advances along the pipeline in stages (in case of error, the pipeline is stopped)
  7. What is Test-Driven Infrastructure?

  8. MARTIN ETMAJER Founder | GetCloudnative e.U. Slide 8 „Make it

    work. Make it right. Make it fast.“ Kent Beck, Creator of Extreme Programming
  9. MARTIN ETMAJER Founder | GetCloudnative e.U. Slide 9 „Make it

    work. Make it right. Make it fast.“ Kent Beck, Creator of Extreme Programming get the code to operate correctly
  10. MARTIN ETMAJER Founder | GetCloudnative e.U. Slide 10 „Make it

    work. Make it right. Make it fast.“ Kent Beck, Creator of Extreme Programming make the code clear and enforce good design
  11. MARTIN ETMAJER Founder | GetCloudnative e.U. Slide 11 „Make it

    work. Make it right. Make it fast.“ Kent Beck, Creator of Extreme Programming optimize
  12. MARTIN ETMAJER Founder | GetCloudnative e.U. Slide 12 The Red-Green-Refactor

    Cycle of Test-Driven Development Write a Failing Test
  13. MARTIN ETMAJER Founder | GetCloudnative e.U. Slide 13 The Red-Green-Refactor

    Cycle of Test-Driven Development Write a Failing Test Make the Test Pass
  14. MARTIN ETMAJER Founder | GetCloudnative e.U. Slide 14 The Red-Green-Refactor

    Cycle of Test-Driven Development Clean up Your Code Write a Failing Test Make the Test Pass
  15. MARTIN ETMAJER Founder | GetCloudnative e.U. Slide 15 The Red-Green-Refactor

    Cycle of Test-Driven Development Write a Failing Test Make the Test Pass Clean up Your Code
  16. MARTIN ETMAJER Founder | GetCloudnative e.U. Slide 16 The Red-Green-Refactor

    Cycle of Test-Driven Development cycle iterates in small increments Write a Failing Test Make the Test Pass Clean up Your Code
  17. MARTIN ETMAJER Founder | GetCloudnative e.U. Slide 17 The Red-Green-Refactor

    Cycle of Test-Driven Development code is designed and tested cycle iterates in small increments Write a Failing Test Make the Test Pass Clean up Your Code
  18. MARTIN ETMAJER Founder | GetCloudnative e.U. Slide 18 The Red-Green-Refactor

    Cycle of Test-Driven Development code is designed and tested cycle iterates in small increments code is of higher quality Write a Failing Test Make the Test Pass Clean up Your Code
  19. MARTIN ETMAJER Founder | GetCloudnative e.U. Slide 19 The Red-Green-Refactor

    Cycle of Test-Driven Development code is designed and tested cycle iterates in small increments code is of higher quality Write a Failing Test Make the Test Pass Clean up Your Code cycle creates regression safety
  20. MARTIN ETMAJER Founder | GetCloudnative e.U. Slide 20 You Want

    to Trust but Verify You want to verify that your automation tool does the job right. You Want to Refactor your Code You want to simplify existing code or integrate a third-party module without compromising quality. You Want to Migrate to a Different Platform You want to migrate to a different operating system or cloud provider. You Want to Migrate to a Different Automation Tool You want to migrate from Bash or Perl to Ansible, Chef, Puppet or Terraform. Test-Driven Infrastructure Practical Reasons
  21. MARTIN ETMAJER Founder | GetCloudnative e.U. Slide 21 „With TDD,

    you express your intentions twice; once as a test, and once as production code. If the two approaches don’t match, your tests fail, and you’ve caught a bug.“ Kent Beck, Creator of Extreme Programming
  22. How to Test-Driven Infrastructure With Terraform, Test Kitchen, AWS &

    awspec
  23. Test Kitchen Infrastructure Integration Testing

  24. MARTIN ETMAJER Founder | GetCloudnative e.U. Slide 24 1. Drivers

    Let you run infrastructure code on various cloud providers and virtualization technologies. § Amazon EC2 § Azure RM § DigitalOcean Test Kitchen Basic Concepts § Docker § Google GCE § Microsoft Hyper-V § Rackspace § Vagrant § VMware vSphere
  25. MARTIN ETMAJER Founder | GetCloudnative e.U. Slide 25 1. Drivers

    Let you run infrastructure code on various cloud providers and virtualization technologies. § Amazon EC2 § Azure RM § DigitalOcean 2. Platforms Represent a configuration of an operating system. § Linux § Windows Test Kitchen Basic Concepts § Docker § Google GCE § Microsoft Hyper-V § Rackspace § Vagrant § VMware vSphere
  26. MARTIN ETMAJER Founder | GetCloudnative e.U. Slide 26 3. Provisioners

    These are the tools needed to converge the enrionment. § Ansible § Chef § Puppet Test Kitchen Basic Concepts § Terraform § SaltStack
  27. MARTIN ETMAJER Founder | GetCloudnative e.U. Slide 27 3. Provisioners

    These are the tools needed to converge the enrionment. § Ansible § Chef § Puppet 4. Test Suites Define the tests you want to run against an instance using a verifier. § Bash § Bats § Cucumber Test Kitchen Basic Concepts § Terraform § SaltStack § InSpec § RSpec § Serverspec
  28. Slide 28 MARTIN ETMAJER Founder | GetCloudnative e.U. How to

    Test-Driven Infrastructure? 1. Create a Terraform Module $ mkdir –p terraform/modules/my $ cd terraform/modules/my $ touch README.md \ main.tf \ variables.tf \ outputs.tf
  29. How to Test-Driven Infrastructure? 2. Install Test Kitchen Dependencies

  30. How to Test-Driven Infrastructure? 2. Install Test Kitchen Dependencies

  31. How to Test-Driven Infrastructure? 2. Install Test Kitchen Dependencies

  32. Just Real Quick… What is RSpec, Serverspec or awspec?

  33. RSpec Behaviour-Driven Development for Ruby

  34. Slide 34 File 1 2 3 4 5 6 7

    8 9 10 11 MARTIN ETMAJER Founder | GetCloudnative e.U. RSpec Behaviour-Driven Development for Ruby File foo_spec.rb 1 require ‘foo’ 2 3 describe Foo do 4 before Foo do 5 @foo = Foo.new 6 end 7 8 it “method #bar does something useful” do 9 @foo.bar should eq “something useful” 10 end 11 end
  35. Serverspec RSpec Tests for Your Servers

  36. Slide 36 File 1 12 2 13 3 14 4

    15 5 16 6 17 7 18 8 19 9 20 10 21 11 22 MARTIN ETMAJER Founder | GetCloudnative e.U. Serverspec RSpec Tests for Your Servers File server_spec.rb 1 require “serverspec” 12 describe service(“bar”) do 2 describe user(“foo”) do 13 it { should be_enabled } 3 it { should exist } 14 it { should be_running } 4 it { should belong_to_group “foo” } 15 end 5 end 16 describe port(8080) do 6 describe file(“/opt/bar”) do 17 it { should be_listening } 7 it { should be_directory } 18 end 8 it { should be_mode 777 } 19 describe command(“apachectl –M”) do 9 it { should be_owned_by “foo” } 20 its(:stdout) { should contain(“proxy_module”) } 10 it { should be_grouped_into “foo” } 21 end 11 end
  37. awspec RSpec Tests for Your AWS Resources

  38. Slide 38 File 1 2 3 4 5 6 7

    8 9 10 11 MARTIN ETMAJER Founder | GetCloudnative e.U. awspec RSpec Tests for Your AWS Resources File ec2_spec.rb 1 require “awspec” 2 describe ec2(“my-ec2-tag-name”) do 3 it { should be_running } 4 its(:instance_id) { should eq “i-ec12345a” } 5 its(:image_id) { should eq “ami-abc12def” } 6 its(:public_ip_address) { should eq “123.0.456.789” } 7 it { should have_security_group(“my-security-group-name”) } 8 it { should belong_to_vpc(“my-vpc”) } 9 it { should belong_to_subnet(“subnet-1234a567”) } 10 it { should have_eip(“123.0.456.789”) } 11 end
  39. Slide 39 MARTIN ETMAJER Founder | GetCloudnative e.U. How to

    Test-Driven Infrastructure? 2. Install Test Kitchen Dependencies $ gem install \ test-kitchen \ kitchen-terraform \ kitchen-verifier-awspec $ kitchen version Test Kitchen version 1.21.2
  40. Slide 40 MARTIN ETMAJER Founder | GetCloudnative e.U. How to

    Test-Driven Infrastructure? 3. Configure Test Kitchen $ kitchen init --driver terraform --provisioner terraform create kitchen.yml create test/integration/default
  41. Slide 41 File 1 12 2 13 3 14 4

    15 5 16 6 17 7 18 8 19 9 20 10 21 11 22 MARTIN ETMAJER Founder | GetCloudnative e.U. How to Test-Driven Infrastructure? 3. Configure Test Kitchen File kitchen.yml 1 driver: 8 suites: 2 name: terraform 9 - name: default 10 run_list: 11 attributes: 3 provisioner: 4 name: terraform 5 platforms: 6 - name: ubuntu-16.04 7 - name: centos-7
  42. Slide 42 File 1 12 2 13 3 14 4

    15 5 16 6 17 7 18 8 19 9 20 10 21 11 22 MARTIN ETMAJER Founder | GetCloudnative e.U. How to Test-Driven Infrastructure? 3. Configure Test Kitchen with kitchen-terraform and kitchen-verifier-awspec File kitchen.yml 1 driver: 11 suites: 2 name: terraform 12 - name: default 3 variable_files: 13 verifier: 4 - terraform-testing.tfvars.json 14 patterns: 5 provisioner: 15 - test/integration/default/*_spec.rb 6 name: terraform 7 platforms: 8 - name: aws 9 verifier: 10 name: awspec
  43. Slide 43 MARTIN ETMAJER Founder | GetCloudnative e.U. How to

    Test-Driven Infrastructure? 3. Configure Test Kitchen $ kitchen list Instance Driver Provisioner Verifier Transport Last Action default-aws Terraform Terraform Awspec Ssh <Not Created>
  44. Slide 44 File 1 2 3 4 5 6 7

    8 9 10 11 MARTIN ETMAJER Founder | GetCloudnative e.U. How to Test-Driven Infrastructure? 5. Configure AWS Credentials File ~/.aws/credentials 1 [staging] 2 aws_access_key_id=[...] 3 aws_secret_access_key=[...]
  45. MARTIN ETMAJER Founder | GetCloudnative e.U. Slide 45 How to

    Test-Driven Infrastructure? 5. Write a Failing Test Write a Failing Test
  46. Slide 46 File 1 12 2 13 3 14 4

    15 5 16 6 17 7 18 8 19 9 20 10 21 11 22 MARTIN ETMAJER Founder | GetCloudnative e.U. How to Test-Driven Infrastructure? 5. Write a Failing Test File kitchen.yml 1 driver: 11 suites: 2 name: terraform 12 - name: default 3 variable_files: 13 verifier: 4 - terraform-testing.tfvars.json 14 patterns: 5 provisioner: 15 - test/integration/default/*_spec.rb 6 name: terraform 7 platforms: 8 - name: aws 9 verifier: 10 name: awspec
  47. Slide 47 File 1 2 3 4 5 6 7

    8 9 10 11 MARTIN ETMAJER Founder | GetCloudnative e.U. How to Test-Driven Infrastructure? 5. Write a Failing Test File test/integration/default/default_spec.rb 1 require ‘awspec’ 2 require_relative ‘./spec_helper’ 3 4 describe “My AWS Deployment” do 5 describe ec2(EC2Helper.get_ec2_instance_id_from_tag_name(“my-aws-ec2-instance”)) do 6 it { should exist } 7 it { should be_running } 8 end 9 end
  48. Slide 48 MARTIN ETMAJER Founder | GetCloudnative e.U. How to

    Test-Driven Infrastructure? 5. Write a Failing Test $ kitchen verify -----> Starting Kitchen (v1.21.2) [...] $$$$$$ Running command `terraform apply [...]` Apply complete! Resources: 0 added, 0 changed, 0 destroyed. -----> Verifying <default-aws>... Running Awspec Error: 'my-aws-ec2-instance' Instance not found Finished in 0.02451 seconds (files took 3.37 seconds to load) 2 examples, 2 failures
  49. Slide 49 MARTIN ETMAJER Founder | GetCloudnative e.U. How to

    Test-Driven Infrastructure? 5. Write a Failing Test $ kitchen list Instance Driver Provisioner Verifier Transport Last Action default-aws Terraform Terraform Awspec Ssh Kitchen:ActionFailed
  50. MARTIN ETMAJER Founder | GetCloudnative e.U. Slide 50 How to

    Test-Driven Infrastructure? 6. Make the Test Pass Write a Failing Test Make the Test Pass
  51. Slide 51 File 1 2 3 4 5 6 7

    8 9 10 11 MARTIN ETMAJER Founder | GetCloudnative e.U. How to Test-Driven Infrastructure? 6. Make the Test Pass File main.tf 1 provider “aws” { 2 region = “us-east-1” 3 } 4 resource “aws_instance” “compute” { 5 ami = “ami-97785bed” 6 instance_type = “t2.micro” 7 tags { 8 Name = “my-aws-ec2-instance” 9 } 10 }
  52. How to Test-Driven Infrastructure? 6. Make the Test Pass $

    kitchen verify -----> Starting Kitchen (v1.21.2) [...] $$$$$$ Running command `terraform apply [...]` Apply complete! Resources: 0 added, 0 changed, 0 destroyed. -----> Verifying <default-aws>... Running Awspec [...] Deployment ec2 'i-02a6a01b0509c55de' should exist should be running Finished in 0.16529 seconds (files took 3.63 seconds to load) 2 examples, 0 failures
  53. Slide 53 MARTIN ETMAJER Founder | GetCloudnative e.U. How to

    Test-Driven Infrastructure? 6. Make the Test Pass $ kitchen list Instance Driver Provisioner Verifier Transport Last Action default-aws Terraform Terraform Awspec Ssh Verified
  54. MARTIN ETMAJER Founder | GetCloudnative e.U. Slide 54 How to

    Test-Driven Infrastructure? 6. Make the Test Pass Write a Failing Test Make the Test Pass Clean up Your Code
  55. MARTIN ETMAJER Founder | GetCloudnative e.U. Slide 55 How to

    Test-Driven Infrastructure? 7. Rinse and Repeat Write a Failing Test Make the Test Pass Clean up Your Code
  56. Prototype Terraform, Test Kitchen, AWS & awspec in a Jenkins

    Pipeline
  57. What Else Is There?

  58. Agile Infrastructure: Because your Infrastructure Deserves Tests, Too Author: Martin

    Etmajer
  59. The Art of Agile Development: Test-Driven Development Author: James Shore