Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Test-Driven Infrastructure 101

Test-Driven Infrastructure 101

Creating better Infrastructure with Terraform, Test Kitchen, AWS & awspec.

GetCloudnative

August 04, 2018
Tweet

More Decks by GetCloudnative

Other Decks in Technology

Transcript

  1. MARTIN ETMAJER
    Founder | GetCloudnative e.U. Version: 20180804
    Creating Better Infrastructure with
    Terraform, Test Kitchen, AWS & awspec
    Test-Driven Infrastructure 101

    View Slide

  2. [email protected]
    @metmajer
    Innovation Technologist & Strategist.
    Founder at GetCloudnative.
    Open Source Maintainer.
    Martin Etmajer

    View Slide

  3. Why Test Infrastructure?

    View Slide

  4. Why Test Infrastructure?
    Because with Infrastructure as Code… you‘re running on code!

    View Slide

  5. MARTIN ETMAJER
    Founder | GetCloudnative e.U. Slide 5
    „Our highest priority is to satisfy the customer through
    early and continuous delivery of valuable software.“
    Agile Manifesto Principle No. 1

    View Slide

  6. MARTIN ETMAJER
    Founder | GetCloudnative e.U. Slide 6
    The Continuous Delivery Deployment Pipeline
    Overview
    Developer Version
    Control
    Production
    Environment
    Code
    check-in
    Application
    release
    query
    check-out
    CI/CD Server (Pipeline Executor)
    Release
    Test
    Deploy
    Build
    working and deployable software
    a build advances along the pipeline in stages
    (in case of error, the pipeline is stopped)

    View Slide

  7. What is Test-Driven Infrastructure?

    View Slide

  8. MARTIN ETMAJER
    Founder | GetCloudnative e.U. Slide 8
    „Make it work. Make it right. Make it fast.“
    Kent Beck, Creator of Extreme Programming

    View Slide

  9. MARTIN ETMAJER
    Founder | GetCloudnative e.U. Slide 9
    „Make it work. Make it right. Make it fast.“
    Kent Beck, Creator of Extreme Programming
    get the code to operate correctly

    View Slide

  10. MARTIN ETMAJER
    Founder | GetCloudnative e.U. Slide 10
    „Make it work. Make it right. Make it fast.“
    Kent Beck, Creator of Extreme Programming
    make the code clear and enforce good design

    View Slide

  11. MARTIN ETMAJER
    Founder | GetCloudnative e.U. Slide 11
    „Make it work. Make it right. Make it fast.“
    Kent Beck, Creator of Extreme Programming
    optimize

    View Slide

  12. MARTIN ETMAJER
    Founder | GetCloudnative e.U. Slide 12
    The Red-Green-Refactor Cycle of Test-Driven Development
    Write a
    Failing
    Test

    View Slide

  13. MARTIN ETMAJER
    Founder | GetCloudnative e.U. Slide 13
    The Red-Green-Refactor Cycle of Test-Driven Development
    Write a
    Failing
    Test
    Make the
    Test Pass

    View Slide

  14. MARTIN ETMAJER
    Founder | GetCloudnative e.U. Slide 14
    The Red-Green-Refactor Cycle of Test-Driven Development
    Clean up
    Your Code
    Write a
    Failing
    Test
    Make the
    Test Pass

    View Slide

  15. MARTIN ETMAJER
    Founder | GetCloudnative e.U. Slide 15
    The Red-Green-Refactor Cycle of Test-Driven Development
    Write a
    Failing
    Test
    Make the
    Test Pass
    Clean up
    Your Code

    View Slide

  16. MARTIN ETMAJER
    Founder | GetCloudnative e.U. Slide 16
    The Red-Green-Refactor Cycle of Test-Driven Development
    cycle iterates in
    small increments
    Write a
    Failing
    Test
    Make the
    Test Pass
    Clean up
    Your Code

    View Slide

  17. MARTIN ETMAJER
    Founder | GetCloudnative e.U. Slide 17
    The Red-Green-Refactor Cycle of Test-Driven Development
    code is designed and tested
    cycle iterates in
    small increments
    Write a
    Failing
    Test
    Make the
    Test Pass
    Clean up
    Your Code

    View Slide

  18. MARTIN ETMAJER
    Founder | GetCloudnative e.U. Slide 18
    The Red-Green-Refactor Cycle of Test-Driven Development
    code is designed and tested
    cycle iterates in
    small increments
    code is of higher quality
    Write a
    Failing
    Test
    Make the
    Test Pass
    Clean up
    Your Code

    View Slide

  19. MARTIN ETMAJER
    Founder | GetCloudnative e.U. Slide 19
    The Red-Green-Refactor Cycle of Test-Driven Development
    code is designed and tested
    cycle iterates in
    small increments
    code is of higher quality
    Write a
    Failing
    Test
    Make the
    Test Pass
    Clean up
    Your Code
    cycle creates
    regression safety

    View Slide

  20. MARTIN ETMAJER
    Founder | GetCloudnative e.U. Slide 20
    You Want to Trust but Verify
    You want to verify that your automation tool does the job right.
    You Want to Refactor your Code
    You want to simplify existing code or integrate a third-party module without compromising quality.
    You Want to Migrate to a Different Platform
    You want to migrate to a different operating system or cloud provider.
    You Want to Migrate to a Different Automation Tool
    You want to migrate from Bash or Perl to Ansible, Chef, Puppet or Terraform.
    Test-Driven Infrastructure
    Practical Reasons

    View Slide

  21. MARTIN ETMAJER
    Founder | GetCloudnative e.U. Slide 21
    „With TDD, you express your intentions twice;
    once as a test, and once as production code.
    If the two approaches don’t match,
    your tests fail, and you’ve caught a bug.“
    Kent Beck, Creator of Extreme Programming

    View Slide

  22. How to Test-Driven Infrastructure
    With Terraform, Test Kitchen, AWS & awspec

    View Slide

  23. Test Kitchen
    Infrastructure Integration Testing

    View Slide

  24. MARTIN ETMAJER
    Founder | GetCloudnative e.U. Slide 24
    1. Drivers
    Let you run infrastructure code on various cloud providers and virtualization technologies.
    § Amazon EC2
    § Azure RM
    § DigitalOcean
    Test Kitchen
    Basic Concepts
    § Docker
    § Google GCE
    § Microsoft Hyper-V
    § Rackspace
    § Vagrant
    § VMware vSphere

    View Slide

  25. MARTIN ETMAJER
    Founder | GetCloudnative e.U. Slide 25
    1. Drivers
    Let you run infrastructure code on various cloud providers and virtualization technologies.
    § Amazon EC2
    § Azure RM
    § DigitalOcean
    2. Platforms
    Represent a configuration of an operating system.
    § Linux
    § Windows
    Test Kitchen
    Basic Concepts
    § Docker
    § Google GCE
    § Microsoft Hyper-V
    § Rackspace
    § Vagrant
    § VMware vSphere

    View Slide

  26. MARTIN ETMAJER
    Founder | GetCloudnative e.U. Slide 26
    3. Provisioners
    These are the tools needed to converge the enrionment.
    § Ansible
    § Chef
    § Puppet
    Test Kitchen
    Basic Concepts
    § Terraform
    § SaltStack

    View Slide

  27. MARTIN ETMAJER
    Founder | GetCloudnative e.U. Slide 27
    3. Provisioners
    These are the tools needed to converge the enrionment.
    § Ansible
    § Chef
    § Puppet
    4. Test Suites
    Define the tests you want to run against an instance using a verifier.
    § Bash
    § Bats
    § Cucumber
    Test Kitchen
    Basic Concepts
    § Terraform
    § SaltStack
    § InSpec
    § RSpec
    § Serverspec

    View Slide

  28. Slide 28
    MARTIN ETMAJER
    Founder | GetCloudnative e.U.
    How to Test-Driven Infrastructure?
    1. Create a Terraform Module
    $ mkdir –p terraform/modules/my
    $ cd terraform/modules/my
    $ touch README.md \
    main.tf \
    variables.tf \
    outputs.tf

    View Slide

  29. How to Test-Driven Infrastructure?
    2. Install Test Kitchen Dependencies

    View Slide

  30. How to Test-Driven Infrastructure?
    2. Install Test Kitchen Dependencies

    View Slide

  31. How to Test-Driven Infrastructure?
    2. Install Test Kitchen Dependencies

    View Slide

  32. Just Real Quick…
    What is RSpec, Serverspec or awspec?

    View Slide

  33. RSpec
    Behaviour-Driven Development for Ruby

    View Slide

  34. Slide 34
    File
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    MARTIN ETMAJER
    Founder | GetCloudnative e.U.
    RSpec
    Behaviour-Driven Development for Ruby
    File foo_spec.rb
    1 require ‘foo’
    2
    3 describe Foo do
    4 before Foo do
    5 @foo = Foo.new
    6 end
    7
    8 it “method #bar does something useful” do
    9 @foo.bar should eq “something useful”
    10 end
    11 end

    View Slide

  35. Serverspec
    RSpec Tests for Your Servers

    View Slide

  36. Slide 36
    File
    1 12
    2 13
    3 14
    4 15
    5 16
    6 17
    7 18
    8 19
    9 20
    10 21
    11 22
    MARTIN ETMAJER
    Founder | GetCloudnative e.U.
    Serverspec
    RSpec Tests for Your Servers
    File server_spec.rb
    1 require “serverspec” 12 describe service(“bar”) do
    2 describe user(“foo”) do 13 it { should be_enabled }
    3 it { should exist } 14 it { should be_running }
    4 it { should belong_to_group “foo” } 15 end
    5 end 16 describe port(8080) do
    6 describe file(“/opt/bar”) do 17 it { should be_listening }
    7 it { should be_directory } 18 end
    8 it { should be_mode 777 } 19 describe command(“apachectl –M”) do
    9 it { should be_owned_by “foo” } 20 its(:stdout) { should contain(“proxy_module”) }
    10 it { should be_grouped_into “foo” } 21 end
    11 end

    View Slide

  37. awspec
    RSpec Tests for Your AWS Resources

    View Slide

  38. Slide 38
    File
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    MARTIN ETMAJER
    Founder | GetCloudnative e.U.
    awspec
    RSpec Tests for Your AWS Resources
    File ec2_spec.rb
    1 require “awspec”
    2 describe ec2(“my-ec2-tag-name”) do
    3 it { should be_running }
    4 its(:instance_id) { should eq “i-ec12345a” }
    5 its(:image_id) { should eq “ami-abc12def” }
    6 its(:public_ip_address) { should eq “123.0.456.789” }
    7 it { should have_security_group(“my-security-group-name”) }
    8 it { should belong_to_vpc(“my-vpc”) }
    9 it { should belong_to_subnet(“subnet-1234a567”) }
    10 it { should have_eip(“123.0.456.789”) }
    11 end

    View Slide

  39. Slide 39
    MARTIN ETMAJER
    Founder | GetCloudnative e.U.
    How to Test-Driven Infrastructure?
    2. Install Test Kitchen Dependencies
    $ gem install \
    test-kitchen \
    kitchen-terraform \
    kitchen-verifier-awspec
    $ kitchen version
    Test Kitchen version 1.21.2

    View Slide

  40. Slide 40
    MARTIN ETMAJER
    Founder | GetCloudnative e.U.
    How to Test-Driven Infrastructure?
    3. Configure Test Kitchen
    $ kitchen init --driver terraform --provisioner terraform
    create kitchen.yml
    create test/integration/default

    View Slide

  41. Slide 41
    File
    1 12
    2 13
    3 14
    4 15
    5 16
    6 17
    7 18
    8 19
    9 20
    10 21
    11 22
    MARTIN ETMAJER
    Founder | GetCloudnative e.U.
    How to Test-Driven Infrastructure?
    3. Configure Test Kitchen
    File kitchen.yml
    1 driver: 8 suites:
    2 name: terraform 9 - name: default
    10 run_list:
    11 attributes:
    3 provisioner:
    4 name: terraform
    5 platforms:
    6 - name: ubuntu-16.04
    7 - name: centos-7

    View Slide

  42. Slide 42
    File
    1 12
    2 13
    3 14
    4 15
    5 16
    6 17
    7 18
    8 19
    9 20
    10 21
    11 22
    MARTIN ETMAJER
    Founder | GetCloudnative e.U.
    How to Test-Driven Infrastructure?
    3. Configure Test Kitchen with kitchen-terraform and kitchen-verifier-awspec
    File kitchen.yml
    1 driver: 11 suites:
    2 name: terraform 12 - name: default
    3 variable_files: 13 verifier:
    4 - terraform-testing.tfvars.json 14 patterns:
    5 provisioner: 15 - test/integration/default/*_spec.rb
    6 name: terraform
    7 platforms:
    8 - name: aws
    9 verifier:
    10 name: awspec

    View Slide

  43. Slide 43
    MARTIN ETMAJER
    Founder | GetCloudnative e.U.
    How to Test-Driven Infrastructure?
    3. Configure Test Kitchen
    $ kitchen list
    Instance Driver Provisioner Verifier Transport Last Action
    default-aws Terraform Terraform Awspec Ssh

    View Slide

  44. Slide 44
    File
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    MARTIN ETMAJER
    Founder | GetCloudnative e.U.
    How to Test-Driven Infrastructure?
    5. Configure AWS Credentials
    File ~/.aws/credentials
    1 [staging]
    2 aws_access_key_id=[...]
    3 aws_secret_access_key=[...]

    View Slide

  45. MARTIN ETMAJER
    Founder | GetCloudnative e.U. Slide 45
    How to Test-Driven Infrastructure?
    5. Write a Failing Test
    Write a
    Failing
    Test

    View Slide

  46. Slide 46
    File
    1 12
    2 13
    3 14
    4 15
    5 16
    6 17
    7 18
    8 19
    9 20
    10 21
    11 22
    MARTIN ETMAJER
    Founder | GetCloudnative e.U.
    How to Test-Driven Infrastructure?
    5. Write a Failing Test
    File kitchen.yml
    1 driver: 11 suites:
    2 name: terraform 12 - name: default
    3 variable_files: 13 verifier:
    4 - terraform-testing.tfvars.json 14 patterns:
    5 provisioner: 15 - test/integration/default/*_spec.rb
    6 name: terraform
    7 platforms:
    8 - name: aws
    9 verifier:
    10 name: awspec

    View Slide

  47. Slide 47
    File
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    MARTIN ETMAJER
    Founder | GetCloudnative e.U.
    How to Test-Driven Infrastructure?
    5. Write a Failing Test
    File test/integration/default/default_spec.rb
    1 require ‘awspec’
    2 require_relative ‘./spec_helper’
    3
    4 describe “My AWS Deployment” do
    5 describe ec2(EC2Helper.get_ec2_instance_id_from_tag_name(“my-aws-ec2-instance”)) do
    6 it { should exist }
    7 it { should be_running }
    8 end
    9 end

    View Slide

  48. Slide 48
    MARTIN ETMAJER
    Founder | GetCloudnative e.U.
    How to Test-Driven Infrastructure?
    5. Write a Failing Test
    $ kitchen verify
    -----> Starting Kitchen (v1.21.2)
    [...]
    $$$$$$ Running command `terraform apply [...]`
    Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
    -----> Verifying ...
    Running Awspec
    Error: 'my-aws-ec2-instance' Instance not found
    Finished in 0.02451 seconds (files took 3.37 seconds to load)
    2 examples, 2 failures

    View Slide

  49. Slide 49
    MARTIN ETMAJER
    Founder | GetCloudnative e.U.
    How to Test-Driven Infrastructure?
    5. Write a Failing Test
    $ kitchen list
    Instance Driver Provisioner Verifier Transport Last Action
    default-aws Terraform Terraform Awspec Ssh Kitchen:ActionFailed

    View Slide

  50. MARTIN ETMAJER
    Founder | GetCloudnative e.U. Slide 50
    How to Test-Driven Infrastructure?
    6. Make the Test Pass
    Write a
    Failing
    Test
    Make the
    Test Pass

    View Slide

  51. Slide 51
    File
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    MARTIN ETMAJER
    Founder | GetCloudnative e.U.
    How to Test-Driven Infrastructure?
    6. Make the Test Pass
    File main.tf
    1 provider “aws” {
    2 region = “us-east-1”
    3 }
    4 resource “aws_instance” “compute” {
    5 ami = “ami-97785bed”
    6 instance_type = “t2.micro”
    7 tags {
    8 Name = “my-aws-ec2-instance”
    9 }
    10 }

    View Slide

  52. How to Test-Driven Infrastructure?
    6. Make the Test Pass
    $ kitchen verify
    -----> Starting Kitchen (v1.21.2)
    [...]
    $$$$$$ Running command `terraform apply [...]`
    Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
    -----> Verifying ...
    Running Awspec
    [...]
    Deployment
    ec2 'i-02a6a01b0509c55de'
    should exist
    should be running
    Finished in 0.16529 seconds (files took 3.63 seconds to load)
    2 examples, 0 failures

    View Slide

  53. Slide 53
    MARTIN ETMAJER
    Founder | GetCloudnative e.U.
    How to Test-Driven Infrastructure?
    6. Make the Test Pass
    $ kitchen list
    Instance Driver Provisioner Verifier Transport Last Action
    default-aws Terraform Terraform Awspec Ssh Verified

    View Slide

  54. MARTIN ETMAJER
    Founder | GetCloudnative e.U. Slide 54
    How to Test-Driven Infrastructure?
    6. Make the Test Pass
    Write a
    Failing
    Test
    Make the
    Test Pass
    Clean up
    Your Code

    View Slide

  55. MARTIN ETMAJER
    Founder | GetCloudnative e.U. Slide 55
    How to Test-Driven Infrastructure?
    7. Rinse and Repeat
    Write a
    Failing
    Test
    Make the
    Test Pass
    Clean up
    Your Code

    View Slide

  56. Prototype
    Terraform, Test Kitchen, AWS & awspec in a Jenkins Pipeline

    View Slide

  57. What Else Is There?

    View Slide

  58. Agile Infrastructure: Because your Infrastructure Deserves Tests, Too
    Author: Martin Etmajer

    View Slide

  59. The Art of Agile Development: Test-Driven Development
    Author: James Shore

    View Slide