Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Beyond Pester 102: Acceptance testing with PowerShell

Glenn
April 26, 2019

Beyond Pester 102: Acceptance testing with PowerShell

Previously in "Beyond Pester 101: Applying testing principles to PowerShell" I talked about unit and integration testing with Pester and PowerShell. Now that we're all experts there, it's time to tackle Acceptance testing, also known as End to End testing.

Glenn

April 26, 2019
Tweet

More Decks by Glenn

Other Decks in Technology

Transcript

  1. @glennsarti glenn-sarti

    View Slide

  2. @glennsarti
    Previously …

    View Slide

  3. @glennsarti
    Unit (White box)
    Integration
    Unit (Black box)
    Characterisation
    Reduce duration
    Remove tests
    Reorder tests
    General
    maintenance

    View Slide

  4. @glennsarti
    Types of tests - Integration
    Image adapted from Test Pyramid – Martin Fowler
    1. Slower
    2. More $$$
    3. Complex

    View Slide

  5. @glennsarti
    Why do I test?

    View Slide

  6. @glennsarti
    To reduce the risk that a
    user will experience an
    unexpected behaviour

    View Slide

  7. @glennsarti
    To reduce the risk that a
    user will experience an
    unexpected behaviour

    View Slide

  8. @glennsarti
    Benefits of testing
    Shorter delivery cycles
    Produce high quality code
    Cheaper

    View Slide

  9. @glennsarti
    To ensure what is created does
    what it’s supposed to do
    Form of documentation
    Benefits of testing

    View Slide

  10. @glennsarti

    View Slide

  11. @glennsarti
    Image adapted from Test Pyramid – Martin Fowler

    View Slide

  12. @glennsarti
    Image adapted from Test Pyramid – Martin Fowler
    ⚠️ Slow

    View Slide

  13. @glennsarti
    Image adapted from Test Pyramid – Martin Fowler
    ⚠️ Expensive

    View Slide

  14. @glennsarti
    Image adapted from Test Pyramid – Martin Fowler
    ⚠️ Complex

    View Slide

  15. @glennsarti
    The cost to fix a bug
    Cost
    to
    Fix
    – Testing in Production – Rob Crowley
    Development Manual Testing From user
    Expensive
    Cheap

    View Slide

  16. @glennsarti
    Fixing a broken test
    Acceptance Tests
    Integration Tests
    Unit Tests
    PowerShell
    being tested
    PAIN

    View Slide

  17. @glennsarti
    Things that can go wrong
    One
    Plethora
    (*) Graph is not to scale or based on actual data
    Acceptance Tests
    Integration Tests
    Unit Tests

    View Slide

  18. @glennsarti


    View Slide

  19. @glennsarti
    Customers don’t measure you on
    how hard you tried, they measure
    you on what you deliver.”

    - Steve Jobs

    View Slide

  20. @glennsarti
    Where’s the value?

    View Slide

  21. @glennsarti
    … there are known knowns …
    known unknowns … also unknown
    unknowns … it is the latter category
    that tend to be the difficult ones
    - Donald Rumsfeld


    View Slide

  22. @glennsarti
    What can we do then…

    View Slide

  23. @glennsarti
    Think Different
    Change Scope

    View Slide

  24. @glennsarti
    Think Different
    Change Scope

    View Slide

  25. @glennsarti
    Cynefin Framework
    “The Cynefin framework (/ˈkʌnɪvɪn/
    KUN-iv-in) is a conceptual framework
    used to aid decision-making”
    - https://www.youtube.com/watch?v=N7oz366X0-8

    View Slide

  26. @glennsarti

    View Slide

  27. @glennsarti
    Complicated
    Complex
    Chaotic Obvious

    View Slide

  28. @glennsarti
    Ordered
    Unordered
    Complicated
    Complex
    Chaotic Obvious

    View Slide

  29. @glennsarti
    Obvious
    Sense – Categorise - Respond
    Unit Tests

    View Slide

  30. @glennsarti
    Complicated
    Sense – Analyse - Respond
    Integration Tests

    View Slide

  31. @glennsarti
    Complex
    Probe – Sense - Respond
    Acceptance Tests

    View Slide

  32. @glennsarti
    Chaotic
    Act – Sense - Respond
    Chaos Engineering

    View Slide

  33. @glennsarti
    Complicated
    Complex
    Obvious
    Unit Tests
    Integration Tests
    Acceptance Tests

    View Slide

  34. @glennsarti
    Think Different
    Change Scope

    View Slide

  35. @glennsarti
    ✔Reduce the scope
    ✔ Reduce complexity
    ✔ Reduce duration
    ✔ Reduce things that can go wrong
    ❌Reduces coverage

    View Slide

  36. @glennsarti
    – Testing in Production – Rob Crowley

    View Slide

  37. @glennsarti
    – Testing in Production – Rob Crowley

    View Slide

  38. @glennsarti
    – Testing in Production – Rob Crowley

    View Slide

  39. @glennsarti
    Let’s write
    some tests

    View Slide

  40. @glennsarti
    Introducing PoshBot
    and Mattermost …

    View Slide

  41. @glennsarti
    Mattermost
    Backend
    Mattermost
    Server
    Mattermost
    Client
    PoshBot
    Docker
    REST API
    PowerShell Web

    View Slide

  42. @glennsarti
    Mattermost
    Backend
    Mattermost
    Server
    Mattermost
    Client
    PoshBot
    OAT
    Tests

    View Slide

  43. @glennsarti
    Mattermost
    Backend
    Mattermost
    Server
    Mattermost
    Client
    PoshBot
    UAT
    Tests

    View Slide

  44. @glennsarti
    Operational Acceptance Testing
    and
    Operation Validation Testing

    View Slide

  45. @glennsarti
    Is the server running?
    Does the server think it's ok?
    Are the critical settings correct?

    View Slide

  46. @glennsarti
    $MatterMostRoot = "http://localhost:8065"
    Describe "Response from ${MatterMostRoot}" {
    it "should return statuscode 200" {
    $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing
    $Result.StatusCode | Should -Be 200 -Because '…’
    }
    it "should have a title of MatterMost" {
    $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing
    $Result.Content | Should -BeLike "*Mattermost*"
    }
    }

    View Slide

  47. @glennsarti
    $MatterMostRoot = "http://localhost:8065"
    Describe "Response from ${MatterMostRoot}" {
    it "should return statuscode 200" {
    $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing
    $Result.StatusCode | Should -Be 200 -Because '…'
    }
    it "should have a title of MatterMost" {
    $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing
    $Result.Content | Should -BeLike "*Mattermost*"
    }
    }

    View Slide

  48. @glennsarti

    View Slide

  49. @glennsarti

    View Slide

  50. @glennsarti
    Operation Validation Framework
    - https://github.com/PowerShell/Operation-Validation-Framework
    A set of tools for executing validation of the
    operation of a system. It provides a way to
    organize and execute Pester tests which are
    written to validate operation … ”

    View Slide

  51. @glennsarti
    PowerShell Module

    View Slide

  52. @glennsarti
    OVF Directories

    View Slide

  53. @glennsarti
    Tests

    View Slide

  54. @glennsarti
    param(
    $MatterMostRoot = "http://localhost:8065"
    )
    Describe "Mattermost Simple Tests" {
    Context "Response from $MatterMostRoot" {
    it "should return statuscode 200" {
    $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing
    $Result.StatusCode | Should -Be 200 -Because '…'
    }
    it "should have a title of MatterMost" {
    $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing
    $Result.Content | Should -BeLike "*Mattermost*"
    }
    }
    }

    View Slide

  55. @glennsarti
    PS> Get-OperationValidation -Path OVF
    Module: …\MatterMostTests\OAT\OVF\ExampleModule
    Version:
    Type: Simple
    Tags: {}
    File: Mattermost-Simple.Tests.ps1
    FilePath: …\Diagnostics\Simple\Mattermost-Simple.Tests.ps1
    Name:
    Mattermost Simple Tests

    View Slide

  56. @glennsarti
    PS> Invoke-OperationValidation -Path OVF
    Module: C:\Source\PSSummit-Pester102\MatterMostTests\OAT\OVF\ExampleModule
    Result Name
    ------ ----
    Passed Mattermost Simple Tests:Response from http://localhost:8065:should return statuscode 200
    Passed Mattermost Simple Tests:Response from http://localhost:8065:should have a title of MatterMost
    Passed Mattermost Comprehensive Tests:Server configuration:should return statuscode 404 for V3 API
    Passed Mattermost Comprehensive Tests:Server configuration:should report MatterMost Server versio...
    Passed Mattermost Comprehensive Tests:Status endpoint at http://localhost:8065/api/v4/system/ping...
    Passed Mattermost Comprehensive Tests:Status endpoint at http://localhost:8065/api/v4/system/ping...
    Passed Mattermost Comprehensive Tests:Users API endpoint:should have more than one user

    View Slide

  57. @glennsarti
    OVF Gotchas (1.2.0)
    Doesn’t require a PSD file
    Single Describe block and text only
    Use the newest version

    View Slide

  58. @glennsarti
    Beyond OVF…
    Sharing
    Tooling
    ChatOps-ing
    - Devopsing In a Microsoft World

    View Slide

  59. @glennsarti
    Compliance
    Acceptance
    Testing

    View Slide

  60. @glennsarti

    View Slide

  61. @glennsarti

    View Slide

  62. @glennsarti
    Metadata
    Test

    View Slide

  63. @glennsarti
    $MatterMostRoot = "http://localhost:8065"
    Describe "Response from ${MatterMostRoot}" {
    it "should return statuscode 200" {
    $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing
    $Result.StatusCode | Should -Be 200 -Because '…’
    }
    it "should have a title of MatterMost" {
    $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing
    $Result.Content | Should -BeLike "*Mattermost*"
    }
    }

    View Slide

  64. @glennsarti
    mattermost_port = 8065
    mattermost_root = "http://localhost:#{mattermost_port}"
    control 'mattermost-basic-1' do
    impact 'critical'
    title 'Mattermost Server: Basic Connectivity'
    desc 'Basic connectivity …'
    tag 'mattermost', 'basic'
    # Basic Connectivity Tests
    describe port(mattermost_port) do
    it { should be_listening }
    end
    describe http(mattermost_root) do
    its('status') { should eq 200 }
    its('body') { should match 'Mattermost' }
    end
    end

    View Slide

  65. @glennsarti
    mattermost_port = 8065
    mattermost_root = "http://localhost:#{mattermost_port}"
    control 'mattermost-basic-1' do
    impact 'critical'
    title 'Mattermost Server: Basic Connectivity'
    desc 'Basic connectivity …'
    tag 'mattermost', 'basic'
    # Basic Connectivity Tests
    describe port(mattermost_port) do
    it { should be_listening }
    end
    describe http(mattermost_root) do
    its('status') { should eq 200 }
    its('body') { should match 'Mattermost' }
    end
    end

    View Slide

  66. @glennsarti
    Inspec
    Requires an installation package
    $Inspec -ne $rspec
    Windows support can be variable

    View Slide

  67. @glennsarti
    Introducing Inspecster!

    View Slide

  68. @glennsarti
    control 'mattermost-basic-1' `
    -impact 'critical' `
    -title 'Mattermost Server: Basic Connectivity' `
    -desc 'Basic connectivity tests to test …' `
    -tags 'mattermost', 'basic' {
    describe "Response from ${MatterMostRoot}" {
    it "should return statuscode 200" {
    $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing
    $Result.StatusCode | Should -Be 200 -Because '…'
    }

    View Slide

  69. View Slide

  70. View Slide

  71. View Slide

  72. @glennsarti
    < take a breath >

    View Slide

  73. @glennsarti
    User Acceptance Testing

    View Slide

  74. @glennsarti
    Mattermost
    Backend
    Mattermost
    Server
    PoshBot
    UAT
    Tests

    View Slide

  75. @glennsarti
    UAT
    Create systems under test
    Run tests
    Destroy systems under test
    1
    2
    3

    View Slide

  76. @glennsarti
    Run docker image
    Configure the Mattermost server
    Create token for the Bot
    Run PoshBot as a different process
    Wait for Poshbot to be ready

    View Slide

  77. @glennsarti
    1

    View Slide

  78. @glennsarti

    View Slide

  79. @glennsarti
    it "should respond to '! help'" {
    $userInfo = Get-UserInformation
    # Post a Message
    $body = @{
    'channel_id' = $userInfo.channelid
    'message' = "! help"
    }
    $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body
    # Give the bot time to respond
    Start-Sleep 2
    $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" `
    -Method GET
    # The Bot should have a reaction on the message
    $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0
    $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark'
    $newMessages = Invoke-MattermostAPI `
    -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" `
    -Method GET

    View Slide

  80. @glennsarti
    it "should respond to '! help'" {
    $userInfo = Get-UserInformation
    # Post a Message
    $body = @{
    'channel_id' = $userInfo.channelid
    'message' = "! help"
    }
    $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body
    # Give the bot time to respond
    Start-Sleep 2
    $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" `
    -Method GET
    # The Bot should have a reaction on the message
    $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0
    $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark'
    $newMessages = Invoke-MattermostAPI `
    -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" `
    -Method GET

    View Slide

  81. @glennsarti
    it "should respond to '! help'" {
    $userInfo = Get-UserInformation
    # Post a Message
    $body = @{
    'channel_id' = $userInfo.channelid
    'message' = "! help"
    }
    $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body
    # Give the bot time to respond
    Start-Sleep 2
    $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" `
    -Method GET
    # The Bot should have a reaction on the message
    $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0
    $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark’
    $newMessages = Invoke-MattermostAPI `
    -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" `
    -Method GET

    View Slide

  82. @glennsarti
    it "should respond to '! help'" {
    $userInfo = Get-UserInformation
    # Post a Message
    $body = @{
    'channel_id' = $userInfo.channelid
    'message' = "! help"
    }
    $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body
    # Give the bot time to respond
    Start-Sleep 2
    $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" `
    -Method GET
    # The Bot should have a reaction on the message
    $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0
    $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark'
    $newMessages = Invoke-MattermostAPI `
    -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" `
    -Method GET

    View Slide

  83. @glennsarti
    # The Bot should have a reaction on the message
    $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0
    $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark'
    $newMessages = Invoke-MattermostAPI `
    -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" `
    -Method GET
    # There should only be one new message
    $newMessages.order.Count | Should -Be 1
    $newPostID = $newMessages.order[0]
    $newPost = $newMessages.posts."$newPostID"
    # The message text should start with...
    $newPost.message | Should -Match "```````nFullCommandName"
    }

    View Slide

  84. @glennsarti
    # The Bot should have a reaction on the message
    $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0
    $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark'
    $newMessages = Invoke-MattermostAPI `
    -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" `
    -Method GET
    # There should only be one new message
    $newMessages.order.Count | Should -Be 1
    $newPostID = $newMessages.order[0]
    $newPost = $newMessages.posts."$newPostID"
    # The message text should start with...
    $newPost.message | Should -Match "```````nFullCommandName"
    }

    View Slide

  85. @glennsarti

    View Slide

  86. @glennsarti
    Stop and remove the container
    Stop the Poshbot Process

    View Slide

  87. @glennsarti
    try {
    & .\Pre-Pester.ps1
    # Add other options e.g. Output file and format
    $result = Invoke-Pester -Script $PSScriptRoot –PassThru
    }
    finally {
    & .\Post-Pester.ps1
    }
    # Return the number of failed tests. Exit Code 0 = Success
    $host.SetShouldExit($result.failedcount)
    1
    2
    3
    4

    View Slide

  88. @glennsarti
    Thinking more different-er …

    View Slide

  89. @glennsarti
    Gherkin

    View Slide

  90. @glennsarti
    it "should respond to '! help'" {
    $userInfo = Get-UserInformation
    # Post a Message
    $body = @{
    'channel_id' = $userInfo.channelid
    'message' = "! help"
    }
    $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body
    # Give the bot time to respond
    Start-Sleep 2
    $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" `
    -Method GET
    # The Bot should have a reaction on the message
    $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0
    $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark'
    $newMessages = Invoke-MattermostAPI `
    -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" `
    -Method GET
    # The Bot should have a reaction on the message
    $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0
    $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark'
    $newMessages = Invoke-MattermostAPI `
    -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" `
    -Method GET
    # There should only be one new message
    $newMessages.order.Count | Should -Be 1
    $newPostID = $newMessages.order[0]
    $newPost = $newMessages.posts."$newPostID"
    # The message text should start with...
    $newPost.message | Should -Match "```````nFullCommandName"
    }

    View Slide

  91. @glennsarti
    Feature: It can respond to help messages
    Background: Login to Mattermost
    Given mattermost server http://localhost:8065
    And mattermost user '[email protected]' with passsword 'Password1'
    Scenario: Sending a valid help message
    When sending a message of ! help
    And waiting 2 seconds
    Then the message should have a reaction of white_check_mark
    And poshbot returns a message that starts with ```````nFullCommandName

    View Slide

  92. @glennsarti
    Feature: It can respond to help messages
    Background: Login to Mattermost
    Given mattermost server http://localhost:8065
    And mattermost user '[email protected]' with passsword 'Password1'
    Scenario: Sending a valid help message
    When sending a message of ! help
    And waiting 2 seconds
    Then the message should have a reaction of white_check_mark
    And poshbot returns a message that starts with ```````nFullCommandName

    View Slide

  93. @glennsarti
    Feature: It can respond to help messages
    Background: Login to Mattermost
    Given mattermost server http://localhost:8065
    And mattermost user '[email protected]' with passsword 'Password1'
    Scenario: Sending a valid help message
    When sending a message of ! help
    And waiting 5 seconds
    Then the message should have a reaction of white_check_mark
    And poshbot returns a message that starts with ```````nFullCommandName

    View Slide

  94. @glennsarti
    Scenario: Sending a help message with an unknown command
    When sending a message of ! help commanddoesntexist
    And waiting 2 seconds
    Then the message should have a reaction of white_check_mark
    And poshbot returns a message that contains No commands found

    View Slide

  95. @glennsarti
    Tiering

    View Slide

  96. @glennsarti
    Instrumentation

    View Slide

  97. @glennsarti
    $Local -eq $Automated

    View Slide

  98. @glennsarti

    View Slide

  99. @glennsarti

    View Slide

  100. THANK YOU!
    Please use the event app or Socio to submit a session rating!
    @glennsarti
    http://glennsarti.github.io/
    https://speakerdeck.com/glennsarti
    https://github.com/glennsarti/PSSummitNA2019-Pester102

    View Slide

  101. @glennsarti
    Resources
    Source Code
    https://github.com/glennsarti/PSSummitNA2019-Pester102
    Cynefin
    https://www.youtube.com/watch?v=N7oz366X0-8
    https://en.wikipedia.org/wiki/Cynefin_framework#cite_note-SnowdenBoone2007-1
    http://cognitive-edge.com/videos/cynefin-framework-introduction/
    Rob Crowley – Testing in production
    https://speakerdeck.com/robcrowley/testing-in-production-for-the-bold-and-true
    Mattermost
    https://api.mattermost.com/
    Images
    https:/
    /unsplash.com

    View Slide

  102. @glennsarti
    Resources
    Operation Validation Framework
    https://github.com/PowerShell/Operation-Validation-Framework
    Devopsing In a Microsoft World
    https://youtu.be/roFVOnRxXns?t=1500
    https://www.slideshare.net/snasello1/devops-days-seattle-2017
    InSpec
    https://www.inspec.io/

    View Slide

  103. @glennsarti
    Resources
    Gherkin
    https://docs.cucumber.io/gherkin/
    https://powershellexplained.com/2017-03-17-Powershell-Gherkin-specification-validation/
    https://powershellexplained.com/2017-04-30-Powershell-Gherkin-advanced-features/
    https://www.youtube.com/watch?v=hCfcLDLTf6g Part 1
    https://www.youtube.com/watch?v=mWFkFU4Hi_g Part 2
    https://github.com/mikejonestechno/devops/tree/master/powershell/demo-bdd

    View Slide