Beyond Pester 102: Acceptance testing with PowerShell

Previously in "Beyond Pester 101: Applying testing principles to PowerShell" I talked about unit and integration testing with Pester and PowerShell. Now that we're all experts there, it's time to tackle Acceptance testing, also known as End to End testing.

Transcript

1. @glennsarti glenn-sarti
   

   View Slide

2. @glennsarti
   Previously …
   

   View Slide

3. @glennsarti
   Unit (White box)
   Integration
   Unit (Black box)
   Characterisation
   Reduce duration
   Remove tests
   Reorder tests
   General
   maintenance
   

   View Slide

4. @glennsarti
   Types of tests - Integration
   Image adapted from Test Pyramid – Martin Fowler
   1. Slower
   2. More $$$
   3. Complex
   

   View Slide

5. @glennsarti
   Why do I test?
   

   View Slide

6. @glennsarti
   To reduce the risk that a
   user will experience an
   unexpected behaviour
   

   View Slide

7. @glennsarti
   To reduce the risk that a
   user will experience an
   unexpected behaviour
   

   View Slide

8. @glennsarti
   Benefits of testing
   Shorter delivery cycles
   Produce high quality code
   Cheaper
   

   View Slide

9. @glennsarti
   To ensure what is created does
   what it’s supposed to do
   Form of documentation
   Benefits of testing
   

   View Slide

10. @glennsarti
    

    View Slide

11. @glennsarti
    Image adapted from Test Pyramid – Martin Fowler
    

    View Slide

12. @glennsarti
    Image adapted from Test Pyramid – Martin Fowler
    ⚠️ Slow
    

    View Slide

13. @glennsarti
    Image adapted from Test Pyramid – Martin Fowler
    ⚠️ Expensive
    

    View Slide

14. @glennsarti
    Image adapted from Test Pyramid – Martin Fowler
    ⚠️ Complex
    

    View Slide

15. @glennsarti
    The cost to fix a bug
    Cost
    to
    Fix
    – Testing in Production – Rob Crowley
    Development Manual Testing From user
    Expensive
    Cheap
    

    View Slide

16. @glennsarti
    Fixing a broken test
    Acceptance Tests
    Integration Tests
    Unit Tests
    PowerShell
    being tested
    PAIN
    
    

    View Slide

17. @glennsarti
    Things that can go wrong
    One
    Plethora
    (*) Graph is not to scale or based on actual data
    Acceptance Tests
    Integration Tests
    Unit Tests
    

    View Slide

18. @glennsarti
    ❌
    ✔
    

    View Slide

19. @glennsarti
    Customers don’t measure you on
    how hard you tried, they measure
    you on what you deliver.”
    “
    - Steve Jobs
    

    View Slide

20. @glennsarti
    Where’s the value?
    

    View Slide

21. @glennsarti
    … there are known knowns …
    known unknowns … also unknown
    unknowns … it is the latter category
    that tend to be the difficult ones
    - Donald Rumsfeld
    “
    ”
    

    View Slide

22. @glennsarti
    What can we do then…
    

    View Slide

23. @glennsarti
    Think Different
    Change Scope
    

    View Slide

24. @glennsarti
    Think Different
    Change Scope
    

    View Slide

25. @glennsarti
    Cynefin Framework
    “The Cynefin framework (/ˈkʌnɪvɪn/
    KUN-iv-in) is a conceptual framework
    used to aid decision-making”
    - https://www.youtube.com/watch?v=N7oz366X0-8
    

    View Slide

26. @glennsarti
    

    View Slide

27. @glennsarti
    Complicated
    Complex
    Chaotic Obvious
    

    View Slide

28. @glennsarti
    Ordered
    Unordered
    Complicated
    Complex
    Chaotic Obvious
    

    View Slide

29. @glennsarti
    Obvious
    Sense – Categorise - Respond
    Unit Tests
    

    View Slide

30. @glennsarti
    Complicated
    Sense – Analyse - Respond
    Integration Tests
    

    View Slide

31. @glennsarti
    Complex
    Probe – Sense - Respond
    Acceptance Tests
    

    View Slide

32. @glennsarti
    Chaotic
    Act – Sense - Respond
    Chaos Engineering
    

    View Slide

33. @glennsarti
    Complicated
    Complex
    Obvious
    Unit Tests
    Integration Tests
    Acceptance Tests
    

    View Slide

34. @glennsarti
    Think Different
    Change Scope
    

    View Slide

35. @glennsarti
    ✔Reduce the scope
    ✔ Reduce complexity
    ✔ Reduce duration
    ✔ Reduce things that can go wrong
    ❌Reduces coverage
    

    View Slide

36. @glennsarti
    – Testing in Production – Rob Crowley
    

    View Slide

37. @glennsarti
    – Testing in Production – Rob Crowley
    

    View Slide

38. @glennsarti
    – Testing in Production – Rob Crowley
    

    View Slide

39. @glennsarti
    Let’s write
    some tests
    

    View Slide

40. @glennsarti
    Introducing PoshBot
    and Mattermost …
    

    View Slide

41. @glennsarti
    Mattermost
    Backend
    Mattermost
    Server
    Mattermost
    Client
    PoshBot
    Docker
    REST API
    PowerShell Web
    

    View Slide

42. @glennsarti
    Mattermost
    Backend
    Mattermost
    Server
    Mattermost
    Client
    PoshBot
    OAT
    Tests
    

    View Slide

43. @glennsarti
    Mattermost
    Backend
    Mattermost
    Server
    Mattermost
    Client
    PoshBot
    UAT
    Tests
    

    View Slide

44. @glennsarti
    Operational Acceptance Testing
    and
    Operation Validation Testing
    

    View Slide

45. @glennsarti
    Is the server running?
    Does the server think it's ok?
    Are the critical settings correct?
    

    View Slide

46. @glennsarti
    $MatterMostRoot = "http://localhost:8065"
    Describe "Response from ${MatterMostRoot}" {
    it "should return statuscode 200" {
    $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing
    $Result.StatusCode | Should -Be 200 -Because '…’
    }
    it "should have a title of MatterMost" {
    $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing
    $Result.Content | Should -BeLike "*Mattermost*"
    }
    }
    

    View Slide

47. @glennsarti
    $MatterMostRoot = "http://localhost:8065"
    Describe "Response from ${MatterMostRoot}" {
    it "should return statuscode 200" {
    $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing
    $Result.StatusCode | Should -Be 200 -Because '…'
    }
    it "should have a title of MatterMost" {
    $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing
    $Result.Content | Should -BeLike "*Mattermost*"
    }
    }
    

    View Slide

48. @glennsarti
    

    View Slide

49. @glennsarti
    

    View Slide

50. @glennsarti
    Operation Validation Framework
    - https://github.com/PowerShell/Operation-Validation-Framework
    A set of tools for executing validation of the
    operation of a system. It provides a way to
    organize and execute Pester tests which are
    written to validate operation … ”
    “
    

    View Slide

51. @glennsarti
    PowerShell Module
    

    View Slide

52. @glennsarti
    OVF Directories
    

    View Slide

53. @glennsarti
    Tests
    

    View Slide

54. @glennsarti
    param(
    $MatterMostRoot = "http://localhost:8065"
    )
    Describe "Mattermost Simple Tests" {
    Context "Response from $MatterMostRoot" {
    it "should return statuscode 200" {
    $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing
    $Result.StatusCode | Should -Be 200 -Because '…'
    }
    it "should have a title of MatterMost" {
    $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing
    $Result.Content | Should -BeLike "*Mattermost*"
    }
    }
    }
    

    View Slide

55. @glennsarti
    PS> Get-OperationValidation -Path OVF
    Module: …\MatterMostTests\OAT\OVF\ExampleModule
    Version:
    Type: Simple
    Tags: {}
    File: Mattermost-Simple.Tests.ps1
    FilePath: …\Diagnostics\Simple\Mattermost-Simple.Tests.ps1
    Name:
    Mattermost Simple Tests
    …
    

    View Slide

56. @glennsarti
    PS> Invoke-OperationValidation -Path OVF
    Module: C:\Source\PSSummit-Pester102\MatterMostTests\OAT\OVF\ExampleModule
    Result Name
    ------ ----
    Passed Mattermost Simple Tests:Response from http://localhost:8065:should return statuscode 200
    Passed Mattermost Simple Tests:Response from http://localhost:8065:should have a title of MatterMost
    Passed Mattermost Comprehensive Tests:Server configuration:should return statuscode 404 for V3 API
    Passed Mattermost Comprehensive Tests:Server configuration:should report MatterMost Server versio...
    Passed Mattermost Comprehensive Tests:Status endpoint at http://localhost:8065/api/v4/system/ping...
    Passed Mattermost Comprehensive Tests:Status endpoint at http://localhost:8065/api/v4/system/ping...
    Passed Mattermost Comprehensive Tests:Users API endpoint:should have more than one user
    

    View Slide

57. @glennsarti
    OVF Gotchas (1.2.0)
    Doesn’t require a PSD file
    Single Describe block and text only
    Use the newest version
    

    View Slide

58. @glennsarti
    Beyond OVF…
    Sharing
    Tooling
    ChatOps-ing
    - Devopsing In a Microsoft World
    

    View Slide

59. @glennsarti
    Compliance
    Acceptance
    Testing
    

    View Slide

60. @glennsarti
    

    View Slide

61. @glennsarti
    

    View Slide

62. @glennsarti
    Metadata
    Test
    

    View Slide

63. @glennsarti
    $MatterMostRoot = "http://localhost:8065"
    Describe "Response from ${MatterMostRoot}" {
    it "should return statuscode 200" {
    $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing
    $Result.StatusCode | Should -Be 200 -Because '…’
    }
    it "should have a title of MatterMost" {
    $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing
    $Result.Content | Should -BeLike "*Mattermost*"
    }
    }
    

    View Slide

64. @glennsarti
    mattermost_port = 8065
    mattermost_root = "http://localhost:#{mattermost_port}"
    control 'mattermost-basic-1' do
    impact 'critical'
    title 'Mattermost Server: Basic Connectivity'
    desc 'Basic connectivity …'
    tag 'mattermost', 'basic'
    # Basic Connectivity Tests
    describe port(mattermost_port) do
    it { should be_listening }
    end
    describe http(mattermost_root) do
    its('status') { should eq 200 }
    its('body') { should match 'Mattermost' }
    end
    end
    

    View Slide

65. @glennsarti
    mattermost_port = 8065
    mattermost_root = "http://localhost:#{mattermost_port}"
    control 'mattermost-basic-1' do
    impact 'critical'
    title 'Mattermost Server: Basic Connectivity'
    desc 'Basic connectivity …'
    tag 'mattermost', 'basic'
    # Basic Connectivity Tests
    describe port(mattermost_port) do
    it { should be_listening }
    end
    describe http(mattermost_root) do
    its('status') { should eq 200 }
    its('body') { should match 'Mattermost' }
    end
    end
    

    View Slide

66. @glennsarti
    Inspec
    Requires an installation package
    $Inspec -ne $rspec
    Windows support can be variable
    

    View Slide

67. @glennsarti
    Introducing Inspecster!
    

    View Slide

68. @glennsarti
    control 'mattermost-basic-1' `
    -impact 'critical' `
    -title 'Mattermost Server: Basic Connectivity' `
    -desc 'Basic connectivity tests to test …' `
    -tags 'mattermost', 'basic' {
    describe "Response from ${MatterMostRoot}" {
    it "should return statuscode 200" {
    $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing
    $Result.StatusCode | Should -Be 200 -Because '…'
    }
    …
    

    View Slide

69. View Slide

70. View Slide

71. View Slide

72. @glennsarti
    < take a breath >
    

    View Slide

73. @glennsarti
    User Acceptance Testing
    

    View Slide

74. @glennsarti
    Mattermost
    Backend
    Mattermost
    Server
    PoshBot
    UAT
    Tests
    

    View Slide

75. @glennsarti
    UAT
    Create systems under test
    Run tests
    Destroy systems under test
    1
    2
    3
    

    View Slide

76. @glennsarti
    Run docker image
    Configure the Mattermost server
    Create token for the Bot
    Run PoshBot as a different process
    Wait for Poshbot to be ready
    

    View Slide

77. @glennsarti
    1
    

    View Slide

78. @glennsarti
    

    View Slide

79. @glennsarti
    it "should respond to '! help'" {
    $userInfo = Get-UserInformation
    # Post a Message
    $body = @{
    'channel_id' = $userInfo.channelid
    'message' = "! help"
    }
    $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body
    # Give the bot time to respond
    Start-Sleep 2
    $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" `
    -Method GET
    # The Bot should have a reaction on the message
    $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0
    $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark'
    $newMessages = Invoke-MattermostAPI `
    -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" `
    -Method GET
    

    View Slide

80. @glennsarti
    it "should respond to '! help'" {
    $userInfo = Get-UserInformation
    # Post a Message
    $body = @{
    'channel_id' = $userInfo.channelid
    'message' = "! help"
    }
    $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body
    # Give the bot time to respond
    Start-Sleep 2
    $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" `
    -Method GET
    # The Bot should have a reaction on the message
    $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0
    $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark'
    $newMessages = Invoke-MattermostAPI `
    -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" `
    -Method GET
    

    View Slide

81. @glennsarti
    it "should respond to '! help'" {
    $userInfo = Get-UserInformation
    # Post a Message
    $body = @{
    'channel_id' = $userInfo.channelid
    'message' = "! help"
    }
    $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body
    # Give the bot time to respond
    Start-Sleep 2
    $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" `
    -Method GET
    # The Bot should have a reaction on the message
    $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0
    $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark’
    $newMessages = Invoke-MattermostAPI `
    -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" `
    -Method GET
    

    View Slide

82. @glennsarti
    it "should respond to '! help'" {
    $userInfo = Get-UserInformation
    # Post a Message
    $body = @{
    'channel_id' = $userInfo.channelid
    'message' = "! help"
    }
    $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body
    # Give the bot time to respond
    Start-Sleep 2
    $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" `
    -Method GET
    # The Bot should have a reaction on the message
    $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0
    $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark'
    $newMessages = Invoke-MattermostAPI `
    -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" `
    -Method GET
    

    View Slide

83. @glennsarti
    # The Bot should have a reaction on the message
    $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0
    $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark'
    $newMessages = Invoke-MattermostAPI `
    -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" `
    -Method GET
    # There should only be one new message
    $newMessages.order.Count | Should -Be 1
    $newPostID = $newMessages.order[0]
    $newPost = $newMessages.posts."$newPostID"
    # The message text should start with...
    $newPost.message | Should -Match "```````nFullCommandName"
    }
    

    View Slide

84. @glennsarti
    # The Bot should have a reaction on the message
    $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0
    $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark'
    $newMessages = Invoke-MattermostAPI `
    -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" `
    -Method GET
    # There should only be one new message
    $newMessages.order.Count | Should -Be 1
    $newPostID = $newMessages.order[0]
    $newPost = $newMessages.posts."$newPostID"
    # The message text should start with...
    $newPost.message | Should -Match "```````nFullCommandName"
    }
    

    View Slide

85. @glennsarti
    

    View Slide

86. @glennsarti
    Stop and remove the container
    Stop the Poshbot Process
    

    View Slide

87. @glennsarti
    try {
    & .\Pre-Pester.ps1
    # Add other options e.g. Output file and format
    $result = Invoke-Pester -Script $PSScriptRoot –PassThru
    }
    finally {
    & .\Post-Pester.ps1
    }
    # Return the number of failed tests. Exit Code 0 = Success
    $host.SetShouldExit($result.failedcount)
    1
    2
    3
    4
    

    View Slide

88. @glennsarti
    Thinking more different-er …
    

    View Slide

89. @glennsarti
    Gherkin
    

    View Slide

90. @glennsarti
    it "should respond to '! help'" {
    $userInfo = Get-UserInformation
    # Post a Message
    $body = @{
    'channel_id' = $userInfo.channelid
    'message' = "! help"
    }
    $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body
    # Give the bot time to respond
    Start-Sleep 2
    $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" `
    -Method GET
    # The Bot should have a reaction on the message
    $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0
    $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark'
    $newMessages = Invoke-MattermostAPI `
    -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" `
    -Method GET
    # The Bot should have a reaction on the message
    $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0
    $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark'
    $newMessages = Invoke-MattermostAPI `
    -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" `
    -Method GET
    # There should only be one new message
    $newMessages.order.Count | Should -Be 1
    $newPostID = $newMessages.order[0]
    $newPost = $newMessages.posts."$newPostID"
    # The message text should start with...
    $newPost.message | Should -Match "```````nFullCommandName"
    }
    

    View Slide

91. @glennsarti
    Feature: It can respond to help messages
    Background: Login to Mattermost
    Given mattermost server http://localhost:8065
    And mattermost user 'user2@example.com' with passsword 'Password1'
    Scenario: Sending a valid help message
    When sending a message of ! help
    And waiting 2 seconds
    Then the message should have a reaction of white_check_mark
    And poshbot returns a message that starts with ```````nFullCommandName
    

    View Slide

92. @glennsarti
    Feature: It can respond to help messages
    Background: Login to Mattermost
    Given mattermost server http://localhost:8065
    And mattermost user 'user2@example.com' with passsword 'Password1'
    Scenario: Sending a valid help message
    When sending a message of ! help
    And waiting 2 seconds
    Then the message should have a reaction of white_check_mark
    And poshbot returns a message that starts with ```````nFullCommandName
    

    View Slide

93. @glennsarti
    Feature: It can respond to help messages
    Background: Login to Mattermost
    Given mattermost server http://localhost:8065
    And mattermost user 'user2@example.com' with passsword 'Password1'
    Scenario: Sending a valid help message
    When sending a message of ! help
    And waiting 5 seconds
    Then the message should have a reaction of white_check_mark
    And poshbot returns a message that starts with ```````nFullCommandName
    

    View Slide

94. @glennsarti
    Scenario: Sending a help message with an unknown command
    When sending a message of ! help commanddoesntexist
    And waiting 2 seconds
    Then the message should have a reaction of white_check_mark
    And poshbot returns a message that contains No commands found
    

    View Slide

95. @glennsarti
    Tiering
    

    View Slide

96. @glennsarti
    Instrumentation
    

    View Slide

97. @glennsarti
    $Local -eq $Automated
    

    View Slide

98. @glennsarti
    

    View Slide

99. @glennsarti
    

    View Slide

100. THANK YOU!
     Please use the event app or Socio to submit a session rating!
     @glennsarti
     http://glennsarti.github.io/
     https://speakerdeck.com/glennsarti
     https://github.com/glennsarti/PSSummitNA2019-Pester102
     

     View Slide

101. @glennsarti
     Resources
     Source Code
     https://github.com/glennsarti/PSSummitNA2019-Pester102
     Cynefin
     https://www.youtube.com/watch?v=N7oz366X0-8
     https://en.wikipedia.org/wiki/Cynefin_framework#cite_note-SnowdenBoone2007-1
     http://cognitive-edge.com/videos/cynefin-framework-introduction/
     Rob Crowley – Testing in production
     https://speakerdeck.com/robcrowley/testing-in-production-for-the-bold-and-true
     Mattermost
     https://api.mattermost.com/
     Images
     https:/
     /unsplash.com
     

     View Slide

102. @glennsarti
     Resources
     Operation Validation Framework
     https://github.com/PowerShell/Operation-Validation-Framework
     Devopsing In a Microsoft World
     https://youtu.be/roFVOnRxXns?t=1500
     https://www.slideshare.net/snasello1/devops-days-seattle-2017
     InSpec
     https://www.inspec.io/
     

     View Slide

103. @glennsarti
     Resources
     Gherkin
     https://docs.cucumber.io/gherkin/
     https://powershellexplained.com/2017-03-17-Powershell-Gherkin-specification-validation/
     https://powershellexplained.com/2017-04-30-Powershell-Gherkin-advanced-features/
     https://www.youtube.com/watch?v=hCfcLDLTf6g Part 1
     https://www.youtube.com/watch?v=mWFkFU4Hi_g Part 2
     https://github.com/mikejonestechno/devops/tree/master/powershell/demo-bdd
     

     View Slide