Beyond Pester 102: Acceptance testing with PowerShell

Aac3dafaab7a7c2063d2526ba5936305?s=47 Glenn
April 26, 2019

Beyond Pester 102: Acceptance testing with PowerShell

Previously in "Beyond Pester 101: Applying testing principles to PowerShell" I talked about unit and integration testing with Pester and PowerShell. Now that we're all experts there, it's time to tackle Acceptance testing, also known as End to End testing.

Aac3dafaab7a7c2063d2526ba5936305?s=128

Glenn

April 26, 2019
Tweet

Transcript

  1. @glennsarti glenn-sarti

  2. @glennsarti Previously …

  3. @glennsarti Unit (White box) Integration Unit (Black box) Characterisation Reduce

    duration Remove tests Reorder tests General maintenance
  4. @glennsarti Types of tests - Integration Image adapted from Test

    Pyramid – Martin Fowler 1. Slower 2. More $$$ 3. Complex
  5. @glennsarti Why do I test?

  6. @glennsarti To reduce the risk that a user will experience

    an unexpected behaviour
  7. @glennsarti To reduce the risk that a user will experience

    an unexpected behaviour
  8. @glennsarti Benefits of testing Shorter delivery cycles Produce high quality

    code Cheaper
  9. @glennsarti To ensure what is created does what it’s supposed

    to do Form of documentation Benefits of testing
  10. @glennsarti

  11. @glennsarti Image adapted from Test Pyramid – Martin Fowler

  12. @glennsarti Image adapted from Test Pyramid – Martin Fowler ⚠️

    Slow
  13. @glennsarti Image adapted from Test Pyramid – Martin Fowler ⚠️

    Expensive
  14. @glennsarti Image adapted from Test Pyramid – Martin Fowler ⚠️

    Complex
  15. @glennsarti The cost to fix a bug Cost to Fix

    – Testing in Production – Rob Crowley Development Manual Testing From user Expensive Cheap
  16. @glennsarti Fixing a broken test Acceptance Tests Integration Tests Unit

    Tests PowerShell being tested PAIN
  17. @glennsarti Things that can go wrong One Plethora (*) Graph

    is not to scale or based on actual data Acceptance Tests Integration Tests Unit Tests
  18. @glennsarti ❌ ✔

  19. @glennsarti Customers don’t measure you on how hard you tried,

    they measure you on what you deliver.” “ - Steve Jobs
  20. @glennsarti Where’s the value?

  21. @glennsarti … there are known knowns … known unknowns …

    also unknown unknowns … it is the latter category that tend to be the difficult ones - Donald Rumsfeld “ ”
  22. @glennsarti What can we do then…

  23. @glennsarti Think Different Change Scope

  24. @glennsarti Think Different Change Scope

  25. @glennsarti Cynefin Framework “The Cynefin framework (/ˈkʌnɪvɪn/ KUN-iv-in) is a

    conceptual framework used to aid decision-making” - https://www.youtube.com/watch?v=N7oz366X0-8
  26. @glennsarti

  27. @glennsarti Complicated Complex Chaotic Obvious

  28. @glennsarti Ordered Unordered Complicated Complex Chaotic Obvious

  29. @glennsarti Obvious Sense – Categorise - Respond Unit Tests

  30. @glennsarti Complicated Sense – Analyse - Respond Integration Tests

  31. @glennsarti Complex Probe – Sense - Respond Acceptance Tests

  32. @glennsarti Chaotic Act – Sense - Respond Chaos Engineering

  33. @glennsarti Complicated Complex Obvious Unit Tests Integration Tests Acceptance Tests

  34. @glennsarti Think Different Change Scope

  35. @glennsarti ✔Reduce the scope ✔ Reduce complexity ✔ Reduce duration

    ✔ Reduce things that can go wrong ❌Reduces coverage
  36. @glennsarti – Testing in Production – Rob Crowley

  37. @glennsarti – Testing in Production – Rob Crowley

  38. @glennsarti – Testing in Production – Rob Crowley

  39. @glennsarti Let’s write some tests

  40. @glennsarti Introducing PoshBot and Mattermost …

  41. @glennsarti Mattermost Backend Mattermost Server Mattermost Client PoshBot Docker REST

    API PowerShell Web
  42. @glennsarti Mattermost Backend Mattermost Server Mattermost Client PoshBot OAT Tests

  43. @glennsarti Mattermost Backend Mattermost Server Mattermost Client PoshBot UAT Tests

  44. @glennsarti Operational Acceptance Testing and Operation Validation Testing

  45. @glennsarti Is the server running? Does the server think it's

    ok? Are the critical settings correct?
  46. @glennsarti $MatterMostRoot = "http://localhost:8065" Describe "Response from ${MatterMostRoot}" { it

    "should return statuscode 200" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.StatusCode | Should -Be 200 -Because '…’ } it "should have a title of MatterMost" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.Content | Should -BeLike "*<title>Mattermost</title>*" } }
  47. @glennsarti $MatterMostRoot = "http://localhost:8065" Describe "Response from ${MatterMostRoot}" { it

    "should return statuscode 200" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.StatusCode | Should -Be 200 -Because '…' } it "should have a title of MatterMost" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.Content | Should -BeLike "*<title>Mattermost</title>*" } }
  48. @glennsarti

  49. @glennsarti

  50. @glennsarti Operation Validation Framework - https://github.com/PowerShell/Operation-Validation-Framework A set of tools

    for executing validation of the operation of a system. It provides a way to organize and execute Pester tests which are written to validate operation … ” “
  51. @glennsarti PowerShell Module

  52. @glennsarti OVF Directories

  53. @glennsarti Tests

  54. @glennsarti param( $MatterMostRoot = "http://localhost:8065" ) Describe "Mattermost Simple Tests"

    { Context "Response from $MatterMostRoot" { it "should return statuscode 200" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.StatusCode | Should -Be 200 -Because '…' } it "should have a title of MatterMost" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.Content | Should -BeLike "*<title>Mattermost</title>*" } } }
  55. @glennsarti PS> Get-OperationValidation -Path OVF Module: …\MatterMostTests\OAT\OVF\ExampleModule Version: Type: Simple

    Tags: {} File: Mattermost-Simple.Tests.ps1 FilePath: …\Diagnostics\Simple\Mattermost-Simple.Tests.ps1 Name: Mattermost Simple Tests …
  56. @glennsarti PS> Invoke-OperationValidation -Path OVF Module: C:\Source\PSSummit-Pester102\MatterMostTests\OAT\OVF\ExampleModule Result Name ------

    ---- Passed Mattermost Simple Tests:Response from http://localhost:8065:should return statuscode 200 Passed Mattermost Simple Tests:Response from http://localhost:8065:should have a title of MatterMost Passed Mattermost Comprehensive Tests:Server configuration:should return statuscode 404 for V3 API Passed Mattermost Comprehensive Tests:Server configuration:should report MatterMost Server versio... Passed Mattermost Comprehensive Tests:Status endpoint at http://localhost:8065/api/v4/system/ping... Passed Mattermost Comprehensive Tests:Status endpoint at http://localhost:8065/api/v4/system/ping... Passed Mattermost Comprehensive Tests:Users API endpoint:should have more than one user
  57. @glennsarti OVF Gotchas (1.2.0) Doesn’t require a PSD file Single

    Describe block and text only Use the newest version
  58. @glennsarti Beyond OVF… Sharing Tooling ChatOps-ing - Devopsing In a

    Microsoft World
  59. @glennsarti Compliance Acceptance Testing

  60. @glennsarti

  61. @glennsarti

  62. @glennsarti Metadata Test

  63. @glennsarti $MatterMostRoot = "http://localhost:8065" Describe "Response from ${MatterMostRoot}" { it

    "should return statuscode 200" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.StatusCode | Should -Be 200 -Because '…’ } it "should have a title of MatterMost" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.Content | Should -BeLike "*<title>Mattermost</title>*" } }
  64. @glennsarti mattermost_port = 8065 mattermost_root = "http://localhost:#{mattermost_port}" control 'mattermost-basic-1' do

    impact 'critical' title 'Mattermost Server: Basic Connectivity' desc 'Basic connectivity …' tag 'mattermost', 'basic' # Basic Connectivity Tests describe port(mattermost_port) do it { should be_listening } end describe http(mattermost_root) do its('status') { should eq 200 } its('body') { should match '<title>Mattermost</title>' } end end
  65. @glennsarti mattermost_port = 8065 mattermost_root = "http://localhost:#{mattermost_port}" control 'mattermost-basic-1' do

    impact 'critical' title 'Mattermost Server: Basic Connectivity' desc 'Basic connectivity …' tag 'mattermost', 'basic' # Basic Connectivity Tests describe port(mattermost_port) do it { should be_listening } end describe http(mattermost_root) do its('status') { should eq 200 } its('body') { should match '<title>Mattermost</title>' } end end
  66. @glennsarti Inspec Requires an installation package $Inspec -ne $rspec Windows

    support can be variable
  67. @glennsarti Introducing Inspecster!

  68. @glennsarti control 'mattermost-basic-1' ` -impact 'critical' ` -title 'Mattermost Server:

    Basic Connectivity' ` -desc 'Basic connectivity tests to test …' ` -tags 'mattermost', 'basic' { describe "Response from ${MatterMostRoot}" { it "should return statuscode 200" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.StatusCode | Should -Be 200 -Because '…' } …
  69. None
  70. None
  71. None
  72. @glennsarti < take a breath >

  73. @glennsarti User Acceptance Testing

  74. @glennsarti Mattermost Backend Mattermost Server PoshBot UAT Tests

  75. @glennsarti UAT Create systems under test Run tests Destroy systems

    under test 1 2 3
  76. @glennsarti Run docker image Configure the Mattermost server Create token

    for the Bot Run PoshBot as a different process Wait for Poshbot to be ready
  77. @glennsarti 1

  78. @glennsarti

  79. @glennsarti it "should respond to '! help'" { $userInfo =

    Get-UserInformation # Post a Message $body = @{ 'channel_id' = $userInfo.channelid 'message' = "! help" } $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body # Give the bot time to respond Start-Sleep 2 $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" ` -Method GET # The Bot should have a reaction on the message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark' $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET
  80. @glennsarti it "should respond to '! help'" { $userInfo =

    Get-UserInformation # Post a Message $body = @{ 'channel_id' = $userInfo.channelid 'message' = "! help" } $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body # Give the bot time to respond Start-Sleep 2 $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" ` -Method GET # The Bot should have a reaction on the message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark' $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET
  81. @glennsarti it "should respond to '! help'" { $userInfo =

    Get-UserInformation # Post a Message $body = @{ 'channel_id' = $userInfo.channelid 'message' = "! help" } $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body # Give the bot time to respond Start-Sleep 2 $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" ` -Method GET # The Bot should have a reaction on the message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark’ $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET
  82. @glennsarti it "should respond to '! help'" { $userInfo =

    Get-UserInformation # Post a Message $body = @{ 'channel_id' = $userInfo.channelid 'message' = "! help" } $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body # Give the bot time to respond Start-Sleep 2 $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" ` -Method GET # The Bot should have a reaction on the message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark' $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET
  83. @glennsarti # The Bot should have a reaction on the

    message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark' $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET # There should only be one new message $newMessages.order.Count | Should -Be 1 $newPostID = $newMessages.order[0] $newPost = $newMessages.posts."$newPostID" # The message text should start with... $newPost.message | Should -Match "```````nFullCommandName" }
  84. @glennsarti # The Bot should have a reaction on the

    message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark' $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET # There should only be one new message $newMessages.order.Count | Should -Be 1 $newPostID = $newMessages.order[0] $newPost = $newMessages.posts."$newPostID" # The message text should start with... $newPost.message | Should -Match "```````nFullCommandName" }
  85. @glennsarti

  86. @glennsarti Stop and remove the container Stop the Poshbot Process

  87. @glennsarti try { & .\Pre-Pester.ps1 # Add other options e.g.

    Output file and format $result = Invoke-Pester -Script $PSScriptRoot –PassThru } finally { & .\Post-Pester.ps1 } # Return the number of failed tests. Exit Code 0 = Success $host.SetShouldExit($result.failedcount) 1 2 3 4
  88. @glennsarti Thinking more different-er …

  89. @glennsarti Gherkin

  90. @glennsarti it "should respond to '! help'" { $userInfo =

    Get-UserInformation # Post a Message $body = @{ 'channel_id' = $userInfo.channelid 'message' = "! help" } $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body # Give the bot time to respond Start-Sleep 2 $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" ` -Method GET # The Bot should have a reaction on the message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark' $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET # The Bot should have a reaction on the message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark' $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET # There should only be one new message $newMessages.order.Count | Should -Be 1 $newPostID = $newMessages.order[0] $newPost = $newMessages.posts."$newPostID" # The message text should start with... $newPost.message | Should -Match "```````nFullCommandName" }
  91. @glennsarti Feature: It can respond to help messages Background: Login

    to Mattermost Given mattermost server http://localhost:8065 And mattermost user 'user2@example.com' with passsword 'Password1' Scenario: Sending a valid help message When sending a message of ! help And waiting 2 seconds Then the message should have a reaction of white_check_mark And poshbot returns a message that starts with ```````nFullCommandName
  92. @glennsarti Feature: It can respond to help messages Background: Login

    to Mattermost Given mattermost server http://localhost:8065 And mattermost user 'user2@example.com' with passsword 'Password1' Scenario: Sending a valid help message When sending a message of ! help And waiting 2 seconds Then the message should have a reaction of white_check_mark And poshbot returns a message that starts with ```````nFullCommandName
  93. @glennsarti Feature: It can respond to help messages Background: Login

    to Mattermost Given mattermost server http://localhost:8065 And mattermost user 'user2@example.com' with passsword 'Password1' Scenario: Sending a valid help message When sending a message of ! help And waiting 5 seconds Then the message should have a reaction of white_check_mark And poshbot returns a message that starts with ```````nFullCommandName
  94. @glennsarti Scenario: Sending a help message with an unknown command

    When sending a message of ! help commanddoesntexist And waiting 2 seconds Then the message should have a reaction of white_check_mark And poshbot returns a message that contains No commands found
  95. @glennsarti Tiering

  96. @glennsarti Instrumentation

  97. @glennsarti $Local -eq $Automated

  98. @glennsarti

  99. @glennsarti

  100. THANK YOU! Please use the event app or Socio to

    submit a session rating! @glennsarti http://glennsarti.github.io/ https://speakerdeck.com/glennsarti https://github.com/glennsarti/PSSummitNA2019-Pester102
  101. @glennsarti Resources Source Code https://github.com/glennsarti/PSSummitNA2019-Pester102 Cynefin https://www.youtube.com/watch?v=N7oz366X0-8 https://en.wikipedia.org/wiki/Cynefin_framework#cite_note-SnowdenBoone2007-1 http://cognitive-edge.com/videos/cynefin-framework-introduction/ Rob

    Crowley – Testing in production https://speakerdeck.com/robcrowley/testing-in-production-for-the-bold-and-true Mattermost https://api.mattermost.com/ Images https:/ /unsplash.com
  102. @glennsarti Resources Operation Validation Framework https://github.com/PowerShell/Operation-Validation-Framework Devopsing In a Microsoft

    World https://youtu.be/roFVOnRxXns?t=1500 https://www.slideshare.net/snasello1/devops-days-seattle-2017 InSpec https://www.inspec.io/
  103. @glennsarti Resources Gherkin https://docs.cucumber.io/gherkin/ https://powershellexplained.com/2017-03-17-Powershell-Gherkin-specification-validation/ https://powershellexplained.com/2017-04-30-Powershell-Gherkin-advanced-features/ https://www.youtube.com/watch?v=hCfcLDLTf6g Part 1 https://www.youtube.com/watch?v=mWFkFU4Hi_g

    Part 2 https://github.com/mikejonestechno/devops/tree/master/powershell/demo-bdd