$30 off During Our Annual Pro Sale. View Details »

Beyond Pester 102: Acceptance testing with PowerShell

Glenn
April 26, 2019

Beyond Pester 102: Acceptance testing with PowerShell

Previously in "Beyond Pester 101: Applying testing principles to PowerShell" I talked about unit and integration testing with Pester and PowerShell. Now that we're all experts there, it's time to tackle Acceptance testing, also known as End to End testing.

Glenn

April 26, 2019
Tweet

More Decks by Glenn

Other Decks in Technology

Transcript

  1. @glennsarti glenn-sarti

  2. @glennsarti Previously …

  3. @glennsarti Unit (White box) Integration Unit (Black box) Characterisation Reduce

    duration Remove tests Reorder tests General maintenance
  4. @glennsarti Types of tests - Integration Image adapted from Test

    Pyramid – Martin Fowler 1. Slower 2. More $$$ 3. Complex
  5. @glennsarti Why do I test?

  6. @glennsarti To reduce the risk that a user will experience

    an unexpected behaviour
  7. @glennsarti To reduce the risk that a user will experience

    an unexpected behaviour
  8. @glennsarti Benefits of testing Shorter delivery cycles Produce high quality

    code Cheaper
  9. @glennsarti To ensure what is created does what it’s supposed

    to do Form of documentation Benefits of testing
  10. @glennsarti

  11. @glennsarti Image adapted from Test Pyramid – Martin Fowler

  12. @glennsarti Image adapted from Test Pyramid – Martin Fowler ⚠️

    Slow
  13. @glennsarti Image adapted from Test Pyramid – Martin Fowler ⚠️

    Expensive
  14. @glennsarti Image adapted from Test Pyramid – Martin Fowler ⚠️

    Complex
  15. @glennsarti The cost to fix a bug Cost to Fix

    – Testing in Production – Rob Crowley Development Manual Testing From user Expensive Cheap
  16. @glennsarti Fixing a broken test Acceptance Tests Integration Tests Unit

    Tests PowerShell being tested PAIN
  17. @glennsarti Things that can go wrong One Plethora (*) Graph

    is not to scale or based on actual data Acceptance Tests Integration Tests Unit Tests
  18. @glennsarti ❌ ✔

  19. @glennsarti Customers don’t measure you on how hard you tried,

    they measure you on what you deliver.” “ - Steve Jobs
  20. @glennsarti Where’s the value?

  21. @glennsarti … there are known knowns … known unknowns …

    also unknown unknowns … it is the latter category that tend to be the difficult ones - Donald Rumsfeld “ ”
  22. @glennsarti What can we do then…

  23. @glennsarti Think Different Change Scope

  24. @glennsarti Think Different Change Scope

  25. @glennsarti Cynefin Framework “The Cynefin framework (/ˈkʌnɪvɪn/ KUN-iv-in) is a

    conceptual framework used to aid decision-making” - https://www.youtube.com/watch?v=N7oz366X0-8
  26. @glennsarti

  27. @glennsarti Complicated Complex Chaotic Obvious

  28. @glennsarti Ordered Unordered Complicated Complex Chaotic Obvious

  29. @glennsarti Obvious Sense – Categorise - Respond Unit Tests

  30. @glennsarti Complicated Sense – Analyse - Respond Integration Tests

  31. @glennsarti Complex Probe – Sense - Respond Acceptance Tests

  32. @glennsarti Chaotic Act – Sense - Respond Chaos Engineering

  33. @glennsarti Complicated Complex Obvious Unit Tests Integration Tests Acceptance Tests

  34. @glennsarti Think Different Change Scope

  35. @glennsarti ✔Reduce the scope ✔ Reduce complexity ✔ Reduce duration

    ✔ Reduce things that can go wrong ❌Reduces coverage
  36. @glennsarti – Testing in Production – Rob Crowley

  37. @glennsarti – Testing in Production – Rob Crowley

  38. @glennsarti – Testing in Production – Rob Crowley

  39. @glennsarti Let’s write some tests

  40. @glennsarti Introducing PoshBot and Mattermost …

  41. @glennsarti Mattermost Backend Mattermost Server Mattermost Client PoshBot Docker REST

    API PowerShell Web
  42. @glennsarti Mattermost Backend Mattermost Server Mattermost Client PoshBot OAT Tests

  43. @glennsarti Mattermost Backend Mattermost Server Mattermost Client PoshBot UAT Tests

  44. @glennsarti Operational Acceptance Testing and Operation Validation Testing

  45. @glennsarti Is the server running? Does the server think it's

    ok? Are the critical settings correct?
  46. @glennsarti $MatterMostRoot = "http://localhost:8065" Describe "Response from ${MatterMostRoot}" { it

    "should return statuscode 200" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.StatusCode | Should -Be 200 -Because '…’ } it "should have a title of MatterMost" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.Content | Should -BeLike "*<title>Mattermost</title>*" } }
  47. @glennsarti $MatterMostRoot = "http://localhost:8065" Describe "Response from ${MatterMostRoot}" { it

    "should return statuscode 200" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.StatusCode | Should -Be 200 -Because '…' } it "should have a title of MatterMost" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.Content | Should -BeLike "*<title>Mattermost</title>*" } }
  48. @glennsarti

  49. @glennsarti

  50. @glennsarti Operation Validation Framework - https://github.com/PowerShell/Operation-Validation-Framework A set of tools

    for executing validation of the operation of a system. It provides a way to organize and execute Pester tests which are written to validate operation … ” “
  51. @glennsarti PowerShell Module

  52. @glennsarti OVF Directories

  53. @glennsarti Tests

  54. @glennsarti param( $MatterMostRoot = "http://localhost:8065" ) Describe "Mattermost Simple Tests"

    { Context "Response from $MatterMostRoot" { it "should return statuscode 200" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.StatusCode | Should -Be 200 -Because '…' } it "should have a title of MatterMost" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.Content | Should -BeLike "*<title>Mattermost</title>*" } } }
  55. @glennsarti PS> Get-OperationValidation -Path OVF Module: …\MatterMostTests\OAT\OVF\ExampleModule Version: Type: Simple

    Tags: {} File: Mattermost-Simple.Tests.ps1 FilePath: …\Diagnostics\Simple\Mattermost-Simple.Tests.ps1 Name: Mattermost Simple Tests …
  56. @glennsarti PS> Invoke-OperationValidation -Path OVF Module: C:\Source\PSSummit-Pester102\MatterMostTests\OAT\OVF\ExampleModule Result Name ------

    ---- Passed Mattermost Simple Tests:Response from http://localhost:8065:should return statuscode 200 Passed Mattermost Simple Tests:Response from http://localhost:8065:should have a title of MatterMost Passed Mattermost Comprehensive Tests:Server configuration:should return statuscode 404 for V3 API Passed Mattermost Comprehensive Tests:Server configuration:should report MatterMost Server versio... Passed Mattermost Comprehensive Tests:Status endpoint at http://localhost:8065/api/v4/system/ping... Passed Mattermost Comprehensive Tests:Status endpoint at http://localhost:8065/api/v4/system/ping... Passed Mattermost Comprehensive Tests:Users API endpoint:should have more than one user
  57. @glennsarti OVF Gotchas (1.2.0) Doesn’t require a PSD file Single

    Describe block and text only Use the newest version
  58. @glennsarti Beyond OVF… Sharing Tooling ChatOps-ing - Devopsing In a

    Microsoft World
  59. @glennsarti Compliance Acceptance Testing

  60. @glennsarti

  61. @glennsarti

  62. @glennsarti Metadata Test

  63. @glennsarti $MatterMostRoot = "http://localhost:8065" Describe "Response from ${MatterMostRoot}" { it

    "should return statuscode 200" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.StatusCode | Should -Be 200 -Because '…’ } it "should have a title of MatterMost" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.Content | Should -BeLike "*<title>Mattermost</title>*" } }
  64. @glennsarti mattermost_port = 8065 mattermost_root = "http://localhost:#{mattermost_port}" control 'mattermost-basic-1' do

    impact 'critical' title 'Mattermost Server: Basic Connectivity' desc 'Basic connectivity …' tag 'mattermost', 'basic' # Basic Connectivity Tests describe port(mattermost_port) do it { should be_listening } end describe http(mattermost_root) do its('status') { should eq 200 } its('body') { should match '<title>Mattermost</title>' } end end
  65. @glennsarti mattermost_port = 8065 mattermost_root = "http://localhost:#{mattermost_port}" control 'mattermost-basic-1' do

    impact 'critical' title 'Mattermost Server: Basic Connectivity' desc 'Basic connectivity …' tag 'mattermost', 'basic' # Basic Connectivity Tests describe port(mattermost_port) do it { should be_listening } end describe http(mattermost_root) do its('status') { should eq 200 } its('body') { should match '<title>Mattermost</title>' } end end
  66. @glennsarti Inspec Requires an installation package $Inspec -ne $rspec Windows

    support can be variable
  67. @glennsarti Introducing Inspecster!

  68. @glennsarti control 'mattermost-basic-1' ` -impact 'critical' ` -title 'Mattermost Server:

    Basic Connectivity' ` -desc 'Basic connectivity tests to test …' ` -tags 'mattermost', 'basic' { describe "Response from ${MatterMostRoot}" { it "should return statuscode 200" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.StatusCode | Should -Be 200 -Because '…' } …
  69. None
  70. None
  71. None
  72. @glennsarti < take a breath >

  73. @glennsarti User Acceptance Testing

  74. @glennsarti Mattermost Backend Mattermost Server PoshBot UAT Tests

  75. @glennsarti UAT Create systems under test Run tests Destroy systems

    under test 1 2 3
  76. @glennsarti Run docker image Configure the Mattermost server Create token

    for the Bot Run PoshBot as a different process Wait for Poshbot to be ready
  77. @glennsarti 1

  78. @glennsarti

  79. @glennsarti it "should respond to '! help'" { $userInfo =

    Get-UserInformation # Post a Message $body = @{ 'channel_id' = $userInfo.channelid 'message' = "! help" } $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body # Give the bot time to respond Start-Sleep 2 $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" ` -Method GET # The Bot should have a reaction on the message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark' $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET
  80. @glennsarti it "should respond to '! help'" { $userInfo =

    Get-UserInformation # Post a Message $body = @{ 'channel_id' = $userInfo.channelid 'message' = "! help" } $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body # Give the bot time to respond Start-Sleep 2 $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" ` -Method GET # The Bot should have a reaction on the message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark' $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET
  81. @glennsarti it "should respond to '! help'" { $userInfo =

    Get-UserInformation # Post a Message $body = @{ 'channel_id' = $userInfo.channelid 'message' = "! help" } $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body # Give the bot time to respond Start-Sleep 2 $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" ` -Method GET # The Bot should have a reaction on the message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark’ $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET
  82. @glennsarti it "should respond to '! help'" { $userInfo =

    Get-UserInformation # Post a Message $body = @{ 'channel_id' = $userInfo.channelid 'message' = "! help" } $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body # Give the bot time to respond Start-Sleep 2 $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" ` -Method GET # The Bot should have a reaction on the message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark' $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET
  83. @glennsarti # The Bot should have a reaction on the

    message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark' $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET # There should only be one new message $newMessages.order.Count | Should -Be 1 $newPostID = $newMessages.order[0] $newPost = $newMessages.posts."$newPostID" # The message text should start with... $newPost.message | Should -Match "```````nFullCommandName" }
  84. @glennsarti # The Bot should have a reaction on the

    message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark' $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET # There should only be one new message $newMessages.order.Count | Should -Be 1 $newPostID = $newMessages.order[0] $newPost = $newMessages.posts."$newPostID" # The message text should start with... $newPost.message | Should -Match "```````nFullCommandName" }
  85. @glennsarti

  86. @glennsarti Stop and remove the container Stop the Poshbot Process

  87. @glennsarti try { & .\Pre-Pester.ps1 # Add other options e.g.

    Output file and format $result = Invoke-Pester -Script $PSScriptRoot –PassThru } finally { & .\Post-Pester.ps1 } # Return the number of failed tests. Exit Code 0 = Success $host.SetShouldExit($result.failedcount) 1 2 3 4
  88. @glennsarti Thinking more different-er …

  89. @glennsarti Gherkin

  90. @glennsarti it "should respond to '! help'" { $userInfo =

    Get-UserInformation # Post a Message $body = @{ 'channel_id' = $userInfo.channelid 'message' = "! help" } $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body # Give the bot time to respond Start-Sleep 2 $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" ` -Method GET # The Bot should have a reaction on the message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark' $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET # The Bot should have a reaction on the message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark' $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET # There should only be one new message $newMessages.order.Count | Should -Be 1 $newPostID = $newMessages.order[0] $newPost = $newMessages.posts."$newPostID" # The message text should start with... $newPost.message | Should -Match "```````nFullCommandName" }
  91. @glennsarti Feature: It can respond to help messages Background: Login

    to Mattermost Given mattermost server http://localhost:8065 And mattermost user 'user2@example.com' with passsword 'Password1' Scenario: Sending a valid help message When sending a message of ! help And waiting 2 seconds Then the message should have a reaction of white_check_mark And poshbot returns a message that starts with ```````nFullCommandName
  92. @glennsarti Feature: It can respond to help messages Background: Login

    to Mattermost Given mattermost server http://localhost:8065 And mattermost user 'user2@example.com' with passsword 'Password1' Scenario: Sending a valid help message When sending a message of ! help And waiting 2 seconds Then the message should have a reaction of white_check_mark And poshbot returns a message that starts with ```````nFullCommandName
  93. @glennsarti Feature: It can respond to help messages Background: Login

    to Mattermost Given mattermost server http://localhost:8065 And mattermost user 'user2@example.com' with passsword 'Password1' Scenario: Sending a valid help message When sending a message of ! help And waiting 5 seconds Then the message should have a reaction of white_check_mark And poshbot returns a message that starts with ```````nFullCommandName
  94. @glennsarti Scenario: Sending a help message with an unknown command

    When sending a message of ! help commanddoesntexist And waiting 2 seconds Then the message should have a reaction of white_check_mark And poshbot returns a message that contains No commands found
  95. @glennsarti Tiering

  96. @glennsarti Instrumentation

  97. @glennsarti $Local -eq $Automated

  98. @glennsarti

  99. @glennsarti

  100. THANK YOU! Please use the event app or Socio to

    submit a session rating! @glennsarti http://glennsarti.github.io/ https://speakerdeck.com/glennsarti https://github.com/glennsarti/PSSummitNA2019-Pester102
  101. @glennsarti Resources Source Code https://github.com/glennsarti/PSSummitNA2019-Pester102 Cynefin https://www.youtube.com/watch?v=N7oz366X0-8 https://en.wikipedia.org/wiki/Cynefin_framework#cite_note-SnowdenBoone2007-1 http://cognitive-edge.com/videos/cynefin-framework-introduction/ Rob

    Crowley – Testing in production https://speakerdeck.com/robcrowley/testing-in-production-for-the-bold-and-true Mattermost https://api.mattermost.com/ Images https:/ /unsplash.com
  102. @glennsarti Resources Operation Validation Framework https://github.com/PowerShell/Operation-Validation-Framework Devopsing In a Microsoft

    World https://youtu.be/roFVOnRxXns?t=1500 https://www.slideshare.net/snasello1/devops-days-seattle-2017 InSpec https://www.inspec.io/
  103. @glennsarti Resources Gherkin https://docs.cucumber.io/gherkin/ https://powershellexplained.com/2017-03-17-Powershell-Gherkin-specification-validation/ https://powershellexplained.com/2017-04-30-Powershell-Gherkin-advanced-features/ https://www.youtube.com/watch?v=hCfcLDLTf6g Part 1 https://www.youtube.com/watch?v=mWFkFU4Hi_g

    Part 2 https://github.com/mikejonestechno/devops/tree/master/powershell/demo-bdd