Share
Embed
Beyond Pester 102: Acceptance testing with PowerShell
Previously in "Beyond Pester 101: Applying testing principles to PowerShell" I talked about unit and integration testing with Pester and PowerShell. Now that we're all experts there, it's time to tackle Acceptance testing, also known as End to End testing.
Glenn
April 26, 2019
Transcript
103 slides
-
@glennsarti Unit (White box) Integration Unit (Black box) Characterisation Reduce
duration Remove tests Reorder tests General maintenance -
@glennsarti Types of tests - Integration Image adapted from Test
Pyramid – Martin Fowler 1. Slower 2. More $$$ 3. Complex -
@glennsarti To reduce the risk that a user will experience
an unexpected behaviour -
@glennsarti To reduce the risk that a user will experience
an unexpected behaviour -
@glennsarti To ensure what is created does what it’s supposed
to do Form of documentation Benefits of testing -
@glennsarti The cost to fix a bug Cost to Fix
– Testing in Production – Rob Crowley Development Manual Testing From user Expensive Cheap -
@glennsarti Fixing a broken test Acceptance Tests Integration Tests Unit
Tests PowerShell being tested PAIN -
@glennsarti Things that can go wrong One Plethora (*) Graph
is not to scale or based on actual data Acceptance Tests Integration Tests Unit Tests -
@glennsarti Customers don’t measure you on how hard you tried,
they measure you on what you deliver.” “ - Steve Jobs -
@glennsarti … there are known knowns … known unknowns …
also unknown unknowns … it is the latter category that tend to be the difficult ones - Donald Rumsfeld “ ” -
@glennsarti Cynefin Framework “The Cynefin framework (/ˈkʌnɪvɪn/ KUN-iv-in) is a
conceptual framework used to aid decision-making” - https://www.youtube.com/watch?v=N7oz366X0-8 -
@glennsarti ✔Reduce the scope ✔ Reduce complexity ✔ Reduce duration
✔ Reduce things that can go wrong ❌Reduces coverage -
@glennsarti Is the server running? Does the server think it's
ok? Are the critical settings correct? -
@glennsarti $MatterMostRoot = "http://localhost:8065" Describe "Response from ${MatterMostRoot}" { it
"should return statuscode 200" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.StatusCode | Should -Be 200 -Because '…’ } it "should have a title of MatterMost" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.Content | Should -BeLike "*<title>Mattermost</title>*" } } -
@glennsarti $MatterMostRoot = "http://localhost:8065" Describe "Response from ${MatterMostRoot}" { it
"should return statuscode 200" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.StatusCode | Should -Be 200 -Because '…' } it "should have a title of MatterMost" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.Content | Should -BeLike "*<title>Mattermost</title>*" } } -
@glennsarti Operation Validation Framework - https://github.com/PowerShell/Operation-Validation-Framework A set of tools
for executing validation of the operation of a system. It provides a way to organize and execute Pester tests which are written to validate operation … ” “ -
@glennsarti param( $MatterMostRoot = "http://localhost:8065" ) Describe "Mattermost Simple Tests"
{ Context "Response from $MatterMostRoot" { it "should return statuscode 200" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.StatusCode | Should -Be 200 -Because '…' } it "should have a title of MatterMost" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.Content | Should -BeLike "*<title>Mattermost</title>*" } } } -
@glennsarti PS> Get-OperationValidation -Path OVF Module: …\MatterMostTests\OAT\OVF\ExampleModule Version: Type: Simple
Tags: {} File: Mattermost-Simple.Tests.ps1 FilePath: …\Diagnostics\Simple\Mattermost-Simple.Tests.ps1 Name: Mattermost Simple Tests … -
@glennsarti PS> Invoke-OperationValidation -Path OVF Module: C:\Source\PSSummit-Pester102\MatterMostTests\OAT\OVF\ExampleModule Result Name ------
---- Passed Mattermost Simple Tests:Response from http://localhost:8065:should return statuscode 200 Passed Mattermost Simple Tests:Response from http://localhost:8065:should have a title of MatterMost Passed Mattermost Comprehensive Tests:Server configuration:should return statuscode 404 for V3 API Passed Mattermost Comprehensive Tests:Server configuration:should report MatterMost Server versio... Passed Mattermost Comprehensive Tests:Status endpoint at http://localhost:8065/api/v4/system/ping... Passed Mattermost Comprehensive Tests:Status endpoint at http://localhost:8065/api/v4/system/ping... Passed Mattermost Comprehensive Tests:Users API endpoint:should have more than one user -
@glennsarti OVF Gotchas (1.2.0) Doesn’t require a PSD file Single
Describe block and text only Use the newest version -
@glennsarti $MatterMostRoot = "http://localhost:8065" Describe "Response from ${MatterMostRoot}" { it
"should return statuscode 200" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.StatusCode | Should -Be 200 -Because '…’ } it "should have a title of MatterMost" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.Content | Should -BeLike "*<title>Mattermost</title>*" } } -
@glennsarti mattermost_port = 8065 mattermost_root = "http://localhost:#{mattermost_port}" control 'mattermost-basic-1' do
impact 'critical' title 'Mattermost Server: Basic Connectivity' desc 'Basic connectivity …' tag 'mattermost', 'basic' # Basic Connectivity Tests describe port(mattermost_port) do it { should be_listening } end describe http(mattermost_root) do its('status') { should eq 200 } its('body') { should match '<title>Mattermost</title>' } end end -
@glennsarti mattermost_port = 8065 mattermost_root = "http://localhost:#{mattermost_port}" control 'mattermost-basic-1' do
impact 'critical' title 'Mattermost Server: Basic Connectivity' desc 'Basic connectivity …' tag 'mattermost', 'basic' # Basic Connectivity Tests describe port(mattermost_port) do it { should be_listening } end describe http(mattermost_root) do its('status') { should eq 200 } its('body') { should match '<title>Mattermost</title>' } end end -
@glennsarti Inspec Requires an installation package $Inspec -ne $rspec Windows
support can be variable -
@glennsarti control 'mattermost-basic-1' ` -impact 'critical' ` -title 'Mattermost Server:
Basic Connectivity' ` -desc 'Basic connectivity tests to test …' ` -tags 'mattermost', 'basic' { describe "Response from ${MatterMostRoot}" { it "should return statuscode 200" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.StatusCode | Should -Be 200 -Because '…' } … -
@glennsarti UAT Create systems under test Run tests Destroy systems
under test 1 2 3 -
@glennsarti Run docker image Configure the Mattermost server Create token
for the Bot Run PoshBot as a different process Wait for Poshbot to be ready -
@glennsarti it "should respond to '! help'" { $userInfo =
Get-UserInformation # Post a Message $body = @{ 'channel_id' = $userInfo.channelid 'message' = "! help" } $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body # Give the bot time to respond Start-Sleep 2 $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" ` -Method GET # The Bot should have a reaction on the message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark' $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET -
@glennsarti it "should respond to '! help'" { $userInfo =
Get-UserInformation # Post a Message $body = @{ 'channel_id' = $userInfo.channelid 'message' = "! help" } $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body # Give the bot time to respond Start-Sleep 2 $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" ` -Method GET # The Bot should have a reaction on the message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark' $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET -
@glennsarti it "should respond to '! help'" { $userInfo =
Get-UserInformation # Post a Message $body = @{ 'channel_id' = $userInfo.channelid 'message' = "! help" } $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body # Give the bot time to respond Start-Sleep 2 $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" ` -Method GET # The Bot should have a reaction on the message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark’ $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET -
@glennsarti it "should respond to '! help'" { $userInfo =
Get-UserInformation # Post a Message $body = @{ 'channel_id' = $userInfo.channelid 'message' = "! help" } $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body # Give the bot time to respond Start-Sleep 2 $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" ` -Method GET # The Bot should have a reaction on the message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark' $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET -
@glennsarti # The Bot should have a reaction on the
message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark' $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET # There should only be one new message $newMessages.order.Count | Should -Be 1 $newPostID = $newMessages.order[0] $newPost = $newMessages.posts."$newPostID" # The message text should start with... $newPost.message | Should -Match "```````nFullCommandName" } -
@glennsarti # The Bot should have a reaction on the
message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark' $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET # There should only be one new message $newMessages.order.Count | Should -Be 1 $newPostID = $newMessages.order[0] $newPost = $newMessages.posts."$newPostID" # The message text should start with... $newPost.message | Should -Match "```````nFullCommandName" } -
@glennsarti try { & .\Pre-Pester.ps1 # Add other options e.g.
Output file and format $result = Invoke-Pester -Script $PSScriptRoot –PassThru } finally { & .\Post-Pester.ps1 } # Return the number of failed tests. Exit Code 0 = Success $host.SetShouldExit($result.failedcount) 1 2 3 4 -
@glennsarti it "should respond to '! help'" { $userInfo =
Get-UserInformation # Post a Message $body = @{ 'channel_id' = $userInfo.channelid 'message' = "! help" } $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body # Give the bot time to respond Start-Sleep 2 $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" ` -Method GET # The Bot should have a reaction on the message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark' $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET # The Bot should have a reaction on the message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark' $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET # There should only be one new message $newMessages.order.Count | Should -Be 1 $newPostID = $newMessages.order[0] $newPost = $newMessages.posts."$newPostID" # The message text should start with... $newPost.message | Should -Match "```````nFullCommandName" } -
@glennsarti Feature: It can respond to help messages Background: Login
to Mattermost Given mattermost server http://localhost:8065 And mattermost user '[email protected]' with passsword 'Password1' Scenario: Sending a valid help message When sending a message of ! help And waiting 2 seconds Then the message should have a reaction of white_check_mark And poshbot returns a message that starts with ```````nFullCommandName -
@glennsarti Feature: It can respond to help messages Background: Login
to Mattermost Given mattermost server http://localhost:8065 And mattermost user '[email protected]' with passsword 'Password1' Scenario: Sending a valid help message When sending a message of ! help And waiting 2 seconds Then the message should have a reaction of white_check_mark And poshbot returns a message that starts with ```````nFullCommandName -
@glennsarti Feature: It can respond to help messages Background: Login
to Mattermost Given mattermost server http://localhost:8065 And mattermost user '[email protected]' with passsword 'Password1' Scenario: Sending a valid help message When sending a message of ! help And waiting 5 seconds Then the message should have a reaction of white_check_mark And poshbot returns a message that starts with ```````nFullCommandName -
@glennsarti Scenario: Sending a help message with an unknown command
When sending a message of ! help commanddoesntexist And waiting 2 seconds Then the message should have a reaction of white_check_mark And poshbot returns a message that contains No commands found -
THANK YOU! Please use the event app or Socio to
submit a session rating! @glennsarti http://glennsarti.github.io/ https://speakerdeck.com/glennsarti https://github.com/glennsarti/PSSummitNA2019-Pester102 -
@glennsarti Resources Source Code https://github.com/glennsarti/PSSummitNA2019-Pester102 Cynefin https://www.youtube.com/watch?v=N7oz366X0-8 https://en.wikipedia.org/wiki/Cynefin_framework#cite_note-SnowdenBoone2007-1 http://cognitive-edge.com/videos/cynefin-framework-introduction/ Rob
Crowley – Testing in production https://speakerdeck.com/robcrowley/testing-in-production-for-the-bold-and-true Mattermost https://api.mattermost.com/ Images https:/ /unsplash.com -
@glennsarti Resources Operation Validation Framework https://github.com/PowerShell/Operation-Validation-Framework Devopsing In a Microsoft
World https://youtu.be/roFVOnRxXns?t=1500 https://www.slideshare.net/snasello1/devops-days-seattle-2017 InSpec https://www.inspec.io/ -
@glennsarti Resources Gherkin https://docs.cucumber.io/gherkin/ https://powershellexplained.com/2017-03-17-Powershell-Gherkin-specification-validation/ https://powershellexplained.com/2017-04-30-Powershell-Gherkin-advanced-features/ https://www.youtube.com/watch?v=hCfcLDLTf6g Part 1 https://www.youtube.com/watch?v=mWFkFU4Hi_g
Part 2 https://github.com/mikejonestechno/devops/tree/master/powershell/demo-bdd