Beyond Pester 102: Acceptance testing with PowerShell

@glennsarti glenn-sarti

@glennsarti Previously …

@glennsarti Unit (White box) Integration Unit (Black box) Characterisation Reduce
duration Remove tests Reorder tests General maintenance

@glennsarti Types of tests - Integration Image adapted from Test
Pyramid – Martin Fowler 1. Slower 2. More $$$ 3. Complex

@glennsarti Why do I test?

@glennsarti To reduce the risk that a user will experience
an unexpected behaviour

@glennsarti Benefits of testing Shorter delivery cycles Produce high quality
code Cheaper

@glennsarti To ensure what is created does what it’s supposed
to do Form of documentation Benefits of testing

@glennsarti

@glennsarti Image adapted from Test Pyramid – Martin Fowler

@glennsarti Image adapted from Test Pyramid – Martin Fowler ⚠️
Slow

Expensive

Complex

@glennsarti The cost to fix a bug Cost to Fix
– Testing in Production – Rob Crowley Development Manual Testing From user Expensive Cheap

@glennsarti Fixing a broken test Acceptance Tests Integration Tests Unit
Tests PowerShell being tested PAIN

@glennsarti Things that can go wrong One Plethora (*) Graph
is not to scale or based on actual data Acceptance Tests Integration Tests Unit Tests

@glennsarti ❌ ✔

@glennsarti Customers don’t measure you on how hard you tried,
they measure you on what you deliver.” “ - Steve Jobs

@glennsarti Where’s the value?

@glennsarti … there are known knowns … known unknowns …
also unknown unknowns … it is the latter category that tend to be the difficult ones - Donald Rumsfeld “ ”

@glennsarti What can we do then…

@glennsarti Think Different Change Scope

@glennsarti Cynefin Framework “The Cynefin framework (/ˈkʌnɪvɪn/ KUN-iv-in) is a
conceptual framework used to aid decision-making” - https://www.youtube.com/watch?v=N7oz366X0-8

@glennsarti

@glennsarti Complicated Complex Chaotic Obvious

@glennsarti Ordered Unordered Complicated Complex Chaotic Obvious

@glennsarti Obvious Sense – Categorise - Respond Unit Tests

@glennsarti Complicated Sense – Analyse - Respond Integration Tests

@glennsarti Complex Probe – Sense - Respond Acceptance Tests

@glennsarti Chaotic Act – Sense - Respond Chaos Engineering

@glennsarti Complicated Complex Obvious Unit Tests Integration Tests Acceptance Tests

@glennsarti Think Different Change Scope

@glennsarti ✔Reduce the scope ✔ Reduce complexity ✔ Reduce duration
✔ Reduce things that can go wrong ❌Reduces coverage

@glennsarti – Testing in Production – Rob Crowley

@glennsarti Let’s write some tests

@glennsarti Introducing PoshBot and Mattermost …

@glennsarti Mattermost Backend Mattermost Server Mattermost Client PoshBot Docker REST
API PowerShell Web

@glennsarti Mattermost Backend Mattermost Server Mattermost Client PoshBot OAT Tests

@glennsarti Mattermost Backend Mattermost Server Mattermost Client PoshBot UAT Tests

@glennsarti Operational Acceptance Testing and Operation Validation Testing

@glennsarti Is the server running? Does the server think it's
ok? Are the critical settings correct?

@glennsarti $MatterMostRoot = "http://localhost:8065" Describe "Response from ${MatterMostRoot}" { it
"should return statuscode 200" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.StatusCode | Should -Be 200 -Because '…’ } it "should have a title of MatterMost" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.Content | Should -BeLike "*<title>Mattermost</title>*" } }

"should return statuscode 200" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.StatusCode | Should -Be 200 -Because '…' } it "should have a title of MatterMost" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.Content | Should -BeLike "*<title>Mattermost</title>*" } }

@glennsarti

@glennsarti Operation Validation Framework - https://github.com/PowerShell/Operation-Validation-Framework A set of tools
for executing validation of the operation of a system. It provides a way to organize and execute Pester tests which are written to validate operation … ” “

@glennsarti PowerShell Module

@glennsarti OVF Directories

@glennsarti Tests

@glennsarti param( $MatterMostRoot = "http://localhost:8065" ) Describe "Mattermost Simple Tests"
{ Context "Response from $MatterMostRoot" { it "should return statuscode 200" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.StatusCode | Should -Be 200 -Because '…' } it "should have a title of MatterMost" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.Content | Should -BeLike "*<title>Mattermost</title>*" } } }

@glennsarti PS> Get-OperationValidation -Path OVF Module: …\MatterMostTests\OAT\OVF\ExampleModule Version: Type: Simple
Tags: {} File: Mattermost-Simple.Tests.ps1 FilePath: …\Diagnostics\Simple\Mattermost-Simple.Tests.ps1 Name: Mattermost Simple Tests …

@glennsarti PS> Invoke-OperationValidation -Path OVF Module: C:\Source\PSSummit-Pester102\MatterMostTests\OAT\OVF\ExampleModule Result Name ------
---- Passed Mattermost Simple Tests:Response from http://localhost:8065:should return statuscode 200 Passed Mattermost Simple Tests:Response from http://localhost:8065:should have a title of MatterMost Passed Mattermost Comprehensive Tests:Server configuration:should return statuscode 404 for V3 API Passed Mattermost Comprehensive Tests:Server configuration:should report MatterMost Server versio... Passed Mattermost Comprehensive Tests:Status endpoint at http://localhost:8065/api/v4/system/ping... Passed Mattermost Comprehensive Tests:Status endpoint at http://localhost:8065/api/v4/system/ping... Passed Mattermost Comprehensive Tests:Users API endpoint:should have more than one user

@glennsarti OVF Gotchas (1.2.0) Doesn’t require a PSD file Single
Describe block and text only Use the newest version

@glennsarti Beyond OVF… Sharing Tooling ChatOps-ing - Devopsing In a
Microsoft World

@glennsarti Compliance Acceptance Testing

@glennsarti

@glennsarti Metadata Test

"should return statuscode 200" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.StatusCode | Should -Be 200 -Because '…’ } it "should have a title of MatterMost" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.Content | Should -BeLike "*<title>Mattermost</title>*" } }

@glennsarti mattermost_port = 8065 mattermost_root = "http://localhost:#{mattermost_port}" control 'mattermost-basic-1' do
impact 'critical' title 'Mattermost Server: Basic Connectivity' desc 'Basic connectivity …' tag 'mattermost', 'basic' # Basic Connectivity Tests describe port(mattermost_port) do it { should be_listening } end describe http(mattermost_root) do its('status') { should eq 200 } its('body') { should match '<title>Mattermost</title>' } end end

@glennsarti Inspec Requires an installation package $Inspec -ne $rspec Windows
support can be variable

@glennsarti Introducing Inspecster!

@glennsarti control 'mattermost-basic-1' ` -impact 'critical' ` -title 'Mattermost Server:
Basic Connectivity' ` -desc 'Basic connectivity tests to test …' ` -tags 'mattermost', 'basic' { describe "Response from ${MatterMostRoot}" { it "should return statuscode 200" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.StatusCode | Should -Be 200 -Because '…' } …

@glennsarti < take a breath >

@glennsarti User Acceptance Testing

@glennsarti Mattermost Backend Mattermost Server PoshBot UAT Tests

@glennsarti UAT Create systems under test Run tests Destroy systems
under test 1 2 3

@glennsarti Run docker image Configure the Mattermost server Create token
for the Bot Run PoshBot as a different process Wait for Poshbot to be ready

@glennsarti 1

@glennsarti

@glennsarti it "should respond to '! help'" { $userInfo =
Get-UserInformation # Post a Message $body = @{ 'channel_id' = $userInfo.channelid 'message' = "! help" } $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body # Give the bot time to respond Start-Sleep 2 $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" ` -Method GET # The Bot should have a reaction on the message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark' $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET

Get-UserInformation # Post a Message $body = @{ 'channel_id' = $userInfo.channelid 'message' = "! help" } $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body # Give the bot time to respond Start-Sleep 2 $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" ` -Method GET # The Bot should have a reaction on the message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark’ $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET

Get-UserInformation # Post a Message $body = @{ 'channel_id' = $userInfo.channelid 'message' = "! help" } $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body # Give the bot time to respond Start-Sleep 2 $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" ` -Method GET # The Bot should have a reaction on the message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark' $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET

@glennsarti # The Bot should have a reaction on the
message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark' $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET # There should only be one new message $newMessages.order.Count | Should -Be 1 $newPostID = $newMessages.order[0] $newPost = $newMessages.posts."$newPostID" # The message text should start with... $newPost.message | Should -Match "```````nFullCommandName" }

@glennsarti

@glennsarti Stop and remove the container Stop the Poshbot Process

@glennsarti try { & .\Pre-Pester.ps1 # Add other options e.g.
Output file and format $result = Invoke-Pester -Script $PSScriptRoot –PassThru } finally { & .\Post-Pester.ps1 } # Return the number of failed tests. Exit Code 0 = Success $host.SetShouldExit($result.failedcount) 1 2 3 4

@glennsarti Thinking more different-er …

@glennsarti Gherkin

Get-UserInformation # Post a Message $body = @{ 'channel_id' = $userInfo.channelid 'message' = "! help" } $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body # Give the bot time to respond Start-Sleep 2 $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" ` -Method GET # The Bot should have a reaction on the message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark' $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET # The Bot should have a reaction on the message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark' $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET # There should only be one new message $newMessages.order.Count | Should -Be 1 $newPostID = $newMessages.order[0] $newPost = $newMessages.posts."$newPostID" # The message text should start with... $newPost.message | Should -Match "```````nFullCommandName" }

@glennsarti Feature: It can respond to help messages Background: Login
to Mattermost Given mattermost server http://localhost:8065 And mattermost user '[email protected]' with passsword 'Password1' Scenario: Sending a valid help message When sending a message of ! help And waiting 2 seconds Then the message should have a reaction of white_check_mark And poshbot returns a message that starts with ```````nFullCommandName

@glennsarti Feature: It can respond to help messages Background: Login
to Mattermost Given mattermost server http://localhost:8065 And mattermost user '[email protected]' with passsword 'Password1' Scenario: Sending a valid help message When sending a message of ! help And waiting 5 seconds Then the message should have a reaction of white_check_mark And poshbot returns a message that starts with ```````nFullCommandName

@glennsarti Scenario: Sending a help message with an unknown command
When sending a message of ! help commanddoesntexist And waiting 2 seconds Then the message should have a reaction of white_check_mark And poshbot returns a message that contains No commands found

@glennsarti Tiering

@glennsarti Instrumentation

@glennsarti $Local -eq $Automated

@glennsarti

THANK YOU! Please use the event app or Socio to
submit a session rating! @glennsarti http://glennsarti.github.io/ https://speakerdeck.com/glennsarti https://github.com/glennsarti/PSSummitNA2019-Pester102

@glennsarti Resources Source Code https://github.com/glennsarti/PSSummitNA2019-Pester102 Cynefin https://www.youtube.com/watch?v=N7oz366X0-8 https://en.wikipedia.org/wiki/Cynefin_framework#cite_note-SnowdenBoone2007-1 http://cognitive-edge.com/videos/cynefin-framework-introduction/ Rob
Crowley – Testing in production https://speakerdeck.com/robcrowley/testing-in-production-for-the-bold-and-true Mattermost https://api.mattermost.com/ Images https:/ /unsplash.com

@glennsarti Resources Operation Validation Framework https://github.com/PowerShell/Operation-Validation-Framework Devopsing In a Microsoft
World https://youtu.be/roFVOnRxXns?t=1500 https://www.slideshare.net/snasello1/devops-days-seattle-2017 InSpec https://www.inspec.io/

@glennsarti Resources Gherkin https://docs.cucumber.io/gherkin/ https://powershellexplained.com/2017-03-17-Powershell-Gherkin-specification-validation/ https://powershellexplained.com/2017-04-30-Powershell-Gherkin-advanced-features/ https://www.youtube.com/watch?v=hCfcLDLTf6g Part 1 https://www.youtube.com/watch?v=mWFkFU4Hi_g
Part 2 https://github.com/mikejonestechno/devops/tree/master/powershell/demo-bdd

Beyond Pester 102: Acceptance testing with PowerShell

Beyond Pester 102: Acceptance testing with PowerShell

More Decks by Glenn

Other Decks in Technology

Featured

Transcript