@glennsarti glenn-sarti 

@glennsarti Previously … 

@glennsarti Unit (White box) Integration Unit (Black box) Characterisation Reduce duration Remove tests Reorder tests General maintenance 

@glennsarti Types of tests - Integration Image adapted from Test Pyramid – Martin Fowler 1. Slower 2. More $$$ 3. Complex 

@glennsarti Why do I test? 

@glennsarti To reduce the risk that a user will experience an unexpected behaviour 

@glennsarti To reduce the risk that a user will experience an unexpected behaviour 

@glennsarti Benefits of testing Shorter delivery cycles Produce high quality code Cheaper 

@glennsarti To ensure what is created does what it’s supposed to do Form of documentation Benefits of testing 

@glennsarti 

@glennsarti Image adapted from Test Pyramid – Martin Fowler 

@glennsarti Image adapted from Test Pyramid – Martin Fowler ⚠️ Slow 

@glennsarti Image adapted from Test Pyramid – Martin Fowler ⚠️ Expensive 

@glennsarti Image adapted from Test Pyramid – Martin Fowler ⚠️ Complex 

@glennsarti The cost to fix a bug Cost to Fix – Testing in Production – Rob Crowley Development Manual Testing From user Expensive Cheap 

@glennsarti Fixing a broken test Acceptance Tests Integration Tests Unit Tests PowerShell being tested PAIN 

@glennsarti Things that can go wrong One Plethora (*) Graph is not to scale or based on actual data Acceptance Tests Integration Tests Unit Tests 

@glennsarti ❌ ✔ 

@glennsarti Customers don’t measure you on how hard you tried, they measure you on what you deliver.” “ - Steve Jobs 

@glennsarti Where’s the value? 

@glennsarti … there are known knowns … known unknowns … also unknown unknowns … it is the latter category that tend to be the difficult ones - Donald Rumsfeld “ ” 

@glennsarti What can we do then… 

@glennsarti Think Different Change Scope 

@glennsarti Think Different Change Scope 

@glennsarti Cynefin Framework “The Cynefin framework (/ˈkʌnɪvɪn/ KUN-iv-in) is a conceptual framework used to aid decision-making” - https://www.youtube.com/watch?v=N7oz366X0-8 

@glennsarti 

@glennsarti Complicated Complex Chaotic Obvious 

@glennsarti Ordered Unordered Complicated Complex Chaotic Obvious 

@glennsarti Obvious Sense – Categorise - Respond Unit Tests 

@glennsarti Complicated Sense – Analyse - Respond Integration Tests 

@glennsarti Complex Probe – Sense - Respond Acceptance Tests 

@glennsarti Chaotic Act – Sense - Respond Chaos Engineering 

@glennsarti Complicated Complex Obvious Unit Tests Integration Tests Acceptance Tests 

@glennsarti Think Different Change Scope 

@glennsarti ✔Reduce the scope ✔ Reduce complexity ✔ Reduce duration ✔ Reduce things that can go wrong ❌Reduces coverage 

@glennsarti – Testing in Production – Rob Crowley 

@glennsarti – Testing in Production – Rob Crowley 

@glennsarti – Testing in Production – Rob Crowley 

@glennsarti Let’s write some tests 

@glennsarti Introducing PoshBot and Mattermost … 

@glennsarti Mattermost Backend Mattermost Server Mattermost Client PoshBot Docker REST API PowerShell Web 

@glennsarti Mattermost Backend Mattermost Server Mattermost Client PoshBot OAT Tests 

@glennsarti Mattermost Backend Mattermost Server Mattermost Client PoshBot UAT Tests 

@glennsarti Operational Acceptance Testing and Operation Validation Testing 

@glennsarti Is the server running? Does the server think it's ok? Are the critical settings correct? 

@glennsarti $MatterMostRoot = "http://localhost:8065" Describe "Response from ${MatterMostRoot}" { it "should return statuscode 200" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.StatusCode | Should -Be 200 -Because '…’ } it "should have a title of MatterMost" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.Content | Should -BeLike "*Mattermost*" } } 

@glennsarti $MatterMostRoot = "http://localhost:8065" Describe "Response from ${MatterMostRoot}" { it "should return statuscode 200" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.StatusCode | Should -Be 200 -Because '…' } it "should have a title of MatterMost" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.Content | Should -BeLike "*Mattermost*" } } 

@glennsarti 

@glennsarti 

@glennsarti Operation Validation Framework - https://github.com/PowerShell/Operation-Validation-Framework A set of tools for executing validation of the operation of a system. It provides a way to organize and execute Pester tests which are written to validate operation … ” “ 

@glennsarti PowerShell Module 

@glennsarti OVF Directories 

@glennsarti Tests 

@glennsarti param( $MatterMostRoot = "http://localhost:8065" ) Describe "Mattermost Simple Tests" { Context "Response from $MatterMostRoot" { it "should return statuscode 200" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.StatusCode | Should -Be 200 -Because '…' } it "should have a title of MatterMost" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.Content | Should -BeLike "*Mattermost*" } } } 

@glennsarti PS> Get-OperationValidation -Path OVF Module: …\MatterMostTests\OAT\OVF\ExampleModule Version: Type: Simple Tags: {} File: Mattermost-Simple.Tests.ps1 FilePath: …\Diagnostics\Simple\Mattermost-Simple.Tests.ps1 Name: Mattermost Simple Tests … 

@glennsarti PS> Invoke-OperationValidation -Path OVF Module: C:\Source\PSSummit-Pester102\MatterMostTests\OAT\OVF\ExampleModule Result Name ------ ---- Passed Mattermost Simple Tests:Response from http://localhost:8065:should return statuscode 200 Passed Mattermost Simple Tests:Response from http://localhost:8065:should have a title of MatterMost Passed Mattermost Comprehensive Tests:Server configuration:should return statuscode 404 for V3 API Passed Mattermost Comprehensive Tests:Server configuration:should report MatterMost Server versio... Passed Mattermost Comprehensive Tests:Status endpoint at http://localhost:8065/api/v4/system/ping... Passed Mattermost Comprehensive Tests:Status endpoint at http://localhost:8065/api/v4/system/ping... Passed Mattermost Comprehensive Tests:Users API endpoint:should have more than one user 

@glennsarti OVF Gotchas (1.2.0) Doesn’t require a PSD file Single Describe block and text only Use the newest version 

@glennsarti Beyond OVF… Sharing Tooling ChatOps-ing - Devopsing In a Microsoft World 

@glennsarti Compliance Acceptance Testing 

@glennsarti 

@glennsarti 

@glennsarti Metadata Test 

@glennsarti $MatterMostRoot = "http://localhost:8065" Describe "Response from ${MatterMostRoot}" { it "should return statuscode 200" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.StatusCode | Should -Be 200 -Because '…’ } it "should have a title of MatterMost" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.Content | Should -BeLike "*Mattermost*" } } 

@glennsarti mattermost_port = 8065 mattermost_root = "http://localhost:#{mattermost_port}" control 'mattermost-basic-1' do impact 'critical' title 'Mattermost Server: Basic Connectivity' desc 'Basic connectivity …' tag 'mattermost', 'basic' # Basic Connectivity Tests describe port(mattermost_port) do it { should be_listening } end describe http(mattermost_root) do its('status') { should eq 200 } its('body') { should match 'Mattermost' } end end 

@glennsarti mattermost_port = 8065 mattermost_root = "http://localhost:#{mattermost_port}" control 'mattermost-basic-1' do impact 'critical' title 'Mattermost Server: Basic Connectivity' desc 'Basic connectivity …' tag 'mattermost', 'basic' # Basic Connectivity Tests describe port(mattermost_port) do it { should be_listening } end describe http(mattermost_root) do its('status') { should eq 200 } its('body') { should match 'Mattermost' } end end 

@glennsarti Inspec Requires an installation package $Inspec -ne $rspec Windows support can be variable 

@glennsarti Introducing Inspecster! 

@glennsarti control 'mattermost-basic-1' ` -impact 'critical' ` -title 'Mattermost Server: Basic Connectivity' ` -desc 'Basic connectivity tests to test …' ` -tags 'mattermost', 'basic' { describe "Response from ${MatterMostRoot}" { it "should return statuscode 200" { $Result = Invoke-WebRequest -Uri $MatterMostRoot -UseBasicParsing $Result.StatusCode | Should -Be 200 -Because '…' } … 

No content

No content

No content

@glennsarti < take a breath > 

@glennsarti User Acceptance Testing 

@glennsarti Mattermost Backend Mattermost Server PoshBot UAT Tests 

@glennsarti UAT Create systems under test Run tests Destroy systems under test 1 2 3 

@glennsarti Run docker image Configure the Mattermost server Create token for the Bot Run PoshBot as a different process Wait for Poshbot to be ready 

@glennsarti 1 

@glennsarti 

@glennsarti it "should respond to '! help'" { $userInfo = Get-UserInformation # Post a Message $body = @{ 'channel_id' = $userInfo.channelid 'message' = "! help" } $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body # Give the bot time to respond Start-Sleep 2 $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" ` -Method GET # The Bot should have a reaction on the message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark' $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET 

@glennsarti it "should respond to '! help'" { $userInfo = Get-UserInformation # Post a Message $body = @{ 'channel_id' = $userInfo.channelid 'message' = "! help" } $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body # Give the bot time to respond Start-Sleep 2 $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" ` -Method GET # The Bot should have a reaction on the message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark' $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET 

@glennsarti it "should respond to '! help'" { $userInfo = Get-UserInformation # Post a Message $body = @{ 'channel_id' = $userInfo.channelid 'message' = "! help" } $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body # Give the bot time to respond Start-Sleep 2 $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" ` -Method GET # The Bot should have a reaction on the message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark’ $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET 

@glennsarti it "should respond to '! help'" { $userInfo = Get-UserInformation # Post a Message $body = @{ 'channel_id' = $userInfo.channelid 'message' = "! help" } $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body # Give the bot time to respond Start-Sleep 2 $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" ` -Method GET # The Bot should have a reaction on the message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark' $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET 

@glennsarti # The Bot should have a reaction on the message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark' $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET # There should only be one new message $newMessages.order.Count | Should -Be 1 $newPostID = $newMessages.order[0] $newPost = $newMessages.posts."$newPostID" # The message text should start with... $newPost.message | Should -Match "```````nFullCommandName" } 

@glennsarti # The Bot should have a reaction on the message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark' $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET # There should only be one new message $newMessages.order.Count | Should -Be 1 $newPostID = $newMessages.order[0] $newPost = $newMessages.posts."$newPostID" # The message text should start with... $newPost.message | Should -Match "```````nFullCommandName" } 

@glennsarti 

@glennsarti Stop and remove the container Stop the Poshbot Process 

@glennsarti try { & .\Pre-Pester.ps1 # Add other options e.g. Output file and format $result = Invoke-Pester -Script $PSScriptRoot –PassThru } finally { & .\Post-Pester.ps1 } # Return the number of failed tests. Exit Code 0 = Success $host.SetShouldExit($result.failedcount) 1 2 3 4 

@glennsarti Thinking more different-er … 

@glennsarti Gherkin 

@glennsarti it "should respond to '! help'" { $userInfo = Get-UserInformation # Post a Message $body = @{ 'channel_id' = $userInfo.channelid 'message' = "! help" } $originalMessage = Invoke-MattermostAPI -Uri 'posts' -Method POST -Body $body # Give the bot time to respond Start-Sleep 2 $updatedMessage = Invoke-MattermostAPI -Uri "posts/$($originalMessage.id)" ` -Method GET # The Bot should have a reaction on the message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark' $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET # The Bot should have a reaction on the message $updatedMessage.metadata.reactions.Count | Should -BeGreaterThan 0 $updatedMessage.metadata.reactions[0].emoji_name | Should -Be 'white_check_mark' $newMessages = Invoke-MattermostAPI ` -Uri "channels/$($userInfo.channelid)/posts?after=$($originalMessage.id)" ` -Method GET # There should only be one new message $newMessages.order.Count | Should -Be 1 $newPostID = $newMessages.order[0] $newPost = $newMessages.posts."$newPostID" # The message text should start with... $newPost.message | Should -Match "```````nFullCommandName" } 

@glennsarti Feature: It can respond to help messages Background: Login to Mattermost Given mattermost server http://localhost:8065 And mattermost user '[email protected]' with passsword 'Password1' Scenario: Sending a valid help message When sending a message of ! help And waiting 2 seconds Then the message should have a reaction of white_check_mark And poshbot returns a message that starts with ```````nFullCommandName 

@glennsarti Feature: It can respond to help messages Background: Login to Mattermost Given mattermost server http://localhost:8065 And mattermost user '[email protected]' with passsword 'Password1' Scenario: Sending a valid help message When sending a message of ! help And waiting 2 seconds Then the message should have a reaction of white_check_mark And poshbot returns a message that starts with ```````nFullCommandName 

@glennsarti Feature: It can respond to help messages Background: Login to Mattermost Given mattermost server http://localhost:8065 And mattermost user '[email protected]' with passsword 'Password1' Scenario: Sending a valid help message When sending a message of ! help And waiting 5 seconds Then the message should have a reaction of white_check_mark And poshbot returns a message that starts with ```````nFullCommandName 

@glennsarti Scenario: Sending a help message with an unknown command When sending a message of ! help commanddoesntexist And waiting 2 seconds Then the message should have a reaction of white_check_mark And poshbot returns a message that contains No commands found 

@glennsarti Tiering 

@glennsarti Instrumentation 

@glennsarti $Local -eq $Automated 

@glennsarti 

@glennsarti 

THANK YOU! Please use the event app or Socio to submit a session rating! @glennsarti http://glennsarti.github.io/ https://speakerdeck.com/glennsarti https://github.com/glennsarti/PSSummitNA2019-Pester102 

@glennsarti Resources Source Code https://github.com/glennsarti/PSSummitNA2019-Pester102 Cynefin https://www.youtube.com/watch?v=N7oz366X0-8 https://en.wikipedia.org/wiki/Cynefin_framework#cite_note-SnowdenBoone2007-1 http://cognitive-edge.com/videos/cynefin-framework-introduction/ Rob Crowley – Testing in production https://speakerdeck.com/robcrowley/testing-in-production-for-the-bold-and-true Mattermost https://api.mattermost.com/ Images https:/ /unsplash.com 

@glennsarti Resources Operation Validation Framework https://github.com/PowerShell/Operation-Validation-Framework Devopsing In a Microsoft World https://youtu.be/roFVOnRxXns?t=1500 https://www.slideshare.net/snasello1/devops-days-seattle-2017 InSpec https://www.inspec.io/ 

@glennsarti Resources Gherkin https://docs.cucumber.io/gherkin/ https://powershellexplained.com/2017-03-17-Powershell-Gherkin-specification-validation/ https://powershellexplained.com/2017-04-30-Powershell-Gherkin-advanced-features/ https://www.youtube.com/watch?v=hCfcLDLTf6g Part 1 https://www.youtube.com/watch?v=mWFkFU4Hi_g Part 2 https://github.com/mikejonestechno/devops/tree/master/powershell/demo-bdd