Beyond Pester 101: Applying testing principles to PowerShell

Slide 1

Slide 1 text

Beyond Pester 101 Applying testing principles to PowerShell Glenn Sarti Software Engineer, Windows Team

Slide 2

Slide 2 text

@glennsarti A familiar st ry…

Slide 3

Slide 3 text

@glennsarti Happiness Time Tests?

Slide 4

Slide 4 text

@glennsarti Happiness Time YAY!

Slide 5

Slide 5 text

@glennsarti Happiness Time Hrmm

Slide 6

Slide 6 text

@glennsarti Happiness Time 

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

@glennsarti Happiness Time ☺

Slide 9

Slide 9 text

@glennsarti Dunning - Kruger Effect Confidence Knowledge / Skills Mt. Stupid Valley of Despair Slope of Enlightenment

Slide 10

Slide 10 text

@glennsarti Beginning the ascent …

Slide 11

Slide 11 text

@glennsarti Why do I test?

Slide 12

Slide 12 text

@glennsarti To reduce the risk that a user will experience an unexpected behaviour

Slide 13

Slide 13 text

@glennsarti To reduce the risk that a user will experience an unexpected behaviour

Slide 14

Slide 14 text

@glennsarti • Shorter delivery cycles • Produce high quality code • Cheaper Benefits of testing

Slide 15

Slide 15 text

@glennsarti • To ensure what is created does what it’s supposed to do • Form of documentation Benefits of testing

Slide 16

Slide 16 text

@glennsarti Kinds of tests?

Slide 17

Slide 17 text

@glennsarti 1.Unit 2. 3. 4. Types of tests

Slide 18

Slide 18 text

@glennsarti Types of tests - Unit Image adapted from Test Pyramid – Martin Fowler 1. Fast 2. Cheap 3. Single concern

Slide 19

Slide 19 text

Slide 20

Slide 20 text

Slide 21

Slide 21 text

@glennsarti function Get-PoshBot { … param( [parameter(ValueFromPipeline, ValueFromPipelineByPropertyName) ] [int[]]$Id = @() ) … process { if ($Id.Count -gt 0) { foreach ($item in $Id) { if ($b = $script:botTracker.$item) { [pscustomobject][ordered]@{ Id = $item Name = $b.Name State = (Get-Job -Id $b.jobId).State InstanceId = $b.InstanceId Config = $b.Config } } } } else { … describe 'Get-PoshBot’ { $script:botTracker.Add(1, @{ … }) $script:botTracker.Add(2, @{ … }) mock Get-Job { return 'Running’ } … it 'Returns a specific job instance’ { $j = Get-PoshBot -Id 1 $j | should not benullorempty $j.Id | should be 1 } } Input Output External External

Slide 22

Slide 22 text

Slide 23

Slide 23 text

@glennsarti Types of tests - Unit Get-Job Inputs Outputs Mocks Get-PoshBot $script:botTracker

Slide 24

Slide 24 text

@glennsarti describe 'Get-PoshBot’ { $script:botTracker.Add(1, @{ … }) $script:botTracker.Add(2, @{ … }) mock Get-Job { return 'Running’ } … it 'Returns a specific job instance’ { $j = Get-PoshBot -Id 1 $j | should not benullorempty $j.Id | should be 1 } } Act Assert Arrange

Slide 25

Slide 25 text

@glennsarti 1.Unit 2. 3. 4. Types of tests Brian Burke: How to love Unit Testing!

Slide 26

Slide 26 text

@glennsarti 1.Unit 2.Integration 3. 4. Types of tests

Slide 27

Slide 27 text

@glennsarti Types of tests - Integration Image adapted from Test Pyramid – Martin Fowler 1. Slower 2. More $$$ 3. Complex

Slide 28

Slide 28 text

@glennsarti Types of tests - Integration

Slide 29

Slide 29 text

@glennsarti Describe "$($script:DSCResourceName)_Integration" { Context 'Disable NetBIOS over TCP/IP’ { $configData = @{ AllNodes = @( @{ NodeName = 'localhost’ InterfaceAlias = $netAdapter.Name Setting = 'Disable’ } ) } It 'Should compile and apply the MOF without throwing’ { { & "$($script:DSCResourceName)_Config" -OutputPath $TestDrive –ConfigurationData … Start-DscConfiguration -Path $TestDrive -ComputerName localhost -Wait –Verbose … } | Should -Not -Throw } It 'Should be able to call Get-DscConfiguration without throwing’ { { Get-DscConfiguration -Verbose -ErrorAction Stop } | Should -Not -Throw }

Slide 30

Slide 30 text

@glennsarti Describe "$($script:DSCResourceName)_Integration" { Context 'Disable NetBIOS over TCP/IP' { $configData = @{ AllNodes = @( @{ NodeName = 'localhost' InterfaceAlias = $netAdapter.Name Setting = 'Disable' } ) } It 'Should compile and apply the MOF without throwing' { { & "$($script:DSCResourceName)_Config" -OutputPath $TestDrive –ConfigurationData … Start-DscConfiguration -Path $TestDrive -ComputerName localhost -Wait –Verbose … } | Should -Not -Throw } It 'Should be able to call Get-DscConfiguration without throwing' { { Get-DscConfiguration -Verbose -ErrorAction Stop } | Should -Not -Throw }

Slide 31

Slide 31 text

@glennsarti Types of tests - Integration Unit Integration Number 1393 140 Duration (s) 440 184 Average (s) 0.316 1.314 Taken from the xNetworking DSC Module Over 4x longer

Slide 32

Slide 32 text

@glennsarti 1.Unit 2.Integration 3.Unit (Again?) 4. Types of tests

Slide 33

Slide 33 text

@glennsarti Types of tests – Unit (Again?) Not all unit tests are the same!

Slide 34

Slide 34 text

@glennsarti Types of tests – Unit (Again?) Function Get-SvcDisplayName($ServiceName ) { (Get-WMIObject -Class Win32_Service | ? { $_.Name -eq $ServiceName }).DisplayName } Describe 'Get-SvcDisplayName’ { Mock Get-WMIObject { @{ 'Name' = 'MockService’; 'DisplayName' = 'MockDisplayName'} } it 'returns the DisplayName of a service’ { Get-SvcDisplayName 'MockService’ | Should Be 'MockDisplayName’ } }

Slide 35

Slide 35 text

@glennsarti Types of tests – Unit (Again?) Function Get-SvcDisplayName($ServiceName ) { } REDACTED Describe 'Get-SvcDisplayName’ { it 'returns the DisplayName of a service' { Get-SvcDisplayName 'WinRM' | Should Be 'Windows Remote Management (WS-Management)' } }

Slide 36

Slide 36 text

@glennsarti Types of tests – Unit (Black box) Black box White box

Slide 37

Slide 37 text

@glennsarti Types of tests – Unit (Black box) Black box White box Behavioural “What it does” Implementation “How it does it”

Slide 38

Slide 38 text

@glennsarti Types of tests – Unit (Black box) Black box White box Behavioural “What it does” Implementation “How it does it” QA Focused Dev Focused

Slide 39

Slide 39 text

@glennsarti Types of tests – Unit (Black box) Black box White box Behavioural “What it does” Implementation “How it does it” Change of mindset Fragile

Slide 40

Slide 40 text

@glennsarti Types of tests – Unit (Black box) Function Get-SvcDisplayName($ServiceName ) { (Get-WMIObject -Class Win32_Service | ? { $_.Name -eq $ServiceName }).DisplayName } Describe 'Get-SvcDisplayName' { Mock Get-WMIObject { @{ 'Name' = 'MockService'; 'DisplayName' = 'MockDisplayName'} } it 'returns the DisplayName of a service' { Get-SvcDisplayName 'MockService' | Should Be 'MockDisplayName' } }

Slide 41

Slide 41 text

@glennsarti Types of tests – Unit (Black box) Function Get-SvcDisplayName($ServiceName ) { (Get-Service –ServiceName $ServiceName | ForEach-Object { $_.DisplayName } } Describe 'Get-SvcDisplayName' { Mock Get-WMIObject { @{ 'Name' = 'MockService'; 'DisplayName' = 'MockDisplayName'} } it 'returns the DisplayName of a service' { Get-SvcDisplayName 'MockService' | Should Be 'MockDisplayName' } }

Slide 42

Slide 42 text

@glennsarti Types of tests – Unit (Black box) White box Black box • Only use parameters • Wrap with private functions

Slide 43

Slide 43 text

@glennsarti Types of tests – Unit (Black box) “Tools do one thing … Tools don’t know where their input data has generated … Tools accept all input only from their parameters … Tools don’t know how their output will be used, “ - The Many Forms of Scripting: which to use - Manning

Slide 44

Slide 44 text

@glennsarti Types of tests – Unit (Black box) Which is better? Which should I use? … and of course

Slide 45

Slide 45 text

@glennsarti Types of tests – Unit (Black box) Should I test private functions?

Slide 46

Slide 46 text

@glennsarti 1.Unit (White box) 2.Integration 3.Unit (Black box) 4. Types of tests

Slide 47

Slide 47 text

@glennsarti 1.Unit (White box) 2.Integration 3.Unit (Black box) 4.Unit (Again again?) Types of tests

Slide 48

Slide 48 text

@glennsarti 1.Unit (White box) 2.Integration 3.Unit (Black box) 4.Characterisation Types of tests

Slide 49

Slide 49 text

@glennsarti Types of tests - Characterisation - Michael C. Feathers “A characterization test is a test that characterizes the actual behavior of a piece of code ... The tests document the actual current behavior of the system”

Slide 50

Slide 50 text

@glennsarti Function Step-ByOne ($value) { Write-Output ($value + 2) } # Unit Test it 'should increment by one' { Step-ByOne 1 | Should be 2 } # Characterisation Test it 'should increment by two' { Step-ByOne 1 | Should be 3 }

Slide 51

Slide 51 text

@glennsarti Types of tests - Characterisation What use are they?

Slide 52

Slide 52 text

@glennsarti Types of tests - Characterisation “… legacy code is simply code without tests.” - Michael C. Feathers

Slide 53

Slide 53 text

@glennsarti Types of tests - Characterisation function Start-LegacyCode { # Really # horrible # code # which # should # be # refactored }

Slide 54

Slide 54 text

@glennsarti Types of tests - Characterisation function Start-LegacyCode { # Really # horrible # code # which # should # be # refactored } Characterisation Tests Characterisation Tests

Slide 55

Slide 55 text

@glennsarti Types of tests - Characterisation function Start-LegacyCode { Start-Good # code # which # should # be # refactored } Characterisation Tests function Start-Good { # Good code } Unit Tests

Slide 56

Slide 56 text

@glennsarti Types of tests - Characterisation function Start-LegacyCode { Start-Good Start-Clean # should # be # refactored } Characterisation Tests function Start-Good { # Good code } function Start-Clean { # Clean code } Unit Tests Unit Tests

Slide 57

Slide 57 text

@glennsarti Types of tests - Characterisation function Start-LegacyCode { Start-Good Start-Clean Start-Nice } Characterisation Tests function Start-Good { # Good code } function Start-Clean { # Clean code } function Start-Nice { # Nice code } Unit Tests Unit Tests Unit Tests

Slide 58

Slide 58 text

@glennsarti Types of tests - Characterisation function Start-LegacyCode { Start-Good Start-Clean Start-Nice } function Start-Good { # Good code } function Start-Clean { # Clean code } function Start-Nice { # Nice code } Unit Tests Unit Tests Unit Tests Unit Tests

Slide 59

Slide 59 text

@glennsarti Types of tests - Characterisation • I can’t get this class into a test harness • I need to make many changes in one area • I need to make a change, but I don’t know what tests to write • I’m changing the same code all over the place • I need to change a monster method and I can’t write tests for it • How do I know that I’m not breaking anything?

Slide 60

Slide 60 text

@glennsarti Types of tests Unit (White box) Tests how a function is supposed to operate Integration Tests how units interact with each other Unit (Black box) Tests the behaviour of a function without knowing how it works Characterisation Used to make changing legacy code safer

Slide 61

Slide 61 text

@glennsarti The second half of the slope …

Slide 62

Slide 62 text

@glennsarti Tending your test suite

Slide 63

Slide 63 text

@glennsarti Reduce duration Remove tests Reorder tests General maintenance

Slide 64

Slide 64 text

@glennsarti Invoke-ParallelPester https://github.com/glennsarti/ ParallelPester/tree/feature-parallel-mode

Slide 65

Slide 65 text

@glennsarti Pester 5 4 3 2 1 Output

Slide 66

Slide 66 text

@glennsarti ParallelPester 5 4 3 2 1 Output

Slide 67

Slide 67 text

@glennsarti 0 10 20 30 40 50 60 70 80 90 1 2 3 4 5 6 7 8 Duration (sec) Parallel Pester

Slide 68

Slide 68 text

@glennsarti Downsides?

Slide 69

Slide 69 text

@glennsarti Garbled log output Describing MSFT_xFirewallProfile\Get-TargetResource Executing script C:\Source\ParallelPester\demo-repos\xNetworking\Tests\Unit\MSFT_xHostsFile.Tests.ps1 [+] Should return current default gateway [+] Should return a "System.Collections.Hashtable" object type [+] Should return true 6.89s [+] Should return correct DNS Client Global Settings values Context Firewall Profile Exists 6.68s 6.67s 6.03s ? ? ?

Slide 70

Slide 70 text

@glennsarti Code Coverage

Slide 71

Slide 71 text

@glennsarti Subtle race conditions • Use TestDrive: ,Get-Random or GUIDs • Explicitly setup testing state Remember - Arrange, Act, Assert

Slide 72

Slide 72 text

@glennsarti Not suitable for everything

Slide 73

Slide 73 text

@glennsarti 0 10 20 30 40 50 60 70 80 90 1 2 3 4 5 6 7 8 Duration (sec) 2 CPU 4 CPU

Slide 74

Slide 74 text

@glennsarti Reduce duration Remove tests Reorder tests General maintenance

Slide 75

Slide 75 text

@glennsarti “Testing is a Forever Thing” - The Pester Book … but tests are mortal!

Slide 76

Slide 76 text

@glennsarti Use data! (Pester Output Files)

Slide 77

Slide 77 text

@glennsarti

Slide 78

Slide 78 text

@glennsarti TESTS 1075

Slide 79

Slide 79 text

@glennsarti /projects/Powershell/xnetworking/history?recordsNumber=10 Invoke-RESTMethod https://ci.appveyor.com/api/ /projects/Powershell/xnetworking/build/${buildVersion} /buildjobs/${jobID}/tests

Slide 80

Slide 80 text

@glennsarti { "list": [ { "buildJobTestId": 0, "name": "should return false", "fileName": "invoking with state disabled b… "framework": "NUnit", "outcome": "passed", "duration": 189, "errorMessage": "", "errorStackTrace": "", "stdOut": "", "stdErr": "", "created": "2017-03-06T21:50:22.6918035+00:00" }, …

Slide 81

Slide 81 text

@glennsarti Build Test Test Test Test Test Test Test Test Test Skipped https://github.com/glennsarti/PSSummit2018

Slide 82

Slide 82 text

@glennsarti 0 200 400 600 800 1000 1200 1400 1600 1800 2.10.314.0 2.10.326.0 2.10.337.0 2.11.348.0 2.12.359.0 2.12.370.0 2.5.186.0 2.5.197.0 2.5.209.0 2.5.222.0 2.5.236.0 2.6.247.0 2.7.259.0 2.7.270.0 2.8.282.0 2.8.295.0 2.9.307.0 3.0.385.0 3.0.396.0 3.1.407.0 3.1.418.0 3.1.429.0 3.1.475.0 3.1.487.0 3.1.498.0 3.1.509.0 3.1.520.0 3.1.532.0 3.1.543.0 3.1.554.0 3.1.565.0 3.1.576.0 3.1.588.0 3.1.599.0 3.1.611.0 3.1.622.0 3.1.633.0 3.1.644.0 3.1.655.0 3.1.666.0 3.1.677.0 3.1.688.0 3.1.699.0 3.1.710.0 3.1.721.0 3.1.732.0 3.1.743.0 3.1.754.0 3.1.765.0 3.1.776.0 3.1.787.0 Build History (DSC Networking module) Passed Failed Skipped

Slide 83

Slide 83 text

@glennsarti 0 5 10 15 20 25 30 35 40 1 2 3 4 5 6 9 10 11 12 13 14 21 22 23 27 28 36 38 39 45 62 Number of Builds Number of Failures Failure Profile (DSC Networking module)

Slide 84

Slide 84 text

@glennsarti 0 5 10 15 20 25 30 35 40 45 Avg Duration (s) Top 100 slowest tests, in last 5 builds Long running integration tests?

Slide 85

Slide 85 text

@glennsarti 0 5 10 15 20 25 30 35 40 45 Avg Duration (s) Tests that have never failed Test that have never failed? 208 times!!

Slide 86

Slide 86 text

@glennsarti ▪ Tests that always fail together? ▪ Which tests fail too often? ▪ Which files tend to fail tests?

Slide 87

Slide 87 text

@glennsarti Reduce duration Remove tests Reorder tests General maintenance

Slide 88

Slide 88 text

@glennsarti Test Tiering HIGH MEDIUM LOW

Slide 89

Slide 89 text

@glennsarti # Tagging a Describe block Describe 'Smoke Tests' -Tag 'High' { It 'Should test something important' { # ... } }

Slide 90

Slide 90 text

@glennsarti Test name Failures "Common Tests -Validate Markdown Files; Context: - Should not have errors in any markdown files" 29 "Common Tests - File Formatting; Context: - Should not contain files without a newline at the end" 16 "MSFT_xNetAdapterRsc_Integration; Context: - should be able to call Get- DscConfiguration without throwing" 11 "Pester - MSFT_xRoute_Add_Integration.Should have set the resource and all the parameters should match" 11 … next 17 integration tests Common Tests - File Formatting; Context: - Should not contain any files with tab characters 10 DSC Networking module

Slide 91

Slide 91 text

@glennsarti PS> Invoke-Pester -Tag 'High' PS> Invoke-Pester -Tag 'High','Medium' PS> Invoke-Pester -ExcludeTag 'Low' Watch out for untagged tests!

Slide 92

Slide 92 text

@glennsarti Test Tiering HIGH MEDIUM LOW Execution Order

Slide 93

Slide 93 text

@glennsarti version: 1.0.{build} environment: matrix: - TEST_TIER: High - TEST_TIER: Medium - TEST_TIER: Low build: off matrix: fast_finish: true install: - ps: Install-Module Pester -SkipPublisherCheck -Force -Confirm:$false test_script: - ps: Invoke-Pester -Tag $ENV:TEST_TIER Example for AppVeyor CI 1 2 3

Slide 94

Slide 94 text

@glennsarti Test Tiering Next Level - Execution Cadence HIGH MEDIUM LOW

Slide 95

Slide 95 text

@glennsarti Reduce duration Remove tests Reorder tests General maintenance

Slide 96

Slide 96 text

@glennsarti Say what you mean and mean what you say. - George S Patton (Probably)

Slide 97

Slide 97 text

@glennsarti “Should not contain files without a newline at the end” DSC Networking module “Should contain files with a newline at the end”

Slide 98

Slide 98 text

@glennsarti The Pester Book it 'when the Web-Mgmt-Service feature is already installed, it attempts to change the WMSvc service startup type to Automatic' { $null = Enable-IISRemoteManagement -ComputerName 'SOMETHING' $assMParams = @{ CommandName = 'Set-Service' Times = 1 Scope = 'It' Exactly = $true ParameterFilter = { $ComputerName -eq 'SOMETHING' } } Assert-MockCalled @assMParams }

Slide 99

Slide 99 text

Slide 100

Slide 100 text

Slide 101

Slide 101 text

Slide 102

Slide 102 text

@glennsarti Wrapping up…

Slide 103

Slide 103 text

@glennsarti Unit (White box) Integration Unit (Black box) Characterisation Reduce duration Remove tests Reorder tests General maintenance

Slide 104

Slide 104 text

@glennsarti Where to now?

Slide 105

Slide 105 text

Please use the event app or Sched.com to submit a session rating! @glennsarti http://glennsarti.github.io/ https://speakerdeck.com/glennsarti

Slide 106

Slide 106 text

Types of software testing https://www.guru99.com/types-of-software-testing.html Pester book https://leanpub.com/pesterbook Images https://unsplash.com Dunning Kruger Effect https://www.xonitek.com/lessons-from-mt-stupid/ Test Pyramid – Martin Fowler https://martinfowler.com/bliki/TestPyramid.html Resources

Slide 107

Slide 107 text

Arrange, Act, Assert http://wiki.c2.com/?ArrangeActAssert Working effectively with Legacy Code https://www.amazon.com/Working-Effectively-Legacy-Michael-Feathers/dp/0131177052 Parallel Pester https://github.com/glennsarti/ParallelPester/tree/feature-parallel-mode PoshBot https://github.com/poshbotio/PoshBot The Many Forms of Scripting: which to use – Manning http://freecontent.manning.com/the-many-forms-of-scripting-which-to-use/ Resources