(PS Conf Asia '18) Beyond Pester 101: Applying testing principles to PowerShell

PowerShell Conference Asia Beyond Pester 101 Applying testing principles to
PowerShell Glenn Sarti Software Engineer, Windows Team

Powershell quick starts, tutorials, API reference, and code examples! Advance
your career with hands- on training at docs.microsoft Check it out here:

We really do! Show some love and join the conversation
by tweeting @MicrosoftTech

PowerShell Conference Asia Beyond Pester 101 Applying testing principles to
PowerShell Glenn Sarti Software Engineer, Windows Team

@glennsarti A familiar st ry…

@glennsarti Happiness Time Tests?

@glennsarti Happiness Time YAY!

@glennsarti Happiness Time Hrmm

@glennsarti Happiness Time 

@glennsarti Happiness Time ☺

@glennsarti Dunning - Kruger Effect Confidence Knowledge / Skills Mt.
Stupid Valley of Despair Slope of Enlightenment

@glennsarti Beginning the ascent …

@glennsarti Why do I test?

@glennsarti To reduce the risk that a user will experience
an unexpected behaviour

@glennsarti • Shorter delivery cycles • Produce high quality code
• Cheaper Benefits of testing

@glennsarti • To ensure what is created does what it’s
supposed to do • Form of documentation Benefits of testing

@glennsarti Kinds of tests?

@glennsarti 1.Unit 2. 3. 4. Types of tests

@glennsarti Types of tests - Unit Image adapted from Test
Pyramid – Martin Fowler 1. Fast 2. Cheap 3. Single concern

@glennsarti function Get-PoshBot { … param( [parameter(ValueFromPipeline, ValueFromPipelineByPropertyName) ] [int[]]$Id
= @() ) … process { if ($Id.Count -gt 0) { foreach ($item in $Id) { if ($b = $script:botTracker.$item) { [pscustomobject][ordered]@{ Id = $item Name = $b.Name State = (Get-Job -Id $b.jobId).State InstanceId = $b.InstanceId Config = $b.Config } } } } else { …

= @() ) … process { if ($Id.Count -gt 0) { foreach ($item in $Id) { if ($b = $script:botTracker.$item) { [pscustomobject][ordered]@{ Id = $item Name = $b.Name State = (Get-Job -Id $b.jobId).State InstanceId = $b.InstanceId Config = $b.Config } } } } else { … Input Output External External

= @() ) … process { if ($Id.Count -gt 0) { foreach ($item in $Id) { if ($b = $script:botTracker.$item) { [pscustomobject][ordered]@{ Id = $item Name = $b.Name State = (Get-Job -Id $b.jobId).State InstanceId = $b.InstanceId Config = $b.Config } } } } else { … describe 'Get-PoshBot’ { $script:botTracker.Add(1, @{ … }) $script:botTracker.Add(2, @{ … }) mock Get-Job { return 'Running’ } … it 'Returns a specific job instance’ { $j = Get-PoshBot -Id 1 $j | should not benullorempty $j.Id | should be 1 } } Input Output External External

@glennsarti Types of tests - Unit Get-Job Inputs Outputs Mocks
Get-PoshBot $script:botTracker

@glennsarti describe 'Get-PoshBot’ { $script:botTracker.Add(1, @{ … }) $script:botTracker.Add(2, @{
… }) mock Get-Job { return 'Running’ } … it 'Returns a specific job instance’ { $j = Get-PoshBot -Id 1 $j | should not benullorempty $j.Id | should be 1 } } Act Assert Arrange

@glennsarti 1.Unit 2.Integration 3. 4. Types of tests

@glennsarti Types of tests - Integration Image adapted from Test
Pyramid – Martin Fowler 1. Slower 2. More $$$ 3. Complex

@glennsarti Types of tests - Integration

@glennsarti Describe "$($script:DSCResourceName)_Integration" { Context 'Disable NetBIOS over TCP/IP’ {
$configData = @{ AllNodes = @( @{ NodeName = 'localhost’ InterfaceAlias = $netAdapter.Name Setting = 'Disable’ } ) } It 'Should compile and apply the MOF without throwing’ { { & "$($script:DSCResourceName)_Config" -OutputPath $TestDrive –ConfigurationData … Start-DscConfiguration -Path $TestDrive -ComputerName localhost -Wait –Verbose … } | Should -Not -Throw } It 'Should be able to call Get-DscConfiguration without throwing’ { { Get-DscConfiguration -Verbose -ErrorAction Stop } | Should -Not -Throw }

@glennsarti Describe "$($script:DSCResourceName)_Integration" { Context 'Disable NetBIOS over TCP/IP' {
$configData = @{ AllNodes = @( @{ NodeName = 'localhost' InterfaceAlias = $netAdapter.Name Setting = 'Disable' } ) } It 'Should compile and apply the MOF without throwing' { { & "$($script:DSCResourceName)_Config" -OutputPath $TestDrive –ConfigurationData … Start-DscConfiguration -Path $TestDrive -ComputerName localhost -Wait –Verbose … } | Should -Not -Throw } It 'Should be able to call Get-DscConfiguration without throwing' { { Get-DscConfiguration -Verbose -ErrorAction Stop } | Should -Not -Throw }

@glennsarti Types of tests - Integration Unit Integration Number 1393
140 Duration (s) 440 184 Average (s) 0.316 1.314 Taken from the xNetworking DSC Module Over 4x longer

@glennsarti 1.Unit 2.Integration 3.Unit (Again?) 4. Types of tests

@glennsarti Types of tests – Unit (Again?) Not all unit
tests are the same!

@glennsarti Types of tests – Unit (Again?) Function Get-SvcDisplayName($ServiceName )
{ (Get-WMIObject -Class Win32_Service | ? { $_.Name -eq $ServiceName }).DisplayName } Describe 'Get-SvcDisplayName’ { Mock Get-WMIObject { @{ 'Name' = 'MockService’; 'DisplayName' = 'MockDisplayName'} } it 'returns the DisplayName of a service’ { Get-SvcDisplayName 'MockService’ | Should Be 'MockDisplayName’ } }

@glennsarti Types of tests – Unit (Again?) Function Get-SvcDisplayName($ServiceName )
{ } REDACTED Describe 'Get-SvcDisplayName’ { it 'returns the DisplayName of a service' { Get-SvcDisplayName 'WinRM' | Should Be 'Windows Remote Management (WS-Management)' } }

@glennsarti Types of tests – Unit (Black box) Black box
White box

White box Behavioural “What it does” Implementation “How it does it”

White box Behavioural “What it does” Implementation “How it does it” QA Focused Dev Focused

White box Behavioural “What it does” Implementation “How it does it” Change of mindset Fragile

@glennsarti Types of tests – Unit (Black box) Function Get-SvcDisplayName($ServiceName
) { (Get-WMIObject -Class Win32_Service | ? { $_.Name -eq $ServiceName }).DisplayName } Describe 'Get-SvcDisplayName' { Mock Get-WMIObject { @{ 'Name' = 'MockService'; 'DisplayName' = 'MockDisplayName'} } it 'returns the DisplayName of a service' { Get-SvcDisplayName 'MockService' | Should Be 'MockDisplayName' } }

@glennsarti Types of tests – Unit (Black box) Function Get-SvcDisplayName($ServiceName
) { (Get-Service –ServiceName $ServiceName | ForEach-Object { $_.DisplayName } } Describe 'Get-SvcDisplayName' { Mock Get-WMIObject { @{ 'Name' = 'MockService'; 'DisplayName' = 'MockDisplayName'} } it 'returns the DisplayName of a service' { Get-SvcDisplayName 'MockService' | Should Be 'MockDisplayName' } }

@glennsarti Types of tests – Unit (Black box) White box
Black box • Only use parameters • Wrap with private functions

@glennsarti Types of tests – Unit (Black box) “Tools do
one thing … Tools don’t know where their input data has generated … Tools accept all input only from their parameters … Tools don’t know how their output will be used, “ - The Many Forms of Scripting: which to use - Manning

@glennsarti Types of tests – Unit (Black box) Which is
better? Which should I use? … and of course

@glennsarti Types of tests – Unit (Black box) Should I
test private functions?

@glennsarti 1.Unit (White box) 2.Integration 3.Unit (Black box) 4. Types
of tests

@glennsarti 1.Unit (White box) 2.Integration 3.Unit (Black box) 4.Unit (Again
again?) Types of tests

@glennsarti 1.Unit (White box) 2.Integration 3.Unit (Black box) 4.Characterisation Types
of tests

@glennsarti Types of tests - Characterisation - Michael C. Feathers
“A characterization test is a test that characterizes the actual behavior of a piece of code ... The tests document the actual current behavior of the system”

@glennsarti Function Step-ByOne ($value) { Write-Output ($value + 2) }
# Unit Test it 'should increment by one' { Step-ByOne 1 | Should be 2 } # Characterisation Test it 'should increment by two' { Step-ByOne 1 | Should be 3 }

@glennsarti Types of tests - Characterisation What use are they?

@glennsarti Types of tests - Characterisation “… legacy code is
simply code without tests.” - Michael C. Feathers

@glennsarti Types of tests - Characterisation function Start-LegacyCode { #
Really # horrible # code # which # should # be # refactored }

@glennsarti Types of tests - Characterisation function Start-LegacyCode { #
Really # horrible # code # which # should # be # refactored } Characterisation Tests Characterisation Tests

@glennsarti Types of tests - Characterisation function Start-LegacyCode { Start-Good
# code # which # should # be # refactored } Characterisation Tests function Start-Good { # Good code } Unit Tests

Start-Clean # should # be # refactored } Characterisation Tests function Start-Good { # Good code } function Start-Clean { # Clean code } Unit Tests Unit Tests

Start-Clean Start-Nice } Characterisation Tests function Start-Good { # Good code } function Start-Clean { # Clean code } function Start-Nice { # Nice code } Unit Tests Unit Tests Unit Tests

Start-Clean Start-Nice } function Start-Good { # Good code } function Start-Clean { # Clean code } function Start-Nice { # Nice code } Unit Tests Unit Tests Unit Tests Unit Tests

@glennsarti Types of tests - Characterisation • I can’t get
this class into a test harness • I need to make many changes in one area • I need to make a change, but I don’t know what tests to write • I’m changing the same code all over the place • I need to change a monster method and I can’t write tests for it • How do I know that I’m not breaking anything?

@glennsarti Types of tests Unit (White box) Tests how a
function is supposed to operate Integration Tests how units interact with each other Unit (Black box) Tests the behaviour of a function without knowing how it works Characterisation Used to make changing legacy code safer

@glennsarti The second half of the slope …

@glennsarti Tending your test suite

@glennsarti Reduce duration Remove tests Reorder tests General maintenance

@glennsarti Invoke-ParallelPester https://github.com/glennsarti/ ParallelPester/tree/feature-parallel-mode

@glennsarti Pester 5 4 3 2 1 Output

@glennsarti ParallelPester 5 4 3 2 1 Output

@glennsarti 0 10 20 30 40 50 60 70 80
90 1 2 3 4 5 6 7 8 Duration (sec) Parallel Pester

@glennsarti Downsides?

@glennsarti Garbled log output Describing MSFT_xFirewallProfile\Get-TargetResource Executing script C:\Source\ParallelPester\demo-repos\xNetworking\Tests\Unit\MSFT_xHostsFile.Tests.ps1 [+]
Should return current default gateway [+] Should return a "System.Collections.Hashtable" object type [+] Should return true 6.89s [+] Should return correct DNS Client Global Settings values Context Firewall Profile Exists 6.68s 6.67s 6.03s ? ? ?

@glennsarti Code Coverage

@glennsarti Subtle race conditions • Use TestDrive: ,Get-Random or GUIDs
• Explicitly setup testing state Remember - Arrange, Act, Assert

@glennsarti Not suitable for everything

@glennsarti 0 10 20 30 40 50 60 70 80
90 1 2 3 4 5 6 7 8 Duration (sec) 2 CPU 4 CPU

@glennsarti “Testing is a Forever Thing” - The Pester Book
… but tests are mortal!

@glennsarti Use data! (Pester Output Files)

@glennsarti

@glennsarti TESTS 1075

@glennsarti /projects/Powershell/xnetworking/history?recordsNumber=10 Invoke-RESTMethod https://ci.appveyor.com/api/ /projects/Powershell/xnetworking/build/${buildVersion} /buildjobs/${jobID}/tests

@glennsarti { "list": [ { "buildJobTestId": 0, "name": "should return
false", "fileName": "invoking with state disabled b… "framework": "NUnit", "outcome": "passed", "duration": 189, "errorMessage": "", "errorStackTrace": "", "stdOut": "", "stdErr": "", "created": "2017-03-06T21:50:22.6918035+00:00" }, …

@glennsarti Build Test Test Test Test Test Test Test Test
Test Skipped https://github.com/glennsarti/PSSummit2018

@glennsarti 0 200 400 600 800 1000 1200 1400 1600
1800 2.10.314.0 2.10.326.0 2.10.337.0 2.11.348.0 2.12.359.0 2.12.370.0 2.5.186.0 2.5.197.0 2.5.209.0 2.5.222.0 2.5.236.0 2.6.247.0 2.7.259.0 2.7.270.0 2.8.282.0 2.8.295.0 2.9.307.0 3.0.385.0 3.0.396.0 3.1.407.0 3.1.418.0 3.1.429.0 3.1.475.0 3.1.487.0 3.1.498.0 3.1.509.0 3.1.520.0 3.1.532.0 3.1.543.0 3.1.554.0 3.1.565.0 3.1.576.0 3.1.588.0 3.1.599.0 3.1.611.0 3.1.622.0 3.1.633.0 3.1.644.0 3.1.655.0 3.1.666.0 3.1.677.0 3.1.688.0 3.1.699.0 3.1.710.0 3.1.721.0 3.1.732.0 3.1.743.0 3.1.754.0 3.1.765.0 3.1.776.0 3.1.787.0 Build History (DSC Networking module) Passed Failed Skipped

@glennsarti 0 5 10 15 20 25 30 35 40
1 2 3 4 5 6 9 10 11 12 13 14 21 22 23 27 28 36 38 39 45 62 Number of Builds Number of Failures Failure Profile (DSC Networking module)

@glennsarti 0 5 10 15 20 25 30 35 40
45 Avg Duration (s) Top 100 slowest tests, in last 5 builds Long running integration tests?

@glennsarti 0 5 10 15 20 25 30 35 40
45 Avg Duration (s) Tests that have never failed Test that have never failed? 208 times!!

@glennsarti ▪ Tests that always fail together? ▪ Which tests
fail too often? ▪ Which files tend to fail tests?

@glennsarti Test Tiering HIGH MEDIUM LOW

@glennsarti # Tagging a Describe block Describe 'Smoke Tests' -Tag
'High' { It 'Should test something important' { # ... } }

@glennsarti Test name Failures "Common Tests -Validate Markdown Files; Context:
- Should not have errors in any markdown files" 29 "Common Tests - File Formatting; Context: - Should not contain files without a newline at the end" 16 "MSFT_xNetAdapterRsc_Integration; Context: - should be able to call Get- DscConfiguration without throwing" 11 "Pester - MSFT_xRoute_Add_Integration.Should have set the resource and all the parameters should match" 11 … next 17 integration tests Common Tests - File Formatting; Context: - Should not contain any files with tab characters 10 DSC Networking module

@glennsarti PS> Invoke-Pester -Tag 'High' PS> Invoke-Pester -Tag 'High','Medium' PS>
Invoke-Pester -ExcludeTag 'Low' Watch out for untagged tests!

@glennsarti Test Tiering HIGH MEDIUM LOW Execution Order

@glennsarti version: 1.0.{build} environment: matrix: - TEST_TIER: High - TEST_TIER:
Medium - TEST_TIER: Low build: off matrix: fast_finish: true install: - ps: Install-Module Pester -SkipPublisherCheck -Force -Confirm:$false test_script: - ps: Invoke-Pester -Tag $ENV:TEST_TIER Example for AppVeyor CI 1 2 3

@glennsarti Test Tiering Next Level - Execution Cadence HIGH MEDIUM
LOW

@glennsarti Say what you mean and mean what you say.
- George S Patton (Probably)

@glennsarti “Should not contain files without a newline at the
end” DSC Networking module “Should contain files with a newline at the end”

@glennsarti The Pester Book it 'when the Web-Mgmt-Service feature is
already installed, it attempts to change the WMSvc service startup type to Automatic' { $null = Enable-IISRemoteManagement -ComputerName 'SOMETHING' $assMParams = @{ CommandName = 'Set-Service' Times = 1 Scope = 'It' Exactly = $true ParameterFilter = { $ComputerName -eq 'SOMETHING' } } Assert-MockCalled @assMParams }

@glennsarti The Pester Book it 'when the Web-Mgmt-Service feature is
already installed, it attempts to change the WMSvc service startup type to Automatic' { $null = Enable-IISRemoteManagement -ComputerName 'SOMETHING' $assMParams = @{ CommandName = 'Set-Service' Times = 1 Scope = 'It' Exactly = $true ParameterFilter = { $ComputerName -eq 'SOMETHING' –and $Name -eq 'WMSvc' –and $StartupType -eq 'Automatic' } } Assert-MockCalled @assMParams }

@glennsarti Wrapping up…

@glennsarti Unit (White box) Integration Unit (Black box) Characterisation Reduce
duration Remove tests Reorder tests General maintenance

@glennsarti Where to now?

PowerShell Conference Singapore 2018 @glennsarti http://glennsarti.github.io/ https://speakerdeck.com/glennsarti Please fill in
your survey – it’s how we do better! #PowerShell #PSConfAsia @Psconf.Asia

Types of software testing https://www.guru99.com/types-of-software-testing.html Pester book https://leanpub.com/pesterbook Images https://unsplash.com
Dunning Kruger Effect https://www.xonitek.com/lessons-from-mt-stupid/ Test Pyramid – Martin Fowler https://martinfowler.com/bliki/TestPyramid.html Resources

Arrange, Act, Assert http://wiki.c2.com/?ArrangeActAssert Working effectively with Legacy Code https://www.amazon.com/Working-Effectively-Legacy-Michael-Feathers/dp/0131177052
Parallel Pester https://github.com/glennsarti/ParallelPester/tree/feature-parallel-mode PoshBot https://github.com/poshbotio/PoshBot The Many Forms of Scripting: which to use – Manning http://freecontent.manning.com/the-many-forms-of-scripting-which-to-use/ Resources

(PS Conf Asia '18) Beyond Pester 101: Applying testing principles to PowerShell

(PS Conf Asia '18) Beyond Pester 101: Applying testing principles to PowerShell

More Decks by Glenn

Other Decks in Technology

Featured

Transcript