Share
Embed
(PS Conf Asia '18) Beyond Pester 101: Applying testing principles to PowerShell
We see a lot talks on testing PowerShell with Pester, but are the tests we write good tests? What makes a test "good"? How do we measure how effective our tests are? This talk will help you answer these questions, including why testing is important and how to apply these principles to your project.
Glenn
October 18, 2018
Transcript
109 slides
-
PowerShell Conference Asia Beyond Pester 101 Applying testing principles to
PowerShell Glenn Sarti Software Engineer, Windows Team -
Powershell quick starts, tutorials, API reference, and code examples! Advance
your career with hands- on training at docs.microsoft Check it out here: -
We really do! Show some love and join the conversation
by tweeting @MicrosoftTech -
PowerShell Conference Asia Beyond Pester 101 Applying testing principles to
PowerShell Glenn Sarti Software Engineer, Windows Team -
@glennsarti Dunning - Kruger Effect Confidence Knowledge / Skills Mt.
Stupid Valley of Despair Slope of Enlightenment -
@glennsarti To reduce the risk that a user will experience
an unexpected behaviour -
@glennsarti To reduce the risk that a user will experience
an unexpected behaviour -
@glennsarti • Shorter delivery cycles • Produce high quality code
• Cheaper Benefits of testing -
@glennsarti • To ensure what is created does what it’s
supposed to do • Form of documentation Benefits of testing -
@glennsarti Types of tests - Unit Image adapted from Test
Pyramid – Martin Fowler 1. Fast 2. Cheap 3. Single concern -
@glennsarti function Get-PoshBot { … param( [parameter(ValueFromPipeline, ValueFromPipelineByPropertyName) ] [int[]]$Id
= @() ) … process { if ($Id.Count -gt 0) { foreach ($item in $Id) { if ($b = $script:botTracker.$item) { [pscustomobject][ordered]@{ Id = $item Name = $b.Name State = (Get-Job -Id $b.jobId).State InstanceId = $b.InstanceId Config = $b.Config } } } } else { … -
@glennsarti function Get-PoshBot { … param( [parameter(ValueFromPipeline, ValueFromPipelineByPropertyName) ] [int[]]$Id
= @() ) … process { if ($Id.Count -gt 0) { foreach ($item in $Id) { if ($b = $script:botTracker.$item) { [pscustomobject][ordered]@{ Id = $item Name = $b.Name State = (Get-Job -Id $b.jobId).State InstanceId = $b.InstanceId Config = $b.Config } } } } else { … Input Output External External -
@glennsarti function Get-PoshBot { … param( [parameter(ValueFromPipeline, ValueFromPipelineByPropertyName) ] [int[]]$Id
= @() ) … process { if ($Id.Count -gt 0) { foreach ($item in $Id) { if ($b = $script:botTracker.$item) { [pscustomobject][ordered]@{ Id = $item Name = $b.Name State = (Get-Job -Id $b.jobId).State InstanceId = $b.InstanceId Config = $b.Config } } } } else { … describe 'Get-PoshBot’ { $script:botTracker.Add(1, @{ … }) $script:botTracker.Add(2, @{ … }) mock Get-Job { return 'Running’ } … it 'Returns a specific job instance’ { $j = Get-PoshBot -Id 1 $j | should not benullorempty $j.Id | should be 1 } } Input Output External External -
@glennsarti function Get-PoshBot { … param( [parameter(ValueFromPipeline, ValueFromPipelineByPropertyName) ] [int[]]$Id
= @() ) … process { if ($Id.Count -gt 0) { foreach ($item in $Id) { if ($b = $script:botTracker.$item) { [pscustomobject][ordered]@{ Id = $item Name = $b.Name State = (Get-Job -Id $b.jobId).State InstanceId = $b.InstanceId Config = $b.Config } } } } else { … describe 'Get-PoshBot’ { $script:botTracker.Add(1, @{ … }) $script:botTracker.Add(2, @{ … }) mock Get-Job { return 'Running’ } … it 'Returns a specific job instance’ { $j = Get-PoshBot -Id 1 $j | should not benullorempty $j.Id | should be 1 } } Input Output External External -
@glennsarti Types of tests - Unit Get-Job Inputs Outputs Mocks
Get-PoshBot $script:botTracker -
@glennsarti describe 'Get-PoshBot’ { $script:botTracker.Add(1, @{ … }) $script:botTracker.Add(2, @{
… }) mock Get-Job { return 'Running’ } … it 'Returns a specific job instance’ { $j = Get-PoshBot -Id 1 $j | should not benullorempty $j.Id | should be 1 } } Act Assert Arrange -
@glennsarti Types of tests - Integration Image adapted from Test
Pyramid – Martin Fowler 1. Slower 2. More $$$ 3. Complex -
@glennsarti Describe "$($script:DSCResourceName)_Integration" { Context 'Disable NetBIOS over TCP/IP’ {
$configData = @{ AllNodes = @( @{ NodeName = 'localhost’ InterfaceAlias = $netAdapter.Name Setting = 'Disable’ } ) } It 'Should compile and apply the MOF without throwing’ { { & "$($script:DSCResourceName)_Config" -OutputPath $TestDrive –ConfigurationData … Start-DscConfiguration -Path $TestDrive -ComputerName localhost -Wait –Verbose … } | Should -Not -Throw } It 'Should be able to call Get-DscConfiguration without throwing’ { { Get-DscConfiguration -Verbose -ErrorAction Stop } | Should -Not -Throw } -
@glennsarti Describe "$($script:DSCResourceName)_Integration" { Context 'Disable NetBIOS over TCP/IP' {
$configData = @{ AllNodes = @( @{ NodeName = 'localhost' InterfaceAlias = $netAdapter.Name Setting = 'Disable' } ) } It 'Should compile and apply the MOF without throwing' { { & "$($script:DSCResourceName)_Config" -OutputPath $TestDrive –ConfigurationData … Start-DscConfiguration -Path $TestDrive -ComputerName localhost -Wait –Verbose … } | Should -Not -Throw } It 'Should be able to call Get-DscConfiguration without throwing' { { Get-DscConfiguration -Verbose -ErrorAction Stop } | Should -Not -Throw } -
@glennsarti Types of tests - Integration Unit Integration Number 1393
140 Duration (s) 440 184 Average (s) 0.316 1.314 Taken from the xNetworking DSC Module Over 4x longer -
@glennsarti Types of tests – Unit (Again?) Not all unit
tests are the same! -
@glennsarti Types of tests – Unit (Again?) Function Get-SvcDisplayName($ServiceName )
{ (Get-WMIObject -Class Win32_Service | ? { $_.Name -eq $ServiceName }).DisplayName } Describe 'Get-SvcDisplayName’ { Mock Get-WMIObject { @{ 'Name' = 'MockService’; 'DisplayName' = 'MockDisplayName'} } it 'returns the DisplayName of a service’ { Get-SvcDisplayName 'MockService’ | Should Be 'MockDisplayName’ } } -
@glennsarti Types of tests – Unit (Again?) Function Get-SvcDisplayName($ServiceName )
{ } REDACTED Describe 'Get-SvcDisplayName’ { it 'returns the DisplayName of a service' { Get-SvcDisplayName 'WinRM' | Should Be 'Windows Remote Management (WS-Management)' } } -
@glennsarti Types of tests – Unit (Black box) Black box
White box Behavioural “What it does” Implementation “How it does it” -
@glennsarti Types of tests – Unit (Black box) Black box
White box Behavioural “What it does” Implementation “How it does it” QA Focused Dev Focused -
@glennsarti Types of tests – Unit (Black box) Black box
White box Behavioural “What it does” Implementation “How it does it” Change of mindset Fragile -
@glennsarti Types of tests – Unit (Black box) Function Get-SvcDisplayName($ServiceName
) { (Get-WMIObject -Class Win32_Service | ? { $_.Name -eq $ServiceName }).DisplayName } Describe 'Get-SvcDisplayName' { Mock Get-WMIObject { @{ 'Name' = 'MockService'; 'DisplayName' = 'MockDisplayName'} } it 'returns the DisplayName of a service' { Get-SvcDisplayName 'MockService' | Should Be 'MockDisplayName' } } -
@glennsarti Types of tests – Unit (Black box) Function Get-SvcDisplayName($ServiceName
) { (Get-Service –ServiceName $ServiceName | ForEach-Object { $_.DisplayName } } Describe 'Get-SvcDisplayName' { Mock Get-WMIObject { @{ 'Name' = 'MockService'; 'DisplayName' = 'MockDisplayName'} } it 'returns the DisplayName of a service' { Get-SvcDisplayName 'MockService' | Should Be 'MockDisplayName' } } -
@glennsarti Types of tests – Unit (Black box) White box
Black box • Only use parameters • Wrap with private functions -
@glennsarti Types of tests – Unit (Black box) “Tools do
one thing … Tools don’t know where their input data has generated … Tools accept all input only from their parameters … Tools don’t know how their output will be used, “ - The Many Forms of Scripting: which to use - Manning -
@glennsarti Types of tests – Unit (Black box) Which is
better? Which should I use? … and of course -
@glennsarti Types of tests – Unit (Black box) Should I
test private functions? -
@glennsarti 1.Unit (White box) 2.Integration 3.Unit (Black box) 4.Unit (Again
again?) Types of tests -
@glennsarti Types of tests - Characterisation - Michael C. Feathers
“A characterization test is a test that characterizes the actual behavior of a piece of code ... The tests document the actual current behavior of the system” -
@glennsarti Function Step-ByOne ($value) { Write-Output ($value + 2) }
# Unit Test it 'should increment by one' { Step-ByOne 1 | Should be 2 } # Characterisation Test it 'should increment by two' { Step-ByOne 1 | Should be 3 } -
@glennsarti Types of tests - Characterisation “… legacy code is
simply code without tests.” - Michael C. Feathers -
@glennsarti Types of tests - Characterisation function Start-LegacyCode { #
Really # horrible # code # which # should # be # refactored } -
@glennsarti Types of tests - Characterisation function Start-LegacyCode { #
Really # horrible # code # which # should # be # refactored } Characterisation Tests Characterisation Tests -
@glennsarti Types of tests - Characterisation function Start-LegacyCode { Start-Good
# code # which # should # be # refactored } Characterisation Tests function Start-Good { # Good code } Unit Tests -
@glennsarti Types of tests - Characterisation function Start-LegacyCode { Start-Good
Start-Clean # should # be # refactored } Characterisation Tests function Start-Good { # Good code } function Start-Clean { # Clean code } Unit Tests Unit Tests -
@glennsarti Types of tests - Characterisation function Start-LegacyCode { Start-Good
Start-Clean Start-Nice } Characterisation Tests function Start-Good { # Good code } function Start-Clean { # Clean code } function Start-Nice { # Nice code } Unit Tests Unit Tests Unit Tests -
@glennsarti Types of tests - Characterisation function Start-LegacyCode { Start-Good
Start-Clean Start-Nice } function Start-Good { # Good code } function Start-Clean { # Clean code } function Start-Nice { # Nice code } Unit Tests Unit Tests Unit Tests Unit Tests -
@glennsarti Types of tests - Characterisation • I can’t get
this class into a test harness • I need to make many changes in one area • I need to make a change, but I don’t know what tests to write • I’m changing the same code all over the place • I need to change a monster method and I can’t write tests for it • How do I know that I’m not breaking anything? -
@glennsarti Types of tests Unit (White box) Tests how a
function is supposed to operate Integration Tests how units interact with each other Unit (Black box) Tests the behaviour of a function without knowing how it works Characterisation Used to make changing legacy code safer -
@glennsarti 0 10 20 30 40 50 60 70 80
90 1 2 3 4 5 6 7 8 Duration (sec) Parallel Pester -
@glennsarti Garbled log output Describing MSFT_xFirewallProfile\Get-TargetResource Executing script C:\Source\ParallelPester\demo-repos\xNetworking\Tests\Unit\MSFT_xHostsFile.Tests.ps1 [+]
Should return current default gateway [+] Should return a "System.Collections.Hashtable" object type [+] Should return true 6.89s [+] Should return correct DNS Client Global Settings values Context Firewall Profile Exists 6.68s 6.67s 6.03s ? ? ? -
@glennsarti Subtle race conditions • Use TestDrive: ,Get-Random or GUIDs
• Explicitly setup testing state Remember - Arrange, Act, Assert -
@glennsarti 0 10 20 30 40 50 60 70 80
90 1 2 3 4 5 6 7 8 Duration (sec) 2 CPU 4 CPU -
@glennsarti “Testing is a Forever Thing” - The Pester Book
… but tests are mortal! -
@glennsarti { "list": [ { "buildJobTestId": 0, "name": "should return
false", "fileName": "invoking with state disabled b… "framework": "NUnit", "outcome": "passed", "duration": 189, "errorMessage": "", "errorStackTrace": "", "stdOut": "", "stdErr": "", "created": "2017-03-06T21:50:22.6918035+00:00" }, … -
@glennsarti Build Test Test Test Test Test Test Test Test
Test Skipped https://github.com/glennsarti/PSSummit2018 -
@glennsarti 0 200 400 600 800 1000 1200 1400 1600
1800 2.10.314.0 2.10.326.0 2.10.337.0 2.11.348.0 2.12.359.0 2.12.370.0 2.5.186.0 2.5.197.0 2.5.209.0 2.5.222.0 2.5.236.0 2.6.247.0 2.7.259.0 2.7.270.0 2.8.282.0 2.8.295.0 2.9.307.0 3.0.385.0 3.0.396.0 3.1.407.0 3.1.418.0 3.1.429.0 3.1.475.0 3.1.487.0 3.1.498.0 3.1.509.0 3.1.520.0 3.1.532.0 3.1.543.0 3.1.554.0 3.1.565.0 3.1.576.0 3.1.588.0 3.1.599.0 3.1.611.0 3.1.622.0 3.1.633.0 3.1.644.0 3.1.655.0 3.1.666.0 3.1.677.0 3.1.688.0 3.1.699.0 3.1.710.0 3.1.721.0 3.1.732.0 3.1.743.0 3.1.754.0 3.1.765.0 3.1.776.0 3.1.787.0 Build History (DSC Networking module) Passed Failed Skipped -
@glennsarti 0 5 10 15 20 25 30 35 40
1 2 3 4 5 6 9 10 11 12 13 14 21 22 23 27 28 36 38 39 45 62 Number of Builds Number of Failures Failure Profile (DSC Networking module) -
@glennsarti 0 5 10 15 20 25 30 35 40
45 Avg Duration (s) Top 100 slowest tests, in last 5 builds Long running integration tests? -
@glennsarti 0 5 10 15 20 25 30 35 40
45 Avg Duration (s) Tests that have never failed Test that have never failed? 208 times!! -
@glennsarti ▪ Tests that always fail together? ▪ Which tests
fail too often? ▪ Which files tend to fail tests? -
@glennsarti # Tagging a Describe block Describe 'Smoke Tests' -Tag
'High' { It 'Should test something important' { # ... } } -
@glennsarti Test name Failures "Common Tests -Validate Markdown Files; Context:
- Should not have errors in any markdown files" 29 "Common Tests - File Formatting; Context: - Should not contain files without a newline at the end" 16 "MSFT_xNetAdapterRsc_Integration; Context: - should be able to call Get- DscConfiguration without throwing" 11 "Pester - MSFT_xRoute_Add_Integration.Should have set the resource and all the parameters should match" 11 … next 17 integration tests Common Tests - File Formatting; Context: - Should not contain any files with tab characters 10 DSC Networking module -
@glennsarti PS> Invoke-Pester -Tag 'High' PS> Invoke-Pester -Tag 'High','Medium' PS>
Invoke-Pester -ExcludeTag 'Low' Watch out for untagged tests! -
@glennsarti version: 1.0.{build} environment: matrix: - TEST_TIER: High - TEST_TIER:
Medium - TEST_TIER: Low build: off matrix: fast_finish: true install: - ps: Install-Module Pester -SkipPublisherCheck -Force -Confirm:$false test_script: - ps: Invoke-Pester -Tag $ENV:TEST_TIER Example for AppVeyor CI 1 2 3 -
@glennsarti Say what you mean and mean what you say.
- George S Patton (Probably) -
@glennsarti “Should not contain files without a newline at the
end” DSC Networking module “Should contain files with a newline at the end” -
@glennsarti The Pester Book it 'when the Web-Mgmt-Service feature is
already installed, it attempts to change the WMSvc service startup type to Automatic' { $null = Enable-IISRemoteManagement -ComputerName 'SOMETHING' $assMParams = @{ CommandName = 'Set-Service' Times = 1 Scope = 'It' Exactly = $true ParameterFilter = { $ComputerName -eq 'SOMETHING' } } Assert-MockCalled @assMParams } -
@glennsarti The Pester Book it 'when the Web-Mgmt-Service feature is
already installed, it attempts to change the WMSvc service startup type to Automatic' { $null = Enable-IISRemoteManagement -ComputerName 'SOMETHING' $assMParams = @{ CommandName = 'Set-Service' Times = 1 Scope = 'It' Exactly = $true ParameterFilter = { $ComputerName -eq 'SOMETHING' } } Assert-MockCalled @assMParams } -
@glennsarti The Pester Book it 'when the Web-Mgmt-Service feature is
already installed, it attempts to change the WMSvc service startup type to Automatic' { $null = Enable-IISRemoteManagement -ComputerName 'SOMETHING' $assMParams = @{ CommandName = 'Set-Service' Times = 1 Scope = 'It' Exactly = $true ParameterFilter = { $ComputerName -eq 'SOMETHING' } } Assert-MockCalled @assMParams } -
@glennsarti The Pester Book it 'when the Web-Mgmt-Service feature is
already installed, it attempts to change the WMSvc service startup type to Automatic' { $null = Enable-IISRemoteManagement -ComputerName 'SOMETHING' $assMParams = @{ CommandName = 'Set-Service' Times = 1 Scope = 'It' Exactly = $true ParameterFilter = { $ComputerName -eq 'SOMETHING' –and $Name -eq 'WMSvc' –and $StartupType -eq 'Automatic' } } Assert-MockCalled @assMParams } -
@glennsarti Unit (White box) Integration Unit (Black box) Characterisation Reduce
duration Remove tests Reorder tests General maintenance -
PowerShell Conference Singapore 2018 @glennsarti http://glennsarti.github.io/ https://speakerdeck.com/glennsarti Please fill in
your survey – it’s how we do better! #PowerShell #PSConfAsia @Psconf.Asia -
Types of software testing https://www.guru99.com/types-of-software-testing.html Pester book https://leanpub.com/pesterbook Images https://unsplash.com
Dunning Kruger Effect https://www.xonitek.com/lessons-from-mt-stupid/ Test Pyramid – Martin Fowler https://martinfowler.com/bliki/TestPyramid.html Resources -
Arrange, Act, Assert http://wiki.c2.com/?ArrangeActAssert Working effectively with Legacy Code https://www.amazon.com/Working-Effectively-Legacy-Michael-Feathers/dp/0131177052
Parallel Pester https://github.com/glennsarti/ParallelPester/tree/feature-parallel-mode PoshBot https://github.com/poshbotio/PoshBot The Many Forms of Scripting: which to use – Manning http://freecontent.manning.com/the-many-forms-of-scripting-which-to-use/ Resources