Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥

(PS Conf Asia '18) Beyond Pester 101: Applying ...

Glenn
October 18, 2018

(PS Conf Asia '18) Beyond Pester 101: Applying testing principles to PowerShell

We see a lot talks on testing PowerShell with Pester, but are the tests we write good tests? What makes a test "good"? How do we measure how effective our tests are? This talk will help you answer these questions, including why testing is important and how to apply these principles to your project.

Glenn

October 18, 2018
Tweet

More Decks by Glenn

Other Decks in Technology

Transcript

  1. PowerShell Conference Asia Beyond Pester 101 Applying testing principles to

    PowerShell Glenn Sarti Software Engineer, Windows Team
  2. Powershell quick starts, tutorials, API reference, and code examples! Advance

    your career with hands- on training at docs.microsoft Check it out here:
  3. PowerShell Conference Asia Beyond Pester 101 Applying testing principles to

    PowerShell Glenn Sarti Software Engineer, Windows Team
  4. @glennsarti Dunning - Kruger Effect Confidence Knowledge / Skills Mt.

    Stupid Valley of Despair Slope of Enlightenment
  5. @glennsarti • To ensure what is created does what it’s

    supposed to do • Form of documentation Benefits of testing
  6. @glennsarti Types of tests - Unit Image adapted from Test

    Pyramid – Martin Fowler 1. Fast 2. Cheap 3. Single concern
  7. @glennsarti function Get-PoshBot { … param( [parameter(ValueFromPipeline, ValueFromPipelineByPropertyName) ] [int[]]$Id

    = @() ) … process { if ($Id.Count -gt 0) { foreach ($item in $Id) { if ($b = $script:botTracker.$item) { [pscustomobject][ordered]@{ Id = $item Name = $b.Name State = (Get-Job -Id $b.jobId).State InstanceId = $b.InstanceId Config = $b.Config } } } } else { …
  8. @glennsarti function Get-PoshBot { … param( [parameter(ValueFromPipeline, ValueFromPipelineByPropertyName) ] [int[]]$Id

    = @() ) … process { if ($Id.Count -gt 0) { foreach ($item in $Id) { if ($b = $script:botTracker.$item) { [pscustomobject][ordered]@{ Id = $item Name = $b.Name State = (Get-Job -Id $b.jobId).State InstanceId = $b.InstanceId Config = $b.Config } } } } else { … Input Output External External
  9. @glennsarti function Get-PoshBot { … param( [parameter(ValueFromPipeline, ValueFromPipelineByPropertyName) ] [int[]]$Id

    = @() ) … process { if ($Id.Count -gt 0) { foreach ($item in $Id) { if ($b = $script:botTracker.$item) { [pscustomobject][ordered]@{ Id = $item Name = $b.Name State = (Get-Job -Id $b.jobId).State InstanceId = $b.InstanceId Config = $b.Config } } } } else { … describe 'Get-PoshBot’ { $script:botTracker.Add(1, @{ … }) $script:botTracker.Add(2, @{ … }) mock Get-Job { return 'Running’ } … it 'Returns a specific job instance’ { $j = Get-PoshBot -Id 1 $j | should not benullorempty $j.Id | should be 1 } } Input Output External External
  10. @glennsarti function Get-PoshBot { … param( [parameter(ValueFromPipeline, ValueFromPipelineByPropertyName) ] [int[]]$Id

    = @() ) … process { if ($Id.Count -gt 0) { foreach ($item in $Id) { if ($b = $script:botTracker.$item) { [pscustomobject][ordered]@{ Id = $item Name = $b.Name State = (Get-Job -Id $b.jobId).State InstanceId = $b.InstanceId Config = $b.Config } } } } else { … describe 'Get-PoshBot’ { $script:botTracker.Add(1, @{ … }) $script:botTracker.Add(2, @{ … }) mock Get-Job { return 'Running’ } … it 'Returns a specific job instance’ { $j = Get-PoshBot -Id 1 $j | should not benullorempty $j.Id | should be 1 } } Input Output External External
  11. @glennsarti describe 'Get-PoshBot’ { $script:botTracker.Add(1, @{ … }) $script:botTracker.Add(2, @{

    … }) mock Get-Job { return 'Running’ } … it 'Returns a specific job instance’ { $j = Get-PoshBot -Id 1 $j | should not benullorempty $j.Id | should be 1 } } Act Assert Arrange
  12. @glennsarti Types of tests - Integration Image adapted from Test

    Pyramid – Martin Fowler 1. Slower 2. More $$$ 3. Complex
  13. @glennsarti Describe "$($script:DSCResourceName)_Integration" { Context 'Disable NetBIOS over TCP/IP’ {

    $configData = @{ AllNodes = @( @{ NodeName = 'localhost’ InterfaceAlias = $netAdapter.Name Setting = 'Disable’ } ) } It 'Should compile and apply the MOF without throwing’ { { & "$($script:DSCResourceName)_Config" -OutputPath $TestDrive –ConfigurationData … Start-DscConfiguration -Path $TestDrive -ComputerName localhost -Wait –Verbose … } | Should -Not -Throw } It 'Should be able to call Get-DscConfiguration without throwing’ { { Get-DscConfiguration -Verbose -ErrorAction Stop } | Should -Not -Throw }
  14. @glennsarti Describe "$($script:DSCResourceName)_Integration" { Context 'Disable NetBIOS over TCP/IP' {

    $configData = @{ AllNodes = @( @{ NodeName = 'localhost' InterfaceAlias = $netAdapter.Name Setting = 'Disable' } ) } It 'Should compile and apply the MOF without throwing' { { & "$($script:DSCResourceName)_Config" -OutputPath $TestDrive –ConfigurationData … Start-DscConfiguration -Path $TestDrive -ComputerName localhost -Wait –Verbose … } | Should -Not -Throw } It 'Should be able to call Get-DscConfiguration without throwing' { { Get-DscConfiguration -Verbose -ErrorAction Stop } | Should -Not -Throw }
  15. @glennsarti Types of tests - Integration Unit Integration Number 1393

    140 Duration (s) 440 184 Average (s) 0.316 1.314 Taken from the xNetworking DSC Module Over 4x longer
  16. @glennsarti Types of tests – Unit (Again?) Function Get-SvcDisplayName($ServiceName )

    { (Get-WMIObject -Class Win32_Service | ? { $_.Name -eq $ServiceName }).DisplayName } Describe 'Get-SvcDisplayName’ { Mock Get-WMIObject { @{ 'Name' = 'MockService’; 'DisplayName' = 'MockDisplayName'} } it 'returns the DisplayName of a service’ { Get-SvcDisplayName 'MockService’ | Should Be 'MockDisplayName’ } }
  17. @glennsarti Types of tests – Unit (Again?) Function Get-SvcDisplayName($ServiceName )

    { } REDACTED Describe 'Get-SvcDisplayName’ { it 'returns the DisplayName of a service' { Get-SvcDisplayName 'WinRM' | Should Be 'Windows Remote Management (WS-Management)' } }
  18. @glennsarti Types of tests – Unit (Black box) Black box

    White box Behavioural “What it does” Implementation “How it does it”
  19. @glennsarti Types of tests – Unit (Black box) Black box

    White box Behavioural “What it does” Implementation “How it does it” QA Focused Dev Focused
  20. @glennsarti Types of tests – Unit (Black box) Black box

    White box Behavioural “What it does” Implementation “How it does it” Change of mindset Fragile
  21. @glennsarti Types of tests – Unit (Black box) Function Get-SvcDisplayName($ServiceName

    ) { (Get-WMIObject -Class Win32_Service | ? { $_.Name -eq $ServiceName }).DisplayName } Describe 'Get-SvcDisplayName' { Mock Get-WMIObject { @{ 'Name' = 'MockService'; 'DisplayName' = 'MockDisplayName'} } it 'returns the DisplayName of a service' { Get-SvcDisplayName 'MockService' | Should Be 'MockDisplayName' } }
  22. @glennsarti Types of tests – Unit (Black box) Function Get-SvcDisplayName($ServiceName

    ) { (Get-Service –ServiceName $ServiceName | ForEach-Object { $_.DisplayName } } Describe 'Get-SvcDisplayName' { Mock Get-WMIObject { @{ 'Name' = 'MockService'; 'DisplayName' = 'MockDisplayName'} } it 'returns the DisplayName of a service' { Get-SvcDisplayName 'MockService' | Should Be 'MockDisplayName' } }
  23. @glennsarti Types of tests – Unit (Black box) White box

    Black box • Only use parameters • Wrap with private functions
  24. @glennsarti Types of tests – Unit (Black box) “Tools do

    one thing … Tools don’t know where their input data has generated … Tools accept all input only from their parameters … Tools don’t know how their output will be used, “ - The Many Forms of Scripting: which to use - Manning
  25. @glennsarti Types of tests – Unit (Black box) Which is

    better? Which should I use? … and of course
  26. @glennsarti Types of tests - Characterisation - Michael C. Feathers

    “A characterization test is a test that characterizes the actual behavior of a piece of code ... The tests document the actual current behavior of the system”
  27. @glennsarti Function Step-ByOne ($value) { Write-Output ($value + 2) }

    # Unit Test it 'should increment by one' { Step-ByOne 1 | Should be 2 } # Characterisation Test it 'should increment by two' { Step-ByOne 1 | Should be 3 }
  28. @glennsarti Types of tests - Characterisation “… legacy code is

    simply code without tests.” - Michael C. Feathers
  29. @glennsarti Types of tests - Characterisation function Start-LegacyCode { #

    Really # horrible # code # which # should # be # refactored }
  30. @glennsarti Types of tests - Characterisation function Start-LegacyCode { #

    Really # horrible # code # which # should # be # refactored } Characterisation Tests Characterisation Tests
  31. @glennsarti Types of tests - Characterisation function Start-LegacyCode { Start-Good

    # code # which # should # be # refactored } Characterisation Tests function Start-Good { # Good code } Unit Tests
  32. @glennsarti Types of tests - Characterisation function Start-LegacyCode { Start-Good

    Start-Clean # should # be # refactored } Characterisation Tests function Start-Good { # Good code } function Start-Clean { # Clean code } Unit Tests Unit Tests
  33. @glennsarti Types of tests - Characterisation function Start-LegacyCode { Start-Good

    Start-Clean Start-Nice } Characterisation Tests function Start-Good { # Good code } function Start-Clean { # Clean code } function Start-Nice { # Nice code } Unit Tests Unit Tests Unit Tests
  34. @glennsarti Types of tests - Characterisation function Start-LegacyCode { Start-Good

    Start-Clean Start-Nice } function Start-Good { # Good code } function Start-Clean { # Clean code } function Start-Nice { # Nice code } Unit Tests Unit Tests Unit Tests Unit Tests
  35. @glennsarti Types of tests - Characterisation • I can’t get

    this class into a test harness • I need to make many changes in one area • I need to make a change, but I don’t know what tests to write • I’m changing the same code all over the place • I need to change a monster method and I can’t write tests for it • How do I know that I’m not breaking anything?
  36. @glennsarti Types of tests Unit (White box) Tests how a

    function is supposed to operate Integration Tests how units interact with each other Unit (Black box) Tests the behaviour of a function without knowing how it works Characterisation Used to make changing legacy code safer
  37. @glennsarti 0 10 20 30 40 50 60 70 80

    90 1 2 3 4 5 6 7 8 Duration (sec) Parallel Pester
  38. @glennsarti Garbled log output Describing MSFT_xFirewallProfile\Get-TargetResource Executing script C:\Source\ParallelPester\demo-repos\xNetworking\Tests\Unit\MSFT_xHostsFile.Tests.ps1 [+]

    Should return current default gateway [+] Should return a "System.Collections.Hashtable" object type [+] Should return true 6.89s [+] Should return correct DNS Client Global Settings values Context Firewall Profile Exists 6.68s 6.67s 6.03s ? ? ?
  39. @glennsarti Subtle race conditions • Use TestDrive: ,Get-Random or GUIDs

    • Explicitly setup testing state Remember - Arrange, Act, Assert
  40. @glennsarti 0 10 20 30 40 50 60 70 80

    90 1 2 3 4 5 6 7 8 Duration (sec) 2 CPU 4 CPU
  41. @glennsarti { "list": [ { "buildJobTestId": 0, "name": "should return

    false", "fileName": "invoking with state disabled b… "framework": "NUnit", "outcome": "passed", "duration": 189, "errorMessage": "", "errorStackTrace": "", "stdOut": "", "stdErr": "", "created": "2017-03-06T21:50:22.6918035+00:00" }, …
  42. @glennsarti Build Test Test Test Test Test Test Test Test

    Test Skipped https://github.com/glennsarti/PSSummit2018
  43. @glennsarti 0 200 400 600 800 1000 1200 1400 1600

    1800 2.10.314.0 2.10.326.0 2.10.337.0 2.11.348.0 2.12.359.0 2.12.370.0 2.5.186.0 2.5.197.0 2.5.209.0 2.5.222.0 2.5.236.0 2.6.247.0 2.7.259.0 2.7.270.0 2.8.282.0 2.8.295.0 2.9.307.0 3.0.385.0 3.0.396.0 3.1.407.0 3.1.418.0 3.1.429.0 3.1.475.0 3.1.487.0 3.1.498.0 3.1.509.0 3.1.520.0 3.1.532.0 3.1.543.0 3.1.554.0 3.1.565.0 3.1.576.0 3.1.588.0 3.1.599.0 3.1.611.0 3.1.622.0 3.1.633.0 3.1.644.0 3.1.655.0 3.1.666.0 3.1.677.0 3.1.688.0 3.1.699.0 3.1.710.0 3.1.721.0 3.1.732.0 3.1.743.0 3.1.754.0 3.1.765.0 3.1.776.0 3.1.787.0 Build History (DSC Networking module) Passed Failed Skipped
  44. @glennsarti 0 5 10 15 20 25 30 35 40

    1 2 3 4 5 6 9 10 11 12 13 14 21 22 23 27 28 36 38 39 45 62 Number of Builds Number of Failures Failure Profile (DSC Networking module)
  45. @glennsarti 0 5 10 15 20 25 30 35 40

    45 Avg Duration (s) Top 100 slowest tests, in last 5 builds Long running integration tests?
  46. @glennsarti 0 5 10 15 20 25 30 35 40

    45 Avg Duration (s) Tests that have never failed Test that have never failed? 208 times!!
  47. @glennsarti ▪ Tests that always fail together? ▪ Which tests

    fail too often? ▪ Which files tend to fail tests?
  48. @glennsarti # Tagging a Describe block Describe 'Smoke Tests' -Tag

    'High' { It 'Should test something important' { # ... } }
  49. @glennsarti Test name Failures "Common Tests -Validate Markdown Files; Context:

    - Should not have errors in any markdown files" 29 "Common Tests - File Formatting; Context: - Should not contain files without a newline at the end" 16 "MSFT_xNetAdapterRsc_Integration; Context: - should be able to call Get- DscConfiguration without throwing" 11 "Pester - MSFT_xRoute_Add_Integration.Should have set the resource and all the parameters should match" 11 … next 17 integration tests Common Tests - File Formatting; Context: - Should not contain any files with tab characters 10 DSC Networking module
  50. @glennsarti PS> Invoke-Pester -Tag 'High' PS> Invoke-Pester -Tag 'High','Medium' PS>

    Invoke-Pester -ExcludeTag 'Low' Watch out for untagged tests!
  51. @glennsarti version: 1.0.{build} environment: matrix: - TEST_TIER: High - TEST_TIER:

    Medium - TEST_TIER: Low build: off matrix: fast_finish: true install: - ps: Install-Module Pester -SkipPublisherCheck -Force -Confirm:$false test_script: - ps: Invoke-Pester -Tag $ENV:TEST_TIER Example for AppVeyor CI 1 2 3
  52. @glennsarti “Should not contain files without a newline at the

    end” DSC Networking module “Should contain files with a newline at the end”
  53. @glennsarti The Pester Book it 'when the Web-Mgmt-Service feature is

    already installed, it attempts to change the WMSvc service startup type to Automatic' { $null = Enable-IISRemoteManagement -ComputerName 'SOMETHING' $assMParams = @{ CommandName = 'Set-Service' Times = 1 Scope = 'It' Exactly = $true ParameterFilter = { $ComputerName -eq 'SOMETHING' } } Assert-MockCalled @assMParams }
  54. @glennsarti The Pester Book it 'when the Web-Mgmt-Service feature is

    already installed, it attempts to change the WMSvc service startup type to Automatic' { $null = Enable-IISRemoteManagement -ComputerName 'SOMETHING' $assMParams = @{ CommandName = 'Set-Service' Times = 1 Scope = 'It' Exactly = $true ParameterFilter = { $ComputerName -eq 'SOMETHING' } } Assert-MockCalled @assMParams }
  55. @glennsarti The Pester Book it 'when the Web-Mgmt-Service feature is

    already installed, it attempts to change the WMSvc service startup type to Automatic' { $null = Enable-IISRemoteManagement -ComputerName 'SOMETHING' $assMParams = @{ CommandName = 'Set-Service' Times = 1 Scope = 'It' Exactly = $true ParameterFilter = { $ComputerName -eq 'SOMETHING' } } Assert-MockCalled @assMParams }
  56. @glennsarti The Pester Book it 'when the Web-Mgmt-Service feature is

    already installed, it attempts to change the WMSvc service startup type to Automatic' { $null = Enable-IISRemoteManagement -ComputerName 'SOMETHING' $assMParams = @{ CommandName = 'Set-Service' Times = 1 Scope = 'It' Exactly = $true ParameterFilter = { $ComputerName -eq 'SOMETHING' –and $Name -eq 'WMSvc' –and $StartupType -eq 'Automatic' } } Assert-MockCalled @assMParams }
  57. @glennsarti Unit (White box) Integration Unit (Black box) Characterisation Reduce

    duration Remove tests Reorder tests General maintenance
  58. Types of software testing https://www.guru99.com/types-of-software-testing.html Pester book https://leanpub.com/pesterbook Images https://unsplash.com

    Dunning Kruger Effect https://www.xonitek.com/lessons-from-mt-stupid/ Test Pyramid – Martin Fowler https://martinfowler.com/bliki/TestPyramid.html Resources
  59. Arrange, Act, Assert http://wiki.c2.com/?ArrangeActAssert Working effectively with Legacy Code https://www.amazon.com/Working-Effectively-Legacy-Michael-Feathers/dp/0131177052

    Parallel Pester https://github.com/glennsarti/ParallelPester/tree/feature-parallel-mode PoshBot https://github.com/poshbotio/PoshBot The Many Forms of Scripting: which to use – Manning http://freecontent.manning.com/the-many-forms-of-scripting-which-to-use/ Resources