(PS Conf Asia '18) Beyond Pester 101: Applying testing principles to PowerShell

Aac3dafaab7a7c2063d2526ba5936305?s=47 Glenn
October 18, 2018

(PS Conf Asia '18) Beyond Pester 101: Applying testing principles to PowerShell

We see a lot talks on testing PowerShell with Pester, but are the tests we write good tests? What makes a test "good"? How do we measure how effective our tests are? This talk will help you answer these questions, including why testing is important and how to apply these principles to your project.

Aac3dafaab7a7c2063d2526ba5936305?s=128

Glenn

October 18, 2018
Tweet

Transcript

  1. PowerShell Conference Asia Beyond Pester 101 Applying testing principles to

    PowerShell Glenn Sarti Software Engineer, Windows Team
  2. Powershell quick starts, tutorials, API reference, and code examples! Advance

    your career with hands- on training at docs.microsoft Check it out here:
  3. We really do! Show some love and join the conversation

    by tweeting @MicrosoftTech
  4. PowerShell Conference Asia Beyond Pester 101 Applying testing principles to

    PowerShell Glenn Sarti Software Engineer, Windows Team
  5. @glennsarti A familiar st ry…

  6. @glennsarti Happiness Time Tests?

  7. @glennsarti Happiness Time YAY!

  8. @glennsarti Happiness Time Hrmm

  9. @glennsarti Happiness Time 

  10. None
  11. @glennsarti Happiness Time ☺

  12. @glennsarti Dunning - Kruger Effect Confidence Knowledge / Skills Mt.

    Stupid Valley of Despair Slope of Enlightenment
  13. @glennsarti Beginning the ascent …

  14. @glennsarti Why do I test?

  15. @glennsarti To reduce the risk that a user will experience

    an unexpected behaviour
  16. @glennsarti To reduce the risk that a user will experience

    an unexpected behaviour
  17. @glennsarti • Shorter delivery cycles • Produce high quality code

    • Cheaper Benefits of testing
  18. @glennsarti • To ensure what is created does what it’s

    supposed to do • Form of documentation Benefits of testing
  19. @glennsarti Kinds of tests?

  20. @glennsarti 1.Unit 2. 3. 4. Types of tests

  21. @glennsarti Types of tests - Unit Image adapted from Test

    Pyramid – Martin Fowler 1. Fast 2. Cheap 3. Single concern
  22. @glennsarti function Get-PoshBot { … param( [parameter(ValueFromPipeline, ValueFromPipelineByPropertyName) ] [int[]]$Id

    = @() ) … process { if ($Id.Count -gt 0) { foreach ($item in $Id) { if ($b = $script:botTracker.$item) { [pscustomobject][ordered]@{ Id = $item Name = $b.Name State = (Get-Job -Id $b.jobId).State InstanceId = $b.InstanceId Config = $b.Config } } } } else { …
  23. @glennsarti function Get-PoshBot { … param( [parameter(ValueFromPipeline, ValueFromPipelineByPropertyName) ] [int[]]$Id

    = @() ) … process { if ($Id.Count -gt 0) { foreach ($item in $Id) { if ($b = $script:botTracker.$item) { [pscustomobject][ordered]@{ Id = $item Name = $b.Name State = (Get-Job -Id $b.jobId).State InstanceId = $b.InstanceId Config = $b.Config } } } } else { … Input Output External External
  24. @glennsarti function Get-PoshBot { … param( [parameter(ValueFromPipeline, ValueFromPipelineByPropertyName) ] [int[]]$Id

    = @() ) … process { if ($Id.Count -gt 0) { foreach ($item in $Id) { if ($b = $script:botTracker.$item) { [pscustomobject][ordered]@{ Id = $item Name = $b.Name State = (Get-Job -Id $b.jobId).State InstanceId = $b.InstanceId Config = $b.Config } } } } else { … describe 'Get-PoshBot’ { $script:botTracker.Add(1, @{ … }) $script:botTracker.Add(2, @{ … }) mock Get-Job { return 'Running’ } … it 'Returns a specific job instance’ { $j = Get-PoshBot -Id 1 $j | should not benullorempty $j.Id | should be 1 } } Input Output External External
  25. @glennsarti function Get-PoshBot { … param( [parameter(ValueFromPipeline, ValueFromPipelineByPropertyName) ] [int[]]$Id

    = @() ) … process { if ($Id.Count -gt 0) { foreach ($item in $Id) { if ($b = $script:botTracker.$item) { [pscustomobject][ordered]@{ Id = $item Name = $b.Name State = (Get-Job -Id $b.jobId).State InstanceId = $b.InstanceId Config = $b.Config } } } } else { … describe 'Get-PoshBot’ { $script:botTracker.Add(1, @{ … }) $script:botTracker.Add(2, @{ … }) mock Get-Job { return 'Running’ } … it 'Returns a specific job instance’ { $j = Get-PoshBot -Id 1 $j | should not benullorempty $j.Id | should be 1 } } Input Output External External
  26. @glennsarti Types of tests - Unit Get-Job Inputs Outputs Mocks

    Get-PoshBot $script:botTracker
  27. @glennsarti describe 'Get-PoshBot’ { $script:botTracker.Add(1, @{ … }) $script:botTracker.Add(2, @{

    … }) mock Get-Job { return 'Running’ } … it 'Returns a specific job instance’ { $j = Get-PoshBot -Id 1 $j | should not benullorempty $j.Id | should be 1 } } Act Assert Arrange
  28. @glennsarti 1.Unit 2.Integration 3. 4. Types of tests

  29. @glennsarti Types of tests - Integration Image adapted from Test

    Pyramid – Martin Fowler 1. Slower 2. More $$$ 3. Complex
  30. @glennsarti Types of tests - Integration

  31. @glennsarti Describe "$($script:DSCResourceName)_Integration" { Context 'Disable NetBIOS over TCP/IP’ {

    $configData = @{ AllNodes = @( @{ NodeName = 'localhost’ InterfaceAlias = $netAdapter.Name Setting = 'Disable’ } ) } It 'Should compile and apply the MOF without throwing’ { { & "$($script:DSCResourceName)_Config" -OutputPath $TestDrive –ConfigurationData … Start-DscConfiguration -Path $TestDrive -ComputerName localhost -Wait –Verbose … } | Should -Not -Throw } It 'Should be able to call Get-DscConfiguration without throwing’ { { Get-DscConfiguration -Verbose -ErrorAction Stop } | Should -Not -Throw }
  32. @glennsarti Describe "$($script:DSCResourceName)_Integration" { Context 'Disable NetBIOS over TCP/IP' {

    $configData = @{ AllNodes = @( @{ NodeName = 'localhost' InterfaceAlias = $netAdapter.Name Setting = 'Disable' } ) } It 'Should compile and apply the MOF without throwing' { { & "$($script:DSCResourceName)_Config" -OutputPath $TestDrive –ConfigurationData … Start-DscConfiguration -Path $TestDrive -ComputerName localhost -Wait –Verbose … } | Should -Not -Throw } It 'Should be able to call Get-DscConfiguration without throwing' { { Get-DscConfiguration -Verbose -ErrorAction Stop } | Should -Not -Throw }
  33. @glennsarti Types of tests - Integration Unit Integration Number 1393

    140 Duration (s) 440 184 Average (s) 0.316 1.314 Taken from the xNetworking DSC Module Over 4x longer
  34. @glennsarti 1.Unit 2.Integration 3.Unit (Again?) 4. Types of tests

  35. @glennsarti Types of tests – Unit (Again?) Not all unit

    tests are the same!
  36. @glennsarti Types of tests – Unit (Again?) Function Get-SvcDisplayName($ServiceName )

    { (Get-WMIObject -Class Win32_Service | ? { $_.Name -eq $ServiceName }).DisplayName } Describe 'Get-SvcDisplayName’ { Mock Get-WMIObject { @{ 'Name' = 'MockService’; 'DisplayName' = 'MockDisplayName'} } it 'returns the DisplayName of a service’ { Get-SvcDisplayName 'MockService’ | Should Be 'MockDisplayName’ } }
  37. @glennsarti Types of tests – Unit (Again?) Function Get-SvcDisplayName($ServiceName )

    { } REDACTED Describe 'Get-SvcDisplayName’ { it 'returns the DisplayName of a service' { Get-SvcDisplayName 'WinRM' | Should Be 'Windows Remote Management (WS-Management)' } }
  38. @glennsarti Types of tests – Unit (Black box) Black box

    White box
  39. @glennsarti Types of tests – Unit (Black box) Black box

    White box Behavioural “What it does” Implementation “How it does it”
  40. @glennsarti Types of tests – Unit (Black box) Black box

    White box Behavioural “What it does” Implementation “How it does it” QA Focused Dev Focused
  41. @glennsarti Types of tests – Unit (Black box) Black box

    White box Behavioural “What it does” Implementation “How it does it” Change of mindset Fragile
  42. @glennsarti Types of tests – Unit (Black box) Function Get-SvcDisplayName($ServiceName

    ) { (Get-WMIObject -Class Win32_Service | ? { $_.Name -eq $ServiceName }).DisplayName } Describe 'Get-SvcDisplayName' { Mock Get-WMIObject { @{ 'Name' = 'MockService'; 'DisplayName' = 'MockDisplayName'} } it 'returns the DisplayName of a service' { Get-SvcDisplayName 'MockService' | Should Be 'MockDisplayName' } }
  43. @glennsarti Types of tests – Unit (Black box) Function Get-SvcDisplayName($ServiceName

    ) { (Get-Service –ServiceName $ServiceName | ForEach-Object { $_.DisplayName } } Describe 'Get-SvcDisplayName' { Mock Get-WMIObject { @{ 'Name' = 'MockService'; 'DisplayName' = 'MockDisplayName'} } it 'returns the DisplayName of a service' { Get-SvcDisplayName 'MockService' | Should Be 'MockDisplayName' } }
  44. @glennsarti Types of tests – Unit (Black box) White box

    Black box • Only use parameters • Wrap with private functions
  45. @glennsarti Types of tests – Unit (Black box) “Tools do

    one thing … Tools don’t know where their input data has generated … Tools accept all input only from their parameters … Tools don’t know how their output will be used, “ - The Many Forms of Scripting: which to use - Manning
  46. @glennsarti Types of tests – Unit (Black box) Which is

    better? Which should I use? … and of course
  47. @glennsarti Types of tests – Unit (Black box) Should I

    test private functions?
  48. @glennsarti 1.Unit (White box) 2.Integration 3.Unit (Black box) 4. Types

    of tests
  49. @glennsarti 1.Unit (White box) 2.Integration 3.Unit (Black box) 4.Unit (Again

    again?) Types of tests
  50. @glennsarti 1.Unit (White box) 2.Integration 3.Unit (Black box) 4.Characterisation Types

    of tests
  51. @glennsarti Types of tests - Characterisation - Michael C. Feathers

    “A characterization test is a test that characterizes the actual behavior of a piece of code ... The tests document the actual current behavior of the system”
  52. @glennsarti Function Step-ByOne ($value) { Write-Output ($value + 2) }

    # Unit Test it 'should increment by one' { Step-ByOne 1 | Should be 2 } # Characterisation Test it 'should increment by two' { Step-ByOne 1 | Should be 3 }
  53. @glennsarti Types of tests - Characterisation What use are they?

  54. @glennsarti Types of tests - Characterisation “… legacy code is

    simply code without tests.” - Michael C. Feathers
  55. @glennsarti Types of tests - Characterisation function Start-LegacyCode { #

    Really # horrible # code # which # should # be # refactored }
  56. @glennsarti Types of tests - Characterisation function Start-LegacyCode { #

    Really # horrible # code # which # should # be # refactored } Characterisation Tests Characterisation Tests
  57. @glennsarti Types of tests - Characterisation function Start-LegacyCode { Start-Good

    # code # which # should # be # refactored } Characterisation Tests function Start-Good { # Good code } Unit Tests
  58. @glennsarti Types of tests - Characterisation function Start-LegacyCode { Start-Good

    Start-Clean # should # be # refactored } Characterisation Tests function Start-Good { # Good code } function Start-Clean { # Clean code } Unit Tests Unit Tests
  59. @glennsarti Types of tests - Characterisation function Start-LegacyCode { Start-Good

    Start-Clean Start-Nice } Characterisation Tests function Start-Good { # Good code } function Start-Clean { # Clean code } function Start-Nice { # Nice code } Unit Tests Unit Tests Unit Tests
  60. @glennsarti Types of tests - Characterisation function Start-LegacyCode { Start-Good

    Start-Clean Start-Nice } function Start-Good { # Good code } function Start-Clean { # Clean code } function Start-Nice { # Nice code } Unit Tests Unit Tests Unit Tests Unit Tests
  61. @glennsarti Types of tests - Characterisation • I can’t get

    this class into a test harness • I need to make many changes in one area • I need to make a change, but I don’t know what tests to write • I’m changing the same code all over the place • I need to change a monster method and I can’t write tests for it • How do I know that I’m not breaking anything?
  62. @glennsarti Types of tests Unit (White box) Tests how a

    function is supposed to operate Integration Tests how units interact with each other Unit (Black box) Tests the behaviour of a function without knowing how it works Characterisation Used to make changing legacy code safer
  63. @glennsarti The second half of the slope …

  64. @glennsarti Tending your test suite

  65. @glennsarti Reduce duration Remove tests Reorder tests General maintenance

  66. @glennsarti Invoke-ParallelPester https://github.com/glennsarti/ ParallelPester/tree/feature-parallel-mode

  67. @glennsarti Pester 5 4 3 2 1 Output

  68. @glennsarti ParallelPester 5 4 3 2 1 Output

  69. @glennsarti 0 10 20 30 40 50 60 70 80

    90 1 2 3 4 5 6 7 8 Duration (sec) Parallel Pester
  70. @glennsarti Downsides?

  71. @glennsarti Garbled log output Describing MSFT_xFirewallProfile\Get-TargetResource Executing script C:\Source\ParallelPester\demo-repos\xNetworking\Tests\Unit\MSFT_xHostsFile.Tests.ps1 [+]

    Should return current default gateway [+] Should return a "System.Collections.Hashtable" object type [+] Should return true 6.89s [+] Should return correct DNS Client Global Settings values Context Firewall Profile Exists 6.68s 6.67s 6.03s ? ? ?
  72. @glennsarti Code Coverage

  73. @glennsarti Subtle race conditions • Use TestDrive: ,Get-Random or GUIDs

    • Explicitly setup testing state Remember - Arrange, Act, Assert
  74. @glennsarti Not suitable for everything

  75. @glennsarti 0 10 20 30 40 50 60 70 80

    90 1 2 3 4 5 6 7 8 Duration (sec) 2 CPU 4 CPU
  76. @glennsarti Reduce duration Remove tests Reorder tests General maintenance

  77. @glennsarti “Testing is a Forever Thing” - The Pester Book

    … but tests are mortal!
  78. @glennsarti Use data! (Pester Output Files)

  79. @glennsarti

  80. @glennsarti TESTS 1075

  81. @glennsarti /projects/Powershell/xnetworking/history?recordsNumber=10 Invoke-RESTMethod https://ci.appveyor.com/api/ /projects/Powershell/xnetworking/build/${buildVersion} /buildjobs/${jobID}/tests

  82. @glennsarti { "list": [ { "buildJobTestId": 0, "name": "should return

    false", "fileName": "invoking with state disabled b… "framework": "NUnit", "outcome": "passed", "duration": 189, "errorMessage": "", "errorStackTrace": "", "stdOut": "", "stdErr": "", "created": "2017-03-06T21:50:22.6918035+00:00" }, …
  83. @glennsarti Build Test Test Test Test Test Test Test Test

    Test Skipped https://github.com/glennsarti/PSSummit2018
  84. @glennsarti 0 200 400 600 800 1000 1200 1400 1600

    1800 2.10.314.0 2.10.326.0 2.10.337.0 2.11.348.0 2.12.359.0 2.12.370.0 2.5.186.0 2.5.197.0 2.5.209.0 2.5.222.0 2.5.236.0 2.6.247.0 2.7.259.0 2.7.270.0 2.8.282.0 2.8.295.0 2.9.307.0 3.0.385.0 3.0.396.0 3.1.407.0 3.1.418.0 3.1.429.0 3.1.475.0 3.1.487.0 3.1.498.0 3.1.509.0 3.1.520.0 3.1.532.0 3.1.543.0 3.1.554.0 3.1.565.0 3.1.576.0 3.1.588.0 3.1.599.0 3.1.611.0 3.1.622.0 3.1.633.0 3.1.644.0 3.1.655.0 3.1.666.0 3.1.677.0 3.1.688.0 3.1.699.0 3.1.710.0 3.1.721.0 3.1.732.0 3.1.743.0 3.1.754.0 3.1.765.0 3.1.776.0 3.1.787.0 Build History (DSC Networking module) Passed Failed Skipped
  85. @glennsarti 0 5 10 15 20 25 30 35 40

    1 2 3 4 5 6 9 10 11 12 13 14 21 22 23 27 28 36 38 39 45 62 Number of Builds Number of Failures Failure Profile (DSC Networking module)
  86. @glennsarti 0 5 10 15 20 25 30 35 40

    45 Avg Duration (s) Top 100 slowest tests, in last 5 builds Long running integration tests?
  87. @glennsarti 0 5 10 15 20 25 30 35 40

    45 Avg Duration (s) Tests that have never failed Test that have never failed? 208 times!!
  88. @glennsarti ▪ Tests that always fail together? ▪ Which tests

    fail too often? ▪ Which files tend to fail tests?
  89. @glennsarti Reduce duration Remove tests Reorder tests General maintenance

  90. @glennsarti Test Tiering HIGH MEDIUM LOW

  91. @glennsarti # Tagging a Describe block Describe 'Smoke Tests' -Tag

    'High' { It 'Should test something important' { # ... } }
  92. @glennsarti Test name Failures "Common Tests -Validate Markdown Files; Context:

    - Should not have errors in any markdown files" 29 "Common Tests - File Formatting; Context: - Should not contain files without a newline at the end" 16 "MSFT_xNetAdapterRsc_Integration; Context: - should be able to call Get- DscConfiguration without throwing" 11 "Pester - MSFT_xRoute_Add_Integration.Should have set the resource and all the parameters should match" 11 … next 17 integration tests Common Tests - File Formatting; Context: - Should not contain any files with tab characters 10 DSC Networking module
  93. @glennsarti PS> Invoke-Pester -Tag 'High' PS> Invoke-Pester -Tag 'High','Medium' PS>

    Invoke-Pester -ExcludeTag 'Low' Watch out for untagged tests!
  94. @glennsarti Test Tiering HIGH MEDIUM LOW Execution Order

  95. @glennsarti version: 1.0.{build} environment: matrix: - TEST_TIER: High - TEST_TIER:

    Medium - TEST_TIER: Low build: off matrix: fast_finish: true install: - ps: Install-Module Pester -SkipPublisherCheck -Force -Confirm:$false test_script: - ps: Invoke-Pester -Tag $ENV:TEST_TIER Example for AppVeyor CI 1 2 3
  96. @glennsarti Test Tiering Next Level - Execution Cadence HIGH MEDIUM

    LOW
  97. @glennsarti Reduce duration Remove tests Reorder tests General maintenance

  98. @glennsarti Say what you mean and mean what you say.

    - George S Patton (Probably)
  99. @glennsarti “Should not contain files without a newline at the

    end” DSC Networking module “Should contain files with a newline at the end”
  100. @glennsarti The Pester Book it 'when the Web-Mgmt-Service feature is

    already installed, it attempts to change the WMSvc service startup type to Automatic' { $null = Enable-IISRemoteManagement -ComputerName 'SOMETHING' $assMParams = @{ CommandName = 'Set-Service' Times = 1 Scope = 'It' Exactly = $true ParameterFilter = { $ComputerName -eq 'SOMETHING' } } Assert-MockCalled @assMParams }
  101. @glennsarti The Pester Book it 'when the Web-Mgmt-Service feature is

    already installed, it attempts to change the WMSvc service startup type to Automatic' { $null = Enable-IISRemoteManagement -ComputerName 'SOMETHING' $assMParams = @{ CommandName = 'Set-Service' Times = 1 Scope = 'It' Exactly = $true ParameterFilter = { $ComputerName -eq 'SOMETHING' } } Assert-MockCalled @assMParams }
  102. @glennsarti The Pester Book it 'when the Web-Mgmt-Service feature is

    already installed, it attempts to change the WMSvc service startup type to Automatic' { $null = Enable-IISRemoteManagement -ComputerName 'SOMETHING' $assMParams = @{ CommandName = 'Set-Service' Times = 1 Scope = 'It' Exactly = $true ParameterFilter = { $ComputerName -eq 'SOMETHING' } } Assert-MockCalled @assMParams }
  103. @glennsarti The Pester Book it 'when the Web-Mgmt-Service feature is

    already installed, it attempts to change the WMSvc service startup type to Automatic' { $null = Enable-IISRemoteManagement -ComputerName 'SOMETHING' $assMParams = @{ CommandName = 'Set-Service' Times = 1 Scope = 'It' Exactly = $true ParameterFilter = { $ComputerName -eq 'SOMETHING' –and $Name -eq 'WMSvc' –and $StartupType -eq 'Automatic' } } Assert-MockCalled @assMParams }
  104. @glennsarti Wrapping up…

  105. @glennsarti Unit (White box) Integration Unit (Black box) Characterisation Reduce

    duration Remove tests Reorder tests General maintenance
  106. @glennsarti Where to now?

  107. PowerShell Conference Singapore 2018 @glennsarti http://glennsarti.github.io/ https://speakerdeck.com/glennsarti Please fill in

    your survey – it’s how we do better! #PowerShell #PSConfAsia @Psconf.Asia
  108. Types of software testing https://www.guru99.com/types-of-software-testing.html Pester book https://leanpub.com/pesterbook Images https://unsplash.com

    Dunning Kruger Effect https://www.xonitek.com/lessons-from-mt-stupid/ Test Pyramid – Martin Fowler https://martinfowler.com/bliki/TestPyramid.html Resources
  109. Arrange, Act, Assert http://wiki.c2.com/?ArrangeActAssert Working effectively with Legacy Code https://www.amazon.com/Working-Effectively-Legacy-Michael-Feathers/dp/0131177052

    Parallel Pester https://github.com/glennsarti/ParallelPester/tree/feature-parallel-mode PoshBot https://github.com/poshbotio/PoshBot The Many Forms of Scripting: which to use – Manning http://freecontent.manning.com/the-many-forms-of-scripting-which-to-use/ Resources