Beyond Pester 101: Applying testing principles to PowerShell

Beyond Pester 101
Applying testing principles to PowerShell
Glenn Sarti
Software Engineer, Windows Team

View Slide

@glennsarti
A familiar st ry…

View Slide

@glennsarti
Happiness
Time
Tests?

View Slide

@glennsarti
Happiness
Time
YAY!

View Slide

@glennsarti
Happiness
Time
Hrmm

View Slide

@glennsarti
Happiness
Time


View Slide

View Slide

@glennsarti
Happiness
Time
☺

View Slide

@glennsarti
Dunning - Kruger Effect
Confidence
Knowledge / Skills
Mt. Stupid
Valley of
Despair
Slope of
Enlightenment

View Slide

@glennsarti
Beginning the
ascent …

View Slide

@glennsarti
Why do I test?

View Slide

@glennsarti
To reduce the risk that a
user will experience an
unexpected behaviour

View Slide

@glennsarti
• Shorter delivery cycles
• Produce high quality code
• Cheaper
Benefits of testing

View Slide

@glennsarti
• To ensure what is created does
what it’s supposed to do
• Form of documentation
Benefits of testing

View Slide

@glennsarti
Kinds of
tests?

View Slide

@glennsarti
1.Unit
2.
3.
4.
Types of tests

View Slide

@glennsarti
Types of tests - Unit
Image adapted from Test Pyramid – Martin Fowler
1. Fast
2. Cheap
3. Single concern

View Slide

@glennsarti
function Get-PoshBot { …
param(
[parameter(ValueFromPipeline,
ValueFromPipelineByPropertyName)
]
[int[]]$Id = @()
)
…
process {
if ($Id.Count -gt 0) {
foreach ($item in $Id) {
if ($b = $script:botTracker.$item) {
[pscustomobject][ordered]@{
Id = $item
Name = $b.Name
State = (Get-Job -Id $b.jobId).State
InstanceId = $b.InstanceId
Config = $b.Config
}
}
}
} else {
…

View Slide

@glennsarti
param(
]
[int[]]$Id = @()
)
…
process {
Id = $item
Name = $b.Name
Config = $b.Config
}
}
}
} else {
…
Input
Output
External
External

View Slide

@glennsarti
param(
]
[int[]]$Id = @()
)
…
process {
Id = $item
Name = $b.Name
Config = $b.Config
}
}
}
} else {
…
describe 'Get-PoshBot’ {
$script:botTracker.Add(1, @{
…
})
…
})
mock Get-Job { return 'Running’ }
…
it 'Returns a specific job instance’ {
$j = Get-PoshBot -Id 1
$j | should not benullorempty
$j.Id | should be 1
}
}
Input
Output
External
External

View Slide

@glennsarti
Types of tests - Unit
Get-Job
Inputs
Outputs
Mocks
Get-PoshBot $script:botTracker

View Slide

@glennsarti
describe 'Get-PoshBot’ {
…
})
…
})
mock Get-Job { return 'Running’ }
…
it 'Returns a specific job instance’ {
$j = Get-PoshBot -Id 1
$j | should not benullorempty
$j.Id | should be 1
}
}
Act
Assert
Arrange

View Slide

@glennsarti
1.Unit
2.
3.
4.
Types of tests
Brian Burke:
How to love Unit Testing!

View Slide

@glennsarti
1.Unit
2.Integration
3.
4.
Types of tests

View Slide

@glennsarti
Types of tests - Integration
Image adapted from Test Pyramid – Martin Fowler
1. Slower
2. More $$$
3. Complex

View Slide

@glennsarti

View Slide

@glennsarti
Describe "$($script:DSCResourceName)_Integration" {
Context 'Disable NetBIOS over TCP/IP’ {
$configData = @{
AllNodes = @(
@{
NodeName = 'localhost’
InterfaceAlias = $netAdapter.Name
Setting = 'Disable’
}
)
}
It 'Should compile and apply the MOF without throwing’ {
{
& "$($script:DSCResourceName)_Config" -OutputPath $TestDrive –ConfigurationData …
Start-DscConfiguration -Path $TestDrive -ComputerName localhost -Wait –Verbose …
} | Should -Not -Throw
}
It 'Should be able to call Get-DscConfiguration without throwing’ {
{ Get-DscConfiguration -Verbose -ErrorAction Stop } | Should -Not -Throw
}

View Slide

@glennsarti
Describe "$($script:DSCResourceName)_Integration" {
Context 'Disable NetBIOS over TCP/IP' {
$configData = @{
AllNodes = @(
@{
NodeName = 'localhost'
InterfaceAlias = $netAdapter.Name
Setting = 'Disable'
}
)
}
It 'Should compile and apply the MOF without throwing' {
{
& "$($script:DSCResourceName)_Config" -OutputPath $TestDrive –ConfigurationData …
Start-DscConfiguration -Path $TestDrive -ComputerName localhost -Wait –Verbose …
} | Should -Not -Throw
}
It 'Should be able to call Get-DscConfiguration without throwing' {
{ Get-DscConfiguration -Verbose -ErrorAction Stop } | Should -Not -Throw
}

View Slide

@glennsarti
Unit Integration
Number 1393 140
Duration (s) 440 184
Average (s) 0.316 1.314
Taken from the xNetworking DSC Module
Over 4x longer

View Slide

@glennsarti
1.Unit
2.Integration
3.Unit (Again?)
4.
Types of tests

View Slide

@glennsarti
Types of tests – Unit (Again?)
Not all unit tests
are the same!

View Slide

@glennsarti
Function Get-SvcDisplayName($ServiceName ) {
(Get-WMIObject -Class Win32_Service |
? { $_.Name -eq $ServiceName }).DisplayName
}
Describe 'Get-SvcDisplayName’ {
Mock Get-WMIObject { @{ 'Name' = 'MockService’;
'DisplayName' = 'MockDisplayName'} }
it 'returns the DisplayName of a service’ {
Get-SvcDisplayName 'MockService’ |
Should Be 'MockDisplayName’
}
}

View Slide

@glennsarti
}
REDACTED
Describe 'Get-SvcDisplayName’ {
it 'returns the DisplayName of a service' {
Get-SvcDisplayName 'WinRM' |
Should Be 'Windows Remote Management (WS-Management)'
}
}

View Slide

@glennsarti
Types of tests – Unit (Black box)
Black box White box

View Slide

@glennsarti
Black box White box
Behavioural
“What it does”
Implementation
“How it does it”

View Slide

@glennsarti
Black box White box
Behavioural
“What it does”
Implementation
QA Focused Dev Focused

View Slide

@glennsarti
Black box White box
Behavioural
“What it does”
Implementation
Change of
mindset
Fragile

View Slide

@glennsarti
(Get-WMIObject -Class Win32_Service |
? { $_.Name -eq $ServiceName }).DisplayName
}
Describe 'Get-SvcDisplayName' {
Mock Get-WMIObject { @{ 'Name' = 'MockService';
Get-SvcDisplayName 'MockService' |
Should Be 'MockDisplayName'
}
}

View Slide

@glennsarti
(Get-Service –ServiceName $ServiceName |
ForEach-Object { $_.DisplayName }
}
Describe 'Get-SvcDisplayName' {
Mock Get-WMIObject { @{ 'Name' = 'MockService';
Get-SvcDisplayName 'MockService' |
Should Be 'MockDisplayName'
}
}

View Slide

@glennsarti
White box Black box
• Only use parameters
• Wrap with private functions

View Slide

@glennsarti
“Tools do one thing … Tools don’t
know where their input data has
generated … Tools accept all input
only from their parameters … Tools
don’t know how their output will be
used, “
- The Many Forms of Scripting: which to use - Manning

View Slide

@glennsarti
Which is better?
Which should I use?
… and of course

View Slide

@glennsarti
Should I test
private functions?

View Slide

@glennsarti
1.Unit (White box)
2.Integration
3.Unit (Black box)
4.
Types of tests

View Slide

@glennsarti
1.Unit (White box)
2.Integration
3.Unit (Black box)
4.Unit (Again again?)
Types of tests

View Slide

@glennsarti
1.Unit (White box)
2.Integration
3.Unit (Black box)
4.Characterisation
Types of tests

View Slide

@glennsarti
Types of tests - Characterisation
- Michael C. Feathers
“A characterization test is a test that
characterizes the actual behavior of a
piece of code ... The tests document
the actual current behavior of the
system”

View Slide

@glennsarti
Function Step-ByOne ($value) {
Write-Output ($value + 2)
}
# Unit Test
it 'should increment by one' {
Step-ByOne 1 | Should be 2
}
# Characterisation Test
it 'should increment by two' {
Step-ByOne 1 | Should be 3
}

View Slide

@glennsarti
What use are they?

View Slide

@glennsarti
“… legacy code is
simply code without
tests.”
- Michael C. Feathers

View Slide

@glennsarti
function Start-LegacyCode {
# Really
# horrible
# code
# which
# should
# be
# refactored
}

View Slide

@glennsarti
# Really
# horrible
# code
# which
# should
# be
# refactored
}
Characterisation Tests

View Slide

@glennsarti
Start-Good
# code
# which
# should
# be
# refactored
}
function Start-Good {
# Good code
}
Unit
Tests

View Slide

@glennsarti
Start-Good
Start-Clean
# should
# be
# refactored
}
# Good code
}
function Start-Clean {
# Clean code
}
Unit
Tests
Unit
Tests

View Slide

@glennsarti
Start-Good
Start-Clean
Start-Nice
}
# Good code
}
# Clean code
}
function Start-Nice {
# Nice code
}
Unit
Tests
Unit
Tests
Unit
Tests

View Slide

@glennsarti
Start-Good
Start-Clean
Start-Nice
}
# Good code
}
# Clean code
}
function Start-Nice {
# Nice code
}
Unit Tests
Unit
Tests
Unit
Tests
Unit
Tests

View Slide

@glennsarti
• I can’t get this class into a test harness
• I need to make many changes in one area
• I need to make a change, but I don’t know what tests to
write
• I’m changing the same code all over the place
• I need to change a monster method and I can’t write
tests for it
• How do I know that I’m not breaking anything?

View Slide

@glennsarti
Types of tests
Unit (White box) Tests how a function is supposed to operate
Integration Tests how units interact with each other
Unit (Black box) Tests the behaviour of a function without
knowing how it works
Characterisation Used to make changing legacy code safer

View Slide

@glennsarti
The second half
of the slope …

View Slide

@glennsarti
Tending your
test suite

View Slide

@glennsarti
Reduce duration
Remove tests
Reorder tests
General maintenance

View Slide

@glennsarti
Invoke-ParallelPester
https://github.com/glennsarti/
ParallelPester/tree/feature-parallel-mode

View Slide

@glennsarti
Pester
5
4
3
2
1
Output

View Slide

@glennsarti
ParallelPester
5
4
3
2
1
Output

View Slide

@glennsarti
0
10
20
30
40
50
60
70
80
90
1 2 3 4 5 6 7 8
Duration (sec)
Parallel
Pester

View Slide

@glennsarti
Downsides?

View Slide

@glennsarti
Garbled log output
Describing MSFT_xFirewallProfile\Get-TargetResource
Executing script C:\Source\ParallelPester\demo-repos\xNetworking\Tests\Unit\MSFT_xHostsFile.Tests.ps1
[+] Should return current default gateway
[+] Should return a "System.Collections.Hashtable" object type [+] Should return true 6.89s
[+] Should return correct DNS Client Global Settings values Context Firewall Profile Exists
6.68s
6.67s
6.03s
? ?
?

View Slide

@glennsarti
Code Coverage

View Slide

@glennsarti
Subtle race conditions
• Use TestDrive: ,Get-Random or GUIDs
• Explicitly setup testing state
Remember - Arrange, Act, Assert

View Slide

@glennsarti
Not suitable for
everything

View Slide

@glennsarti
0
10
20
30
40
50
60
70
80
90
1 2 3 4 5 6 7 8
Duration (sec)
2 CPU
4 CPU

View Slide

@glennsarti
Reduce duration
Remove tests
Reorder tests
General maintenance

View Slide

@glennsarti
“Testing is a Forever Thing”
- The Pester Book
… but tests are mortal!

View Slide

@glennsarti
Use data!
(Pester Output Files)

View Slide

@glennsarti

View Slide

@glennsarti
TESTS 1075

View Slide

@glennsarti
/projects/Powershell/xnetworking/history?recordsNumber=10
Invoke-RESTMethod
https://ci.appveyor.com/api/
/projects/Powershell/xnetworking/build/${buildVersion}
/buildjobs/${jobID}/tests

View Slide

@glennsarti
{
"list": [
{
"buildJobTestId": 0,
"name": "should return false",
"fileName": "invoking with state disabled b…
"framework": "NUnit",
"outcome": "passed",
"duration": 189,
"errorMessage": "",
"errorStackTrace": "",
"stdOut": "",
"stdErr": "",
"created": "2017-03-06T21:50:22.6918035+00:00"
},
…

View Slide

@glennsarti
Build Test
Test
Test
Test
Test
Test
Test
Test
Test
Skipped
https://github.com/glennsarti/PSSummit2018

View Slide

@glennsarti
0
200
400
600
800
1000
1200
1400
1600
1800
2.10.314.0
2.10.326.0
2.10.337.0
2.11.348.0
2.12.359.0
2.12.370.0
2.5.186.0
2.5.197.0
2.5.209.0
2.5.222.0
2.5.236.0
2.6.247.0
2.7.259.0
2.7.270.0
2.8.282.0
2.8.295.0
2.9.307.0
3.0.385.0
3.0.396.0
3.1.407.0
3.1.418.0
3.1.429.0
3.1.475.0
3.1.487.0
3.1.498.0
3.1.509.0
3.1.520.0
3.1.532.0
3.1.543.0
3.1.554.0
3.1.565.0
3.1.576.0
3.1.588.0
3.1.599.0
3.1.611.0
3.1.622.0
3.1.633.0
3.1.644.0
3.1.655.0
3.1.666.0
3.1.677.0
3.1.688.0
3.1.699.0
3.1.710.0
3.1.721.0
3.1.732.0
3.1.743.0
3.1.754.0
3.1.765.0
3.1.776.0
3.1.787.0
Build History (DSC Networking module)
Passed Failed Skipped

View Slide

@glennsarti
0
5
10
15
20
25
30
35
40
1 2 3 4 5 6 9 10 11 12 13 14 21 22 23 27 28 36 38 39 45 62
Number of Builds
Number of Failures
Failure Profile (DSC Networking module)

View Slide

@glennsarti
0
5
10
15
20
25
30
35
40
45
Avg Duration (s)
Top 100 slowest tests, in last 5 builds
Long running integration tests?

View Slide

@glennsarti
0
5
10
15
20
25
30
35
40
45
Avg Duration (s)
Tests that have never failed
Test that have never failed?
208 times!!

View Slide

@glennsarti
▪ Tests that always fail together?
▪ Which tests fail too often?
▪ Which files tend to fail tests?

View Slide

@glennsarti
Reduce duration
Remove tests
Reorder tests
General maintenance

View Slide

@glennsarti
Test Tiering
HIGH MEDIUM LOW

View Slide

@glennsarti
# Tagging a Describe block
Describe 'Smoke Tests' -Tag 'High' {
It 'Should test something important' {
# ...
}
}

View Slide

@glennsarti
Test name Failures
"Common Tests -Validate Markdown Files; Context: - Should not have errors
in any markdown files"
29
"Common Tests - File Formatting; Context: - Should not contain files without
a newline at the end"
16
"MSFT_xNetAdapterRsc_Integration; Context: - should be able to call Get-
DscConfiguration without throwing"
11
"Pester - MSFT_xRoute_Add_Integration.Should have set the resource and all
the parameters should match"
11
… next 17 integration tests
Common Tests - File Formatting; Context: - Should not contain any files with
tab characters
10
DSC Networking module

View Slide

@glennsarti
PS> Invoke-Pester -Tag 'High'
PS> Invoke-Pester -Tag 'High','Medium'
PS> Invoke-Pester -ExcludeTag 'Low'
Watch out for untagged tests!

View Slide

@glennsarti
Test Tiering
HIGH
MEDIUM
LOW
Execution Order

View Slide

@glennsarti
version: 1.0.{build}
environment:
matrix:
- TEST_TIER: High
- TEST_TIER: Medium
- TEST_TIER: Low
build: off
matrix:
fast_finish: true
install:
- ps: Install-Module Pester -SkipPublisherCheck -Force -Confirm:$false
test_script:
- ps: Invoke-Pester -Tag $ENV:TEST_TIER
Example for AppVeyor CI
1
2
3

View Slide

@glennsarti
Test Tiering
Next Level - Execution Cadence
HIGH
MEDIUM
LOW

View Slide

@glennsarti
Reduce duration
Remove tests
Reorder tests
General maintenance

View Slide

@glennsarti
Say what you mean and
mean what you say.
- George S Patton (Probably)

View Slide

@glennsarti
“Should not contain files without a newline at
the end”
DSC Networking module
“Should contain files with a newline at the end”

View Slide

@glennsarti
The Pester Book
it 'when the Web-Mgmt-Service feature is already installed, it attempts to change
the WMSvc service startup type to Automatic' {
$null = Enable-IISRemoteManagement -ComputerName 'SOMETHING'
$assMParams = @{
CommandName = 'Set-Service'
Times = 1
Scope = 'It'
Exactly = $true
ParameterFilter = { $ComputerName -eq 'SOMETHING' }
} Assert-MockCalled @assMParams
}

View Slide

@glennsarti
The Pester Book
it 'when the Web-Mgmt-Service feature is already installed, it attempts to change
the WMSvc service startup type to Automatic' {
$null = Enable-IISRemoteManagement -ComputerName 'SOMETHING'
$assMParams = @{
CommandName = 'Set-Service'
Times = 1
Scope = 'It'
Exactly = $true
ParameterFilter = { $ComputerName -eq 'SOMETHING' –and
$Name -eq 'WMSvc' –and
$StartupType -eq 'Automatic'
}
} Assert-MockCalled @assMParams
}

View Slide

@glennsarti
Wrapping up…

View Slide

@glennsarti
Unit (White box)
Integration
Unit (Black box)
Characterisation
Reduce duration
Remove tests
Reorder tests
General
maintenance

View Slide

@glennsarti
Where to now?

View Slide

Please use the event app or Sched.com
to submit a session rating!
@glennsarti
http://glennsarti.github.io/
https://speakerdeck.com/glennsarti

View Slide

Types of software testing
https://www.guru99.com/types-of-software-testing.html
Pester book
https://leanpub.com/pesterbook
Images
https://unsplash.com
Dunning Kruger Effect
https://www.xonitek.com/lessons-from-mt-stupid/
Test Pyramid – Martin Fowler
https://martinfowler.com/bliki/TestPyramid.html
Resources

View Slide

Arrange, Act, Assert
http://wiki.c2.com/?ArrangeActAssert
Working effectively with Legacy Code
https://www.amazon.com/Working-Effectively-Legacy-Michael-Feathers/dp/0131177052
Parallel Pester
https://github.com/glennsarti/ParallelPester/tree/feature-parallel-mode
PoshBot
https://github.com/poshbotio/PoshBot
The Many Forms of Scripting: which to use – Manning
http://freecontent.manning.com/the-many-forms-of-scripting-which-to-use/
Resources

View Slide

Beyond Pester 101: Applying testing principles to PowerShell

Beyond Pester 101: Applying testing principles to PowerShell

Glenn

More Decks by Glenn

Other Decks in Technology

Featured

Transcript