Getting Testy With Perl
Steven Lembark
Workhorse Computing
[email protected]
Slide 2
Slide 2 text
Cultured Perl
Perl's “test culture” is part of the language.
Test::* modules easily added to Perl code.
Smoke testing provided as part of CPAN.
Reporter accumulates results from installers.
Language structure, modules work together.
Slide 3
Slide 3 text
Where there's smoke, there's Perl.
CPAN::Reporter send back success/fail.
CPAN testers: variety of platforms, daily reports.
cpan-testers & cpan-testers-discuss mailing lists.
http://cpantesters.org/
Community effort at testing is unique to Perl.
Slide 4
Slide 4 text
Excellent Reference
Perl Testing: A developer's Notebook,
O'Reilly Press
Wonderful example of a good howto-book.
Includes modules, ways to use them with variations.
Good cover-to-cover read.
Slide 5
Slide 5 text
Perl is easy to test
Test::* modules do most of the work.
Use the wheels, don't re-invent:
Test::Simple, Test::More, Test::Deep,
Test::Builder, Test::Coverage, Object::Exercise
Perl adds introspection.
Slide 6
Slide 6 text
One-stop shopping
Perl as a great glue language.
Use perl to test other programs.
Have other programs output TAP.
Combined with “inline” you can test almost anything!
Slide 7
Slide 7 text
Test your tests!
Devel::Coverage have you tested all of the code?
Test::Builder roll your own tests.
Build you own re-usable test components:
Test::Harness::Straps
Test::Builder::Tester
Slide 8
Slide 8 text
“Test” code can do more
Pre-load fixed data into a database.
Validate the working environment.
Execute daily cleanups in cron.
Slide 9
Slide 9 text
Testing simply
Count hardwired:
use Test::Simple tests => 6;
Difference in count will be an error.
Slide 10
Slide 10 text
Its all “ok”
use Test::Simple tests => 2;
# ok( boolean, message );
ok 1 == 1, “Count: '1' (1)”;
ok 2 == 1, “Count: '2' (1)”;
Output like:
ok 1 - Count: '1' (1)
not ok 2 - Count: '2' (1)
Slide 11
Slide 11 text
Its all “ok”
use Test::Simple tests => 2;
# ok( boolean, message );
ok 1, 'It passed!';
Output like:
1..2
ok 1 - It passed!
# Looks like you planned 2 tests but ran 1.
Slide 12
Slide 12 text
Seen these before?
“not ok 10 - Count”
“not ok 11 - Format”
“not ok 5 - File or directory not found.”
OK, now what?
Slide 13
Slide 13 text
Designing tests:
Test what matters.
Report something useful.
Slide 14
Slide 14 text
Isolate the failure
What to fix:
ok -e $path , “Existing: '$path'”;
ok -d _ , “Directory: '$path'”;
ok -r _ , “Readable: '$path'”;
ok -w _ , “Writeable '$path'”;
Slide 15
Slide 15 text
Q: Why all of the quotes?
ok -e $path, “Existing dir: $path”
Slide 16
Slide 16 text
Q: Why all of the quotes?
ok -e $path, “Existing dir: $path”
not ok 1 - Existing dir: /var/tmp
not ok 1 Existing dir: /var/tmp
Slide 17
Slide 17 text
Q: Why all of the quotes?
ok -e $path, “Existing dir: '$path'”
not ok 1 - Existing dir: '/var/tmp '
not ok 1 Existing dir: '/var/tmp
'
Slide 18
Slide 18 text
Good general format
Include what you found, what you expect.
You'll need them both with “not ok”:
ok $got eq $want, 'Name: '$got' ($want)”;
not ok 99 – Name: '' (Jow Bloe)
not ok 99 – Name: '1 Main St.' (Jow Bloe)
not ok 99 – Name: 'Jow Bloe' ()
not ok 99 – Name: 'Jow Bloe' (User::Name=0x1A...
Slide 19
Slide 19 text
Test::Simple may be enough
Generate a plan.
“ok” generates TAP output.
Test count is fixed.
This works for many day-to-day tests.
Slide 20
Slide 20 text
But Wait! There's More!
Test::More
plan, done_testing test counts
is, isnt, like, unlike stringy “ok”
pass, fail skip the boolean
use_ok, require_ok module test
note, diag, explain messages
Slide 21
Slide 21 text
Flexable count
Set count with “plan”.
Compute from test input:
plan tests => 42;
plan tests => 3 * keys $config;
plan tests => 2 * values $query_output;
Test count varys?
Skip plan.
Slide 22
Slide 22 text
When you're done testing
Just say so:
done_testing;
Generates final report.
Plan count is checked here.
No plan, no count check.
Slide 23
Slide 23 text
Notes show up with “prove -v”
note “read_config from '$config_path'”;
my $path = read_config 'tmp_dir';
ok -e $path, “Exists: '$path' (tmp_dir)”;
$ prove -v;
# read_config from './etc/pass1.conf'
not ok 1 – Exists: '/var/tmp/foobar ' (tmp_dir)
Slide 24
Slide 24 text
Diagnostics show why tests failed
Show up without “-v”.
ok … or diag “Oopsie...”;
ok grep /blah/, @test_input, 'Found blah'
or diag “Missing 'blah':”, @test_input;
ok $dbh, “Connected”
or diag “Failed connect: '$dsn'”;
Slide 25
Slide 25 text
Explain shows exactly what failed
“explain” shows nested structure.
Use with “note” to show setup details.
With “diag” shows extra info on failure.
my $dbh = DBI->connect( @argz );
ok $dbh, 'Database connected'
or diag 'Connect args:', explain \@argz;
Slide 26
Slide 26 text
Stringy “ok”
like, not_like use regexen.
saves “=~” syntax in the tests:
like $stuff, qr/milk/, “Got: 'milk'?”
or diag “Have: '$stuff' instead”;
not ok 1 - Got: 'milk'?
# Have 'very new cheese' instead
Slide 27
Slide 27 text
Variable number of tests
If-logic generates variable number of tests.
Skip “plan”, use “done_testing”.
Slide 28
Slide 28 text
Taking it pass/fail
if( $obj = eval { $class->constructify } )
{
# validate object contents
}
else
{
# nothing more to test
fail “Failed constructify: $@”;
}
Slide 29
Slide 29 text
Test expected errors
eval
{
$obj->new( $junk );
fail "No exception: foo( $junk )?";
1
}
or do
{
# validate error handling
...
}
Slide 30
Slide 30 text
Controlling test cycle
Abort if everything will fail.
Avoid expensive, specialized, un-necesary tests.
Saves extraneous code in all of the tests.
Slide 31
Slide 31 text
BAIL_OUT: Knowing when to give up
Aborts all testing.
Unsafe or guaranteed failure.
Useful in early test for environment, arg checks.
Slide 32
Slide 32 text
BAIL_OUT
BAIL_OUT 'Do not run tests as su!' unless $>;
BAIL_OUT 'Missing $DB' unless $ENV{DB};
-e $test_dir or mkdir $dir, 0777
or BAIL_OUT “Failed mkdir: '$dir', $!”;
-d $test_dir or BAIL_OUT “Missing: '$dir'”;
-r _ or BAIL_OUT “Un-readable: '$dir'”;
-w _ or BAIL_OUT “Un-writeable: '$dir'”;
Slide 33
Slide 33 text
“SKIP” blocks
Skip a block with test count and message.
Adjust the plan test count.
Expensive or specialized tests.
Slide 34
Slide 34 text
Skip external tests
SKIP:
{
skip “No database (TEST_DB)”, 8 if ! $ENV{ TEST_DB };
# or … no network available...
# or … no server handle...
...
}
Slide 35
Slide 35 text
Skip expensive tests
SKIP:
{
skip “Used for internal development only”, 12
unless $ENV{ EXPENSIVE_TESTS };
# test plan count reduced by 12
...
}
Slide 36
Slide 36 text
Skip dangerous tests
SKIP:
{
skip “Unsafe as superuser”, 22 unless $>;
# at this point the tests are not running su.
...
}
Slide 37
Slide 37 text
You'll usually use Test::More
note & diag nearly always worth using.
plan & done_testing makes life easier.
Still have “ok” for the majority of work.
Slide 38
Slide 38 text
Testing Structures
“like” syntax with nested structure:
use Test::Deep;
cmp_deeply $found, $expect, $message;
Great for testing parser or grammar outcome.
Slide 39
Slide 39 text
Devel::Cover: what didn't you check?
All of the else blocks?
All of the “||=” assignments?
All of the “... or die ...” branches?
Devel::Cover bookkeeps running code.
Reports what you didn't test.
Tells you what test to write next.
Slide 40
Slide 40 text
Similar to NYTProf:
Run your program & summarize the results:
$ cover -test;
or
$ perl -MDevel::Cover ./t/foo.t;
$ cover;
Running “cover” generates the report.
Slide 41
Slide 41 text
Introspection simplifies testing
Want to test database connect failure.
Can't assume SQL::Lite.
Flat file databases are messy.
Guarantee that something fails?
You could write an operation that should fail.
Then again, it might not...
Slide 42
Slide 42 text
Making sure you fail
Override class methods.
Override Perl functions.
Result:
Mock objects,
Mock methods,
Mock perl.
Slide 43
Slide 43 text
Mock Objects
Test your wrapper handling failure?
Override DBI::connect with sub { die … }.
No guess: You know it's going to fail.
Slide 44
Slide 44 text
Mock Modules
Your test:
*DBI::connect = sub { die '…' };
my $status = eval { $obj->make_connection };
my $err = $@;
# test $status, $err, $obj...
Slide 45
Slide 45 text
Force an exception
use Symbol qw( qualify_to_ref );
# e.g., force_exception 'Invalid username', 'connect', 'DBI';
# ( name, packcage ) or ( “package::name” )
sub force_exception
{
chomp( my $messsage = shift );
my $ref = qualify_to_ref @_;
undef &{ *$ref };
*{ $ref } = sub { die “$message\n” };
return
}
Slide 46
Slide 46 text
Testing multiple failures
for( @testz )
{
my( $msg, $expect ) = @$_;
force_exception $msg, 'DBI::connect';
my $status = eval { $wrapper->connect };
my $err = $@;
# your tests here
}
Slide 47
Slide 47 text
Avoid caller cleanup
Override with “local”
sub force_exception
{
my ( $proto, $method, $pkg, $name, $msg ) = splice @_, 0, 5;
my $ref = qualify_to_ref $name, $pkg;
local *{ $ref } = sub { die “$msg” };
# exit from block cleans up local override.
# caller checks return, $@, $proto.
eval { $proto->$method( @_ ) }
}
Slide 48
Slide 48 text
Mock Perl
Override perl: *Core::Global::
Is “exit” is called?
my $exits = 0;
*Core::Global::exit = sub { $exits = 1 };
eval { frobnicate $arg };
ok $exits, “Frobnicate exits ($exits)”;
Slide 49
Slide 49 text
Devel::Cover & Mocking
The biggest reason for mock anything:
Force an outcome to test a branch.
Iterate:
Test with Devel::Cover.
See what still needs testing.
Mock object/method/function forces the branch.
Slide 50
Slide 50 text
Automated testing
Lazyness is a virtue.
Avoid cut+paste.
Let Perl do the work!
Slide 51
Slide 51 text
Example: Sanity check modules
Use symlinks.
Validate modules compile.
Check package argument.
require_ok $package; # ok if require-able.
use_ok $package; # ok if use-able.
Slide 52
Slide 52 text
Make the links
Path below ./lib.
Replace slashes with dashes.
Add a leading “01”.
Symlink them all to a generic baseline test.
Symlinks look like:
01-Wcurve-Util.t → generic-01-t
Slide 53
Slide 53 text
use FindBin::libs;
use Test::More;
use File::Basename qw( basename );
my $madness = basename $0, '.t'; # 01-WCurve-Util
$madness =~ s{^ 01-}{}x; # WCurve-Util
$madness =~ s{ \W+ }{::}gx; # WCUrve::Util
if( use_ok $madness )
{
# check for correct package argument.
ok $madness->can( 'VERSION' ), “$maddness has a method”;
ok $a = $madness->VERSION, “$madness is '$a'”;
}
done_testing;
./t/generic-01-t
Similar checks for config files
Add “00-” tests to reads config files.
Combine “ok” with Config::Any.
Now make-generic-links tests config & code.
“prove” runs them all in one pass.
Slide 56
Slide 56 text
The Validation Two-Step
Use with git for sanity check:
git pull &&
./t/make-generic-links &&
prove --jobs=2 –-state=save &&
git tag -a “prove/$date” &&
git push &&
git push –-tag ;
Slide 57
Slide 57 text
Exercise for healthy objects
Data-driven testing for a single object.
Data includes method, data, expected return.
Module iterates the tests, reports results.
When tests or expected values are generated.
Slide 58
Slide 58 text
use Object::Exercise;
my @testz =
(
[
[ qw( name ) ], # method + args
[ q{Jow Bloe} ], # expected return
],
[
[ qw( street ) ],
[ q{1 Nuwere St} ],
],
[
[ qw( street number ) ], # $obj->address( 'number' )
[ qw( 1 ) ],
'Check street number' # hardwired message
],
);
Person->new( 'Jow Blow' => '1 Nuwere St' )->$exercise( @testz );
Slide 59
Slide 59 text
Load fixed data
Flat file -> arrayrefs
“insert” as method, [1] as return.
Load the data with:
$sth->$exercise( @data )
Get “ok” message for each of the data record.
Slide 60
Slide 60 text
Roll your own: Test::Builder
Mirrors Test::More with a singleton object.
my $test = Test::Builder->new;
$test->ok( $boolean, $success_message )
or $test->diag( $more_info );
Spread single test across multiple modules.
Slide 61
Slide 61 text
Testing Testing
Test::Build::Tester wraps your tests.
Forces test to return not-ok in order to test it.
Ignores the output of the test being tested.
Validate the test plan.
Slide 62
Slide 62 text
Getting the most out of prove
Save state: re-run failed tests:
$prove –state=save;
# fix something...
$prove –state=save,failed;
Parallel execution with “--jobs”.
“perldoc” is your friend!
Slide 63
Slide 63 text
Great, but my shop uses
Test multiple languages with Inline.
Include C, C++, Python...
Multiple Perl versions.
Mix languages to test interfaces.
Slide 64
Slide 64 text
One-stop teting for your API lib's
Need to test multi-language support?
Use Perl to move data between them.
Unit-test C talking to Java.
Slide 65
Slide 65 text
Example: Testing your C code
use Inline C;
use Test::Simple tests => 1;
my $a = 12 + 34;
my $b = add( 12, 34 );
ok $a == $b, "Add: '$b' ($a)";
__END__
__C__
int add(int x, int y)
{
return x + y;
}
Slide 66
Slide 66 text
Example: Testing your C code
$ prove -v add.t
add.t ..
1..1
ok 1 - Add: '46' (46)
ok
All tests successful.
Files=1, Tests=1, 0 wallclock secs
( 0.05 usr 0.00 sys + 0.09 cusr 0.00 csys = ...
Result: PASS
Slide 67
Slide 67 text
Example: Testing your C code
You can call into a library.
Write a whole program.
Inline builds the interface.
Slide 68
Slide 68 text
Test Anything Protcol
Inline Language Support Modules add languages.
Inline supports:
C, C++, Java, Python, Ruby, Tcl,
Assembler, Basic, Guile, Befunge, Octave,
Awk, BC, TT (Template Toolkit), WebChat,
and (of course) Perl.
Slide 69
Slide 69 text
Really: Any Thing
Say you had a quad-rotor UAV...
You'd write a Perl API for it, wouldn't you?
UAV::Pilot::ARDrone, for example.
But then you'd have to test it...
Slide 70
Slide 70 text
First: Define the test actions
my @actionz =
(
[
takeoff => 10
],
[
wave => 8000
],
[
flip_left => 5000
],
[
land => 5000,
send => $expect
],
);
plan tests => 2 + @actionz;
Basic set of actions:
Takeoff, wobble, flip,
and land.
Final “send” validates
end-of-channel.
Slide 71
Slide 71 text
Control interfaces
BAIL_OUT
avoids running
without controls.
my $driver
= UAV::Pilot::ARDrone::Driver->new
(
{
host => $HOST,
}
)
or BAIL_OUT "Failed construct";
eval
{
$driver->connect;
pass "Connected to '$HOST'";
}
or BAIL_OUT "Failed connect: $@";
Slide 72
Slide 72 text
Error handler for actions
Attempt the last
action (land) on
errors.
sub oops
{
eval { execute $actionz[ -1 ] }
or
BAIL_OUT "Literal crash expected";
}
Slide 73
Slide 73 text
sub execute
{
state $a = $event;
my ( $ctrl_op, $time, $cv_op, $cv_val ) = @$_;
$a = $a->add_timer
(
{
duration => $time,
duration_units => $event->UNITS_MILLISECOND,
cb =>
sub
{
$control->$ctrl_op or die "Failed: $ctrl_op";
$cv->$cv_op( $cv_val ) if $cv_op;
pass "Execute: '$ctrl_op' ($cv_op)";
}
}
)
Slide 74
Slide 74 text
Execute the actions
“note” here
describes what
to expect.
for( @actionz )
{
note "Prepare:\n", explain $_;
eval { execute $_ }
or
oops;
}
$event->init_event_loop;
“pass” shows what happens
# Prepare:
# [
# 'land',
# 5000,
# 'send',
# '123'
# ]
ok 2 - Execute: 'takeoff' ()
ok 3 - Execute: 'wave' ()
Slide 79
Slide 79 text
“pass” shows what happens
# Prepare:
# [
# 'land',
# 5000,
# 'send',
# '123'
# ]
ok 2 - Execute: 'takeoff' ()
ok 3 - Execute: 'wave' ()
ok 4 - Execute: 'flip_left' ()
Slide 80
Slide 80 text
“pass” shows what happens
# Prepare:
# [
# 'land',
# 5000,
# 'send',
# '123'
# ]
ok 2 - Execute: 'takeoff' ()
ok 3 - Execute: 'wave' ()
ok 4 - Execute: 'flip_left' ()
ok 5 - Execute: 'land' (send)
Slide 81
Slide 81 text
“pass” shows what happens
# Prepare:
# [
# 'land',
# 5000,
# 'send',
# '123'
# ]
ok 2 - Execute: 'takeoff' ()
ok 3 - Execute: 'wave' ()
ok 4 - Execute: 'flip_left' ()
ok 5 - Execute: 'land' (send)
ok 6 - Recv completed '123' (123)
Slide 82
Slide 82 text
Cultured Perls
The Developer's Notebook is a great resource.
POD for Test::More == wheels you don't re-invent.
POD for prove == make life faster, easier.
cpantesters.org: test results for modules by platform.
Stay cultured, be testy: use perl.