Getting Testy With Perl

Perl's testing is built into the core language, and the culture of testing has pushed users to create simple, effective tools for authoring and running tests. Perl's tools for introspection also makes it easy to test specific outcomes by overriding library or even core functions to return specific outcomes. For true flexibility, add the Inline module and test other languages or interfaces between them.

This talk describes the basics of Perl's testing modules for common data types and structure, overloading for complete testing, using coverage modules to validate the completeness and accuracy of the testing itself, Inline for testing other languages.

The example of executing this is on YouTube at:

https://www.youtube.com/watch?v=IXNwofjIZUM&list=PLA9_Hq3zhoFy5jmj6_Fd2YMuOjsovaIh-&index=54

Transcript

1. Getting Testy With Perl Steven Lembark Workhorse Computing lembark@wrkhors.com
2. Cultured Perl Perl's “test culture” is part of the language. Test::* modules easily added to Perl code. Smoke testing provided as part of CPAN. Reporter accumulates results from installers. Language structure, modules work together.
3. Where there's smoke, there's Perl. CPAN::Reporter send back success/fail. CPAN testers: variety of platforms, daily reports. cpan-testers & cpan-testers-discuss mailing lists. http://cpantesters.org/ Community effort at testing is unique to Perl.
4. Excellent Reference Perl Testing: A developer's Notebook, O'Reilly Press Wonderful example of a good howto-book. Includes modules, ways to use them with variations. Good cover-to-cover read.
5. Perl is easy to test Test::* modules do most of the work. Use the wheels, don't re-invent: Test::Simple, Test::More, Test::Deep, Test::Builder, Test::Coverage, Object::Exercise Perl adds introspection.
6. One-stop shopping Perl as a great glue language. Use perl to test other programs. Have other programs output TAP. Combined with “inline” you can test almost anything!
7. Test your tests! Devel::Coverage have you tested all of the code? Test::Builder roll your own tests. Build you own re-usable test components: Test::Harness::Straps Test::Builder::Tester
8. “Test” code can do more Pre-load fixed data into a database. Validate the working environment. Execute daily cleanups in cron.
9. Testing simply Count hardwired: use Test::Simple tests => 6; Difference in count will be an error.
10. Its all “ok” use Test::Simple tests => 2; # ok( boolean, message ); ok 1 == 1, “Count: '1' (1)”; ok 2 == 1, “Count: '2' (1)”; Output like: ok 1 - Count: '1' (1) not ok 2 - Count: '2' (1)
11. Its all “ok” use Test::Simple tests => 2; # ok( boolean, message ); ok 1, 'It passed!'; Output like: 1..2 ok 1 - It passed! # Looks like you planned 2 tests but ran 1.
12. Seen these before? “not ok 10 - Count” “not ok 11 - Format” “not ok 5 - File or directory not found.” OK, now what?
13. Designing tests: Test what matters. Report something useful.
14. Isolate the failure What to fix: ok -e $path , “Existing: '$path'”; ok -d _ , “Directory: '$path'”; ok -r _ , “Readable: '$path'”; ok -w _ , “Writeable '$path'”;
15. Q: Why all of the quotes? ok -e $path, “Existing dir: $path”
16. Q: Why all of the quotes? ok -e $path, “Existing dir: $path” not ok 1 - Existing dir: /var/tmp not ok 1 Existing dir: /var/tmp
17. Q: Why all of the quotes? ok -e $path, “Existing dir: '$path'” not ok 1 - Existing dir: '/var/tmp ' not ok 1 Existing dir: '/var/tmp '
18. Good general format Include what you found, what you expect. You'll need them both with “not ok”: ok $got eq $want, 'Name: '$got' ($want)”; not ok 99 – Name: '' (Jow Bloe) not ok 99 – Name: '1 Main St.' (Jow Bloe) not ok 99 – Name: 'Jow Bloe' () not ok 99 – Name: 'Jow Bloe' (User::Name=0x1A...
19. Test::Simple may be enough Generate a plan. “ok” generates TAP output. Test count is fixed. This works for many day-to-day tests.
20. But Wait! There's More! Test::More plan, done_testing test counts is, isnt, like, unlike stringy “ok” pass, fail skip the boolean use_ok, require_ok module test note, diag, explain messages
21. Flexable count Set count with “plan”. Compute from test input: plan tests => 42; plan tests => 3 * keys $config; plan tests => 2 * values $query_output; Test count varys? Skip plan.
22. When you're done testing Just say so: done_testing; Generates final report. Plan count is checked here. No plan, no count check.
23. Notes show up with “prove -v” note “read_config from '$config_path'”; my $path = read_config 'tmp_dir'; ok -e $path, “Exists: '$path' (tmp_dir)”; $ prove -v; # read_config from './etc/pass1.conf' not ok 1 – Exists: '/var/tmp/foobar ' (tmp_dir)
24. Diagnostics show why tests failed Show up without “-v”. ok … or diag “Oopsie...”; ok grep /blah/, @test_input, 'Found blah' or diag “Missing 'blah':”, @test_input; ok $dbh, “Connected” or diag “Failed connect: '$dsn'”;
25. Explain shows exactly what failed “explain” shows nested structure. Use with “note” to show setup details. With “diag” shows extra info on failure. my $dbh = DBI->connect( @argz ); ok $dbh, 'Database connected' or diag 'Connect args:', explain \@argz;
26. Stringy “ok” like, not_like use regexen. saves “=~” syntax in the tests: like $stuff, qr/milk/, “Got: 'milk'?” or diag “Have: '$stuff' instead”; not ok 1 - Got: 'milk'? # Have 'very new cheese' instead
27. Variable number of tests If-logic generates variable number of tests. Skip “plan”, use “done_testing”.
28. Taking it pass/fail if( $obj = eval { $class->constructify } ) { # validate object contents } else { # nothing more to test fail “Failed constructify: $@”; }
29. Test expected errors eval { $obj->new( $junk ); fail "No exception: foo( $junk )?"; 1 } or do { # validate error handling ... }
30. Controlling test cycle Abort if everything will fail. Avoid expensive, specialized, un-necesary tests. Saves extraneous code in all of the tests.
31. BAIL_OUT: Knowing when to give up Aborts all testing. Unsafe or guaranteed failure. Useful in early test for environment, arg checks.
32. BAIL_OUT BAIL_OUT 'Do not run tests as su!' unless $>; BAIL_OUT 'Missing $DB' unless $ENV{DB}; -e $test_dir or mkdir $dir, 0777 or BAIL_OUT “Failed mkdir: '$dir', $!”; -d $test_dir or BAIL_OUT “Missing: '$dir'”; -r _ or BAIL_OUT “Un-readable: '$dir'”; -w _ or BAIL_OUT “Un-writeable: '$dir'”;
33. “SKIP” blocks Skip a block with test count and message. Adjust the plan test count. Expensive or specialized tests.
34. Skip external tests SKIP: { skip “No database (TEST_DB)”, 8 if ! $ENV{ TEST_DB }; # or … no network available... # or … no server handle... ... }
35. Skip expensive tests SKIP: { skip “Used for internal development only”, 12 unless $ENV{ EXPENSIVE_TESTS }; # test plan count reduced by 12 ... }
36. Skip dangerous tests SKIP: { skip “Unsafe as superuser”, 22 unless $>; # at this point the tests are not running su. ... }
37. You'll usually use Test::More note & diag nearly always worth using. plan & done_testing makes life easier. Still have “ok” for the majority of work.
38. Testing Structures “like” syntax with nested structure: use Test::Deep; cmp_deeply $found, $expect, $message; Great for testing parser or grammar outcome.
39. Devel::Cover: what didn't you check? All of the else blocks? All of the “||=” assignments? All of the “... or die ...” branches? Devel::Cover bookkeeps running code. Reports what you didn't test. Tells you what test to write next.
40. Similar to NYTProf: Run your program & summarize the results: $ cover -test; or $ perl -MDevel::Cover ./t/foo.t; $ cover; Running “cover” generates the report.
41. Introspection simplifies testing Want to test database connect failure. Can't assume SQL::Lite. Flat file databases are messy. Guarantee that something fails? You could write an operation that should fail. Then again, it might not...
42. Making sure you fail Override class methods. Override Perl functions. Result: Mock objects, Mock methods, Mock perl.
43. Mock Objects Test your wrapper handling failure? Override DBI::connect with sub { die … }. No guess: You know it's going to fail.
44. Mock Modules Your test: *DBI::connect = sub { die '…' }; my $status = eval { $obj->make_connection }; my $err = $@; # test $status, $err, $obj...
45. Force an exception use Symbol qw( qualify_to_ref ); # e.g., force_exception 'Invalid username', 'connect', 'DBI'; # ( name, packcage ) or ( “package::name” ) sub force_exception { chomp( my $messsage = shift ); my $ref = qualify_to_ref @_; undef &{ *$ref }; *{ $ref } = sub { die “$message\n” }; return }
46. Testing multiple failures for( @testz ) { my( $msg, $expect ) = @$_; force_exception $msg, 'DBI::connect'; my $status = eval { $wrapper->connect }; my $err = $@; # your tests here }
47. Avoid caller cleanup Override with “local” sub force_exception { my ( $proto, $method, $pkg, $name, $msg ) = splice @_, 0, 5; my $ref = qualify_to_ref $name, $pkg; local *{ $ref } = sub { die “$msg” }; # exit from block cleans up local override. # caller checks return, $@, $proto. eval { $proto->$method( @_ ) } }
48. Mock Perl Override perl: *Core::Global::<name> Is “exit” is called? my $exits = 0; *Core::Global::exit = sub { $exits = 1 }; eval { frobnicate $arg }; ok $exits, “Frobnicate exits ($exits)”;
49. Devel::Cover & Mocking The biggest reason for mock anything: Force an outcome to test a branch. Iterate: Test with Devel::Cover. See what still needs testing. Mock object/method/function forces the branch.
50. Automated testing Lazyness is a virtue. Avoid cut+paste. Let Perl do the work!
51. Example: Sanity check modules Use symlinks. Validate modules compile. Check package argument. require_ok $package; # ok if require-able. use_ok $package; # ok if use-able.
52. Make the links Path below ./lib. Replace slashes with dashes. Add a leading “01”. Symlink them all to a generic baseline test. Symlinks look like: 01-Wcurve-Util.t → generic-01-t
53. use FindBin::libs; use Test::More; use File::Basename qw( basename ); my $madness = basename $0, '.t'; # 01-WCurve-Util $madness =~ s{^ 01-}{}x; # WCurve-Util $madness =~ s{ \W+ }{::}gx; # WCUrve::Util if( use_ok $madness ) { # check for correct package argument. ok $madness->can( 'VERSION' ), “$maddness has a method”; ok $a = $madness->VERSION, “$madness is '$a'”; } done_testing; ./t/generic-01-t
54. ./t/make-generic-links #!/bin/bash cd $(dirname $0); rm 01-*.t; find ../lib/ -name '*.pm' | perl -n \ -e 'chomp;' \ -e 's{^ .+ /lib/}{}x' \ -e 's{.pm $}{.t}x' \ -e 'tr{/}{-} \ -e 'symlink “01-generic-t” => “01-$_”'; exit 0;
55. Similar checks for config files Add “00-” tests to reads config files. Combine “ok” with Config::Any. Now make-generic-links tests config & code. “prove” runs them all in one pass.
56. The Validation Two-Step Use with git for sanity check: git pull && ./t/make-generic-links && prove --jobs=2 –-state=save && git tag -a “prove/$date” && git push && git push –-tag ;
57. Exercise for healthy objects Data-driven testing for a single object. Data includes method, data, expected return. Module iterates the tests, reports results. When tests or expected values are generated.
58. use Object::Exercise; my @testz = ( [ [ qw( name ) ], # method + args [ q{Jow Bloe} ], # expected return ], [ [ qw( street ) ], [ q{1 Nuwere St} ], ], [ [ qw( street number ) ], # $obj->address( 'number' ) [ qw( 1 ) ], 'Check street number' # hardwired message ], ); Person->new( 'Jow Blow' => '1 Nuwere St' )->$exercise( @testz );
59. Load fixed data Flat file -> arrayrefs “insert” as method, [1] as return. Load the data with: $sth->$exercise( @data ) Get “ok” message for each of the data record.
60. Roll your own: Test::Builder Mirrors Test::More with a singleton object. my $test = Test::Builder->new; $test->ok( $boolean, $success_message ) or $test->diag( $more_info ); Spread single test across multiple modules.
61. Testing Testing Test::Build::Tester wraps your tests. Forces test to return not-ok in order to test it. Ignores the output of the test being tested. Validate the test plan.
62. Getting the most out of prove Save state: re-run failed tests: $prove –state=save; # fix something... $prove –state=save,failed; Parallel execution with “--jobs”. “perldoc” is your friend!
63. Great, but my shop uses <pick one> Test multiple languages with Inline. Include C, C++, Python... Multiple Perl versions. Mix languages to test interfaces.
64. One-stop teting for your API lib's Need to test multi-language support? Use Perl to move data between them. Unit-test C talking to Java.
65. Example: Testing your C code use Inline C; use Test::Simple tests => 1; my $a = 12 + 34; my $b = add( 12, 34 ); ok $a == $b, "Add: '$b' ($a)"; __END__ __C__ int add(int x, int y) { return x + y; }
66. Example: Testing your C code $ prove -v add.t add.t .. 1..1 ok 1 - Add: '46' (46) ok All tests successful. Files=1, Tests=1, 0 wallclock secs ( 0.05 usr 0.00 sys + 0.09 cusr 0.00 csys = ... Result: PASS
67. Example: Testing your C code You can call into a library. Write a whole program. Inline builds the interface.
68. Test Anything Protcol Inline Language Support Modules add languages. Inline supports: C, C++, Java, Python, Ruby, Tcl, Assembler, Basic, Guile, Befunge, Octave, Awk, BC, TT (Template Toolkit), WebChat, and (of course) Perl.
69. Really: Any Thing Say you had a quad-rotor UAV... You'd write a Perl API for it, wouldn't you? UAV::Pilot::ARDrone, for example. But then you'd have to test it...
70. First: Define the test actions my @actionz = ( [ takeoff => 10 ], [ wave => 8000 ], [ flip_left => 5000 ], [ land => 5000, send => $expect ], ); plan tests => 2 + @actionz; Basic set of actions: Takeoff, wobble, flip, and land. Final “send” validates end-of-channel.
71. Control interfaces BAIL_OUT avoids running without controls. my $driver = UAV::Pilot::ARDrone::Driver->new ( { host => $HOST, } ) or BAIL_OUT "Failed construct"; eval { $driver->connect; pass "Connected to '$HOST'"; } or BAIL_OUT "Failed connect: $@";
72. Error handler for actions Attempt the last action (land) on errors. sub oops { eval { execute $actionz[ -1 ] } or BAIL_OUT "Literal crash expected"; }
73. sub execute { state $a = $event; my ( $ctrl_op, $time, $cv_op, $cv_val ) = @$_; $a = $a->add_timer ( { duration => $time, duration_units => $event->UNITS_MILLISECOND, cb => sub { $control->$ctrl_op or die "Failed: $ctrl_op"; $cv->$cv_op( $cv_val ) if $cv_op; pass "Execute: '$ctrl_op' ($cv_op)"; } } )
74. Execute the actions “note” here describes what to expect. for( @actionz ) { note "Prepare:\n", explain $_; eval { execute $_ } or oops; } $event->init_event_loop;
75. Check the end-of-channel my $found = $cv->recv; if( $found eq $expect ) { pass "Recv completed '$found' ($expect)"; } else { fail "Recv incomplete ($found), send emergency land"; oops; } done_testing;
76. Execute the tests $ perl t/01-test-flight.t 1..6 # Connect to: '192.168.1.1' (UAV::Pilot::ARDrone::Driver) ok 1 - Connected to '192.168.1.1' # U::P::EasyEvent has a socket # Prepare: # [ # 'takeoff', # 10 # ] # Prepare: # [ # 'wave'
77. “pass” shows what happens # Prepare: # [ # 'land', # 5000, # 'send', # '123' # ] ok 2 - Execute: 'takeoff' ()
78. “pass” shows what happens # Prepare: # [ # 'land', # 5000, # 'send', # '123' # ] ok 2 - Execute: 'takeoff' () ok 3 - Execute: 'wave' ()
79. “pass” shows what happens # Prepare: # [ # 'land', # 5000, # 'send', # '123' # ] ok 2 - Execute: 'takeoff' () ok 3 - Execute: 'wave' () ok 4 - Execute: 'flip_left' ()
80. “pass” shows what happens # Prepare: # [ # 'land', # 5000, # 'send', # '123' # ] ok 2 - Execute: 'takeoff' () ok 3 - Execute: 'wave' () ok 4 - Execute: 'flip_left' () ok 5 - Execute: 'land' (send)
81. “pass” shows what happens # Prepare: # [ # 'land', # 5000, # 'send', # '123' # ] ok 2 - Execute: 'takeoff' () ok 3 - Execute: 'wave' () ok 4 - Execute: 'flip_left' () ok 5 - Execute: 'land' (send) ok 6 - Recv completed '123' (123)
82. Cultured Perls The Developer's Notebook is a great resource. POD for Test::More == wheels you don't re-invent. POD for prove == make life faster, easier. cpantesters.org: test results for modules by platform. Stay cultured, be testy: use perl.