Getting Testy With Perl

Perl's testing is built into the core language, and the culture of testing has pushed users to create simple, effective tools for authoring and running tests. Perl's tools for introspection also makes it easy to test specific outcomes by overriding library or even core functions to return specific outcomes. For true flexibility, add the Inline module and test other languages or interfaces between them.

This talk describes the basics of Perl's testing modules for common data types and structure, overloading for complete testing, using coverage modules to validate the completeness and accuracy of the testing itself, Inline for testing other languages.

The example of executing this is on YouTube at:

https://www.youtube.com/watch?v=IXNwofjIZUM&list=PLA9_Hq3zhoFy5jmj6_Fd2YMuOjsovaIh-&index=54

Transcript

1. Getting Testy With Perl
   Steven Lembark
   Workhorse Computing
   lembark@wrkhors.com
   

   View Slide

2. Cultured Perl
   Perl's “test culture” is part of the language.
   Test::* modules easily added to Perl code.
   Smoke testing provided as part of CPAN.
   Reporter accumulates results from installers.
   Language structure, modules work together.
   

   View Slide

3. Where there's smoke, there's Perl.
   CPAN::Reporter send back success/fail.
   CPAN testers: variety of platforms, daily reports.
   cpan-testers & cpan-testers-discuss mailing lists.
   http://cpantesters.org/
   Community effort at testing is unique to Perl.
   

   View Slide

4. Excellent Reference
   Perl Testing: A developer's Notebook,
   O'Reilly Press
   Wonderful example of a good howto-book.
   Includes modules, ways to use them with variations.
   Good cover-to-cover read.
   

   View Slide

5. Perl is easy to test
   Test::* modules do most of the work.
   Use the wheels, don't re-invent:
   Test::Simple, Test::More, Test::Deep,
   Test::Builder, Test::Coverage, Object::Exercise
   Perl adds introspection.
   

   View Slide

6. One-stop shopping
   Perl as a great glue language.
   Use perl to test other programs.
   Have other programs output TAP.
   Combined with “inline” you can test almost anything!
   

   View Slide

7. Test your tests!
   Devel::Coverage have you tested all of the code?
   Test::Builder roll your own tests.
   Build you own re-usable test components:
   Test::Harness::Straps
   Test::Builder::Tester
   

   View Slide

8. “Test” code can do more
   Pre-load fixed data into a database.
   Validate the working environment.
   Execute daily cleanups in cron.
   

   View Slide

9. Testing simply
   Count hardwired:
   use Test::Simple tests => 6;
   Difference in count will be an error.
   

   View Slide

10. Its all “ok”
    use Test::Simple tests => 2;
    # ok( boolean, message );
    ok 1 == 1, “Count: '1' (1)”;
    ok 2 == 1, “Count: '2' (1)”;
    Output like:
    ok 1 - Count: '1' (1)
    not ok 2 - Count: '2' (1)
    

    View Slide

11. Its all “ok”
    use Test::Simple tests => 2;
    # ok( boolean, message );
    ok 1, 'It passed!';
    Output like:
    1..2
    ok 1 - It passed!
    # Looks like you planned 2 tests but ran 1.
    

    View Slide

12. Seen these before?
    “not ok 10 - Count”
    “not ok 11 - Format”
    “not ok 5 - File or directory not found.”
    OK, now what?
    

    View Slide

13. Designing tests:
    Test what matters.
    Report something useful.
    

    View Slide

14. Isolate the failure
    What to fix:
    ok -e $path , “Existing: '$path'”;
    ok -d _ , “Directory: '$path'”;
    ok -r _ , “Readable: '$path'”;
    ok -w _ , “Writeable '$path'”;
    

    View Slide

15. Q: Why all of the quotes?
    ok -e $path, “Existing dir: $path”
    

    View Slide

16. Q: Why all of the quotes?
    ok -e $path, “Existing dir: $path”
    not ok 1 - Existing dir: /var/tmp
    not ok 1 Existing dir: /var/tmp
    

    View Slide

17. Q: Why all of the quotes?
    ok -e $path, “Existing dir: '$path'”
    not ok 1 - Existing dir: '/var/tmp '
    not ok 1 Existing dir: '/var/tmp
    '
    

    View Slide

18. Good general format
    Include what you found, what you expect.
    You'll need them both with “not ok”:
    ok $got eq $want, 'Name: '$got' ($want)”;
    not ok 99 – Name: '' (Jow Bloe)
    not ok 99 – Name: '1 Main St.' (Jow Bloe)
    not ok 99 – Name: 'Jow Bloe' ()
    not ok 99 – Name: 'Jow Bloe' (User::Name=0x1A...
    

    View Slide

19. Test::Simple may be enough
    Generate a plan.
    “ok” generates TAP output.
    Test count is fixed.
    This works for many day-to-day tests.
    

    View Slide

20. But Wait! There's More!
    Test::More
    plan, done_testing test counts
    is, isnt, like, unlike stringy “ok”
    pass, fail skip the boolean
    use_ok, require_ok module test
    note, diag, explain messages
    

    View Slide

21. Flexable count
    Set count with “plan”.
    Compute from test input:
    plan tests => 42;
    plan tests => 3 * keys $config;
    plan tests => 2 * values $query_output;
    Test count varys?
    Skip plan.
    

    View Slide

22. When you're done testing
    Just say so:
    done_testing;
    Generates final report.
    Plan count is checked here.
    No plan, no count check.
    

    View Slide

23. Notes show up with “prove -v”
    note “read_config from '$config_path'”;
    my $path = read_config 'tmp_dir';
    ok -e $path, “Exists: '$path' (tmp_dir)”;
    $ prove -v;
    # read_config from './etc/pass1.conf'
    not ok 1 – Exists: '/var/tmp/foobar ' (tmp_dir)
    

    View Slide

24. Diagnostics show why tests failed
    Show up without “-v”.
    ok … or diag “Oopsie...”;
    ok grep /blah/, @test_input, 'Found blah'
    or diag “Missing 'blah':”, @test_input;
    ok $dbh, “Connected”
    or diag “Failed connect: '$dsn'”;
    

    View Slide

25. Explain shows exactly what failed
    “explain” shows nested structure.
    Use with “note” to show setup details.
    With “diag” shows extra info on failure.
    my $dbh = DBI->connect( @argz );
    ok $dbh, 'Database connected'
    or diag 'Connect args:', explain \@argz;
    

    View Slide

26. Stringy “ok”
    like, not_like use regexen.
    saves “=~” syntax in the tests:
    like $stuff, qr/milk/, “Got: 'milk'?”
    or diag “Have: '$stuff' instead”;
    not ok 1 - Got: 'milk'?
    # Have 'very new cheese' instead
    

    View Slide

27. Variable number of tests
    If-logic generates variable number of tests.
    Skip “plan”, use “done_testing”.
    

    View Slide

28. Taking it pass/fail
    if( $obj = eval { $class->constructify } )
    {
    # validate object contents
    }
    else
    {
    # nothing more to test
    fail “Failed constructify: $@”;
    }
    

    View Slide

29. Test expected errors
    eval
    {
    $obj->new( $junk );
    fail "No exception: foo( $junk )?";
    1
    }
    or do
    {
    # validate error handling
    ...
    }
    

    View Slide

30. Controlling test cycle
    Abort if everything will fail.
    Avoid expensive, specialized, un-necesary tests.
    Saves extraneous code in all of the tests.
    

    View Slide

31. BAIL_OUT: Knowing when to give up
    Aborts all testing.
    Unsafe or guaranteed failure.
    Useful in early test for environment, arg checks.
    

    View Slide

32. BAIL_OUT
    BAIL_OUT 'Do not run tests as su!' unless $>;
    BAIL_OUT 'Missing $DB' unless $ENV{DB};
    -e $test_dir or mkdir $dir, 0777
    or BAIL_OUT “Failed mkdir: '$dir', $!”;
    -d $test_dir or BAIL_OUT “Missing: '$dir'”;
    -r _ or BAIL_OUT “Un-readable: '$dir'”;
    -w _ or BAIL_OUT “Un-writeable: '$dir'”;
    

    View Slide

33. “SKIP” blocks
    Skip a block with test count and message.
    Adjust the plan test count.
    Expensive or specialized tests.
    

    View Slide

34. Skip external tests
    SKIP:
    {
    skip “No database (TEST_DB)”, 8 if ! $ENV{ TEST_DB };
    # or … no network available...
    # or … no server handle...
    ...
    }
    

    View Slide

35. Skip expensive tests
    SKIP:
    {
    skip “Used for internal development only”, 12
    unless $ENV{ EXPENSIVE_TESTS };
    # test plan count reduced by 12
    ...
    }
    

    View Slide

36. Skip dangerous tests
    SKIP:
    {
    skip “Unsafe as superuser”, 22 unless $>;
    # at this point the tests are not running su.
    ...
    }
    

    View Slide

37. You'll usually use Test::More
    note & diag nearly always worth using.
    plan & done_testing makes life easier.
    Still have “ok” for the majority of work.
    

    View Slide

38. Testing Structures
    “like” syntax with nested structure:
    use Test::Deep;
    cmp_deeply $found, $expect, $message;
    Great for testing parser or grammar outcome.
    

    View Slide

39. Devel::Cover: what didn't you check?
    All of the else blocks?
    All of the “||=” assignments?
    All of the “... or die ...” branches?
    Devel::Cover bookkeeps running code.
    Reports what you didn't test.
    Tells you what test to write next.
    

    View Slide

40. Similar to NYTProf:
    Run your program & summarize the results:
    $ cover -test;
    or
    $ perl -MDevel::Cover ./t/foo.t;
    $ cover;
    Running “cover” generates the report.
    

    View Slide

41. Introspection simplifies testing
    Want to test database connect failure.
    Can't assume SQL::Lite.
    Flat file databases are messy.
    Guarantee that something fails?
    You could write an operation that should fail.
    Then again, it might not...
    

    View Slide

42. Making sure you fail
    Override class methods.
    Override Perl functions.
    Result:
    Mock objects,
    Mock methods,
    Mock perl.
    

    View Slide

43. Mock Objects
    Test your wrapper handling failure?
    Override DBI::connect with sub { die … }.
    No guess: You know it's going to fail.
    

    View Slide

44. Mock Modules
    Your test:
    *DBI::connect = sub { die '…' };
    my $status = eval { $obj->make_connection };
    my $err = $@;
    # test $status, $err, $obj...
    

    View Slide

45. Force an exception
    use Symbol qw( qualify_to_ref );
    # e.g., force_exception 'Invalid username', 'connect', 'DBI';
    # ( name, packcage ) or ( “package::name” )
    sub force_exception
    {
    chomp( my $messsage = shift );
    my $ref = qualify_to_ref @_;
    undef &{ *$ref };
    *{ $ref } = sub { die “$message\n” };
    return
    }
    

    View Slide

46. Testing multiple failures
    for( @testz )
    {
    my( $msg, $expect ) = @$_;
    force_exception $msg, 'DBI::connect';
    my $status = eval { $wrapper->connect };
    my $err = $@;
    # your tests here
    }
    

    View Slide

47. Avoid caller cleanup
    Override with “local”
    sub force_exception
    {
    my ( $proto, $method, $pkg, $name, $msg ) = splice @_, 0, 5;
    my $ref = qualify_to_ref $name, $pkg;
    local *{ $ref } = sub { die “$msg” };
    # exit from block cleans up local override.
    # caller checks return, $@, $proto.
    eval { $proto->$method( @_ ) }
    }
    

    View Slide

48. Mock Perl
    Override perl: *Core::Global::
    Is “exit” is called?
    my $exits = 0;
    *Core::Global::exit = sub { $exits = 1 };
    eval { frobnicate $arg };
    ok $exits, “Frobnicate exits ($exits)”;
    

    View Slide

49. Devel::Cover & Mocking
    The biggest reason for mock anything:
    Force an outcome to test a branch.
    Iterate:
    Test with Devel::Cover.
    See what still needs testing.
    Mock object/method/function forces the branch.
    

    View Slide

50. Automated testing
    Lazyness is a virtue.
    Avoid cut+paste.
    Let Perl do the work!
    

    View Slide

51. Example: Sanity check modules
    Use symlinks.
    Validate modules compile.
    Check package argument.
    require_ok $package; # ok if require-able.
    use_ok $package; # ok if use-able.
    

    View Slide

52. Make the links
    Path below ./lib.
    Replace slashes with dashes.
    Add a leading “01”.
    Symlink them all to a generic baseline test.
    Symlinks look like:
    01-Wcurve-Util.t → generic-01-t
    

    View Slide

53. use FindBin::libs;
    use Test::More;
    use File::Basename qw( basename );
    my $madness = basename $0, '.t'; # 01-WCurve-Util
    $madness =~ s{^ 01-}{}x; # WCurve-Util
    $madness =~ s{ \W+ }{::}gx; # WCUrve::Util
    if( use_ok $madness )
    {
    # check for correct package argument.
    ok $madness->can( 'VERSION' ), “$maddness has a method”;
    ok $a = $madness->VERSION, “$madness is '$a'”;
    }
    done_testing;
    ./t/generic-01-t
    

    View Slide

54. ./t/make-generic-links
    #!/bin/bash
    cd $(dirname $0);
    rm 01-*.t;
    find ../lib/ -name '*.pm' |
    perl -n \
    -e 'chomp;' \
    -e 's{^ .+ /lib/}{}x' \
    -e 's{.pm $}{.t}x' \
    -e 'tr{/}{-} \
    -e 'symlink “01-generic-t” => “01-$_”';
    exit 0;
    

    View Slide

55. Similar checks for config files
    Add “00-” tests to reads config files.
    Combine “ok” with Config::Any.
    Now make-generic-links tests config & code.
    “prove” runs them all in one pass.
    

    View Slide

56. The Validation Two-Step
    Use with git for sanity check:
    git pull &&
    ./t/make-generic-links &&
    prove --jobs=2 –-state=save &&
    git tag -a “prove/$date” &&
    git push &&
    git push –-tag ;
    

    View Slide

57. Exercise for healthy objects
    Data-driven testing for a single object.
    Data includes method, data, expected return.
    Module iterates the tests, reports results.
    When tests or expected values are generated.
    

    View Slide

58. use Object::Exercise;
    my @testz =
    (
    [
    [ qw( name ) ], # method + args
    [ q{Jow Bloe} ], # expected return
    ],
    [
    [ qw( street ) ],
    [ q{1 Nuwere St} ],
    ],
    [
    [ qw( street number ) ], # $obj->address( 'number' )
    [ qw( 1 ) ],
    'Check street number' # hardwired message
    ],
    );
    Person->new( 'Jow Blow' => '1 Nuwere St' )->$exercise( @testz );
    

    View Slide

59. Load fixed data
    Flat file -> arrayrefs
    “insert” as method, [1] as return.
    Load the data with:
    $sth->$exercise( @data )
    Get “ok” message for each of the data record.
    

    View Slide

60. Roll your own: Test::Builder
    Mirrors Test::More with a singleton object.
    my $test = Test::Builder->new;
    $test->ok( $boolean, $success_message )
    or $test->diag( $more_info );
    Spread single test across multiple modules.
    

    View Slide

61. Testing Testing
    Test::Build::Tester wraps your tests.
    Forces test to return not-ok in order to test it.
    Ignores the output of the test being tested.
    Validate the test plan.
    

    View Slide

62. Getting the most out of prove
    Save state: re-run failed tests:
    $prove –state=save;
    # fix something...
    $prove –state=save,failed;
    Parallel execution with “--jobs”.
    “perldoc” is your friend!
    

    View Slide

63. Great, but my shop uses
    Test multiple languages with Inline.
    Include C, C++, Python...
    Multiple Perl versions.
    Mix languages to test interfaces.
    

    View Slide

64. One-stop teting for your API lib's
    Need to test multi-language support?
    Use Perl to move data between them.
    Unit-test C talking to Java.
    

    View Slide

65. Example: Testing your C code
    use Inline C;
    use Test::Simple tests => 1;
    my $a = 12 + 34;
    my $b = add( 12, 34 );
    ok $a == $b, "Add: '$b' ($a)";
    __END__
    __C__
    int add(int x, int y)
    {
    return x + y;
    }
    

    View Slide

66. Example: Testing your C code
    $ prove -v add.t
    add.t ..
    1..1
    ok 1 - Add: '46' (46)
    ok
    All tests successful.
    Files=1, Tests=1, 0 wallclock secs
    ( 0.05 usr 0.00 sys + 0.09 cusr 0.00 csys = ...
    Result: PASS
    

    View Slide

67. Example: Testing your C code
    You can call into a library.
    Write a whole program.
    Inline builds the interface.
    

    View Slide

68. Test Anything Protcol
    Inline Language Support Modules add languages.
    Inline supports:
    C, C++, Java, Python, Ruby, Tcl,
    Assembler, Basic, Guile, Befunge, Octave,
    Awk, BC, TT (Template Toolkit), WebChat,
    and (of course) Perl.
    

    View Slide

69. Really: Any Thing
    Say you had a quad-rotor UAV...
    You'd write a Perl API for it, wouldn't you?
    UAV::Pilot::ARDrone, for example.
    But then you'd have to test it...
    

    View Slide

70. First: Define the test actions
    my @actionz =
    (
    [
    takeoff => 10
    ],
    [
    wave => 8000
    ],
    [
    flip_left => 5000
    ],
    [
    land => 5000,
    send => $expect
    ],
    );
    plan tests => 2 + @actionz;
    Basic set of actions:
    Takeoff, wobble, flip,
    and land.
    Final “send” validates
    end-of-channel.
    

    View Slide

71. Control interfaces
    BAIL_OUT
    avoids running
    without controls.
    my $driver
    = UAV::Pilot::ARDrone::Driver->new
    (
    {
    host => $HOST,
    }
    )
    or BAIL_OUT "Failed construct";
    eval
    {
    $driver->connect;
    pass "Connected to '$HOST'";
    }
    or BAIL_OUT "Failed connect: $@";
    

    View Slide

72. Error handler for actions
    Attempt the last
    action (land) on
    errors.
    sub oops
    {
    eval { execute $actionz[ -1 ] }
    or
    BAIL_OUT "Literal crash expected";
    }
    

    View Slide

73. sub execute
    {
    state $a = $event;
    my ( $ctrl_op, $time, $cv_op, $cv_val ) = @$_;
    $a = $a->add_timer
    (
    {
    duration => $time,
    duration_units => $event->UNITS_MILLISECOND,
    cb =>
    sub
    {
    $control->$ctrl_op or die "Failed: $ctrl_op";
    $cv->$cv_op( $cv_val ) if $cv_op;
    pass "Execute: '$ctrl_op' ($cv_op)";
    }
    }
    )
    

    View Slide

74. Execute the actions
    “note” here
    describes what
    to expect.
    for( @actionz )
    {
    note "Prepare:\n", explain $_;
    eval { execute $_ }
    or
    oops;
    }
    $event->init_event_loop;
    

    View Slide

75. Check the end-of-channel
    my $found = $cv->recv;
    if( $found eq $expect )
    {
    pass "Recv completed '$found' ($expect)";
    }
    else
    {
    fail "Recv incomplete ($found), send emergency land";
    oops;
    }
    done_testing;
    

    View Slide

76. Execute the tests
    $ perl t/01-test-flight.t
    1..6
    # Connect to: '192.168.1.1'
    (UAV::Pilot::ARDrone::Driver)
    ok 1 - Connected to '192.168.1.1'
    # U::P::EasyEvent has a socket
    # Prepare:
    # [
    # 'takeoff',
    # 10
    # ]
    # Prepare:
    # [
    # 'wave'
    

    View Slide

77. “pass” shows what happens
    # Prepare:
    # [
    # 'land',
    # 5000,
    # 'send',
    # '123'
    # ]
    ok 2 - Execute: 'takeoff' ()
    

    View Slide

78. “pass” shows what happens
    # Prepare:
    # [
    # 'land',
    # 5000,
    # 'send',
    # '123'
    # ]
    ok 2 - Execute: 'takeoff' ()
    ok 3 - Execute: 'wave' ()
    

    View Slide

79. “pass” shows what happens
    # Prepare:
    # [
    # 'land',
    # 5000,
    # 'send',
    # '123'
    # ]
    ok 2 - Execute: 'takeoff' ()
    ok 3 - Execute: 'wave' ()
    ok 4 - Execute: 'flip_left' ()
    

    View Slide

80. “pass” shows what happens
    # Prepare:
    # [
    # 'land',
    # 5000,
    # 'send',
    # '123'
    # ]
    ok 2 - Execute: 'takeoff' ()
    ok 3 - Execute: 'wave' ()
    ok 4 - Execute: 'flip_left' ()
    ok 5 - Execute: 'land' (send)
    

    View Slide

81. “pass” shows what happens
    # Prepare:
    # [
    # 'land',
    # 5000,
    # 'send',
    # '123'
    # ]
    ok 2 - Execute: 'takeoff' ()
    ok 3 - Execute: 'wave' ()
    ok 4 - Execute: 'flip_left' ()
    ok 5 - Execute: 'land' (send)
    ok 6 - Recv completed '123' (123)
    

    View Slide

82. Cultured Perls
    The Developer's Notebook is a great resource.
    POD for Test::More == wheels you don't re-invent.
    POD for prove == make life faster, easier.
    cpantesters.org: test results for modules by platform.
    Stay cultured, be testy: use perl.
    

    View Slide