Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Getting Testy With Perl

Getting Testy With Perl

Perl's testing is built into the core language, and the culture of testing has pushed users to create simple, effective tools for authoring and running tests. Perl's tools for introspection also makes it easy to test specific outcomes by overriding library or even core functions to return specific outcomes. For true flexibility, add the Inline module and test other languages or interfaces between them.

This talk describes the basics of Perl's testing modules for common data types and structure, overloading for complete testing, using coverage modules to validate the completeness and accuracy of the testing itself, Inline for testing other languages.

The example of executing this is on YouTube at:

https://www.youtube.com/watch?v=IXNwofjIZUM&list=PLA9_Hq3zhoFy5jmj6_Fd2YMuOjsovaIh-&index=54

Steven Lembark
PRO

July 09, 2022
Tweet

More Decks by Steven Lembark

Other Decks in Technology

Transcript

  1. Getting Testy With Perl
    Steven Lembark
    Workhorse Computing
    [email protected]

    View Slide

  2. Cultured Perl
    Perl's “test culture” is part of the language.
    Test::* modules easily added to Perl code.
    Smoke testing provided as part of CPAN.
    Reporter accumulates results from installers.
    Language structure, modules work together.

    View Slide

  3. Where there's smoke, there's Perl.
    CPAN::Reporter send back success/fail.
    CPAN testers: variety of platforms, daily reports.
    cpan-testers & cpan-testers-discuss mailing lists.
    http://cpantesters.org/
    Community effort at testing is unique to Perl.

    View Slide

  4. Excellent Reference
    Perl Testing: A developer's Notebook,
    O'Reilly Press
    Wonderful example of a good howto-book.
    Includes modules, ways to use them with variations.
    Good cover-to-cover read.

    View Slide

  5. Perl is easy to test
    Test::* modules do most of the work.
    Use the wheels, don't re-invent:
    Test::Simple, Test::More, Test::Deep,
    Test::Builder, Test::Coverage, Object::Exercise
    Perl adds introspection.

    View Slide

  6. One-stop shopping
    Perl as a great glue language.
    Use perl to test other programs.
    Have other programs output TAP.
    Combined with “inline” you can test almost anything!

    View Slide

  7. Test your tests!
    Devel::Coverage have you tested all of the code?
    Test::Builder roll your own tests.
    Build you own re-usable test components:
    Test::Harness::Straps
    Test::Builder::Tester

    View Slide

  8. “Test” code can do more
    Pre-load fixed data into a database.
    Validate the working environment.
    Execute daily cleanups in cron.

    View Slide

  9. Testing simply
    Count hardwired:
    use Test::Simple tests => 6;
    Difference in count will be an error.

    View Slide

  10. Its all “ok”
    use Test::Simple tests => 2;
    # ok( boolean, message );
    ok 1 == 1, “Count: '1' (1)”;
    ok 2 == 1, “Count: '2' (1)”;
    Output like:
    ok 1 - Count: '1' (1)
    not ok 2 - Count: '2' (1)

    View Slide

  11. Its all “ok”
    use Test::Simple tests => 2;
    # ok( boolean, message );
    ok 1, 'It passed!';
    Output like:
    1..2
    ok 1 - It passed!
    # Looks like you planned 2 tests but ran 1.

    View Slide

  12. Seen these before?
    “not ok 10 - Count”
    “not ok 11 - Format”
    “not ok 5 - File or directory not found.”
    OK, now what?

    View Slide

  13. Designing tests:
    Test what matters.
    Report something useful.

    View Slide

  14. Isolate the failure
    What to fix:
    ok -e $path , “Existing: '$path'”;
    ok -d _ , “Directory: '$path'”;
    ok -r _ , “Readable: '$path'”;
    ok -w _ , “Writeable '$path'”;

    View Slide

  15. Q: Why all of the quotes?
    ok -e $path, “Existing dir: $path”

    View Slide

  16. Q: Why all of the quotes?
    ok -e $path, “Existing dir: $path”
    not ok 1 - Existing dir: /var/tmp
    not ok 1 Existing dir: /var/tmp

    View Slide

  17. Q: Why all of the quotes?
    ok -e $path, “Existing dir: '$path'”
    not ok 1 - Existing dir: '/var/tmp '
    not ok 1 Existing dir: '/var/tmp
    '

    View Slide

  18. Good general format
    Include what you found, what you expect.
    You'll need them both with “not ok”:
    ok $got eq $want, 'Name: '$got' ($want)”;
    not ok 99 – Name: '' (Jow Bloe)
    not ok 99 – Name: '1 Main St.' (Jow Bloe)
    not ok 99 – Name: 'Jow Bloe' ()
    not ok 99 – Name: 'Jow Bloe' (User::Name=0x1A...

    View Slide

  19. Test::Simple may be enough
    Generate a plan.
    “ok” generates TAP output.
    Test count is fixed.
    This works for many day-to-day tests.

    View Slide

  20. But Wait! There's More!
    Test::More
    plan, done_testing test counts
    is, isnt, like, unlike stringy “ok”
    pass, fail skip the boolean
    use_ok, require_ok module test
    note, diag, explain messages

    View Slide

  21. Flexable count
    Set count with “plan”.
    Compute from test input:
    plan tests => 42;
    plan tests => 3 * keys $config;
    plan tests => 2 * values $query_output;
    Test count varys?
    Skip plan.

    View Slide

  22. When you're done testing
    Just say so:
    done_testing;
    Generates final report.
    Plan count is checked here.
    No plan, no count check.

    View Slide

  23. Notes show up with “prove -v”
    note “read_config from '$config_path'”;
    my $path = read_config 'tmp_dir';
    ok -e $path, “Exists: '$path' (tmp_dir)”;
    $ prove -v;
    # read_config from './etc/pass1.conf'
    not ok 1 – Exists: '/var/tmp/foobar ' (tmp_dir)

    View Slide

  24. Diagnostics show why tests failed
    Show up without “-v”.
    ok … or diag “Oopsie...”;
    ok grep /blah/, @test_input, 'Found blah'
    or diag “Missing 'blah':”, @test_input;
    ok $dbh, “Connected”
    or diag “Failed connect: '$dsn'”;

    View Slide

  25. Explain shows exactly what failed
    “explain” shows nested structure.
    Use with “note” to show setup details.
    With “diag” shows extra info on failure.
    my $dbh = DBI->connect( @argz );
    ok $dbh, 'Database connected'
    or diag 'Connect args:', explain \@argz;

    View Slide

  26. Stringy “ok”
    like, not_like use regexen.
    saves “=~” syntax in the tests:
    like $stuff, qr/milk/, “Got: 'milk'?”
    or diag “Have: '$stuff' instead”;
    not ok 1 - Got: 'milk'?
    # Have 'very new cheese' instead

    View Slide

  27. Variable number of tests
    If-logic generates variable number of tests.
    Skip “plan”, use “done_testing”.

    View Slide

  28. Taking it pass/fail
    if( $obj = eval { $class->constructify } )
    {
    # validate object contents
    }
    else
    {
    # nothing more to test
    fail “Failed constructify: [email protected]”;
    }

    View Slide

  29. Test expected errors
    eval
    {
    $obj->new( $junk );
    fail "No exception: foo( $junk )?";
    1
    }
    or do
    {
    # validate error handling
    ...
    }

    View Slide

  30. Controlling test cycle
    Abort if everything will fail.
    Avoid expensive, specialized, un-necesary tests.
    Saves extraneous code in all of the tests.

    View Slide

  31. BAIL_OUT: Knowing when to give up
    Aborts all testing.
    Unsafe or guaranteed failure.
    Useful in early test for environment, arg checks.

    View Slide

  32. BAIL_OUT
    BAIL_OUT 'Do not run tests as su!' unless $>;
    BAIL_OUT 'Missing $DB' unless $ENV{DB};
    -e $test_dir or mkdir $dir, 0777
    or BAIL_OUT “Failed mkdir: '$dir', $!”;
    -d $test_dir or BAIL_OUT “Missing: '$dir'”;
    -r _ or BAIL_OUT “Un-readable: '$dir'”;
    -w _ or BAIL_OUT “Un-writeable: '$dir'”;

    View Slide

  33. “SKIP” blocks
    Skip a block with test count and message.
    Adjust the plan test count.
    Expensive or specialized tests.

    View Slide

  34. Skip external tests
    SKIP:
    {
    skip “No database (TEST_DB)”, 8 if ! $ENV{ TEST_DB };
    # or … no network available...
    # or … no server handle...
    ...
    }

    View Slide

  35. Skip expensive tests
    SKIP:
    {
    skip “Used for internal development only”, 12
    unless $ENV{ EXPENSIVE_TESTS };
    # test plan count reduced by 12
    ...
    }

    View Slide

  36. Skip dangerous tests
    SKIP:
    {
    skip “Unsafe as superuser”, 22 unless $>;
    # at this point the tests are not running su.
    ...
    }

    View Slide

  37. You'll usually use Test::More
    note & diag nearly always worth using.
    plan & done_testing makes life easier.
    Still have “ok” for the majority of work.

    View Slide

  38. Testing Structures
    “like” syntax with nested structure:
    use Test::Deep;
    cmp_deeply $found, $expect, $message;
    Great for testing parser or grammar outcome.

    View Slide

  39. Devel::Cover: what didn't you check?
    All of the else blocks?
    All of the “||=” assignments?
    All of the “... or die ...” branches?
    Devel::Cover bookkeeps running code.
    Reports what you didn't test.
    Tells you what test to write next.

    View Slide

  40. Similar to NYTProf:
    Run your program & summarize the results:
    $ cover -test;
    or
    $ perl -MDevel::Cover ./t/foo.t;
    $ cover;
    Running “cover” generates the report.

    View Slide

  41. Introspection simplifies testing
    Want to test database connect failure.
    Can't assume SQL::Lite.
    Flat file databases are messy.
    Guarantee that something fails?
    You could write an operation that should fail.
    Then again, it might not...

    View Slide

  42. Making sure you fail
    Override class methods.
    Override Perl functions.
    Result:
    Mock objects,
    Mock methods,
    Mock perl.

    View Slide

  43. Mock Objects
    Test your wrapper handling failure?
    Override DBI::connect with sub { die … }.
    No guess: You know it's going to fail.

    View Slide

  44. Mock Modules
    Your test:
    *DBI::connect = sub { die '…' };
    my $status = eval { $obj->make_connection };
    my $err = [email protected];
    # test $status, $err, $obj...

    View Slide

  45. Force an exception
    use Symbol qw( qualify_to_ref );
    # e.g., force_exception 'Invalid username', 'connect', 'DBI';
    # ( name, packcage ) or ( “package::name” )
    sub force_exception
    {
    chomp( my $messsage = shift );
    my $ref = qualify_to_ref @_;
    undef &{ *$ref };
    *{ $ref } = sub { die “$message\n” };
    return
    }

    View Slide

  46. Testing multiple failures
    for( @testz )
    {
    my( $msg, $expect ) = @$_;
    force_exception $msg, 'DBI::connect';
    my $status = eval { $wrapper->connect };
    my $err = [email protected];
    # your tests here
    }

    View Slide

  47. Avoid caller cleanup
    Override with “local”
    sub force_exception
    {
    my ( $proto, $method, $pkg, $name, $msg ) = splice @_, 0, 5;
    my $ref = qualify_to_ref $name, $pkg;
    local *{ $ref } = sub { die “$msg” };
    # exit from block cleans up local override.
    # caller checks return, [email protected], $proto.
    eval { $proto->$method( @_ ) }
    }

    View Slide

  48. Mock Perl
    Override perl: *Core::Global::
    Is “exit” is called?
    my $exits = 0;
    *Core::Global::exit = sub { $exits = 1 };
    eval { frobnicate $arg };
    ok $exits, “Frobnicate exits ($exits)”;

    View Slide

  49. Devel::Cover & Mocking
    The biggest reason for mock anything:
    Force an outcome to test a branch.
    Iterate:
    Test with Devel::Cover.
    See what still needs testing.
    Mock object/method/function forces the branch.

    View Slide

  50. Automated testing
    Lazyness is a virtue.
    Avoid cut+paste.
    Let Perl do the work!

    View Slide

  51. Example: Sanity check modules
    Use symlinks.
    Validate modules compile.
    Check package argument.
    require_ok $package; # ok if require-able.
    use_ok $package; # ok if use-able.

    View Slide

  52. Make the links
    Path below ./lib.
    Replace slashes with dashes.
    Add a leading “01”.
    Symlink them all to a generic baseline test.
    Symlinks look like:
    01-Wcurve-Util.t → generic-01-t

    View Slide

  53. use FindBin::libs;
    use Test::More;
    use File::Basename qw( basename );
    my $madness = basename $0, '.t'; # 01-WCurve-Util
    $madness =~ s{^ 01-}{}x; # WCurve-Util
    $madness =~ s{ \W+ }{::}gx; # WCUrve::Util
    if( use_ok $madness )
    {
    # check for correct package argument.
    ok $madness->can( 'VERSION' ), “$maddness has a method”;
    ok $a = $madness->VERSION, “$madness is '$a'”;
    }
    done_testing;
    ./t/generic-01-t

    View Slide

  54. ./t/make-generic-links
    #!/bin/bash
    cd $(dirname $0);
    rm 01-*.t;
    find ../lib/ -name '*.pm' |
    perl -n \
    -e 'chomp;' \
    -e 's{^ .+ /lib/}{}x' \
    -e 's{.pm $}{.t}x' \
    -e 'tr{/}{-} \
    -e 'symlink “01-generic-t” => “01-$_”';
    exit 0;

    View Slide

  55. Similar checks for config files
    Add “00-” tests to reads config files.
    Combine “ok” with Config::Any.
    Now make-generic-links tests config & code.
    “prove” runs them all in one pass.

    View Slide

  56. The Validation Two-Step
    Use with git for sanity check:
    git pull &&
    ./t/make-generic-links &&
    prove --jobs=2 –-state=save &&
    git tag -a “prove/$date” &&
    git push &&
    git push –-tag ;

    View Slide

  57. Exercise for healthy objects
    Data-driven testing for a single object.
    Data includes method, data, expected return.
    Module iterates the tests, reports results.
    When tests or expected values are generated.

    View Slide

  58. use Object::Exercise;
    my @testz =
    (
    [
    [ qw( name ) ], # method + args
    [ q{Jow Bloe} ], # expected return
    ],
    [
    [ qw( street ) ],
    [ q{1 Nuwere St} ],
    ],
    [
    [ qw( street number ) ], # $obj->address( 'number' )
    [ qw( 1 ) ],
    'Check street number' # hardwired message
    ],
    );
    Person->new( 'Jow Blow' => '1 Nuwere St' )->$exercise( @testz );

    View Slide

  59. Load fixed data
    Flat file -> arrayrefs
    “insert” as method, [1] as return.
    Load the data with:
    $sth->$exercise( @data )
    Get “ok” message for each of the data record.

    View Slide

  60. Roll your own: Test::Builder
    Mirrors Test::More with a singleton object.
    my $test = Test::Builder->new;
    $test->ok( $boolean, $success_message )
    or $test->diag( $more_info );
    Spread single test across multiple modules.

    View Slide

  61. Testing Testing
    Test::Build::Tester wraps your tests.
    Forces test to return not-ok in order to test it.
    Ignores the output of the test being tested.
    Validate the test plan.

    View Slide

  62. Getting the most out of prove
    Save state: re-run failed tests:
    $prove –state=save;
    # fix something...
    $prove –state=save,failed;
    Parallel execution with “--jobs”.
    “perldoc” is your friend!

    View Slide

  63. Great, but my shop uses
    Test multiple languages with Inline.
    Include C, C++, Python...
    Multiple Perl versions.
    Mix languages to test interfaces.

    View Slide

  64. One-stop teting for your API lib's
    Need to test multi-language support?
    Use Perl to move data between them.
    Unit-test C talking to Java.

    View Slide

  65. Example: Testing your C code
    use Inline C;
    use Test::Simple tests => 1;
    my $a = 12 + 34;
    my $b = add( 12, 34 );
    ok $a == $b, "Add: '$b' ($a)";
    __END__
    __C__
    int add(int x, int y)
    {
    return x + y;
    }

    View Slide

  66. Example: Testing your C code
    $ prove -v add.t
    add.t ..
    1..1
    ok 1 - Add: '46' (46)
    ok
    All tests successful.
    Files=1, Tests=1, 0 wallclock secs
    ( 0.05 usr 0.00 sys + 0.09 cusr 0.00 csys = ...
    Result: PASS

    View Slide

  67. Example: Testing your C code
    You can call into a library.
    Write a whole program.
    Inline builds the interface.

    View Slide

  68. Test Anything Protcol
    Inline Language Support Modules add languages.
    Inline supports:
    C, C++, Java, Python, Ruby, Tcl,
    Assembler, Basic, Guile, Befunge, Octave,
    Awk, BC, TT (Template Toolkit), WebChat,
    and (of course) Perl.

    View Slide

  69. Really: Any Thing
    Say you had a quad-rotor UAV...
    You'd write a Perl API for it, wouldn't you?
    UAV::Pilot::ARDrone, for example.
    But then you'd have to test it...

    View Slide

  70. First: Define the test actions
    my @actionz =
    (
    [
    takeoff => 10
    ],
    [
    wave => 8000
    ],
    [
    flip_left => 5000
    ],
    [
    land => 5000,
    send => $expect
    ],
    );
    plan tests => 2 + @actionz;
    Basic set of actions:
    Takeoff, wobble, flip,
    and land.
    Final “send” validates
    end-of-channel.

    View Slide

  71. Control interfaces
    BAIL_OUT
    avoids running
    without controls.
    my $driver
    = UAV::Pilot::ARDrone::Driver->new
    (
    {
    host => $HOST,
    }
    )
    or BAIL_OUT "Failed construct";
    eval
    {
    $driver->connect;
    pass "Connected to '$HOST'";
    }
    or BAIL_OUT "Failed connect: [email protected]";

    View Slide

  72. Error handler for actions
    Attempt the last
    action (land) on
    errors.
    sub oops
    {
    eval { execute $actionz[ -1 ] }
    or
    BAIL_OUT "Literal crash expected";
    }

    View Slide

  73. sub execute
    {
    state $a = $event;
    my ( $ctrl_op, $time, $cv_op, $cv_val ) = @$_;
    $a = $a->add_timer
    (
    {
    duration => $time,
    duration_units => $event->UNITS_MILLISECOND,
    cb =>
    sub
    {
    $control->$ctrl_op or die "Failed: $ctrl_op";
    $cv->$cv_op( $cv_val ) if $cv_op;
    pass "Execute: '$ctrl_op' ($cv_op)";
    }
    }
    )

    View Slide

  74. Execute the actions
    “note” here
    describes what
    to expect.
    for( @actionz )
    {
    note "Prepare:\n", explain $_;
    eval { execute $_ }
    or
    oops;
    }
    $event->init_event_loop;

    View Slide

  75. Check the end-of-channel
    my $found = $cv->recv;
    if( $found eq $expect )
    {
    pass "Recv completed '$found' ($expect)";
    }
    else
    {
    fail "Recv incomplete ($found), send emergency land";
    oops;
    }
    done_testing;

    View Slide

  76. Execute the tests
    $ perl t/01-test-flight.t
    1..6
    # Connect to: '192.168.1.1'
    (UAV::Pilot::ARDrone::Driver)
    ok 1 - Connected to '192.168.1.1'
    # U::P::EasyEvent has a socket
    # Prepare:
    # [
    # 'takeoff',
    # 10
    # ]
    # Prepare:
    # [
    # 'wave'

    View Slide

  77. “pass” shows what happens
    # Prepare:
    # [
    # 'land',
    # 5000,
    # 'send',
    # '123'
    # ]
    ok 2 - Execute: 'takeoff' ()

    View Slide

  78. “pass” shows what happens
    # Prepare:
    # [
    # 'land',
    # 5000,
    # 'send',
    # '123'
    # ]
    ok 2 - Execute: 'takeoff' ()
    ok 3 - Execute: 'wave' ()

    View Slide

  79. “pass” shows what happens
    # Prepare:
    # [
    # 'land',
    # 5000,
    # 'send',
    # '123'
    # ]
    ok 2 - Execute: 'takeoff' ()
    ok 3 - Execute: 'wave' ()
    ok 4 - Execute: 'flip_left' ()

    View Slide

  80. “pass” shows what happens
    # Prepare:
    # [
    # 'land',
    # 5000,
    # 'send',
    # '123'
    # ]
    ok 2 - Execute: 'takeoff' ()
    ok 3 - Execute: 'wave' ()
    ok 4 - Execute: 'flip_left' ()
    ok 5 - Execute: 'land' (send)

    View Slide

  81. “pass” shows what happens
    # Prepare:
    # [
    # 'land',
    # 5000,
    # 'send',
    # '123'
    # ]
    ok 2 - Execute: 'takeoff' ()
    ok 3 - Execute: 'wave' ()
    ok 4 - Execute: 'flip_left' ()
    ok 5 - Execute: 'land' (send)
    ok 6 - Recv completed '123' (123)

    View Slide

  82. Cultured Perls
    The Developer's Notebook is a great resource.
    POD for Test::More == wheels you don't re-invent.
    POD for prove == make life faster, easier.
    cpantesters.org: test results for modules by platform.
    Stay cultured, be testy: use perl.

    View Slide