Why you ever meta data you didn’t like: Standardized testing without copies.

Many of the basic tests we perform on libraries and executables are pretty standard: does it compile? Is the package spelled properly? Rather than handle the repetition with copies, standardize the tests with metadata that doesn't need to be copied and edited with each file.

This talk describes how to encode test specifics into the basenames of files, allowing symlinks and file globs to do most of the work in driving and partitioning the tests.

Transcript

1. Why you ever meta data you didn’t like:
   Testing twisty little passages all alike.
   Steven Lembark
   Workhorse Computing
   lemark@wrkhors.com
   

   View Slide

2. What is testing?
   Smoke, white- & black-box, integration, regression...
   Set up controlled environment.
   See if software fails.
   Lather, rinse repeat...
   Controls include data, environment, handlers.
   

   View Slide

3. The usual way: Template tests
   Write a test.
   Make it work.
   Copy it.
   Edit it.
   Copy it.
   Edit it.
   Copy...
   

   View Slide

4. “Red flags”
   Cut + paste
   Works until you find a bug...
   Need a change...
   Update 45 files.
   

   View Slide

5. Common result: A mess
   Single directory.
   Semi-random basenames.
   Difficult to run in order.
   

   View Slide

6. First piece of metadata
   Filesystem is powerful medicine.
   Cures all sorts of problems.
   

   View Slide

7. Basenames
   prove uses lexical order.
   100-perlcritic.t
   101-perltidy.t
   102-pod.t
   103-podt.t
   10-init.t
   200-player-TestRoo.t
   201-TestRoo-Action.t
   201-TestRoo-Actor.t
   201-TestRoo-Adventure.t
   201-TestRoo-Base.t
   201-TestRoo-Item.t
   201-TestRoo-Location.t
   201-TestRoo-Player.t
   202-TestClassMoose.t
   20-player.t
   300-black-hole.t
   60-command.t
   

   View Slide

8. Basenames
   prove uses lexical order.
   Might not be what you
   want.
   100-perlcritic.t
   101-perltidy.t
   102-pod.t
   103-podt.t
   10-init.t
   200-player-TestRoo.t
   201-TestRoo-Action.t
   201-TestRoo-Actor.t
   201-TestRoo-Adventure.t
   201-TestRoo-Base.t
   201-TestRoo-Item.t
   201-TestRoo-Location.t
   201-TestRoo-Player.t
   202-TestClassMoose.t
   20-player.t
   300-black-hole.t
   60-command.t
   

   View Slide

9. Basenames
   prove uses lexical order.
   Might not be what you
   want.
   Use consistent names.
   010-init.t
   020-player.t
   060-command.t
   100-perlcritic.t
   101-perltidy.t
   102-pod.t
   103-podt.t
   200-player-TestRoo.t
   201-TestRoo-Action.t
   201-TestRoo-Actor.t
   201-TestRoo-Adventure.t
   201-TestRoo-Base.t
   201-TestRoo-Item.t
   201-TestRoo-Location.t
   201-TestRoo-Player.t
   202-TestClassMoose.t
   300-black-hole.t
   

   View Slide

10. Directories
    Need every test every time?
    010-init.t
    020-player.t
    060-command.t
    100-perlcritic.t
    101-perltidy.t
    102-pod.t
    103-podt.t
    200-player-TestRoo.t
    201-TestRoo-Action.t
    201-TestRoo-Actor.t
    201-TestRoo-Adventure.t
    201-TestRoo-Base.t
    201-TestRoo-Item.t
    201-TestRoo-Location.t
    201-TestRoo-Player.t
    202-TestClassMoose.t
    300-black-hole.t
    

    View Slide

11. Directories
    Need every test every time?
    What about:
    Quick tests for check-in.
    Complete for release.
    010-init.t
    020-player.t
    060-command.t
    100-perlcritic.t
    101-perltidy.t
    102-pod.t
    103-podt.t
    200-player-TestRoo.t
    201-TestRoo-Action.t
    201-TestRoo-Actor.t
    201-TestRoo-Adventure.t
    201-TestRoo-Base.t
    201-TestRoo-Item.t
    201-TestRoo-Location.t
    201-TestRoo-Player.t
    202-TestClassMoose.t
    300-black-hole.t
    

    View Slide

12. Directories
    Need every test every time?
    What about:
    prove && commit;
    prove -r && release;
    010-init.t
    020-player.t
    060-command.t
    100-perlcritic.t
    101-perltidy.t
    102-pod.t
    103-podt.t
    200-Modules
    300-Missions
    

    View Slide

13. Directories
    Need every test every time?
    What about:
    prove && commit;
    prove -r && release;
    01-init.t
    02-player.t
    06-command.t
    10-perlcritic.t
    11-perltidy.t
    12-pod.t
    13-podt.t
    20-Modules
    30-Missions
    

    View Slide

14. Directories
    Need every test every time?
    What about:
    prove && commit;
    prove -r && release;
    01-init.t
    02-player.t
    06-command.t
    10-perlcritic.t
    11-perltidy.t
    12-pod.t
    13-podt.t
    20-Modules
    30-Missions
    

    View Slide

15. Directories
    Need every test every time?
    Baseline & config checks.
    prove t/0*t;
    01-init.t
    02-player.t
    06-command.t
    10-perlcritic.t
    11-perltidy.t
    12-pod.t
    13-podt.t
    20-Modules
    30-Missions
    

    View Slide

16. Directories
    Need every test every time?
    Simple integration tests:
    prove -v t/1*t;
    01-init.t
    02-player.t
    06-command.t
    10-perlcritic.t
    11-perltidy.t
    12-pod.t
    13-podt.t
    20-Modules
    30-Missions
    

    View Slide

17. Tests don’t need to be stupid.
    Adding a little logic avoids copying.
    Lazy: Write once, recycle in place.
    

    View Slide

18. Choose your mission
    Defines the map, monsters, goal.
    The mission files start with:
    my $pkg = 'Adventure';
    my $path = 't/etc/EmptyMap00.yaml';
    

    View Slide

19. Choose your mission
    Defines the map, monsters, goal.
    One set of tests starts with:
    Same tests for any config file.
    my $pkg = 'Adventure';
    my $path = 't/etc/EmptyMap00.yaml';
    

    View Slide

20. Choose your mission
    Defines the map, monsters, goal.
    Don’t copy the file:
    my $pkg = 'Adventure';
    my $base = basename $0, ‘.t’;
    my $path = “t/etc/$base.yaml”;
    

    View Slide

21. Choose your mission
    Defines the map, monsters, goal.
    Don’t copy the file:
    Symlink them all to “./bin/01-mission_t”.
    my $pkg = 'Adventure';
    my $base = basename $0, ‘.t’;
    my $path = “t/etc/$base.yaml”;
    

    View Slide

22. Choose your mission
    cd $(dirname $0)/../20-Missions;
    for i in ../etc/*-mission.yaml;
    do
    base=$(basename $i .yaml);
    ln -fs ../bin/01-mission_t ./01-$base.t;
    done
    One “test” for many configs:
    

    View Slide

23. Generic tests: well-formed Perl
    “use_ok” is unfairly maligned.
    It does something quite useful.
    Better off if all modules pass it.
    

    View Slide

24. Generic tests: well-formed Perl
    “use_ok” is unfairly maligned.
    It does something quite useful.
    Better off if all modules pass it.
    And have a working package name.
    With a version.
    

    View Slide

25. Generic tests: well-formed Perl
    use Test::More;
    use_ok ‘Frobnicate’;
    can_ok Frobnicate => ‘VERSION’;
    ok Frobnicate->VERSION,
    “Frobnicate has a version”;
    done_testing;
    __END__
    

    View Slide

26. Generic tests: well-formed Perl
    use Test::More;
    use_ok ‘Frobnicate’;
    can_ok Frobnicate => ‘VERSION’;
    ok Frobnicate->VERSION,
    “Frobnicate has a version”;
    done_testing;
    __END__
    

    View Slide

27. Generic tests: well-formed Perl
    use Test::More;
    my $package= ‘Frobnicate’;
    use_ok $package;
    can_ok $package => ‘VERSION’;
    ok $frobnicate->VERSION,
    “$package has a version”;
    done_testing;
    

    View Slide

28. Generic tests: well-formed Perl
    my $package= ‘Frobnicate’;
    

    View Slide

29. Generic tests: well-formed Perl
    my $base = basename $0, ‘.t’;
    my @partz = split /\W+/, $base;
    my $package
    = join ‘::’, @partz[ 1 .. $#partz ];
    

    View Slide

30. Generic tests: well-formed Perl
    ln -fs ../bin/01-generic_t \
    ./01-Acme-Eyedrops.t;
    

    View Slide

31. Test them all
    cd $(dirname $0)/..;
    rm -f 01*.t;
    find .. -name $glob |
    perl -n \
    -E 'state $path = ( glob "./bin/01-*_t" )[0];' \
    -E 'chomp;' \
    -E 'my @a = split m{[/]}, substr $_, 3;' \
    -E 'my $b = join "-", @a;' \
    -E 'symlink $path => "01-$b.t" or warn' ;
    

    View Slide

32. Testing groups of files
    Runs same test on all modules:
    prove 10-*.t ;
    Test class & children:
    prove *-Parent-*.t;
    find t -name $glob | xargs prove;
    

    View Slide

33. Combine with commits
    Git tags mark prove success.
    Combine prove with git tags to track progress:
    prove && git tag …
    Merge tags into main branch for Q/A.
    Use “prove -r” in master for Q/A pass.
    

    View Slide

34. Generic tests: well formed configs.
    Ever fat-finger some JSON?
    Leave out an XML tag?
    Mis-quote an .ini?
    

    View Slide

35. Generic tests: config files.
    Ever fat-finger some JSON?
    Leave out an XML tag?
    Mis-quote an .ini?
    Then waste debugging code to find it?
    

    View Slide

36. Generic tests: config files
    To the rescue: Config::Any.
    If its read-able, we can check it.
    At least for readabiliy...
    

    View Slide

37. Generic tests: config files
    Same basic trick: symlink a reader.
    my $base = basename $0;
    my @partz= split /\W+/, $base;
    my $test = join ‘~’ => ‘01’, @partz;
    symlink ‘../bin/00-config_t’ =>
    $test . ‘.t’;
    

    View Slide

38. #!/bin/bash
    cd $(dirname $0)/..; # run from ./t/bin
    rm -f 0*.t; # remove generic tests
    i='-1';
    for glob in '*.yaml' '*.pm' # test config & modules
    do
    export j="0$((++i))";
    echo "Pass: $j";
    ls ./bin/$j-*_t;
    find .. -name $glob |
    perl -n \
    -E 'state $path = ( glob "./bin/$ENV{j}-*_t" )[0];' \
    -E 'chomp;' \
    -E 'my @a = split m{[/]}, substr $_, 3;' \
    -E 'my $b = join "-", @a;' \
    -E 'symlink $path => "$ENV{j}-$b.t" or warn' ;
    done
    exit 0;
    

    View Slide

39. Re-duce, Re-use, Re-cycle
    Tests only depend on “.pm”.
    Re-use on multiple directories.
    Across projects.
    

    View Slide

40. Metadata: ./t is for testing
    Ever test a production database?
    Destructively?
    Ouch...
    Metadata: test configs are in ./t/etc.
    

    View Slide

41. Metadata: ./t is for testing
    # tests find ./t/lib/Foo/Config.pm
    # ./bin files find ./lib/Foo/Config.pm
    use FindBin::libs;
    use Foo::Config;
    

    View Slide

42. Metadata: ./t is for testing
    # tests prefer ./t/etc
    use FindBin::libs qw( base=etc export );
    my $base = ‘Database.config.yaml’;
    my $found
    = first {-e “$_/$base” } @etc
    or die “Oops... no database config.\n”;
    my $path = “$found/$base”;
    

    View Slide

43. Looking inside yourself
    Or, at least, inside of Perl.
    Overloads.
    Closures.
    Non-OO Polymorphism.
    

    View Slide

44. Dispatch via scalar.
    Say you want to test Madness for a Method.
    $madness->method( … );
    

    View Slide

45. Dispatch via scalar.
    Say you want to test Madness for a Method.
    $madness->$method( … ); # Perl5
    $method can be text or a subref...
    

    View Slide

46. Aside: Dispatch in Perl6
    my $code = sub { … };
    $madness.$code;
    $madness.$code( @argz );
    my $name = ‘subname’;
    $madness.”$name”();
    $madness.”$name”( @argz );
    

    View Slide

47. Testing many methods
    Iterate an object over many methods:
    $object->$_( @argz )
    for @methodz;
    

    View Slide

48. Example: Object::Exercise
    my @plan =
    (
    [
    [ method => ( ‘a’, ‘b’ ) ],
    [ ‘Expected Return Value’ ],
    [ ‘Your message here’ ],
    ],
    

    View Slide

49. Example: Object::Exercise
    my @plan =
    (
    [
    [ method => ( ‘a’, ‘b’ ) ],
    [ ‘Expected Return Value’ ],
    [ ‘Your message here’ ],
    ],
    

    View Slide

50. Example: Object::Exercise
    use Object::Exercise;
    my @plan = ( [ … ], [ … ], … );
    $madness
    ->new( … )
    ->$exercise( @plan );
    

    View Slide

51. Example: Object::Exercise
    use Object::Exercise;
    my @plan = generate_plan( … );
    $madness
    ->new( … )
    ->$exercise( @plan );
    

    View Slide

52. Example: Object::Exercise
    my @plan = generate_plan( … );
    Generate plan can read a YAML file...
    Query a database...
    Read a symlink...
    

    View Slide

53. Multiple tests in YAML
    ---
    -
    - - method
    - a
    - b
    - - Expected Return Value
    - - Your Message Here
    -
    - - frobnicate
    - ...
    

    View Slide

54. Testing system failures
    Check for file-read failures.
    Bad way: Hack the filesystem.
    Beter way: Hack Perl CORE.
    

    View Slide

55. All politics is local
    So are values.
    “local” provides scoped values.
    local $\ = “\n”;
    local *STDOUT = $fh;
    

    View Slide

56. Perl Testers Notebook
    Great book, even with the fake coffee stains.
    One nice technique described in detail:
    Hacking CORE.
    Say you want to open to fail.
    

    View Slide

57. Hack open???
    sub
    {
    # see Perl Testing Notebook
    local *CORE::open
    = sub { die “No such file.\n” };
    $madness->$method( @_ );
    }
    

    View Slide

58. Fail on one specific path
    sub fail_on_open
    {
    my $pkg = shift;
    my $path = shift;
    my $open = $pkg->can( ‘open_config’ );
    sub
    {
    $_[1] eq $path
    and die “Failed open: ‘$path’\n”;
    goto &$open;
    }
    }
    

    View Slide

59. Wrap a method to fail on a specific file
    my @plan
    = map
    {
    my $sub = fail_on_open $pkg, $_;
    [
    [ $sub, “$_” ],undef,[ “Fail: $_” ]
    ]
    }
    glob “/etc/frobnicate/*.config.*”;
    

    View Slide

60. Testing runt reads
    .ini or data files may not have bookends.
    Lacking a closing marker, can you detect runts?
    Bad test: Write hacked files to temp dir’s.
    Better test: Hack your reader.
    

    View Slide

61. Dispatch a partial read
    my $path = shift;
    sub
    {
    $_[1] eq $path or goto &$wrapped;
    my $data = &$wrapped;
    substr $data, 0, rand length $data
    }
    

    View Slide

62. my $method = ‘do_something’;
    my @plan
    = map
    {
    my $path = $_;
    my $runt = gen_runt_read $path;
    my $ref = qualify_to_ref read_cfg => $pkg;
    my $sub
    = sub
    {
    local *{ $ref } = $runt;
    $object->$method( $path )
    };
    [
    [ $sub, $path ], undef, [ “Failed read: ‘$path’” ]
    ]
    }
    glob $glob;
    $object->$exercise( @plan ); # verify failing on each conf
    

    View Slide

63. Checking mods
    Overload stat to return zero size, hacked mods.
    Force fail on -s, -r checks for data files.
    Return non-existant or zero UID, GID.
    

    View Slide

64. Mapped tests
    Good: One test file, one test result.
    Bad: Have to test them all each time.
    Alternatives:
    Symlink individual tests.
    Break glob-lists into smaller pieces.
    

    View Slide

65. Aside: “Testable” code.
    Monolithic code is harder to test.
    Testing a find-and-check-and-validate-and-open-
    and-read-and-close-and-evaluate-and-install-values-
    and-use-values-and-return is hard.
    Faking an open is relatively easy.
    So is faking a read.
    Un-testable code is less maintainable.
    

    View Slide

66. Adventure: Your mission...
    Given a data file, play the game.
    Twisty little passages, nasty trolls, you name it.
    

    View Slide

67. Sanity checking size
    Generic wrapper: call a method, check size of
    object.
    Same basic wrapper:
    Store size.
    Call something.
    Re-examine size.
    

    View Slide

68. ---
    name: Empty Map 00
    namespace : EmptyMap00
    locations:
    blackhole:
    name: BlackHole
    description : You are standing in a small Black Hole.
    exits :
    Out : blackhole
    items: {}
    player:
    location : blackhole
    items : {}
    

    View Slide

69. ---
    name: Empty Map 00
    namespace : EmptyMap00
    locations:
    blackhole:
    name: BlackHole
    description : You are standing in a small Black Hole.
    exits :
    Out : blackhole
    items: {}
    player:
    location : blackhole
    items : {}
    

    View Slide

70. Entering a black hole
    use FindBin::libs qw(base=etc export scalar);
    my $madness = ‘Adventure’;
    use_ok $madness;
    $madness->init(“$etc/blackhole.yaml);
    my $player = $madness->player;
    is_ok ‘blackhole’, $player->location;
    $player->location_object
    ->use_exit('blackhole');
    is_ok ‘blackhole’, $player->location;
    

    View Slide

71. Look for memory leaks
    ok $player->move( blackhole ), “Can move out”
    for 1 .. 1_000_000;
    

    View Slide

72. Look for memory leaks
    ok $player->move( out ), “Can move out”
    for 1 .. 1_000_000;
    Downside: a million OK’s.
    

    View Slide

73. Avoid a million OK’s.
    my $expect = ‘out’;
    my $found= ‘’;
    my $i;
    for $i ( 1 .. 1_000_000 )
    {
    $player->move( $expect );
    $found = $player->location;
    $expect eq $found or last;
    }
    is $found, $expect, “’$found’-‘$expect’ at $i”;
    

    View Slide

74. How big are you?
    Memory footprint:
    sum_size
    {
    sum map { size $_ } @_
    }
    

    View Slide

75. Yes, guys, size() does matter
    my $loc = $player->location_object;
    my $prior = 1.1*sum_size $madness, $player;
    $loc->use_exit('blackhole')
    for( 1 .. 1_000_000 );
    my $after = sum_size $madness, $player;
    ok $after < $prior, “$after < $prior”;
    

    View Slide

76. Generic Tests
    Init the game with a mission file.
    Check the initial location.
    Write a black-hole file with 1 .. N stages.
    Check that the size doesn’t grow.
    

    View Slide

77. my $base = basename $0, ‘.t’;
    my $limit = ( split / \W /x, $base )[ -1 ];
    my $path= make_daisy_chain_map $limit;
    Adventure->init( $path );
    my $player = Adventure->player;
    my $loc = $player->location_object;
    my $prior = 1.1 * size $player;
    for my $i ( 1 .. $limit )
    {
    my $next = ‘room_’ . $i;
    $loc->use_exit( $next );
    $next eq $loc->location or last;
    my $after = size $player;
    $prior > $after or last;
    }
    

    View Slide

78. Playing with the web
    Ever have test a web app?
    But didn’t have a “back end”?
    You are doing it wrong!
    

    View Slide

79. Selenium Sandwich
    Sandwich the browser between tasty layers of Perl:
    use Plack;
    use Selenium;
    

    View Slide

80. Metadata-driven selenium testing
    Four structs define an iteration:
    [
    [ DOM contents ],
    [ Expected values ],
    [ return struct ],
    [ result DOM ]
    ]
    

    View Slide

81. Failing successfully
    Return the HTTP failure code:
    [
    [ DOM contents ],
    [ Expected values ],
    [ 404, “Oops...”, ],
    [ result DOM ]
    ]
    

    View Slide

82. Summary
    Avoid “red flags”.
    Use data to drive tests.
    Metadata to generate the data.
    Abstract your tests.
    

    View Slide

83. Bedside reading
    Object::Exercise
    Selenium Sandwich
    Getting Testy with Perl
    https://www.slideshare.net/lembark
    

    View Slide

84. Bedside reading
    Adventure:
    git@github.com:rizen/Adventure.git
    

    View Slide