Why you ever meta data you didn’t like: Standardized testing without copies.

Many of the basic tests we perform on libraries and executables are pretty standard: does it compile? Is the package spelled properly? Rather than handle the repetition with copies, standardize the tests with metadata that doesn't need to be copied and edited with each file.

This talk describes how to encode test specifics into the basenames of files, allowing symlinks and file globs to do most of the work in driving and partitioning the tests.

Transcript

1. Why you ever meta data you didn’t like: Testing twisty little passages all alike. Steven Lembark Workhorse Computing lemark@wrkhors.com
2. What is testing? Smoke, white- & black-box, integration, regression... Set up controlled environment. See if software fails. Lather, rinse repeat... Controls include data, environment, handlers.
3. The usual way: Template tests Write a test. Make it work. Copy it. Edit it. Copy it. Edit it. Copy...
4. “Red flags” Cut + paste Works until you find a bug... Need a change... Update 45 files.
5. Common result: A mess Single directory. Semi-random basenames. Difficult to run in order.
6. First piece of metadata Filesystem is powerful medicine. Cures all sorts of problems.
7. Basenames prove uses lexical order. 100-perlcritic.t 101-perltidy.t 102-pod.t 103-podt.t 10-init.t 200-player-TestRoo.t 201-TestRoo-Action.t 201-TestRoo-Actor.t 201-TestRoo-Adventure.t 201-TestRoo-Base.t 201-TestRoo-Item.t 201-TestRoo-Location.t 201-TestRoo-Player.t 202-TestClassMoose.t 20-player.t 300-black-hole.t 60-command.t
8. Basenames prove uses lexical order. Might not be what you want. 100-perlcritic.t 101-perltidy.t 102-pod.t 103-podt.t 10-init.t 200-player-TestRoo.t 201-TestRoo-Action.t 201-TestRoo-Actor.t 201-TestRoo-Adventure.t 201-TestRoo-Base.t 201-TestRoo-Item.t 201-TestRoo-Location.t 201-TestRoo-Player.t 202-TestClassMoose.t 20-player.t 300-black-hole.t 60-command.t
9. Basenames prove uses lexical order. Might not be what you want. Use consistent names. 010-init.t 020-player.t 060-command.t 100-perlcritic.t 101-perltidy.t 102-pod.t 103-podt.t 200-player-TestRoo.t 201-TestRoo-Action.t 201-TestRoo-Actor.t 201-TestRoo-Adventure.t 201-TestRoo-Base.t 201-TestRoo-Item.t 201-TestRoo-Location.t 201-TestRoo-Player.t 202-TestClassMoose.t 300-black-hole.t
10. Directories Need every test every time? 010-init.t 020-player.t 060-command.t 100-perlcritic.t 101-perltidy.t 102-pod.t 103-podt.t 200-player-TestRoo.t 201-TestRoo-Action.t 201-TestRoo-Actor.t 201-TestRoo-Adventure.t 201-TestRoo-Base.t 201-TestRoo-Item.t 201-TestRoo-Location.t 201-TestRoo-Player.t 202-TestClassMoose.t 300-black-hole.t
11. Directories Need every test every time? What about: Quick tests for check-in. Complete for release. 010-init.t 020-player.t 060-command.t 100-perlcritic.t 101-perltidy.t 102-pod.t 103-podt.t 200-player-TestRoo.t 201-TestRoo-Action.t 201-TestRoo-Actor.t 201-TestRoo-Adventure.t 201-TestRoo-Base.t 201-TestRoo-Item.t 201-TestRoo-Location.t 201-TestRoo-Player.t 202-TestClassMoose.t 300-black-hole.t
12. Directories Need every test every time? What about: prove && commit; prove -r && release; 010-init.t 020-player.t 060-command.t 100-perlcritic.t 101-perltidy.t 102-pod.t 103-podt.t 200-Modules 300-Missions
13. Directories Need every test every time? What about: prove && commit; prove -r && release; 01-init.t 02-player.t 06-command.t 10-perlcritic.t 11-perltidy.t 12-pod.t 13-podt.t 20-Modules 30-Missions
14. Directories Need every test every time? What about: prove && commit; prove -r && release; 01-init.t 02-player.t 06-command.t 10-perlcritic.t 11-perltidy.t 12-pod.t 13-podt.t 20-Modules 30-Missions
15. Directories Need every test every time? Baseline & config checks. prove t/0*t; 01-init.t 02-player.t 06-command.t 10-perlcritic.t 11-perltidy.t 12-pod.t 13-podt.t 20-Modules 30-Missions
16. Directories Need every test every time? Simple integration tests: prove -v t/1*t; 01-init.t 02-player.t 06-command.t 10-perlcritic.t 11-perltidy.t 12-pod.t 13-podt.t 20-Modules 30-Missions
17. Tests don’t need to be stupid. Adding a little logic avoids copying. Lazy: Write once, recycle in place.
18. Choose your mission Defines the map, monsters, goal. The mission files start with: my $pkg = 'Adventure'; my $path = 't/etc/EmptyMap00.yaml';
19. Choose your mission Defines the map, monsters, goal. One set of tests starts with: Same tests for any config file. my $pkg = 'Adventure'; my $path = 't/etc/EmptyMap00.yaml';
20. Choose your mission Defines the map, monsters, goal. Don’t copy the file: my $pkg = 'Adventure'; my $base = basename $0, ‘.t’; my $path = “t/etc/$base.yaml”;
21. Choose your mission Defines the map, monsters, goal. Don’t copy the file: Symlink them all to “./bin/01-mission_t”. my $pkg = 'Adventure'; my $base = basename $0, ‘.t’; my $path = “t/etc/$base.yaml”;
22. Choose your mission cd $(dirname $0)/../20-Missions; for i in ../etc/*-mission.yaml; do base=$(basename $i .yaml); ln -fs ../bin/01-mission_t ./01-$base.t; done One “test” for many configs:
23. Generic tests: well-formed Perl “use_ok” is unfairly maligned. It does something quite useful. Better off if all modules pass it.
24. Generic tests: well-formed Perl “use_ok” is unfairly maligned. It does something quite useful. Better off if all modules pass it. And have a working package name. With a version.
25. Generic tests: well-formed Perl use Test::More; use_ok ‘Frobnicate’; can_ok Frobnicate => ‘VERSION’; ok Frobnicate->VERSION, “Frobnicate has a version”; done_testing; __END__
26. Generic tests: well-formed Perl use Test::More; use_ok ‘Frobnicate’; can_ok Frobnicate => ‘VERSION’; ok Frobnicate->VERSION, “Frobnicate has a version”; done_testing; __END__
27. Generic tests: well-formed Perl use Test::More; my $package= ‘Frobnicate’; use_ok $package; can_ok $package => ‘VERSION’; ok $frobnicate->VERSION, “$package has a version”; done_testing;
28. Generic tests: well-formed Perl my $package= ‘Frobnicate’;
29. Generic tests: well-formed Perl my $base = basename $0, ‘.t’; my @partz = split /\W+/, $base; my $package = join ‘::’, @partz[ 1 .. $#partz ];
30. Generic tests: well-formed Perl ln -fs ../bin/01-generic_t \ ./01-Acme-Eyedrops.t;
31. Test them all cd $(dirname $0)/..; rm -f 01*.t; find .. -name $glob | perl -n \ -E 'state $path = ( glob "./bin/01-*_t" )[0];' \ -E 'chomp;' \ -E 'my @a = split m{[/]}, substr $_, 3;' \ -E 'my $b = join "-", @a;' \ -E 'symlink $path => "01-$b.t" or warn' ;
32. Testing groups of files Runs same test on all modules: prove 10-*.t ; Test class & children: prove *-Parent-*.t; find t -name $glob | xargs prove;
33. Combine with commits Git tags mark prove success. Combine prove with git tags to track progress: prove && git tag … Merge tags into main branch for Q/A. Use “prove -r” in master for Q/A pass.
34. Generic tests: well formed configs. Ever fat-finger some JSON? Leave out an XML tag? Mis-quote an .ini?
35. Generic tests: config files. Ever fat-finger some JSON? Leave out an XML tag? Mis-quote an .ini? Then waste debugging code to find it?
36. Generic tests: config files To the rescue: Config::Any. If its read-able, we can check it. At least for readabiliy...
37. Generic tests: config files Same basic trick: symlink a reader. my $base = basename $0; my @partz= split /\W+/, $base; my $test = join ‘~’ => ‘01’, @partz; symlink ‘../bin/00-config_t’ => $test . ‘.t’;
38. #!/bin/bash cd $(dirname $0)/..; # run from ./t/bin rm -f 0*.t; # remove generic tests i='-1'; for glob in '*.yaml' '*.pm' # test config & modules do export j="0$((++i))"; echo "Pass: $j"; ls ./bin/$j-*_t; find .. -name $glob | perl -n \ -E 'state $path = ( glob "./bin/$ENV{j}-*_t" )[0];' \ -E 'chomp;' \ -E 'my @a = split m{[/]}, substr $_, 3;' \ -E 'my $b = join "-", @a;' \ -E 'symlink $path => "$ENV{j}-$b.t" or warn' ; done exit 0;
39. Re-duce, Re-use, Re-cycle Tests only depend on “.pm”. Re-use on multiple directories. Across projects.
40. Metadata: ./t is for testing Ever test a production database? Destructively? Ouch... Metadata: test configs are in ./t/etc.
41. Metadata: ./t is for testing # tests find ./t/lib/Foo/Config.pm # ./bin files find ./lib/Foo/Config.pm use FindBin::libs; use Foo::Config;
42. Metadata: ./t is for testing # tests prefer ./t/etc use FindBin::libs qw( base=etc export ); my $base = ‘Database.config.yaml’; my $found = first {-e “$_/$base” } @etc or die “Oops... no database config.\n”; my $path = “$found/$base”;
43. Looking inside yourself Or, at least, inside of Perl. Overloads. Closures. Non-OO Polymorphism.
44. Dispatch via scalar. Say you want to test Madness for a Method. $madness->method( … );
45. Dispatch via scalar. Say you want to test Madness for a Method. $madness->$method( … ); # Perl5 $method can be text or a subref...
46. Aside: Dispatch in Perl6 my $code = sub { … }; $madness.$code; $madness.$code( @argz ); my $name = ‘subname’; $madness.”$name”(); $madness.”$name”( @argz );
47. Testing many methods Iterate an object over many methods: $object->$_( @argz ) for @methodz;
48. Example: Object::Exercise my @plan = ( [ [ method => ( ‘a’, ‘b’ ) ], [ ‘Expected Return Value’ ], [ ‘Your message here’ ], ],
49. Example: Object::Exercise my @plan = ( [ [ method => ( ‘a’, ‘b’ ) ], [ ‘Expected Return Value’ ], [ ‘Your message here’ ], ],
50. Example: Object::Exercise use Object::Exercise; my @plan = ( [ … ], [ … ], … ); $madness ->new( … ) ->$exercise( @plan );
51. Example: Object::Exercise use Object::Exercise; my @plan = generate_plan( … ); $madness ->new( … ) ->$exercise( @plan );
52. Example: Object::Exercise my @plan = generate_plan( … ); Generate plan can read a YAML file... Query a database... Read a symlink...
53. Multiple tests in YAML --- - - - method - a - b - - Expected Return Value - - Your Message Here - - - frobnicate - ...
54. Testing system failures Check for file-read failures. Bad way: Hack the filesystem. Beter way: Hack Perl CORE.
55. All politics is local So are values. “local” provides scoped values. local $\ = “\n”; local *STDOUT = $fh;
56. Perl Testers Notebook Great book, even with the fake coffee stains. One nice technique described in detail: Hacking CORE. Say you want to open to fail.
57. Hack open??? sub { # see Perl Testing Notebook local *CORE::open = sub { die “No such file.\n” }; $madness->$method( @_ ); }
58. Fail on one specific path sub fail_on_open { my $pkg = shift; my $path = shift; my $open = $pkg->can( ‘open_config’ ); sub { $_[1] eq $path and die “Failed open: ‘$path’\n”; goto &$open; } }
59. Wrap a method to fail on a specific file my @plan = map { my $sub = fail_on_open $pkg, $_; [ [ $sub, “$_” ],undef,[ “Fail: $_” ] ] } glob “/etc/frobnicate/*.config.*”;
60. Testing runt reads .ini or data files may not have bookends. Lacking a closing marker, can you detect runts? Bad test: Write hacked files to temp dir’s. Better test: Hack your reader.
61. Dispatch a partial read my $path = shift; sub { $_[1] eq $path or goto &$wrapped; my $data = &$wrapped; substr $data, 0, rand length $data }
62. my $method = ‘do_something’; my @plan = map { my $path = $_; my $runt = gen_runt_read $path; my $ref = qualify_to_ref read_cfg => $pkg; my $sub = sub { local *{ $ref } = $runt; $object->$method( $path ) }; [ [ $sub, $path ], undef, [ “Failed read: ‘$path’” ] ] } glob $glob; $object->$exercise( @plan ); # verify failing on each conf
63. Checking mods Overload stat to return zero size, hacked mods. Force fail on -s, -r checks for data files. Return non-existant or zero UID, GID.
64. Mapped tests Good: One test file, one test result. Bad: Have to test them all each time. Alternatives: Symlink individual tests. Break glob-lists into smaller pieces.
65. Aside: “Testable” code. Monolithic code is harder to test. Testing a find-and-check-and-validate-and-open- and-read-and-close-and-evaluate-and-install-values- and-use-values-and-return is hard. Faking an open is relatively easy. So is faking a read. Un-testable code is less maintainable.
66. Adventure: Your mission... Given a data file, play the game. Twisty little passages, nasty trolls, you name it.
67. Sanity checking size Generic wrapper: call a method, check size of object. Same basic wrapper: Store size. Call something. Re-examine size.
68. --- name: Empty Map 00 namespace : EmptyMap00 locations: blackhole: name: BlackHole description : You are standing in a small Black Hole. exits : Out : blackhole items: {} player: location : blackhole items : {}
69. --- name: Empty Map 00 namespace : EmptyMap00 locations: blackhole: name: BlackHole description : You are standing in a small Black Hole. exits : Out : blackhole items: {} player: location : blackhole items : {}
70. Entering a black hole use FindBin::libs qw(base=etc export scalar); my $madness = ‘Adventure’; use_ok $madness; $madness->init(“$etc/blackhole.yaml); my $player = $madness->player; is_ok ‘blackhole’, $player->location; $player->location_object ->use_exit('blackhole'); is_ok ‘blackhole’, $player->location;
71. Look for memory leaks ok $player->move( blackhole ), “Can move out” for 1 .. 1_000_000;
72. Look for memory leaks ok $player->move( out ), “Can move out” for 1 .. 1_000_000; Downside: a million OK’s.
73. Avoid a million OK’s. my $expect = ‘out’; my $found= ‘’; my $i; for $i ( 1 .. 1_000_000 ) { $player->move( $expect ); $found = $player->location; $expect eq $found or last; } is $found, $expect, “’$found’-‘$expect’ at $i”;
74. How big are you? Memory footprint: sum_size { sum map { size $_ } @_ }
75. Yes, guys, size() does matter my $loc = $player->location_object; my $prior = 1.1*sum_size $madness, $player; $loc->use_exit('blackhole') for( 1 .. 1_000_000 ); my $after = sum_size $madness, $player; ok $after < $prior, “$after < $prior”;
76. Generic Tests Init the game with a mission file. Check the initial location. Write a black-hole file with 1 .. N stages. Check that the size doesn’t grow.
77. my $base = basename $0, ‘.t’; my $limit = ( split / \W /x, $base )[ -1 ]; my $path= make_daisy_chain_map $limit; Adventure->init( $path ); my $player = Adventure->player; my $loc = $player->location_object; my $prior = 1.1 * size $player; for my $i ( 1 .. $limit ) { my $next = ‘room_’ . $i; $loc->use_exit( $next ); $next eq $loc->location or last; my $after = size $player; $prior > $after or last; }
78. Playing with the web Ever have test a web app? But didn’t have a “back end”? You are doing it wrong!
79. Selenium Sandwich Sandwich the browser between tasty layers of Perl: use Plack; use Selenium;
80. Metadata-driven selenium testing Four structs define an iteration: [ [ DOM contents ], [ Expected values ], [ return struct ], [ result DOM ] ]
81. Failing successfully Return the HTTP failure code: [ [ DOM contents ], [ Expected values ], [ 404, “Oops...”, ], [ result DOM ] ]
82. Summary Avoid “red flags”. Use data to drive tests. Metadata to generate the data. Abstract your tests.
83. Bedside reading Object::Exercise Selenium Sandwich Getting Testy with Perl https://www.slideshare.net/lembark
84. Bedside reading Adventure: git@github.com:rizen/Adventure.git