Smoke & Mirrors: Containerizing Perl Testing

Smoke testing can be hard on the systems performing it -- you never know exactly what's going to happen when you run random code in your environment. Docker provides a useful environment for smoke testing, but requires a specialized installation and configuration if the CPAN module to update and manage the code properly.

This talk describes the operation of smoke testing using docker as a platform and the configuration of docker and Perl used to make it work.

Transcript

1. Smoke & Mirrors: Containerizing Perl Testing Steven Lembark Workhorse Computing lembark@wrkhors.com
2. What is “Smoking”? • High-level sanity check, not detailed analysis. Does it catch fire when we power it on? • CPAN runs tests as part of installation. We are unusual that way. • CPAN::Reporter reports results of tests. • CPAN::Reporter::Smoker builds with reporting.
3. Why do you smoke? • Verify personal modules. • Validate project platform. • Addict (CPAN smoker).
4. What do you smoke? • Whatever is in your sandbox. • A project and all of its supporting code. • All of CPAN, or the RECENTS file.
5. How do you smoke? • “make -wk -C myproject all test” in your sandbox. • Nightly build, git hooks, Travis CI. • CPAN or CPANM smoke testers.
6. Risks of Smoking • Pull in arbitrary code. Maybe broken. Possibly malicious. More likely just stupid system( rm -rf / *.foobar );
7. Second-hand Smoke • Permanent side effects. • Left over test files. • Wasted disk space. • Bandwidth. • Still possible with personal tests: Dependent modules.
8. Protect Yourself • Install into a dedicated sandbox. • Disposable environment.
9. One approach: Virtual Machines • Heavy footprint. • Fully configure a running machine to run a test. • Inefficient use of hardware.
10. Lightweigit: “chroot” • Filesystem isolation only. • Tests can interfere with network, processes. • Files pollute permenant storage. • No resource limits.
11. Another approach: Process isolation. • Keep processes from interfering with one another. • Lightweight: Share logger, daemons, filesytems... • More efficient use of CPU. • Less work to set up. • Not always easy to do right. Jails & Zones
12. Better way: lxc • Approach via IBM. Mainframe LPAR ported to linux. • Uses process groups. • Excellent control. • One problem: People used them like VM's. • Two problem: lxc has hundreds of switches.
13. lxc for the Masses • Docker: 80/20 of lxc. Medium isolation. • Generate an “image”. Read-only stack of AUFS mounts. • Run a “container”. Mountpoint + resources + isolation. • Decent manglement tools.
14. AUFS with lxc • Images are layered. Read-only. • "Container layer" updated. discarded. Good and bad: No permenant updates.
15. Not so nice • Images look good for smoking: Read-only underneath: minimal harm. Re-cycle O/S layer with Perly build. Allows mix'n match testing. • Catch: No fixes or upgrades to smokebox. Changes require complete rebuild.
16. One fix: Volumes • Described as “data volumes”. • Can store anything. • Like Perl, build directory, CPAN mirror. • Mountpoints and symlinks. Dir's or files from O/S into container. • Mount RO or RW. Maintain Perl & Smoker.
17. Hybrid approach • O/S is an image. • Perl, CPAN on a volume. O/S tools build and maintain perl. perl maintains /var/lib/CPAN
18. perly volumes • Tempting: perlbrew dir's as volumes. Problems with storage in /home. Hard to share across users. Storage depends on user account. Multiple accounts duplicate space. • Alternate: Let Docker manage the volumes Simpler sharing. /var/lib/docker [hopefully] has more space. Manage via docker.
19. Perl Testers Wiki • Good instructions for CPAN::Reporter::Smoker. • Start with an O/S image and get a working smoker. • CPANM has its own version. "minismokebox"
20. Grab a distro • Docker Hub has hundreds of distro images. • I'll use Gentoo here: it includes build environment. No need to install gcc, make... “stage3” is the starting point.
21. Installing Stage3 • docker import $tarball into gentoo-stage3:YYYYMMDD. • Layer custom files into smoker-gentoo-stage3:lYYYYMMDD. root's .exrc .bashrc • Used to bulid perl, run smoke.
22. Building perl, smoker • Volumes for /opt, /var/tmp & script files. Standard paths simplify build scripts. • “docker run” with command of build script. build perl. install & run minicpan. configure reporting & smoker. CPAN update to test reporting.
23. Aside: --tmpfs • docker run –tmpfs has a problem: The volume has execution turned off. Fine for data. Useless for build. • Avoid the issue with a tmpfs volume. Volume must be private to build.
24. Creating a tmpfs • Private tmp volume via PID. • tmpfs usually cleaned up automatically on the way out. • Use volume rm to be sure. tmp=”var-tmp-$BASHPID”; $docker volume create --driver=local --opt type=tmpfs --opt device=tmpfs --opt o=size=320M $tmp && docker run ... -v $tmp:/var/tmp ... ; # may fail due to non-existant volume. docker volume rm $tmp;
25. Volumes of Volumes • Temp volume -v $tmp:/var/tmp • Perl tarball -v $PWD/perl‑5.24.1.tar.gz:/var/tmp/perl.tar.gz • Build script -v $PWD/build-perl:/var/tmp/build-perl • Perl install -v opt-perl-5.24.1:/opt • CPAN cache -v /var/lib/CPAN:/var/lib/CPAN
26. Volumes of Volumes • Temp volume -v $tmp:/var/tmp • Perl tarball -v $PWD/perl‑5.24.1.tar.gz:/var/tmp/perl.tar.gz • Build script -v $PWD/build-perl:/var/tmp/build-perl • Perl install -v opt-perl-5.24.1:/opt • CPAN cache -v /var/lib/CPAN:/var/lib/CPAN
27. Volumes of Volumes • Temp volume -v $tmp:/var/tmp • Perl tarball -v $PWD/perl‑5.24.1.tar.gz:/var/tmp/perl.tar.gz • Build script -v $PWD/build-perl:/var/tmp/build-perl • Perl install -v opt-perl-5.24.1:/opt • CPAN cache -v /var/lib/CPAN:/var/lib/CPAN
28. Volumes of Volumes • Temp volume -v $tmp:/var/tmp • Perl tarball -v $PWD/perl‑5.24.1.tar.gz:/var/tmp/perl.tar.gz • Build script -v $PWD/build-perl:/var/tmp/build-perl • Perl install -v opt-perl-5.24.1:/opt • CPAN cache -v /var/lib/CPAN:/var/lib/CPAN
29. Start the ball rolling • Assemble a container. • Dispatch the build script. • “-t” avoids issues with core test that uses column/row counts. • “-a” gets output from container into log. $docker volume create \ --driver=local \ --opt type=tmpfs \ --opt device=tmpfs \ --opt o=size=320M \ $tmp && $docker volume create \ --driver=local \ $perl && $docker run --rm \ -t \ --name=”build-$perl”\ -a stdout \ -a stderr \ "${vols[@]}" \ $image \ /var/tmp/build-perl \ 2>&1 | tee build.out; docker volume rm $tmp;
30. Building Perl • The fun starts in /var/tmp/build-perl • Mostly here-scripts. • Install & configure smoketest engine. • Includes common non-smoke modules.
31. Building Perl HOME='/var/lib/CPAN'; ( cd /var/tmp; gzip -dc < perl.tar.gz | tar xf -; cd perl*; jobs=$(grep '^processor' /proc/cpuinfo | wc -l); export TEST_JOBS=$jobs; ./Configure \ -de \ -Dprefix=/opt/perl5 \ -Dman1dir='none' \ -Dman3dir='none' \ -Doptimize='-O2 -pipe -march=native' \ && /usr/bin/make -j$jobs all test_harness install; ) || exit -1; rm -rf /var/tmp/perl*; • Generic “perl.tar.gz” via docker volume mapping. • /opt/perl5 is in external volume. • Don't knead man pages.
32. Building Perl HOME='/var/lib/CPAN'; • /home is in the O/S image! Changes in container layer are discarded!! • Persistent files go in /var/lib/CPAN or /opt.
33. Building Perl TEST_JOBS=$jobs ... /usr/bin/make -j$jobs all test_harness install; Files=2410, Tests=851044, 223 wallclock secs (172.97 usr 22.15 sys + 862.57 cusr 57.18 csys = 1114.87 CPU)
34. Aside: Testing perl builds • Use tmpfs for /opt instead of disk volume. Fast, disposable storage for validating your build scripts. • git clone <whatever> /scratch; cd /scratch/whatever; • docker run … -v /scratch/whatever:/var/tmp/whatever … ; • build script does cd /var/tmp/whatever; make all test install;
35. Testing bleeding edge perl • Replace tar xvf with volume: -v /gitclone/perl-5.25.X:/var/tmp/perl • Rest of it stays the same. Configure and make run quickly. Minimal overhead if core modules don't change. Lots “up to date” messages from CPAN.
36. Minor setups • “h2ph” validates $PATH. • Pick your remote server. export PATH=/opt/perl5/bin:$PATH; h2ph -r -l /usr/include || exit -1; # at this point install seems usable lib=$(ls -d /opt/perl5/lib/5* | tail -1); site=$(ls -d /opt/perl5/lib/site_perl/5* | tail -1); cpan='/var/lib/CPAN'; build='/var/tmp/CPAN'; remote='http://mirror.uic.edu/CPAN/'; local="$cpan/sources/";
37. CPAN uses /var/lib, /var/tmp • Push CPAN config onto stable storage. • CPAN config is specific to perl version. perl -MCPAN -e 'shell'<<CPAN; yes o conf cpan_home $cpan o conf histfile $cpan/histfile o conf prefs_dir $cpan/prefs o conf build_dir $build o conf keep_source_where $local o conf auto_commit yes o conf build_dir_reuse yes o conf check_sigs yes o conf inhibit_startup_message yes o conf halt_on_failure no o conf build_cache 1 o conf index_expire 1 o conf commit CPAN
38. TLC • Update CPAN. • Deal with modules that require user input, special choices. ( CPAN_RUN_SHELL_TEST_WITHOUT_EXPECT=1; perl -MCPAN -e 'install CPAN'; perl -MCPAN -e shell <<CPAN; install Module::Signature 2 CPAN AUTOMATED_TESTING=1; # avoid interactive prompt perl -MCPAN -e 'shell <<CPAN; install Term::Readline::Perl CPAN )
39. • Common. • Smoke modules. • Iterate them with ${modules[@]}. modules=( YAML::XS JSON::XS CPAN::SQLite IO::Socket::SSL Net::SSL Net::SSLeay HTTP::Date Test::Most Metabase::Resource Module::Version Module::Version Bundle::CPAN Log::Log4perl Bundle::CPAN CPAN::Mini CPAN::Reporter CPAN::Testers Test::RequiresInternet WWW::RobotRules LWP::UserAgent WWW::Mechanize CPAN::Reporter::Smoker Bundle::CPAN::Reporter::Smoker::Tests Bundle::CPANReporter2 App::SmokeBox::Mini App::cpanminus::reporter ); Other modules
40. Generate ID file for Reporter • ~ is /var/lib/CPAN Not /home! ( mkdir ~/.cpantesters; cd ~/.cpantesters; metabase-profile <<END; Steven Lembark lembark@wrkhors.com password/secret END chmod 0400 metabase_id.json; )
41. Configure CPAN::Reporter # Generate test reports if CPAN::Reporter is installed (yes/no)? [no] yes # Would you like me configure CPAN::Reporter now? [yes] <enter> # email_from? [] example@example.com # edit_report? [default:ask/no pass/na:no] <enter> # send_report? [default:ask/yes pass/na:yes] <enter> # transport? [Metabase uri https://metabase.cpantesters.org/api/v1/ id_file metabase_id.json] <enter> # Would you like to run 'metabase-profile' now to create '/root/.cpanreporter/metabase_id.json'? [y] <enter> perl -MCPAN -e shell <<CPAN; o conf init test_report yes yes lembark@wrkhors.com no yes CPAN
42. Voyeurism • Fixing the build uses “docker exec”. • Attach to a running container. • Does not require ssh. • Allows fixing, re-running the build.
43. Voyeurism • Fixing the build uses “docker exec”. • Attach to a running container. • Does not require ssh. • Allows fixing, re-running the build. • Or just watching it: docker exec -it build-perl-5.24.1 'top';
44. Access a running container • Say you get something like: t/07_plugins.t ........... ok t/08_since_epoch.t ....... Can't locate YAML/Syck.pm in @INC (you may need to install the YAML::Syck module) (@INC contains: /var/tmp/CPAN/minismokebox-0.66-0/blib/lib /var/tmp/CPAN/minismokebox-0.66-0/blib/arch /opt/perl5/lib/site_perl/5.24.1/x86_64- linux /opt/perl5/lib/site_perl/5.24.1 /opt/perl5/lib/5.24.1/x86_64-linux /opt/perl5/lib/5.24.1 .) at t/08_since_epoch.t line 12. BEGIN failed--compilation aborted at t/08_since_epoch.t line 12. # Looks like your test exited with 2 before it could output anything. t/08_since_epoch.t ....... Dubious, test returned 2 (wstat 512, 0x200) Failed 11/11 subtests
45. Access a running container • Say you get something like: • Q: How do you fix it? t/07_plugins.t ........... ok t/08_since_epoch.t ....... Can't locate YAML/Syck.pm in @INC (you may need to install the YAML::Syck module) (@INC contains: /var/tmp/CPAN/minismokebox-0.66-0/blib/lib /var/tmp/CPAN/minismokebox-0.66-0/blib/arch /opt/perl5/lib/site_perl/5.24.1/x86_64- linux /opt/perl5/lib/site_perl/5.24.1 /opt/perl5/lib/5.24.1/x86_64-linux /opt/perl5/lib/5.24.1 .) at t/08_since_epoch.t line 12. BEGIN failed--compilation aborted at t/08_since_epoch.t line 12. # Looks like your test exited with 2 before it could output anything. t/08_since_epoch.t ....... Dubious, test returned 2 (wstat 512, 0x200) Failed 11/11 subtests
46. Access a running container • Exec into the container. • Use CPAN to install the module. • Which leaves you with: docker exec -it build-perl-5.24.1 bash –-login;
47. Aside: Container ID • docker ps shows what is running. Includes the container id & name: $ docker ps; CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 493d9f285faf jekyll:5000/workhorse/gentoo-stage3-amd64 "/bin/bash -c /var..." 28 minutes ago Up 28 minutes build-perl-5.24.1 # same result: $ docker exec -it 493d9f285faf 'bash login'; ‑‑ root@493d9f285faf ~ # $ docker exec -it build-perl-5.4.21 'bash login'; ‑‑ root@493d9f285faf ~ #
48. Through the looking glass • Inside the container you are root. Kernel's proc struct has unchanged EUID. • Limited view of the system. Only container proc's in “ps” or “top”. Nice on a crowded server. Runlevel restrictions are ineffective! • Run “perl -MCPAN” or “make install”. Helpful if a non-essential test fails.
49. Through the Looking Glass • Outside docker you have a specific PID & EUID. • Inside docker the initial process has EUID 0 and PID 1. • EUID 0 is manageable. • PID 1 requires that the smoketest proc reap children. Otherwise you end up with defunct proc's. Will eventually eat all proc slots with long-running container. One fix is Docker::Reaper.
50. Remember: • Changes to the image go to a separate AUFS layer. • They are discarded. • Fixing files in /etc/ will rescue this build, not the next one. • Changes to /opt & /var/lib/CPAN are persistent.
51. TMTOWTDI: minicpan • Outside of docker. Maintain CPAN cache in /var/lib. • Inside docker. Use /var/lib.
52. Running repeated jobs • Docker can automatically restart running jobs. • docker run -d ‑‑restart=always ... Downside: Lots of proc's started up at once. Issues with tmpfs on system restarts. • --restart=unless-stopped Allows stopping volumes on the way down. Start singly on way back up.
53. Smoketest lifecyle • Run minicpan once on /var/lib/CPAN. Global cache of modules. • Run smoker with O/S images & perly vol's. • This will only test *one* kernel! • Use VM's running docker for multi-kernel smoke tests.
54. Zombie apocolypse • Container startup command is process #1. • When it dies the container exits. • It has to reap all children. Not a big issue for small, single-purpose containers. Big problem for smoketests.
55. Fork & Reap • Parent propagates signals. • Exits with $? from child process. • Loop reaps zombies. Without this process slots will be exhausted. If that happens: log on as SU and stop, rm the container. for(;;) { my $pid = wait; $pid != -1 or die "Lost child pid ($child)"; $pid != $child and next; return $? }
56. Smoking • Volumes for perl CPAN /var/tmp • Run smoker. Never exits. vols=( "-v $perl:/opt:ro" "-v $cpan:$cpan" "-v $tmp:/var/tmp" ); ( $docker volume create \ ... $tmp \ && exec $run \ -t \ --name="$cname" \ ${vols[@]} \ $image \ "/var/lib/CPAN/.cpantesters/docker-smoker" ) 2>&1 | tee $log;
57. Propsed change • Smoker processes recents and exit. Simplifies iterating multiple perl verisons. • Goal: Product of images, perl vol's. Automate bleed test.
58. Summary • Smoking can be hazardous. • LXC provides lightweight solution. • Mix O/S image, perl volume. O/S is fixed. Volume updated to maintain smoker. • git@github.com:lembark/docker-smoker.git Mostly shell kwikhaks. Working to make it cleaner, perly.