Smoke & Mirrors: Containerizing Perl Testing Steven Lembark Workhorse Computing [email protected] 

What is “Smoking”? ● High-level sanity check, not detailed analysis. Does it catch fire when we power it on? ● CPAN runs tests as part of installation. We are unusual that way. ● CPAN::Reporter reports results of tests. ● CPAN::Reporter::Smoker builds with reporting. 

Why do you smoke? ● Verify personal modules. ● Validate project platform. ● Addict (CPAN smoker). 

What do you smoke? ● Whatever is in your sandbox. ● A project and all of its supporting code. ● All of CPAN, or the RECENTS file. 

How do you smoke? ● “make -wk -C myproject all test” in your sandbox. ● Nightly build, git hooks, Travis CI. ● CPAN or CPANM smoke testers. 

Risks of Smoking ● Pull in arbitrary code. Maybe broken. Possibly malicious. More likely just stupid system( rm -rf / *.foobar ); 

Second-hand Smoke ● Permanent side effects. ● Left over test files. ● Wasted disk space. ● Bandwidth. ● Still possible with personal tests: Dependent modules. 

Protect Yourself ● Install into a dedicated sandbox. ● Disposable environment. 

One approach: Virtual Machines ● Heavy footprint. ● Fully configure a running machine to run a test. ● Inefficient use of hardware. 

Lightweigit: “chroot” ● Filesystem isolation only. ● Tests can interfere with network, processes. ● Files pollute permenant storage. ● No resource limits. 

Another approach: Process isolation. ● Keep processes from interfering with one another. ● Lightweight: Share logger, daemons, filesytems... ● More efficient use of CPU. ● Less work to set up. ● Not always easy to do right. Jails & Zones 

Better way: lxc ● Approach via IBM. Mainframe LPAR ported to linux. ● Uses process groups. ● Excellent control. ● One problem: People used them like VM's. ● Two problem: lxc has hundreds of switches. 

lxc for the Masses ● Docker: 80/20 of lxc. Medium isolation. ● Generate an “image”. Read-only stack of AUFS mounts. ● Run a “container”. Mountpoint + resources + isolation. ● Decent manglement tools. 

AUFS with lxc ● Images are layered. Read-only. ● "Container layer" updated. discarded. Good and bad: No permenant updates. 

Not so nice ● Images look good for smoking: Read-only underneath: minimal harm. Re-cycle O/S layer with Perly build. Allows mix'n match testing. ● Catch: No fixes or upgrades to smokebox. Changes require complete rebuild. 

One fix: Volumes ● Described as “data volumes”. ● Can store anything. ● Like Perl, build directory, CPAN mirror. ● Mountpoints and symlinks. Dir's or files from O/S into container. ● Mount RO or RW. Maintain Perl & Smoker. 

Hybrid approach ● O/S is an image. ● Perl, CPAN on a volume. O/S tools build and maintain perl. perl maintains /var/lib/CPAN 

perly volumes ● Tempting: perlbrew dir's as volumes. Problems with storage in /home. Hard to share across users. Storage depends on user account. Multiple accounts duplicate space. ● Alternate: Let Docker manage the volumes Simpler sharing. /var/lib/docker [hopefully] has more space. Manage via docker. 

Perl Testers Wiki ● Good instructions for CPAN::Reporter::Smoker. ● Start with an O/S image and get a working smoker. ● CPANM has its own version. "minismokebox" 

Grab a distro ● Docker Hub has hundreds of distro images. ● I'll use Gentoo here: it includes build environment. No need to install gcc, make... “stage3” is the starting point. 

Installing Stage3 ● docker import $tarball into gentoo-stage3:YYYYMMDD. ● Layer custom files into smoker-gentoo-stage3:lYYYYMMDD. root's .exrc .bashrc ● Used to bulid perl, run smoke. 

Building perl, smoker ● Volumes for /opt, /var/tmp & script files. Standard paths simplify build scripts. ● “docker run” with command of build script. build perl. install & run minicpan. configure reporting & smoker. CPAN update to test reporting. 

Aside: --tmpfs ● docker run –tmpfs has a problem: The volume has execution turned off. Fine for data. Useless for build. ● Avoid the issue with a tmpfs volume. Volume must be private to build. 

Creating a tmpfs ● Private tmp volume via PID. ● tmpfs usually cleaned up automatically on the way out. ● Use volume rm to be sure. tmp=”var-tmp-$BASHPID”; $docker volume create --driver=local --opt type=tmpfs --opt device=tmpfs --opt o=size=320M $tmp && docker run ... -v $tmp:/var/tmp ... ; # may fail due to non-existant volume. docker volume rm $tmp; 

Volumes of Volumes ● Temp volume -v $tmp:/var/tmp ● Perl tarball -v $PWD/perl‑5.24.1.tar.gz:/var/tmp/perl.tar.gz ● Build script -v $PWD/build-perl:/var/tmp/build-perl ● Perl install -v opt-perl-5.24.1:/opt ● CPAN cache -v /var/lib/CPAN:/var/lib/CPAN 

Volumes of Volumes ● Temp volume -v $tmp:/var/tmp ● Perl tarball -v $PWD/perl‑5.24.1.tar.gz:/var/tmp/perl.tar.gz ● Build script -v $PWD/build-perl:/var/tmp/build-perl ● Perl install -v opt-perl-5.24.1:/opt ● CPAN cache -v /var/lib/CPAN:/var/lib/CPAN 

Volumes of Volumes ● Temp volume -v $tmp:/var/tmp ● Perl tarball -v $PWD/perl‑5.24.1.tar.gz:/var/tmp/perl.tar.gz ● Build script -v $PWD/build-perl:/var/tmp/build-perl ● Perl install -v opt-perl-5.24.1:/opt ● CPAN cache -v /var/lib/CPAN:/var/lib/CPAN 

Volumes of Volumes ● Temp volume -v $tmp:/var/tmp ● Perl tarball -v $PWD/perl‑5.24.1.tar.gz:/var/tmp/perl.tar.gz ● Build script -v $PWD/build-perl:/var/tmp/build-perl ● Perl install -v opt-perl-5.24.1:/opt ● CPAN cache -v /var/lib/CPAN:/var/lib/CPAN 

Start the ball rolling ● Assemble a container. ● Dispatch the build script. ● “-t” avoids issues with core test that uses column/row counts. ● “-a” gets output from container into log. $docker volume create \ --driver=local \ --opt type=tmpfs \ --opt device=tmpfs \ --opt o=size=320M \ $tmp && $docker volume create \ --driver=local \ $perl && $docker run --rm \ -t \ --name=”build-$perl”\ -a stdout \ -a stderr \ "${vols[@]}" \ $image \ /var/tmp/build-perl \ 2>&1 | tee build.out; docker volume rm $tmp; 

Building Perl ● The fun starts in /var/tmp/build-perl ● Mostly here-scripts. ● Install & configure smoketest engine. ● Includes common non-smoke modules. 

Building Perl HOME='/var/lib/CPAN'; ( cd /var/tmp; gzip -dc < perl.tar.gz | tar xf -; cd perl*; jobs=$(grep '^processor' /proc/cpuinfo | wc -l); export TEST_JOBS=$jobs; ./Configure \ -de \ -Dprefix=/opt/perl5 \ -Dman1dir='none' \ -Dman3dir='none' \ -Doptimize='-O2 -pipe -march=native' \ && /usr/bin/make -j$jobs all test_harness install; ) || exit -1; rm -rf /var/tmp/perl*; ● Generic “perl.tar.gz” via docker volume mapping. ● /opt/perl5 is in external volume. ● Don't knead man pages. 

Building Perl HOME='/var/lib/CPAN'; ● /home is in the O/S image! Changes in container layer are discarded!! ● Persistent files go in /var/lib/CPAN or /opt. 

Building Perl TEST_JOBS=$jobs ... /usr/bin/make -j$jobs all test_harness install; Files=2410, Tests=851044, 223 wallclock secs (172.97 usr 22.15 sys + 862.57 cusr 57.18 csys = 1114.87 CPU) 

Aside: Testing perl builds ● Use tmpfs for /opt instead of disk volume. Fast, disposable storage for validating your build scripts. ● git clone /scratch; cd /scratch/whatever; ● docker run … -v /scratch/whatever:/var/tmp/whatever … ; ● build script does cd /var/tmp/whatever; make all test install; 

Testing bleeding edge perl ● Replace tar xvf with volume: -v /gitclone/perl-5.25.X:/var/tmp/perl ● Rest of it stays the same. Configure and make run quickly. Minimal overhead if core modules don't change. Lots “up to date” messages from CPAN. 

Minor setups ● “h2ph” validates $PATH. ● Pick your remote server. export PATH=/opt/perl5/bin:$PATH; h2ph -r -l /usr/include || exit -1; # at this point install seems usable lib=$(ls -d /opt/perl5/lib/5* | tail -1); site=$(ls -d /opt/perl5/lib/site_perl/5* | tail -1); cpan='/var/lib/CPAN'; build='/var/tmp/CPAN'; remote='http://mirror.uic.edu/CPAN/'; local="$cpan/sources/"; 

CPAN uses /var/lib, /var/tmp ● Push CPAN config onto stable storage. ● CPAN config is specific to perl version. perl -MCPAN -e 'shell'<

TLC ● Update CPAN. ● Deal with modules that require user input, special choices. ( CPAN_RUN_SHELL_TEST_WITHOUT_EXPECT=1; perl -MCPAN -e 'install CPAN'; perl -MCPAN -e shell <

● Common. ● Smoke modules. ● Iterate them with ${modules[@]}. modules=( YAML::XS JSON::XS CPAN::SQLite IO::Socket::SSL Net::SSL Net::SSLeay HTTP::Date Test::Most Metabase::Resource Module::Version Module::Version Bundle::CPAN Log::Log4perl Bundle::CPAN CPAN::Mini CPAN::Reporter CPAN::Testers Test::RequiresInternet WWW::RobotRules LWP::UserAgent WWW::Mechanize CPAN::Reporter::Smoker Bundle::CPAN::Reporter::Smoker::Tests Bundle::CPANReporter2 App::SmokeBox::Mini App::cpanminus::reporter ); Other modules 

Generate ID file for Reporter ● ~ is /var/lib/CPAN Not /home! ( mkdir ~/.cpantesters; cd ~/.cpantesters; metabase-profile <

Configure CPAN::Reporter # Generate test reports if CPAN::Reporter is installed (yes/no)? [no] yes # Would you like me configure CPAN::Reporter now? [yes] # email_from? [] [email protected] # edit_report? [default:ask/no pass/na:no] # send_report? [default:ask/yes pass/na:yes] # transport? [Metabase uri https://metabase.cpantesters.org/api/v1/ id_file metabase_id.json] # Would you like to run 'metabase-profile' now to create '/root/.cpanreporter/metabase_id.json'? [y] perl -MCPAN -e shell <

Voyeurism ● Fixing the build uses “docker exec”. ● Attach to a running container. ● Does not require ssh. ● Allows fixing, re-running the build. 

Voyeurism ● Fixing the build uses “docker exec”. ● Attach to a running container. ● Does not require ssh. ● Allows fixing, re-running the build. ● Or just watching it: docker exec -it build-perl-5.24.1 'top'; 

Access a running container ● Say you get something like: t/07_plugins.t ........... ok t/08_since_epoch.t ....... Can't locate YAML/Syck.pm in @INC (you may need to install the YAML::Syck module) (@INC contains: /var/tmp/CPAN/minismokebox-0.66-0/blib/lib /var/tmp/CPAN/minismokebox-0.66-0/blib/arch /opt/perl5/lib/site_perl/5.24.1/x86_64- linux /opt/perl5/lib/site_perl/5.24.1 /opt/perl5/lib/5.24.1/x86_64-linux /opt/perl5/lib/5.24.1 .) at t/08_since_epoch.t line 12. BEGIN failed--compilation aborted at t/08_since_epoch.t line 12. # Looks like your test exited with 2 before it could output anything. t/08_since_epoch.t ....... Dubious, test returned 2 (wstat 512, 0x200) Failed 11/11 subtests 

Access a running container ● Say you get something like: ● Q: How do you fix it? t/07_plugins.t ........... ok t/08_since_epoch.t ....... Can't locate YAML/Syck.pm in @INC (you may need to install the YAML::Syck module) (@INC contains: /var/tmp/CPAN/minismokebox-0.66-0/blib/lib /var/tmp/CPAN/minismokebox-0.66-0/blib/arch /opt/perl5/lib/site_perl/5.24.1/x86_64- linux /opt/perl5/lib/site_perl/5.24.1 /opt/perl5/lib/5.24.1/x86_64-linux /opt/perl5/lib/5.24.1 .) at t/08_since_epoch.t line 12. BEGIN failed--compilation aborted at t/08_since_epoch.t line 12. # Looks like your test exited with 2 before it could output anything. t/08_since_epoch.t ....... Dubious, test returned 2 (wstat 512, 0x200) Failed 11/11 subtests 

Access a running container ● Exec into the container. ● Use CPAN to install the module. ● Which leaves you with: docker exec -it build-perl-5.24.1 bash –-login; 

Aside: Container ID ● docker ps shows what is running. Includes the container id & name: $ docker ps; CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 493d9f285faf jekyll:5000/workhorse/gentoo-stage3-amd64 "/bin/bash -c /var..." 28 minutes ago Up 28 minutes build-perl-5.24.1 # same result: $ docker exec -it 493d9f285faf 'bash login'; ‑‑ root@493d9f285faf ~ # $ docker exec -it build-perl-5.4.21 'bash login'; ‑‑ root@493d9f285faf ~ # 

Through the looking glass ● Inside the container you are root. Kernel's proc struct has unchanged EUID. ● Limited view of the system. Only container proc's in “ps” or “top”. Nice on a crowded server. Runlevel restrictions are ineffective! ● Run “perl -MCPAN” or “make install”. Helpful if a non-essential test fails. 

Through the Looking Glass ● Outside docker you have a specific PID & EUID. ● Inside docker the initial process has EUID 0 and PID 1. ● EUID 0 is manageable. ● PID 1 requires that the smoketest proc reap children. Otherwise you end up with defunct proc's. Will eventually eat all proc slots with long-running container. One fix is Docker::Reaper. 

Remember: ● Changes to the image go to a separate AUFS layer. ● They are discarded. ● Fixing files in /etc/ will rescue this build, not the next one. ● Changes to /opt & /var/lib/CPAN are persistent. 

TMTOWTDI: minicpan ● Outside of docker. Maintain CPAN cache in /var/lib. ● Inside docker. Use /var/lib. 

Running repeated jobs ● Docker can automatically restart running jobs. ● docker run -d ‑‑restart=always ... Downside: Lots of proc's started up at once. Issues with tmpfs on system restarts. ● --restart=unless-stopped Allows stopping volumes on the way down. Start singly on way back up. 

Smoketest lifecyle ● Run minicpan once on /var/lib/CPAN. Global cache of modules. ● Run smoker with O/S images & perly vol's. ● This will only test *one* kernel! ● Use VM's running docker for multi-kernel smoke tests. 

Zombie apocolypse ● Container startup command is process #1. ● When it dies the container exits. ● It has to reap all children. Not a big issue for small, single-purpose containers. Big problem for smoketests. 

Fork & Reap ● Parent propagates signals. ● Exits with $? from child process. ● Loop reaps zombies. Without this process slots will be exhausted. If that happens: log on as SU and stop, rm the container. for(;;) { my $pid = wait; $pid != -1 or die "Lost child pid ($child)"; $pid != $child and next; return $? } 

Smoking ● Volumes for perl CPAN /var/tmp ● Run smoker. Never exits. vols=( "-v $perl:/opt:ro" "-v $cpan:$cpan" "-v $tmp:/var/tmp" ); ( $docker volume create \ ... $tmp \ && exec $run \ -t \ --name="$cname" \ ${vols[@]} \ $image \ "/var/lib/CPAN/.cpantesters/docker-smoker" ) 2>&1 | tee $log; 

Propsed change ● Smoker processes recents and exit. Simplifies iterating multiple perl verisons. ● Goal: Product of images, perl vol's. Automate bleed test. 

Summary ● Smoking can be hazardous. ● LXC provides lightweight solution. ● Mix O/S image, perl volume. O/S is fixed. Volume updated to maintain smoker. ● [email protected]:lembark/docker-smoker.git Mostly shell kwikhaks. Working to make it cleaner, perly.