Smoke & Mirrors: Containerizing Perl Testing

Smoke testing can be hard on the systems performing it -- you never know exactly what's going to happen when you run random code in your environment. Docker provides a useful environment for smoke testing, but requires a specialized installation and configuration if the CPAN module to update and manage the code properly.

This talk describes the operation of smoke testing using docker as a platform and the configuration of docker and Perl used to make it work.

Transcript

1. Smoke & Mirrors:
   Containerizing Perl Testing
   Steven Lembark
   Workhorse Computing
   lembark@wrkhors.com
   

   View Slide

2. What is “Smoking”?
   ●
   High-level sanity check, not detailed analysis.
   Does it catch fire when we power it on?
   ●
   CPAN runs tests as part of installation.
   We are unusual that way.
   ●
   CPAN::Reporter reports results of tests.
   ●
   CPAN::Reporter::Smoker builds with reporting.
   

   View Slide

3. Why do you smoke?
   ●
   Verify personal modules.
   ●
   Validate project platform.
   ●
   Addict (CPAN smoker).
   

   View Slide

4. What do you smoke?
   ●
   Whatever is in your sandbox.
   ●
   A project and all of its supporting code.
   ●
   All of CPAN, or the RECENTS file.
   

   View Slide

5. How do you smoke?
   ●
   “make -wk -C myproject all test” in your sandbox.
   ●
   Nightly build, git hooks, Travis CI.
   ●
   CPAN or CPANM smoke testers.
   

   View Slide

6. Risks of Smoking
   ●
   Pull in arbitrary code.
   Maybe broken.
   Possibly malicious.
   More likely just stupid
   system( rm -rf / *.foobar );
   

   View Slide

7. Second-hand Smoke
   ●
   Permanent side effects.
   ●
   Left over test files.
   ●
   Wasted disk space.
   ●
   Bandwidth.
   ●
   Still possible with personal tests:
   Dependent modules.
   

   View Slide

8. Protect Yourself
   ●
   Install into a dedicated sandbox.
   ●
   Disposable environment.
   

   View Slide

9. One approach: Virtual Machines
   ●
   Heavy footprint.
   ●
   Fully configure a running machine to run a test.
   ●
   Inefficient use of hardware.
   

   View Slide

10. Lightweigit: “chroot”
    ●
    Filesystem isolation only.
    ●
    Tests can interfere with network, processes.
    ●
    Files pollute permenant storage.
    ●
    No resource limits.
    

    View Slide

11. Another approach: Process isolation.
    ●
    Keep processes from interfering with one another.
    ●
    Lightweight: Share logger, daemons, filesytems...
    ●
    More efficient use of CPU.
    ●
    Less work to set up.
    ●
    Not always easy to do right.
    Jails & Zones
    

    View Slide

12. Better way: lxc
    ●
    Approach via IBM.
    Mainframe LPAR ported to linux.
    ●
    Uses process groups.
    ●
    Excellent control.
    ●
    One problem: People used them like VM's.
    ●
    Two problem: lxc has hundreds of switches.
    

    View Slide

13. lxc for the Masses
    ●
    Docker: 80/20 of lxc.
    Medium isolation.
    ●
    Generate an “image”.
    Read-only stack of AUFS mounts.
    ●
    Run a “container”.
    Mountpoint + resources + isolation.
    ●
    Decent manglement tools.
    

    View Slide

14. AUFS with lxc
    ●
    Images are layered.
    Read-only.
    ●
    "Container layer"
    updated.
    discarded.
    Good and bad:
    No permenant updates.
    

    View Slide

15. Not so nice
    ●
    Images look good for smoking:
    Read-only underneath: minimal harm.
    Re-cycle O/S layer with Perly build.
    Allows mix'n match testing.
    ●
    Catch:
    No fixes or upgrades to smokebox.
    Changes require complete rebuild.
    

    View Slide

16. One fix: Volumes
    ●
    Described as “data volumes”.
    ●
    Can store anything.
    ●
    Like Perl, build directory, CPAN mirror.
    ●
    Mountpoints and symlinks.
    Dir's or files from O/S into container.
    ●
    Mount RO or RW.
    Maintain Perl & Smoker.
    

    View Slide

17. Hybrid approach
    ●
    O/S is an image.
    ●
    Perl, CPAN on a volume.
    O/S tools build and maintain perl.
    perl maintains /var/lib/CPAN
    

    View Slide

18. perly volumes
    ●
    Tempting: perlbrew dir's as volumes.
    Problems with storage in /home.
    Hard to share across users.
    Storage depends on user account.
    Multiple accounts duplicate space.
    ●
    Alternate: Let Docker manage the volumes
    Simpler sharing.
    /var/lib/docker [hopefully] has more space.
    Manage via docker.
    

    View Slide

19. Perl Testers Wiki
    ●
    Good instructions for CPAN::Reporter::Smoker.
    ●
    Start with an O/S image and get a working smoker.
    ●
    CPANM has its own version.
    "minismokebox"
    

    View Slide

20. Grab a distro
    ●
    Docker Hub has hundreds of distro images.
    ●
    I'll use Gentoo here: it includes build environment.
    No need to install gcc, make...
    “stage3” is the starting point.
    

    View Slide

21. Installing Stage3
    ●
    docker import $tarball into gentoo-stage3:YYYYMMDD.
    ●
    Layer custom files into smoker-gentoo-stage3:lYYYYMMDD.
    root's .exrc .bashrc
    ●
    Used to bulid perl, run smoke.
    

    View Slide

22. Building perl, smoker
    ●
    Volumes for /opt, /var/tmp & script files.
    Standard paths simplify build scripts.
    ●
    “docker run” with command of build script.
    build perl.
    install & run minicpan.
    configure reporting & smoker.
    CPAN update to test reporting.
    

    View Slide

23. Aside: --tmpfs
    ●
    docker run –tmpfs has a problem:
    The volume has execution turned off.
    Fine for data.
    Useless for build.
    ●
    Avoid the issue with a tmpfs volume.
    Volume must be private to build.
    

    View Slide

24. Creating a tmpfs
    ●
    Private tmp
    volume via PID.
    ●
    tmpfs usually
    cleaned up
    automatically on
    the way out.
    ●
    Use volume rm
    to be sure.
    tmp=”var-tmp-$BASHPID”;
    $docker volume create
    --driver=local
    --opt type=tmpfs
    --opt device=tmpfs
    --opt o=size=320M
    $tmp
    &&
    docker run
    ...
    -v $tmp:/var/tmp
    ...
    ;
    # may fail due to non-existant volume.
    docker volume rm $tmp;
    

    View Slide

25. Volumes of Volumes
    ●
    Temp volume -v $tmp:/var/tmp
    ●
    Perl tarball -v $PWD/perl‑5.24.1.tar.gz:/var/tmp/perl.tar.gz
    ●
    Build script -v $PWD/build-perl:/var/tmp/build-perl
    ●
    Perl install -v opt-perl-5.24.1:/opt
    ●
    CPAN cache -v /var/lib/CPAN:/var/lib/CPAN
    

    View Slide

26. Volumes of Volumes
    ●
    Temp volume -v $tmp:/var/tmp
    ●
    Perl tarball -v $PWD/perl‑5.24.1.tar.gz:/var/tmp/perl.tar.gz
    ●
    Build script -v $PWD/build-perl:/var/tmp/build-perl
    ●
    Perl install -v opt-perl-5.24.1:/opt
    ●
    CPAN cache -v /var/lib/CPAN:/var/lib/CPAN
    

    View Slide

27. Volumes of Volumes
    ●
    Temp volume -v $tmp:/var/tmp
    ●
    Perl tarball -v $PWD/perl‑5.24.1.tar.gz:/var/tmp/perl.tar.gz
    ●
    Build script -v $PWD/build-perl:/var/tmp/build-perl
    ●
    Perl install -v opt-perl-5.24.1:/opt
    ●
    CPAN cache -v /var/lib/CPAN:/var/lib/CPAN
    

    View Slide

28. Volumes of Volumes
    ●
    Temp volume -v $tmp:/var/tmp
    ●
    Perl tarball -v $PWD/perl‑5.24.1.tar.gz:/var/tmp/perl.tar.gz
    ●
    Build script -v $PWD/build-perl:/var/tmp/build-perl
    ●
    Perl install -v opt-perl-5.24.1:/opt
    ●
    CPAN cache -v /var/lib/CPAN:/var/lib/CPAN
    

    View Slide

29. Start the ball rolling
    ●
    Assemble a container.
    ●
    Dispatch the build script.
    ●
    “-t” avoids issues with
    core test that uses
    column/row counts.
    ●
    “-a” gets output from
    container into log.
    $docker volume create \
    --driver=local \
    --opt type=tmpfs \
    --opt device=tmpfs \
    --opt o=size=320M \
    $tmp &&
    $docker volume create \
    --driver=local \
    $perl &&
    $docker run --rm \
    -t \
    --name=”build-$perl”\
    -a stdout \
    -a stderr \
    "${vols[@]}" \
    $image \
    /var/tmp/build-perl \
    2>&1 | tee build.out;
    docker volume rm $tmp;
    

    View Slide

30. Building Perl
    ●
    The fun starts in /var/tmp/build-perl
    ●
    Mostly here-scripts.
    ●
    Install & configure smoketest engine.
    ●
    Includes common non-smoke modules.
    

    View Slide

31. Building Perl
    HOME='/var/lib/CPAN';
    (
    cd /var/tmp;
    gzip -dc < perl.tar.gz | tar xf -;
    cd perl*;
    jobs=$(grep '^processor' /proc/cpuinfo | wc -l);
    export TEST_JOBS=$jobs;
    ./Configure \
    -de \
    -Dprefix=/opt/perl5 \
    -Dman1dir='none' \
    -Dman3dir='none' \
    -Doptimize='-O2 -pipe -march=native' \
    &&
    /usr/bin/make -j$jobs all test_harness install;
    ) ||
    exit -1;
    rm -rf /var/tmp/perl*;
    ●
    Generic
    “perl.tar.gz” via
    docker volume
    mapping.
    ●
    /opt/perl5 is in
    external
    volume.
    ●
    Don't knead
    man pages.
    

    View Slide

32. Building Perl
    HOME='/var/lib/CPAN';
    ●
    /home is in the
    O/S image!
    Changes in
    container layer
    are discarded!!
    ●
    Persistent files
    go in
    /var/lib/CPAN or
    /opt.
    

    View Slide

33. Building Perl
    TEST_JOBS=$jobs ... /usr/bin/make -j$jobs all test_harness
    install;
    Files=2410, Tests=851044, 223 wallclock secs
    (172.97 usr 22.15 sys + 862.57 cusr 57.18 csys = 1114.87 CPU)
    

    View Slide

34. Aside: Testing perl builds
    ●
    Use tmpfs for /opt instead of disk volume.
    Fast, disposable storage for validating your build scripts.
    ●
    git clone /scratch;
    cd /scratch/whatever;
    ●
    docker run … -v /scratch/whatever:/var/tmp/whatever … ;
    ●
    build script does cd /var/tmp/whatever; make all test install;
    

    View Slide

35. Testing bleeding edge perl
    ●
    Replace tar xvf with volume:
    -v /gitclone/perl-5.25.X:/var/tmp/perl
    ●
    Rest of it stays the same.
    Configure and make run quickly.
    Minimal overhead if core modules don't change.
    Lots “up to date” messages from CPAN.
    

    View Slide

36. Minor setups
    ●
    “h2ph”
    validates
    $PATH.
    ●
    Pick your
    remote
    server.
    export PATH=/opt/perl5/bin:$PATH;
    h2ph -r -l /usr/include || exit -1;
    # at this point install seems usable
    lib=$(ls -d /opt/perl5/lib/5* | tail -1);
    site=$(ls -d /opt/perl5/lib/site_perl/5* | tail -1);
    cpan='/var/lib/CPAN';
    build='/var/tmp/CPAN';
    remote='http://mirror.uic.edu/CPAN/';
    local="$cpan/sources/";
    

    View Slide

37. CPAN uses /var/lib, /var/tmp
    ●
    Push CPAN config onto
    stable storage.
    ●
    CPAN config is specific
    to perl version.
    perl -MCPAN -e 'shell'<yes
    o conf cpan_home $cpan
    o conf histfile $cpan/histfile
    o conf prefs_dir $cpan/prefs
    o conf build_dir $build
    o conf keep_source_where $local
    o conf auto_commit yes
    o conf build_dir_reuse yes
    o conf check_sigs yes
    o conf inhibit_startup_message yes
    o conf halt_on_failure no
    o conf build_cache 1
    o conf index_expire 1
    o conf commit
    CPAN
    

    View Slide

38. TLC
    ●
    Update CPAN.
    ●
    Deal with modules
    that require user
    input, special
    choices.
    (
    CPAN_RUN_SHELL_TEST_WITHOUT_EXPECT=1;
    perl -MCPAN -e 'install CPAN';
    perl -MCPAN -e shell <install Module::Signature
    2
    CPAN
    AUTOMATED_TESTING=1; # avoid interactive prompt
    perl -MCPAN -e 'shell <install Term::Readline::Perl
    CPAN
    )
    

    View Slide

39. ●
    Common.
    ●
    Smoke modules.
    ●
    Iterate them with
    ${modules[@]}.
    modules=(
    YAML::XS
    JSON::XS
    CPAN::SQLite
    IO::Socket::SSL
    Net::SSL
    Net::SSLeay
    HTTP::Date
    Test::Most
    Metabase::Resource
    Module::Version
    Module::Version Bundle::CPAN
    Log::Log4perl
    Bundle::CPAN
    CPAN::Mini
    CPAN::Reporter
    CPAN::Testers
    Test::RequiresInternet
    WWW::RobotRules
    LWP::UserAgent
    WWW::Mechanize
    CPAN::Reporter::Smoker
    Bundle::CPAN::Reporter::Smoker::Tests
    Bundle::CPANReporter2
    App::SmokeBox::Mini
    App::cpanminus::reporter
    );
    Other modules
    

    View Slide

40. Generate ID file for Reporter
    ●
    ~ is /var/lib/CPAN
    Not /home!
    (
    mkdir ~/.cpantesters;
    cd ~/.cpantesters;
    metabase-profile <Steven Lembark
    lembark@wrkhors.com
    password/secret
    END
    chmod 0400 metabase_id.json;
    )
    

    View Slide

41. Configure CPAN::Reporter
    # Generate test reports if CPAN::Reporter is installed (yes/no)? [no] yes
    # Would you like me configure CPAN::Reporter now? [yes]
    # email_from? [] example@example.com
    # edit_report? [default:ask/no pass/na:no]
    # send_report? [default:ask/yes pass/na:yes]
    # transport? [Metabase uri https://metabase.cpantesters.org/api/v1/ id_file metabase_id.json]
    # Would you like to run 'metabase-profile' now to create '/root/.cpanreporter/metabase_id.json'? [y]
    perl -MCPAN -e shell <o conf init test_report
    yes
    yes
    lembark@wrkhors.com
    no
    yes
    CPAN
    

    View Slide

42. Voyeurism
    ●
    Fixing the build uses “docker exec”.
    ●
    Attach to a running container.
    ●
    Does not require ssh.
    ●
    Allows fixing, re-running the build.
    

    View Slide

43. Voyeurism
    ●
    Fixing the build uses “docker exec”.
    ●
    Attach to a running container.
    ●
    Does not require ssh.
    ●
    Allows fixing, re-running the build.
    ●
    Or just watching it:
    docker exec -it build-perl-5.24.1 'top';
    

    View Slide

44. Access a running container
    ●
    Say you get something like:
    t/07_plugins.t ........... ok
    t/08_since_epoch.t ....... Can't locate YAML/Syck.pm in @INC (you may need to install
    the YAML::Syck module) (@INC contains: /var/tmp/CPAN/minismokebox-0.66-0/blib/lib
    /var/tmp/CPAN/minismokebox-0.66-0/blib/arch /opt/perl5/lib/site_perl/5.24.1/x86_64-
    linux /opt/perl5/lib/site_perl/5.24.1 /opt/perl5/lib/5.24.1/x86_64-linux /opt/perl5/lib/5.24.1 .)
    at t/08_since_epoch.t line 12.
    BEGIN failed--compilation aborted at t/08_since_epoch.t line 12.
    # Looks like your test exited with 2 before it could output anything.
    t/08_since_epoch.t ....... Dubious, test returned 2 (wstat 512, 0x200)
    Failed 11/11 subtests
    

    View Slide

45. Access a running container
    ●
    Say you get something like:
    ●
    Q: How do you fix it?
    t/07_plugins.t ........... ok
    t/08_since_epoch.t ....... Can't locate YAML/Syck.pm in @INC (you may need to install
    the YAML::Syck module) (@INC contains: /var/tmp/CPAN/minismokebox-0.66-0/blib/lib
    /var/tmp/CPAN/minismokebox-0.66-0/blib/arch /opt/perl5/lib/site_perl/5.24.1/x86_64-
    linux /opt/perl5/lib/site_perl/5.24.1 /opt/perl5/lib/5.24.1/x86_64-linux /opt/perl5/lib/5.24.1 .)
    at t/08_since_epoch.t line 12.
    BEGIN failed--compilation aborted at t/08_since_epoch.t line 12.
    # Looks like your test exited with 2 before it could output anything.
    t/08_since_epoch.t ....... Dubious, test returned 2 (wstat 512, 0x200)
    Failed 11/11 subtests
    

    View Slide

46. Access a running container
    ●
    Exec into the container.
    ●
    Use CPAN to install the module.
    ●
    Which leaves you with:
    docker exec -it build-perl-5.24.1 bash –-login;
    

    View Slide

47. Aside: Container ID
    ●
    docker ps shows what is running.
    Includes the container id & name:
    $ docker ps;
    CONTAINER ID IMAGE COMMAND
    CREATED STATUS PORTS NAMES
    493d9f285faf jekyll:5000/workhorse/gentoo-stage3-amd64 "/bin/bash -c /var..."
    28 minutes ago Up 28 minutes build-perl-5.24.1
    # same result:
    $ docker exec -it 493d9f285faf 'bash login';
    ‑‑
    root@493d9f285faf ~ #
    $ docker exec -it build-perl-5.4.21 'bash login';
    ‑‑
    root@493d9f285faf ~ #
    

    View Slide

48. Through the looking glass
    ●
    Inside the container you are root.
    Kernel's proc struct has unchanged EUID.
    ●
    Limited view of the system.
    Only container proc's in “ps” or “top”.
    Nice on a crowded server.
    Runlevel restrictions are ineffective!
    ●
    Run “perl -MCPAN” or “make install”.
    Helpful if a non-essential test fails.
    

    View Slide

49. Through the Looking Glass
    ●
    Outside docker you have a specific PID & EUID.
    ●
    Inside docker the initial process has EUID 0 and PID 1.
    ●
    EUID 0 is manageable.
    ●
    PID 1 requires that the smoketest proc reap children.
    Otherwise you end up with defunct proc's.
    Will eventually eat all proc slots with long-running container.
    One fix is Docker::Reaper.
    

    View Slide

50. Remember:
    ●
    Changes to the image go to a separate AUFS layer.
    ●
    They are discarded.
    ●
    Fixing files in /etc/ will rescue this build, not the next one.
    ●
    Changes to /opt & /var/lib/CPAN are persistent.
    

    View Slide

51. TMTOWTDI: minicpan
    ●
    Outside of docker.
    Maintain CPAN cache in /var/lib.
    ●
    Inside docker.
    Use /var/lib.
    

    View Slide

52. Running repeated jobs
    ●
    Docker can automatically restart running jobs.
    ●
    docker run -d ‑‑restart=always ...
    Downside: Lots of proc's started up at once.
    Issues with tmpfs on system restarts.
    ●
    --restart=unless-stopped
    Allows stopping volumes on the way down.
    Start singly on way back up.
    

    View Slide

53. Smoketest lifecyle
    ●
    Run minicpan once on /var/lib/CPAN.
    Global cache of modules.
    ●
    Run smoker with O/S images & perly vol's.
    ●
    This will only test *one* kernel!
    ●
    Use VM's running docker for multi-kernel smoke tests.
    

    View Slide

54. Zombie apocolypse
    ●
    Container startup command is process #1.
    ●
    When it dies the container exits.
    ●
    It has to reap all children.
    Not a big issue for small, single-purpose containers.
    Big problem for smoketests.
    

    View Slide

55. Fork & Reap
    ●
    Parent propagates signals.
    ●
    Exits with $? from child process.
    ●
    Loop reaps zombies.
    Without this process slots will be
    exhausted.
    If that happens: log on as SU
    and stop, rm the container.
    for(;;)
    {
    my $pid = wait;
    $pid != -1
    or die "Lost child pid ($child)";
    $pid != $child
    and next;
    return $?
    }
    

    View Slide

56. Smoking
    ●
    Volumes for
    perl
    CPAN
    /var/tmp
    ●
    Run smoker.
    Never exits.
    vols=(
    "-v $perl:/opt:ro"
    "-v $cpan:$cpan"
    "-v $tmp:/var/tmp"
    );
    (
    $docker volume create \
    ...
    $tmp \
    &&
    exec $run \
    -t \
    --name="$cname" \
    ${vols[@]} \
    $image \
    "/var/lib/CPAN/.cpantesters/docker-smoker"
    ) 2>&1 |
    tee $log;
    

    View Slide

57. Propsed change
    ●
    Smoker processes recents and exit.
    Simplifies iterating multiple perl verisons.
    ●
    Goal:
    Product of images, perl vol's.
    Automate bleed test.
    

    View Slide

58. Summary
    ●
    Smoking can be hazardous.
    ●
    LXC provides lightweight solution.
    ●
    Mix O/S image, perl volume.
    O/S is fixed.
    Volume updated to maintain smoker.
    ●
    git@github.com:lembark/docker-smoker.git
    Mostly shell kwikhaks.
    Working to make it cleaner, perly.
    

    View Slide