Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Smoke & Mirrors: Containerizing Perl Testing

Smoke & Mirrors: Containerizing Perl Testing

Smoke testing can be hard on the systems performing it -- you never know exactly what's going to happen when you run random code in your environment. Docker provides a useful environment for smoke testing, but requires a specialized installation and configuration if the CPAN module to update and manage the code properly.

This talk describes the operation of smoke testing using docker as a platform and the configuration of docker and Perl used to make it work.

Steven Lembark
PRO

July 09, 2022
Tweet

More Decks by Steven Lembark

Other Decks in Technology

Transcript

  1. Smoke & Mirrors:
    Containerizing Perl Testing
    Steven Lembark
    Workhorse Computing
    [email protected]

    View Slide

  2. What is “Smoking”?

    High-level sanity check, not detailed analysis.
    Does it catch fire when we power it on?

    CPAN runs tests as part of installation.
    We are unusual that way.

    CPAN::Reporter reports results of tests.

    CPAN::Reporter::Smoker builds with reporting.

    View Slide

  3. Why do you smoke?

    Verify personal modules.

    Validate project platform.

    Addict (CPAN smoker).

    View Slide

  4. What do you smoke?

    Whatever is in your sandbox.

    A project and all of its supporting code.

    All of CPAN, or the RECENTS file.

    View Slide

  5. How do you smoke?

    “make -wk -C myproject all test” in your sandbox.

    Nightly build, git hooks, Travis CI.

    CPAN or CPANM smoke testers.

    View Slide

  6. Risks of Smoking

    Pull in arbitrary code.
    Maybe broken.
    Possibly malicious.
    More likely just stupid
    system( rm -rf / *.foobar );

    View Slide

  7. Second-hand Smoke

    Permanent side effects.

    Left over test files.

    Wasted disk space.

    Bandwidth.

    Still possible with personal tests:
    Dependent modules.

    View Slide

  8. Protect Yourself

    Install into a dedicated sandbox.

    Disposable environment.

    View Slide

  9. One approach: Virtual Machines

    Heavy footprint.

    Fully configure a running machine to run a test.

    Inefficient use of hardware.

    View Slide

  10. Lightweigit: “chroot”

    Filesystem isolation only.

    Tests can interfere with network, processes.

    Files pollute permenant storage.

    No resource limits.

    View Slide

  11. Another approach: Process isolation.

    Keep processes from interfering with one another.

    Lightweight: Share logger, daemons, filesytems...

    More efficient use of CPU.

    Less work to set up.

    Not always easy to do right.
    Jails & Zones

    View Slide

  12. Better way: lxc

    Approach via IBM.
    Mainframe LPAR ported to linux.

    Uses process groups.

    Excellent control.

    One problem: People used them like VM's.

    Two problem: lxc has hundreds of switches.

    View Slide

  13. lxc for the Masses

    Docker: 80/20 of lxc.
    Medium isolation.

    Generate an “image”.
    Read-only stack of AUFS mounts.

    Run a “container”.
    Mountpoint + resources + isolation.

    Decent manglement tools.

    View Slide

  14. AUFS with lxc

    Images are layered.
    Read-only.

    "Container layer"
    updated.
    discarded.
    Good and bad:
    No permenant updates.

    View Slide

  15. Not so nice

    Images look good for smoking:
    Read-only underneath: minimal harm.
    Re-cycle O/S layer with Perly build.
    Allows mix'n match testing.

    Catch:
    No fixes or upgrades to smokebox.
    Changes require complete rebuild.

    View Slide

  16. One fix: Volumes

    Described as “data volumes”.

    Can store anything.

    Like Perl, build directory, CPAN mirror.

    Mountpoints and symlinks.
    Dir's or files from O/S into container.

    Mount RO or RW.
    Maintain Perl & Smoker.

    View Slide

  17. Hybrid approach

    O/S is an image.

    Perl, CPAN on a volume.
    O/S tools build and maintain perl.
    perl maintains /var/lib/CPAN

    View Slide

  18. perly volumes

    Tempting: perlbrew dir's as volumes.
    Problems with storage in /home.
    Hard to share across users.
    Storage depends on user account.
    Multiple accounts duplicate space.

    Alternate: Let Docker manage the volumes
    Simpler sharing.
    /var/lib/docker [hopefully] has more space.
    Manage via docker.

    View Slide

  19. Perl Testers Wiki

    Good instructions for CPAN::Reporter::Smoker.

    Start with an O/S image and get a working smoker.

    CPANM has its own version.
    "minismokebox"

    View Slide

  20. Grab a distro

    Docker Hub has hundreds of distro images.

    I'll use Gentoo here: it includes build environment.
    No need to install gcc, make...
    “stage3” is the starting point.

    View Slide

  21. Installing Stage3

    docker import $tarball into gentoo-stage3:YYYYMMDD.

    Layer custom files into smoker-gentoo-stage3:lYYYYMMDD.
    root's .exrc .bashrc

    Used to bulid perl, run smoke.

    View Slide

  22. Building perl, smoker

    Volumes for /opt, /var/tmp & script files.
    Standard paths simplify build scripts.

    “docker run” with command of build script.
    build perl.
    install & run minicpan.
    configure reporting & smoker.
    CPAN update to test reporting.

    View Slide

  23. Aside: --tmpfs

    docker run –tmpfs has a problem:
    The volume has execution turned off.
    Fine for data.
    Useless for build.

    Avoid the issue with a tmpfs volume.
    Volume must be private to build.

    View Slide

  24. Creating a tmpfs

    Private tmp
    volume via PID.

    tmpfs usually
    cleaned up
    automatically on
    the way out.

    Use volume rm
    to be sure.
    tmp=”var-tmp-$BASHPID”;
    $docker volume create
    --driver=local
    --opt type=tmpfs
    --opt device=tmpfs
    --opt o=size=320M
    $tmp
    &&
    docker run
    ...
    -v $tmp:/var/tmp
    ...
    ;
    # may fail due to non-existant volume.
    docker volume rm $tmp;

    View Slide

  25. Volumes of Volumes

    Temp volume -v $tmp:/var/tmp

    Perl tarball -v $PWD/perl‑5.24.1.tar.gz:/var/tmp/perl.tar.gz

    Build script -v $PWD/build-perl:/var/tmp/build-perl

    Perl install -v opt-perl-5.24.1:/opt

    CPAN cache -v /var/lib/CPAN:/var/lib/CPAN

    View Slide

  26. Volumes of Volumes

    Temp volume -v $tmp:/var/tmp

    Perl tarball -v $PWD/perl‑5.24.1.tar.gz:/var/tmp/perl.tar.gz

    Build script -v $PWD/build-perl:/var/tmp/build-perl

    Perl install -v opt-perl-5.24.1:/opt

    CPAN cache -v /var/lib/CPAN:/var/lib/CPAN

    View Slide

  27. Volumes of Volumes

    Temp volume -v $tmp:/var/tmp

    Perl tarball -v $PWD/perl‑5.24.1.tar.gz:/var/tmp/perl.tar.gz

    Build script -v $PWD/build-perl:/var/tmp/build-perl

    Perl install -v opt-perl-5.24.1:/opt

    CPAN cache -v /var/lib/CPAN:/var/lib/CPAN

    View Slide

  28. Volumes of Volumes

    Temp volume -v $tmp:/var/tmp

    Perl tarball -v $PWD/perl‑5.24.1.tar.gz:/var/tmp/perl.tar.gz

    Build script -v $PWD/build-perl:/var/tmp/build-perl

    Perl install -v opt-perl-5.24.1:/opt

    CPAN cache -v /var/lib/CPAN:/var/lib/CPAN

    View Slide

  29. Start the ball rolling

    Assemble a container.

    Dispatch the build script.

    “-t” avoids issues with
    core test that uses
    column/row counts.

    “-a” gets output from
    container into log.
    $docker volume create \
    --driver=local \
    --opt type=tmpfs \
    --opt device=tmpfs \
    --opt o=size=320M \
    $tmp &&
    $docker volume create \
    --driver=local \
    $perl &&
    $docker run --rm \
    -t \
    --name=”build-$perl”\
    -a stdout \
    -a stderr \
    "${vols[@]}" \
    $image \
    /var/tmp/build-perl \
    2>&1 | tee build.out;
    docker volume rm $tmp;

    View Slide

  30. Building Perl

    The fun starts in /var/tmp/build-perl

    Mostly here-scripts.

    Install & configure smoketest engine.

    Includes common non-smoke modules.

    View Slide

  31. Building Perl
    HOME='/var/lib/CPAN';
    (
    cd /var/tmp;
    gzip -dc < perl.tar.gz | tar xf -;
    cd perl*;
    jobs=$(grep '^processor' /proc/cpuinfo | wc -l);
    export TEST_JOBS=$jobs;
    ./Configure \
    -de \
    -Dprefix=/opt/perl5 \
    -Dman1dir='none' \
    -Dman3dir='none' \
    -Doptimize='-O2 -pipe -march=native' \
    &&
    /usr/bin/make -j$jobs all test_harness install;
    ) ||
    exit -1;
    rm -rf /var/tmp/perl*;

    Generic
    “perl.tar.gz” via
    docker volume
    mapping.

    /opt/perl5 is in
    external
    volume.

    Don't knead
    man pages.

    View Slide

  32. Building Perl
    HOME='/var/lib/CPAN';

    /home is in the
    O/S image!
    Changes in
    container layer
    are discarded!!

    Persistent files
    go in
    /var/lib/CPAN or
    /opt.

    View Slide

  33. Building Perl
    TEST_JOBS=$jobs ... /usr/bin/make -j$jobs all test_harness
    install;
    Files=2410, Tests=851044, 223 wallclock secs
    (172.97 usr 22.15 sys + 862.57 cusr 57.18 csys = 1114.87 CPU)

    View Slide

  34. Aside: Testing perl builds

    Use tmpfs for /opt instead of disk volume.
    Fast, disposable storage for validating your build scripts.

    git clone /scratch;
    cd /scratch/whatever;

    docker run … -v /scratch/whatever:/var/tmp/whatever … ;

    build script does cd /var/tmp/whatever; make all test install;

    View Slide

  35. Testing bleeding edge perl

    Replace tar xvf with volume:
    -v /gitclone/perl-5.25.X:/var/tmp/perl

    Rest of it stays the same.
    Configure and make run quickly.
    Minimal overhead if core modules don't change.
    Lots “up to date” messages from CPAN.

    View Slide

  36. Minor setups

    “h2ph”
    validates
    $PATH.

    Pick your
    remote
    server.
    export PATH=/opt/perl5/bin:$PATH;
    h2ph -r -l /usr/include || exit -1;
    # at this point install seems usable
    lib=$(ls -d /opt/perl5/lib/5* | tail -1);
    site=$(ls -d /opt/perl5/lib/site_perl/5* | tail -1);
    cpan='/var/lib/CPAN';
    build='/var/tmp/CPAN';
    remote='http://mirror.uic.edu/CPAN/';
    local="$cpan/sources/";

    View Slide

  37. CPAN uses /var/lib, /var/tmp

    Push CPAN config onto
    stable storage.

    CPAN config is specific
    to perl version.
    perl -MCPAN -e 'shell'<yes
    o conf cpan_home $cpan
    o conf histfile $cpan/histfile
    o conf prefs_dir $cpan/prefs
    o conf build_dir $build
    o conf keep_source_where $local
    o conf auto_commit yes
    o conf build_dir_reuse yes
    o conf check_sigs yes
    o conf inhibit_startup_message yes
    o conf halt_on_failure no
    o conf build_cache 1
    o conf index_expire 1
    o conf commit
    CPAN

    View Slide

  38. TLC

    Update CPAN.

    Deal with modules
    that require user
    input, special
    choices.
    (
    CPAN_RUN_SHELL_TEST_WITHOUT_EXPECT=1;
    perl -MCPAN -e 'install CPAN';
    perl -MCPAN -e shell <install Module::Signature
    2
    CPAN
    AUTOMATED_TESTING=1; # avoid interactive prompt
    perl -MCPAN -e 'shell <install Term::Readline::Perl
    CPAN
    )

    View Slide


  39. Common.

    Smoke modules.

    Iterate them with
    ${modules[@]}.
    modules=(
    YAML::XS
    JSON::XS
    CPAN::SQLite
    IO::Socket::SSL
    Net::SSL
    Net::SSLeay
    HTTP::Date
    Test::Most
    Metabase::Resource
    Module::Version
    Module::Version Bundle::CPAN
    Log::Log4perl
    Bundle::CPAN
    CPAN::Mini
    CPAN::Reporter
    CPAN::Testers
    Test::RequiresInternet
    WWW::RobotRules
    LWP::UserAgent
    WWW::Mechanize
    CPAN::Reporter::Smoker
    Bundle::CPAN::Reporter::Smoker::Tests
    Bundle::CPANReporter2
    App::SmokeBox::Mini
    App::cpanminus::reporter
    );
    Other modules

    View Slide

  40. Generate ID file for Reporter

    ~ is /var/lib/CPAN
    Not /home!
    (
    mkdir ~/.cpantesters;
    cd ~/.cpantesters;
    metabase-profile <Steven Lembark
    [email protected]
    password/secret
    END
    chmod 0400 metabase_id.json;
    )

    View Slide

  41. Configure CPAN::Reporter
    # Generate test reports if CPAN::Reporter is installed (yes/no)? [no] yes
    # Would you like me configure CPAN::Reporter now? [yes]
    # email_from? [] [email protected]
    # edit_report? [default:ask/no pass/na:no]
    # send_report? [default:ask/yes pass/na:yes]
    # transport? [Metabase uri https://metabase.cpantesters.org/api/v1/ id_file metabase_id.json]
    # Would you like to run 'metabase-profile' now to create '/root/.cpanreporter/metabase_id.json'? [y]
    perl -MCPAN -e shell <o conf init test_report
    yes
    yes
    [email protected]
    no
    yes
    CPAN

    View Slide

  42. Voyeurism

    Fixing the build uses “docker exec”.

    Attach to a running container.

    Does not require ssh.

    Allows fixing, re-running the build.

    View Slide

  43. Voyeurism

    Fixing the build uses “docker exec”.

    Attach to a running container.

    Does not require ssh.

    Allows fixing, re-running the build.

    Or just watching it:
    docker exec -it build-perl-5.24.1 'top';

    View Slide

  44. Access a running container

    Say you get something like:
    t/07_plugins.t ........... ok
    t/08_since_epoch.t ....... Can't locate YAML/Syck.pm in @INC (you may need to install
    the YAML::Syck module) (@INC contains: /var/tmp/CPAN/minismokebox-0.66-0/blib/lib
    /var/tmp/CPAN/minismokebox-0.66-0/blib/arch /opt/perl5/lib/site_perl/5.24.1/x86_64-
    linux /opt/perl5/lib/site_perl/5.24.1 /opt/perl5/lib/5.24.1/x86_64-linux /opt/perl5/lib/5.24.1 .)
    at t/08_since_epoch.t line 12.
    BEGIN failed--compilation aborted at t/08_since_epoch.t line 12.
    # Looks like your test exited with 2 before it could output anything.
    t/08_since_epoch.t ....... Dubious, test returned 2 (wstat 512, 0x200)
    Failed 11/11 subtests

    View Slide

  45. Access a running container

    Say you get something like:

    Q: How do you fix it?
    t/07_plugins.t ........... ok
    t/08_since_epoch.t ....... Can't locate YAML/Syck.pm in @INC (you may need to install
    the YAML::Syck module) (@INC contains: /var/tmp/CPAN/minismokebox-0.66-0/blib/lib
    /var/tmp/CPAN/minismokebox-0.66-0/blib/arch /opt/perl5/lib/site_perl/5.24.1/x86_64-
    linux /opt/perl5/lib/site_perl/5.24.1 /opt/perl5/lib/5.24.1/x86_64-linux /opt/perl5/lib/5.24.1 .)
    at t/08_since_epoch.t line 12.
    BEGIN failed--compilation aborted at t/08_since_epoch.t line 12.
    # Looks like your test exited with 2 before it could output anything.
    t/08_since_epoch.t ....... Dubious, test returned 2 (wstat 512, 0x200)
    Failed 11/11 subtests

    View Slide

  46. Access a running container

    Exec into the container.

    Use CPAN to install the module.

    Which leaves you with:
    docker exec -it build-perl-5.24.1 bash –-login;

    View Slide

  47. Aside: Container ID

    docker ps shows what is running.
    Includes the container id & name:
    $ docker ps;
    CONTAINER ID IMAGE COMMAND
    CREATED STATUS PORTS NAMES
    493d9f285faf jekyll:5000/workhorse/gentoo-stage3-amd64 "/bin/bash -c /var..."
    28 minutes ago Up 28 minutes build-perl-5.24.1
    # same result:
    $ docker exec -it 493d9f285faf 'bash login';
    ‑‑
    [email protected] ~ #
    $ docker exec -it build-perl-5.4.21 'bash login';
    ‑‑
    [email protected] ~ #

    View Slide

  48. Through the looking glass

    Inside the container you are root.
    Kernel's proc struct has unchanged EUID.

    Limited view of the system.
    Only container proc's in “ps” or “top”.
    Nice on a crowded server.
    Runlevel restrictions are ineffective!

    Run “perl -MCPAN” or “make install”.
    Helpful if a non-essential test fails.

    View Slide

  49. Through the Looking Glass

    Outside docker you have a specific PID & EUID.

    Inside docker the initial process has EUID 0 and PID 1.

    EUID 0 is manageable.

    PID 1 requires that the smoketest proc reap children.
    Otherwise you end up with defunct proc's.
    Will eventually eat all proc slots with long-running container.
    One fix is Docker::Reaper.

    View Slide

  50. Remember:

    Changes to the image go to a separate AUFS layer.

    They are discarded.

    Fixing files in /etc/ will rescue this build, not the next one.

    Changes to /opt & /var/lib/CPAN are persistent.

    View Slide

  51. TMTOWTDI: minicpan

    Outside of docker.
    Maintain CPAN cache in /var/lib.

    Inside docker.
    Use /var/lib.

    View Slide

  52. Running repeated jobs

    Docker can automatically restart running jobs.

    docker run -d ‑‑restart=always ...
    Downside: Lots of proc's started up at once.
    Issues with tmpfs on system restarts.

    --restart=unless-stopped
    Allows stopping volumes on the way down.
    Start singly on way back up.

    View Slide

  53. Smoketest lifecyle

    Run minicpan once on /var/lib/CPAN.
    Global cache of modules.

    Run smoker with O/S images & perly vol's.

    This will only test *one* kernel!

    Use VM's running docker for multi-kernel smoke tests.

    View Slide

  54. Zombie apocolypse

    Container startup command is process #1.

    When it dies the container exits.

    It has to reap all children.
    Not a big issue for small, single-purpose containers.
    Big problem for smoketests.

    View Slide

  55. Fork & Reap

    Parent propagates signals.

    Exits with $? from child process.

    Loop reaps zombies.
    Without this process slots will be
    exhausted.
    If that happens: log on as SU
    and stop, rm the container.
    for(;;)
    {
    my $pid = wait;
    $pid != -1
    or die "Lost child pid ($child)";
    $pid != $child
    and next;
    return $?
    }

    View Slide

  56. Smoking

    Volumes for
    perl
    CPAN
    /var/tmp

    Run smoker.
    Never exits.
    vols=(
    "-v $perl:/opt:ro"
    "-v $cpan:$cpan"
    "-v $tmp:/var/tmp"
    );
    (
    $docker volume create \
    ...
    $tmp \
    &&
    exec $run \
    -t \
    --name="$cname" \
    ${vols[@]} \
    $image \
    "/var/lib/CPAN/.cpantesters/docker-smoker"
    ) 2>&1 |
    tee $log;

    View Slide

  57. Propsed change

    Smoker processes recents and exit.
    Simplifies iterating multiple perl verisons.

    Goal:
    Product of images, perl vol's.
    Automate bleed test.

    View Slide

  58. Summary

    Smoking can be hazardous.

    LXC provides lightweight solution.

    Mix O/S image, perl volume.
    O/S is fixed.
    Volume updated to maintain smoker.

    [email protected]:lembark/docker-smoker.git
    Mostly shell kwikhaks.
    Working to make it cleaner, perly.

    View Slide