Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Test your Docker images with Python by Jamie He...

Pycon ZA
October 11, 2018

Test your Docker images with Python by Jamie Hewland

As more and more software is packaged in Docker images, it has become increasingly important that the Dockerfiles and scripts that these images are built from are correct. If Docker images are built and deployed as part of an automated pipeline, it is also important that they continue to work as expected when changes are made upstream.

Start testing your Docker images without relying on Bash scripts! We’ll cover why we decided to write a testing library and how to use it. We’ll also talk about some of the test fixtures we developed for common infrastructure such as RabbitMQ and PostgreSQL. Finally, we’ll explore some of the limitations and workarounds of creating a test environment of Docker containers.

Some of the best tools for working with Docker are already written in Python, for example, docker-compose. Bringing together the Python ecosystems around Docker and test frameworks, we created a new Python library called Seaworthy. Seaworthy can be used to verify that a Docker image works as expected in an isolated environment. It provides rich tools for asserting on processes, logs, and HTTP requests.

Pycon ZA

October 11, 2018
Tweet

More Decks by Pycon ZA

Other Decks in Programming

Transcript

  1. Introduction • Site Reliability Engineer (SRE) and Tech Ambassador at

    Praekelt.org • Developed continuous integration (CI) workflows for Docker images • 3rd PyConZA, 2nd talk on Docker
  2. What’s a (Linux) container? Isolation of a process’ view of

    its operating environment (via namespaces) Limitation/prioritization of resources (via cgroups) •Block I/O •Networking… •CPU •Memory •Networking •Mounted filesystems… •Process trees •User IDs
  3. Docker containers Docker is the most popular container technology. •

    Batteries included • Easy-to-use, lots of sensible defaults • Layered filesystem: containers can start up very quickly and share a lot of filesystem data • Images available for all the software you know & love
  4. Why containers? Consistent portability • A clean way to package

    software • With (almost) everything it needs to run • With a single, (hopefully) simple entry-point • Limit access to resources
 Eliminates “but it works on my machine”
  5. Docker daemon • All interactions with Docker happen via the

    daemon • Docker has an HTTP API: Docker Engine API
  6. Seaworthy • A testing library we wrote in Python •

    Integrates with Docker using the Docker for Python SDK • Interact with Docker daemon programmatically • Define various Docker resources as test fixtures • Containers, networks, and volumes • Handles creation/teardown of resources • Prebuilt fixtures for common containers: PostgreSQL, RabbitMQ, Redis • Tools for ensuring containers fully started • Leverages existing Python testing libraries • unittest, pytest, testtools
  7. Resource Definitions • Define a resource before it is created

    • So that we know how to create/tear it down • e.g. ContainerDefinition, VolumeDefinition • Docker SDK/daemon API complicated • Attempt to make Seaworthy closer to docker CLI or docker- compose. • Wrappers around Docker SDK types.
  8. Helpers • Docker SDK structured as models and collections of

    models • e.g. Container (model), ContainerCollection (collection) • Definitions wrap models, “helpers” wrap collections • Helpers (try to) track all resources created • So there aren’t any containers/networks/volumes lying around after tests • If you use the built-in pytest fixtures you don’t have to worry about helpers much
  9. Sensible defaults • Seaworthy does some things by default to

    make life easier • Inspired by docker-compose and docker CLI • Pull missing images by default • Create dedicated bridge network for containers • Make containers available by name • Shorter forms of image names, volume mounts
  10. Waiting for containers to start • Wait for certain log

    line(s) to appear • Stream logs line-by-line, match against pattern(s) with a timeout • Simplest and most effective • Wait for HTTP response • Wait for container to be “running” for some time
  11. HTTP client • Often useful to be able to make

    requests against a container • Seaworthy can create a Requests-based client to make requests against forwarded ports
  12. Writing tests Assert on: • Command return codes, output •

    stdout/stderr • HTTP responses • Process trees • Info from Docker about the container • More…
  13. Competitors • Testcontainers • Java/JUnit • Selenium WebDriver containers •

    https://www.testcontainers.org • Google’s Container Structure Tests • Declarative tests: YAML • Commands + file existence, contents, metadata • https://github.com/GoogleContainerTools/container-structure-test
  14. pytest • pytest can do many things • Mostly interested

    in its fixture functionality • Easy setup/teardown • Easily adjustable scope • Can inspect which test requested the fixture • Other tools for annotating tests
  15. pytest • Built-in fixture “factories” • Call pytest_fixture() on any

    definition instance • Default helpers used automatically • Annotate tests to only run on systems with Docker
  16. testtools • Various extensions to Python test framework • Mostly

    interested in its matchers • Make it possible/ easier to assert complex things
  17. Performance • Container things are often slow • Slow tests

    don’t get run • Resetting state is a big task • e.g. recreating a PostgreSQL database • Python not the bottleneck • Docker daemon not good at parallelism
  18. Performance workarounds • Have to carefully choose container fixture scope

    • Longer scope (e.g. module), faster tests • Shorter scope (e.g. function), less state between tests • Concept of “cleanable” state • e.g. drop and recreate table instead of restarting container • Call clean() on container with long scope when necessary
  19. What’s in a container? • Seaworthy relies on docker exec

    calls to do many things • What if executable is not in the container? • Seaworthy uses ps to list the running processes • Debian Jessie image had procps pre-installed, Stretch doesn’t • Could read /proc in the container • What if there weren’t even basic Unix tools? • How to check the presence/contents/metadata of a file? • Could stream tarball from daemon, check files
  20. Fixture ordering • Resources often have strict ordering requirements •

    e.g. create networks, volumes before containers that use them • Can make fixture setup even slower • pytest lets fixtures depend on other fixtures • May be other libraries that can do fixture ordering better • docker-compose handles this nicely because it is declarative
  21. Container state management • Most database images will start up

    with empty state, then create default user/role/tables etc. • Potential to improve performance by doing less at startup • zanox/mysql allows applying SQL code while building image • docker commit: Save container filesystem state • docker checkpoint: Save container execution state (experimental)
  22. Multi-container patterns • Application pod • Kubernetes concept • Co-located

    containers • Integration between 2 containers • In many cases this should be transparent: e.g. “service mesh” • Some cases tighter integration, e.g. • Webserver serving files at certain paths • Vault agent token handshake Ambassador pattern Sidecar pattern Adapter pattern Design Patterns for Container-based Distributed Systems by B. Burns
  23. Kubernetes • Could try run containers for test on an

    external cluster • Kubernetes platform now de-facto standard • But is complex, with many variables • Greater potential for parallelising tests • Just run lots of containers operating independently • Would be sensitive to scheduling latency • Is there something like this already???
  24. Is this recreating production? • Recreating production with Docker containers

    on a dev machine is a nice idea • In practice many limitations • The only real production environment is production itself • Seaworthy aimed at testing Dockerfiles, entry point scripts, basic config • But you could do a full integration test suite if you wanted
  25. This is a difficult problem • Seaworthy works well for

    the limited cases we’ve used it • Needs more users and contributors • Docker/container space move quickly • Driven by large companies with more resources • It’s been a learning experience • Technical challenges, finding quirks in popular software • Lots of ideas that may never be built • Need a community to build sizeable open source projects