Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Test your Docker images with Python

Test your Docker images with Python

As more and more software is packaged in Docker images, it has become increasingly important that the Dockerfiles and scripts that these images are built from are correct. If Docker images are built and deployed as part of an automated pipeline, it is also important that they continue to work as expected when changes are made upstream.

Start testing your Docker images without relying on Bash scripts! We’ll cover why we decided to write a testing library and how to use it. We’ll also talk about some of the test fixtures we developed for common infrastructure such as RabbitMQ and PostgreSQL. Finally, we’ll explore some of the limitations and workarounds of creating a test environment of Docker containers.

Some of the best tools for working with Docker are already written in Python, for example, docker-compose. Bringing together the Python ecosystems around Docker and test frameworks, we created a new Python library called Seaworthy. Seaworthy can be used to verify that a Docker image works as expected in an isolated environment. It provides rich tools for asserting on processes, logs, and HTTP requests.


Jamie Hewland

October 12, 2018


  1. Test your Docker images with Python Jamie Hewland PyConZA 2018

  2. Welcome 00 Thanks for coming to a talk about testing

  3. Introduction • Site Reliability Engineer (SRE) and Tech Ambassador at

    Praekelt.org • Developed continuous integration (CI) workflows for Docker images • 3rd PyConZA, 2nd talk on Docker
  4. What’s a (Linux) container? Isolation of a process’ view of

    its operating environment (via namespaces) Limitation/prioritization of resources (via cgroups) •Block I/O •Networking… •CPU •Memory •Networking •Mounted filesystems… •Process trees •User IDs
  5. Docker containers Docker is the most popular container technology. •

    Batteries included • Easy-to-use, lots of sensible defaults • Layered filesystem: containers can start up very quickly and share a lot of filesystem data • Images available for all the software you know & love
  6. ubuntu debian redis rabbitmq > $ docker run python postgres

    sentry nginx nodejs
  7. Why containers? Consistent portability • A clean way to package

    software • With (almost) everything it needs to run • With a single, (hopefully) simple entry-point • Limit access to resources
 Eliminates “but it works on my machine”
  8. Docker terminology Terms “image” and “container” often conflated

  9. Docker daemon • All interactions with Docker happen via the

    daemon • Docker has an HTTP API: Docker Engine API
  10. The problem 01 Dockerfiles are code— code should be tested

  11. Dockerfiles

  12. Entry point scripts…

  13. Config… 12-factor app environment variables

  14. How is this usually solved? Bash scripts

  15. How is this usually solved? Bash Automated Testing System (Bats)

  16. How is this usually solved? docker- compose (still need something

    to assert expectations—probably Bash)
  17. Bash scripts Google’s Shell Style Guide google.github.io/ styleguide/shell.xml

  18. A solution 02 Seaworthy

  19. Seaworthy • A testing library we wrote in Python •

    Integrates with Docker using the Docker for Python SDK • Interact with Docker daemon programmatically • Define various Docker resources as test fixtures • Containers, networks, and volumes • Handles creation/teardown of resources • Prebuilt fixtures for common containers: PostgreSQL, RabbitMQ, Redis • Tools for ensuring containers fully started • Leverages existing Python testing libraries • unittest, pytest, testtools
  20. Resource Definitions • Define a resource before it is created

    • So that we know how to create/tear it down • e.g. ContainerDefinition, VolumeDefinition • Docker SDK/daemon API complicated • Attempt to make Seaworthy closer to docker CLI or docker- compose. • Wrappers around Docker SDK types.
  21. Resource Definitions

  22. Helpers • Docker SDK structured as models and collections of

    models • e.g. Container (model), ContainerCollection (collection) • Definitions wrap models, “helpers” wrap collections • Helpers (try to) track all resources created • So there aren’t any containers/networks/volumes lying around after tests • If you use the built-in pytest fixtures you don’t have to worry about helpers much
  23. Sensible defaults • Seaworthy does some things by default to

    make life easier • Inspired by docker-compose and docker CLI • Pull missing images by default • Create dedicated bridge network for containers • Make containers available by name • Shorter forms of image names, volume mounts
  24. Waiting for containers to start • Wait for certain log

    line(s) to appear • Stream logs line-by-line, match against pattern(s) with a timeout • Simplest and most effective • Wait for HTTP response • Wait for container to be “running” for some time
  25. HTTP client • Often useful to be able to make

    requests against a container • Seaworthy can create a Requests-based client to make requests against forwarded ports
  26. Writing tests Assert on: • Command return codes, output •

    stdout/stderr • HTTP responses • Process trees • Info from Docker about the container • More…
  27. Competitors • Testcontainers • Java/JUnit • Selenium WebDriver containers •

    https://www.testcontainers.org • Google’s Container Structure Tests • Declarative tests: YAML • Commands + file existence, contents, metadata • https://github.com/GoogleContainerTools/container-structure-test
  28. Python testing libraries 03 pytest and testtools

  29. pytest • pytest can do many things • Mostly interested

    in its fixture functionality • Easy setup/teardown • Easily adjustable scope • Can inspect which test requested the fixture • Other tools for annotating tests
  30. pytest • Built-in fixture “factories” • Call pytest_fixture() on any

    definition instance • Default helpers used automatically • Annotate tests to only run on systems with Docker
  31. testtools • Various extensions to Python test framework • Mostly

    interested in its matchers • Make it possible/ easier to assert complex things
  32. testtools • Process tree assertions

  33. Challenges 04 Managing state, container boundaries

  34. Performance • Container things are often slow • Slow tests

    don’t get run • Resetting state is a big task • e.g. recreating a PostgreSQL database • Python not the bottleneck • Docker daemon not good at parallelism
  35. Performance workarounds • Have to carefully choose container fixture scope

    • Longer scope (e.g. module), faster tests • Shorter scope (e.g. function), less state between tests • Concept of “cleanable” state • e.g. drop and recreate table instead of restarting container • Call clean() on container with long scope when necessary
  36. Performance workarounds

  37. What’s in a container? • Seaworthy relies on docker exec

    calls to do many things • What if executable is not in the container? • Seaworthy uses ps to list the running processes • Debian Jessie image had procps pre-installed, Stretch doesn’t • Could read /proc in the container • What if there weren’t even basic Unix tools? • How to check the presence/contents/metadata of a file? • Could stream tarball from daemon, check files
  38. Fixture ordering • Resources often have strict ordering requirements •

    e.g. create networks, volumes before containers that use them • Can make fixture setup even slower • pytest lets fixtures depend on other fixtures • May be other libraries that can do fixture ordering better • docker-compose handles this nicely because it is declarative
  39. Further work 05 Optimisations and pod support

  40. Container state management • Most database images will start up

    with empty state, then create default user/role/tables etc. • Potential to improve performance by doing less at startup • zanox/mysql allows applying SQL code while building image • docker commit: Save container filesystem state • docker checkpoint: Save container execution state (experimental)
  41. Multi-container patterns • Application pod • Kubernetes concept • Co-located

    containers • Integration between 2 containers • In many cases this should be transparent: e.g. “service mesh” • Some cases tighter integration, e.g. • Webserver serving files at certain paths • Vault agent token handshake Ambassador pattern Sidecar pattern Adapter pattern Design Patterns for Container-based Distributed Systems by B. Burns
  42. Kubernetes • Could try run containers for test on an

    external cluster • Kubernetes platform now de-facto standard • But is complex, with many variables • Greater potential for parallelising tests • Just run lots of containers operating independently • Would be sensitive to scheduling latency • Is there something like this already???
  43. Conclusions 06 …or should you write a testing library?

  44. Is this recreating production? • Recreating production with Docker containers

    on a dev machine is a nice idea • In practice many limitations • The only real production environment is production itself • Seaworthy aimed at testing Dockerfiles, entry point scripts, basic config • But you could do a full integration test suite if you wanted
  45. This is a difficult problem • Seaworthy works well for

    the limited cases we’ve used it • Needs more users and contributors • Docker/container space move quickly • Driven by large companies with more resources • It’s been a learning experience • Technical challenges, finding quirks in popular software • Lots of ideas that may never be built • Need a community to build sizeable open source projects
  46. Thank you Seaworthy praekeltfoundation/seaworthy seaworthy.readthedocs.io @praekeltorg medium.com/mobileforgood Me @jayhewland youtu.be/T2hooQzvurQ