Test your Docker images with Python by Jamie Hewland

As more and more software is packaged in Docker images, it has become increasingly important that the Dockerfiles and scripts that these images are built from are correct. If Docker images are built and deployed as part of an automated pipeline, it is also important that they continue to work as expected when changes are made upstream.

Start testing your Docker images without relying on Bash scripts! We’ll cover why we decided to write a testing library and how to use it. We’ll also talk about some of the test fixtures we developed for common infrastructure such as RabbitMQ and PostgreSQL. Finally, we’ll explore some of the limitations and workarounds of creating a test environment of Docker containers.

Some of the best tools for working with Docker are already written in Python, for example, docker-compose. Bringing together the Python ecosystems around Docker and test frameworks, we created a new Python library called Seaworthy. Seaworthy can be used to verify that a Docker image works as expected in an isolated environment. It provides rich tools for asserting on processes, logs, and HTTP requests.

Transcript

1. Test your Docker images with Python Jamie Hewland PyConZA 2018
2. Welcome 00 Thanks for coming to a talk about testing
3. Introduction • Site Reliability Engineer (SRE) and Tech Ambassador at Praekelt.org • Developed continuous integration (CI) workflows for Docker images • 3rd PyConZA, 2nd talk on Docker
4. What’s a (Linux) container? Isolation of a process’ view of its operating environment (via namespaces) Limitation/prioritization of resources (via cgroups) •Block I/O •Networking… •CPU •Memory •Networking •Mounted filesystems… •Process trees •User IDs
5. Docker containers Docker is the most popular container technology. • Batteries included • Easy-to-use, lots of sensible defaults • Layered filesystem: containers can start up very quickly and share a lot of filesystem data • Images available for all the software you know & love
6. ubuntu debian redis rabbitmq > $ docker run python postgres sentry nginx nodejs
7. Why containers? Consistent portability • A clean way to package software • With (almost) everything it needs to run • With a single, (hopefully) simple entry-point • Limit access to resources
 Eliminates “but it works on my machine”
8. Docker terminology Terms “image” and “container” often conflated
9. Docker daemon • All interactions with Docker happen via the daemon • Docker has an HTTP API: Docker Engine API
10. The problem 01 Dockerfiles are code— code should be tested
11. Dockerfiles
12. Entry point scripts…
13. Config… 12-factor app environment variables
14. How is this usually solved? Bash scripts
15. How is this usually solved? Bash Automated Testing System (Bats)
16. How is this usually solved? docker- compose (still need something to assert expectations—probably Bash)
17. Bash scripts Google’s Shell Style Guide google.github.io/ styleguide/shell.xml
18. A solution 02 Seaworthy
19. Seaworthy • A testing library we wrote in Python • Integrates with Docker using the Docker for Python SDK • Interact with Docker daemon programmatically • Define various Docker resources as test fixtures • Containers, networks, and volumes • Handles creation/teardown of resources • Prebuilt fixtures for common containers: PostgreSQL, RabbitMQ, Redis • Tools for ensuring containers fully started • Leverages existing Python testing libraries • unittest, pytest, testtools
20. Resource Definitions • Define a resource before it is created • So that we know how to create/tear it down • e.g. ContainerDefinition, VolumeDefinition • Docker SDK/daemon API complicated • Attempt to make Seaworthy closer to docker CLI or docker- compose. • Wrappers around Docker SDK types.
21. Resource Definitions
22. Helpers • Docker SDK structured as models and collections of models • e.g. Container (model), ContainerCollection (collection) • Definitions wrap models, “helpers” wrap collections • Helpers (try to) track all resources created • So there aren’t any containers/networks/volumes lying around after tests • If you use the built-in pytest fixtures you don’t have to worry about helpers much
23. Sensible defaults • Seaworthy does some things by default to make life easier • Inspired by docker-compose and docker CLI • Pull missing images by default • Create dedicated bridge network for containers • Make containers available by name • Shorter forms of image names, volume mounts
24. Waiting for containers to start • Wait for certain log line(s) to appear • Stream logs line-by-line, match against pattern(s) with a timeout • Simplest and most effective • Wait for HTTP response • Wait for container to be “running” for some time
25. HTTP client • Often useful to be able to make requests against a container • Seaworthy can create a Requests-based client to make requests against forwarded ports
26. Writing tests Assert on: • Command return codes, output • stdout/stderr • HTTP responses • Process trees • Info from Docker about the container • More…
27. Competitors • Testcontainers • Java/JUnit • Selenium WebDriver containers • https://www.testcontainers.org • Google’s Container Structure Tests • Declarative tests: YAML • Commands + file existence, contents, metadata • https://github.com/GoogleContainerTools/container-structure-test
28. Python testing libraries 03 pytest and testtools
29. pytest • pytest can do many things • Mostly interested in its fixture functionality • Easy setup/teardown • Easily adjustable scope • Can inspect which test requested the fixture • Other tools for annotating tests
30. pytest • Built-in fixture “factories” • Call pytest_fixture() on any definition instance • Default helpers used automatically • Annotate tests to only run on systems with Docker
31. testtools • Various extensions to Python test framework • Mostly interested in its matchers • Make it possible/ easier to assert complex things
32. testtools • Process tree assertions
33. Challenges 04 Managing state, container boundaries
34. Performance • Container things are often slow • Slow tests don’t get run • Resetting state is a big task • e.g. recreating a PostgreSQL database • Python not the bottleneck • Docker daemon not good at parallelism
35. Performance workarounds • Have to carefully choose container fixture scope • Longer scope (e.g. module), faster tests • Shorter scope (e.g. function), less state between tests • Concept of “cleanable” state • e.g. drop and recreate table instead of restarting container • Call clean() on container with long scope when necessary
36. Performance workarounds
37. What’s in a container? • Seaworthy relies on docker exec calls to do many things • What if executable is not in the container? • Seaworthy uses ps to list the running processes • Debian Jessie image had procps pre-installed, Stretch doesn’t • Could read /proc in the container • What if there weren’t even basic Unix tools? • How to check the presence/contents/metadata of a file? • Could stream tarball from daemon, check files
38. Fixture ordering • Resources often have strict ordering requirements • e.g. create networks, volumes before containers that use them • Can make fixture setup even slower • pytest lets fixtures depend on other fixtures • May be other libraries that can do fixture ordering better • docker-compose handles this nicely because it is declarative
39. Further work 05 Optimisations and pod support
40. Container state management • Most database images will start up with empty state, then create default user/role/tables etc. • Potential to improve performance by doing less at startup • zanox/mysql allows applying SQL code while building image • docker commit: Save container filesystem state • docker checkpoint: Save container execution state (experimental)
41. Multi-container patterns • Application pod • Kubernetes concept • Co-located containers • Integration between 2 containers • In many cases this should be transparent: e.g. “service mesh” • Some cases tighter integration, e.g. • Webserver serving files at certain paths • Vault agent token handshake Ambassador pattern Sidecar pattern Adapter pattern Design Patterns for Container-based Distributed Systems by B. Burns
42. Kubernetes • Could try run containers for test on an external cluster • Kubernetes platform now de-facto standard • But is complex, with many variables • Greater potential for parallelising tests • Just run lots of containers operating independently • Would be sensitive to scheduling latency • Is there something like this already???
43. Conclusions 06 …or should you write a testing library?
44. Is this recreating production? • Recreating production with Docker containers on a dev machine is a nice idea • In practice many limitations • The only real production environment is production itself • Seaworthy aimed at testing Dockerfiles, entry point scripts, basic config • But you could do a full integration test suite if you wanted
45. This is a difficult problem • Seaworthy works well for the limited cases we’ve used it • Needs more users and contributors • Docker/container space move quickly • Driven by large companies with more resources • It’s been a learning experience • Technical challenges, finding quirks in popular software • Lots of ideas that may never be built • Need a community to build sizeable open source projects
46. Thank you Seaworthy praekeltfoundation/seaworthy seaworthy.readthedocs.io @praekeltorg medium.com/mobileforgood Me @jayhewland youtu.be/T2hooQzvurQ Questions?