ContainerDays 2019 - Write Maintainable Integration Tests with Docker

Integration tests are extremely valuable but very hard to maintain and to write. You need to provision and manage the lifecycle for the dependencies that your application uses such as caches and databases. Testcontainer is an open source community all focused on solving this kind of problems across many languages. Gianluca is the maintainer of the Golang implementation that uses the Docker API to expose a test-friendly library that you can use in your test cases. During this talk, he will show you how to use testcontainer to write unit tests in Go with very low overhead.

Transcript

1. Gianluca Arbezzano Write maintainable test with Docker
2. © 2019 InfluxData. All rights reserved. 2 Who am I? Gianluca Arbezzano Site Reliability Engineer @InfluxData • http:/ /gianarb.it • @gianarb What I like: • I make dirty hacks that look awesome • I grow my vegetables • Travel for fun and work
3. At least not integration tests Testing is not a solved issue
4. Old school applications (the greaters)
5. Today’s applications
6. The “mock everything” era is over we have too many integration points...
7. This doesn’t mean we need to stop doing unit tests! They still matter a lot! Disclaimer
8. •jUnit •PHPUnit •unittest •go “testing” There are frameworks for unit tests
9. $ docker-compose up -d $ make test-integration 1. Orchestration Issue 2. Validation issue 3. Flakiness 4. No Parallelization What about integration tests?
10. We can do better
11. We need to remember that Docker has an API
14. DIND docker run \ -v /var/run/docker.sock:/var/run/docker.sock ...
15. Over the DOCKER CLI dockerd -H tcp://10.120.0.12
16. The SDK ctx := context.Background() cli, err := client.NewClientWithOpts(client.FromEnv) if err != nil { panic(err) } cli.NegotiateAPIVersion(ctx) reader, err := cli.ImagePull(ctx, "docker.io/library/alpine", types.ImagePullOptions{}) if err != nil { panic(err) } io.Copy(os.Stdout, reader)
17. Programmatically provision your integration tests config = MySqlContainer('mysql:5.7.17') with config as mysql: e = sqlalchemy.create_engine(mysql.get_connection_url()) result = e.execute("select version()")
18. github.com/testcontaine rs TestContainers
20. Where to find us: https://testcontainers.slack.com @testcontainers on Twitter
21. testcontainers-go
22. func TestNginxLatestReturn(t *testing.T) { ctx := context.Background() req := testcontainers.ContainerRequest{ Image: "nginx", ExposedPorts: []string{"80/tcp"}, } nginxC, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{ ContainerRequest: req, Started: true, }) if err != nil { t.Error(err) } defer nginxC.Terminate(ctx) ip, err := nginxC.Host(ctx) if err != nil { t.Error(err) } port, err := nginxC.MappedPort(ctx, "80") if err != nil { t.Error(err) } resp, err := http.Get(fmt.Sprintf("http://%s:%s", ip, port.Port())) if resp.StatusCode != http.StatusOK { t.Errorf("Expected status code %d. Got %d.", http.StatusOK, resp.StatusCode) } }
23. ctx := context.Background() req := ContainerRequest{ Image: "mysql:latest", ExposedPorts: []string{"3306/tcp", "33060/tcp"}, Env: map[string]string{ "MYSQL_ROOT_PASSWORD": "password", "MYSQL_DATABASE": "database", }, WaitingFor: wait.ForLog("port: 3306 MySQL Community Server - GPL"), } mysqlC, _ := GenericContainer(ctx, testcontainers.GenericContainerRequest{ ContainerRequest: req, Started: true, }) connectionString := fmt.Sprintf("%s:%s@tcp(%s:%d)/%s?tls=skip-verify", "root", "password", host, port, "database") db, err := sql.Open("mysql", connectionString)
24. testcontainers-java
25. /** Integration test for Redis-backed cache implementation. */ public class RedisBackedCacheTest { @Rule public GenericContainer redis = new GenericContainer("redis:3.0.6").withExposedPorts(6379); private Cache cache; @Before public void setUp() throws Exception { Jedis jedis = new Jedis(redis.getContainerIpAddress(), redis.getMappedPort(6379)); cache = new RedisBackedCache(jedis, "test"); } @Test public void testFindingAnInsertedValue() { cache.put("foo", "FOO"); Optional<String> foundObject = cache.get("foo", String.class); assertTrue("When an object in the cache is retrieved, it can be found", foundObject.isPresent()); assertEquals("When we put a String in to the cache and retrieve it, the value is the same", "FOO", foundObject.get()); } https://github.com/testcontainers/testcontainers-java/tree/master/examples
26. container.execInContainer("touch", "/somefile.txt"); new GenericContainer(...).withEnv("API_TOKEN", "foo") new GenericContainer(...) .withClasspathResourceMapping("redis.conf", "/etc/redis.conf", BindMode.READ_ONLY)
27. public GenericContainer nginxWithHttpWait = new GenericContainer("nginx:1.9.4") .withExposedPorts(80) .waitingFor(Wait.forHttp("/")); Wait.forHttp("/") .forStatusCode(200) .forStatusCode(301)
28. @Rule public GenericContainer dslContainer = new GenericContainer( new ImageFromDockerfile() .withFileFromString("folder/someFile.txt", "hello") .withFileFromClasspath("test.txt", "mappable-resource/test-resource.txt") .withFileFromClasspath("Dockerfile", "mappable-dockerfile/Dockerfile"))
29. public class ElasticsearchStorageRule extends ExternalResource { static final Logger LOGGER = LoggerFactory.getLogger(ElasticsearchStorageRule.class); static final int ELASTICSEARCH_PORT = 9200; final String image; final String index; GenericContainer container; Closer closer = Closer.create(); public ElasticsearchStorageRule(String image, String index) { this.image = image; this.index = index; } @Override protected void before() { try { LOGGER.info("Starting docker image " + image); container = new GenericContainer(image) .withExposedPorts(ELASTICSEARCH_PORT) .waitingFor(new HttpWaitStrategy().forPath("/")); container.start(); if (Boolean.valueOf(System.getenv("ES_DEBUG"))) { container.followOutput(new Slf4jLogConsumer(LoggerFactory.getLogger(image))); } System.out.println("Starting docker image " + image); } catch (RuntimeException e) { LOGGER.warn("Couldn't start docker image " + image + ": " + e.getMessage(), e); } https://github.com/apache/incubator-zipkin/blob/b8646142fa15c8c5f47ff2a2a48dc663c7bb65b3/zipkin-storage/elasticsearch/src/test/java/zipkin2/elasticsearch/inte gration/ElasticsearchStorageRule.java#L30
30. testcontainers/moby-ryuk
32. Write your own test util functions You can write your packages that include functions to spin up and configure your environments.
33. testcontainers-go has canned containers
34. ctx := context.Background() k := &KubeKindContainer{} err := k.Start(ctx) if err != nil { t.Fatal(err.Error()) } defer k.Terminate(ctx) clientset, err := k.GetClientset() if err != nil { t.Fatal(err.Error()) } ns, err := clientset.CoreV1().Namespaces().Get("default", metav1.GetOptions{}) if err != nil { t.Fatal(err.Error()) } https://github.com/testcontainers/testcontainers-go/pull/67
35. Links and Credits • https://medium.zenika.com/dockerize-your-integration-tests-8d26a7425baa • https://www.testcontainers.org/ • https://github.com/bmuschko/testcontainers-demo • https://gianarb.it/blog/testcontainers-go • https://rnorth.org/better-junit-selenium-testing-with-docker-and-testcontainers • https://github.com/testcontainers
36. @gianarb Thanks