Powering a build pipeline with Docker and Jenkins Benjamin Muschko

AUTOMATED ASCENT bmuschko bmuschko bmuschko.com About the speaker automatedascent.com

As a Java developer, I want to use Docker but…

Find your Docker Zen

Agenda Typical Continuous Delivery uses cases How can Docker help? Show me some code! The glue as code in Jenkins

Typical workflows

Dockerizing a Java application

Pushing an image to a registry

Image as fixture for testing

Running application stacks

Docker as part of the pipeline

Docker’s CLI is great but do I need to operate it by hand?

Pick the right tool for the job

Docker CLI Other options

Show me something beyond Hello World!

Sample architecture whiteboard Client Web application Spring Boot Web service Spring Boot Database PostgreSQL

Demo Time

Containerization

Pick a small base image

Copy app archive or files?

Controlling runtime behavior

Is application healthy?

Version appropriately 4.2.6

Dockerfile for Spring Boot app FROM openjdk:jre-alpine COPY todo-webservice-1.0.jar↵ /app/todo-webservice-1.0.jar
 ENTRYPOINT ["java"]
 CMD ["-jar", "/app/todo-webservice-1.0.jar"]
 HEALTHCHECK CMD wget --quiet --tries=1 --spider↵ http://localhost:8080/actuator/health || exit 1
 EXPOSE 8080

Dockerfile with layers for app files FROM openjdk:jre-alpine WORKDIR /app COPY libs libs/ COPY resources resources/ COPY classes classes/ ENTRYPOINT ["java", "-cp",↵ "/app/resources:/app/classes:/app/libs/*",↵ "com.bmuschko.MyApplication"]
 HEALTHCHECK CMD wget --quiet --tries=1 --spider↵ http://localhost:8080/actuator/health || exit 1
 EXPOSE 8080

Building and running image # Build image from Dockerfile
 docker build -t my-todo-web-service:1.0.0 .
 
 # Run built image in container
 docker run -d -p 8080:8080 my-todo-web-service:1.0.0

Containerization with Google Jib Build images without the need for Docker Engine https://github.com/GoogleContainerTools/jib

No need for Docker Engine

FROM … COPY … EXPOSE … ENTRYPOINT … Dockerfile is optional

Layers for cacheability

Integration with build tools

Google Jib in Maven build 
 ...
 
 
 
 com.google.cloud.tools
 jib-maven-plugin
 1.6.0
 
 
 bmuschko/my-web-service:1.0.0
 
 
 
 
 
 $ ./mvnw compile jib:dockerBuild

Google Jib in Gradle build plugins { id 'java' id 'com.google.cloud.tools.jib' version ‘1.6.0' } jib.to.image = 'bmuschko/my-web-service:1.0.0' $ ./gradlew jibDockerBuild

Demo Time

Pushing image # Log into Docker registry
 docker login --username=bmuschko↵ —email=benjamin.muschko@gmail.com
 
 # Tag the image
 docker tag bb38976d03cf bmuschko/todo-web-service:1.0.0
 
 # Push image to Docker registry
 docker push bmuschko/todo-web-service

Using Google Jib $ ./gradlew jib $ ./mvnw compile jib:build

Demo Time

Integrated Testing

2 unit tests, 0 integration tests

Reproducible environment

Slow startup times

Isolated & cross-platform

The obvious choice

Pull image and start/stop container # Pull image from registry (or build locally)
 docker pull bmuschko/todo-web-service:1.0.0
 
 # Start container
 docker run -d -p 8080:8080 -name todo-web-service↵ my-todo-web-service:1.0.0
 
 # Stop container
 docker container stop todo-web-service
 
 # Remove container
 docker container rm todo-web-service

Testing with TestContainers Managing lightweight, throwaway Docker containers for testing https://www.testcontainers.org/

Docker Engine Communication

Docker Environment Discovery

Automatic container cleanup

Multi-container coordination

Testcontainers in Maven build 
 org.testcontainers
 testcontainers
 1.12.0
 test


Testcontainers in Gradle build repositories {
 mavenCentral()
 }
 
 dependencies {
 testImplementation 'org.testcontainers:junit-jupiter:1.12.0'
 testImplementation 'org.testcontainers:postgresql:1.12.0'
 testRuntime 'org.postgresql:postgresql'
 }

Integration test with JUnit 5 @DataJpaTest
 public class ToDoRepositoryIntegrationTest {
 @Autowired
 private ToDoRepository repository;
 
 @Test
 public void testCanSaveNewToDoItem() {
 ToDoItem toDoItem = createToDoItem("Buy milk");
 assertNull(toDoItem.getId());
 repository.save(toDoItem);
 assertNotNull(toDoItem.getId());
 }
 
 private ToDoItem createToDoItem(String name) {
 ToDoItem toDoItem = new ToDoItem();
 toDoItem.setName(name);
 return toDoItem;
 }
 } Class under test

Containerized database as fixture @Testcontainers
 @AutoConfigureTestDatabase(replace = NONE)
 @ContextConfiguration(initializers = { ToDoRepositoryIntegrationTest.Initializer.class })
 public class ToDoRepositoryIntegrationTest {
 @Container
 public static PostgreSQLContainer postgreSQLContainer = createDbContainer();
 
 private static PostgreSQLContainer createDbContainer() {
 return new PostgreSQLContainer("postgres:9.6.10-alpine")
 .withUsername("postgres")
 .withPassword("postgres")
 .withDatabaseName("todo");
 }
 
 static class Initializer implements ApplicationContextInitializer {
 @Override
 public void initialize(ConfigurableApplicationContext configurableApplicationContext) {
 TestPropertyValues.of(
 "spring.datasource.url=" + postgreSQLContainer.getJdbcUrl(),
 "spring.datasource.username=" + postgreSQLContainer.getUsername(),
 "spring.datasource.password=" + postgreSQLContainer.getPassword(),
 "spring.datasource.driver-class-name=org.postgresql.Driver",
 "spring.jpa.generate-ddl=true"
 ).applyTo(configurableApplicationContext.getEnvironment());
 }
 }
 } Create database container Inject database Configuration

Functional test with JUnit 5 public class ToDoWebServiceFunctionalTest {
 @Test
 @DisplayName("can retrieve all items before and after inserting new ones")
 void retrieveAllItems() {
 String allItems = getAllItems();
 assertEquals("[]", allItems);
 
 ToDoItem toDoItem = new ToDoItem();
 toDoItem.setName("Buy milk");
 toDoItem.setCompleted(false);
 insertItem(toDoItem);
 
 allItems = getAllItems();
 assertEquals("[{\"id\":1,\"name\":\"Buy milk\",\"completed\":false}]", allItems);
 } } Calls to HTTP endpoints

Containerized application as fixture @Testcontainers
 public class ToDoWebServiceFunctionalTest {
 @Container
 private static GenericContainer appContainer = createContainer();
 
 private static GenericContainer createContainer() {
 return new GenericContainer(buildImageDockerfile())
 .withExposedPorts(8080)
 .waitingFor(Wait.forHttp("/actuator/health")
 .forStatusCode(200));
 }
 
 private static ImageFromDockerfile buildImageDockerfile() {
 return new ImageFromDockerfile()
 .withFileFromFile(ARCHIVE_NAME, new File(DISTRIBUTION_DIR, ARCHIVE_NAME))
 .withDockerfileFromBuilder(builder -> builder
 .from("openjdk:jre-alpine")
 .copy(ARCHIVE_NAME, "/app/" + ARCHIVE_NAME)
 .entryPoint("java", "-jar", "/app/" + ARCHIVE_NAME)
 .build());
 }
 } Build container on-the-fly

Demo Time

Running a multi-container app # Start composed apps
 docker-compose up
 
 # Stop composed apps
 docker-compose down

Compose test with JUnit 5 @ExtendWith(SpringExtension.class) @SpringBootTest
 public class ToDoAppIntegrationTest {
 @Autowired
 private ToDoService service;
 
 @Test
 public void canCreateNewItemAndRetrieveIt() {
 ToDoItem newItem = newItem("Buy milk");
 assertNull(newItem.getId());
 service.save(newItem);
 assertNotNull(newItem.getId());
 ToDoItem retrievedItem = service.findOne(newItem.getId());
 assertEquals(newItem, retrievedItem);
 } } Class under test

Compose fixture @Testcontainers
 public class ToDoAppImplIntegrationTest {
 private static final Logger logger = LoggerFactory.getLogger(ToDoAppImplIntegrationTest.class);
 private final static String WEB_SERVICE_NAME = "webservice_1";
 private final static int WEB_SERVICE_PORT = 8080;
 private final static String DATABASE_NAME = "database_1";
 private final static int DATABASE_PORT = 5432;
 
 @Container
 public static DockerComposeContainer environment = createComposeContainer();
 
 private static DockerComposeContainer createComposeContainer() {
 return new DockerComposeContainer(new File("src/integrationTest/resources/compose-test.yml"))
 .withLogConsumer(WEB_SERVICE_NAME, new Slf4jLogConsumer(logger))
 .withExposedService(WEB_SERVICE_NAME, WEB_SERVICE_PORT,
 Wait.forHttp("/actuator/health")
 .forStatusCode(200))
 .withExposedService(DATABASE_NAME, DATABASE_PORT,
 Wait.forLogMessage(".*database system is ready to accept connections.*\\s", 2));
 } } Create Compose container

Compose test file version: "2.0"
 services:
 database:
 image: "postgres:9.6.10-alpine"
 environment:
 - POSTGRES_USER=postgres
 - POSTGRES_PASSWORD=postgres
 - POSTGRES_DB=todo
 webservice:
 image: "bmuschko/todo-web-service:1.0.0"
 environment:
 - SPRING_PROFILES_ACTIVE=dev
 depends_on:
 - database

Demo Time

Glueing together the pipeline

Jenkins plugins Optional: Blue Ocean Pipeline Suite

Standard Pipeline Blue Ocean Pipeline

Pipeline as code Jenkinsfile Jenkins Pipeline

Invoke build with Wrapper pipeline { stages { stage('Compile & Unit Tests') { steps { gradlew('clean', 'test') } } } ... } def gradlew(String... args) { sh "./gradlew ${args.join(' ')} -s" }

Credentials Environment variables

Injecting credentials & env vars stage('Push Image') { environment { DOCKER_USERNAME = "${env.DOCKER_USERNAME}" DOCKER_PASSWORD = credentials('DOCKER_PASSWORD') DOCKER_EMAIL = "${env.DOCKER_EMAIL}" } steps { gradlew('dockerPushImage') } }

Scaling job execution

Configuration Execution

Containerized pipeline stages pipeline {
 agent {
 kubernetes {
 defaultContainer 'jnlp'
 yaml """
 apiVersion: v1
 kind: Pod
 metadata:
 labels:
 some-label: some-label-value
 spec:
 containers:
 - name: maven
 image: maven:alpine
 command:
 - cat
 tty: true
 - name: busybox
 image: busybox
 command:
 - cat
 tty: true
 """
 }
 }
 } stages {
 stage('Run in Maven container') {
 steps {
 container('maven') {
 sh 'mvn -version'
 }
 }
 }
 stage('Run in Busybox container') {
 steps {
 container('busybox') {
 sh '/bin/busybox'
 }
 }
 }
 stage('Run in default container') {
 steps {
 sh "echo 'Running in default container'"
 }
 }
 }

Demo Time

Resources github.com/bmuschko/todo-web-service github.com/bmuschko/todo-web-app github.com/bmuschko/jenkins-with-kubernetes

Thank you! Please ask questions…