Powering a build pipeline with Docker and Jenkins

Continuous Delivery is key to delivering software fast and frequently. Jenkins 2 has made great strides in providing a solution for modeling a build pipeline as code. In addition, Docker invoked from a Gradle build can help with implementing certain aspects of your pipeline. Combining the benefits of these tools makes it possible to implement a streamlined and bulletproof process.

In this demo-driven session, you will learn how to construct and operate a declarative pipeline with Jenkins. You’ll see a Spring Boot application taken all the way through the pipeline, from building the Docker image to pushing the image to a registry, using it for integration testing, and finally deploying the application to a Docker Swarm or Kubernetes.

Transcript

1. Powering a build pipeline with Docker and Jenkins Benjamin Muschko
2. AUTOMATED ASCENT bmuschko bmuschko bmuschko.com About the speaker automatedascent.com
3. As a Java developer, I want to use Docker but…
4. Find your Docker Zen
5. Agenda Typical Continuous Delivery uses cases How can Docker help? Show me some code! The glue as code in Jenkins
6. Typical workflows
7. Dockerizing a Java application
8. Pushing an image to a registry
9. Image as fixture for testing
10. Running application stacks
11. Docker as part of the pipeline
12. Docker’s CLI is great but do I need to operate it by hand?
13. Pick the right tool for the job
14. Docker CLI Other options
15. Show me something beyond Hello World!
16. Sample architecture whiteboard Client Web application Spring Boot Web service Spring Boot Database PostgreSQL
17. Demo Time
18. Containerization
19. Pick a small base image
20. Copy app archive or files?
21. Controlling runtime behavior
22. Is application healthy?
23. Version appropriately 4.2.6
24. Dockerfile for Spring Boot app FROM openjdk:jre-alpine COPY todo-webservice-1.0.jar↵ /app/todo-webservice-1.0.jar
 ENTRYPOINT ["java"]
 CMD ["-jar", "/app/todo-webservice-1.0.jar"]
 HEALTHCHECK CMD wget --quiet --tries=1 --spider↵ http://localhost:8080/actuator/health || exit 1
 EXPOSE 8080
25. Dockerfile with layers for app files FROM openjdk:jre-alpine WORKDIR /app COPY libs libs/ COPY resources resources/ COPY classes classes/ ENTRYPOINT ["java", "-cp",↵ "/app/resources:/app/classes:/app/libs/*",↵ "com.bmuschko.MyApplication"]
 HEALTHCHECK CMD wget --quiet --tries=1 --spider↵ http://localhost:8080/actuator/health || exit 1
 EXPOSE 8080
26. Building and running image # Build image from Dockerfile
 docker build -t my-todo-web-service:1.0.0 .
 
 # Run built image in container
 docker run -d -p 8080:8080 my-todo-web-service:1.0.0
27. Containerization with Google Jib Build images without the need for Docker Engine https://github.com/GoogleContainerTools/jib
28. No need for Docker Engine
29. FROM … COPY … EXPOSE … ENTRYPOINT … Dockerfile is optional
30. Layers for cacheability
31. Integration with build tools
32. Google Jib in Maven build <project>
 ...
 <build>
 <plugins>
 <plugin>
 <groupId>com.google.cloud.tools</groupId>
 <artifactId>jib-maven-plugin</artifactId>
 <version>1.6.0</version>
 <configuration>
 <to>
 <image>bmuschko/my-web-service:1.0.0</image>
 </to>
 </configuration>
 </plugin>
 </plugins>
 </build>
 </project> $ ./mvnw compile jib:dockerBuild
33. Google Jib in Gradle build plugins { id 'java' id 'com.google.cloud.tools.jib' version ‘1.6.0' } jib.to.image = 'bmuschko/my-web-service:1.0.0' $ ./gradlew jibDockerBuild
34. Demo Time
35. Pushing image # Log into Docker registry
 docker login --username=bmuschko↵ —email=benjamin.muschko@gmail.com
 
 # Tag the image
 docker tag bb38976d03cf bmuschko/todo-web-service:1.0.0
 
 # Push image to Docker registry
 docker push bmuschko/todo-web-service
36. Using Google Jib $ ./gradlew jib $ ./mvnw compile jib:build
37. Demo Time
38. Integrated Testing
39. 2 unit tests, 0 integration tests
40. Reproducible environment
41. Slow startup times
42. Isolated & cross-platform
43. The obvious choice
44. Pull image and start/stop container # Pull image from registry (or build locally)
 docker pull bmuschko/todo-web-service:1.0.0
 
 # Start container
 docker run -d -p 8080:8080 -name todo-web-service↵ my-todo-web-service:1.0.0
 
 # Stop container
 docker container stop todo-web-service
 
 # Remove container
 docker container rm todo-web-service
45. Testing with TestContainers Managing lightweight, throwaway Docker containers for testing https://www.testcontainers.org/
46. Docker Engine Communication
47. Docker Environment Discovery
48. Automatic container cleanup
49. Multi-container coordination
50. Testcontainers in Maven build <dependency>
 <groupId>org.testcontainers</groupId>
 <artifactId>testcontainers</artifactId>
 <version>1.12.0</version>
 <scope>test</scope>
 </dependency>
51. Testcontainers in Gradle build repositories {
 mavenCentral()
 }
 
 dependencies {
 testImplementation 'org.testcontainers:junit-jupiter:1.12.0'
 testImplementation 'org.testcontainers:postgresql:1.12.0'
 testRuntime 'org.postgresql:postgresql'
 }
52. Integration test with JUnit 5 @DataJpaTest
 public class ToDoRepositoryIntegrationTest {
 @Autowired
 private ToDoRepository repository;
 
 @Test
 public void testCanSaveNewToDoItem() {
 ToDoItem toDoItem = createToDoItem("Buy milk");
 assertNull(toDoItem.getId());
 repository.save(toDoItem);
 assertNotNull(toDoItem.getId());
 }
 
 private ToDoItem createToDoItem(String name) {
 ToDoItem toDoItem = new ToDoItem();
 toDoItem.setName(name);
 return toDoItem;
 }
 } Class under test
53. Containerized database as fixture @Testcontainers
 @AutoConfigureTestDatabase(replace = NONE)
 @ContextConfiguration(initializers = { ToDoRepositoryIntegrationTest.Initializer.class })
 public class ToDoRepositoryIntegrationTest {
 @Container
 public static PostgreSQLContainer postgreSQLContainer = createDbContainer();
 
 private static PostgreSQLContainer createDbContainer() {
 return new PostgreSQLContainer("postgres:9.6.10-alpine")
 .withUsername("postgres")
 .withPassword("postgres")
 .withDatabaseName("todo");
 }
 
 static class Initializer implements ApplicationContextInitializer<ConfigurableApplicationContext> {
 @Override
 public void initialize(ConfigurableApplicationContext configurableApplicationContext) {
 TestPropertyValues.of(
 "spring.datasource.url=" + postgreSQLContainer.getJdbcUrl(),
 "spring.datasource.username=" + postgreSQLContainer.getUsername(),
 "spring.datasource.password=" + postgreSQLContainer.getPassword(),
 "spring.datasource.driver-class-name=org.postgresql.Driver",
 "spring.jpa.generate-ddl=true"
 ).applyTo(configurableApplicationContext.getEnvironment());
 }
 }
 } Create database container Inject database Configuration
54. Functional test with JUnit 5 public class ToDoWebServiceFunctionalTest {
 @Test
 @DisplayName("can retrieve all items before and after inserting new ones")
 void retrieveAllItems() {
 String allItems = getAllItems();
 assertEquals("[]", allItems);
 
 ToDoItem toDoItem = new ToDoItem();
 toDoItem.setName("Buy milk");
 toDoItem.setCompleted(false);
 insertItem(toDoItem);
 
 allItems = getAllItems();
 assertEquals("[{\"id\":1,\"name\":\"Buy milk\",\"completed\":false}]", allItems);
 } } Calls to HTTP endpoints
55. Containerized application as fixture @Testcontainers
 public class ToDoWebServiceFunctionalTest {
 @Container
 private static GenericContainer appContainer = createContainer();
 
 private static GenericContainer createContainer() {
 return new GenericContainer(buildImageDockerfile())
 .withExposedPorts(8080)
 .waitingFor(Wait.forHttp("/actuator/health")
 .forStatusCode(200));
 }
 
 private static ImageFromDockerfile buildImageDockerfile() {
 return new ImageFromDockerfile()
 .withFileFromFile(ARCHIVE_NAME, new File(DISTRIBUTION_DIR, ARCHIVE_NAME))
 .withDockerfileFromBuilder(builder -> builder
 .from("openjdk:jre-alpine")
 .copy(ARCHIVE_NAME, "/app/" + ARCHIVE_NAME)
 .entryPoint("java", "-jar", "/app/" + ARCHIVE_NAME)
 .build());
 }
 } Build container on-the-fly
56. Demo Time
57. Running a multi-container app # Start composed apps
 docker-compose up
 
 # Stop composed apps
 docker-compose down
58. Compose test with JUnit 5 @ExtendWith(SpringExtension.class) @SpringBootTest
 public class ToDoAppIntegrationTest {
 @Autowired
 private ToDoService service;
 
 @Test
 public void canCreateNewItemAndRetrieveIt() {
 ToDoItem newItem = newItem("Buy milk");
 assertNull(newItem.getId());
 service.save(newItem);
 assertNotNull(newItem.getId());
 ToDoItem retrievedItem = service.findOne(newItem.getId());
 assertEquals(newItem, retrievedItem);
 } } Class under test
59. Compose fixture @Testcontainers
 public class ToDoAppImplIntegrationTest {
 private static final Logger logger = LoggerFactory.getLogger(ToDoAppImplIntegrationTest.class);
 private final static String WEB_SERVICE_NAME = "webservice_1";
 private final static int WEB_SERVICE_PORT = 8080;
 private final static String DATABASE_NAME = "database_1";
 private final static int DATABASE_PORT = 5432;
 
 @Container
 public static DockerComposeContainer environment = createComposeContainer();
 
 private static DockerComposeContainer createComposeContainer() {
 return new DockerComposeContainer(new File("src/integrationTest/resources/compose-test.yml"))
 .withLogConsumer(WEB_SERVICE_NAME, new Slf4jLogConsumer(logger))
 .withExposedService(WEB_SERVICE_NAME, WEB_SERVICE_PORT,
 Wait.forHttp("/actuator/health")
 .forStatusCode(200))
 .withExposedService(DATABASE_NAME, DATABASE_PORT,
 Wait.forLogMessage(".*database system is ready to accept connections.*\\s", 2));
 } } Create Compose container
60. Compose test file version: "2.0"
 services:
 database:
 image: "postgres:9.6.10-alpine"
 environment:
 - POSTGRES_USER=postgres
 - POSTGRES_PASSWORD=postgres
 - POSTGRES_DB=todo
 webservice:
 image: "bmuschko/todo-web-service:1.0.0"
 environment:
 - SPRING_PROFILES_ACTIVE=dev
 depends_on:
 - database
61. Demo Time
62. Glueing together the pipeline
63. Jenkins plugins Optional: Blue Ocean Pipeline Suite
64. Standard Pipeline Blue Ocean Pipeline
65. Pipeline as code Jenkinsfile Jenkins Pipeline
66. Invoke build with Wrapper pipeline { stages { stage('Compile & Unit Tests') { steps { gradlew('clean', 'test') } } } ... } def gradlew(String... args) { sh "./gradlew ${args.join(' ')} -s" }
67. Credentials Environment variables
68. Injecting credentials & env vars stage('Push Image') { environment { DOCKER_USERNAME = "${env.DOCKER_USERNAME}" DOCKER_PASSWORD = credentials('DOCKER_PASSWORD') DOCKER_EMAIL = "${env.DOCKER_EMAIL}" } steps { gradlew('dockerPushImage') } }
69. Scaling job execution
70. Configuration Execution
71. Containerized pipeline stages pipeline {
 agent {
 kubernetes {
 defaultContainer 'jnlp'
 yaml """
 apiVersion: v1
 kind: Pod
 metadata:
 labels:
 some-label: some-label-value
 spec:
 containers:
 - name: maven
 image: maven:alpine
 command:
 - cat
 tty: true
 - name: busybox
 image: busybox
 command:
 - cat
 tty: true
 """
 }
 }
 } stages {
 stage('Run in Maven container') {
 steps {
 container('maven') {
 sh 'mvn -version'
 }
 }
 }
 stage('Run in Busybox container') {
 steps {
 container('busybox') {
 sh '/bin/busybox'
 }
 }
 }
 stage('Run in default container') {
 steps {
 sh "echo 'Running in default container'"
 }
 }
 }
72. Demo Time
73. Resources github.com/bmuschko/todo-web-service github.com/bmuschko/todo-web-app github.com/bmuschko/jenkins-with-kubernetes
74. Thank you! Please ask questions…