Powering a build pipeline with Docker and Jenkins

Continuous Delivery is key to delivering software fast and frequently. Jenkins 2 has made great strides in providing a solution for modeling a build pipeline as code. In addition, Docker invoked from a Gradle build can help with implementing certain aspects of your pipeline. Combining the benefits of these tools makes it possible to implement a streamlined and bulletproof process.

In this demo-driven session, you will learn how to construct and operate a declarative pipeline with Jenkins. You’ll see a Spring Boot application taken all the way through the pipeline, from building the Docker image to pushing the image to a registry, using it for integration testing, and finally deploying the application to a Docker Swarm or Kubernetes.

Transcript

1. Powering a build pipeline
   with Docker and Jenkins
   Benjamin Muschko
   

   View Slide

2. AUTOMATED
   ASCENT
   bmuschko
   bmuschko
   bmuschko.com
   About the speaker
   automatedascent.com
   

   View Slide

3. As a Java developer,
   I want to use Docker but…
   

   View Slide

4. Find your Docker Zen
   

   View Slide

5. Agenda
   Typical Continuous Delivery uses cases
   How can Docker help?
   Show me some code!
   The glue as code in Jenkins
   

   View Slide

6. Typical workflows
   

   View Slide

7. Dockerizing a Java application
   

   View Slide

8. Pushing an image to a registry
   

   View Slide

9. Image as fixture for testing
   

   View Slide

10. Running application stacks
    

    View Slide

11. Docker as part of the pipeline
    

    View Slide

12. Docker’s CLI is great
    but do I need to operate it by hand?
    

    View Slide

13. Pick the right tool for the job
    

    View Slide

14. Docker CLI
    Other options
    

    View Slide

15. Show me something
    beyond Hello World!
    

    View Slide

16. Sample architecture whiteboard
    Client Web application
    Spring
    Boot
    Web service
    Spring
    Boot
    Database
    PostgreSQL
    

    View Slide

17. Demo Time
    

    View Slide

18. Containerization
    

    View Slide

19. Pick a small base image
    

    View Slide

20. Copy app archive or files?
    

    View Slide

21. Controlling runtime behavior
    

    View Slide

22. Is application healthy?
    

    View Slide

23. Version appropriately
    4.2.6
    

    View Slide

24. Dockerfile for Spring Boot app
    FROM openjdk:jre-alpine
    COPY todo-webservice-1.0.jar↵
    /app/todo-webservice-1.0.jar

    ENTRYPOINT ["java"]

    CMD ["-jar", "/app/todo-webservice-1.0.jar"]

    HEALTHCHECK CMD wget --quiet --tries=1 --spider↵
    http://localhost:8080/actuator/health || exit 1

    EXPOSE 8080
    

    View Slide

25. Dockerfile with layers for app files
    FROM openjdk:jre-alpine
    WORKDIR /app
    COPY libs libs/
    COPY resources resources/
    COPY classes classes/
    ENTRYPOINT ["java", "-cp",↵
    "/app/resources:/app/classes:/app/libs/*",↵
    "com.bmuschko.MyApplication"]

    HEALTHCHECK CMD wget --quiet --tries=1 --spider↵
    http://localhost:8080/actuator/health || exit 1

    EXPOSE 8080
    

    View Slide

26. Building and running image
    # Build image from Dockerfile

    docker build -t my-todo-web-service:1.0.0 .

    

    # Run built image in container

    docker run -d -p 8080:8080 my-todo-web-service:1.0.0
    

    View Slide

27. Containerization with Google Jib
    Build images without the need
    for Docker Engine
    https://github.com/GoogleContainerTools/jib
    

    View Slide

28. No need for Docker Engine
    

    View Slide

29. FROM …
    COPY …
    EXPOSE …
    ENTRYPOINT …
    Dockerfile is optional
    

    View Slide

30. Layers for cacheability
    

    View Slide

31. Integration with build tools
    

    View Slide

32. Google Jib in Maven build
    

    ...

    

    

    

    com.google.cloud.tools

    jib-maven-plugin

    1.6.0

    

    

    bmuschko/my-web-service:1.0.0

    

    

    

    

    

    $ ./mvnw compile jib:dockerBuild
    

    View Slide

33. Google Jib in Gradle build
    plugins {
    id 'java'
    id 'com.google.cloud.tools.jib' version ‘1.6.0'
    }
    jib.to.image = 'bmuschko/my-web-service:1.0.0'
    $ ./gradlew jibDockerBuild
    

    View Slide

34. Demo Time
    

    View Slide

35. Pushing image
    # Log into Docker registry

    docker login --username=bmuschko↵
    —email=benjamin.muschko@gmail.com

    

    # Tag the image

    docker tag bb38976d03cf bmuschko/todo-web-service:1.0.0

    

    # Push image to Docker registry

    docker push bmuschko/todo-web-service
    

    View Slide

36. Using Google Jib
    $ ./gradlew jib
    $ ./mvnw compile jib:build
    

    View Slide

37. Demo Time
    

    View Slide

38. Integrated Testing
    

    View Slide

39. 2 unit tests, 0 integration tests
    

    View Slide

40. Reproducible environment
    

    View Slide

41. Slow startup times
    

    View Slide

42. Isolated & cross-platform
    

    View Slide

43. 
    The obvious choice
    

    View Slide

44. Pull image and start/stop container
    # Pull image from registry (or build locally)

    docker pull bmuschko/todo-web-service:1.0.0

    

    # Start container

    docker run -d -p 8080:8080 -name todo-web-service↵
    my-todo-web-service:1.0.0

    

    # Stop container

    docker container stop todo-web-service

    

    # Remove container

    docker container rm todo-web-service
    

    View Slide

45. Testing with TestContainers
    Managing lightweight, throwaway
    Docker containers for testing
    https://www.testcontainers.org/
    

    View Slide

46. Docker Engine Communication
    

    View Slide

47. Docker Environment Discovery
    

    View Slide

48. Automatic container cleanup
    

    View Slide

49. Multi-container coordination
    

    View Slide

50. Testcontainers in Maven build
    

    org.testcontainers

    testcontainers

    1.12.0

    test

    
    

    View Slide

51. Testcontainers in Gradle build
    repositories {

    mavenCentral()

    }

    

    dependencies {

    testImplementation 'org.testcontainers:junit-jupiter:1.12.0'

    testImplementation 'org.testcontainers:postgresql:1.12.0'

    testRuntime 'org.postgresql:postgresql'

    }
    

    View Slide

52. Integration test with JUnit 5
    @DataJpaTest

    public class ToDoRepositoryIntegrationTest {

    @Autowired

    private ToDoRepository repository;

    

    @Test

    public void testCanSaveNewToDoItem() {

    ToDoItem toDoItem = createToDoItem("Buy milk");

    assertNull(toDoItem.getId());

    repository.save(toDoItem);

    assertNotNull(toDoItem.getId());

    }

    

    private ToDoItem createToDoItem(String name) {

    ToDoItem toDoItem = new ToDoItem();

    toDoItem.setName(name);

    return toDoItem;

    }

    }
    Class under test
    

    View Slide

53. Containerized database as fixture
    @Testcontainers

    @AutoConfigureTestDatabase(replace = NONE)

    @ContextConfiguration(initializers = { ToDoRepositoryIntegrationTest.Initializer.class })

    public class ToDoRepositoryIntegrationTest {

    @Container

    public static PostgreSQLContainer postgreSQLContainer = createDbContainer();

    

    private static PostgreSQLContainer createDbContainer() {

    return new PostgreSQLContainer("postgres:9.6.10-alpine")

    .withUsername("postgres")

    .withPassword("postgres")

    .withDatabaseName("todo");

    }

    

    static class Initializer implements ApplicationContextInitializer {

    @Override

    public void initialize(ConfigurableApplicationContext configurableApplicationContext) {

    TestPropertyValues.of(

    "spring.datasource.url=" + postgreSQLContainer.getJdbcUrl(),

    "spring.datasource.username=" + postgreSQLContainer.getUsername(),

    "spring.datasource.password=" + postgreSQLContainer.getPassword(),

    "spring.datasource.driver-class-name=org.postgresql.Driver",

    "spring.jpa.generate-ddl=true"

    ).applyTo(configurableApplicationContext.getEnvironment());

    }

    }

    }
    Create database container
    Inject database
    Configuration
    

    View Slide

54. Functional test with JUnit 5
    public class ToDoWebServiceFunctionalTest {

    @Test

    @DisplayName("can retrieve all items before and after inserting new ones")

    void retrieveAllItems() {

    String allItems = getAllItems();

    assertEquals("[]", allItems);

    

    ToDoItem toDoItem = new ToDoItem();

    toDoItem.setName("Buy milk");

    toDoItem.setCompleted(false);

    insertItem(toDoItem);

    

    allItems = getAllItems();

    assertEquals("[{\"id\":1,\"name\":\"Buy milk\",\"completed\":false}]", allItems);

    }
    }
    Calls to HTTP endpoints
    

    View Slide

55. Containerized application as fixture
    @Testcontainers

    public class ToDoWebServiceFunctionalTest {

    @Container

    private static GenericContainer appContainer = createContainer();

    

    private static GenericContainer createContainer() {

    return new GenericContainer(buildImageDockerfile())

    .withExposedPorts(8080)

    .waitingFor(Wait.forHttp("/actuator/health")

    .forStatusCode(200));

    }

    

    private static ImageFromDockerfile buildImageDockerfile() {

    return new ImageFromDockerfile()

    .withFileFromFile(ARCHIVE_NAME, new File(DISTRIBUTION_DIR, ARCHIVE_NAME))

    .withDockerfileFromBuilder(builder -> builder

    .from("openjdk:jre-alpine")

    .copy(ARCHIVE_NAME, "/app/" + ARCHIVE_NAME)

    .entryPoint("java", "-jar", "/app/" + ARCHIVE_NAME)

    .build());

    }

    }
    Build container on-the-fly
    

    View Slide

56. Demo Time
    

    View Slide

57. Running a multi-container app
    # Start composed apps

    docker-compose up

    

    # Stop composed apps

    docker-compose down
    

    View Slide

58. Compose test with JUnit 5
    @ExtendWith(SpringExtension.class)
    @SpringBootTest

    public class ToDoAppIntegrationTest {

    @Autowired

    private ToDoService service;

    

    @Test

    public void canCreateNewItemAndRetrieveIt() {

    ToDoItem newItem = newItem("Buy milk");

    assertNull(newItem.getId());

    service.save(newItem);

    assertNotNull(newItem.getId());

    ToDoItem retrievedItem = service.findOne(newItem.getId());

    assertEquals(newItem, retrievedItem);

    }
    }
    Class under test
    

    View Slide

59. Compose fixture
    @Testcontainers

    public class ToDoAppImplIntegrationTest {

    private static final Logger logger = LoggerFactory.getLogger(ToDoAppImplIntegrationTest.class);

    private final static String WEB_SERVICE_NAME = "webservice_1";

    private final static int WEB_SERVICE_PORT = 8080;

    private final static String DATABASE_NAME = "database_1";

    private final static int DATABASE_PORT = 5432;

    

    @Container

    public static DockerComposeContainer environment = createComposeContainer();

    

    private static DockerComposeContainer createComposeContainer() {

    return new DockerComposeContainer(new File("src/integrationTest/resources/compose-test.yml"))

    .withLogConsumer(WEB_SERVICE_NAME, new Slf4jLogConsumer(logger))

    .withExposedService(WEB_SERVICE_NAME, WEB_SERVICE_PORT,

    Wait.forHttp("/actuator/health")

    .forStatusCode(200))

    .withExposedService(DATABASE_NAME, DATABASE_PORT,

    Wait.forLogMessage(".*database system is ready to accept connections.*\\s", 2));

    }
    }
    Create Compose container
    

    View Slide

60. Compose test file
    version: "2.0"

    services:

    database:

    image: "postgres:9.6.10-alpine"

    environment:

    - POSTGRES_USER=postgres

    - POSTGRES_PASSWORD=postgres

    - POSTGRES_DB=todo

    webservice:

    image: "bmuschko/todo-web-service:1.0.0"

    environment:

    - SPRING_PROFILES_ACTIVE=dev

    depends_on:

    - database
    

    View Slide

61. Demo Time
    

    View Slide

62. Glueing together the pipeline
    

    View Slide

63. Jenkins plugins
    Optional: Blue Ocean
    Pipeline Suite
    

    View Slide

64. Standard Pipeline
    Blue Ocean Pipeline
    

    View Slide

65. Pipeline as code
    Jenkinsfile Jenkins Pipeline
    

    View Slide

66. Invoke build with Wrapper
    pipeline {
    stages {
    stage('Compile & Unit Tests') {
    steps {
    gradlew('clean', 'test')
    }
    }
    }
    ...
    }
    def gradlew(String... args) {
    sh "./gradlew ${args.join(' ')} -s"
    }
    

    View Slide

67. Credentials
    Environment variables
    

    View Slide

68. Injecting credentials & env vars
    stage('Push Image') {
    environment {
    DOCKER_USERNAME = "${env.DOCKER_USERNAME}"
    DOCKER_PASSWORD = credentials('DOCKER_PASSWORD')
    DOCKER_EMAIL = "${env.DOCKER_EMAIL}"
    }
    steps {
    gradlew('dockerPushImage')
    }
    }
    

    View Slide

69. Scaling job execution
    

    View Slide

70. Configuration Execution
    

    View Slide

71. Containerized pipeline stages
    pipeline {

    agent {

    kubernetes {

    defaultContainer 'jnlp'

    yaml """

    apiVersion: v1

    kind: Pod

    metadata:

    labels:

    some-label: some-label-value

    spec:

    containers:

    - name: maven

    image: maven:alpine

    command:

    - cat

    tty: true

    - name: busybox

    image: busybox

    command:

    - cat

    tty: true

    """

    }

    }

    }
    stages {

    stage('Run in Maven container') {

    steps {

    container('maven') {

    sh 'mvn -version'

    }

    }

    }

    stage('Run in Busybox container') {

    steps {

    container('busybox') {

    sh '/bin/busybox'

    }

    }

    }

    stage('Run in default container') {

    steps {

    sh "echo 'Running in default container'"

    }

    }

    }
    

    View Slide

72. Demo Time
    

    View Slide

73. Resources
    github.com/bmuschko/todo-web-service
    github.com/bmuschko/todo-web-app
    github.com/bmuschko/jenkins-with-kubernetes
    

    View Slide

74. Thank you!
    Please ask questions…
    

    View Slide