$30 off During Our Annual Pro Sale. View Details »

Powering a build pipeline with Docker and Jenkins

Powering a build pipeline with Docker and Jenkins

Continuous Delivery is key to delivering software fast and frequently. Jenkins 2 has made great strides in providing a solution for modeling a build pipeline as code. In addition, Docker invoked from a Gradle build can help with implementing certain aspects of your pipeline. Combining the benefits of these tools makes it possible to implement a streamlined and bulletproof process.

In this demo-driven session, you will learn how to construct and operate a declarative pipeline with Jenkins. You’ll see a Spring Boot application taken all the way through the pipeline, from building the Docker image to pushing the image to a registry, using it for integration testing, and finally deploying the application to a Docker Swarm or Kubernetes.

Benjamin Muschko

September 16, 2019
Tweet

More Decks by Benjamin Muschko

Other Decks in Programming

Transcript

  1. Powering a build pipeline
    with Docker and Jenkins
    Benjamin Muschko

    View Slide

  2. AUTOMATED
    ASCENT
    bmuschko
    bmuschko
    bmuschko.com
    About the speaker
    automatedascent.com

    View Slide

  3. As a Java developer,
    I want to use Docker but…

    View Slide

  4. Find your Docker Zen

    View Slide

  5. Agenda
    Typical Continuous Delivery uses cases
    How can Docker help?
    Show me some code!
    The glue as code in Jenkins

    View Slide

  6. Typical workflows

    View Slide

  7. Dockerizing a Java application

    View Slide

  8. Pushing an image to a registry

    View Slide

  9. Image as fixture for testing

    View Slide

  10. Running application stacks

    View Slide

  11. Docker as part of the pipeline

    View Slide

  12. Docker’s CLI is great
    but do I need to operate it by hand?

    View Slide

  13. Pick the right tool for the job

    View Slide

  14. Docker CLI
    Other options

    View Slide

  15. Show me something
    beyond Hello World!

    View Slide

  16. Sample architecture whiteboard
    Client Web application
    Spring
    Boot
    Web service
    Spring
    Boot
    Database
    PostgreSQL

    View Slide

  17. Demo Time

    View Slide

  18. Containerization

    View Slide

  19. Pick a small base image

    View Slide

  20. Copy app archive or files?

    View Slide

  21. Controlling runtime behavior

    View Slide

  22. Is application healthy?

    View Slide

  23. Version appropriately
    4.2.6

    View Slide

  24. Dockerfile for Spring Boot app
    FROM openjdk:jre-alpine
    COPY todo-webservice-1.0.jar↵
    /app/todo-webservice-1.0.jar

    ENTRYPOINT ["java"]

    CMD ["-jar", "/app/todo-webservice-1.0.jar"]

    HEALTHCHECK CMD wget --quiet --tries=1 --spider↵
    http://localhost:8080/actuator/health || exit 1

    EXPOSE 8080

    View Slide

  25. Dockerfile with layers for app files
    FROM openjdk:jre-alpine
    WORKDIR /app
    COPY libs libs/
    COPY resources resources/
    COPY classes classes/
    ENTRYPOINT ["java", "-cp",↵
    "/app/resources:/app/classes:/app/libs/*",↵
    "com.bmuschko.MyApplication"]

    HEALTHCHECK CMD wget --quiet --tries=1 --spider↵
    http://localhost:8080/actuator/health || exit 1

    EXPOSE 8080

    View Slide

  26. Building and running image
    # Build image from Dockerfile

    docker build -t my-todo-web-service:1.0.0 .


    # Run built image in container

    docker run -d -p 8080:8080 my-todo-web-service:1.0.0

    View Slide

  27. Containerization with Google Jib
    Build images without the need
    for Docker Engine
    https://github.com/GoogleContainerTools/jib

    View Slide

  28. No need for Docker Engine

    View Slide

  29. FROM …
    COPY …
    EXPOSE …
    ENTRYPOINT …
    Dockerfile is optional

    View Slide

  30. Layers for cacheability

    View Slide

  31. Integration with build tools

    View Slide

  32. Google Jib in Maven build

    ...




    com.google.cloud.tools

    jib-maven-plugin

    1.6.0



    bmuschko/my-web-service:1.0.0






    $ ./mvnw compile jib:dockerBuild

    View Slide

  33. Google Jib in Gradle build
    plugins {
    id 'java'
    id 'com.google.cloud.tools.jib' version ‘1.6.0'
    }
    jib.to.image = 'bmuschko/my-web-service:1.0.0'
    $ ./gradlew jibDockerBuild

    View Slide

  34. Demo Time

    View Slide

  35. Pushing image
    # Log into Docker registry

    docker login --username=bmuschko↵
    [email protected]

    # Tag the image

    docker tag bb38976d03cf bmuschko/todo-web-service:1.0.0


    # Push image to Docker registry

    docker push bmuschko/todo-web-service

    View Slide

  36. Using Google Jib
    $ ./gradlew jib
    $ ./mvnw compile jib:build

    View Slide

  37. Demo Time

    View Slide

  38. Integrated Testing

    View Slide

  39. 2 unit tests, 0 integration tests

    View Slide

  40. Reproducible environment

    View Slide

  41. Slow startup times

    View Slide

  42. Isolated & cross-platform

    View Slide


  43. The obvious choice

    View Slide

  44. Pull image and start/stop container
    # Pull image from registry (or build locally)

    docker pull bmuschko/todo-web-service:1.0.0


    # Start container

    docker run -d -p 8080:8080 -name todo-web-service↵
    my-todo-web-service:1.0.0


    # Stop container

    docker container stop todo-web-service


    # Remove container

    docker container rm todo-web-service

    View Slide

  45. Testing with TestContainers
    Managing lightweight, throwaway
    Docker containers for testing
    https://www.testcontainers.org/

    View Slide

  46. Docker Engine Communication

    View Slide

  47. Docker Environment Discovery

    View Slide

  48. Automatic container cleanup

    View Slide

  49. Multi-container coordination

    View Slide

  50. Testcontainers in Maven build

    org.testcontainers

    testcontainers

    1.12.0

    test


    View Slide

  51. Testcontainers in Gradle build
    repositories {

    mavenCentral()

    }


    dependencies {

    testImplementation 'org.testcontainers:junit-jupiter:1.12.0'

    testImplementation 'org.testcontainers:postgresql:1.12.0'

    testRuntime 'org.postgresql:postgresql'

    }

    View Slide

  52. Integration test with JUnit 5
    @DataJpaTest

    public class ToDoRepositoryIntegrationTest {

    @Autowired

    private ToDoRepository repository;


    @Test

    public void testCanSaveNewToDoItem() {

    ToDoItem toDoItem = createToDoItem("Buy milk");

    assertNull(toDoItem.getId());

    repository.save(toDoItem);

    assertNotNull(toDoItem.getId());

    }


    private ToDoItem createToDoItem(String name) {

    ToDoItem toDoItem = new ToDoItem();

    toDoItem.setName(name);

    return toDoItem;

    }

    }
    Class under test

    View Slide

  53. Containerized database as fixture
    @Testcontainers

    @AutoConfigureTestDatabase(replace = NONE)

    @ContextConfiguration(initializers = { ToDoRepositoryIntegrationTest.Initializer.class })

    public class ToDoRepositoryIntegrationTest {

    @Container

    public static PostgreSQLContainer postgreSQLContainer = createDbContainer();


    private static PostgreSQLContainer createDbContainer() {

    return new PostgreSQLContainer("postgres:9.6.10-alpine")

    .withUsername("postgres")

    .withPassword("postgres")

    .withDatabaseName("todo");

    }


    static class Initializer implements ApplicationContextInitializer {

    @Override

    public void initialize(ConfigurableApplicationContext configurableApplicationContext) {

    TestPropertyValues.of(

    "spring.datasource.url=" + postgreSQLContainer.getJdbcUrl(),

    "spring.datasource.username=" + postgreSQLContainer.getUsername(),

    "spring.datasource.password=" + postgreSQLContainer.getPassword(),

    "spring.datasource.driver-class-name=org.postgresql.Driver",

    "spring.jpa.generate-ddl=true"

    ).applyTo(configurableApplicationContext.getEnvironment());

    }

    }

    }
    Create database container
    Inject database
    Configuration

    View Slide

  54. Functional test with JUnit 5
    public class ToDoWebServiceFunctionalTest {

    @Test

    @DisplayName("can retrieve all items before and after inserting new ones")

    void retrieveAllItems() {

    String allItems = getAllItems();

    assertEquals("[]", allItems);


    ToDoItem toDoItem = new ToDoItem();

    toDoItem.setName("Buy milk");

    toDoItem.setCompleted(false);

    insertItem(toDoItem);


    allItems = getAllItems();

    assertEquals("[{\"id\":1,\"name\":\"Buy milk\",\"completed\":false}]", allItems);

    }
    }
    Calls to HTTP endpoints

    View Slide

  55. Containerized application as fixture
    @Testcontainers

    public class ToDoWebServiceFunctionalTest {

    @Container

    private static GenericContainer appContainer = createContainer();


    private static GenericContainer createContainer() {

    return new GenericContainer(buildImageDockerfile())

    .withExposedPorts(8080)

    .waitingFor(Wait.forHttp("/actuator/health")

    .forStatusCode(200));

    }


    private static ImageFromDockerfile buildImageDockerfile() {

    return new ImageFromDockerfile()

    .withFileFromFile(ARCHIVE_NAME, new File(DISTRIBUTION_DIR, ARCHIVE_NAME))

    .withDockerfileFromBuilder(builder -> builder

    .from("openjdk:jre-alpine")

    .copy(ARCHIVE_NAME, "/app/" + ARCHIVE_NAME)

    .entryPoint("java", "-jar", "/app/" + ARCHIVE_NAME)

    .build());

    }

    }
    Build container on-the-fly

    View Slide

  56. Demo Time

    View Slide

  57. Running a multi-container app
    # Start composed apps

    docker-compose up


    # Stop composed apps

    docker-compose down

    View Slide

  58. Compose test with JUnit 5
    @ExtendWith(SpringExtension.class)
    @SpringBootTest

    public class ToDoAppIntegrationTest {

    @Autowired

    private ToDoService service;


    @Test

    public void canCreateNewItemAndRetrieveIt() {

    ToDoItem newItem = newItem("Buy milk");

    assertNull(newItem.getId());

    service.save(newItem);

    assertNotNull(newItem.getId());

    ToDoItem retrievedItem = service.findOne(newItem.getId());

    assertEquals(newItem, retrievedItem);

    }
    }
    Class under test

    View Slide

  59. Compose fixture
    @Testcontainers

    public class ToDoAppImplIntegrationTest {

    private static final Logger logger = LoggerFactory.getLogger(ToDoAppImplIntegrationTest.class);

    private final static String WEB_SERVICE_NAME = "webservice_1";

    private final static int WEB_SERVICE_PORT = 8080;

    private final static String DATABASE_NAME = "database_1";

    private final static int DATABASE_PORT = 5432;


    @Container

    public static DockerComposeContainer environment = createComposeContainer();


    private static DockerComposeContainer createComposeContainer() {

    return new DockerComposeContainer(new File("src/integrationTest/resources/compose-test.yml"))

    .withLogConsumer(WEB_SERVICE_NAME, new Slf4jLogConsumer(logger))

    .withExposedService(WEB_SERVICE_NAME, WEB_SERVICE_PORT,

    Wait.forHttp("/actuator/health")

    .forStatusCode(200))

    .withExposedService(DATABASE_NAME, DATABASE_PORT,

    Wait.forLogMessage(".*database system is ready to accept connections.*\\s", 2));

    }
    }
    Create Compose container

    View Slide

  60. Compose test file
    version: "2.0"

    services:

    database:

    image: "postgres:9.6.10-alpine"

    environment:

    - POSTGRES_USER=postgres

    - POSTGRES_PASSWORD=postgres

    - POSTGRES_DB=todo

    webservice:

    image: "bmuschko/todo-web-service:1.0.0"

    environment:

    - SPRING_PROFILES_ACTIVE=dev

    depends_on:

    - database

    View Slide

  61. Demo Time

    View Slide

  62. Glueing together the pipeline

    View Slide

  63. Jenkins plugins
    Optional: Blue Ocean
    Pipeline Suite

    View Slide

  64. Standard Pipeline
    Blue Ocean Pipeline

    View Slide

  65. Pipeline as code
    Jenkinsfile Jenkins Pipeline

    View Slide

  66. Invoke build with Wrapper
    pipeline {
    stages {
    stage('Compile & Unit Tests') {
    steps {
    gradlew('clean', 'test')
    }
    }
    }
    ...
    }
    def gradlew(String... args) {
    sh "./gradlew ${args.join(' ')} -s"
    }

    View Slide

  67. Credentials
    Environment variables

    View Slide

  68. Injecting credentials & env vars
    stage('Push Image') {
    environment {
    DOCKER_USERNAME = "${env.DOCKER_USERNAME}"
    DOCKER_PASSWORD = credentials('DOCKER_PASSWORD')
    DOCKER_EMAIL = "${env.DOCKER_EMAIL}"
    }
    steps {
    gradlew('dockerPushImage')
    }
    }

    View Slide

  69. Scaling job execution

    View Slide

  70. Configuration Execution

    View Slide

  71. Containerized pipeline stages
    pipeline {

    agent {

    kubernetes {

    defaultContainer 'jnlp'

    yaml """

    apiVersion: v1

    kind: Pod

    metadata:

    labels:

    some-label: some-label-value

    spec:

    containers:

    - name: maven

    image: maven:alpine

    command:

    - cat

    tty: true

    - name: busybox

    image: busybox

    command:

    - cat

    tty: true

    """

    }

    }

    }
    stages {

    stage('Run in Maven container') {

    steps {

    container('maven') {

    sh 'mvn -version'

    }

    }

    }

    stage('Run in Busybox container') {

    steps {

    container('busybox') {

    sh '/bin/busybox'

    }

    }

    }

    stage('Run in default container') {

    steps {

    sh "echo 'Running in default container'"

    }

    }

    }

    View Slide

  72. Demo Time

    View Slide

  73. Resources
    github.com/bmuschko/todo-web-service
    github.com/bmuschko/todo-web-app
    github.com/bmuschko/jenkins-with-kubernetes

    View Slide

  74. Thank you!
    Please ask questions…

    View Slide