Powering a build pipeline with Docker and Jenkins

Powering a build pipeline with Docker and Jenkins

Continuous Delivery is key to delivering software fast and frequently. Jenkins 2 has made great strides in providing a solution for modeling a build pipeline as code. In addition, Docker invoked from a Gradle build can help with implementing certain aspects of your pipeline. Combining the benefits of these tools makes it possible to implement a streamlined and bulletproof process.

In this demo-driven session, you will learn how to construct and operate a declarative pipeline with Jenkins. You’ll see a Spring Boot application taken all the way through the pipeline, from building the Docker image to pushing the image to a registry, using it for integration testing, and finally deploying the application to a Docker Swarm or Kubernetes.

8f2248c6bfcc6df39a2cd8edf4267cb5?s=128

Benjamin Muschko

September 16, 2019
Tweet

Transcript

  1. Powering a build pipeline with Docker and Jenkins Benjamin Muschko

  2. AUTOMATED ASCENT bmuschko bmuschko bmuschko.com About the speaker automatedascent.com

  3. As a Java developer, I want to use Docker but…

  4. Find your Docker Zen

  5. Agenda Typical Continuous Delivery uses cases How can Docker help?

    Show me some code! The glue as code in Jenkins
  6. Typical workflows

  7. Dockerizing a Java application

  8. Pushing an image to a registry

  9. Image as fixture for testing

  10. Running application stacks

  11. Docker as part of the pipeline

  12. Docker’s CLI is great but do I need to operate

    it by hand?
  13. Pick the right tool for the job

  14. Docker CLI Other options

  15. Show me something beyond Hello World!

  16. Sample architecture whiteboard Client Web application Spring Boot Web service

    Spring Boot Database PostgreSQL
  17. Demo Time

  18. Containerization

  19. Pick a small base image

  20. Copy app archive or files?

  21. Controlling runtime behavior

  22. Is application healthy?

  23. Version appropriately 4.2.6

  24. Dockerfile for Spring Boot app FROM openjdk:jre-alpine COPY todo-webservice-1.0.jar↵ /app/todo-webservice-1.0.jar


    ENTRYPOINT ["java"]
 CMD ["-jar", "/app/todo-webservice-1.0.jar"]
 HEALTHCHECK CMD wget --quiet --tries=1 --spider↵ http://localhost:8080/actuator/health || exit 1
 EXPOSE 8080
  25. Dockerfile with layers for app files FROM openjdk:jre-alpine WORKDIR /app

    COPY libs libs/ COPY resources resources/ COPY classes classes/ ENTRYPOINT ["java", "-cp",↵ "/app/resources:/app/classes:/app/libs/*",↵ "com.bmuschko.MyApplication"]
 HEALTHCHECK CMD wget --quiet --tries=1 --spider↵ http://localhost:8080/actuator/health || exit 1
 EXPOSE 8080
  26. Building and running image # Build image from Dockerfile
 docker

    build -t my-todo-web-service:1.0.0 .
 
 # Run built image in container
 docker run -d -p 8080:8080 my-todo-web-service:1.0.0
  27. Containerization with Google Jib Build images without the need for

    Docker Engine https://github.com/GoogleContainerTools/jib
  28. No need for Docker Engine

  29. FROM … COPY … EXPOSE … ENTRYPOINT … Dockerfile is

    optional
  30. Layers for cacheability

  31. Integration with build tools

  32. Google Jib in Maven build <project>
 ...
 <build>
 <plugins>
 <plugin>


    <groupId>com.google.cloud.tools</groupId>
 <artifactId>jib-maven-plugin</artifactId>
 <version>1.6.0</version>
 <configuration>
 <to>
 <image>bmuschko/my-web-service:1.0.0</image>
 </to>
 </configuration>
 </plugin>
 </plugins>
 </build>
 </project> $ ./mvnw compile jib:dockerBuild
  33. Google Jib in Gradle build plugins { id 'java' id

    'com.google.cloud.tools.jib' version ‘1.6.0' } jib.to.image = 'bmuschko/my-web-service:1.0.0' $ ./gradlew jibDockerBuild
  34. Demo Time

  35. Pushing image # Log into Docker registry
 docker login --username=bmuschko↵

    —email=benjamin.muschko@gmail.com
 
 # Tag the image
 docker tag bb38976d03cf bmuschko/todo-web-service:1.0.0
 
 # Push image to Docker registry
 docker push bmuschko/todo-web-service
  36. Using Google Jib $ ./gradlew jib $ ./mvnw compile jib:build

  37. Demo Time

  38. Integrated Testing

  39. 2 unit tests, 0 integration tests

  40. Reproducible environment

  41. Slow startup times

  42. Isolated & cross-platform

  43. The obvious choice

  44. Pull image and start/stop container # Pull image from registry

    (or build locally)
 docker pull bmuschko/todo-web-service:1.0.0
 
 # Start container
 docker run -d -p 8080:8080 -name todo-web-service↵ my-todo-web-service:1.0.0
 
 # Stop container
 docker container stop todo-web-service
 
 # Remove container
 docker container rm todo-web-service
  45. Testing with TestContainers Managing lightweight, throwaway Docker containers for testing

    https://www.testcontainers.org/
  46. Docker Engine Communication

  47. Docker Environment Discovery

  48. Automatic container cleanup

  49. Multi-container coordination

  50. Testcontainers in Maven build <dependency>
 <groupId>org.testcontainers</groupId>
 <artifactId>testcontainers</artifactId>
 <version>1.12.0</version>
 <scope>test</scope>
 </dependency>

  51. Testcontainers in Gradle build repositories {
 mavenCentral()
 }
 
 dependencies

    {
 testImplementation 'org.testcontainers:junit-jupiter:1.12.0'
 testImplementation 'org.testcontainers:postgresql:1.12.0'
 testRuntime 'org.postgresql:postgresql'
 }
  52. Integration test with JUnit 5 @DataJpaTest
 public class ToDoRepositoryIntegrationTest {


    @Autowired
 private ToDoRepository repository;
 
 @Test
 public void testCanSaveNewToDoItem() {
 ToDoItem toDoItem = createToDoItem("Buy milk");
 assertNull(toDoItem.getId());
 repository.save(toDoItem);
 assertNotNull(toDoItem.getId());
 }
 
 private ToDoItem createToDoItem(String name) {
 ToDoItem toDoItem = new ToDoItem();
 toDoItem.setName(name);
 return toDoItem;
 }
 } Class under test
  53. Containerized database as fixture @Testcontainers
 @AutoConfigureTestDatabase(replace = NONE)
 @ContextConfiguration(initializers =

    { ToDoRepositoryIntegrationTest.Initializer.class })
 public class ToDoRepositoryIntegrationTest {
 @Container
 public static PostgreSQLContainer postgreSQLContainer = createDbContainer();
 
 private static PostgreSQLContainer createDbContainer() {
 return new PostgreSQLContainer("postgres:9.6.10-alpine")
 .withUsername("postgres")
 .withPassword("postgres")
 .withDatabaseName("todo");
 }
 
 static class Initializer implements ApplicationContextInitializer<ConfigurableApplicationContext> {
 @Override
 public void initialize(ConfigurableApplicationContext configurableApplicationContext) {
 TestPropertyValues.of(
 "spring.datasource.url=" + postgreSQLContainer.getJdbcUrl(),
 "spring.datasource.username=" + postgreSQLContainer.getUsername(),
 "spring.datasource.password=" + postgreSQLContainer.getPassword(),
 "spring.datasource.driver-class-name=org.postgresql.Driver",
 "spring.jpa.generate-ddl=true"
 ).applyTo(configurableApplicationContext.getEnvironment());
 }
 }
 } Create database container Inject database Configuration
  54. Functional test with JUnit 5 public class ToDoWebServiceFunctionalTest {
 @Test


    @DisplayName("can retrieve all items before and after inserting new ones")
 void retrieveAllItems() {
 String allItems = getAllItems();
 assertEquals("[]", allItems);
 
 ToDoItem toDoItem = new ToDoItem();
 toDoItem.setName("Buy milk");
 toDoItem.setCompleted(false);
 insertItem(toDoItem);
 
 allItems = getAllItems();
 assertEquals("[{\"id\":1,\"name\":\"Buy milk\",\"completed\":false}]", allItems);
 } } Calls to HTTP endpoints
  55. Containerized application as fixture @Testcontainers
 public class ToDoWebServiceFunctionalTest {
 @Container


    private static GenericContainer appContainer = createContainer();
 
 private static GenericContainer createContainer() {
 return new GenericContainer(buildImageDockerfile())
 .withExposedPorts(8080)
 .waitingFor(Wait.forHttp("/actuator/health")
 .forStatusCode(200));
 }
 
 private static ImageFromDockerfile buildImageDockerfile() {
 return new ImageFromDockerfile()
 .withFileFromFile(ARCHIVE_NAME, new File(DISTRIBUTION_DIR, ARCHIVE_NAME))
 .withDockerfileFromBuilder(builder -> builder
 .from("openjdk:jre-alpine")
 .copy(ARCHIVE_NAME, "/app/" + ARCHIVE_NAME)
 .entryPoint("java", "-jar", "/app/" + ARCHIVE_NAME)
 .build());
 }
 } Build container on-the-fly
  56. Demo Time

  57. Running a multi-container app # Start composed apps
 docker-compose up


    
 # Stop composed apps
 docker-compose down
  58. Compose test with JUnit 5 @ExtendWith(SpringExtension.class) @SpringBootTest
 public class ToDoAppIntegrationTest

    {
 @Autowired
 private ToDoService service;
 
 @Test
 public void canCreateNewItemAndRetrieveIt() {
 ToDoItem newItem = newItem("Buy milk");
 assertNull(newItem.getId());
 service.save(newItem);
 assertNotNull(newItem.getId());
 ToDoItem retrievedItem = service.findOne(newItem.getId());
 assertEquals(newItem, retrievedItem);
 } } Class under test
  59. Compose fixture @Testcontainers
 public class ToDoAppImplIntegrationTest {
 private static final

    Logger logger = LoggerFactory.getLogger(ToDoAppImplIntegrationTest.class);
 private final static String WEB_SERVICE_NAME = "webservice_1";
 private final static int WEB_SERVICE_PORT = 8080;
 private final static String DATABASE_NAME = "database_1";
 private final static int DATABASE_PORT = 5432;
 
 @Container
 public static DockerComposeContainer environment = createComposeContainer();
 
 private static DockerComposeContainer createComposeContainer() {
 return new DockerComposeContainer(new File("src/integrationTest/resources/compose-test.yml"))
 .withLogConsumer(WEB_SERVICE_NAME, new Slf4jLogConsumer(logger))
 .withExposedService(WEB_SERVICE_NAME, WEB_SERVICE_PORT,
 Wait.forHttp("/actuator/health")
 .forStatusCode(200))
 .withExposedService(DATABASE_NAME, DATABASE_PORT,
 Wait.forLogMessage(".*database system is ready to accept connections.*\\s", 2));
 } } Create Compose container
  60. Compose test file version: "2.0"
 services:
 database:
 image: "postgres:9.6.10-alpine"
 environment:


    - POSTGRES_USER=postgres
 - POSTGRES_PASSWORD=postgres
 - POSTGRES_DB=todo
 webservice:
 image: "bmuschko/todo-web-service:1.0.0"
 environment:
 - SPRING_PROFILES_ACTIVE=dev
 depends_on:
 - database
  61. Demo Time

  62. Glueing together the pipeline

  63. Jenkins plugins Optional: Blue Ocean Pipeline Suite

  64. Standard Pipeline Blue Ocean Pipeline

  65. Pipeline as code Jenkinsfile Jenkins Pipeline

  66. Invoke build with Wrapper pipeline { stages { stage('Compile &

    Unit Tests') { steps { gradlew('clean', 'test') } } } ... } def gradlew(String... args) { sh "./gradlew ${args.join(' ')} -s" }
  67. Credentials Environment variables

  68. Injecting credentials & env vars stage('Push Image') { environment {

    DOCKER_USERNAME = "${env.DOCKER_USERNAME}" DOCKER_PASSWORD = credentials('DOCKER_PASSWORD') DOCKER_EMAIL = "${env.DOCKER_EMAIL}" } steps { gradlew('dockerPushImage') } }
  69. Scaling job execution

  70. Configuration Execution

  71. Containerized pipeline stages pipeline {
 agent {
 kubernetes {
 defaultContainer

    'jnlp'
 yaml """
 apiVersion: v1
 kind: Pod
 metadata:
 labels:
 some-label: some-label-value
 spec:
 containers:
 - name: maven
 image: maven:alpine
 command:
 - cat
 tty: true
 - name: busybox
 image: busybox
 command:
 - cat
 tty: true
 """
 }
 }
 } stages {
 stage('Run in Maven container') {
 steps {
 container('maven') {
 sh 'mvn -version'
 }
 }
 }
 stage('Run in Busybox container') {
 steps {
 container('busybox') {
 sh '/bin/busybox'
 }
 }
 }
 stage('Run in default container') {
 steps {
 sh "echo 'Running in default container'"
 }
 }
 }
  72. Demo Time

  73. Resources github.com/bmuschko/todo-web-service github.com/bmuschko/todo-web-app github.com/bmuschko/jenkins-with-kubernetes

  74. Thank you! Please ask questions…