Ship your containers fast with GitLab CI

This talk was persented during the Cloud Native and Kubernetes Karlsruhe Meetup (12/11/2018)

https://www.meetup.com/de-DE/Cloud-Native-and-Kubernetes-Karlsruhe/events/25607037

If you interested in more follow me: (https://twitter.com/solidnerd)

Build, Ship and Run your container from source to production with your own pipeline. In the era of cloud and dynamic infrastructure our integration and deployment pipelines for containers needs to be build up fast and easy as well. This talk will present concepts for integration and deployment of containers with GitLab CI. Create GitLab Runners for the different uses cases of containers and cloud native environments. We will also discuss a simple setup of your own CI/CD infrastructure that can be build up in less than 10 minutes.

Transcript

1. Ship containers fast with GitLab CI Niclas Mietz    niclas.mietz@bee42.com solidnerd solidnerd 1
2. CI, CD, CD? 2 . 1
3. Continuous Integration Software development practice where code is integrated regularly through a process that includes building and testing of the sourcecode 2 . 2
4. Continuous Delivery Implies to the CI a process to releasing integrated code with manual approvements into targeted software environments 2 . 3
5. Continuous Deployment Implies to the CI a process to releasing integrated code automatically into targeted software environments 2 . 4
6. What is GitLab ? An alternative to git hostings like GitHub GitHub Enterprise Gitea and others ... https://about.gitlab.com/ 3 . 1
7. Why more ? OpenCore (Sources of Community and Enterprise Edition are available for everyone) Track uni ed issues Being able to do Code Reviews directly in the UI Build CI/CD Pipeline into a single UI Works awlessly with Container Ecosystem like Docker or Kubernetes Automatic website generation (Pages) Every 22nd a new release, with new innovations and xes An API for Automation 3 . 2
8. Official and Unoffical GitLab Images (of cial) (unof cial) https://hub.docker.com/r/gitlab/gitlab-ce/ https://hub.docker.com/r/sameersbn/gitlab/ Currently sameersbn/gitlab is more cloud native agnostic 3 . 3
9. sameersbn/gitlab 12M+ Docker Hub Pulls 5,3k Stars on GitHub 336 releases on GitHub 143 contributors Big active Community https://github.com/sameersbn/docker-gitlab 3 . 4
10. Deep Dive Into GitLab 4
11. GitLab Container Architecture (sameersbn/gitlab) Unicor n (GitLab Rails) Postgr eSQL Sidekiq (GitLab Rails) GitLab Shell GitLab W orkorse NGINX GitLab Pages SSH HTTP HTTPS TCP 8090 TCP 8181 TCP 80, 443 TCP 8080 TCP 5432 Redis TCP 8080 TCP 22 TCP 6379 GitLab Container 5
12. GitLab CI 6 . 1
13. GitLab CI Compoments .gitlab-ci.yml de nes how the pipeline for a project looks like GitLab Runner executes the jobs that are de ned in the .gitlab-ci.yml GitLab itself as coordinator 6 . 2
14. GitLab CI Architecture 6 . 3
15. GitLab Runner GitLab Runner could be an isolated machine or an isolated virtual machine that picks up builds through the coordinator API of GitLab CI Con guration of the runner is placed in a le that is called config.toml 6 . 4
16. Shared vs. Specific Shared runners can run jobs for every project that has enabled shared runner Speci c runners run only for speci c jobs, they need to be enabled 6 . 5
17. Executors Executors are a special avor of a Runner De nes how a runner executes his jobs from GitLab 7 . 1
18. Executor Flavors Shell Parallels VirtualBox SSH 7 . 2
19. Executor with Container Flavor Docker Docker Machine (w/ autoscaling) Kubernetes 7 . 3
20. Docker Executor Docker Executor has three different ways to work with the Docker Engine Normal Wormhole DinD (Docker in Docker) 7 . 4
21. Normal 7 . 5
22. Wormhole 7 . 6
23. DinD (Docker in Docker) 7 . 7
24. Sharing your Artifacts (Images) Docker Distribution The Docker toolset to pack, ship, store, and deliver content 8 . 1
25. Simple registry deployment 8 . 2
26. GitLab Registry Authentication Workflow 8 . 3
27. Demo 9 . 1
28. Goal 9 . 2
29. Goal with Kubernetes 9 . 3
30. Using different ways to build java images 10 . 1
31. Simple Docker Build FROM maven:3.5.0-jdk-8 RUN mkdir -p /usr/src/app WORKDIR /usr/src/app ARG MAVEN_OPTS="-Dmaven.repo.local=.m2/repository -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jM COPY pom.xml /usr/src/app RUN mvn verify clean --fail-never COPY . /usr/src/app RUN mvn package \ && mkdir -p /usr/jar \ && mv /usr/src/app/target/helloworld-0.0.1-SNAPSHOT.jar /usr/jar/app.jar \ && rm -rf /usr/src/app/target/ ENTRYPOINT ["java", "-Djava.security.egd=file:/dev/./urandom","-jar", "/usr/jar/app.jar"] 10 . 2
32. Simple Docker Build (CI) stages: - docker build:docker-image: stage: docker image: docker:latest script: - docker login -u gitlab-ci-token -p ${CI_JOB_TOKEN} ${CI_REGISTRY} - docker image build -t ${CI_REGISTRY_IMAGE}:latest -t ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHA:0:5} . - docker image push ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHA:0:5} 10 . 3
33. Build Pattern (Build) # Dockerfile.build FROM maven:3.5.0-jdk-8 RUN mkdir -p /usr/src/app WORKDIR /usr/src/app ARG MAVEN_OPTS="-Dmaven.repo.local=.m2/repository -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jM COPY pom.xml /usr/src/app RUN mvn verify clean --fail-never # COPY now all sources COPY . /usr/src/app RUN mvn package 10 . 4
34. Build Pattern (Run) #Dockerfile.run FROM openjdk:8u131-jre-alpine COPY target/helloworld-0.0.1-SNAPSHOT.jar /app.jar EXPOSE 8080 ENTRYPOINT ["java", "-Djava.security.egd=file:/dev/./urandom","-jar"] CMD ["/app.jar"] 10 . 5
35. Build Pattern (CI) stages: - docker build:docker-image: stage: docker image: docker:latest script: - docker login -u gitlab-ci-token -p ${CI_JOB_TOKEN} ${CI_REGISTRY} - docker image build -t ${CI_REGISTRY_IMAGE}/java-build-img -f Dockerfile.build . - docker container create --name ${CI_PROJECT_NAME}-java-build-cont ${CI_REGISTRY_IMAGE}/java-build-img - mkdir -p $(pwd)/target/ - docker container cp ${CI_PROJECT_NAME}-java-build-cont:/usr/src/app/target/helloworld-0.0.1-SNAPSHOT.jar $(pwd) - docker container rm -f ${CI_PROJECT_NAME}-java-build-cont - docker image build -t ${CI_REGISTRY_IMAGE}:latest -t ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHA:0:5} -f Dockerfile.ru - docker image push ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHA:0:5} 10 . 6
36. Multistage Build FROM maven:3.5.0-jdk-8 as build RUN mkdir -p /usr/src/app WORKDIR /usr/src/app ARG MAVEN_OPTS="-Dmaven.repo.local=.m2/repository -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jM COPY pom.xml /usr/src/app RUN mvn verify clean --fail-never # COPY now all sources COPY . /usr/src/app RUN mvn package FROM openjdk:8u131-jre-alpine COPY --from=build /usr/src/app/target/helloworld-0.0.1-SNAPSHOT.jar /app.jar ENTRYPOINT ["java", "-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"] 10 . 7
37. Multistage Build (CI) stages: - docker build:docker-image: stage: docker image: docker:latest script: - docker login -u gitlab-ci-token -p ${CI_JOB_TOKEN} ${CI_REGISTRY} - docker image build -t ${CI_REGISTRY_IMAGE}:latest -t ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHA:0:5} . - docker image push ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHA:0:5} 10 . 8
38. Good Docker Images 11 . 1
39. Base Images Use a Base Image to controll your Packages in your Pipeline 11 . 2
40. Comparsion of Base Images image size centos:7 193MB ubuntu:16.04 118MB debian:stretch 100MB debian:strech-slim 55.3MB alpine:3.6 3.97MB bitnami/minideb 53.6MB 11 . 3
41.  1. Tags 2. Labels 3. Digest 11 . 4
42. Tags Using tags to determine which image will be used Pointing to a speci cy image that will be used to run the container 11 . 5
43. Tags docker container -it --rm run alpine docker container -it --rm run alpine:latest docker container -it --rm run alpine:3.7 docker container -it --rm run alpine:3.8 11 . 6
44. Combine Tags with CI System (GitLab CI) docker: stage: build image: docker:latest script: - docker image build -t $CI_REGISTRY_IMAGE -t $CI_REGISTRY_IMAGE:$(git rev-parse --short HEAD) . - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY - docker push $CI_REGISTRY_IMAGE:latest - docker push $CI_REGISTRY_IMAGE:$(git rev-parse --short HEAD) 11 . 7
45. Combine Tags with CI System (GitLab CI) docker: stage: build image: docker:latest script: - docker image build -t $CI_REGISTRY_IMAGE -t $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA:0:5} . - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY - docker push $CI_REGISTRY_IMAGE:latest - docker push $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA:0:5} 11 . 8
46. Label A label is key=value pair to add more meta-information to the ressource 11 . 9
47. Label Standards http://label-schema.org/rc1/ 11 . 10
48. Labels in Dockerfile ARG BUILD_DATE ARG VCS_REF ARG USER ARG REPO_URL ARG NAME ARG LICENSE LABEL org.label-schema.build-date=$BUILD_DATE \ org.label-schema.docker.dockerfile="/Dockerfile" \ org.label-schema.license="$LICENSE" \ org.label-schema.name="$NAME" \ org.label-schema.vendor="$USER" \ org.label-schema.url="https://github.com/solidnerd/goinc/" \ org.label-schema.vcs-ref=$VCS_REF \ org.label-schema.vcs-url="$REPO_URL" \ org.label-schema.vcs-type="git" 11 . 11
49. Using Labels in gitlab-ci.yml build: image: docker before_script: - apk add --no-cache util-linux pciutils usbutils coreutils binutils findutils grep script: - docker build --build-arg LICENSE="MIT" --build-arg NAME=$CI_PROJECT_NAME --build-arg BUILD_DATE="$(date --rfc- 11 . 12
50. Using Digests A content-addressable identi er for Images docker run namespace/image@sha256:digest #Dockerfile FROM namespace/image@sha256:digest 11 . 13
51. User Switch All containers are running in the normal case as root user 11 . 14
52. Add User Switch to Dockerfile FROM alpine:3.8 ENV GO_USER=go \ GO_HOME=/app \ GO_UID=1000 \ GO_GID=1000 RUN addgroup -S $GO_USER -g ${GO_GID} \ && adduser -S \ -g $GO_USER \ -h $GO_HOME \ -u ${GO_UID} \ $GO_USER USER ${GO_USER} WORKDIR ${GO_HOME} COPY mybinary ${GO_HOME}/mybinary ENTRYPOINT ["${GO_HOME}/mybinary"] 11 . 15
53. Use Scratch FROM scratch COPY mybinary /mybinary ENTRYPOINT ["/mybinary"] 11 . 16
54. Use a Healthcheck FROM maven:3.5.0-jdk-8 as build RUN mkdir -p /usr/src/app WORKDIR /usr/src/app ARG MAVEN_OPTS="-Dmaven.repo.local=.m2/repository -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jM COPY pom.xml /usr/src/app RUN mvn verify clean --fail-never # COPY now all sources COPY . /usr/src/app RUN mvn package FROM openjdk:8u131-jre-slim RUN apt-get update && apt-get install -y --no-install-recommends curl \ && rm -rf /var/lib/apt/lists/ COPY --from=build /usr/src/app/target/hellowrold-0.0.1-SNAPSHOT.jar /app.jar HEALTHCHECK --interval=15s --timeout=15s --retries=3 \ CMD curl -f http://localhost:8080/health || exit 1 ENTRYPOINT ["java", "-Djava.security.egd=file:/dev/./urandom","-jar", "/app.jar"] 11 . 17
55. Good Docker Images Own User (with User Switch) Contains only the application Labels to determine the content Healthcheck 12
56. Build Images without Docker 13 . 1
57. Another Approach with Kaniko Kaniko is an open source tool to build container images within containers can be Kubernetes or Docker without a daemon Builds every general Docker le The resulting Image after the build can be directly pushed to registry or exported as tarball It doesn't require any particular privilege for building the image Is able to remove timestamps to perfom repeatable builds 13 . 2
58. Kaniko with GitLab CI 13 . 3
59. Kaniko with GitLab CI (cont.) # .gitlab-ci.yml build:kaniko: stage: build image: name: gcr.io/kaniko-project/executor:debug-v0.6.0 entrypoint: ["/busybox/sh","-c"] before_script: - export KANIKO_DOCKERFILE="Dockerfile" - export KANIKO_DESTINATION="$CI_REGISTRY_IMAGE/$CI_JOB_NAME:$CI_COMMIT_REF_NAME" variables: KANIKO_DOCKERFILE: $CI_PROJECT_DIR/Dockerfile KANIKO_DESTINATION: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME script: - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\" - /kaniko/executor --context "$CI_PROJECT_DIR/$CI_JOB_NAME" --destination "$KANIKO_DESTINATION" 13 . 4
60. Open the doors for Q&A    niclas.mietz@bee42.com solidnerd solidnerd 14