Ship your containers fast with GitLab CI

This talk was persented during the Cloud Native and Kubernetes Karlsruhe Meetup (12/11/2018)

https://www.meetup.com/de-DE/Cloud-Native-and-Kubernetes-Karlsruhe/events/25607037

If you interested in more follow me: (https://twitter.com/solidnerd)

Build, Ship and Run your container from source to production with your own pipeline. In the era of cloud and dynamic infrastructure our integration and deployment pipelines for containers needs to be build up fast and easy as well. This talk will present concepts for integration and deployment of containers with GitLab CI. Create GitLab Runners for the different uses cases of containers and cloud native environments. We will also discuss a simple setup of your own CI/CD infrastructure that can be build up in less than 10 minutes.

Transcript

1. Ship containers fast with
   GitLab CI
   Niclas Mietz
   
   
   
   niclas.mietz@bee42.com
   solidnerd
   solidnerd
   1
   

   View Slide

2. CI, CD, CD?
   2 . 1
   

   View Slide

3. Continuous Integration
   Software development practice where code is integrated
   regularly through a process that includes building and testing
   of the sourcecode
   2 . 2
   

   View Slide

4. Continuous Delivery
   Implies to the CI a process to releasing integrated code with
   manual approvements into targeted software environments
   2 . 3
   

   View Slide

5. Continuous Deployment
   Implies to the CI a process to releasing integrated code
   automatically into targeted software environments
   2 . 4
   

   View Slide

6. What is GitLab ?
   An alternative to git hostings like
   GitHub
   GitHub Enterprise
   Gitea
   and others ...
   https://about.gitlab.com/
   3 . 1
   

   View Slide

7. Why more ?
   OpenCore (Sources of Community and Enterprise Edition are
   available for everyone)
   Track uni ed issues
   Being able to do Code Reviews directly in the UI
   Build CI/CD Pipeline into a single UI
   Works awlessly with Container Ecosystem like Docker or
   Kubernetes
   Automatic website generation (Pages)
   Every 22nd a new release, with new innovations and xes
   An API for Automation
   3 . 2
   

   View Slide

8. Official and Unoffical GitLab
   Images
   (of cial)
   (unof cial)
   https://hub.docker.com/r/gitlab/gitlab-ce/
   https://hub.docker.com/r/sameersbn/gitlab/
   Currently sameersbn/gitlab is more cloud
   native agnostic
   3 . 3
   

   View Slide

9. sameersbn/gitlab
   12M+ Docker Hub Pulls
   5,3k Stars on GitHub
   336 releases on GitHub
   143 contributors
   Big active Community
   https://github.com/sameersbn/docker-gitlab
   3 . 4
   

   View Slide

10. Deep Dive Into
    GitLab
    4
    

    View Slide

11. GitLab Container Architecture
    (sameersbn/gitlab)
    Unicor n
    (GitLab Rails)
    Postgr eSQL
    Sidekiq
    (GitLab Rails)
    GitLab Shell GitLab W orkorse NGINX
    GitLab Pages
    SSH
    HTTP
    HTTPS
    TCP 8090
    TCP 8181
    TCP 80, 443
    TCP 8080
    TCP 5432
    Redis
    TCP 8080
    TCP 22
    TCP 6379
    GitLab Container
    5
    

    View Slide

12. GitLab CI
    6 . 1
    

    View Slide

13. GitLab CI Compoments
    .gitlab-ci.yml de nes how the pipeline for a project
    looks like
    GitLab Runner executes the jobs that are de ned in the
    .gitlab-ci.yml
    GitLab itself as coordinator
    6 . 2
    

    View Slide

14. GitLab CI Architecture
    6 . 3
    

    View Slide

15. GitLab Runner
    GitLab Runner could be an isolated machine or an isolated
    virtual machine that picks up builds through the coordinator
    API of GitLab CI
    Con guration of the runner is placed in a le that is called
    config.toml
    6 . 4
    

    View Slide

16. Shared vs. Specific
    Shared runners can run jobs for every project that has
    enabled shared runner
    Speci c runners run only for speci c jobs, they need to be
    enabled
    6 . 5
    

    View Slide

17. Executors
    Executors are a special avor of a Runner
    De nes how a runner executes his jobs from GitLab
    7 . 1
    

    View Slide

18. Executor Flavors
    Shell
    Parallels
    VirtualBox
    SSH
    7 . 2
    

    View Slide

19. Executor with Container Flavor
    Docker
    Docker Machine (w/ autoscaling)
    Kubernetes
    7 . 3
    

    View Slide

20. Docker Executor
    Docker Executor has three different ways to work with the
    Docker Engine
    Normal
    Wormhole
    DinD (Docker in Docker)
    7 . 4
    

    View Slide

21. Normal
    7 . 5
    

    View Slide

22. Wormhole
    7 . 6
    

    View Slide

23. DinD (Docker in Docker)
    7 . 7
    

    View Slide

24. Sharing your Artifacts (Images)
    Docker Distribution
    The Docker toolset to pack, ship, store, and
    deliver content
    8 . 1
    

    View Slide

25. Simple registry deployment
    8 . 2
    

    View Slide

26. GitLab Registry Authentication
    Workflow
    8 . 3
    

    View Slide

27. Demo
    9 . 1
    

    View Slide

28. Goal
    9 . 2
    

    View Slide

29. Goal with Kubernetes
    9 . 3
    

    View Slide

30. Using different ways to build
    java images
    10 . 1
    

    View Slide

31. Simple Docker Build
    FROM maven:3.5.0-jdk-8
    RUN mkdir -p /usr/src/app
    WORKDIR /usr/src/app
    ARG MAVEN_OPTS="-Dmaven.repo.local=.m2/repository -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jM
    COPY pom.xml /usr/src/app
    RUN mvn verify clean --fail-never
    COPY . /usr/src/app
    RUN mvn package \
    && mkdir -p /usr/jar \
    && mv /usr/src/app/target/helloworld-0.0.1-SNAPSHOT.jar /usr/jar/app.jar \
    && rm -rf /usr/src/app/target/
    ENTRYPOINT ["java", "-Djava.security.egd=file:/dev/./urandom","-jar", "/usr/jar/app.jar"]
    10 . 2
    

    View Slide

32. Simple Docker Build (CI)
    stages:
    - docker
    build:docker-image:
    stage: docker
    image: docker:latest
    script:
    - docker login -u gitlab-ci-token -p ${CI_JOB_TOKEN} ${CI_REGISTRY}
    - docker image build -t ${CI_REGISTRY_IMAGE}:latest -t ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHA:0:5} .
    - docker image push ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHA:0:5}
    10 . 3
    

    View Slide

33. Build Pattern (Build)
    # Dockerfile.build
    FROM maven:3.5.0-jdk-8
    RUN mkdir -p /usr/src/app
    WORKDIR /usr/src/app
    ARG MAVEN_OPTS="-Dmaven.repo.local=.m2/repository -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jM
    COPY pom.xml /usr/src/app
    RUN mvn verify clean --fail-never
    # COPY now all sources
    COPY . /usr/src/app
    RUN mvn package
    10 . 4
    

    View Slide

34. Build Pattern (Run)
    #Dockerfile.run
    FROM openjdk:8u131-jre-alpine
    COPY target/helloworld-0.0.1-SNAPSHOT.jar /app.jar
    EXPOSE 8080
    ENTRYPOINT ["java", "-Djava.security.egd=file:/dev/./urandom","-jar"]
    CMD ["/app.jar"]
    10 . 5
    

    View Slide

35. Build Pattern (CI)
    stages:
    - docker
    build:docker-image:
    stage: docker
    image: docker:latest
    script:
    - docker login -u gitlab-ci-token -p ${CI_JOB_TOKEN} ${CI_REGISTRY}
    - docker image build -t ${CI_REGISTRY_IMAGE}/java-build-img -f Dockerfile.build .
    - docker container create --name ${CI_PROJECT_NAME}-java-build-cont ${CI_REGISTRY_IMAGE}/java-build-img
    - mkdir -p $(pwd)/target/
    - docker container cp ${CI_PROJECT_NAME}-java-build-cont:/usr/src/app/target/helloworld-0.0.1-SNAPSHOT.jar $(pwd)
    - docker container rm -f ${CI_PROJECT_NAME}-java-build-cont
    - docker image build -t ${CI_REGISTRY_IMAGE}:latest -t ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHA:0:5} -f Dockerfile.ru
    - docker image push ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHA:0:5}
    10 . 6
    

    View Slide

36. Multistage Build
    FROM maven:3.5.0-jdk-8 as build
    RUN mkdir -p /usr/src/app
    WORKDIR /usr/src/app
    ARG MAVEN_OPTS="-Dmaven.repo.local=.m2/repository -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jM
    COPY pom.xml /usr/src/app
    RUN mvn verify clean --fail-never
    # COPY now all sources
    COPY . /usr/src/app
    RUN mvn package
    FROM openjdk:8u131-jre-alpine
    COPY --from=build /usr/src/app/target/helloworld-0.0.1-SNAPSHOT.jar /app.jar
    ENTRYPOINT ["java", "-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
    10 . 7
    

    View Slide

37. Multistage Build (CI)
    stages:
    - docker
    build:docker-image:
    stage: docker
    image: docker:latest
    script:
    - docker login -u gitlab-ci-token -p ${CI_JOB_TOKEN} ${CI_REGISTRY}
    - docker image build -t ${CI_REGISTRY_IMAGE}:latest -t ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHA:0:5} .
    - docker image push ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHA:0:5}
    10 . 8
    

    View Slide

38. Good Docker Images
    11 . 1
    

    View Slide

39. Base Images
    Use a Base Image to controll your Packages in your Pipeline
    11 . 2
    

    View Slide

40. Comparsion of Base Images
    image size
    centos:7 193MB
    ubuntu:16.04 118MB
    debian:stretch 100MB
    debian:strech-slim 55.3MB
    alpine:3.6 3.97MB
    bitnami/minideb 53.6MB
    11 . 3
    

    View Slide

41. 
    1. Tags
    2. Labels
    3. Digest
    11 . 4
    

    View Slide

42. Tags
    Using tags to determine which image will be used
    Pointing to a speci cy image that will be used to run the
    container
    11 . 5
    

    View Slide

43. Tags
    docker container -it --rm run alpine
    docker container -it --rm run alpine:latest
    docker container -it --rm run alpine:3.7
    docker container -it --rm run alpine:3.8
    11 . 6
    

    View Slide

44. Combine Tags with CI System
    (GitLab CI)
    docker:
    stage: build
    image: docker:latest
    script:
    - docker image build -t $CI_REGISTRY_IMAGE -t $CI_REGISTRY_IMAGE:$(git rev-parse --short HEAD) .
    - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY
    - docker push $CI_REGISTRY_IMAGE:latest
    - docker push $CI_REGISTRY_IMAGE:$(git rev-parse --short HEAD)
    11 . 7
    

    View Slide

45. Combine Tags with CI System
    (GitLab CI)
    docker:
    stage: build
    image: docker:latest
    script:
    - docker image build -t $CI_REGISTRY_IMAGE -t $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA:0:5} .
    - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY
    - docker push $CI_REGISTRY_IMAGE:latest
    - docker push $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA:0:5}
    11 . 8
    

    View Slide

46. Label
    A label is key=value pair to add more meta-information to the
    ressource
    11 . 9
    

    View Slide

47. Label Standards
    http://label-schema.org/rc1/
    11 . 10
    

    View Slide

48. Labels in Dockerfile
    ARG BUILD_DATE
    ARG VCS_REF
    ARG USER
    ARG REPO_URL
    ARG NAME
    ARG LICENSE
    LABEL org.label-schema.build-date=$BUILD_DATE \
    org.label-schema.docker.dockerfile="/Dockerfile" \
    org.label-schema.license="$LICENSE" \
    org.label-schema.name="$NAME" \
    org.label-schema.vendor="$USER" \
    org.label-schema.url="https://github.com/solidnerd/goinc/" \
    org.label-schema.vcs-ref=$VCS_REF \
    org.label-schema.vcs-url="$REPO_URL" \
    org.label-schema.vcs-type="git"
    11 . 11
    

    View Slide

49. Using Labels in gitlab-ci.yml
    build:
    image: docker
    before_script:
    - apk add --no-cache util-linux pciutils usbutils coreutils binutils findutils grep
    script:
    - docker build --build-arg LICENSE="MIT" --build-arg NAME=$CI_PROJECT_NAME --build-arg BUILD_DATE="$(date --rfc-
    11 . 12
    

    View Slide

50. Using Digests
    A content-addressable identi er for Images
    docker run namespace/image@sha256:digest
    #Dockerfile
    FROM namespace/image@sha256:digest
    11 . 13
    

    View Slide

51. User Switch
    All containers are running in the normal case as root user
    11 . 14
    

    View Slide

52. Add User Switch to Dockerfile
    FROM alpine:3.8
    ENV GO_USER=go \
    GO_HOME=/app \
    GO_UID=1000 \
    GO_GID=1000
    RUN addgroup -S $GO_USER -g ${GO_GID} \
    && adduser -S \
    -g $GO_USER \
    -h $GO_HOME \
    -u ${GO_UID} \
    $GO_USER
    USER ${GO_USER}
    WORKDIR ${GO_HOME}
    COPY mybinary ${GO_HOME}/mybinary
    ENTRYPOINT ["${GO_HOME}/mybinary"]
    11 . 15
    

    View Slide

53. Use Scratch
    FROM scratch
    COPY mybinary /mybinary
    ENTRYPOINT ["/mybinary"]
    11 . 16
    

    View Slide

54. Use a Healthcheck
    FROM maven:3.5.0-jdk-8 as build
    RUN mkdir -p /usr/src/app
    WORKDIR /usr/src/app
    ARG MAVEN_OPTS="-Dmaven.repo.local=.m2/repository -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jM
    COPY pom.xml /usr/src/app
    RUN mvn verify clean --fail-never
    # COPY now all sources
    COPY . /usr/src/app
    RUN mvn package
    FROM openjdk:8u131-jre-slim
    RUN apt-get update && apt-get install -y --no-install-recommends curl \
    && rm -rf /var/lib/apt/lists/
    COPY --from=build /usr/src/app/target/hellowrold-0.0.1-SNAPSHOT.jar /app.jar
    HEALTHCHECK --interval=15s --timeout=15s --retries=3 \
    CMD curl -f http://localhost:8080/health || exit 1
    ENTRYPOINT ["java", "-Djava.security.egd=file:/dev/./urandom","-jar", "/app.jar"]
    11 . 17
    

    View Slide

55. Good Docker Images
    Own User (with User Switch)
    Contains only the application
    Labels to determine the content
    Healthcheck
    12
    

    View Slide

56. Build Images without Docker
    13 . 1
    

    View Slide

57. Another Approach with Kaniko
    Kaniko is an open source tool to build container images
    within containers
    can be Kubernetes or Docker without a daemon
    Builds every general Docker le
    The resulting Image after the build can be directly pushed to
    registry or exported as tarball
    It doesn't require any particular privilege for building the
    image
    Is able to remove timestamps to perfom repeatable builds
    13 . 2
    

    View Slide

58. Kaniko with GitLab CI
    13 . 3
    

    View Slide

59. Kaniko with GitLab CI (cont.)
    # .gitlab-ci.yml
    build:kaniko:
    stage: build
    image:
    name: gcr.io/kaniko-project/executor:debug-v0.6.0
    entrypoint: ["/busybox/sh","-c"]
    before_script:
    - export KANIKO_DOCKERFILE="Dockerfile"
    - export KANIKO_DESTINATION="$CI_REGISTRY_IMAGE/$CI_JOB_NAME:$CI_COMMIT_REF_NAME"
    variables:
    KANIKO_DOCKERFILE: $CI_PROJECT_DIR/Dockerfile
    KANIKO_DESTINATION: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME
    script:
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"
    - /kaniko/executor --context "$CI_PROJECT_DIR/$CI_JOB_NAME" --destination "$KANIKO_DESTINATION"
    13 . 4
    

    View Slide

60. Open the doors for Q&A
    
    
    
    niclas.mietz@bee42.com
    solidnerd
    solidnerd
    14
    

    View Slide