Create multi-stage, multi-arch Go container images for your heterogeneous Kubernetes clusters

In this talk, Peter will talk about how Docker was extended from x86 Linux to the ARM platforms for your Go applications. They will cover the work and architecture that makes it possible to run Golang with Docker on different CPU architectures. Peter will also demo some of the new features of the current Docker CE engine to manage Kubernetes Clusters with both x86 and Raspberry PI Linux nodes.

Building multi-arch container images with Docker is a very special build process.

​
How you can use your locale developer environment to build images for a hybride Kubernetes Cluster?

- What it means for a Docker image to be build multi-arch?
- How are multi-arch images push to a registry and maintained?
- How does Docker correctly deploy and schedule go applications on heterogeneous Kubernetes or Docker swarm cluster?
-How you can run and tests those hybride go applications?

* https://www.godays.io/program

Transcript

1. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion 1 Create multi-stage, multi-arch Go container images for your heterogeneous Kubernetes clusters
2. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Golang 2 Simple and helpful… Go was designed at Google ] by Robert Griesemer, Rob Pike, and Ken Thompson in 2007 to improve programming productivity in an era of multicore, networked machines and extremely large codebases. Container technologies like Docker and Kubernetes powered by golang
3. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Why build GO apps 
 with containers? 3 •Use or implement OpenSource projects that written in Go •Ship your apps with container is state of the art •Build software for different environments (OS, CPU, …) •Use multiple golang version in parallel •Solve your golang dependency problems •Reduce your size of your app shipments for better scale
4. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion What’s the problem? 4 •compiling is slow What we need? •Speed up compiling •Reduce size •Multi Arch •Pipeline Builds •Running at Kubernetes
5. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Native build 5 FROM golang:1.11 RUN go get github.com/golang/dep/cmd/dep WORKDIR /go/src/github.com/bee42/whoamI ADD . . # Download deps RUN dep ensure # Install RUN go install github.com/bee42/whoamI ENTRYPOINT /go/bin/whoamI >50 sec
6. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion With cached deps 6 FROM golang:1.11 RUN go get github.com/golang/dep/cmd/dep WORKDIR /go/src/github.com/bee42/whomaI # Download deps ADD Gopkg.* ./ RUN dep ensure --vendor-only # Install source ADD . . RUN go install github.com/bee42/whomaI # Keep the container open ENTRYPOINT tail -f /dev/null 24 secs
7. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion With cached obj - I 7 # Start builder ARG baseImage="golang:1.11" FROM ${baseImage} as builder RUN go get github.com/golang/dep/cmd/dep WORKDIR /go/src/github.com/bee42/whoamI # Download deps ADD Gopkg.* ./ RUN dep ensure --vendor-only # Install source ADD . . RUN go install github.com/bee42/whoamI # Done builder
 … 24 secs
8. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion With cached obj - II 8 # Start builder ARG baseImage="golang:1.11" FROM ${baseImage} as builder … # Start obj-cache FROM golang:1.11 as obj-cache COPY --from=builder /root/.cache /root/.cache # Done obj-cache # Start main FROM builder ENTRYPOINT /go/bin/whoamI # Done main
9. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Use cached obj 9 cacheobjs-base: if [ "$(shell docker images bee42/whoamI/cacheobjs-base -q)" = "" ]; then \ docker build -t bee42/whoamI/cacheobjs-base -f Dockerfile.cacheobjs --target=obj-cache .; \ fi; cacheobjs: cacheobjs-base docker build --build-arg baseImage=bee42/whoamI/cacheobjs-base \ -t bee42/whoamI/cacheobjs \ -f Dockerfile.cacheobjs . docker run --rm -it bee42/whoamI/cacheobjs 8 secs
10. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Taily Build 10 FROM golang:1.11 RUN go get github.com/golang/dep/cmd/dep WORKDIR /go/src/github.com/bee42/whoamI # Download deps ADD Gopkg.* ./ RUN dep ensure --vendor-only # Install source ADD . . RUN go install github.com/bee42/whoamI # Keep the container open ENTRYPOINT tail -f /dev/null
11. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Use taily build 11 tailybuild-base: if [ "$(shell docker ps --filter=name=tailybuild -q)" = "" ]; then \ docker build -t bee42/whomaI/tailybuild-base -f Dockerfile.tailybuild .; \ docker run --name tailybuild -d bee42/whoami/tailybuild-base; \ fi; tailybuild: tailybuild-base docker exec -it tailybuild rm -fR whoamI docker cp cmd tailybuild:/go/src/github.com/bee42/whoamI/ docker exec -it tailybuild go install github.com/bee42/whoamI docker exec -it tailybuild /go/bin/whomaI 3 secs
12. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Taily Mount 12 FROM golang:1.11 RUN go get github.com/golang/dep/cmd/dep WORKDIR /go/src/github.com/bee42/whoamI # Download deps ADD Gopkg.* ./ RUN dep ensure --vendor-only # Install source ADD . . RUN go install github.com/bee42/whoamI # Keep the container open ENTRYPOINT tail -f /dev/null
13. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Use taily mount 13 tailymount-base: if [ "$(shell docker ps --filter=name=tailymount -q)" = "" ]; then \ docker build -t bee42/whoamI/tailymount-base -f Dockerfile.tailymount .; \ docker run --mount type=bind,source=$(shell pwd)/cmd,target=/go/src/github.com/bee42/whoamI \ --name tailymount -d bee42/whoamI/tailymount-base; \ fi; tailymount: tailymount-base docker exec -it tailymount go install github.com/bee42/whoamI docker exec -it tailymount /go/bin/whoamI 2.4 secs
14. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Yeah > Save life time… 14 • https://medium.com/windmill-engineering/tips-tricks-for-making-your-golang-container-builds-10x-faster-4cc618a43827
15. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Reduce Size 15 •Prepare a Tool Container •Multi Stage Build •Compression •Improve quality
16. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Choose the right base images 16 https://www.codacy.com/blog/five-ways-to-slim-your-docker-images/
17. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Strategies to reduce image size 17 •Think Carefully About Your Application’s Needs •Use a Small Base Image •Use as Few Layers As Possible •Use .dockerignore files •Squash Docker Images
18. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion go dep tool base container 18 ARG BASE_IMAGE=${BASE_IMAGE:-golang:1.11.5-alpine3.8} FROM ${BASE_IMAGE} LABEL maintainer="Peter Rossbach <peter.rossbach@bee42.com>" ARG DEP_VERSION=${DEP_VERSION:-0.5.0} RUN apk update; \ apk add --no-cache \ ca-certificates \ curl \ git \ make \ openssl; \ curl -L -s https://github.com/golang/dep/releases/download/v${DEP_VERSION}/dep-linux-amd64 \ -o /bin/dep; \ chmod +x /bin/dep; \ rm -rf /var/cache/apk/*; \ rm -rf /tmp/*;
19. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Builder Image 19 FROM bee42.com/containers/tools/go-dep:1.11.5-alpine3.8 as builder ARG TARGET_ARCH=${TARGET_ARCH:-amd64} ARG APP=${APP:-bee42.com/containers/examples/k8s-client/blinkt} ENV CGO_ENABLED=0 ENV APP_GOPATH $GOPATH/src/$APP WORKDIR $APP_GOPATH RUN mkdir -p $APP_GOPATH COPY vendor/ $APP_GOPATH/vendor/ COPY Gopkg.* $APP_GOPATH/ COPY *.go $APP_GOPATH RUN cd $APP_GOPATH && \ GOOS=linux GOARCH=${TARGET_ARCH} GOARM=${GOARM:-7} go build -a --installsuffix cgo -- ldflags="-s" -o blinkt # Resulting App …
20. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Resulting Image 20 FROM bee42.com/containers/tools/go-dep:1.11.5-alpine3.8 as builder … # Resulting App FROM alpine:v3.8 COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt COPY --from=builder /app/blinkt /app/blinkt WORKDIR /app ENTRYPOINT ["/app/blinkt"]
21. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion UPX 21 https://upx.github.io UPX achieves an excellent compression ratio and offers very fast decompression.
22. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion UPX Package Compression 22 # Optimize binary size FROM alpine:v3.8 as packager ARG APP=${APP:-bee42.com/containers/examples/k8s-client/blinkt} ENV APP_GOPATH /go/src/$APP ARG TARGET_ARCH=${TARGET_ARCH:-amd64} ARG UPX_VERSION=${UPX_VERSION:-3.95} RUN apk add --no-cache xz binutils curl && echo ${TARGET_ARCH} RUN curl -sL -o /tmp/upx-${UPX_VERSION}-${TARGET_ARCH}_linux.tar.xz \ https://github.com/upx/upx/releases/download/v${UPX_VERSION}/upx-${UPX_VERSION}-$ {TARGET_ARCH}_linux.tar.xz && \ xz -d -c /tmp/upx-${UPX_VERSION}-${TARGET_ARCH}_linux.tar.xz | \ tar -xOf - upx-${UPX_VERSION}-${TARGET_ARCH}_linux/upx > /bin/upx && \ chmod a+x /bin/upx && \ rm /tmp/upx-${UPX_VERSION}-${TARGET_ARCH}_linux.tar.xz COPY --from=builder $APP_GOPATH/blinkt /app/blinkt RUN cd /app && \ strip --strip-unneeded blinkt && \ upx blinkt # Resulting App
23. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Use Upx package compression 23 # Optimize binary size FROM alpine:v3.8 as packager … # Resulting App FROM alpine:v3.8 COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt COPY --from=packager /app/blinkt /app/blinkt WORKDIR /app ENTRYPOINT ["/app/blinkt"] Safe 20-40% image size
24. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion 24 Build Cross Compiled Binaries: 
 qemu static docker run --rm --privileged multiarch/qemu-user-static:register --reset https://hub.docker.com/r/multiarch/qemu-user-static for target_arch in aarch64 arm x86_64; do wget -N https://github.com/multiarch/qemu-user-static/releases/download/v2.9.1-1/x86_64_qemu-$ {target_arch}-static.tar.gz tar -xvf x86_64_qemu-${target_arch}-static.tar.gz done https://lobradov.github.io/Building-docker-multiarch-images/ Registry kernel modules Build with emulation binary
25. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Multiarch Docker hub 25 https://hub.docker.com/u/multiarch/ https://github.com/multiarch
26. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Multi Arch build matrix 26 https://doi-janky.infosiftr.net/job/multiarch/job/arm32v7/job/httpd/
27. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Build with separate Dockerfiles 27 cat >Dockerfile.amd64 <<EOF FROM amd64/alpine:3.7 # Not necessary for the arch where host and target are the same # COPY qemu-x86_64-static /usr/bin/ RUN apk --no-cache --update add nginx EXPOSE 80 CMD ["nginx", "-g", "daemon off;"] EOF
 
 cat >Dockerfile.arm32v6 <<EOF FROM arm32v6/alpine:3.7 COPY qemu-arm-static /usr/bin/ RUN apk --no-cache --update add nginx EXPOSE 80 CMD ["nginx", "-g", "daemon off;“] EOF
28. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Resulting Image 28 for arch in amd64 arm32v6; do docker build -f Dockerfile.${arch} -t bee42/nginx:${arch}-latest . docker push bee42/nginx:${arch}-latest done
29. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Builder Image with Args 29 ARG GOLANG_TARGET=${GOLANG_TARGET:-bee42.com/containers/tools/go-dep:1.11.5-alpine3.8} ARG TARGET=${TARGET:-multiarch/alpine:armhf-v3.8} FROM ${GOLANG_TARGET} as builder ARG TARGET_ARCH=${TARGET_ARCH:-arm} ARG APP=${APP:-bee42.com/containers/examples/k8s-client/blinkt} ENV CGO_ENABLED=0 ENV APP_GOPATH $GOPATH/src/$APP WORKDIR $APP_GOPATH RUN mkdir -p $APP_GOPATH COPY vendor/ $APP_GOPATH/vendor/ COPY Gopkg.* $APP_GOPATH/ COPY *.go $APP_GOPATH RUN cd $APP_GOPATH && \ GOOS=linux GOARCH=${TARGET_ARCH} GOARM=${GOARM:-7} go build -a --installsuffix cgo -- ldflags="-s" -o blinkt # Resulting App …
30. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Resulting Image 30 ARG GOLANG_TARGET=${GOLANG_TARGET:-bee42.com/containers/tools/go-dep:1.11.5-alpine3.8} ARG TARGET=${TARGET:-multiarch/alpine:armhf-v3.8} FROM ${GOLANG_TARGET} as builder … # Resulting App FROM ${TARGET} COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt COPY --from=buillder /app/blinkt /app/blinkt WORKDIR /app ENTRYPOINT ["/app/blinkt"]
31. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Build with Multi Arch Images 31 build: case $${arch} in \ amd64 ) target_image="multiarch/alpine:amd64-v3.8" ;; \ arm ) target_image="multiarch/alpine:armhf-v3.8" ;; \ arm64 ) target_image="multiarch/alpine:arm64-v3.8" ;; \ esac ; \ docker image build --no-cache \ --build-arg TARGET=$${target_image} \ --build-arg TARGET_ARCH=$${arch} \ -t $(DOCKER_IMAGE):$(DOCKER_TAG)-$${arch} . ; \
32. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Build with Mutli Arch Images 32 build-multiarch: @for arch in $(MULTIARCH); do \ case $${arch} in \ amd64 ) target_image="multiarch/alpine:amd64-v3.8" ;; \ arm ) target_image="multiarch/alpine:armhf-v3.8" ;; \ arm64 ) target_image="multiarch/alpine:arm64-v3.8" ;; \ esac ; \ docker image build --no-cache \ --build-arg TARGET=$${target_image} \ --build-arg TARGET_ARCH=$${arch} \ --build-arg VERSION=`cat VERSION` \ --build-arg VCS_REF=$(DOCKER_TAG) \ --build-arg BUILD_DATE=`date -u +"%Y-%m-%dT%H:%M:%SZ"` \ -t $(DOCKER_IMAGE):$(DOCKER_TAG)-$${arch} . ; \ done
33. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion 33
34. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Create Multiarch Manifest 34 push-multiarch: @echo "Create and push multiarch manifest: " @for arch in $(MULTIARCH); do \ docker image push $(DOCKER_IMAGE):$(DOCKER_TAG)-$${arch} ; \ done @docker manifest create $(DOCKER_IMAGE):$(MANIFEST_TAG) \ $(DOCKER_IMAGE):$(DOCKER_TAG)-amd64 \ $(DOCKER_IMAGE):$(DOCKER_TAG)-arm \ $(DOCKER_IMAGE):$(DOCKER_TAG)-arm64 @for arch in $(MULTIARCH); do \ case $${arch} in \ amd64 ) manifest_annotate="" ;; \ arm ) manifest_annotate="--os linux --arch arm" ;; \ arm64 ) manifest_annotate="--os linux --arch arm64 --variant armv8" ;; \ esac ; \ docker manifest annotate $(DOCKER_IMAGE):$(MANIFEST_TAG) $(DOCKER_IMAGE):$(DOCKER_TAG)- $${arch} $${manifest_annotate} ;\ done @docker manifest push $(DOCKER_IMAGE):$(MANIFEST_TAG)
35. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Docker 18.09 
 client experimental feature 35 mkdir -p ~/.docker cat > "$HOME/.docker/config.json" <<EOF { "experimental": "enabled" } EOF https://docs.docker.com/engine/reference/commandline/manifest/ https://docs.docker.com/registry/spec/manifest-v2-2/
36. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Add Metadata 36 # Metadata ARG VCS_REF ARG BUILD_DATE ARG VERSION # Metadata LABEL maintainer="bee42 cloud native crew <cloud-native@bee42.com>" \ org.opencontainers.image.title="blinkt" \ org.opencontainers.image.version="${VERSION}" \ org.opencontainers.image.revision="${VCS_REF}" \ org.opencontainers.image.created="${BUILD_DATE}" \ org.opencontainers.image.url="https://r-gitlab.bee42.com/containers/examples/k8s-client/blinkt/" \ org.opencontainers.image.source="https://gitlab.bee42.com/containers/examples/k8s-client/blinkt/" \ org.opencontainers.image.authors="bee42 cloud native crew <cloud-native@bee42.com>" \ org.opencontainers.image.vendor="bee42 solutions gmbh" \ org.opencontainers.image.licenses="Apache-2.0" \ com.bee42.image.type="service-stateless" \ https://github.com/opencontainers/image-spec/blob/master/annotations.md
37. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion 37
38. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion 38 • Kubernetes is a container orchestrator. • It’s how to run containers at scale. • It’s a very active open-source platform with lots of contributors, start at 6. June 2014 • Originally developed by Google and 
 donated to Cloud Native Computing Foundation
39. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion 39
40. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion 40
41. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion 41
42. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion 42 https://github.com/bee42/kubernetes-on-embedded https://blog.hypriot.com/post/setup-kubernetes-raspberry-pi-cluster/ Blinkt - Demo https://github.com/apprenda/blinkt-k8s-controller https://github.com/StefanScherer/swarm-monitor
43. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion 43 Your Mac Ethernet Switch bee42-crew— 03-001 bee42- crew-03-002 bee42- crew-03-003 Edge Max DNS 192.168.42.31 192.168.42.32 192.168.42.33 192.168.42.101 192.168.42.1 Master Nodes Raspberry PI 3+ armv7 bee42-crew— 03-004 192.168.42.34 Nodes Raspberry PI 3+ arm64 bee42-crew— 03-005 192.168.42.35 Nodes UP Board amd64 192.168.1.230
44. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Kubernetes Deploy Blinkt 44
45. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Facts of the multistage 
 and multi arch adventure 45 • Speed up with golang inside container is possible. • Caching the dependencies • Use compiling inside run container • Reduce of images size with multistage build (copy resulting binary) • Reduce size with UPX • Copy Binary • Multi Arch image building • Use qemu-user-static at intel • Sometimes you need native builder instances • Create image manifests to use same image reference inside your deployment resources • Add metadata to your images • Setup and maintain a heterogeneous machine cluster is hard. • Compiling your own kernel • Manuell setup of machines • Wait for features
46. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion rethink IT Build herogenous kubernetes cluster with embedded machines is a funny adventure… 46
47. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Kubernetes poster pre registration started https://tinyurl.com/y9js3p7w 47 delivery starts at 42ten day of the year 2019 PREVIEW PREVIEW WE
48. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Cloud Native System Architect & bee42 founder Peter Roßbach @PRossbach peter.rossbach@bee42.com https://bee42.com https://devops-gathering.io 48 Save the date… #DOG19 11.-13. March 2019 at Bochum
49. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion bee42 Trainings 49 https://bee42.com/de/trainings
50. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion 50 We hiring :-) 
 https://bit.ly/2K8DtRu 
 jobs@bee42.com
 @bee42solutions