Hybrid Kubernetes Cluster on Embedded Devices

In this talk, Peter will talk about how Docker was extended from x86 Linux to the ARM platforms for your hyprid kubernetes clusters.

Peter will also demo some of the new features of the current Docker CE engine to manage Kubernetes Clusters with both x86 and Raspberry PI Linux nodes.

Transcript

1. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion 1 Hybrid Kubernetes Cluster 
 on Embedded Devices
2. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Docker on ARM 2 Simple and helpful… First rumor at mid 2014 Hybrid OS Feb 2015 Docker Con 2015 - Raspberry PI Httpd Challenge Docker Engine support ARM mid 2016 Kubernetes on ARM starts at Dec 2015 Docker support Multi Arch Binary mid 2017
3. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Challenges 3 •Compiling things, on small embedded boards needs time •Cross Compilation is your friend •Find ready to use images on the official registry •Multi Arch Binaries •Reduce image Size •Control hardware devices •Build a cluster of raspberry PI’s •Let Kubernetes install on PI’s
4. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion 4 • Adventures needs clever friends….
5. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Reduce Size 5 •Prepare a Tool Container •Multi Stage Build •Compression •Improve quality
6. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Choose the right base images 6 https://www.codacy.com/blog/five-ways-to-slim-your-docker-images/
7. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Strategies to reduce image size 7 •Think Carefully About Your Application’s Needs •Use a Small Base Image •Use as Few Layers As Possible •Use .dockerignore files •Squash Docker Images
8. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Builder Pattern 8 Tool Base Image Compile Package Prepare Resulting Image Source
9. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion go dep tool base container 9 ARG BASE_IMAGE=${BASE_IMAGE:-golang:1.11.5-alpine3.8} FROM ${BASE_IMAGE} LABEL maintainer="Peter Rossbach <peter.rossbach@bee42.com>" ARG DEP_VERSION=${DEP_VERSION:-0.5.0} RUN apk update; \ apk add --no-cache \ ca-certificates \ curl \ git \ make \ openssl; \ curl -L -s https://github.com/golang/dep/releases/download/v${DEP_VERSION}/dep-linux-amd64 \ -o /bin/dep; \ chmod +x /bin/dep; \ rm -rf /var/cache/apk/*; \ rm -rf /tmp/*;
10. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Builder Image 10 FROM bee42.com/containers/tools/go-dep:1.11.5-alpine3.8 as builder ARG TARGET_ARCH=${TARGET_ARCH:-amd64} ARG APP=${APP:-bee42.com/containers/examples/k8s-client/blinkt} ENV CGO_ENABLED=0 ENV APP_GOPATH $GOPATH/src/$APP WORKDIR $APP_GOPATH RUN mkdir -p $APP_GOPATH COPY vendor/ $APP_GOPATH/vendor/ COPY Gopkg.* $APP_GOPATH/ COPY *.go $APP_GOPATH RUN cd $APP_GOPATH && \ GOOS=linux GOARCH=${TARGET_ARCH} GOARM=${GOARM:-7} go build -a --installsuffix cgo -- ldflags="-s" -o blinkt # Resulting App …
11. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Resulting Image 11 FROM bee42.com/containers/tools/go-dep:1.11.5-alpine3.8 as builder … # Resulting App FROM alpine:v3.8 COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt COPY --from=builder /app/blinkt /app/blinkt WORKDIR /app ENTRYPOINT ["/app/blinkt"]
12. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion UPX 12 https://upx.github.io UPX achieves an excellent compression ratio and offers very fast decompression.
13. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion UPX Package Compression 13 # Optimize binary size FROM alpine:v3.8 as packager ARG APP=${APP:-bee42.com/containers/examples/k8s-client/blinkt} ENV APP_GOPATH /go/src/$APP ARG TARGET_ARCH=${TARGET_ARCH:-amd64} ARG UPX_VERSION=${UPX_VERSION:-3.95} RUN apk add --no-cache xz binutils curl && echo ${TARGET_ARCH} RUN curl -sL -o /tmp/upx-${UPX_VERSION}-${TARGET_ARCH}_linux.tar.xz \ https://github.com/upx/upx/releases/download/v${UPX_VERSION}/upx-${UPX_VERSION}-$ {TARGET_ARCH}_linux.tar.xz && \ xz -d -c /tmp/upx-${UPX_VERSION}-${TARGET_ARCH}_linux.tar.xz | \ tar -xOf - upx-${UPX_VERSION}-${TARGET_ARCH}_linux/upx > /bin/upx && \ chmod a+x /bin/upx && \ rm /tmp/upx-${UPX_VERSION}-${TARGET_ARCH}_linux.tar.xz COPY --from=builder $APP_GOPATH/blinkt /app/blinkt RUN cd /app && \ strip --strip-unneeded blinkt && \ upx blinkt # Resulting App
14. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Use Upx package compression 14 # Optimize binary size FROM alpine:v3.8 as packager … # Resulting App FROM alpine:v3.8 COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt COPY --from=packager /app/blinkt /app/blinkt WORKDIR /app ENTRYPOINT ["/app/blinkt"] Safe 20-40% image size
15. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion 15 Build Cross Compiled Binaries: 
 qemu static at your ADM64 Boxes docker run --rm --privileged multiarch/qemu-user-static:register --reset https://hub.docker.com/r/multiarch/qemu-user-static for target_arch in aarch64 arm x86_64; do wget -N https://github.com/multiarch/qemu-user-static/releases/download/v2.9.1-1/x86_64_qemu-$ {target_arch}-static.tar.gz tar -xvf x86_64_qemu-${target_arch}-static.tar.gz done https://lobradov.github.io/Building-docker-multiarch-images/ Registry kernel modules Build with emulation binary
16. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Multiarch Docker hub 16 https://hub.docker.com/u/multiarch/ https://github.com/multiarch
17. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Multi Arch build matrix 17 https://doi-janky.infosiftr.net/job/multiarch/job/arm32v7/job/httpd/
18. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Build with separate Dockerfiles 18 cat >Dockerfile.amd64 <<EOF FROM amd64/alpine:3.7 # Not necessary for the arch where host and target are the same # COPY qemu-x86_64-static /usr/bin/ RUN apk --no-cache --update add nginx EXPOSE 80 CMD ["nginx", "-g", "daemon off;"] EOF
 
 cat >Dockerfile.arm32v6 <<EOF FROM arm32v6/alpine:3.7 COPY qemu-arm-static /usr/bin/ RUN apk --no-cache --update add nginx EXPOSE 80 CMD ["nginx", "-g", "daemon off;“] EOF
19. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Resulting Image 19 for arch in amd64 arm32v6; do docker build -f Dockerfile.${arch} -t bee42/nginx:${arch}-latest . docker push bee42/nginx:${arch}-latest done
20. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Builder Image with Args 20 ARG GOLANG_TARGET=${GOLANG_TARGET:-bee42.com/containers/tools/go-dep:1.11.5-alpine3.8} ARG TARGET=${TARGET:-multiarch/alpine:armhf-v3.8} FROM ${GOLANG_TARGET} as builder ARG TARGET_ARCH=${TARGET_ARCH:-arm} ARG APP=${APP:-bee42.com/containers/examples/k8s-client/blinkt} ENV CGO_ENABLED=0 ENV APP_GOPATH $GOPATH/src/$APP WORKDIR $APP_GOPATH RUN mkdir -p $APP_GOPATH COPY vendor/ $APP_GOPATH/vendor/ COPY Gopkg.* $APP_GOPATH/ COPY *.go $APP_GOPATH RUN cd $APP_GOPATH && \ GOOS=linux GOARCH=${TARGET_ARCH} GOARM=${GOARM:-7} go build -a --installsuffix cgo -- ldflags="-s" -o blinkt # Resulting App …
21. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Resulting Image 21 ARG GOLANG_TARGET=${GOLANG_TARGET:-bee42.com/containers/tools/go-dep:1.11.5-alpine3.8} ARG TARGET=${TARGET:-multiarch/alpine:armhf-v3.8} FROM ${GOLANG_TARGET} as builder … # Resulting App FROM ${TARGET} COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt COPY --from=buillder /app/blinkt /app/blinkt WORKDIR /app ENTRYPOINT ["/app/blinkt"]
22. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Build with Multi Arch Images 22 build: case $${arch} in \ amd64 ) target_image="multiarch/alpine:amd64-v3.8" ;; \ arm ) target_image="multiarch/alpine:armhf-v3.8" ;; \ arm64 ) target_image="multiarch/alpine:arm64-v3.8" ;; \ esac ; \ docker image build --no-cache \ --build-arg TARGET=$${target_image} \ --build-arg TARGET_ARCH=$${arch} \ -t $(DOCKER_IMAGE):$(DOCKER_TAG)-$${arch} . ; \
23. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Build with Mutli Arch Images 23 build-multiarch: @for arch in $(MULTIARCH); do \ case $${arch} in \ amd64 ) target_image="multiarch/alpine:amd64-v3.8" ;; \ arm ) target_image="multiarch/alpine:armhf-v3.8" ;; \ arm64 ) target_image="multiarch/alpine:arm64-v3.8" ;; \ esac ; \ docker image build --no-cache \ --build-arg TARGET=$${target_image} \ --build-arg TARGET_ARCH=$${arch} \ --build-arg VERSION=`cat VERSION` \ --build-arg VCS_REF=$(DOCKER_TAG) \ --build-arg BUILD_DATE=`date -u +"%Y-%m-%dT%H:%M:%SZ"` \ -t $(DOCKER_IMAGE):$(DOCKER_TAG)-$${arch} . ; \ done
24. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion 24
25. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Create Multiarch Manifest 25 push-multiarch: @echo "Create and push multiarch manifest: " @for arch in $(MULTIARCH); do \ docker image push $(DOCKER_IMAGE):$(DOCKER_TAG)-$${arch} ; \ done @docker manifest create $(DOCKER_IMAGE):$(MANIFEST_TAG) \ $(DOCKER_IMAGE):$(DOCKER_TAG)-amd64 \ $(DOCKER_IMAGE):$(DOCKER_TAG)-arm \ $(DOCKER_IMAGE):$(DOCKER_TAG)-arm64 @for arch in $(MULTIARCH); do \ case $${arch} in \ amd64 ) manifest_annotate="" ;; \ arm ) manifest_annotate="--os linux --arch arm" ;; \ arm64 ) manifest_annotate="--os linux --arch arm64 --variant armv8" ;; \ esac ; \ docker manifest annotate $(DOCKER_IMAGE):$(MANIFEST_TAG) $(DOCKER_IMAGE):$(DOCKER_TAG)- $${arch} $${manifest_annotate} ;\ done @docker manifest push $(DOCKER_IMAGE):$(MANIFEST_TAG)
26. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Docker 18.09 
 client experimental feature 26 mkdir -p ~/.docker cat > "$HOME/.docker/config.json" <<EOF { "experimental": "enabled" } EOF https://docs.docker.com/engine/reference/commandline/manifest/ https://docs.docker.com/registry/spec/manifest-v2-2/
27. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Add Metadata 27 # Metadata ARG VCS_REF ARG BUILD_DATE ARG VERSION # Metadata LABEL maintainer="bee42 cloud native crew <cloud-native@bee42.com>" \ org.opencontainers.image.title="blinkt" \ org.opencontainers.image.version="${VERSION}" \ org.opencontainers.image.revision="${VCS_REF}" \ org.opencontainers.image.created="${BUILD_DATE}" \ org.opencontainers.image.url="https://r-gitlab.bee42.com/containers/examples/k8s-client/blinkt/" \ org.opencontainers.image.source="https://gitlab.bee42.com/containers/examples/k8s-client/blinkt/" \ org.opencontainers.image.authors="bee42 cloud native crew <cloud-native@bee42.com>" \ org.opencontainers.image.vendor="bee42 solutions gmbh" \ org.opencontainers.image.licenses="Apache-2.0" \ com.bee42.image.type="service-stateless" \ https://github.com/opencontainers/image-spec/blob/master/annotations.md
28. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Control devices with docker 28 docker run --privileged -d blinkt docker run --device /dev/gpiomem -d blinkt You can perform GPIO with user privileges by interacting with the virtual files under /sys/class/gpio. Less Privileged Add device Use the sysfs GPIO interface docker run -v /sys:/sys -d blinkt
29. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion 29 • Kubernetes is a container orchestrator. • It’s how to run containers at scale. • It’s a very active open-source platform with lots of contributors, start at 6. June 2014 • Originally developed by Google and 
 donated to Cloud Native Computing Foundation
30. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion 30
31. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion 31
32. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion 32 https://github.com/bee42/kubernetes-on-embedded https://blog.hypriot.com/post/setup-kubernetes-raspberry-pi-cluster/ Blinkt - Demo https://github.com/apprenda/blinkt-k8s-controller https://github.com/StefanScherer/swarm-monitor
33. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion 33 Your Mac Ethernet Switch bee42-crew— 03-001 bee42- crew-03-002 bee42- crew-03-003 Edge Max DNS 192.168.42.31 192.168.42.32 192.168.42.33 192.168.42.101 192.168.42.1 Master Nodes Raspberry PI 3+ armv7 bee42-crew— 03-004 192.168.42.34 Nodes Raspberry PI 3+ arm64 bee42-crew— 03-005 192.168.42.35 Nodes UP Board amd64 192.168.1.230
34. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion 34 OS Root FS Image-raw-builder rpi kernel Image-builder https://github.com/hypriot/image-builder-rpi Flash ISO Ansible/Kubeadm K8s-Master K8s-Node https://github.com/bee42/kubernetes-on-embedded https://github.com/DieterReuter/image-builder-rpi64 Flash USB Manuel Install Ubuntu Update Kernel Actvate Devices Kubeadm K8s-Node
35. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion 35 API Server blinkt-k8s-controller <DaemonSet> Pods App Pods template: metadata: labels: app: httpd blinkt: show blinktColor: 00FF00 Blink device manage watch Set LED nodeSelector: deviceType: blinkt tolerations: - effect: NoSchedule key: node-role.kubernetes.io/master Blinkt to go
36. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion 36
37. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Kubernetes Deploy Blinkt 37
38. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion More to add… 38
39. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Sometimes 
 Master need more time to boot 39 sudo sed -i 's/failureThreshold: 8/failureThreshold: 20/g' /etc/kubernetes/manifests/kube-apiserver.yaml sudo sed -i 's/initialDelaySeconds: [0-9]\+/initialDelaySeconds: 360/' /etc/kubernetes/manifests/kube- apiserver.yaml
40. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Facts hybrid K8s adventure 40 •Timing problems to install the K8s master •Wrong kernel version to supports weave •Build your own OS with virtual machines and tricks… •Build multi arch binaries need time •Use Boards with IO PIN from Raspberry •Use Pimoroni python libs or gobots •Learn stop, boot reinstall a K8s Clusters
41. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion rethink IT Build hyprid kubernetes cluster with embedded machines is a funny adventure… 41
42. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Kubernetes poster pre registration started https://tinyurl.com/y9js3p7w 42 delivery starts at 42ten day of the year 2019 PREVIEW PREVIEW WE Ask me to buy today a printed copy…
43. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion Cloud Native System Architect & bee42 founder Peter Roßbach @PRossbach peter.rossbach@bee42.com https://bee42.com https://devops-gathering.io 43 #DOG19 11.-13. March 2019 at Bochum Discount Code: KubeCologne-15
44. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion bee42 Trainings 44 https://bee42.com/de/trainings 25.-26 February 2019 Berlin: KubeCologne-K8s-20 https://bee42.com/de/events/container-lab-mit-kubernetes-berlin/
45. Copyright 2019 bee42 solutions gmbh <peter.rossbach@bee42.com> @PRossbach rethink IT - We improve your systems with passion 45 We hiring :-) 
 https://bit.ly/2K8DtRu 
 jobs@bee42.com
 @bee42solutions