Hybrid Kubernetes Cluster  on Embedded Devices

Slide 1

Slide 1 text

Slide 2

Slide 2 text

Copyright 2019 bee42 solutions gmbh @PRossbach rethink IT - We improve your systems with passion Docker on ARM 2 Simple and helpful… First rumor at mid 2014 Hybrid OS Feb 2015 Docker Con 2015 - Raspberry PI Httpd Challenge Docker Engine support ARM mid 2016 Kubernetes on ARM starts at Dec 2015 Docker support Multi Arch Binary mid 2017

Slide 3

Slide 3 text

Copyright 2019 bee42 solutions gmbh @PRossbach rethink IT - We improve your systems with passion Challenges 3 •Compiling things, on small embedded boards needs time •Cross Compilation is your friend •Find ready to use images on the oﬃcial registry •Multi Arch Binaries •Reduce image Size •Control hardware devices •Build a cluster of raspberry PI’s •Let Kubernetes install on PI’s

Slide 4

Slide 4 text

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Slide 7

Slide 7 text

Copyright 2019 bee42 solutions gmbh @PRossbach rethink IT - We improve your systems with passion Strategies to reduce image size 7 •Think Carefully About Your Application’s Needs •Use a Small Base Image •Use as Few Layers As Possible •Use .dockerignore ﬁles •Squash Docker Images

Slide 8

Slide 8 text

Slide 9

Slide 9 text

Copyright 2019 bee42 solutions gmbh @PRossbach rethink IT - We improve your systems with passion go dep tool base container 9 ARG BASE_IMAGE=${BASE_IMAGE:-golang:1.11.5-alpine3.8} FROM ${BASE_IMAGE} LABEL maintainer="Peter Rossbach " ARG DEP_VERSION=${DEP_VERSION:-0.5.0} RUN apk update; \ apk add --no-cache \ ca-certiﬁcates \ curl \ git \ make \ openssl; \ curl -L -s https://github.com/golang/dep/releases/download/v${DEP_VERSION}/dep-linux-amd64 \ -o /bin/dep; \ chmod +x /bin/dep; \ rm -rf /var/cache/apk/*; \ rm -rf /tmp/*;

Slide 10

Slide 10 text

Copyright 2019 bee42 solutions gmbh @PRossbach rethink IT - We improve your systems with passion Builder Image 10 FROM bee42.com/containers/tools/go-dep:1.11.5-alpine3.8 as builder ARG TARGET_ARCH=${TARGET_ARCH:-amd64} ARG APP=${APP:-bee42.com/containers/examples/k8s-client/blinkt} ENV CGO_ENABLED=0 ENV APP_GOPATH $GOPATH/src/$APP WORKDIR $APP_GOPATH RUN mkdir -p $APP_GOPATH COPY vendor/ $APP_GOPATH/vendor/ COPY Gopkg.* $APP_GOPATH/ COPY *.go $APP_GOPATH RUN cd $APP_GOPATH && \ GOOS=linux GOARCH=${TARGET_ARCH} GOARM=${GOARM:-7} go build -a --installsuﬃx cgo -- ldﬂags="-s" -o blinkt # Resulting App …

Slide 11

Slide 11 text

Copyright 2019 bee42 solutions gmbh @PRossbach rethink IT - We improve your systems with passion Resulting Image 11 FROM bee42.com/containers/tools/go-dep:1.11.5-alpine3.8 as builder … # Resulting App FROM alpine:v3.8 COPY --from=builder /etc/ssl/certs/ca-certiﬁcates.crt /etc/ssl/certs/ca-certiﬁcates.crt COPY --from=builder /app/blinkt /app/blinkt WORKDIR /app ENTRYPOINT ["/app/blinkt"]

Slide 12

Slide 12 text

Copyright 2019 bee42 solutions gmbh @PRossbach rethink IT - We improve your systems with passion UPX 12 https://upx.github.io UPX achieves an excellent compression ratio and offers very fast decompression.

Slide 13

Slide 13 text

Copyright 2019 bee42 solutions gmbh @PRossbach rethink IT - We improve your systems with passion UPX Package Compression 13 # Optimize binary size FROM alpine:v3.8 as packager ARG APP=${APP:-bee42.com/containers/examples/k8s-client/blinkt} ENV APP_GOPATH /go/src/$APP ARG TARGET_ARCH=${TARGET_ARCH:-amd64} ARG UPX_VERSION=${UPX_VERSION:-3.95} RUN apk add --no-cache xz binutils curl && echo ${TARGET_ARCH} RUN curl -sL -o /tmp/upx-${UPX_VERSION}-${TARGET_ARCH}_linux.tar.xz \ https://github.com/upx/upx/releases/download/v${UPX_VERSION}/upx-${UPX_VERSION}-$ {TARGET_ARCH}_linux.tar.xz && \ xz -d -c /tmp/upx-${UPX_VERSION}-${TARGET_ARCH}_linux.tar.xz | \ tar -xOf - upx-${UPX_VERSION}-${TARGET_ARCH}_linux/upx > /bin/upx && \ chmod a+x /bin/upx && \ rm /tmp/upx-${UPX_VERSION}-${TARGET_ARCH}_linux.tar.xz COPY --from=builder $APP_GOPATH/blinkt /app/blinkt RUN cd /app && \ strip --strip-unneeded blinkt && \ upx blinkt # Resulting App

Slide 14

Slide 14 text

Copyright 2019 bee42 solutions gmbh @PRossbach rethink IT - We improve your systems with passion Use Upx package compression 14 # Optimize binary size FROM alpine:v3.8 as packager … # Resulting App FROM alpine:v3.8 COPY --from=build /etc/ssl/certs/ca-certiﬁcates.crt /etc/ssl/certs/ca-certiﬁcates.crt COPY --from=packager /app/blinkt /app/blinkt WORKDIR /app ENTRYPOINT ["/app/blinkt"] Safe 20-40% image size

Slide 15

Slide 15 text

Copyright 2019 bee42 solutions gmbh @PRossbach rethink IT - We improve your systems with passion 15 Build Cross Compiled Binaries:   qemu static at your ADM64 Boxes docker run --rm --privileged multiarch/qemu-user-static:register --reset https://hub.docker.com/r/multiarch/qemu-user-static for target_arch in aarch64 arm x86_64; do wget -N https://github.com/multiarch/qemu-user-static/releases/download/v2.9.1-1/x86_64_qemu-$ {target_arch}-static.tar.gz tar -xvf x86_64_qemu-${target_arch}-static.tar.gz done https://lobradov.github.io/Building-docker-multiarch-images/ Registry kernel modules Build with emulation binary

Slide 16

Slide 16 text

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Copyright 2019 bee42 solutions gmbh @PRossbach rethink IT - We improve your systems with passion Resulting Image 19 for arch in amd64 arm32v6; do docker build -f Dockerﬁle.${arch} -t bee42/nginx:${arch}-latest . docker push bee42/nginx:${arch}-latest done

Slide 20

Slide 20 text

Copyright 2019 bee42 solutions gmbh @PRossbach rethink IT - We improve your systems with passion Builder Image with Args 20 ARG GOLANG_TARGET=${GOLANG_TARGET:-bee42.com/containers/tools/go-dep:1.11.5-alpine3.8} ARG TARGET=${TARGET:-multiarch/alpine:armhf-v3.8} FROM ${GOLANG_TARGET} as builder ARG TARGET_ARCH=${TARGET_ARCH:-arm} ARG APP=${APP:-bee42.com/containers/examples/k8s-client/blinkt} ENV CGO_ENABLED=0 ENV APP_GOPATH $GOPATH/src/$APP WORKDIR $APP_GOPATH RUN mkdir -p $APP_GOPATH COPY vendor/ $APP_GOPATH/vendor/ COPY Gopkg.* $APP_GOPATH/ COPY *.go $APP_GOPATH RUN cd $APP_GOPATH && \ GOOS=linux GOARCH=${TARGET_ARCH} GOARM=${GOARM:-7} go build -a --installsuﬃx cgo -- ldﬂags="-s" -o blinkt # Resulting App …

Slide 21

Slide 21 text

Copyright 2019 bee42 solutions gmbh @PRossbach rethink IT - We improve your systems with passion Resulting Image 21 ARG GOLANG_TARGET=${GOLANG_TARGET:-bee42.com/containers/tools/go-dep:1.11.5-alpine3.8} ARG TARGET=${TARGET:-multiarch/alpine:armhf-v3.8} FROM ${GOLANG_TARGET} as builder … # Resulting App FROM ${TARGET} COPY --from=builder /etc/ssl/certs/ca-certiﬁcates.crt /etc/ssl/certs/ca-certiﬁcates.crt COPY --from=buillder /app/blinkt /app/blinkt WORKDIR /app ENTRYPOINT ["/app/blinkt"]

Slide 22

Slide 22 text

Copyright 2019 bee42 solutions gmbh @PRossbach rethink IT - We improve your systems with passion Build with Multi Arch Images 22 build: case $${arch} in \ amd64 ) target_image="multiarch/alpine:amd64-v3.8" ;; \ arm ) target_image="multiarch/alpine:armhf-v3.8" ;; \ arm64 ) target_image="multiarch/alpine:arm64-v3.8" ;; \ esac ; \ docker image build --no-cache \ --build-arg TARGET=$${target_image} \ --build-arg TARGET_ARCH=$${arch} \ -t $(DOCKER_IMAGE):$(DOCKER_TAG)-$${arch} . ; \

Slide 23

Slide 23 text

Copyright 2019 bee42 solutions gmbh @PRossbach rethink IT - We improve your systems with passion Build with Mutli Arch Images 23 build-multiarch: @for arch in $(MULTIARCH); do \ case $${arch} in \ amd64 ) target_image="multiarch/alpine:amd64-v3.8" ;; \ arm ) target_image="multiarch/alpine:armhf-v3.8" ;; \ arm64 ) target_image="multiarch/alpine:arm64-v3.8" ;; \ esac ; \ docker image build --no-cache \ --build-arg TARGET=$${target_image} \ --build-arg TARGET_ARCH=$${arch} \ --build-arg VERSION=`cat VERSION` \ --build-arg VCS_REF=$(DOCKER_TAG) \ --build-arg BUILD_DATE=`date -u +"%Y-%m-%dT%H:%M:%SZ"` \ -t $(DOCKER_IMAGE):$(DOCKER_TAG)-$${arch} . ; \ done

Slide 24

Slide 24 text

Slide 25

Slide 25 text

Copyright 2019 bee42 solutions gmbh @PRossbach rethink IT - We improve your systems with passion Create Multiarch Manifest 25 push-multiarch: @echo "Create and push multiarch manifest: " @for arch in $(MULTIARCH); do \ docker image push $(DOCKER_IMAGE):$(DOCKER_TAG)-$${arch} ; \ done @docker manifest create $(DOCKER_IMAGE):$(MANIFEST_TAG) \ $(DOCKER_IMAGE):$(DOCKER_TAG)-amd64 \ $(DOCKER_IMAGE):$(DOCKER_TAG)-arm \ $(DOCKER_IMAGE):$(DOCKER_TAG)-arm64 @for arch in $(MULTIARCH); do \ case $${arch} in \ amd64 ) manifest_annotate="" ;; \ arm ) manifest_annotate="--os linux --arch arm" ;; \ arm64 ) manifest_annotate="--os linux --arch arm64 --variant armv8" ;; \ esac ; \ docker manifest annotate $(DOCKER_IMAGE):$(MANIFEST_TAG) $(DOCKER_IMAGE):$(DOCKER_TAG)- $${arch} $${manifest_annotate} ;\ done @docker manifest push $(DOCKER_IMAGE):$(MANIFEST_TAG)

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Copyright 2019 bee42 solutions gmbh @PRossbach rethink IT - We improve your systems with passion Add Metadata 27 # Metadata ARG VCS_REF ARG BUILD_DATE ARG VERSION # Metadata LABEL maintainer="bee42 cloud native crew " \ org.opencontainers.image.title="blinkt" \ org.opencontainers.image.version="${VERSION}" \ org.opencontainers.image.revision="${VCS_REF}" \ org.opencontainers.image.created="${BUILD_DATE}" \ org.opencontainers.image.url="https://r-gitlab.bee42.com/containers/examples/k8s-client/blinkt/" \ org.opencontainers.image.source="https://gitlab.bee42.com/containers/examples/k8s-client/blinkt/" \ org.opencontainers.image.authors="bee42 cloud native crew " \ org.opencontainers.image.vendor="bee42 solutions gmbh" \ org.opencontainers.image.licenses="Apache-2.0" \ com.bee42.image.type="service-stateless" \ https://github.com/opencontainers/image-spec/blob/master/annotations.md

Slide 28

Slide 28 text

Copyright 2019 bee42 solutions gmbh @PRossbach rethink IT - We improve your systems with passion Control devices with docker 28 docker run --privileged -d blinkt docker run --device /dev/gpiomem -d blinkt You can perform GPIO with user privileges by interacting with the virtual ﬁles under /sys/class/gpio. Less Privileged Add device Use the sysfs GPIO interface docker run -v /sys:/sys -d blinkt

Slide 29

Slide 29 text

Copyright 2019 bee42 solutions gmbh @PRossbach rethink IT - We improve your systems with passion 29 • Kubernetes is a container orchestrator. • It’s how to run containers at scale. • It’s a very active open-source platform with lots of contributors, start at 6. June 2014 • Originally developed by Google and   donated to Cloud Native Computing Foundation

Slide 30

Slide 30 text

Slide 31

Slide 31 text

Slide 32

Slide 32 text

Copyright 2019 bee42 solutions gmbh @PRossbach rethink IT - We improve your systems with passion 32 https://github.com/bee42/kubernetes-on-embedded https://blog.hypriot.com/post/setup-kubernetes-raspberry-pi-cluster/ Blinkt - Demo https://github.com/apprenda/blinkt-k8s-controller https://github.com/StefanScherer/swarm-monitor

Slide 33

Slide 33 text

Copyright 2019 bee42 solutions gmbh @PRossbach rethink IT - We improve your systems with passion 33 Your Mac Ethernet Switch bee42-crew— 03-001 bee42- crew-03-002 bee42- crew-03-003 Edge Max DNS 192.168.42.31 192.168.42.32 192.168.42.33 192.168.42.101 192.168.42.1 Master Nodes Raspberry PI 3+ armv7 bee42-crew— 03-004 192.168.42.34 Nodes Raspberry PI 3+ arm64 bee42-crew— 03-005 192.168.42.35 Nodes UP Board amd64 192.168.1.230

Slide 34

Slide 34 text

Copyright 2019 bee42 solutions gmbh @PRossbach rethink IT - We improve your systems with passion 34 OS Root FS Image-raw-builder rpi kernel Image-builder https://github.com/hypriot/image-builder-rpi Flash ISO Ansible/Kubeadm K8s-Master K8s-Node https://github.com/bee42/kubernetes-on-embedded https://github.com/DieterReuter/image-builder-rpi64 Flash USB Manuel Install Ubuntu Update Kernel Actvate Devices Kubeadm K8s-Node

Slide 35

Slide 35 text

Copyright 2019 bee42 solutions gmbh @PRossbach rethink IT - We improve your systems with passion 35 API Server blinkt-k8s-controller Pods App Pods template: metadata: labels: app: httpd blinkt: show blinktColor: 00FF00 Blink device manage watch Set LED nodeSelector: deviceType: blinkt tolerations: - eﬀect: NoSchedule key: node-role.kubernetes.io/master Blinkt to go

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Slide 38

Slide 38 text

Slide 39

Slide 39 text

Copyright 2019 bee42 solutions gmbh @PRossbach rethink IT - We improve your systems with passion Sometimes   Master need more time to boot 39 sudo sed -i 's/failureThreshold: 8/failureThreshold: 20/g' /etc/kubernetes/manifests/kube-apiserver.yaml sudo sed -i 's/initialDelaySeconds: [0-9]\+/initialDelaySeconds: 360/' /etc/kubernetes/manifests/kube- apiserver.yaml

Slide 40

Slide 40 text

Copyright 2019 bee42 solutions gmbh @PRossbach rethink IT - We improve your systems with passion Facts hybrid K8s adventure 40 •Timing problems to install the K8s master •Wrong kernel version to supports weave •Build your own OS with virtual machines and tricks… •Build multi arch binaries need time •Use Boards with IO PIN from Raspberry •Use Pimoroni python libs or gobots •Learn stop, boot reinstall a K8s Clusters

Slide 41

Slide 41 text

Slide 42

Slide 42 text

Copyright 2019 bee42 solutions gmbh @PRossbach rethink IT - We improve your systems with passion Kubernetes poster pre registration started https://tinyurl.com/y9js3p7w 42 delivery starts at 42ten day of the year 2019 PREVIEW PREVIEW WE Ask me to buy today a printed copy…

Slide 43

Slide 43 text

Copyright 2019 bee42 solutions gmbh @PRossbach rethink IT - We improve your systems with passion Cloud Native System Architect & bee42 founder Peter Roßbach @PRossbach [email protected] https://bee42.com https://devops-gathering.io 43 #DOG19 11.-13. March 2019 at Bochum Discount Code: KubeCologne-15

Slide 44

Slide 44 text

Copyright 2019 bee42 solutions gmbh @PRossbach rethink IT - We improve your systems with passion bee42 Trainings 44 https://bee42.com/de/trainings 25.-26 February 2019 Berlin: KubeCologne-K8s-20 https://bee42.com/de/events/container-lab-mit-kubernetes-berlin/