Hybrid Kubernetes Cluster on Embedded Devices

In this talk, Peter will talk about how Docker was extended from x86 Linux to the ARM platforms for your hyprid kubernetes clusters.

Peter will also demo some of the new features of the current Docker CE engine to manage Kubernetes Clusters with both x86 and Raspberry PI Linux nodes.

Transcript

1. Copyright 2019 bee42 solutions gmbh @PRossbach
   rethink IT - We improve your systems with passion
   1
   Hybrid Kubernetes Cluster 

   on Embedded Devices
   

   View Slide

2. Copyright 2019 bee42 solutions gmbh @PRossbach
   rethink IT - We improve your systems with passion
   Docker on ARM
   2
   Simple and helpful…
   First rumor at mid 2014
   Hybrid OS Feb 2015
   Docker Con 2015 - Raspberry PI Httpd Challenge
   Docker Engine support ARM mid 2016
   Kubernetes on ARM starts at Dec 2015
   Docker support Multi Arch Binary mid 2017
   

   View Slide

3. Copyright 2019 bee42 solutions gmbh @PRossbach
   rethink IT - We improve your systems with passion
   Challenges
   3
   •Compiling things, on small embedded boards needs time
   •Cross Compilation is your friend
   •Find ready to use images on the official registry
   •Multi Arch Binaries
   •Reduce image Size
   •Control hardware devices
   •Build a cluster of raspberry PI’s
   •Let Kubernetes install on PI’s
   

   View Slide

4. Copyright 2019 bee42 solutions gmbh @PRossbach
   rethink IT - We improve your systems with passion
   4
   •
   Adventures needs clever friends….
   

   View Slide

5. Copyright 2019 bee42 solutions gmbh @PRossbach
   rethink IT - We improve your systems with passion
   Reduce Size
   5
   •Prepare a Tool Container
   •Multi Stage Build
   •Compression
   •Improve quality
   

   View Slide

6. Copyright 2019 bee42 solutions gmbh @PRossbach
   rethink IT - We improve your systems with passion
   Choose the right base images
   6
   https://www.codacy.com/blog/five-ways-to-slim-your-docker-images/
   

   View Slide

7. Copyright 2019 bee42 solutions gmbh @PRossbach
   rethink IT - We improve your systems with passion
   Strategies to reduce image size
   7
   •Think Carefully About Your Application’s Needs
   •Use a Small Base Image
   •Use as Few Layers As Possible
   •Use .dockerignore files
   •Squash Docker Images
   

   View Slide

8. Copyright 2019 bee42 solutions gmbh @PRossbach
   rethink IT - We improve your systems with passion
   Builder Pattern
   8
   Tool Base Image Compile Package
   Prepare Resulting
   Image
   Source
   

   View Slide

9. Copyright 2019 bee42 solutions gmbh @PRossbach
   rethink IT - We improve your systems with passion
   go dep tool base container
   9
   ARG BASE_IMAGE=${BASE_IMAGE:-golang:1.11.5-alpine3.8}
   FROM ${BASE_IMAGE}
   LABEL maintainer="Peter Rossbach "
   ARG DEP_VERSION=${DEP_VERSION:-0.5.0}
   RUN apk update; \
   apk add --no-cache \
   ca-certificates \
   curl \
   git \
   make \
   openssl; \
   curl -L -s https://github.com/golang/dep/releases/download/v${DEP_VERSION}/dep-linux-amd64 \
   -o /bin/dep; \
   chmod +x /bin/dep; \
   rm -rf /var/cache/apk/*; \
   rm -rf /tmp/*;
   

   View Slide

10. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Builder Image
    10
    FROM bee42.com/containers/tools/go-dep:1.11.5-alpine3.8 as builder
    ARG TARGET_ARCH=${TARGET_ARCH:-amd64}
    ARG APP=${APP:-bee42.com/containers/examples/k8s-client/blinkt}
    ENV CGO_ENABLED=0
    ENV APP_GOPATH $GOPATH/src/$APP
    WORKDIR $APP_GOPATH
    RUN mkdir -p $APP_GOPATH
    COPY vendor/ $APP_GOPATH/vendor/
    COPY Gopkg.* $APP_GOPATH/
    COPY *.go $APP_GOPATH
    RUN cd $APP_GOPATH && \
    GOOS=linux GOARCH=${TARGET_ARCH} GOARM=${GOARM:-7} go build -a --installsuffix cgo --
    ldflags="-s" -o blinkt
    # Resulting App
    …
    

    View Slide

11. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Resulting Image
    11
    FROM bee42.com/containers/tools/go-dep:1.11.5-alpine3.8 as builder
    …
    # Resulting App
    FROM alpine:v3.8
    COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
    COPY --from=builder /app/blinkt /app/blinkt
    WORKDIR /app
    ENTRYPOINT ["/app/blinkt"]
    

    View Slide

12. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    UPX
    12
    https://upx.github.io
    UPX achieves an excellent compression ratio and offers very fast decompression.
    

    View Slide

13. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    UPX Package Compression
    13
    # Optimize binary size
    FROM alpine:v3.8 as packager
    ARG APP=${APP:-bee42.com/containers/examples/k8s-client/blinkt}
    ENV APP_GOPATH /go/src/$APP
    ARG TARGET_ARCH=${TARGET_ARCH:-amd64}
    ARG UPX_VERSION=${UPX_VERSION:-3.95}
    RUN apk add --no-cache xz binutils curl && echo ${TARGET_ARCH}
    RUN curl -sL -o /tmp/upx-${UPX_VERSION}-${TARGET_ARCH}_linux.tar.xz \
    https://github.com/upx/upx/releases/download/v${UPX_VERSION}/upx-${UPX_VERSION}-$
    {TARGET_ARCH}_linux.tar.xz && \
    xz -d -c /tmp/upx-${UPX_VERSION}-${TARGET_ARCH}_linux.tar.xz | \
    tar -xOf - upx-${UPX_VERSION}-${TARGET_ARCH}_linux/upx > /bin/upx && \
    chmod a+x /bin/upx && \
    rm /tmp/upx-${UPX_VERSION}-${TARGET_ARCH}_linux.tar.xz
    COPY --from=builder $APP_GOPATH/blinkt /app/blinkt
    RUN cd /app && \
    strip --strip-unneeded blinkt && \
    upx blinkt
    # Resulting App
    

    View Slide

14. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Use Upx package compression
    14
    # Optimize binary size
    FROM alpine:v3.8 as packager
    …
    # Resulting App
    FROM alpine:v3.8
    COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
    COPY --from=packager /app/blinkt /app/blinkt
    WORKDIR /app
    ENTRYPOINT ["/app/blinkt"]
    Safe 20-40% image size
    

    View Slide

15. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    15
    Build Cross Compiled Binaries: 

    qemu static at your ADM64 Boxes
    docker run --rm --privileged multiarch/qemu-user-static:register --reset
    https://hub.docker.com/r/multiarch/qemu-user-static
    for target_arch in aarch64 arm x86_64; do
    wget -N https://github.com/multiarch/qemu-user-static/releases/download/v2.9.1-1/x86_64_qemu-$
    {target_arch}-static.tar.gz
    tar -xvf x86_64_qemu-${target_arch}-static.tar.gz
    done
    https://lobradov.github.io/Building-docker-multiarch-images/
    Registry kernel modules
    Build with emulation binary
    

    View Slide

16. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Multiarch Docker hub
    16
    https://hub.docker.com/u/multiarch/
    https://github.com/multiarch
    

    View Slide

17. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Multi Arch build matrix
    17
    https://doi-janky.infosiftr.net/job/multiarch/job/arm32v7/job/httpd/
    

    View Slide

18. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Build with separate Dockerfiles
    18
    cat >Dockerfile.amd64 <FROM amd64/alpine:3.7
    # Not necessary for the arch where host and target are the same
    # COPY qemu-x86_64-static /usr/bin/
    RUN apk --no-cache --update add nginx
    EXPOSE 80
    CMD ["nginx", "-g", "daemon off;"]
    EOF

    

    cat >Dockerfile.arm32v6 <FROM arm32v6/alpine:3.7
    COPY qemu-arm-static /usr/bin/
    RUN apk --no-cache --update add nginx
    EXPOSE 80
    CMD ["nginx", "-g", "daemon off;“]
    EOF
    

    View Slide

19. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Resulting Image
    19
    for arch in amd64 arm32v6; do
    docker build -f Dockerfile.${arch} -t bee42/nginx:${arch}-latest .
    docker push bee42/nginx:${arch}-latest
    done
    

    View Slide

20. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Builder Image with Args
    20
    ARG GOLANG_TARGET=${GOLANG_TARGET:-bee42.com/containers/tools/go-dep:1.11.5-alpine3.8}
    ARG TARGET=${TARGET:-multiarch/alpine:armhf-v3.8}
    FROM ${GOLANG_TARGET} as builder
    ARG TARGET_ARCH=${TARGET_ARCH:-arm}
    ARG APP=${APP:-bee42.com/containers/examples/k8s-client/blinkt}
    ENV CGO_ENABLED=0
    ENV APP_GOPATH $GOPATH/src/$APP
    WORKDIR $APP_GOPATH
    RUN mkdir -p $APP_GOPATH
    COPY vendor/ $APP_GOPATH/vendor/
    COPY Gopkg.* $APP_GOPATH/
    COPY *.go $APP_GOPATH
    RUN cd $APP_GOPATH && \
    GOOS=linux GOARCH=${TARGET_ARCH} GOARM=${GOARM:-7} go build -a --installsuffix cgo --
    ldflags="-s" -o blinkt
    # Resulting App
    …
    

    View Slide

21. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Resulting Image
    21
    ARG GOLANG_TARGET=${GOLANG_TARGET:-bee42.com/containers/tools/go-dep:1.11.5-alpine3.8}
    ARG TARGET=${TARGET:-multiarch/alpine:armhf-v3.8}
    FROM ${GOLANG_TARGET} as builder
    …
    # Resulting App
    FROM ${TARGET}
    COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
    COPY --from=buillder /app/blinkt /app/blinkt
    WORKDIR /app
    ENTRYPOINT ["/app/blinkt"]
    

    View Slide

22. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Build with Multi Arch Images
    22
    build:
    case $${arch} in \
    amd64 ) target_image="multiarch/alpine:amd64-v3.8" ;; \
    arm ) target_image="multiarch/alpine:armhf-v3.8" ;; \
    arm64 ) target_image="multiarch/alpine:arm64-v3.8" ;; \
    esac ; \
    docker image build --no-cache \
    --build-arg TARGET=$${target_image} \
    --build-arg TARGET_ARCH=$${arch} \
    -t $(DOCKER_IMAGE):$(DOCKER_TAG)-$${arch} . ; \
    

    View Slide

23. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Build with Mutli Arch Images
    23
    build-multiarch:
    @for arch in $(MULTIARCH); do \
    case $${arch} in \
    amd64 ) target_image="multiarch/alpine:amd64-v3.8" ;; \
    arm ) target_image="multiarch/alpine:armhf-v3.8" ;; \
    arm64 ) target_image="multiarch/alpine:arm64-v3.8" ;; \
    esac ; \
    docker image build --no-cache \
    --build-arg TARGET=$${target_image} \
    --build-arg TARGET_ARCH=$${arch} \
    --build-arg VERSION=`cat VERSION` \
    --build-arg VCS_REF=$(DOCKER_TAG) \
    --build-arg BUILD_DATE=`date -u +"%Y-%m-%dT%H:%M:%SZ"` \
    -t $(DOCKER_IMAGE):$(DOCKER_TAG)-$${arch} . ; \
    done
    

    View Slide

24. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    24
    

    View Slide

25. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Create Multiarch Manifest
    25
    push-multiarch:
    @echo "Create and push multiarch manifest: "
    @for arch in $(MULTIARCH); do \
    docker image push $(DOCKER_IMAGE):$(DOCKER_TAG)-$${arch} ; \
    done
    @docker manifest create $(DOCKER_IMAGE):$(MANIFEST_TAG) \
    $(DOCKER_IMAGE):$(DOCKER_TAG)-amd64 \
    $(DOCKER_IMAGE):$(DOCKER_TAG)-arm \
    $(DOCKER_IMAGE):$(DOCKER_TAG)-arm64
    @for arch in $(MULTIARCH); do \
    case $${arch} in \
    amd64 ) manifest_annotate="" ;; \
    arm ) manifest_annotate="--os linux --arch arm" ;; \
    arm64 ) manifest_annotate="--os linux --arch arm64 --variant armv8" ;; \
    esac ; \
    docker manifest annotate $(DOCKER_IMAGE):$(MANIFEST_TAG) $(DOCKER_IMAGE):$(DOCKER_TAG)-
    $${arch} $${manifest_annotate} ;\
    done
    @docker manifest push $(DOCKER_IMAGE):$(MANIFEST_TAG)
    

    View Slide

26. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Docker 18.09 

    client experimental feature
    26
    mkdir -p ~/.docker
    cat > "$HOME/.docker/config.json" <{
    "experimental": "enabled"
    }
    EOF
    https://docs.docker.com/engine/reference/commandline/manifest/
    https://docs.docker.com/registry/spec/manifest-v2-2/
    

    View Slide

27. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Add Metadata
    27
    # Metadata
    ARG VCS_REF
    ARG BUILD_DATE
    ARG VERSION
    # Metadata
    LABEL maintainer="bee42 cloud native crew " \
    org.opencontainers.image.title="blinkt" \
    org.opencontainers.image.version="${VERSION}" \
    org.opencontainers.image.revision="${VCS_REF}" \
    org.opencontainers.image.created="${BUILD_DATE}" \
    org.opencontainers.image.url="https://r-gitlab.bee42.com/containers/examples/k8s-client/blinkt/" \
    org.opencontainers.image.source="https://gitlab.bee42.com/containers/examples/k8s-client/blinkt/" \
    org.opencontainers.image.authors="bee42 cloud native crew " \
    org.opencontainers.image.vendor="bee42 solutions gmbh" \
    org.opencontainers.image.licenses="Apache-2.0" \
    com.bee42.image.type="service-stateless" \
    https://github.com/opencontainers/image-spec/blob/master/annotations.md
    

    View Slide

28. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Control devices with docker
    28
    docker run --privileged -d blinkt
    docker run --device /dev/gpiomem -d blinkt
    You can perform GPIO with user privileges by interacting with the virtual files under /sys/class/gpio.
    Less Privileged
    Add device
    Use the sysfs GPIO interface
    docker run -v /sys:/sys -d blinkt
    

    View Slide

29. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    29
    • Kubernetes is a container orchestrator.
    • It’s how to run containers at scale.
    • It’s a very active open-source platform with lots of
    contributors, start at 6. June 2014
    • Originally developed by Google and 

    donated to Cloud Native Computing Foundation
    

    View Slide

30. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    30
    

    View Slide

31. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    31
    

    View Slide

32. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    32
    https://github.com/bee42/kubernetes-on-embedded
    https://blog.hypriot.com/post/setup-kubernetes-raspberry-pi-cluster/
    Blinkt - Demo
    https://github.com/apprenda/blinkt-k8s-controller
    https://github.com/StefanScherer/swarm-monitor
    

    View Slide

33. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    33
    Your Mac Ethernet Switch
    bee42-crew—
    03-001
    bee42-
    crew-03-002
    bee42-
    crew-03-003
    Edge Max
    DNS
    192.168.42.31
    192.168.42.32
    192.168.42.33
    192.168.42.101 192.168.42.1
    Master
    Nodes
    Raspberry PI 3+
    armv7
    bee42-crew—
    03-004
    192.168.42.34
    Nodes
    Raspberry PI 3+
    arm64 bee42-crew—
    03-005
    192.168.42.35
    Nodes
    UP Board
    amd64
    192.168.1.230
    

    View Slide

34. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    34
    OS Root FS
    Image-raw-builder
    rpi kernel
    Image-builder
    https://github.com/hypriot/image-builder-rpi
    Flash ISO
    Ansible/Kubeadm
    K8s-Master
    K8s-Node
    https://github.com/bee42/kubernetes-on-embedded
    https://github.com/DieterReuter/image-builder-rpi64
    Flash USB
    Manuel Install Ubuntu
    Update Kernel
    Actvate Devices
    Kubeadm
    K8s-Node
    

    View Slide

35. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    35
    API Server
    blinkt-k8s-controller
    Pods
    App Pods
    template:
    metadata:
    labels:
    app: httpd
    blinkt: show
    blinktColor: 00FF00
    Blink device
    manage
    watch
    Set LED
    nodeSelector:
    deviceType: blinkt
    tolerations:
    - effect: NoSchedule
    key: node-role.kubernetes.io/master
    Blinkt to go
    

    View Slide

36. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    36
    

    View Slide

37. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Kubernetes Deploy Blinkt
    37
    

    View Slide

38. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    More to add…
    38
    

    View Slide

39. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Sometimes 

    Master need more time to boot
    39
    sudo sed -i 's/failureThreshold: 8/failureThreshold: 20/g' /etc/kubernetes/manifests/kube-apiserver.yaml
    sudo sed -i 's/initialDelaySeconds: [0-9]\+/initialDelaySeconds: 360/' /etc/kubernetes/manifests/kube-
    apiserver.yaml
    

    View Slide

40. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Facts hybrid K8s adventure
    40
    •Timing problems to install the K8s master
    •Wrong kernel version to supports weave
    •Build your own OS with virtual machines and tricks…
    •Build multi arch binaries need time
    •Use Boards with IO PIN from Raspberry
    •Use Pimoroni python libs or gobots
    •Learn stop, boot reinstall a K8s Clusters
    

    View Slide

41. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    rethink IT
    Build hyprid kubernetes cluster
    with embedded machines is a
    funny adventure…
    41
    

    View Slide

42. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Kubernetes poster
    pre registration started
    
    https://tinyurl.com/y9js3p7w
    42
    delivery starts at 42ten day of the year 2019
    PREVIEW
    PREVIEW
    WE
    Ask me to buy today a printed copy…
    

    View Slide

43. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Cloud Native System Architect
    &
    bee42 founder
    Peter Roßbach
    
    @PRossbach
    
    peter.rossbach@bee42.com
    
    https://bee42.com
    
    https://devops-gathering.io
    
    43
    #DOG19 11.-13. March 2019 at Bochum
    Discount Code: KubeCologne-15
    

    View Slide

44. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    bee42 Trainings
    44
    https://bee42.com/de/trainings
    25.-26 February 2019 Berlin: KubeCologne-K8s-20
    https://bee42.com/de/events/container-lab-mit-kubernetes-berlin/
    

    View Slide

45. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    45
    We hiring :-)
    

    https://bit.ly/2K8DtRu 

    jobs@bee42.com

    @bee42solutions
    

    View Slide