$30 off During Our Annual Pro Sale. View Details »

Hybrid Kubernetes Cluster 
on Embedded Devices

Hybrid Kubernetes Cluster 
on Embedded Devices

In this talk, Peter will talk about how Docker was extended from x86 Linux to the ARM platforms for your hyprid kubernetes clusters.

Peter will also demo some of the new features of the current Docker CE engine to manage Kubernetes Clusters with both x86 and Raspberry PI Linux nodes.

Peter Rossbach

March 09, 2019
Tweet

More Decks by Peter Rossbach

Other Decks in Technology

Transcript

  1. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    1
    Hybrid Kubernetes Cluster 

    on Embedded Devices

    View Slide

  2. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Docker on ARM
    2
    Simple and helpful…
    First rumor at mid 2014
    Hybrid OS Feb 2015
    Docker Con 2015 - Raspberry PI Httpd Challenge
    Docker Engine support ARM mid 2016
    Kubernetes on ARM starts at Dec 2015
    Docker support Multi Arch Binary mid 2017

    View Slide

  3. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Challenges
    3
    •Compiling things, on small embedded boards needs time
    •Cross Compilation is your friend
    •Find ready to use images on the official registry
    •Multi Arch Binaries
    •Reduce image Size
    •Control hardware devices
    •Build a cluster of raspberry PI’s
    •Let Kubernetes install on PI’s

    View Slide

  4. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    4

    Adventures needs clever friends….

    View Slide

  5. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Reduce Size
    5
    •Prepare a Tool Container
    •Multi Stage Build
    •Compression
    •Improve quality

    View Slide

  6. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Choose the right base images
    6
    https://www.codacy.com/blog/five-ways-to-slim-your-docker-images/

    View Slide

  7. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Strategies to reduce image size
    7
    •Think Carefully About Your Application’s Needs
    •Use a Small Base Image
    •Use as Few Layers As Possible
    •Use .dockerignore files
    •Squash Docker Images

    View Slide

  8. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Builder Pattern
    8
    Tool Base Image Compile Package
    Prepare Resulting
    Image
    Source

    View Slide

  9. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    go dep tool base container
    9
    ARG BASE_IMAGE=${BASE_IMAGE:-golang:1.11.5-alpine3.8}
    FROM ${BASE_IMAGE}
    LABEL maintainer="Peter Rossbach "
    ARG DEP_VERSION=${DEP_VERSION:-0.5.0}
    RUN apk update; \
    apk add --no-cache \
    ca-certificates \
    curl \
    git \
    make \
    openssl; \
    curl -L -s https://github.com/golang/dep/releases/download/v${DEP_VERSION}/dep-linux-amd64 \
    -o /bin/dep; \
    chmod +x /bin/dep; \
    rm -rf /var/cache/apk/*; \
    rm -rf /tmp/*;

    View Slide

  10. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Builder Image
    10
    FROM bee42.com/containers/tools/go-dep:1.11.5-alpine3.8 as builder
    ARG TARGET_ARCH=${TARGET_ARCH:-amd64}
    ARG APP=${APP:-bee42.com/containers/examples/k8s-client/blinkt}
    ENV CGO_ENABLED=0
    ENV APP_GOPATH $GOPATH/src/$APP
    WORKDIR $APP_GOPATH
    RUN mkdir -p $APP_GOPATH
    COPY vendor/ $APP_GOPATH/vendor/
    COPY Gopkg.* $APP_GOPATH/
    COPY *.go $APP_GOPATH
    RUN cd $APP_GOPATH && \
    GOOS=linux GOARCH=${TARGET_ARCH} GOARM=${GOARM:-7} go build -a --installsuffix cgo --
    ldflags="-s" -o blinkt
    # Resulting App

    View Slide

  11. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Resulting Image
    11
    FROM bee42.com/containers/tools/go-dep:1.11.5-alpine3.8 as builder

    # Resulting App
    FROM alpine:v3.8
    COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
    COPY --from=builder /app/blinkt /app/blinkt
    WORKDIR /app
    ENTRYPOINT ["/app/blinkt"]

    View Slide

  12. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    UPX
    12
    https://upx.github.io
    UPX achieves an excellent compression ratio and offers very fast decompression.

    View Slide

  13. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    UPX Package Compression
    13
    # Optimize binary size
    FROM alpine:v3.8 as packager
    ARG APP=${APP:-bee42.com/containers/examples/k8s-client/blinkt}
    ENV APP_GOPATH /go/src/$APP
    ARG TARGET_ARCH=${TARGET_ARCH:-amd64}
    ARG UPX_VERSION=${UPX_VERSION:-3.95}
    RUN apk add --no-cache xz binutils curl && echo ${TARGET_ARCH}
    RUN curl -sL -o /tmp/upx-${UPX_VERSION}-${TARGET_ARCH}_linux.tar.xz \
    https://github.com/upx/upx/releases/download/v${UPX_VERSION}/upx-${UPX_VERSION}-$
    {TARGET_ARCH}_linux.tar.xz && \
    xz -d -c /tmp/upx-${UPX_VERSION}-${TARGET_ARCH}_linux.tar.xz | \
    tar -xOf - upx-${UPX_VERSION}-${TARGET_ARCH}_linux/upx > /bin/upx && \
    chmod a+x /bin/upx && \
    rm /tmp/upx-${UPX_VERSION}-${TARGET_ARCH}_linux.tar.xz
    COPY --from=builder $APP_GOPATH/blinkt /app/blinkt
    RUN cd /app && \
    strip --strip-unneeded blinkt && \
    upx blinkt
    # Resulting App

    View Slide

  14. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Use Upx package compression
    14
    # Optimize binary size
    FROM alpine:v3.8 as packager

    # Resulting App
    FROM alpine:v3.8
    COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
    COPY --from=packager /app/blinkt /app/blinkt
    WORKDIR /app
    ENTRYPOINT ["/app/blinkt"]
    Safe 20-40% image size

    View Slide

  15. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    15
    Build Cross Compiled Binaries: 

    qemu static at your ADM64 Boxes
    docker run --rm --privileged multiarch/qemu-user-static:register --reset
    https://hub.docker.com/r/multiarch/qemu-user-static
    for target_arch in aarch64 arm x86_64; do
    wget -N https://github.com/multiarch/qemu-user-static/releases/download/v2.9.1-1/x86_64_qemu-$
    {target_arch}-static.tar.gz
    tar -xvf x86_64_qemu-${target_arch}-static.tar.gz
    done
    https://lobradov.github.io/Building-docker-multiarch-images/
    Registry kernel modules
    Build with emulation binary

    View Slide

  16. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Multiarch Docker hub
    16
    https://hub.docker.com/u/multiarch/
    https://github.com/multiarch

    View Slide

  17. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Multi Arch build matrix
    17
    https://doi-janky.infosiftr.net/job/multiarch/job/arm32v7/job/httpd/

    View Slide

  18. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Build with separate Dockerfiles
    18
    cat >Dockerfile.amd64 <FROM amd64/alpine:3.7
    # Not necessary for the arch where host and target are the same
    # COPY qemu-x86_64-static /usr/bin/
    RUN apk --no-cache --update add nginx
    EXPOSE 80
    CMD ["nginx", "-g", "daemon off;"]
    EOF


    cat >Dockerfile.arm32v6 <FROM arm32v6/alpine:3.7
    COPY qemu-arm-static /usr/bin/
    RUN apk --no-cache --update add nginx
    EXPOSE 80
    CMD ["nginx", "-g", "daemon off;“]
    EOF

    View Slide

  19. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Resulting Image
    19
    for arch in amd64 arm32v6; do
    docker build -f Dockerfile.${arch} -t bee42/nginx:${arch}-latest .
    docker push bee42/nginx:${arch}-latest
    done

    View Slide

  20. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Builder Image with Args
    20
    ARG GOLANG_TARGET=${GOLANG_TARGET:-bee42.com/containers/tools/go-dep:1.11.5-alpine3.8}
    ARG TARGET=${TARGET:-multiarch/alpine:armhf-v3.8}
    FROM ${GOLANG_TARGET} as builder
    ARG TARGET_ARCH=${TARGET_ARCH:-arm}
    ARG APP=${APP:-bee42.com/containers/examples/k8s-client/blinkt}
    ENV CGO_ENABLED=0
    ENV APP_GOPATH $GOPATH/src/$APP
    WORKDIR $APP_GOPATH
    RUN mkdir -p $APP_GOPATH
    COPY vendor/ $APP_GOPATH/vendor/
    COPY Gopkg.* $APP_GOPATH/
    COPY *.go $APP_GOPATH
    RUN cd $APP_GOPATH && \
    GOOS=linux GOARCH=${TARGET_ARCH} GOARM=${GOARM:-7} go build -a --installsuffix cgo --
    ldflags="-s" -o blinkt
    # Resulting App

    View Slide

  21. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Resulting Image
    21
    ARG GOLANG_TARGET=${GOLANG_TARGET:-bee42.com/containers/tools/go-dep:1.11.5-alpine3.8}
    ARG TARGET=${TARGET:-multiarch/alpine:armhf-v3.8}
    FROM ${GOLANG_TARGET} as builder

    # Resulting App
    FROM ${TARGET}
    COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
    COPY --from=buillder /app/blinkt /app/blinkt
    WORKDIR /app
    ENTRYPOINT ["/app/blinkt"]

    View Slide

  22. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Build with Multi Arch Images
    22
    build:
    case $${arch} in \
    amd64 ) target_image="multiarch/alpine:amd64-v3.8" ;; \
    arm ) target_image="multiarch/alpine:armhf-v3.8" ;; \
    arm64 ) target_image="multiarch/alpine:arm64-v3.8" ;; \
    esac ; \
    docker image build --no-cache \
    --build-arg TARGET=$${target_image} \
    --build-arg TARGET_ARCH=$${arch} \
    -t $(DOCKER_IMAGE):$(DOCKER_TAG)-$${arch} . ; \

    View Slide

  23. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Build with Mutli Arch Images
    23
    build-multiarch:
    @for arch in $(MULTIARCH); do \
    case $${arch} in \
    amd64 ) target_image="multiarch/alpine:amd64-v3.8" ;; \
    arm ) target_image="multiarch/alpine:armhf-v3.8" ;; \
    arm64 ) target_image="multiarch/alpine:arm64-v3.8" ;; \
    esac ; \
    docker image build --no-cache \
    --build-arg TARGET=$${target_image} \
    --build-arg TARGET_ARCH=$${arch} \
    --build-arg VERSION=`cat VERSION` \
    --build-arg VCS_REF=$(DOCKER_TAG) \
    --build-arg BUILD_DATE=`date -u +"%Y-%m-%dT%H:%M:%SZ"` \
    -t $(DOCKER_IMAGE):$(DOCKER_TAG)-$${arch} . ; \
    done

    View Slide

  24. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    24

    View Slide

  25. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Create Multiarch Manifest
    25
    push-multiarch:
    @echo "Create and push multiarch manifest: "
    @for arch in $(MULTIARCH); do \
    docker image push $(DOCKER_IMAGE):$(DOCKER_TAG)-$${arch} ; \
    done
    @docker manifest create $(DOCKER_IMAGE):$(MANIFEST_TAG) \
    $(DOCKER_IMAGE):$(DOCKER_TAG)-amd64 \
    $(DOCKER_IMAGE):$(DOCKER_TAG)-arm \
    $(DOCKER_IMAGE):$(DOCKER_TAG)-arm64
    @for arch in $(MULTIARCH); do \
    case $${arch} in \
    amd64 ) manifest_annotate="" ;; \
    arm ) manifest_annotate="--os linux --arch arm" ;; \
    arm64 ) manifest_annotate="--os linux --arch arm64 --variant armv8" ;; \
    esac ; \
    docker manifest annotate $(DOCKER_IMAGE):$(MANIFEST_TAG) $(DOCKER_IMAGE):$(DOCKER_TAG)-
    $${arch} $${manifest_annotate} ;\
    done
    @docker manifest push $(DOCKER_IMAGE):$(MANIFEST_TAG)

    View Slide

  26. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Docker 18.09 

    client experimental feature
    26
    mkdir -p ~/.docker
    cat > "$HOME/.docker/config.json" <{
    "experimental": "enabled"
    }
    EOF
    https://docs.docker.com/engine/reference/commandline/manifest/
    https://docs.docker.com/registry/spec/manifest-v2-2/

    View Slide

  27. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Add Metadata
    27
    # Metadata
    ARG VCS_REF
    ARG BUILD_DATE
    ARG VERSION
    # Metadata
    LABEL maintainer="bee42 cloud native crew " \
    org.opencontainers.image.title="blinkt" \
    org.opencontainers.image.version="${VERSION}" \
    org.opencontainers.image.revision="${VCS_REF}" \
    org.opencontainers.image.created="${BUILD_DATE}" \
    org.opencontainers.image.url="https://r-gitlab.bee42.com/containers/examples/k8s-client/blinkt/" \
    org.opencontainers.image.source="https://gitlab.bee42.com/containers/examples/k8s-client/blinkt/" \
    org.opencontainers.image.authors="bee42 cloud native crew " \
    org.opencontainers.image.vendor="bee42 solutions gmbh" \
    org.opencontainers.image.licenses="Apache-2.0" \
    com.bee42.image.type="service-stateless" \
    https://github.com/opencontainers/image-spec/blob/master/annotations.md

    View Slide

  28. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Control devices with docker
    28
    docker run --privileged -d blinkt
    docker run --device /dev/gpiomem -d blinkt
    You can perform GPIO with user privileges by interacting with the virtual files under /sys/class/gpio.
    Less Privileged
    Add device
    Use the sysfs GPIO interface
    docker run -v /sys:/sys -d blinkt

    View Slide

  29. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    29
    • Kubernetes is a container orchestrator.
    • It’s how to run containers at scale.
    • It’s a very active open-source platform with lots of
    contributors, start at 6. June 2014
    • Originally developed by Google and 

    donated to Cloud Native Computing Foundation

    View Slide

  30. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    30

    View Slide

  31. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    31

    View Slide

  32. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    32
    https://github.com/bee42/kubernetes-on-embedded
    https://blog.hypriot.com/post/setup-kubernetes-raspberry-pi-cluster/
    Blinkt - Demo
    https://github.com/apprenda/blinkt-k8s-controller
    https://github.com/StefanScherer/swarm-monitor

    View Slide

  33. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    33
    Your Mac Ethernet Switch
    bee42-crew—
    03-001
    bee42-
    crew-03-002
    bee42-
    crew-03-003
    Edge Max
    DNS
    192.168.42.31
    192.168.42.32
    192.168.42.33
    192.168.42.101 192.168.42.1
    Master
    Nodes
    Raspberry PI 3+
    armv7
    bee42-crew—
    03-004
    192.168.42.34
    Nodes
    Raspberry PI 3+
    arm64 bee42-crew—
    03-005
    192.168.42.35
    Nodes
    UP Board
    amd64
    192.168.1.230

    View Slide

  34. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    34
    OS Root FS
    Image-raw-builder
    rpi kernel
    Image-builder
    https://github.com/hypriot/image-builder-rpi
    Flash ISO
    Ansible/Kubeadm
    K8s-Master
    K8s-Node
    https://github.com/bee42/kubernetes-on-embedded
    https://github.com/DieterReuter/image-builder-rpi64
    Flash USB
    Manuel Install Ubuntu
    Update Kernel
    Actvate Devices
    Kubeadm
    K8s-Node

    View Slide

  35. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    35
    API Server
    blinkt-k8s-controller
    Pods
    App Pods
    template:
    metadata:
    labels:
    app: httpd
    blinkt: show
    blinktColor: 00FF00
    Blink device
    manage
    watch
    Set LED
    nodeSelector:
    deviceType: blinkt
    tolerations:
    - effect: NoSchedule
    key: node-role.kubernetes.io/master
    Blinkt to go

    View Slide

  36. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    36

    View Slide

  37. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Kubernetes Deploy Blinkt
    37

    View Slide

  38. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    More to add…
    38

    View Slide

  39. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Sometimes 

    Master need more time to boot
    39
    sudo sed -i 's/failureThreshold: 8/failureThreshold: 20/g' /etc/kubernetes/manifests/kube-apiserver.yaml
    sudo sed -i 's/initialDelaySeconds: [0-9]\+/initialDelaySeconds: 360/' /etc/kubernetes/manifests/kube-
    apiserver.yaml

    View Slide

  40. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Facts hybrid K8s adventure
    40
    •Timing problems to install the K8s master
    •Wrong kernel version to supports weave
    •Build your own OS with virtual machines and tricks…
    •Build multi arch binaries need time
    •Use Boards with IO PIN from Raspberry
    •Use Pimoroni python libs or gobots
    •Learn stop, boot reinstall a K8s Clusters

    View Slide

  41. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    rethink IT
    Build hyprid kubernetes cluster
    with embedded machines is a
    funny adventure…
    41

    View Slide

  42. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Kubernetes poster
    pre registration started

    https://tinyurl.com/y9js3p7w
    42
    delivery starts at 42ten day of the year 2019
    PREVIEW
    PREVIEW
    WE
    Ask me to buy today a printed copy…

    View Slide

  43. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    Cloud Native System Architect
    &
    bee42 founder
    Peter Roßbach

    @PRossbach

    [email protected]

    https://bee42.com

    https://devops-gathering.io

    43
    #DOG19 11.-13. March 2019 at Bochum
    Discount Code: KubeCologne-15

    View Slide

  44. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    bee42 Trainings
    44
    https://bee42.com/de/trainings
    25.-26 February 2019 Berlin: KubeCologne-K8s-20
    https://bee42.com/de/events/container-lab-mit-kubernetes-berlin/

    View Slide

  45. Copyright 2019 bee42 solutions gmbh @PRossbach
    rethink IT - We improve your systems with passion
    45
    We hiring :-)

    https://bit.ly/2K8DtRu 

    [email protected]
    @bee42solutions

    View Slide