Shiny New Container Tools

Are the Dockerfile and the Docker Daemon already obsolete?

In fact, there are some promising new tools: GraalVM creates small binary files from Java byte code, Buildah creates OCI-compliant container images without any docker, and gVisor protects the Linux kernel by running containers in a real sandbox.

What problems do these tools fix? Is it worth using them? What has to be considered?

1. INNOQ / Shiny New Container Tools Shiny New Container Tools
2. INNOQ / Shiny New Container Tools "Simplicity is prerequisite for reliability." - Dijkstra CHRISTOPHER SCHMIDT @fakod senior consultant at INNOQ Schweiz GmbH Kubernetes Container Architecture Training Consulting
3. INNOQ / Shiny New Container Tools What we go through (tool kaleidoscope) Container Image • Docker • Kaniko • buildah Runtime • Docker • Kata Container • gVisor CI • Jenkins / Gitlab CI • Jenkins-x • KNative Build • Argo
4. INNOQ / Shiny New Container Tools ContainerConf 2018 Container Runtimes
5. INNOQ / Shiny New Container Tools Container Trends • Standardization through OCI • Trying to get rid of Docker • Separation of Runtime and Build • Specific runtimes that only support Kubernetes CRI • Image builds running in a (unprivileged) container • Increased isolation
6. INNOQ / Shiny New Container Tools Open Container Initiative The OCI is a lightweight, open governance structure (project), formed under the auspices of the Linux Foundation, for the express purpose of creating open industry standards around container formats and runtime. The OCI currently contains two specifications: • the Runtime Specification (runtime-spec) • the Image Specification (image-spec)
7. INNOQ / Shiny New Container Tools Runtime Specification (runtime-spec) aims to specify • the configuration • execution environment • lifecycle of a container e.g. runc - Reference implementation of OCI runtime
8. INNOQ / Shiny New Container Tools Image Specification (image-spec) defines an OCI Image, consisting of • a manifest • an image index (optional) • a set of filesystem layers • a configuration
9. INNOQ / Shiny New Container Tools Container Runtimes Stronger Container Isolation?
10. INNOQ / Shiny New Container Tools What is Docker Infrastructure OS Hypervisor Guest OS Guest OS bin / lib bin / lib Apps Apps Infrastructure OS Docker Engine bin / lib bin / lib Apps Apps VMs Container
11. INNOQ / Shiny New Container Tools Container Isolation isolation Application Container Infrastructure Linux Kernel • cgroups • namespaces OS Union FS
12. INNOQ / Shiny New Container Tools Rule-based Execution e.g. Seccomp, SELinux and AppArmor fine-grained security policy for an application or container → Pod Security Policies on Kubernetes Application Container Kernel limited system calls rule enforcement
13. INNOQ / Shiny New Container Tools INNOQ / Shiny New Container Tools Pod Security Policies apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: restricted annotations: seccomp.security.alpha.kubernetes.io/defaultProfileName: 'docker/default' apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default' spec: privileged: false requiredDropCapabilities: - ALL runAsUser: rule: 'MustRunAsNonRoot' supplementalGroups: rule: 'MustRunAs' ranges: - min: 1 max: 65535 fsGroup: rule: 'MustRunAs' ranges: - min: 1 max: 65535 readOnlyRootFilesystem: false
14. INNOQ / Shiny New Container Tools Kernel “Proxy” seccomp on steroids Application Container Kernel “Proxy” Host Kernel system calls limited system calls
15. INNOQ / Shiny New Container Tools VM (Virtual Hardware) Machine-level Virtualization Application Container Guest Kernel Host Kernel system calls system calls
16. INNOQ / Shiny New Container Tools Container Runtimes Docker
17. INNOQ / Shiny New Container Tools Docker -- Architecture • Docker Client CLI: ◦ Uses the Docker REST API to interact with the Docker Daemon • Dockerd: ◦ Communicates with other daemons to manage Docker services • Containerd: ◦ Manages e.g. images, containers, networks, and volumes ◦ Manages container lifecycle • Runc: ◦ CLI tool for spawning and running containers according to the OCI runtime-spec
18. INNOQ / Shiny New Container Tools Docker -- To run a container... dockerd containerd containerd-shim Application … dockerd creates the image, passes it to containerd … containerd calls containerd-shim that uses runC to run the container … containerd-shim allows the runC to exit after it starts the container
19. INNOQ / Shiny New Container Tools Container Runtimes Kata Container
20. INNOQ / Shiny New Container Tools Kata Container • Runs on multiple hypervisors • Compatible with ◦ OCI specification for Docker containers ◦ Container Runtime Interface (CRI) for Kubernetes • Project has six components: ◦ Agent, Runtime, Proxy, Shim, Kernel and packaging of QEMU 2.11
21. INNOQ / Shiny New Container Tools Kata Container why? • Container share ◦ Kernel ◦ I/O ◦ Network ◦ Memory → A compromised container can also threaten other container or the host itself
22. INNOQ / Shiny New Container Tools Kata Runtime Kata Container -- Architecture QEMU/Kernel-based Virtual Machine VM for each container/pod → a VM that looks and feels like a container. dockerd containerd containerd-shim QEMU* proxy shim
23. INNOQ / Shiny New Container Tools Kata Container -- Architecture VM agent gRPC Server Kata Runtime container management commands agent: Spawn containers and processes inside this virtual machine, on behalf of the runtime running on the host proxy container virtio serial interface
24. INNOQ / Shiny New Container Tools Container Runtimes gVisor
25. INNOQ / Shiny New Container Tools gVisor • Intercepts application system calls • Implements most kernel primitives • will not allow the application to directly control the system calls it makes • Allows it to provide a flexible resource footprint • Lowering the fixed costs of virtualization • But lower application compatibility and higher per system-call overhead Application gVisor Host Kernel system calls limited system calls
26. INNOQ / Shiny New Container Tools Container Runtimes rkt
27. INNOQ / Shiny New Container Tools rkt • Can download, cryptographically verify and run container • no centralized "init" daemon (launches containers directly from client commands) • can optionally use lkvm or qemu-kvm as an additional security measure • Supports a native POD implementation • Cannot build images
28. INNOQ / Shiny New Container Tools stage1 rkt Architecture -- stages stage0 rkt systemd-nspawn systemd systemd-journal stage2 app1 stage2 app2 • Stage0: invoking process • stage0 -> stage1: An ordinary exec is being used to replace the stage0 process with the stage1 entrypoint • stage1 -> stage2: The stage1 entrypoint uses its mechanism to invoke the stage2 app executables.
29. INNOQ / Shiny New Container Tools rkt -- Running with KVM Alternative stage1 • to run a pod within a virtual machine with ◦ own operating system kernel ◦ hypervisor isolation • support for KVM hypervisor - LKVM or QEMU • contains a Linux kernel that is executed under hypervisor (LKVM or QEMU)
30. INNOQ / Shiny New Container Tools ContainerConf 2018 Building Images
31. INNOQ / Shiny New Container Tools Docker build... • Needs Docker • Issues with running it in a container ◦ Has to map docker socket ◦ Or fancy DinD tricks • Dockerfile only
32. INNOQ / Shiny New Container Tools img Builds Images... • Standalone, daemon-less, unprivileged • OCI compatible • Using Dockerfile • More cache-efficient • while executing multiple build stages concurrently But still requires a privileged container
33. INNOQ / Shiny New Container Tools buildah • Specializes in building OCI images • No daemon • Can use Dockerfile, but ... • ...allows other scripting languages (e.g. bash script) as well ◦ Allows more control of generated layers ◦ Allows to “code” an image build
34. INNOQ / Shiny New Container Tools Kaniko • Builds container images from a Dockerfile • Works as a container (e.g. in a Kubernetes cluster) • Runs without any special privileges or permissions • Does not need to be a privileged container -> use it for K8s builds like KNative Build or Argo
35. INNOQ / Shiny New Container Tools ContainerConf 2018 Application Containerization
36. INNOQ / Shiny New Container Tools Container Wishlist • Application with... ◦ Little (or no) dependencies ◦ Fast startup, fast teardown ◦ Little runtime resources • Small images • Secure Fast -> flexible cluster usage, little MTBF Small images -> faster image download, less resources, little MTBF
37. INNOQ / Shiny New Container Tools JVM Binaries Why? • Compile once, run anywhere is no longer needed • Compiled ahead-of-time ◦ No warm-up phase as for just-in-time compilers ◦ Instant startup time • Lower resource consumption • Seamless interop with native code • Little dependencies
38. INNOQ / Shiny New Container Tools JVM Binaries -- GraalVM 1. High-performance modern Java 2. Low-footprint, fast-startup Java 3. Combine JavaScript, Java, Ruby, and R 4. Run native languages on the JVM 5. Tools that work across all languages 6. Extend a JVM-based application 7. Extend a native application 8. Java code as a native library 9. Polyglot in the database 10. Create your own language https://medium.com/graalvm/graalvm-ten-things-12d9111f307d
39. INNOQ / Shiny New Container Tools JVM Binaries -- others • Scala Native • Kotlin/Native • ... Webassembly (?)
40. INNOQ / Shiny New Container Tools Google Container Tools: Base Images Distroless: • Contain only the application and its runtime dependencies • do not contain... ◦ package managers, shells or any other programs • For... ◦ mostly-statically compiled languages like Go, Rust or D ◦ openjdk-based Java runtime ◦ . . .
41. INNOQ / Shiny New Container Tools Container Vulnerability Scanning e.g. Clair: • Vulnerability data is continuously imported • Correlated with the indexed image contents e.g. Yair: • Interacts with Clair • Designed for the execution inside a CI Job
42. INNOQ / Shiny New Container Tools ContainerConf 2018 CI/CD
43. INNOQ / Shiny New Container Tools Trends CI/CD • Highest possible level of automation • GitHub has hooks and an API • There is a Kubernetes cluster anyway • Cloud Native … means ◦ No single build processor ◦ Using Kubernetes primitives (CRD, POD etc.) ◦ Should support Microservice/SCS
44. INNOQ / Shiny New Container Tools CI/CD Tool Classification • Traditional dedicated CI/CD server, executes scripts in socalled runners, provides separate UI ◦ Jenkins, GitLab CI • CI/CD using Github primitives (using its API), without a dedicated UI. Often called Cloud Native GitOps or ChatOps ◦ Prow, Jenkins-x Serverless • In-Cluster CI/CD workflow engines, with dedicated UI, working with Git Repos (may use Github trigger) ◦ KNative-Build, Argo
45. INNOQ / Shiny New Container Tools GitOps / ChatOps GitHub API WebHook Prow KNative Build CRD Jenkins Serverless Container
46. INNOQ / Shiny New Container Tools GitOps / ChatOps
47. INNOQ / Shiny New Container Tools INNOQ / Shiny New Container Tools apiVersion: build.knative.dev/v1alpha1 kind: BuildTemplate metadata: name: kaniko spec: parameters: - name: IMAGE description: The name of the image to push - name: DOCKERFILE description: Path to the Dockerfile to build. default: /workspace/Dockerfile steps: - name: build-and-push image: gcr.io/kaniko-project/executor args: - --dockerfile=${DOCKERFILE} - --destination=${IMAGE} - --skip-tls-verify - name: deploy image: tazjin/kontemplate args: ['kubectl', 'apply', '-f', '/workspace/deploy.yaml'] KNative Build Template
48. INNOQ / Shiny New Container Tools INNOQ / Shiny New Container Tools apiVersion: build.knative.dev/v1alpha1 kind: Build metadata: name: test-build spec: serviceAccountName: build-bot source: git: url: https://github.com/FaKod/knative-build-test.git revision: master template: name: kaniko arguments: - name: IMAGE value: registry.knative-build.svc.cluster.local:5000/app-from-source:latest KNative Build
49. INNOQ / Shiny New Container Tools INNOQ / Shiny New Container Tools apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: generateName: go-build-ci- spec: entrypoint: influxdb-ci arguments: parameters: - name: repo value: . . . templates: - name: checkout inputs: artifacts: - name: source path: /src git: repo: "{{workflow.parameters.repo}}" revision: "{{workflow.parameters.revision}}" outputs: artifacts: - name: source path: /src container: image: golang:1.9.2 command: ["/bin/sh", "-c"] args: ["cd /src && git status && ls -l"] Argo Workflow - name: build inputs: artifacts: - name: source path: /workspace container: image: gcr.io/kaniko-project/executor args: - --destination={{workflow.parameters.image}} - --skip-tls-verify
50. INNOQ / Shiny New Container Tools Recommendation • Git Repos are the source of truth • Use your Kubernetes cluster (“inversion of configuration”) • Use your build script (Dockerfile) ◦ Multi stage builds ◦ Unit tests
51. INNOQ / Shiny New Container Tools Finally...
52. INNOQ / Shiny New Container Tools Finally... • Runtimes and security policies are about protecting the host • runc is less secure than gVisor is less secure than Kata • runc is faster than gVisor is faster than Kata • Building images is something different than running them • Binaries have advantages • Scan your images • be Cloud Native