Docker Storage drivers for production

Docker Storage drivers for production

A small presentation of various issues with Docker, its storage drivers, Kernel capabilities and Kubernetes interaction

Fc1aaf1aea290852ccd0131463d3fe57?s=128

Benjamin Allot

October 23, 2018
Tweet

Transcript

  1. 2.

    FOR ANY STANDARD x86 SERVER PLATFORM 100% SOFTWARE SOLUTION PRIVATE

    CLOUD PUBLIC CLOUDS …… MULTI-CLOUD DATA CONTROLLER TO ACCESS AND MANAGE DATA ACROSS CLOUDS object & file storage in a single system · peer to peer architecture · unlimited scalability · unbounded scale-out performance · most adaptive set of robust data protection mechanisms · autonomous self-healing · designed in close collaboration with the biggest (cloud-scale) service providers in the world a single, unified API across all clouds to simplify application development · the only multi-cloud data management solution independent of the storage system · stores data in standard cloud format to make the data consumable directly by native cloud apps and services · true multi-cloud IT · global search across all managed data independent of cloud location …… … … …
  2. 3.

    Benjamin Allot Docker • Adoption early in 2014 • Tooling

    at Scality for our builds, then CI • S3 API compatible server deployed with docker Tech Leader of “Setup team” Trained as a Developer Unix Sysadmin for a living
  3. 7.

    • The good: The layers are stored efficiently thanks to

    a “copy on write” mechanism Docker : Storage Drivers
  4. 8.

    • The good: The layers are stored efficiently thanks to

    a “copy on write” mechanism • The bad: There is several storage drivers, each with their strengths and weaknesses Docker : Storage Drivers
  5. 9.

    • The good: The layers are stored efficiently thanks to

    a “copy on write” mechanism • The bad: There is several storage drivers, each with their strengths and weaknesses • The ugly: Bad combination of Docker, Storage Driver and Kernel can lead to issues (mostly upon stop) Docker : Storage Drivers
  6. 11.

    Btrfs/Zfs AUFS DEVICE- MAPPER OVERLAY OVERLAY2 Require disk formatting Not

    supported by kernel anymore since 3.18 Warning : bad performance for local-lvm Run out of inode easily Require disabling selinux, require Centos7.4 (kernel 3.10.0-693) Docker storage driver of choice = overlay2 - Best performance/stability with less requirements - With docker < 18.02, detection over kernel capabilities for overlay2 is buggy (require force storage driver for docker 17.03) - Educated bet on future Docker Storage Driver: which and why ?
  7. 12.

    • Configure it with /etc/docker/daemon.json file ◦ “storage-driver”: “overlay2” ◦

    “storage-opts”: [“overlay2.override_kernel_check=true”] • Double check the compatibility matrix • On RedHat/CentOS be wary of: ◦ XFS with a specific mkfs option mandatory with OverlayFS for /var/lib/docker ◦ Device or resource busy when stopping a container ◦ How to Setup OverlayFS on RedHat Docker : Storage Drivers
  8. 13.

    Docker : Storage Drivers issues Storage Driver Number of issues

    Number of issues open Device Mapper 184 41 Overlay (1 & 2) 150 30 Zfs 26 7 Btrfs 46 14 Aufs 87 24 (last 12th September 2017)
  9. 14.

    • Check the Storage Driver to use before deploying on

    production • Be wary of Kernel capabilities and Docker “default” Storage Driver decision • Future might be to use directly containerd 1.1+ (interesting history of graph drivers and why it isn’t supported “as is” in containerd) Docker : Summary
  10. 15.

    • To deploy our S3 connector (RING product) • To

    replicate object into several clouds with Zenko (Open Source) ◦ We used to use Docker Swarm ◦ Now we do Kubernetes What production ?
  11. 16.

    Why Kubernetes ? - Run everywhere, on any cloud =>

    provide API abstraction - Control plane run server side (compared to docker compose) - Self-healing - Auto-scaling (of pods, of cluster, of resources requests) - Huge set of plugins (centralised logging, monitoring, ingress) - Big community - Docker announcement to support Kubernetes in 2017 - Customers trust and want it
  12. 17.

    • An opinionated Kubernetes distribution with a focus on long-term

    on-prem deployments • A commitment to bare-metal • Open Source : check here MetalK8s • KOPS • EKS (one day ?) AWS • GKE GCP • AKS Azure • ? Bare Metal
  13. 18.

    1) Inventory precheck 2) ping (connectivity check) 3) precheck about

    centos (kernel) 4) precheck on storage 5) create lvm vg/lv 6) => call kubespray ! <= 7) register lv into kubespray 8) deploy nginx ingress 9) deploy prometheus + grafana 10) deploy elasticsearch + kibana MetalK8s: quality Inventory n etcd % 2 = 1 n master > 1 n node > 0 Connectivity
  14. 20.

    $ cat /proc/cgroups #subsys_name hierarchy num_cgroups enabled cpuset 10 3741

    1 cpu 2 3971 1 cpuacct 2 3971 1 memory 4 3971 1 devices 7 3971 1 freezer 8 3741 1 net_cls 3 3741 1 blkio 5 3971 1 perf_event 6 3741 1 hugetlb 11 3741 1 pids 9 3971 1 net_prio 3 3741 1 Cgroup : No space left on device ?
  15. 21.

    • A kernel cgroup bug for kernel prior to 4.0

    identified for Moby What we found so far
  16. 22.

    • A kernel cgroup bug for kernel < 4.0 •

    A runc change of behavior with kernel memory accounting What we found so far
  17. 23.

    • A kernel cgroup bug for kernel prior to 4.0

    identified for Moby • A runc change of behavior with kernel memory accounting • A Kubernetes bug identifying the issue What we found so far
  18. 24.

    • A kernel cgroup bug for kernel prior to 4.0

    identified for Moby • A runc change of behavior with kernel memory accounting • A Kubernetes bug identifying the issue • More precisely : this commit is responsible of the “bug” What we found so far
  19. 25.

    • A kernel cgroup bug for kernel prior to 4.0

    identified for Moby • A runc change of behavior with kernel memory accounting • A Kubernetes bug identifying the issue • More precisely : this commit is responsible of the “bug” • A chinese page describing the issue (thank you google translate) What we found so far
  20. 27.

    • Use a recent kernel, even on CentOS • Wait

    for the fix to be backported What’s next
  21. 28.

    • Use a recent kernel, even on CentOS • Wait

    for the fix to be backported • Reboot regularly your servers What’s next
  22. 29.

    • Use a recent kernel, even on CentOS • Wait

    for the fix to be backported • Reboot regularly your servers • Recompile your kernel without “CONFIG_MEMCG_KMEM” What’s next