FOR ANY STANDARD x86 SERVER PLATFORM 100% SOFTWARE SOLUTION PRIVATE CLOUD PUBLIC CLOUDS …… MULTI-CLOUD DATA CONTROLLER TO ACCESS AND MANAGE DATA ACROSS CLOUDS object & file storage in a single system · peer to peer architecture · unlimited scalability · unbounded scale-out performance · most adaptive set of robust data protection mechanisms · autonomous self-healing · designed in close collaboration with the biggest (cloud-scale) service providers in the world a single, unified API across all clouds to simplify application development · the only multi-cloud data management solution independent of the storage system · stores data in standard cloud format to make the data consumable directly by native cloud apps and services · true multi-cloud IT · global search across all managed data independent of cloud location …… … … …
Benjamin Allot Docker ● Adoption early in 2014 ● Tooling at Scality for our builds, then CI ● S3 API compatible server deployed with docker Tech Leader of “Setup team” Trained as a Developer Unix Sysadmin for a living
● The good: The layers are stored efficiently thanks to a “copy on write” mechanism ● The bad: There is several storage drivers, each with their strengths and weaknesses Docker : Storage Drivers
● The good: The layers are stored efficiently thanks to a “copy on write” mechanism ● The bad: There is several storage drivers, each with their strengths and weaknesses ● The ugly: Bad combination of Docker, Storage Driver and Kernel can lead to issues (mostly upon stop) Docker : Storage Drivers
Btrfs/Zfs AUFS DEVICE- MAPPER OVERLAY OVERLAY2 Require disk formatting Not supported by kernel anymore since 3.18 Warning : bad performance for local-lvm Run out of inode easily Require disabling selinux, require Centos7.4 (kernel 3.10.0-693) Docker storage driver of choice = overlay2 - Best performance/stability with less requirements - With docker < 18.02, detection over kernel capabilities for overlay2 is buggy (require force storage driver for docker 17.03) - Educated bet on future Docker Storage Driver: which and why ?
● Configure it with /etc/docker/daemon.json file ○ “storage-driver”: “overlay2” ○ “storage-opts”: [“overlay2.override_kernel_check=true”] ● Double check the compatibility matrix ● On RedHat/CentOS be wary of: ○ XFS with a specific mkfs option mandatory with OverlayFS for /var/lib/docker ○ Device or resource busy when stopping a container ○ How to Setup OverlayFS on RedHat Docker : Storage Drivers
● Check the Storage Driver to use before deploying on production ● Be wary of Kernel capabilities and Docker “default” Storage Driver decision ● Future might be to use directly containerd 1.1+ (interesting history of graph drivers and why it isn’t supported “as is” in containerd) Docker : Summary
● To deploy our S3 connector (RING product) ● To replicate object into several clouds with Zenko (Open Source) ○ We used to use Docker Swarm ○ Now we do Kubernetes What production ?
Why Kubernetes ? - Run everywhere, on any cloud => provide API abstraction - Control plane run server side (compared to docker compose) - Self-healing - Auto-scaling (of pods, of cluster, of resources requests) - Huge set of plugins (centralised logging, monitoring, ingress) - Big community - Docker announcement to support Kubernetes in 2017 - Customers trust and want it
● An opinionated Kubernetes distribution with a focus on long-term on-prem deployments ● A commitment to bare-metal ● Open Source : check here MetalK8s ● KOPS ● EKS (one day ?) AWS ● GKE GCP ● AKS Azure ● ? Bare Metal
("/sys/fs/cgroup/devices/kubepods/burstable/pod6ee68e26-9bed-11e8-b370-f403435bf038 /a1ac00006e2cd56faf6b14212c6a881371d9e1a683c2852fd47295fda4b00954": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/devices/kubepods/burstable/pod6ee68e26-9bed-11e8-b370-f403435bf038/a 1ac00006e2cd56faf6b14212c6a881371d9e1a683c2852fd47295fda4b00954: no space left on device One tiny little problem ….
● A kernel cgroup bug for kernel prior to 4.0 identified for Moby ● A runc change of behavior with kernel memory accounting ● A Kubernetes bug identifying the issue What we found so far
● A kernel cgroup bug for kernel prior to 4.0 identified for Moby ● A runc change of behavior with kernel memory accounting ● A Kubernetes bug identifying the issue ● More precisely : this commit is responsible of the “bug” What we found so far
● A kernel cgroup bug for kernel prior to 4.0 identified for Moby ● A runc change of behavior with kernel memory accounting ● A Kubernetes bug identifying the issue ● More precisely : this commit is responsible of the “bug” ● A chinese page describing the issue (thank you google translate) What we found so far
● Use a recent kernel, even on CentOS ● Wait for the fix to be backported ● Reboot regularly your servers ● Recompile your kernel without “CONFIG_MEMCG_KMEM” What’s next