More Terrible Ideas for Containers? The Ideal and the Real Linux Container

Today’s best practices for building and deploying Linux containers do not always align with the messy realities of containers used in production. In the wild, the vast majority of containers include entire OS filesystems, applications with complex and often poorly understood dependencies, and a plethora of libraries, with all of their attendant security vulnerability and maintenance concerns.

Container orchestration systems like Kubernetes encourage us to treat containers as immutable black boxes. From these building blocks, developers can assemble, integrate, and scale distributed applications with relative ease. Containers as black boxes, is that a brilliant idea or, considering what might be lurking inside, a terrible idea? The answer is both, and after this talk, you will understand why.

Transcript

1. More Terrible Ideas for Containers? The Ideal and the Real Linux Container Andy Henroid <andy.henroid@puppet.com>
2. Who am I?
3. This Talk Linux Containers Big Promises Broken Dreams? Band-Aids & Better Solutions
4. Linux & Open Source Software: Astounding range & variety of SW Dependencies accumulate quickly Yes “DLL Hell” is a Linux problem See also: RPM Hell, Java JAR Hell, Ruby Gem Hell, … And many disparate solutions to these 4 Containers: Escape from Dependency Hell
5. Orchestrators like Kubernetes add: ü Dynamic Scalability ü Resource Efficiency ü High Availability ü Uniform Service Discovery ü Portability And more... Treating each container/pod as a Black Box, a homogenous unit to schedule in the cluster 5 Containers + Orchestrator = Massive Scalability
6. Lightweight Single-purpose Immutable Great starting point for a micro-service 6 The Ideal Linux Container 4.2MB 1 11 Image Size Image Layers Packages Alpine 3.5.2 : A minimal Docker image based on Alpine Linux with a complete package index
7. Heavyweight General-purpose Mutable Flexible, reusable But fast approaching VM size 7 The Real Linux Container Ideal to… far from ideal 4M 11 0.4K 129M 104 8K 190M 140 11K 240M 171 13K 716M 191 25K Image Size Packages Files Alpine Debian Nginx (Debian) Fedora GoLang (Debian)
8. Container tooling makes it trivial to “ADD” But there is no “REMOVE” Images grow quickly And never smaller 8 Image Size & Layers 231M 171 436M 172 Image Size Packages Fedora Fedora (after "yum update")
9. 9 Riffing on a Cool Idea ~330K Dockerfiles on GitHub
10. Open-source https://github.com/anchore/anchore Image size & layers Image family tree Linux distro Packages & files CVEs And more… 10 Variations on a Theme: Opening the Black Box Ideal to.…........... alarming 449 0 8267 707 11191 1435 12792 720 25497 10720 Files Unmanaged Files Alpine Debian Nginx (Debian) Fedora GoLang (Debian)
11. We analyzed :latest image from >5000 top repos 136 Official + many more Wide variety: Base OS images, services, middleware, languages, … 600M+ to 10K pulls 11 Docker Hub Survey † Full data set at https://github.com/ahenroid/docker-hub-survey Fun Fact: Top 2% public Hub repos account for >99% of all Docker pulls 23.3% 2.0% 1.5% 3.1% 1.6% 31.6% 29.5% 6.6% 0.8% alpine buildroot busybox Misc. Other debian ubuntu centos fedora Lightweight = 31.5% Images Heavyweight = 68.5% Images Base Container Image
12. Median stats: Expansive gulf between lightweights & heavyweights 12 Docker Hub Survey 64M 20 577M 245 Image Size Packages Lightweight Heavyweight 4K 0 23K 5K Files Unmanaged Files Lightweight Heavyweight † Full data set at https://github.com/ahenroid/docker-hub-survey
13. Massive Scalability? Maybe. Escape from Dependency Hell? Deferred, but not actually solved. Ahead for operators: Inventory and maintenance nightmare Always Vulnerable Always Updating Always Rebuilding Even the best containers fall short 13 Container Big Promises
14. Labels, embedding Dockerfiles, and similar band-aids Pro: Bound to container image Cons: Not automatic Not standardized No rich data format Quickly out-of-date 14 Advice (Part I): Know What’s Inside
15. Container Scanning Static image scanning Everyone is doing it… Open source, hosted services, integrated with container registries Live container scanning Fewer options today 15 Advice (Part I): Know What’s Inside
16. Software development principles encouraging Fast Delivery Open Source: Every library you will ever need is here, ready to integrate Decomposing traditional monolithic apps is hard, fraught work 16 Quick Interlude: How Did We Get Here?
17. Start small and build carefully, thoughtfully FROM scratch, or more realistically, Alpine Separate & design containers by purpose Front-end web service, Jenkins build slave, ... No “Golden Container Image” Know your dependencies & question over time Be willing to redesign/refactor as your containers evolve Automated tools to identify dependency creep 17 Advice (Part II): Good Engineering Practices Engineering?!
18. Other Talks Continuous Delivery to Azure with Docker Using Redis with other DBs Creating custom Postgresql packages in Alpine Linux for use with Docker/Containers/VMs Libral: towards a system management API for Linux Spice up your dev environment with containers Container Images @ FB with Btrfs
19. Questions?
20. The shortest path to better software.