© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Samuel Karp, Amazon Container Services – @samuelkarp April 27, 2019 – LinuxFest Nortwest Linux Container Primitives cgroups, namespaces, and more!

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Agenda Container primitives overview Control groups (cgroups) Namespaces Union filesystems Runtimes

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Linux container primitives

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Containers are an abstraction over several different Linux technologies

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Linux Kernel Container runtime Container 1 Container 2 Container 3 Container 4 Container 5 Container 6 Namespaces Control groups Union filesystem

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Control groups

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What do control groups (cgroups) do? • Organize all processes in the system • Limit or prioritize resource utilization • Account for resource usage and gather utilization data

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Subsystems • Control groups is an abstract framework • Subsystems are concrete implementations • Different subsystems can organize processes separately • Most subsystems are resource controllers Examples of subsystems: • Memory • CPU time • Block I/O • Number of discrete processes (pids) • CPU & memory pinning • Freezer (used by docker pause) • Devices • Network priority

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Hierarchical representation • Independent subsystem hierarchies • Every pid is represented exactly once in each subsystem • New processes inherit cgroups from their parents ├── blkio │ └── docker │ └── b211c37 ├── cpu,cpuacct │ └── docker │ └── b211c37 ├── cpuset │ └── docker │ └── b211c37 ├── devices │ └── docker │ └── b211c37 ├── freezer │ └── docker │ └── b211c37 ├── hugetlb │ └── docker │ └── b211c37 ├── memory │ └── docker │ └── b211c37

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. cgroup virtual filesystem • Typically mounted at /sys/fs/cgroup • tasks virtual file holds all pids in the cgroup • Other files have settings and utilization data ├── cgroup.clone_children ├── cgroup.procs ├── cgroup.sane_behavior ├── cpuacct.stat ├── cpuacct.usage ├── cpuacct.usage_all ├── cpuacct.usage_percpu ├── cpuacct.usage_percpu_sys ├── cpuacct.usage_percpu_user ├── cpuacct.usage_sys ├── cpuacct.usage_user ├── cpu.cfs_period_us ├── cpu.cfs_quota_us ├── cpu.rt_period_us ├── cpu.rt_runtime_us ├── cpu.shares ├── cpu.stat ├── notify_on_release ├── release_agent └── tasks

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Demo

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What can you use cgroups for? • cgroups can be used independently of containers • cgroups control resource limits for processes • Monitor processes and organize them • Be careful not to break any assumptions your container runtime or orchestrator might have

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Further reading • Linux: Documentation/cgroup-v1

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Namespaces

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What do namespaces do? • Isolation mechanism for resources • Changes to resources within namespace are (generally) invisible outside the namespace

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What namespaces are available? • Network • Filesystem (mounts) • Processes (pid) • Inter-process communication (ipc) • Hostname and domain name (uts) • User and group IDs • cgroup

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Namespace sharing Process A Process B Process C Process D pid:[2] pid:[1] pid:[3] net:[4] net:[5] net:[6] mount:[7] mount:[8]

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Network namespace • Frequently used in containers • docker run uses a separate network namespace per container • Multiple containers can share a network namespace • Kubernetes pods • Amazon ECS tasks with the awsvpc networking mode

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Mount namespace • Used for giving containers their own filesystem • Container image is mounted as the root filesystem • More about filesystems to come! bash-4.2# mount overlay on / type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2 /l/Q5EBZ7CIJYELLG2MBKZIRRFWW6:/var/lib/docker/ overlay2/l/PKATP76T57BQZ5D44JXYFIB26E,upperdir =/var/lib/docker/overlay2/88816f9510a9ff38b31e aaceccbef6ffc9cc3c06bcc451f9684850db5ee1b152/d iff,workdir=/var/lib/docker/overlay2/88816f951 0a9ff38b31eaaceccbef6ffc9cc3c06bcc451f9684850d b5ee1b152/work) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev type tmpfs (rw,nosuid,size=65536k,mode=755) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmx mode=666) sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,relatime,mode=755)

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. procfs virtual filesystem • Namespaces are visible in /proc • Files are symbolic links to the namespace • The link contains the namespace type and inode number to identify the namespace $ readlink /proc/$$/ns/* cgroup:[4026531835] ipc:[4026531839] mnt:[4026531840] net:[4026531993] pid:[4026531836] user:[4026531837] uts:[4026531838]

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Creating namespaces • clone(2) and unshare(2) • CLONE_NEW* flags to specify which namespaces • clone(2) is for new processes to create new namespaces • unshare(2) is for existing processes to create new namespaces

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Persisting namespaces • The kernel automatically garbage-collects namespaces by reference-counting • New namespace remains open as long as • a process runs or • a mount is open • Bind-mount a file in /proc/$$/ns to another place on the filesystem $ mount \ --bind /proc/$$/ns/net \ /var/run/netns/lfnw

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Entering namespaces • Open a file from /proc/$$/ns (or a bind- mount) • Pass to setns(2) to enter the existing namespace • Namespace remains open as long as the process is running, even if the original file goes away • nsenter(1) is a command for doing this interactively • ip-netns(8) works specifically for network namespaces

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Demo

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. How can you leverage this? • Use nsenter or ip netns to troubleshoot container networking • Monitor containers by entering the pid namespace • Access binaries in your containers with the mount namespace

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Further reading • man 7 namespaces • man 7 pid_namespaces • man 7 user_namespaces

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Images, layers, and union filesystems

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Filesystem images • Images are representations of a filesystem • Images are popular for virtualization and container systems • Docker helped popularize the concept of layers • A union filesystem is one where two or more filesystems are joined together in a unified view

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Top layer (read-write) Intermediate layer (read-only) Base layer (read-only) How Docker layers work • A copy-on-write view of your files • New files exist only in the top layer • When a file is modified, it is copied up to the top layer • Unmodified files exist in whatever layer they were added/modified • Deleted files are hidden, but still exist

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Union filesystems • Popular in container runtimes (like Docker) to implement layers • Efficient use of storage when starting multiple containers with identical images • Efficient use of storage when making minor modifications to images

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Overlay filesystem • Joins two directories (upper and lower) to form a union • Uses file name to describe the files • When writing to the overlay • lowerdir is not modified, all changes go to upperdir • Existing files are copied-up to the upperdir for modificiation • Whole file is copied, not just blocks • “Deleting” a file in the upperdir creates a whiteout • Files: character devices with 0/0 device number • Directories: xattr “trusted.overlay.opaque” set to “y”

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Overlay filesystem (continued) • An upperdir can have multiple lowerdirs • Overlay filesystems can be created with mount(2) • You can examine the mounts with • mount(8) • /proc/mounts • /proc/$$/mountinfo

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Docker’s overlay driver • Docker’s default layer storage uses the overlay filesystem • upperdir, lowerdir, and diff directories are in /var/lib/docker/overlay2

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Demo

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. How can you leverage this? • Locate files in your layers • Examine which files and layers contribute to your disk usage • Understand the impact of writable files in your containers and how to reduce # du -h . | sort -hr 753M . 211M ./e33f37/diff 211M ./e33f37 204M ./e33f37/diff/usr 169M ./f87973/diff … # ls ./f87973 diff link # ls ./e33f37 diff link lower work Base layer! Intermediate layer!

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Further reading • Linux: Documentation/filesystems/overlay.txt

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Runtimes

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What is a container runtime? • A software tool that configures Linux primitives to create and run containers on a host • Examples include: • Docker • containerd • CRI-O • rkt • systemd-nspawn • Open Containers Initiative (OCI) aims to standardize container runtimes, image format, and distribution • The OCI reference implementation (runc) powers Docker, containerd, and CRI O ‑

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. OCI runtime spec • Containers are “bundles” • Filesystem • JSON document • Filesystem can be a union • JSON document describes • cgroups • Namespaces • Additional mounts • Linux capabilities • Linux security modules • And more • Hooks can modify the bundle { "ociVersion": "1.0.1", ⋮ "root": { "path": "/var/lib/docker/overlay2/03004c/merged" }, ⋮ "hooks": { "prestart": [{"path": "/proc/9306/exe"}] }, "linux": { "resources": { "cpu": {"shares": 0}, "pids": {“limit": 0}, ⋮ }, "cgroupsPath": "/docker/bd5cebc8950c", "namespaces": [ {"type": "mount"}, {"type": "network"}, ⋮ ], ⋮ }

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. OCI runtime hooks • Hooks run • Before a container starts • After a container starts • After a container stops • Hooks can modify the filesystem, modify the JSON file, or take other actions • Hooks run sequentially, in an order defined in the JSON file • Docker generates a bundle without hooks • Docker does let you specify your own runtime • Your runtime could inject hooks, then execute the real runtime • This is how Nvidia’s container runtime works

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Demo

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Questions?

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. A brief note before we finish — Session surveys provide valuable information to speakers Feedback that is very helpful: • Topics you were excited to learn about • Suggestions for improving understanding and clarity Feedback that is extremely unhelpful: • Comments unrelated to talk content (please refer to the LinuxFest Northwest Code of Conduct) The “hallway track” is always open! Feedback and questions welcome (skarp@amazon.com, @samuelkarp) For support, use the AWS Forums or contact AWS Support

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank you! Samuel Karp @samuelkarp