Japan Community Day at Kubecon + CloudNativeCon Japan 2026: Learning Container Privilege Control by Building My Own Low-Level Container Runtime

Transcript

1. Learning Container Privilege Control by Building My Own Low-Level Container Runtime Software Engineer, LY Corporation Ayako Hayasaka © LY Corporation
2. Ayako Hayasaka LY Corporation (Japan, Tokyo) SWAT Unit Software Engineer • Web backend engineering • • @ternbusty Kotlin, Java Personal OSS project • MonakaFS: host-independent virtual filesystem for WebAssembly • kontainer-runtime: low-level container runtime written in Kotlin/Native • Likes • © LY Corporation Kotlin, Java, WebAssembly 2 2
3. Presentation description © LY Corporation 3
4. Presentation description What is “Container Privilege Control” ? Why Java? © LY Corporation What is GraalVM Native Image? 4
5. Agenda 1 What a low-level container runtime does 2 Why Java? What is GraalVM Native Image? 3 Namespace isolation 4 Linux privilege basics 5 Dropping privileges in the right order © LY Corporation 5
6. What a low-level container runtime does © LY Corporation 6
7. What is a container? An independent execution environment, isolated from the host • It runs as a process • The kernel is shared with the host © LY Corporation process process OS hardware 7
8. How does it differ from a plain process? Isolated resources Own root filesystem Assorted restrictions Other containers and different from the so it cannot misbehave the host are invisible host’s from inside cgroup namespaces capabilities (pid, user, network…etc) © LY Corporation pivot_root AppArmor seccomp SELinux 8
9. What is a low-level container runtime? user / kubelet ↓ requests High-level runtime (CRI runtime) manages containers · images · networks, e.g. containerd ↓ create / start / kill / delete / state Low-level runtime (OCI runtime) builds the isolated environment, e.g. runc © LY Corporation The high-level runtime issues commands by executing the low-level binary. 9
10. What instructions does a low-level runtime receive? • Let's look at how a container is created, in detail $ sudo runc create --bundle test-bundle test-container the low-level runtime's binary the operation to run specifies the directory the name of the container to create • A bundle contains: • the container's root filesystem • a config.json file with the container's configuration © LY Corporation 10
11. create & start runc containerd create command success isolating namespaces, applying pivot_root… and create container (created) start command success let the container process execute the command which is specified to run at the container startup execve container (running) © LY Corporation 11
12. Java & GraalVM Native Image © LY Corporation 12
13. How plain Java runs JVM (HotSpot VM) compile Java Interpreter .java .class (bytecode) JIT compiler call syscall Machine code OS call syscall Runtime compilation for hot paths © LY Corporation 13
14. How plain Java runs JVM (HotSpot VM) compile Java Interpreter .java .class (bytecode) JIT compiler Runtime compilation for hot paths © LY Corporation 🙅 takes more than 100ms to start up call syscall Machine code OS call syscall 🙅 runtime compilation doesn’t take effect because a container runtime exits immediately after each operation 14
15. GraalVM Native Image © LY Corporation https://www.graalvm.org/latest/reference-manual/native-image/ 15
16. GraalVM Native Image compile .java native-image (build tool) .class (bytecode) Substrate VM Machine code embedded runtime services (GC, thread management, etc.) AOT-compiled application code elf file (native binary) • AOT compiled, no JVM, no bytecode at runtime • Fast start, single binary, ideal for CLI tools • Runtime spins up extra threads (reference header, cleaner) → runs multithreaded (like Go). • How is C interoperability? © LY Corporation 16
17. C interoperability (getpid) public static void run() throws Throwable { Linker linker = Linker.nativeLinker(); MethodHandle getpid = linker.downcallHandle( linker.defaultLookup().find("getpid").orElseThrow(), FunctionDescriptor.of(JAVA_INT)); } System.out.println("pid = " + (int) getpid.invoke()); A callable Java-side entry point built from the function's address and signature. Tells Native Image at build public static final class Registration implements Feature { time which signatures need @Override AOT-generated call stubs public void duringSetup(DuringSetupAccess access) { RuntimeForeignAccess.registerForDowncall(FunctionDescriptor.of(JAVA_INT)); } } © LY Corporation 17
18. C interoperability (calling syscalls directly) private static final long SYS_GETPID = 172; public static void run() throws Throwable { Linker linker = Linker.nativeLinker(); MethodHandle syscall = linker.downcallHandle( linker.defaultLookup().find("syscall").orElseThrow(), FunctionDescriptor.of(JAVA_LONG, JAVA_LONG)); } Change getpid -> syscall System.out.println("pid = " + (long) syscall.invoke(SYS_GETPID)); Call by specifying syscall num public static final class Registration implements Feature { @Override public void duringSetup(DuringSetupAccess access) { RuntimeForeignAccess.registerForDowncall(FunctionDescriptor.of(JAVA_LONG, JAVA_LONG)); } } © LY Corporation 18
19. C interoperability (writing a file) public static void run() throws Throwable { // method handle registration (omit) String path = “/tmp/arena_example.txt”; byte[] content = “Hello from Java FFM with Arena!¥n".getBytes(StandardCharsets.UTF_8); try (Arena arena = Arena.ofConfined()) { MemorySegment file = (MemorySegment) fopen.invoke( arena.allocateFrom(path), arena.allocateFrom("w")); if (file.equals(MemorySegment.NULL)) throw new RuntimeException("fopen failed: " + path); long written = (long) fwrite.invoke( arena.allocateFrom(JAVA_BYTE, content), 1L, (long) content.length, file); fclose.invoke(file); System.out.println("Wrote " + written + " bytes to " + path); } } // Register for downcall is needed (omit) © LY Corporation 19
20. Meet takoyaki • takoyaki, an OCI container runtime in Java + GraalVM Native Image • https://github.com/ternbusty/takoyaki • Partial implementation of the OCI Runtime Spec • Major caveats • cgroup v2 only • No systemd cgroup driver © LY Corporation 20
21. Demo: create, start, kill, delete, state © LY Corporation 21
22. Demo with containerd © LY Corporation 22
23. Namespace Isolation © LY Corporation 23
24. How do you isolate namespaces? unshare --user --mount --pid --ipc –uts ... Call unshare with the flag you want. Done! © LY Corporation 24
25. How do you isolate namespaces? unshare --user --mount --pid --ipc –uts ... Call unshare with the flag you want. Done! Things were not that simple! © LY Corporation 25
26. The multithreading problem (1) You can’t unshare(2) in an ordinary way! • Premise: the native-image runtime starts multithreaded. • Call unshare(2) for namespace isolation and…? • For example, when you try to isolate user namespace, you get an error © LY Corporation https://man7.org/linux/man-pages/man2/unshare.2.html 26
27. The multithreading problem (1) You can’t unshare(2) in an ordinary way! • Premise: the native-image runtime starts multithreaded. • Call unshare(2) for namespace isolation and…? • Even without an error, only the calling thread enters the new namespace. The other threads stay behind in the old one. • Figure: each thread's namespace after unsharing the network namespace © LY Corporation Only this thread has moved to a different namespace https://github.com/ternbusty/graalvm-native-image-playground 27
28. The multithreading problem (2) Fork can cause problems • Premise: with a PID namespace, simply calling unshare does not cause the namespace switch. The process forked afterwards is the one that enters the new namespace, making a fork mandatory. • fork(2) copies only the calling thread and every other runtime thread is gone for the child thread. • Reference handler and cleaner threads, which free cleaner-managed native resources like Arena.ofAuto(), are lost for the child thread and never work again. → That memory is never freed again, causing leaks. © LY Corporation https://github.com/ternbusty/graalvm-native-image-playground 28
29. The multithreading problem (2) A tricky solution • Cleanly give up on doing the namespace / fork work in Java! • Arrange for a C program to run before the Java runtime starts • C function marked __attribute__((constructor)) is placed in the ELFʼs .init_array section and runs before the program's main entry point. • This means it can run while the process is still single-threaded. • The first launch starts the Java runtime normally, then re-execs its own binary so this constructor path fires. © LY Corporation 29
30. runc has the same problem • runc (written in Go) hits the same wall • Go also runs multithreaded by default • It dodges the problem by implementing one part in C © LY Corporation https://github.com/opencontainers/runc/blob/f73814296c013f82f72252d6e4e2d917090028f2/libcontainer/nzsenter/nsexec.c 30
31. How to create a container (Incomplete Version) Main Process Stage 1 Stage 2 Parse results are written to env vars Parse Configs clone and execute itself again Separate namespaces Host-side tasks Stage1 is written in C after separating pid namespaces, clone again pivot-root Other container-side tasks Wait for start signals © LY Corporation execve to be a running container… 31
32. The multithreading problem (3) • Substrate VM initialization fails by an error at the beginning of Stage 2. → re-execve is required. Substrate VM source code This causes an error because tid is not updated from Stage1 after clone(2) and this tid is no longer recognized after pid namespace isolation glibc source code • cgo calls the same function, but it avoids this failure by simply skipping the check of the return value. cgo source code © LY Corporation 32
33. How to create a container Main Process Stage 1 Stage 2 Parse results are written to env vars Parse Configs clone and execute itself again Separate namespaces Host-side tasks Stage1 is written in C after separating pid namespaces, clone again execve itself again pivot-root Other container-side tasks Wait for start signals © LY Corporation execve to be a running container… 33
34. Linux privilege basics (After namespace isolation) © LY Corporation 34
35. What if we don’t drop privilege after pivot_root? A container can change the host clock at will A container can read the host's /proc/kcore (raw kernel memory) A container can attach an eBPF program to sys_enter_write and capture every process's writes, including host © LY Corporation 35
36. Let’s talk about “privilege”! • Experiment: I wrote a small binary (port-demo) that listens on the port passed as an argument and waits for connections. • Results: Binding to port 8080 works, but binding to port 80 fails with "permission denied”. We seem to be missing some kind of "privilege". So what exactly is the "privilege" we are talking about here? © LY Corporation https://github.com/ternbusty/port-demo 36
37. Capability Root’s power is split into 41 capabilities in Linux Network CAP_NET_ADMIN Files CAP_CHOWN CAP_MKNOD Process & IDs CAP_SETUID CAP_SETGID System CAP_SYS_BOOT CAP_SYS_TIME Others CAP_SYS_ADMIN … © LY Corporation CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_DAC_OVERRIDE CAP_KILL … … … … https://man7.org/linux/man-pages/man7/capabilities.7.html 37
38. Capability Root’s power is split into 41 capabilities in Linux Network CAP_NET_ADMIN Files CAP_CHOWN CAP_MKNOD Process & IDs CAP_SETUID CAP_SETGID System CAP_SYS_BOOT CAP_SYS_TIME Others CAP_SYS_ADMIN … © LY Corporation CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_DAC_OVERRIDE CAP_KILL … … … … https://man7.org/linux/man-pages/man7/capabilities.7.html 38
39. Capability check when binding to port 80 • A process*1 has its own set of capabilities in its task_struct. • Kernel checks whether the process has the required capability when the process tries to do something. Example: In function __inet_bind, kernel checks whether the process has CAP_NET_BIND_SERVICE. © LY Corporation https://github.com/torvalds/linux/blob/v6.12/net/ipv4/af_inet.c *1: Strictly speaking, it’s possessed by a thread. 39
40. Capability check when binding to port 80 • A process (thread) has its own set of capabilities in its task_struct • Kernel checks whether the process has the required capability when the process tries to do something What are those 5 sets? Example: in function __inet_bind, kernel checks whether the process has CAP_NET_BIND_SERVICE © LY Corporation https://github.com/torvalds/linux/blob/v6.12/include/linux/cred.h 40
41. Capability sets per process: Permitted & effective Permitted can be raised into effective Effective active right now © LY Corporation Raised into effective for the operation, dropped back to permitted when done Kernel checks whether CAP_NET_BIND_SERVICE is in Effective 41
42. Getting CAP_NET_BIND_SERVICE into effective(1): Run as root The process has a huge number of capabilities in its effective set → 🙆 Success! (*) I extended the binary to print • the uid (euid) the kernel sees • the capabilities the process holds We only want to bind a port, so isn't this handing it far too much privilege? © LY Corporation Couldn't we pick just the CAP_NET_BIND_SERVICE? 42
43. Capability sets per process: Inheritable & ambient set You can hand capabilities down from a process using ambient and inheritable set execve Process Process (after execve) Effective: Effective: Permitted: Permitted: Ambient: Inheritable: • Ambient: carried into effective after execve • Inheritable: carried into permmited after execve *1 © LY Corporation *1: Not always. Restricted by executable file capabilities’ inheritable set. 43
44. Getting CAP_NET_BIND_SERVICE into effective (2): Using ambient set The process has CAP_NET_BIND_SERVICE in its effective set handed down from the ambient set → 🙆 Success! © LY Corporation (*) In the command, keep=1 is equivalent to enabling PR_SET_KEEPCAPS, and user=ubuntu is equivalent to setresuid(2) in later slides. This is very similar to what happens when creating a container. 44
45. Getting CAP_NET_BIND_SERVICE into effective (3): Using setuid bit • You can also take the approach of touching the executable file itself. • If you set the setuid bit on an executable, then: • The kernel runs the process with the effective UID of the file's owner. • When that owner is root, the effective UID becomes 0, and the kernel then grants the full capability set on execve, just like running as root. • This can be regarded as a kind of privilege escalation. © LY Corporation 🙆 Success! 45
46. Getting CAP_NET_BIND_SERVICE into effective (3): using setuid bit (supplement) • The setuid bit may not be familiar, but… • For example, when a user runs passwd • The user themself has uid 1000 • /usr/bin/passwd is owned by root and has the setuid bit set, which turns the effective UID into 0 • The permission check on /etc/shadow then passes as euid 0, allowing it to operate on a file that is owned by root • sudo also has this bit set, which lets it run as uid 0 © LY Corporation 46
47. Executable files also possess capability sets. execve Process (after execve) Process Inheritable: Executable file Permitted: Inheritable: Permitted: When executing a file, its file capability permitted set is added to the process’s new permitted set after execve. *1 Can be regarded as privilege escalation © LY Corporation *1: Not always. Discuss later. 47
48. Capabilities of executable files: Examples • ping actually needs CAP_NET_RAW, yet ordinary users run it normally. Why? • Because the ping binary has the capability in its file permitted set and gives the capability to the process! It raises CAP_NET_RAW into effective only when needed, and keeps it parked in permitted otherwise. (experiment with ubuntu 24.04) © LY Corporation 48
49. Getting CAP_NET_BIND_SERVICE into effective (4): Using file capabilities Since the executable file has CAP_NET_BIND_SERVICE in its capability, the process has it in its effective set. → 🙆 Success! © LY Corporation 49
50. When exec-ing a file with strong capabilities… execve Process Inheritable: Process (after execve) Executable file Inheritable: Permitted: Permitted: ↑ a very strong capability like CAP_SYS_ADMIN © LY Corporation I don’t want this process to have too strong capabilities… (especially when it is a container process) results in a very powerful 💪 process 50
51. Bounding set can save you Only capabilities included in the bounding set is added to the process’s new permitted set. execve Process (after execve) Process Inheritable: Bounding: Executable file Inheritable: Permitted: Permitted: doesn’t have in permitted Now, I can set an upper limit of the capabilities that the process can possess! © LY Corporation 51
52. Capability sets per process: Summary Bounding: upper limit Permitted can be raised into effective Effective active right now © LY Corporation Ambient carried into effective after execve Inheritable carried into permitted after execve 52
53. AppArmor • Mandatory Access Control using Linux Security Module • Can control path based access control • Can also express non-file rules (capabilities, network, signals…) • Can be set to executable files to restrict their behavior, but can also be used to confine processes’ behavior • Load a profile, then write the profile name to proc/self/attr/apparmor/exec • The profile applies at the next execve. • Switching to a less restrictive profile can be a privilege escalation. © LY Corporation https://github.com/moby/profiles/blob/main/apparmor/template.go 53
54. Seccomp • Can define actions when specific syscalls are called from the process • Actions: ALLOW, ERRNO, KILL_THREAD / KILL_PROCESS, NOTIFY, LOG, TRAP default action is ERRNO … Safe syscalls are allowed. © LY Corporation Conditional allow: some syscalls are allowed only if the process has an enough capability when the filter is loaded. https://github.com/moby/profiles/blob/main/seccomp/default.json 54
55. Dropping privileges in the right order © LY Corporation 55
56. Privilege management for a container Right after namespace isolation and pivot_root, the process has too strong privileges. UID: 0 Capabilities Syscalls Filesystem Bounding: Can call anything Can access host’s /proc Permitted: Effective: drop privileges based on config UID: 1000 Capabilities Bounding: Inheritable: Ambient: © LY Corporation Syscalls Filesystem Can call safer syscalls, deny others Cannot access host’s /proc No New Privilege enabled We must drop privileges while preventing privilege escalation
57. Dropping privileges in the right order: Simple case Drop Bounding capabilities • • • UID 0 -> UID 1000 calls prctl(2) requires CAP_SETPCAPS UID 0 -> UID 1000 → • calls setresuid(2) • requires CAP_SETUID • side effect: drops all capabilities in permitted, effective, ambient sets Drop Bounding capabilities order 🙅 Already lost capability CAP_SETPCAPS, fail • Drop Bounding capabilities → UID 0 -> UID 1000 order 🙆 succeed! If we don’t drop privileges in the right order, we lose the privilege to drop privileges! © LY Corporation 57
58. More complex case (1): Order of capabilities setting and setresuid Set ambient capabilities • • • UID 0 -> UID 1000 calls prctl(2) requires the capability to be in the permitted and inheritable sets UID 0 -> UID 1000 → • calls setresuid(2) • requires CAP_SETUID • side effect: drops all capabilities in permitted, ambient, effective sets Set ambient capabilities order 🙅 Already lost all capabilities in permitted, fail • Set ambient capabilities → UID 0 -> UID 1000 order 🙅 No error, but ambient set is cleared, wrong © LY Corporation Neither order works. Reordering alone cannot fix this. 58
59. PR_SET_KEEPCAPS saves us How can we realize Ambient: Failure pattern (recap) UID 0 -> UID 1000 Permitted: ? → Set ambient capabilities 🙅 fail Permitted: empty Ambient: empty Success pattern with PR_SET_KEEPCAPS Permitted: Set ambient capabilities PR_SET_KEEPCAPS → UID 0 -> UID 1000 Permitted: Permitted: Permitted: Ambient: empty Ambient: → 🙆 succeed! © LY Corporation Setting PR_SET_KEEPCAPS preserves permitted across the uid change, enabling ambient setting after setresuid 59
60. More complex case (2): Order of seccomp and AppArmor Load seccomp filter • • calls seccomp(2) requires CAP_SYS_ADMIN or no_new_privs flag is set Why seccomp filter loads requires CAP_SYS_ADMIN? © LY Corporation Register AppArmor config • • writes the profile name to proc/self/attr/apparmor/exec requires no_new_privs flag is NOT set What is no_new_privs flag? These questions are related to a famous attack pattern exploiting seccomp filter. 60
61. Attack using seccomp filter 🙆 Safe pattern without seccomp A binary with setuid bit raised uid 1000 start running the binary do_privileged_operation(); setresuid(1000, …); uid 0 Do some operations which need privileges execvp(argv[1], &argv[1]); uid 1000 © LY Corporation Then, run dangerous operations without privileges 61
62. Attack using seccomp filter 🙅 Attack exploiting seccomp filter Before executing the binary, load seccomp filter that forces to return ERRNO 0 when setresuid is called (setresuid is not actually executed, but looks like it succeeded) A binary with setuid bit raised uid 1000 start running the binary do_privileged_operation(); setresuid(1000, …); execvp(argv[1], &argv[1]); setresuid is silently skipped uid 0 © LY Corporation Do some operations which need privilege uid 0 Dangerous operations are done with uid 0 seccomp filter could prevent a privileged program from dropping its privileges, thereby turning seccomp itself into a privilege-escalation vector. 62
63. Possible defense against seccomp filter exploit • One possible defense: require CAP_SYS_ADMIN to load seccomp filters. However, this would prevent unprivileged users from using seccomp to restrict their own processes. • Since the earlier attack started with privilege escalation, the problem can be solved by forbidding privilege escalations before filter load. • The no_new_privs flag provides this guarantee. • Once set, it prevents privilege escalations through setuid bits/setgid bits, file capabilities, or less restrictive AppArmor profiles (what we covered today!). → Solution: a process with no_new_privs flag enabled can also load seccomp filters without CAP_SYS_ADMIN. © LY Corporation 63
64. The required order (no_new_privs enabled path) Register AppArmor config • • write the profile name to proc/self/attr/apparmor/exec requiring no_new_privs flag is NOT set when it can be regarded as privilege escalation Enable no_new_privs flag Load seccomp filter • • © LY Corporation calls seccomp(2) requires CAP_SYS_ADMIN or no_new_privs flag is set 64
65. Overview of the init process after pivot_root when the config specifies non-root uid and sets no_new_privs flag as enabled pivot_root setRootReadonly mask/readOnly requires CAP_SYS_ADMIN requires CAP_SYS_ADMIN oom_score_adj requires CAP_SYS_RESOURCES capability drop (bounding) requires CAP_SETPCAP sethostname requires CAP_SYS_ADMIN PR_SET_ KEEPCAPS loopback up requires CAP_NET_ADMIN AppArmor staging setresuid PR_SET_NO_PRIVS capability set (ambient) PR_SET_DUMPABLE=0 seccomp_load execve © LY Corporation 65
66. Summary • I built a container runtime with Java and GraalVM Native Image. • For namespace isolation, Java's multithreaded nature gave me real trouble, and the FFI to C was underwhelming as well. It was not unusable, but it was pretty rough. • In the second half I organized the privilege management. • Container privilege control is many separate mechanisms tangled together. At first I did not understand the dependencies. I read runc's implementation and coded the parts I had heard of first, so I got stuck quite a bit. • Following a container runtime's implementation teaches you the historical background of privilege control, and that is genuinely fun. Come build a container runtime! © LY Corporation 66
67. Thank You © LY Corporation Emoji graphics: Blob Emoji (Based on Noto Emoji by Google) Licensed under the Apache License 2.0