bypass4netns: Accelerating TCP/IP Communications in Rootless Containers AINTEC 2024 August 9, 2024 Naoki Matsumoto (Kyoto University, Japan) Akihiro Suda (NTT Software Innovation Center, Japan)

Background: Rootless Containers Container runtimes can have vulnerabilities and it sometimes breaks container isolation. In 2024, CVE-2024-21626(fd leaking for /sys/fs/cgroup in runc) was reported. Rootless containers: Running container runtimes without root. → They mitigates attacks against container runtimes and a host. 2 Process Container Runtime Process Container Runtime Normal user Root Rootful container Rootless container

Background: Rootless Containers Container runtimes can have vulnerabilities and it sometimes breaks container isolation. In 2024, CVE-2024-21626(fd leaking for /sys/fs/cgroup in runc) was reported. Rootless containers: Running container runtimes without root. → They mitigates attacks against container runtimes and a host. 2 Process Container Runtime Isolation broken! Process Container Runtime Normal user Root Rootful container Rootless container

Background: Rootless Containers Container runtimes can have vulnerabilities and it sometimes breaks container isolation. In 2024, CVE-2024-21626(fd leaking for /sys/fs/cgroup in runc) was reported. Rootless containers: Running container runtimes without root. → They mitigates attacks against container runtimes and a host. 2 Process Container Runtime Isolation broken! Attacker Process Process Container Runtime Normal user Root Rootful container Rootless container

Background: Rootless Containers Container runtimes can have vulnerabilities and it sometimes breaks container isolation. In 2024, CVE-2024-21626(fd leaking for /sys/fs/cgroup in runc) was reported. Rootless containers: Running container runtimes without root. → They mitigates attacks against container runtimes and a host. 2 Process Container Runtime Malicious operations with root! Isolation broken! Attacker Process Process Container Runtime Normal user Root Rootful container Rootless container

Background: Rootless Containers Container runtimes can have vulnerabilities and it sometimes breaks container isolation. In 2024, CVE-2024-21626(fd leaking for /sys/fs/cgroup in runc) was reported. Rootless containers: Running container runtimes without root. → They mitigates attacks against container runtimes and a host. 2 Process Container Runtime Malicious operations with root! Isolation broken! Attacker Process Process Container Runtime Isolation broken! Normal user Root Rootful container Rootless container

Background: Rootless Containers Container runtimes can have vulnerabilities and it sometimes breaks container isolation. In 2024, CVE-2024-21626(fd leaking for /sys/fs/cgroup in runc) was reported. Rootless containers: Running container runtimes without root. → They mitigates attacks against container runtimes and a host. 2 Process Container Runtime Malicious operations with root! Isolation broken! Attacker Process Process Container Runtime Isolation broken! Attacker Process Normal user Root Rootful container Rootless container

Background: Rootless Containers Container runtimes can have vulnerabilities and it sometimes breaks container isolation. In 2024, CVE-2024-21626(fd leaking for /sys/fs/cgroup in runc) was reported. Rootless containers: Running container runtimes without root. → They mitigates attacks against container runtimes and a host. 2 Process Container Runtime Malicious operations with root! Isolation broken! Attacker Process Process Container Runtime Malicious operations with normal user! Isolation broken! Attacker Process Normal user Root Rootful container Rootless container

Background: Rootless Containers Container runtimes can have vulnerabilities and it sometimes breaks container isolation. In 2024, CVE-2024-21626(fd leaking for /sys/fs/cgroup in runc) was reported. Rootless containers: Running container runtimes without root. → They mitigates attacks against container runtimes and a host. 2 Process Container Runtime Malicious operations with root! Isolation broken! Attacker Process Process Container Runtime Malicious operations with normal user! Isolation broken! Attacker Process Normal user Root Rootful container Rootless container More secure!

Performance Issues in Rootless Containers Rootful container’s network requires root privileges. → slirp4netns and RootlessKit provides rootless container networking. These components causes network performance degradation. 3 Performance degradation occurs!

Our Solution: bypass4netns bypass4netns bypasses these components with rootless. • It creates a socket on a host and switches sockets in a container to it. • The switched socket is not slowed down by rootless components. → It provides significant performance improvements. 4 Bypassing bottleneck!

More details: Requirements in Rootless Containers Rootless containers have following requirements. 5

More details: Requirements in Rootless Containers Rootless containers have following requirements. • Fully rootless: Every components should run without root privileges. 5

More details: Requirements in Rootless Containers Rootless containers have following requirements. • Fully rootless: Every components should run without root privileges. • No dedicated kernel modules: Implementing a secure and stable kernel module is too hard. Dedicated kernel modules increases maintenance costs. 5

More details: Requirements in Rootless Containers Rootless containers have following requirements. • Fully rootless: Every components should run without root privileges. • No dedicated kernel modules: Implementing a secure and stable kernel module is too hard. Dedicated kernel modules increases maintenance costs. • No application modifications: Rootless containers aims to work as same as ordinary rootful containers. 5

Related Works Other approaches cannot work with fully rootless. e.g.) binary with SUID, dedicated kernel modules, eBFP-based approach. LD_PRELOAD cannot handle static-linked binaries = Lack of compatibility LD_PRELOAD hooks libc’s functions calling, but static-linked binaries do not call them. 6

Seccomp and Seccomp Notify bypass4netns uses “Seccomp Notify” to switch sockets. • Secure computing mode (Seccomp) filters syscalls with a static policy. seccomp_unotify(2) (Seccomp Notify) filters syscalls dynamically. • Supervisor process determines whether to execute syscalls or not. 7

Switching Socket’s Reference with Seccomp Notify Seccomp Notify provides SECCOMP_IOCTL_NOTIF_ADDFD. • It switches the target of a file descriptor to other file description like a socket. 8 Rootless Container Process Target syscall Socket‘s file descriptor bypass4netns Socket switching module A socket in a container Seccomp Notify 5.execution

Switching Socket’s Reference with Seccomp Notify Seccomp Notify provides SECCOMP_IOCTL_NOTIF_ADDFD. • It switches the target of a file descriptor to other file description like a socket. 8 Rootless Container Process Target syscall Socket‘s file descriptor bypass4netns Socket switching module A socket in a container Seccomp Notify 5.execution

Switching Socket’s Reference with Seccomp Notify Seccomp Notify provides SECCOMP_IOCTL_NOTIF_ADDFD. • It switches the target of a file descriptor to other file description like a socket. 8 Rootless Container Process Target syscall Socket‘s file descriptor bypass4netns Socket switching module A socket in a container Seccomp Notify 5.execution 1. Notify

Switching Socket’s Reference with Seccomp Notify Seccomp Notify provides SECCOMP_IOCTL_NOTIF_ADDFD. • It switches the target of a file descriptor to other file description like a socket. 8 Rootless Container Process Target syscall Socket‘s file descriptor bypass4netns Socket switching module A socket in a container A socket allocated on a host Seccomp Notify 5.execution 1. Notify 2. Allocate a socket

Switching Socket’s Reference with Seccomp Notify Seccomp Notify provides SECCOMP_IOCTL_NOTIF_ADDFD. • It switches the target of a file descriptor to other file description like a socket. 8 Rootless Container Process Target syscall Socket‘s file descriptor bypass4netns Socket switching module A socket in a container A socket allocated on a host Seccomp Notify 5.execution 1. Notify 2. Allocate a socket 3. Socket switching (SECCOMP_IOCTL_NOTIF_ADDFD)

Switching Socket’s Reference with Seccomp Notify Seccomp Notify provides SECCOMP_IOCTL_NOTIF_ADDFD. • It switches the target of a file descriptor to other file description like a socket. 8 Rootless Container Process Target syscall Socket‘s file descriptor bypass4netns Socket switching module A socket in a container A socket allocated on a host Seccomp Notify 5.execution 4. Socket switched 1. Notify 2. Allocate a socket 3. Socket switching (SECCOMP_IOCTL_NOTIF_ADDFD)

Switching Socket’s Reference with Seccomp Notify Hooking syscalls with Seccomp Notify causes huge overhead (about 100x). → Hooking every syscalls causes another performance issues. 9 336 29494 0 5000 10000 15000 20000 25000 30000 35000 w/o Seccomp Notify w/ SeccompNotify Execution time (ns) getpid(2) execution time

Investigation: Which syscalls need to be hooked? We investigated socket-related syscalls in iperf3, wget, nginx, httpd. bypss4netns hooks following syscalls. • (2) Configuration: records sockets configuration. • (3) Connection: checks endpoint and performs switching. • (4) Status: writes dummy values when needed. • (7) Close: cleanups the socket’s information. bypass4netns does not hook communication syscalls like recv(2),send(2) → Overhead by Seccomp Notify does not affect the performance. 10

Example: Connecting to External Endpoints bypass4netns performs switching when the destination is external endpoints. External Endpoints: Not other containers and loopback address 11 memset((char *)&end_addr, 0, sizeof(end_addr)); end_addr.sin_family = AF_INET; end_addr.sin_port = htons(80); end_addr.sin_addr.s_addr = inet_addr("133.3.254.6"); The endpoint is external Flow of switching in bypass4netns

Example: Connecting to External Endpoints bypass4netns performs switching when the destination is external endpoints. External Endpoints: Not other containers and loopback address 11 memset((char *)&end_addr, 0, sizeof(end_addr)); end_addr.sin_family = AF_INET; end_addr.sin_port = htons(80); end_addr.sin_addr.s_addr = inet_addr("133.3.254.6"); sockfd = socket(AF_INET, SOCK_STREAM, 0); The endpoint is external Flow of switching in bypass4netns (1) Creation

Example: Connecting to External Endpoints bypass4netns performs switching when the destination is external endpoints. External Endpoints: Not other containers and loopback address 11 memset((char *)&end_addr, 0, sizeof(end_addr)); end_addr.sin_family = AF_INET; end_addr.sin_port = htons(80); end_addr.sin_addr.s_addr = inet_addr("133.3.254.6"); sockfd = socket(AF_INET, SOCK_STREAM, 0); int buf_size = 4096; setsockopt(sockfd, SOL_SOCKET, SO_SNDBUF, (void *)&buf_size, sizeof(int)); The endpoint is external Flow of switching in bypass4netns (1) Creation (2) Configuration

Example: Connecting to External Endpoints bypass4netns performs switching when the destination is external endpoints. External Endpoints: Not other containers and loopback address 11 memset((char *)&end_addr, 0, sizeof(end_addr)); end_addr.sin_family = AF_INET; end_addr.sin_port = htons(80); end_addr.sin_addr.s_addr = inet_addr("133.3.254.6"); sockfd = socket(AF_INET, SOCK_STREAM, 0); int buf_size = 4096; setsockopt(sockfd, SOL_SOCKET, SO_SNDBUF, (void *)&buf_size, sizeof(int)); connect(sockfd, (struct sockaddr *)&end_addr, sizeof(end_addr)); The endpoint is external Flow of switching in bypass4netns (1) Creation (2) Configuration (3) Connection

Example: Connecting to External Endpoints bypass4netns performs switching when the destination is external endpoints. External Endpoints: Not other containers and loopback address 11 memset((char *)&end_addr, 0, sizeof(end_addr)); end_addr.sin_family = AF_INET; end_addr.sin_port = htons(80); end_addr.sin_addr.s_addr = inet_addr("133.3.254.6"); sockfd = socket(AF_INET, SOCK_STREAM, 0); int buf_size = 4096; setsockopt(sockfd, SOL_SOCKET, SO_SNDBUF, (void *)&buf_size, sizeof(int)); connect(sockfd, (struct sockaddr *)&end_addr, sizeof(end_addr)); // “sockfd” is already switched! The endpoint is external Flow of switching in bypass4netns (1) Creation (2) Configuration (3) Connection

Example: Connecting to External Endpoints bypass4netns performs switching when the destination is external endpoints. External Endpoints: Not other containers and loopback address 11 memset((char *)&end_addr, 0, sizeof(end_addr)); end_addr.sin_family = AF_INET; end_addr.sin_port = htons(80); end_addr.sin_addr.s_addr = inet_addr("133.3.254.6"); sockfd = socket(AF_INET, SOCK_STREAM, 0); int buf_size = 4096; setsockopt(sockfd, SOL_SOCKET, SO_SNDBUF, (void *)&buf_size, sizeof(int)); connect(sockfd, (struct sockaddr *)&end_addr, sizeof(end_addr)); // “sockfd” is already switched! getpeername(sockfd, (struct sockaddr*)&peer_addr, &len); printf("peer: %s¥n", inet_ntoa(peer_addr.sin_addr)); The endpoint is external Flow of switching in bypass4netns (1) Creation (2) Configuration (3) Connection (4) Status v

Example: Connecting to External Endpoints bypass4netns performs switching when the destination is external endpoints. External Endpoints: Not other containers and loopback address 11 memset((char *)&end_addr, 0, sizeof(end_addr)); end_addr.sin_family = AF_INET; end_addr.sin_port = htons(80); end_addr.sin_addr.s_addr = inet_addr("133.3.254.6"); sockfd = socket(AF_INET, SOCK_STREAM, 0); int buf_size = 4096; setsockopt(sockfd, SOL_SOCKET, SO_SNDBUF, (void *)&buf_size, sizeof(int)); connect(sockfd, (struct sockaddr *)&end_addr, sizeof(end_addr)); // “sockfd” is already switched! getpeername(sockfd, (struct sockaddr*)&peer_addr, &len); printf("peer: %s¥n", inet_ntoa(peer_addr.sin_addr)); sprintf(buffer, "GET / HTTP/1.1¥r¥n¥r¥n"); send(sockfd, buffer, strlen(buffer), 0); The endpoint is external Flow of switching in bypass4netns (1) Creation (2) Configuration (3) Connection (4) Status Other syscalls

Example: Connecting to External Endpoints bypass4netns performs switching when the destination is external endpoints. External Endpoints: Not other containers and loopback address 11 memset((char *)&end_addr, 0, sizeof(end_addr)); end_addr.sin_family = AF_INET; end_addr.sin_port = htons(80); end_addr.sin_addr.s_addr = inet_addr("133.3.254.6"); sockfd = socket(AF_INET, SOCK_STREAM, 0); int buf_size = 4096; setsockopt(sockfd, SOL_SOCKET, SO_SNDBUF, (void *)&buf_size, sizeof(int)); connect(sockfd, (struct sockaddr *)&end_addr, sizeof(end_addr)); // “sockfd” is already switched! getpeername(sockfd, (struct sockaddr*)&peer_addr, &len); printf("peer: %s¥n", inet_ntoa(peer_addr.sin_addr)); sprintf(buffer, "GET / HTTP/1.1¥r¥n¥r¥n"); send(sockfd, buffer, strlen(buffer), 0); close(sockfd); The endpoint is external Flow of switching in bypass4netns (1) Creation (2) Configuration (3) Connection (4) Status Other syscalls (7) Close

Accelerating Multi-node Communications Usernetes (Rootless Kubernetes) also has the same performance issue. • Creating VXLAN interface on a host requires root privileges. • VXLAN interfaces in rootless containers are used. → slirp4netns and RootlessKit process VXLAN packets and it causes overhead. 12 Causes performance issue!

Accelerating Multi-node Communications Rewriting the address enables faster multi-node communications. 1. A published port is bound with the host’s address. 2. The address and port are shared among hosts via KVS. 3. bypass4netns hooks the connection to the socket. 4. Rewrites connect(2)’s the container’s address to the host’s address. 13 C1 bypass4netns C2 bypass4netns HostA HostB

Accelerating Multi-node Communications Rewriting the address enables faster multi-node communications. 1. A published port is bound with the host’s address. 2. The address and port are shared among hosts via KVS. 3. bypass4netns hooks the connection to the socket. 4. Rewrites connect(2)’s the container’s address to the host’s address. 13 C1 bypass4netns C2 Socket binding on 80/tcp bypass4netns HostA HostB

Accelerating Multi-node Communications Rewriting the address enables faster multi-node communications. 1. A published port is bound with the host’s address. 2. The address and port are shared among hosts via KVS. 3. bypass4netns hooks the connection to the socket. 4. Rewrites connect(2)’s the container’s address to the host’s address. 13 C1 bypass4netns C2 Socket binding on 80/tcp Socket binded on 12345/tcp bypass4netns HostA HostB

Accelerating Multi-node Communications Rewriting the address enables faster multi-node communications. 1. A published port is bound with the host’s address. 2. The address and port are shared among hosts via KVS. 3. bypass4netns hooks the connection to the socket. 4. Rewrites connect(2)’s the container’s address to the host’s address. 13 C1 bypass4netns C2 Socket binding on 80/tcp Socket binded on 12345/tcp bypass4netns Container Host C2:80/tcp HostB:12345/tcp KVS HostA HostB

Accelerating Multi-node Communications Rewriting the address enables faster multi-node communications. 1. A published port is bound with the host’s address. 2. The address and port are shared among hosts via KVS. 3. bypass4netns hooks the connection to the socket. 4. Rewrites connect(2)’s the container’s address to the host’s address. 13 C1 Socket connecting to C2:80/tcp bypass4netns C2 Socket binding on 80/tcp Socket binded on 12345/tcp bypass4netns Container Host C2:80/tcp HostB:12345/tcp KVS HostA HostB

Accelerating Multi-node Communications Rewriting the address enables faster multi-node communications. 1. A published port is bound with the host’s address. 2. The address and port are shared among hosts via KVS. 3. bypass4netns hooks the connection to the socket. 4. Rewrites connect(2)’s the container’s address to the host’s address. 13 C1 Socket connecting to C2:80/tcp bypass4netns C2 Socket binding on 80/tcp Socket binded on 12345/tcp bypass4netns Container Host C2:80/tcp HostB:12345/tcp KVS HostA HostB HostB:12345/tcp

Accelerating Multi-node Communications Rewriting the address enables faster multi-node communications. 1. A published port is bound with the host’s address. 2. The address and port are shared among hosts via KVS. 3. bypass4netns hooks the connection to the socket. 4. Rewrites connect(2)’s the container’s address to the host’s address. 13 C1 Socket connecting to C2:80/tcp Socket connecting to HostB:12345/tcp bypass4netns C2 Socket binding on 80/tcp Socket binded on 12345/tcp bypass4netns Container Host C2:80/tcp HostB:12345/tcp KVS HostA HostB HostB:12345/tcp

Accelerating Multi-node Communications Rewriting the address enables faster multi-node communications. 1. A published port is bound with the host’s address. 2. The address and port are shared among hosts via KVS. 3. bypass4netns hooks the connection to the socket. 4. Rewrites connect(2)’s the container’s address to the host’s address. 13 C1 Socket connecting to C2:80/tcp Socket connecting to HostB:12345/tcp bypass4netns C2 Socket binding on 80/tcp Socket binded on 12345/tcp bypass4netns Container Host C2:80/tcp HostB:12345/tcp KVS HostA HostB HostB:12345/tcp

Evaluation We evaluated bypass4netns • Does it work without modifying applications? • Does it provide faster communications and better performance? Evaluations were performed with 2 virtual machines. • CPU: AMD EPYC 7452 (2.35GHz base clock, 8 core assigned) • Memory: 16GB • OS: Ubuntu 22.04 14

Application Compatibility We used official images or original source codes. Evaluations show • Most of tested applications worked. → Our approach works in real applications without modifications. • It worked including static linked applications. → bypass4netns works with apps that LD_PRELOAD cannot handle. Not working case exists • Seccomp Notify sometimes dropped syscalls silently. (with Go 1.21.3 HTTP Client) → This happens when running more than 2 HTTP Clients with goroutines. 15

Performance Evaluation We measured the performance in 6 networks. • rootful-pfd: Rootful container, publish ports via port-forwarding(iptables) • rootless-pfd: Rootless container, publish ports via port-forwarding(RootlessKit) • b4ns-pfd: Rootless container, publish ports via bypass4netns • rootful-vxlan: Rootful container, VXLAN interface on a host is used. • rootless-vxlan: Rootless container, VXLAN interface in a container is used. • b4ns-multimode: Rootless container, bypass4netns runs in multi-node mode. 16 rootless-vxlan b4ns-multinode rootful-vxlan

Performance Evaluation: iperf3 Containers with bypass4netns achieved the best performance. • b4ns-pfd achieved 30x faster throughput(19.7 Gbps) than rootless-pfd(570 Mbps) • b4ns-pfd achieved better performance than rootful-pfd (17.9 Gbps). → b4ns-pfd bypassed components like a bridge and it brought better performance. • b4ns-multinode achieved the best performance in multi-node communications. 17

Performance Evaluation: Redis Redis performance benchmark with redis-benchmark • b4ns-pfd achieved 2x and b4ns-multimode achieved 3x better performance. • Rootless with bypass4netns achieved better performance than rootful. → We consider that same reason with iperf3 caused this. • bypass4netns provides better performance in a real used application. 18

Performance Evaluation: MySQL MySQL benchmark with sysbench’s OLTP benchmark. • No significant performance improvements were observed. • The latency was slightly reduced. • Rootless without bypass4netns provided almost the same performance as rootful. → Network performance seems not to be dominant in OLTP with RDB. 19

Summary Objective: Accelerating TCP/IP communications in Rootless containers. • Current rootless containers have communication performance issues. Solution: bypass4netns, fully rootless socket switching based approach. • Switching sockets in rootless containers to host’s sockets. • We also proposed an approach to accelerate multi-node communication. Evaluation: 30x faster throughput and performance improvements in applications. bypass4netns removes performance tradeoffs and makes container security better. Upstream implementation is available at https://github.com/rootless-containers/bypass4netns 20

Appendix 21

Requirements in Rootless Containers Rootless containers have following requirements. • Fully rootless: Every components should run without root privileges. • No dedicated kernel modules: Implementing a secure and stable kernel module is too hard. Dedicated kernel modules lack compatibility and increase maintenance costs. • No application modifications: Rootless containers aims to work as same as ordinary rootful containers. 22

More details: Rootless Containers Networking Creating network interfaces (veth) requires a host’s root privilege. → Rootless network components are used instead of veth IFs. • RootlessKit: handles incoming connections for published ports. • slirp4netns: handles outgoing connections to external endpoints. 23

Issues in Rootless Containers Networking These components have tradeoffs: performance degradation • RootlessKit: Relaying payloads between a host and a container. • slirp4netns: Extracts payloads from packets and sends it via host’s socket. 24

Publishing Ports with Socket Switching Once a socket is switched, the socket exists in a host’s NetNS. → Other containers cannot connect to the socket with container’s address. bypass4netns rewrites an address to the sockets in hooking connect(2). • Rewrites the destination address via /proc/PID/mem. • Returns the container’s address when getpeername(2) called. 25

Performance Evaluation: Nginx Static file distribution benchmark with nginx and Go HTTP Client • Client binary was statically linked. → The result shows that bypass4netns can handle static linked binary correctly. • Containers with bypass4netns provides almost same performance as rootful-pfd. 26

Discussion: Security Consideration bypass4netns bypasses iptables and other networking components. → Access control with iptables does not work. bypass4netns provides dynamic connectivity tracing mechanism. • Tracing agents checks connectivity between containers dynamically. • bypass4netns checks the status before socket switching. 27

Discussion: As a Versatile Socket Switching Method Seccomp Notify can be used as a versatile socket switching method. • LD_PRELOAD: Cannot handle static linked binaries. • Seccomp Notify: Can handle any binaries but cause huge overheads. → bypass4netns avoids this overhead by handling only required syscalls. bypass4netns provides the ability to switch TCP sockets to other sockets. • No application modifications is required. • Providing dynamic switching depending on the peer address. e.g. ) Switching connections to UNIX Domain sockets or RDMA sockets. → Faster and lighter communication 28

Future Task: Kubernetes/Usernetes Support bypass4netns accelerates Pod-to-Pod communication. • Our approach allocates all sockets on the host → This may lead port starvation or unintended connection. • Another approach is to use sidecar-proxy to relay communications. → The proxy can cause some performance degradation, but reliable. 29 Our approach Another approach Validating and relaying connections with proxy Switching only proxy’s socket Need many sockets! Unintended connection!