Bring Cloud-Native Networking to FreeBSD Jails: Porting Calico from Linux/EuroBSDCon2025

EuroBSDCon 2025

Transcript

1. Bring Cloud-Native Networking to FreeBSD Jails: Porting Calico from Linux Sep. 27 / EuroBSDCon 2025 Souma Sakaguchi (Future University Hakodate, Japan) This is a part of joint work with Shintaro Suzuki (Future University Hakodate) Yuki Nakata (SAKURA internet inc. / Future Univ. Hakodate) Katsuya Matsubara (Future University Hakodate)
2. 2 Who am I Souma Sakaguchi • Master’s student at Future University Hakodate, Japan • Interests: Container Networking / Virtualization / OS • Research Topic: Lightweight Security Sandboxing for Containers • Email: mokemoke1319@gmail.com
3. Proposed Solution: • Transplant Linux Containers onto FreeBSD • Provide the Compatibility of Linux Containers by Jail and Linuxulator • Transparently Apply Capsicum to the Linux Containers 3 Academic Work: Lightweight Security Sandboxing for Containers [1] Objective: Mitigate exploits of vulnerabilities in the Linux kernel while preserving the containers' lightweight nature [1] Y. Nakata, S. Suzuki and K. Matsubara: Reducing Attack Surface with Container Transplantation for Lightweight Sandboxing, Proceedings of the 14th ACM SIGOPS Asia-Pacific Workshop on Systems. pp. 58-64 (2023). Invalidate exploits of Linux kernel vulnerabilities with less overheads because of the kernel implementation defferencies Reduces attack surface (syscalls) by Capusicum
4. 4 Academic Work: FreeBSD as a Linux-compat Runtime [1] [1] Y. Nakata, S. Suzuki and K. Matsubara: Reducing Attack Surface with Container Transplantation for Lightweight Sandboxing, Proceedings of the 14th ACM SIGOPS Asia-Pacific Workshop on Systems. pp. 58-64 (2023). Comparison of sandboxed Linux runtimes and FreeBSD’s linuxularotor # of supported syscalls unixbench scores BETTER BETTER
5. 5 Academic Work: Can FreeBSD Mitigate Linux Vulnerability Exploits? [2] [1] Y. Nakata, S. Suzuki and K. Matsubara: Reducing Attack Surface with Container Transplantation for Lightweight Sandboxing, Proceedings of the 14th ACM SIGOPS Asia-Pacific Workshop on Systems. pp. 58-64 (2023). Access to files that should not be readable Execute attack code on Linux Linux
6. 6 Academic Work: Can FreeBSD Mitigate Linux Vulnerability Exploits? [2 [2] Y. Nakata, S. Suzuki and K. Matsubara: Reducing Attack Surface with Container Transplantation for Lightweight Sandboxing, Proceedings of the 14th ACM SIGOPS Asia-Pacific Workshop on Systems. pp. 58-64 (2023). Execute attack code on Linuxulator Failed to read ! FreeBSD’s Linuxurator
7. 7 Motivation: Importance of Container Networking On cloud computing platform, Container Networking is crucial. • Microservices Architecture - Assigning a Container to each Microservice - The overall service is composed of processes that container communication, triggered by user access • Frequent container communication [2] [2] J. Lewis and M. Fowler: Microservices, 2014, https://martinfowler.com/articles/microservices.html (Accessed on 2025/7/24) Container Container Container Container Container Container Microservices Architecture
8. 8 Current Situation Around Container Networking Containers and their supporting technologies are dominated by Linux コンテナ コンテナ コンテナ コンテナ コンテナ コンテナ Microservices Architecture • We want a rich container environment just like Linux! ◦ Jail with vnet only implements primitive container network ◦ Lacks functionality to transplant Linux Container ecosystems [2] J. Lewis and M. Fowler: Microservices, 2014, https://martinfowler.com/articles/microservices.html (Accessed on 2025/7/24) Microservices Architecture
9. 9 Requirements for Container Networking on FreeBSD Container-to-Container on single node Container-to-Internet • Isolating network resources • Assigning network device to Isolated spaces • Assigning IP addresses • Clouds consist of multiple nodes • Routing control to containers on different nodes • Accessing to external resources from containers • Packet Control Using Filtering and NAT Container-to-Container among nodes
10. 10 Porting Container Networking From Linux To FreeBSD What Should I Do Container-to-Container on single node Container-to-Internet Container-to-Container among nodes
11. 11 Approach: Porting Linux Standard CNI Plugins Porting Calico to FreeBSD for containers communication • Calico: The de facto standard[4] CNI Plugin and Modules - Configuration network device and route Container-to-Container on single node - Route Advertisement to containers by BGP Container-to-Container among nodes - Packet filtering and NAT Container-to-Internet [4] Datadog: 11 Facts About Real-World Container Use, 2020, https://www.datadoghq.com/ja/container-report-2020/ (Accessed on 2025/7/22)
12. 12 [4] Datadog: 11 Facts About Real-World Container Use, 2020, https://www.datadoghq.com/ja/container-report-2020/ (Accessed on 2025/7/22) Approach: Porting Linux Standard CNI Plugins Porting Calico to FreeBSD for containers communication • Calico: The de facto standard[4] CNI Plugin and Modules - Configuration network device and route Container-to-Container on single node - Route Advertisement to containers by BGP Container-to-Container among nodes - Packet filtering and NAT Container-to-Internet
13. 13 [4] Datadog: 11 Facts About Real-World Container Use, 2020, https://www.datadoghq.com/ja/container-report-2020/ (Accessed on 2025/7/22) Approach: Porting Linux Standard CNI Plugins Porting Calico to FreeBSD for containers communication • Calico: The de facto standard[4] CNI Plugin and Modules - Configuration network device and route Container-to-Container on single node - Route Advertisement to containers by BGP Container-to-Container among nodes - Packet filtering and NAT Container-to-Internet
14. 14 [4] Datadog: 11 Facts About Real-World Container Use, 2020, https://www.datadoghq.com/ja/container-report-2020/ (Accessed on 2025/7/22) Approach: Porting Linux Standard CNI Plugins Porting Calico to FreeBSD for containers communication • Calico: The de facto standard[4] CNI Plugin and Modules - Configuration network device and route Container-to-Container on single node - Route Advertisement to containers by BGP Container-to-Container among nodes - Packet filtering and NAT Container-to-Internet
15. 15 Calico Modules to Meet the Requirements etcd: Key-Value store with configurations for Calico modules calicoctl: Register Calico configurations to etcd Calico-BIRD: BGP Configuration and route control Felix: Management of route control and packet diltering iptables: Firewall and NAT configuration on the Linux kernel nerdctl: Controlling container runtime and CNI plugin Calico Plugin: CNI Plugins that configure IP addresses and network devices Container Runtime: Container management, such as execution and deletion
16. 16 Issues of Porting Calico To FreeBSD Calicoの 独自実装版BIRD Issue 3: Can't use iptables in Linux Issue 2: Can't use the network configuration in Linux Issue 1: Can't share Network Resources
17. 17 Issue 1: Network Namespace vs VNET VNET • Isolating network resources per Jail ◦ Like Linux Network Namespace • Sharing network resources among multiple containers ◦ Jail and VNET have a one-to-one mapping　 New Share Inherit VNET cannot share network resources among multiple Jails Container Container Container Container Container Container
18. 18 How To Share Network Resources Nested Multiple Jails within a VNET Jail • Multiple jails without isolated network run within a Vnet Jail • Implemented this mechanism in the low-level container runtime runj
19. 19 VNET Library for Calico Implementing a library that enables VNET operations instead of Network Namespace operations • Implement VNET library useable by Calico modules - Network Namespace library compatible - https://github.com/oss-fun/vnet func New() (NsHandle, error) { err := unix.Unshare(unix.CLONE_NEWNET) if err != nil { return -1, err } return Get() } ex. function of Creating Network Namespace
20. VNET Library for Calico Implementing a library that enables VNET operations instead of Network Namespace operations • Implement VNET library useable by Calico modules - Network Namespace library compatible - https://github.com/oss-fun/vnet func New() (NsHandle, error) { err := unix.Unshare(unix.CLONE_NEWNET) if err != nil { return -1, err } return Get() } ex. function of Creating Network Namespace 20 jid, _, errno := unix.Syscall( unix.SYS_JAIL_SET, uintptr(unsafe.Pointer(&iov[0])), uintptr(len(iov)), uintptr(JAIL_CREATE|JAIL_ATTACH), ) Create Jails Based on Pre-prepared VNET Jail Configurations
21. Issue 2: BIRD / Calico Plugin / Felix: Need for an Alternative to Netlink 21 Netlink, used for network configuration, is not fully available on FreeBSD • Netlink: Network Resource Management Provided by the Linux Kernel Request Network Configurations via Netlink through the Library BIRD - Route Advertisement Using BGP Calico Plugin - Assign IP addresses - Create network devices Felix - Routing table operations - Monitoring network devices
22. Reproducing Behavior Equivalent to Netlink 22 Implemented a wrapper library on FreeBSD capable of Netlink-equivalent behavior feature • Routing Socket: Netlink-Compatible routing operations available on FreeBSD - Only routing table operations are possible with an equivalent function • ioctl(AF_INET): Functions for managing network resources, as used by “ifconfig” - Create network devices - Assign IP addresses • https://github.com/oss-fun/netlink
23. Routing Table Operations by Routing Socket 23 Reimplemented constants, functions, and structures with different values between Linux and FreeBSD • Usable with Minor Modifications #define AF_NETLINK 16 #define PF_ROUTE 17 Netlink (Linux) Routing Socket (FreeBSD) Differences in constants for creating communication sockets with the same role type NlMsghdr struct { Len uint32 Type uint16 Flags uint16 Seq uint32 Pid uint32 } Defined the message header structure used for Routing Socket
24. Network Resource Management by ioctl(AF_INET) 24 Defined Functions with same Interface for FreeBSD func (h *Handle) LinkAdd(link Link) error ex. function of creating a link device 1. Prepared a message requesting link device creation 2. Request to Netlink ｜ Create link device 3. Receive results from Netlink Linux
25. FreeBSD 1. Create socket for ioctl 2. Create link device by ioctl system call 3. Receive results 25 func (h *Handle) LinkAdd(link Link) error _, _, errno := unix.Syscall( unix.SYS_IOCTL, uintptr(fd), uintptr(unix.SIOCIFCREATE2), uintptr(unsafe.Pointer(&ifr)), ) Network Resource Management by ioctl(AF_INET) Defined Functions with same Interface for FreeBSD ex. function of creating a link device
26. Issue 3: Felix: Need for an Alternative to Iptables 26 iptables, used for packet filtering and NAT, is unavailable on FreeBSD. • iptables: Feature of Packet filtering・NAT on Linux - Apply the network policy defined by the user - For communication with external hosts Packet Control and NAT via Rules chain: Set of Rules chain & Rule Jump mark: Marking to Packets ipset: IP Address Pool Management
27. ipfw 27 Modified Felix to use ipfw with functionality equivalent to iptables • ipfw: Packet Filtering and NAT functionality implemented in the FreeBSD kernel - Replicating iptables functionality using ipfw capabilities - Implementation needs to account for differences in specification methods and syntax Packet Control and NAT via Rules RuleNumber & Rule Jump Tag: Marking to Packets Table: IP Address Pool Management
28. Imitating the Concept of iptables Tables 28 Convert Rules Separately for Incoming, Outgoing, and Forwarded Packets ipfw add 1 call 10 ip from not me to me in ipfw add 2 call 20 ip from me to not me out ipfw add 3 call 30 ip from not me to not me ipfw add 4 skipto 65535 ip from any to any ipfw add 10 count // in packet ipfw add 11 call （PREROUTING-raw ） ipfw add 12 call （PREROUTING-mangle ） ipfw add 13 call （PREROUTING-nat ） ipfw add 14 call （INPUT-filter ） ipfw add 19 skipto 65535 ip from any to any ipfw add 20 count // out packet ipfw add 21 call （OUTPUT-raw ） ipfw add 22 call （OUTPUT-nat ） ipfw add 23 call （OUTPUT-filter ） ipfw add 24 call （POSTROUTING-mangle ） ipfw add 25 call （POSTROUTING-nat ） ipfw add 29 skipto 65535 ip from any to any ipfw add 30 count // forward packet ipfw add 31 call （PREROUTING-raw ） ipfw add 32 call （PREROUTING-mangle ） ipfw add 33 call （PREROUTING-nat ） ipfw add 34 call （FORWARD-filter ） ipfw add 35 call （POSTROUTING-mangle ） ipfw add 36 call （POSTROUTING-nat ） ipfw add 60 skipto 65535 ip from any to any （some_chain ） ipfw add 65535 allow ip from any to any Convert into a Single Rule
29. Instead of “Chain”: Management Rules by Rule Number 29 Allow Packets to ◦◦ Allow Packets to ×× Jamp chain_A Drop Packets from □□ … Drop Pakctets to △△ Drop Packets from▲▲ … Return Main Rules chain_A iptables（Linux） • Management rules by chain • Give custom names to chains and Rule jump with a specified chain name • Return to the calling chain Chain-Equivalent Behavior by Rule Number Jumps
30. 30 ipfw（FreeBSD） • Always assigns rule number • Rule jump with a rule number instead of chain jamp (call Action & skipto Action) • Return to the calling rule (return Action) Instead of “Chain”: Management Rules by Rule Number Chain-Equivalent Behavior by Rule Number Jumps Create Rule Groups by Treating Rules Numbered 100 and Above as Chains 1: Allow Packets to ◦◦ 2: Allow Packets to ×× 3: Jamp Rule 100 4: Drop Packets from □□ … 100: Drop Pakctets to △△ 101: Drop Packets from▲▲ … 199: Return Main Rules Rule 100…
31. Packet Control via Rules 31 Equivalent functionality despite differences in syntax ex. Deny packets to UDP and Port 4789 ( = VXLAN Traffic) iptables -A INPUT -p udp --dport 4789 -j DROP ipfw add 100 deny udp from any to any dst-port 4789 ipfw（FreeBSD） iptables（Linux） By implementing differences in syntax, the same rules can be reproduced
32. NAT via Rules 32 Communication from Containers to the Outside Is Possible nat 1 ip from table(cali40masq-ipam-pools) to not table(cali40all-ipam-pools) via em0 nat 1 ip from any to any in via em0 ipfw nat 1 config if em0 unreg_only ipfw（FreeBSD） Outgoing Packets Incoming Packets
33. Instead of “Mark”: Tag 33 Equivalent Functionality Despite Differences in Available Types iptables（Linux） • mark: marked with a uint32 ipfw（FreeBSD） • Tag: marked with a uint16 iptables -t mangle -A INPUT -j MARK --set-mark 0x1 iptables -A INPUT -m mark --mark 0x1 -j ACCEPT ipfw add 10 count tag 1 ip from any to any ipfw add 11 allow ip from any to any tagged 1 “Mark” used by Calico is sufficient with half the bit width
34. Instead of “ipset”: Table 34 Equivalent Functionality iptables（Linux） ipfw（FreeBSD） ipfw add 100 Deny ip from table(calico_ip) to any iptables -A INPUT -m set --match-set calico_ip -j Drop Name: calico-ip Members: 192.168.10.1 172.168.0.0/16 --- table(calico_ip), set(0) --- 192.168.10.1 172.168.0.0/16
35. 35 Implement Felix Felix Manage Chain Struct Manage Table Struct Communication with etcd Rules Rules Save and Restore Rule Synchronization Filtering by eBPF Communication with k8s Using wireguard Existing implementation fixed unsupport User land Kernel ipfw add 1 call 10 ip from not me to me in ipfw add 2 call 20 ip from me to not me out ipfw add 3 call 30 ip from not me to not me ipfw add 4 skipto 65535 ip from any to any ipfw add 10 count // in packet …
36. Container-to-Container on single node: Setup etcd 36
37. Container-to-Container on single node: Setup etcd 37 apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: default-ipv4-ippool spec: cidr: 172.16.166.0/24 ipipMode: Never vxlanMode: Never natOutgoing: true disabled: false Range of IP addresses used by containers
38. Container-to-Container on single node: Setup etcd 38 etcdctl put /calico/resources/v3/projectcalico.org/clusterinformations/default '{ "kind": "ClusterInformation", "apiVersion": "projectcalico.org/v3", "metadata": { "name": "default" }, "spec": { "datastoreReady": true } }' Store Cluster Information into etcd
39. Container-to-Container on single node: Setup etcd 39 etcdctl put /calico/resources/v3/projectcalico.org/nodes/node1 '{ "kind": "Node", "apiVersion": "projectcalico.org/v3", "metadata": { "name": "node1" }, "spec": { "bgp": { "ipv4Address": "192.168.100.11/24" } }, "status": {} }' Store Node Information such as host IP into etcd
40. Container-to-Container on single node: Run Container-A 40
41. Container-to-Container on single node: Run Container-A 41 { "cniVersion": "1.0.0", "name": "99-calico", "plugins": [ { "datastore_type": "etcdv3", "etcd_endpoints": "http://127.0.0.1:2379, "log_level": "DEBUG", "log_file_path": "/var/log/calico/cni/cni.log", "type": "calico", "ipam": { "type": "calico-ipam" } } ] } Configured to get information from etcd Calico Plugin, Calico-IPAM Plugin
42. Container-to-Container on single node: Run Container-A 42 An IP address is assigned to eth0 created by Calico
43. Container-to-Container on single node: Run Container-B 43
44. Container-to-Container on single node: Run Container-B 44 apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: default-ipv4-ippool spec: cidr: 192.168.100.0/24 ipipMode: Never vxlanMode: Never natOutgoing: true disabled: false Change the IP address range, deploy containers on a different network
45. Container-to-Container on single node: Run Container-B 45 Different IP address is assigned to ContainerA
46. Container-to-Container on single node: Ping Test 46
47. Container-to-Container among nodes: Setup BIRD 47
48. Container-to-Container among nodes: Setup BIRD 48 log syslog all; log stderr all; router id 192.168.100.11; protocol kernel { scan time 20; export all; } protocol direct { interface "*"; } protocol bgp node2 { local as 64512; neighbor 192.168.100.12 as 64512; export all; import all; } registers Neighbor Nodes for BGP Route Advertisement
49. Container-to-Container among nodes: Setup BIRD 49 Each node can perform route control via the BGP protocol
50. Container-to-Container among nodes: Ping Test 50
51. Container-to-Internet: Setup Felix 51
52. Container-to-Internet: Setup Felix 52 Verify that NAT rules are applied Verify the rule structure for each chain ex. cali-F-INPUT Manage IP addresses collectively using table
53. Container-to-Internet: Ping Test 53
54. 54 Summary • Outcome: An implementation of the major CNI plugin ‘Calico’ for FreeBSD • Challenge & Solution ◦ Vnet can't share network resources between Jails ✅ Use a nested Vnet jail to share a private network among jails ◦ Existing Netlink implementation for FreeBSD has lacks on functionality ✅Re-implement a wrapper library capable of netlink-equivalent behavior using the routing socket and ioctls ◦ No Iptables in FreeBSD ✅Convert iptables-style routing rules for IPFW
55. Implementation Code of Calico for FreeBSD 55 Calico Plugin [branch: feature/cni-plugin] - https://github.com/oss-fun/souma-calico/tree/feature/cni-plugin calicoctl [branch: feature/calicoctl] - https://github.com/oss-fun/souma-calico/tree/feature/calicoctl Felix [branch: feature/felix] - https://github.com/oss-fun/souma-calico/tree/feature/felix calico-BIRD [branch: dev/freebsd] - https://github.com/oss-fun/souma-calico-bird/tree/dev/freebsd netlink Package [branch: develop] - https://github.com/oss-fun/netlink/tree/develop vnet Package [branch: main] - https://github.com/oss-fun/vnet