Cilium & eBPF - Networking, Security & Observability

eBPF is on everyone’s lips these days. It’s been identified as one of the top 5 trends in Cloud Native by the CNCF technical committee. eBPF-based Cilium is now the default data plane for GKE. Even Microsoft just announced a big initiative around adding eBPF to Windows. But what exactly is it and how can you take advantage of it?

eBPF is one of the fastest emerging kernel technologies. Projects like Cilium leverage eBPF to provide highly scalable and efficient networking, security and observability for containers and microservices. This talk will provide an introduction to both eBPF and Cilium. We’ll explore how eBPF enables the dynamic insertion of powerful security, visibility, and control logic within the OS kernel, then dive into how Cilium leverages the power of eBPF to deliver efficient load balancing, multi-cluster networking, security policies, deep observability, transparent encryption and much much more.

Presented at the DFW Kubernetes meetup on 2020-06-23:

https://community.cncf.io/events/details/cncf-dallas-kubernetes-meetup-presents-cilium-ebpf-cloud-native-networking-security-observability/

For more information, visit:
https://ebpf.io
https://cilium.io
https://isovalent.com

Transcript

1. ebpf.io cilium.io isovalent.com Cilium & eBPF Cloud-Native Networking, Security & Observability Joe Stringer Software Engineer, Isovalent
2. What is eBPF? Highly efficient sandboxed virtual machine in the kernel, making it more programmable at native execution speed. Jointly maintained by Isovalent and Facebook with collaborations from Google, Red Hat, Netflix, CloudFlare, and many others. 2 $ clang -target bpf -emit-llvm -S \ 32-bit-example.c $ llc -march=bpf 32-bit-example.ll $ cat 32-bit-example.s cal: r1 = *(u32 *)(r1 + 0) r2 = *(u32 *)(r2 + 0) r2 += r1 *(u32 *)(r3 + 0) = r2 exit
3. Run eBPF programs on events 3 Process NIC Disk Process IO Read Send network packet connect() Sockets TCP/IP Network Device TCP retrans read() File Descriptor VFS Block Device Attachment points • Kernel functions (kprobes) • Userspace functions (uprobe) • System calls • Tracepoints • Sockets (data level) • Network devices (packet level) • Network device (DMA level) [XDP] • ...
4. eBPF Helpers 4 bpf_get_prandom_u32() eBPF helpers: • Stable kernel API exposed to eBPF programs to interact with the kernel • Includes ability to: ◦ Get process/cgroup context ◦ Manipulate network packets and forwarding ◦ Access eBPF maps ◦ Access socket data ◦ Send metrics to user space ◦ ... bpf_skb_store_bytes() bpf_redirect() bpf_get_current_pid_tgid() bpf_perf_event_output()
5. Process eBPF Maps 5 Maps eBPF map use cases: • Hold program state • Share state between programs • Share state with user space • Export metrics & statistics • Configure programs Map types: • Hash tables • Arrays • LRU (Least recently used) • Ring buffer • Stack trace • LPM (Longest prefix match)
6. Kprobe Example 6 Example: eBPF program using gobpf/bcc:
7. eBPF Loading 7 Verifier: • Checks safety of imported programs ◦ Access ◦ Bounds ◦ Termination JIT Compiler: • Ensures native execution performance without requiring to understand CPU • Compiles eBPF bytecode to CPU architecture specific instruction set Supported architectures: • X86_64, arm64, ppc64, s390x, mips64, sparc64, arm Byte code Byte code x86_64 generic Byte code generic JIT Verifier
8. 631 Daniel Borkmann (Isovalent, Maintainer) 591 Andrii Nakryiko (Facebook, Maintainer) 441 Alexei Starovoitov (Facebook, Maintainer) 285 Yonghong Song (Facebook) 222 Martin KaFai Lau (Facebook) 209 Jakub Kicinski (Facebook) 205 Stanislav Fomichev (Google) 181 John Fastabend (Isovalent) 174 Quentin Monnet (Isovalent) 142 Jesper Dangaard Brouer (Red Hat) 8 eBPF Contributors Interested? https:/ /ebpf.io/contribute Top contributors of the total 446 contributors to Linux eBPF from January 2016 to April 2021.
9. eBPF Users Load balancing, network security, traffic optimization, tracing 9 QoS, traffic optimization, network security, profiling Performance troubleshooting, tracing, networking Networking, advanced security, load-balancing, traffic optimization DDoS mitigation, QoS, traffic optimization Container runtime security, behavior analysis Profiling, tracing, working on iptables replacement upstream Networking, security and load-balancing for Kubernetes Uses BPF to isolate Chrome plugins
10. Cilium is open source software for transparently providing and securing the network and API connectivity between application services deployed in cloud-native environments like Kubernetes. What is Cilium? The foundation of Cilium is eBPF, which enables the dynamic insertion of powerful security, visibility, and networking control logic within the kernel. Besides providing traditional network level security, the flexibility of eBPF enables security on API and process level to secure communication within a container or pod. Read More 10
11. Networking
12. cluster Kubernetes Networking node pod container 12 node pod container pod container Networking (CNI) Networking plugin • Network devices • IP Address Management • Intra-node connectivity • Inter-node connectivity Kube Proxy • Services • iptables or ipvs • Service discovery kube-proxy kube-proxy
13. cluster Kubernetes Networking node pod container 13 node pod container pod container Cilium • Agent on each node • Tunneling or Direct Routing • eBPF native dataplane • kube-proxy replacement.
14. Kubernetes Services 14 app1 app2 ClusterIP 10.0.1.10 10.0.1.11 172.0.2.3 app2 10.0.1.12 East-west connectivity • Durable abstraction • Connect applications • Ephemeral addresses • High churn • Iptables or ipvs
15. eBPF-based kube-proxy • Linear List • All rules have to be replaced as a whole • Per-CPU Hash table Kubernetes Services 15
16. 16 node 1 pod 10.0.1.0/24 192.168.0.11/24 node 2 pod pod 192.168.0.12/24 ??? 10.0.2.0/24 Load Balancing
17. 17 node 1 pod 10.0.1.0/24 192.168.0.11/24 node 2 pod pod 192.168.0.12/24 10.0.2.0/24 NodePort 30000 NodePort 30000 LoadBalancer node 192.168.0.10/24 Load Balancing Batteries not included • NodePort • Cloud integrations
18. LoadBalancer node 18 192.168.0.10/24 node 1 pod 10.0.1.0/24 192.168.0.11/24 node 2 pod pod 192.168.0.12/24 10.0.2.0/24 Load Balancing Cilium • Pod-aware • Maglev • Standalone or distributed
19. 19 Platform Integration
20. 20 Cluster Mesh Shared Services Cluster 1 Cluster 2 pod-1 pod-2 pod-3 vault-1 vault-2 Vault Service pod-1 pod-2 Vault Service Vault Service
21. Security
22. Kubernetes / k8s CRD Worker Node Worker Node 22 backend frontend Attach Identity Verify Identity Eth IPv6 TCP Payload Frontend Identity - frontend Identities: - backend - Policy - Deployments HTTP Request Pkt Pkt ... Identity-based Security
23. Worker Node Worker Node 23 backend frontend HTTP Request Worker Node Worker Node backend frontend HTTP Request Port 80 Worker Node Worker Node backend frontend HTTP Request Allow HTTP GET /public API-aware Authorization L3 L4 L7
24. 24 Worker Node Worker Node kube-dns frontend HTTP Request DNS-aware Policy → Resolve www.mydomain.io ← A 130.57.66.19 www.mydomain.io Allowed egress: 130.57.66.19 apiVersion: "cilium.io/v2" kind: CiliumNetworkPolicy [...] specs: - endpointSelector: matchLabels: app: frontend egress: - toFQDNs: - matchName: "*.mydomain.io" toPorts: - ports: - port: "443" protocol: TCP Curious? https:/ /networkpolicy.io
25. L3 Matching Capabilities Kubernetes • Pod labels • Namespace name & labels • ServiceAccount name • Service names • Cluster name DNS Names • FQDN and regular expression CIDR • CIDR blocks with exceptions Cloud Providers (AWS,GKE) • EC2 labels • VPC/Subnet name/tags • Security group name Logical Entities • Everything inside cluster • Everything outside cluster • Local host • ... 25
26. Observability
27. 27 What is Observability? ‒ Show me the traffic between two applications. ‒ Show me why traffic is dropped. ‒ Show me what my application is doing! ‒ Do I see an increase in errors when certain conditions happen? ‒ … Is it DNS? ‒ Kubernetes doesn’t provide tooling to answer these questions.
28. 28 HUBBLE Pod Pod ‒ Service Dependency Maps ‒ Flow Display and Filtering ‒ Network Policy Viewer ‒ Detailed Flow Visibility ‒ Extensive Filtering ‒ JSON output ‒ Built-in Metrics for Operations & Application Monitoring HUBBLE METRICS What is Hubble?
29. Service Map 29
30. 30 $ kubectl get pods NAME READY STATUS RESTARTS AGE tiefighter 1/1 Running 0 2m34s xwing 1/1 Running 0 2m34s deathstar-5b7489bc84-crlxh 1/1 Running 0 2m34s deathstar-5b7489bc84-j7qwq 1/1 Running 0 2m34s $ hubble observe --follow -l class=xwing # DNS lookup to coredns default/xwing:41391 -> kube-system/coredns-66bff467f8-28dgp:53 to-proxy FORWARDED (UDP) kube-system/coredns-66bff467f8-28dgp:53 -> default/xwing:41391 to-endpoint FORWARDED (UDP) # ... # Successful HTTPS request to www.disney.com default/xwing:37836 -> www.disney.com:443 to-stack FORWARDED (TCP Flags: SYN) www.disney.com:443 -> default/xwing:37836 to-endpoint FORWARDED (TCP Flags: SYN, ACK) www.disney.com:443 -> default/xwing:37836 to-endpoint FORWARDED (TCP Flags: ACK, FIN) default/xwing:37836 -> www.disney.com:443 to-stack FORWARDED (TCP Flags: RST) # ... # Blocked HTTP request to deathstar backend default/xwing:49610 -> default/deathstar:80 Policy denied DROPPED (TCP Flags: SYN) Flow Metadata ‒ Ethernet headers ‒ IP & ICMP headers ‒ UDP/TCP ports, TCP flags ‒ HTTP, DNS, Kafka, ... Kubernetes ‒ Pod names and labels ‒ Service names ‒ Worker node names DNS (if available) ‒ Fully qualified domain names for source and destination Cilium ‒ Security identities and endpoints ‒ Drop reasons ‒ Policy verdict matches Flow Visibility
31. 31 DNS Visibility $ hubble observe -t l7 -o json --follow | \ jq -r 'select(.l7.dns.rcode!=null) | .destination.namespace +"/"+ .destination.pod_name + "\t" + .Summary' starwars/jar-jar-binks-6945758d55-pjdt4 DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Proxy unknown-galaxy.starwars.svc.cluster.local. AAAA) starwars/jar-jar-binks-6945758d55-pjdt4 DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Proxy unknown-galaxy.starwars.svc.cluster.local. A) starwars/jar-jar-binks-6945758d55-pjdt4 DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Proxy unknown-galaxy.svc.cluster.local. A) starwars/jar-jar-binks-6945758d55-pjdt4 DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Proxy unknown-galaxy.svc.cluster.local. AAAA) starwars/jar-jar-binks-6945758d55-pjdt4 DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Proxy unknown-galaxy.cluster.local. A) starwars/jar-jar-binks-6945758d55-pjdt4 DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Proxy unknown-galaxy.cluster.local. AAAA) starwars/jar-jar-binks-6945758d55-pjdt4 DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Proxy unknown-galaxy. A) starwars/jar-jar-binks-6945758d55-pjdt4 DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Proxy unknown-galaxy. AAAA)
32. Pod 32 Target HUBBLE Parser ‒ HTTP, TLS, gRPC, Kafka ‒ many more Metadata $ kubectl annotate pod tiefighter \ io.cilium.proxy-visibility="<Egress/80/TCP/HTTP>" pod/tiefighter annotated $ hubble observe --from-pod tiefighter \ --http-status 200 --output json | jq '.l7' { "type": "RESPONSE", "latency_ns": "4342402", "http": { "code": 200, "method": "POST", "url": "http://deathstar/v1/request-landing", "protocol": "HTTP/1.1", "headers": [ { "key": "User-Agent", "value": "curl/7.52.1" }, ... ] } } L7 Visibility
33. 33 More info in KubeCon EU 2018 slides: Accelerating Envoy and Istio with Cilium and the Linux Kernel https:/ /bit.ly/2G7DfIY Filter Chain Proxy Service TLS Endpoint TLS Clear Encrypted Handshake Encryption TLS Deferred encryption Transparent SSL Visibility
34. ‒ Open Metrics ‒ Scraping endpoint for e.g. Prometheus ‒ Metrics for Flow Events ‒ e.g. HTTP, DNS, TCP, ICMP ‒ Visualization with Grafana ‒ Example dashboards available ‒ Customizable & Extendable Grafana 34 HUBBLE Node HUBBLE Node Metrics Metrics Prometheus Storage Hubble Metrics
35. Cilium Use Cases 35 Container Networking • Highly efficient and flexible networking • Routing, Overlay, Cloud-provider native • IPv4, IPv6, NAT46 • Multi cluster routing • Wireguard or ipsec encryption Service Load balancing: • Highly scalable L3-L4 load balancing • Kubernetes services (replaces kube-proxy) • Multi-cluster • Service affinity (prefer zones) Microservices Security • Identity-based network security • API-aware security (HTTP, gRPC, Kafka, Cassandra, memcached, ..) • DNS-aware policies • Encryption • SSL data visibility Servicemesh: • Minimize overhead when injecting servicemesh sidecar proxies • Istio integration
36. Who is using Cilium? 36
37. 37 More Information: eBPF: https:/ /ebpf.io/ Cilium: https:/ /cilium.io/ Slack: https:/ /cilium.io/slack GitHub: https:/ /github.com/cilium/cilium Twitter: @ciliumproject, @joestringernz Thank you!