It's Not Always the Network (But Here's How to Prove It): Kubernetes Packet Capture for SREs

SREcon26 Americas presentation
https://www.usenix.org/conference/srecon26americas/presentation/shibuya

Transcript

1. 1 It's Not Always the Network (But Here's How to Prove It): Kubernetes Packet Capture for SREs Mitsuhiro Shibuya Mercari, Inc. @m4buya
2. 2 Introduction Agenda The Best-Effort Reality Underneath the Abstraction The SRE's Capture Toolkit for Kubernetes The Next Frontier: eBPF 2. 3. 4. 1. Conclusion 5.
3. 3 Introduction Agenda The Best-Effort Reality Underneath the Abstraction The SRE's Capture Toolkit for Kubernetes The Next Frontier: eBPF 2. 3. 4. 1. Conclusion 5.
4. 4 Introduction Me @ Network team
5. 5 Introduction Me @ Network team
6. 6 Introduction Me @ Network team We think it might be a network issue.
7. 7 Introduction Me @ Network team
8. 8 Introduction Me @ Network team It had nothing to do with the network.
9. 9 Introduction • Distributed systems require the network • Every operation uses the network • → Every failure implicates the network
10. 10 Introduction Everyone needs to be ready to perform deep network investigations!
11. 11 Scope and Disclaimer ✅ Kubernetes environment 󰢄 Bare metal, VMs ✅ Capture packets 󰢄 Deep packet analysis Disclaimer: Packet capture can expose sensitive data!
12. 12 Introduction Agenda The Best-Effort Reality Underneath the Abstraction The SRE's Capture Toolkit for Kubernetes The Next Frontier: eBPF 2. 3. 4. 1. Conclusion 5.
13. 13 Application Logic vs. Network Reality What our code expects: What the network provides: A reliable, ordered stream of data. A "best-effort" delivery of individual packets. Underneath the Abstraction
14. 14 The Realities of Best-Effort Networking ❌ No guaranteed latency or delivery order ❌ No guaranteed throughput
15. 15 These are not bugs. This is the network operating as designed. The Realities of Best-Effort Networking
16. 16 Introduction Agenda The Best-Effort Reality Underneath the Abstraction The SRE's Capture Toolkit for Kubernetes The Next Frontier: eBPF 2. 3. 4. 1. Conclusion 5.
17. 17 Packet Capture Strategies - Agenda Strategy 1: The On-Node Approach Strategy 2: The Binary Upload Approach (ksniff) Strategy 3: The In-Pod Approach (kubectl debug)
18. 18 Strategy 1: The On-Node Approach Caveat: You must have shell access with root privileges to the Kubernetes nodes.
19. 19 2 Observation Points Root netns Pod1 netns Pod2 netns eth0 veth111 eth0 veth222 eth0 Observation Point 1: Node's Network Interface Observation Point 2: Inside the Namespace (nsenter)
20. 20 Capturing on the Node's Interface Root netns Pod1 netns Pod2 netns eth0 veth111 eth0 veth222 eth0 Observation Point 1: Node's Network Interface Observation Point 2: Inside the Namespace (nsenter)
21. 21 Capturing on the Node's Interface # 1. First, get the pod's node and IP address. $ kubectl get pod $POD_NAME -o wide # 2. SSH into the correct node. $ ssh my-user@$NODE_NAME # 3. On the node, run tcpdump on the main interface, # filtering by the pod's IP. # tcpdump -i eth0 -n -w capture.pcap "host $POD_IP" BPF Filter: The "what" to capture Write to file No DNS Resolution Interface to listen on
22. 22 Node When to use this? When you suspect the problem is outside of the application container. Examples: • Pod A can't reach Pod B on a different node. • A NetworkPolicy is unexpectedly blocking traffic. Pod Pod 💣❓
23. 23 Capturing with nsenter Root netns Pod1 netns Pod2 netns eth0 veth111 eth0 veth222 eth0 Observation Point 1: Node's Network Interface Observation Point 2: Inside the Namespace (nsenter)
24. 24 Capturing with nsenter # 1. SSH into the node. $ ssh my-user@$NODE_NAME # 2. On the node, find the pod's PID using 'crictl'. # crictl pods --name $POD_NAME -q --state Ready | xargs crictl inspectp -o go-template --template '{{.info.pid}}' # 3. Use nsenter to run tcpdump inside the pod's network namespace. # nsenter -t $PID -n -- tcpdump -i eth0 -n -w capture.pcap
25. 25 Node When to use this? When you suspect the problem is inside of the application container. Examples: • Your application fails to start, complaining it can't bind to port 8080. • Istio service mesh sidecar is causing issues. ◦ Capturing on the lo interface allows you to see mTLS traffic unencrypted Pod Pod 💣❓
26. 26 Recap: 2 Observation Points Root netns Pod1 netns Pod2 netns eth0 veth111 eth0 veth222 eth0 Observation Point 1: Node's Network Interface Observation Point 2: Inside the Namespace (nsenter)
27. 27 Strategy 2: The Binary Upload Approach (ksniff) Pod eth0 tcpdump Statically-built tcpdump binaries Capture Upload # Conceptually: $ kubectl cp static-tcpdump ... && kubectl exec ... -- /path/to/tcpdump ...
28. 28 ksniff in Action $ kubectl sniff -n istioinaction webapp-64bf64c478-hrnhm
29. 29 ksniff - downsides: • Stated as non-production-ready • Modifies a running container's filesystem • Possible compatibility issues with the static binary
30. 30 Strategy 3: The In-Pod Approach (kubectl debug) Pod eth0 Capture # Creates an ephemeral container (GA since Kubernetes 1.25) $ kubectl debug -it my-pod --image=nicolaka/netshoot -- sh Ephemeral container tcpdump
31. 31 How Ephemeral Containers Work Network namespace Mount namespace (filesystem) App container ./my-app Mount namespace (filesystem) Ephemeral container /bin/sh Creates
32. 32 netshoot: network "swiss-army" container Includes troubleshooting tools like: • tcpdump • dig • curl • nmap … and much more! https://github.com/nicolaka/netshoot
33. 33 In-Pod Approach - Live Demo! https://youtu.be/C8ZCBusd-Eg
34. 34 In-Pod Approach - Command Examples # Or pipe into Wireshark directly $ kubectl -n istioinaction debug $POD_NAME -iq --image=nicolaka/netshoot -- tcpdump -i any -n -w - | wireshark -k -i - Write to stdout Capture on all interfaces Read from stdin Start capture immediately # Open netshoot shell $ kubectl debug -it -n istioinaction $POD_NAME --image=nicolaka/netshoot Use stdin, allocate TTY Use netshoot image Use stdin, but suppress messages
35. 35 Introduction Agenda The Best-Effort Reality Underneath the Abstraction The SRE's Capture Toolkit for Kubernetes The Next Frontier: eBPF 2. 3. 4. 1. Conclusion 5.
36. 36 The Next Frontier: eBPF
37. 37 Hubble - Live Demo! https://youtu.be/YsL6n03Yz84
38. 38 From speculative capture to targeted capture The New Workflow Hubble Brings:
39. 39 Introduction Agenda The Best-Effort Reality Underneath the Abstraction The SRE's Capture Toolkit for Kubernetes The Next Frontier: eBPF 2. 3. 4. 1. Conclusion 5.
40. 40 Summary: Choosing a Right Tool 🔐 Strategy 1: The On-Node Approach. Requires high privileges. • eth0 on nodes: for capturing traffic as the node sees it [outside pod] • nsenter: for capturing traffic as the application sees it [inside pod] 🏪 Strategy 2: The Binary Upload Approach (ksniff). [inside pod] Convenient, but be mindful of its intrusiveness. 🌟 Strategy 3: The In-Pod Approach (kubectl debug). [inside pod] A Kubernetes-native approach, with flexibility and ease-of-use. 🚀 Keep an eye on eBPF, the future of network observability.
41. 41 Now you will be ready to answer the question! Is it the Network?
42. 42 Now you will be ready to answer the question! It was / wasn't the Network!
43. 43 Thank you! Capturing Network Packets in Kubernetes - Mercari Engineering Blog 🔎 Kubernetes packet capture