How do I troubleshooting on container, more than docker?

#CNTUG #Taiwan #podman #docker #kubernetes #debug #troubleshooting

FB Cloud Native Taiwan User Group:
https://www.facebook.com/groups/cloudnative.tw/

Slide:
https://docs.google.com/presentation/d/1bz9Dh8Lcq4-0OJNwiSADKrXQuVzXJ9a6GDCW8LSyC4Q/edit#slide=id.p

Transcript

1. How do I Troubleshooting on Container, more than Docker? Phil Huang 黃秉鈞 <pichuang@pichuang.com.tw> HKOSCon 2020 , Webinar, June 12, 2020
2. # whois Phil Huang 黃秉鈞 • OpenSource Community Member ◦ Cloud Native Taiwan User Group (CNTUG) ◦ SDNDS-TW • Red Hat HK/TW Solution Architect ◦ Ansible IT Automation ◦ OpenShift Container Platform ◦ Software-Defined Networking (SDN) ◦ Network Function Virtualization (NFV) • Personal Blog ◦ https://blog.pichuang.com.tw Ref: https://www.linkedin.com/in/phil-huang-09b09895/
3. Cloud Native Taiwan User Group Ref: https://www.facebook.com/groups/cloudnative.tw/ More than 2700+ member in here!!! 記得要 加入 分享 一同參與
4. Agenda • Prologue ◦ Container is Linux ◦ Why Podman ◦ Why Network Namespace is Important? • Container? Docker? ◦ Understanding of Container Networking Model/Interface ◦ 3 Useful container network mode for troubleshooting • Pod? Kubernetes? ◦ Understanding of 5 Kubernetes Network Traffic ◦ How to do Kubernetes network troubleshooting? • How to obtain or get a debug container?
5. Prologue
6. Containers are Linux Ref: http://www.brendangregg.com/linuxperf.html https://www.redhat.com/en/blog/containers-are-linux • A linux container is nothing more than a process that runs on Linux ◦ Linux namespaces ◦ cgroups ◦ SELinux ◦ seccomp • It also mean you can use Linux performance tools to do some troubleshooting works
7. Why Podman? • Support multiple image formats including the OCI and Docker image formats • 3 Benefits ◦ Daemonless container engine ◦ Provides a familiar command experience compatible with the Docker CLI ◦ Build and run rootless containers as non-root • How to start? ◦ dnf install -y podman ◦ alias docker=podman Ref: https://speakerdeck.com/pichuang/the-first-journey-from-docker-to-podman
8. Why Network Namespace is Important? • Container uses many Linux namespace technologies for isolation resource, such as user namspace / process / mnt / net ... • For network isolation, container uses Linux network namespace technology • Each network namespace can have its own: ◦ Network interface ◦ Routing tables ◦ Firewall rules ◦ DNS lookup ◦ IP address ◦ Subnets ◦ ... Ref: http://redhatgov.io/workshops/containers_the_hard_way/
9. Docker ? Container ? • Docker Registry => Container Registry • Docker Images => Container Images • Docker Containers => OCI Containers • Dockerfile => Containerfile Ref: https://dwalsh.fedorapeople.org/ReplacingDockerWithPodman.pdf OCI: Open Container Initiative
10. Container Networking Mode/Interface • Why need container networking? ◦ Allow containers to communicate to host machine ◦ Containers need to talk to Internet ◦ Containers can attach to multi networks • Explore the nature of communication between container resource, instead of focusing on the implementation details for specific container networking standards ◦ Docker use Container Network Model (CNM) ◦ Podman use Container Network Interface (CNI) • 3 Useful container network mode for troubleshooting ◦ Bridge mode ◦ Container mode ◦ Host mode Ref: https://www.nuagenetworks.net/blog/container-networking-standards/
11. 3 Useful Container Network Mode for Troubleshooting
12. Bridge Mode # podman run -it quay.io/pichuang/debug-container • Create a network namespace on the default bridge
13. Container Mode # podman run -it --net container:<container_name> quay.io/pichuang/debug-container • Reuse another container’s network namespace
14. Host Mode # podman run -it --net host quay.io/pichuang/debug-container • Use host network namespace
15. Kubernetes Networking Model • Kubernetes project DOES NOT HAVE a network model default implementation • There are multiple implementations in the world, all of which must follow CNI spec ◦ The most popular CNI plugins in community: Flannel and Calico • Explore the nature of communication between Kubernetes resource, instead of focusing on the implementation details for each of CNI plugins • 5 Kubernetes network traffic • 4 level debugging methods CNI: Container Network Interface
16. Undestanding of 5 Kubernetes Network Traffic
17. Basic Concept: 1 ~ n Containers per Pod • Pods are the smallest deployable units in Kubernetes • A Pod is a group of one or more containers with shared storage&network
18. Traffic Model: Container to Container • A network namespace provides a new network stack for all the containers per Pod • Containers within a Pod share an IP Address and port space
19. Traffic Model: Pod-to-Pod in the same node • Connecting namespaces using a linux bridge • Every Pod gets its own IP address in a flat shared networking space
20. Traffic Model: Pod-to-Pod across different nodes • Kubernetes uses iptables to handle many networking and port forwarding rules ◦ iptables ◦ routing tables
21. Traffic Model: Pod-to-Service • Kubernetes Service ◦ Internal load balancer that routes the traffic to Pods • When load balancing for multiple backend pods, it uses unweighted round-robin scheduling
22. Traffic Model: Service-to-Internet • Ingress ◦ Routing Internet traffic to Kubernetes • Egress ◦ Routing traffic to the Internet
23. How to do Kubernetes Network Troubleshooting?
24. Running Container Level Debugging # Get a Shell to a Running Container kubectl exec -it pod/welcome-1-5h7z6 -- /bin/bash Ref: https://kubernetes.io/docs/tasks/debug-application-cluster/get-shell-running-container/
25. Running Container Level Debugging
26. Namespace Level Debugging # Running one Pod in namespace and specific node kubectl run ocp-debug-container --image quay.io/pichuang/debug-container --restart=Never -it --attach --rm --overrides='{ "apiVersion": "v1", "spec": { "nodeSelector":{"kubernetes.io/hostname":"compute-1"}}}' Debug Pod Namespace
27. Namespace Level Debugging
28. Node Level Debugging # Running one Pod on specific Node kubectl run ocp-debug-container --image quay.io/pichuang/debug-container --restart=Never -it --attach --rm --overrides='{ "apiVersion": "v1", "spec": { "nodeSelector":{"kubernetes.io/hostname":"compute-1"}, "hostNetwork": true}}' Debug Pod Container OS
29. Node Level Debugging
30. Running Pods Level Debugging # After kubectl v1.18, you can enable “kubectl alpha debug” to use ephemeral containers kubectl alpha debug -it ephemeral-demo --image=busybox --target=ephemeral-demo # Or use kubectl-debug Ref: https://github.com/aylei/kubectl-debug
31. Running Pods Level Debugging
32. Environment • OpenShift v4.4.4 • Kubernetes v1.17.1 Ref: https://k9scli.io/
33. How to obtain or make a debug container?
34. Make Your Company-Wide Debug Container • Super easy! It just put some network diagnostic tools into a container • Or get the container from netshoot: a Docker + Kubernetes network trouble-shooting swiss-army container Ref: https://github.com/pichuang/debug-container
35. References 1. GitHub - nicolaka/netshoot 2. Container Bare Metal for Networking 3. A Guide to the Kubernetes Networking Model 4. Quay - pichuang/debug-container 5. Troubleshooting from Container to Any 6. Linux Containers the Hard Way
36. Q&A Tifa: 是否加入 Cloud Native Taiwan User Group ? You: [Y/y]