How do I troubleshooting on container, more than docker [OpenShift Special Edition]

#CNTUG #Taiwan #podman #docker #kubernetes #debug #troubleshooting

Red Hat Taiwan Facebook
https://www.facebook.com/RedHatTaiwan/

Slide:
https://docs.google.com/presentation/d/1vEldz9yEfb-vdaGOCyTrS84iAZPBBsKcCiScVkpdtBg/edit?usp=sharing

Transcript

1. How do I Troubleshooting on Container, more than Docker? Phil Huang <phil.huang@redhat.com> Solution Architect, Red Hat 2020/06/12 Special Edition
2. # whois Phil Huang 黃秉鈞 • Red Hat HK/TW Solution Architect ◦ Ansible IT Automation ◦ OpenShift Container Platform ◦ Software-Defined Networking (SDN) ◦ Network Function Virtualization (NFV) • OpenSource Community Member ◦ Cloud Native Taiwan User Group (CNTUG) ◦ SDNDS-TW • Personal Blog ◦ https://blog.pichuang.com.tw Ref: https://www.linkedin.com/in/phil-huang-09b09895/
3. Agenda • Prologue ◦ Container is Linux ◦ Why Podman ◦ Why Network Namespace is Important? • Container? Docker? ◦ Understanding of Container Networking Model/Interface ◦ 3 Useful container network mode for troubleshooting • Pod? Kubernetes? OpenShift? ◦ Understanding of 5 OpenShift Network Traffic ◦ How to do OpenShift network troubleshooting? • How to obtain or get a debug container?
4. Prologue
5. Containers are Linux Ref: http://www.brendangregg.com/linuxperf.html https://www.redhat.com/en/blog/containers-are-linux • A linux container is nothing more than a process that runs on Linux ◦ Linux namespaces ◦ cgroups ◦ SELinux ◦ seccomp • It also mean you can use Linux performance tools to do some troubleshooting works
6. Why Podman? • Support multiple image formats including the OCI and Docker image formats • 3 Benefits ◦ Daemonless container engine ◦ Provides a familiar command experience compatible with the Docker CLI ◦ Build and run rootless containers as non-root • How to start? ◦ dnf install -y podman ◦ alias docker=podman Ref: https://speakerdeck.com/pichuang/the-first-journey-from-docker-to-podman
7. Why Network Namespace is Important? • Container uses many Linux namespace technologies for isolation resource, such as user namspace / process / mnt / net ... • For network isolation, container uses Linux network namespace technology • Each network namespace can have its own: ◦ Network interface ◦ Routing tables ◦ Firewall rules ◦ DNS lookup ◦ IP address ◦ Subnets ◦ ... Ref: http://redhatgov.io/workshops/containers_the_hard_way/
8. Docker ? Container ? • Docker Registry => Container Registry • Docker Images => Container Images • Docker Containers => OCI Containers • Dockerfile => Containerfile Ref: https://dwalsh.fedorapeople.org/ReplacingDockerWithPodman.pdf OCI: Open Container Initiative
9. Container Networking Interface • Why need container networking? ◦ Allow containers to communicate to host machine ◦ Containers need to talk to Internet ◦ Containers can attach to multi networks • Explore the nature of communication between container resource, instead of focusing on the implementation details for specific container networking standards ◦ Docker use Container Network Model (CNM) ◦ Podman use Container Network Interface (CNI) • 3 Useful container network mode for troubleshooting ◦ Bridge mode ◦ Container mode ◦ Host mode Ref: https://www.nuagenetworks.net/blog/container-networking-standards/
10. 3 Useful Container Network Interface for Troubleshooting
11. $ man podman run • Implement 7 network modes • By default, use bridge mode
12. Bridge Mode # podman run -it quay.io/pichuang/debug-container • Create a network namespace on the default bridge
13. Container Mode # podman run -it --net container:<container_name> quay.io/pichuang/debug-container • Reuse another container’s network namespace
14. Host Mode # podman run -it --net host quay.io/pichuang/debug-container • Use host network namespace
15. Kubernetes Networking Model • Kubernetes project DOES NOT HAVE a network model default implementation • There are multiple implementations in the world, all of which must follow CNI spec ◦ The most popular CNI plugins in community: Flannel and Calico • Explore the nature of communication between Kubernetes resource, instead of focusing on the implementation details for each of CNI plugins • 5 Kubernetes network traffic • 4 level debugging methods CNI: Container Network Interface
16. Enterprise Grade Kubernetes Platform
17. Undestanding of 5 Kubernetes Network Traffic
18. Basic Concept: 1 ~ n Containers per Pod • Pods are the smallest deployable units in Kubernetes • A Pod is a group of one or more containers with shared storage&network
19. Traffic Model: Container to Container • A network namespace provides a new network stack for all the containers per Pod • Containers within a Pod share an IP Address and port space
20. Traffic Model: Pod-to-Pod in the same node • Connecting namespaces using a linux bridge • Every Pod gets its own IP address in a flat shared networking space
21. Traffic Model: Pod-to-Pod across different nodes • Kubernetes uses iptables to handle many networking and port forwarding rules ◦ iptables ◦ routing tables
22. Traffic Model: Pod-to-Service • Kubernetes Service ◦ Internal load balancer that routes the traffic to Pods • When load balancing for multiple backend pods, it uses unweighted round-robin scheduling
23. Traffic Model: Service-to-Internet • Ingress ◦ Routing Internet traffic to Kubernetes • Egress ◦ Routing traffic to the Internet
24. How to do OpenShift Network Troubleshooting?
25. Running Container Level Debugging # Get a Shell to a Running Container oc rsh welcome-1-xqbm9 -- /bin/bash Ref: https://kubernetes.io/docs/tasks/debug-application-cluster/get-shell-running-container/
26. Running Container Level Debugging
27. Namespace Level Debugging # Running one Pod in namespace and specific node oc run ocp-debug-container --image quay.io/pichuang/debug-container --restart=Never -it --attach --rm --overrides='{ "apiVersion": "v1", "spec": { "nodeSelector":{"kubernetes.io/hostname":"compute-1"}}}' Debug Pod Namespace
28. Namespace Level Debugging
29. Node Level Debugging # Running one Pod on specific Node oc run ocp-debug-container --image quay.io/pichuang/debug-container --restart=Never -it --attach --rm --overrides='{ "apiVersion": "v1", "spec": { "nodeSelector":{"kubernetes.io/hostname":"compute-1"}, "hostNetwork": true}}' Debug Pod Container OS
30. Node Level Debugging
31. Running Pods Level Debugging # Cloning specific Pod and exec it oc debug pod/productpage-v1-597b74b4c-xzf92 Ref: https://blog.pichuang.com.tw/20200521-openshift-with-coreos-part-7/
32. Running Pods Level Debugging
33. Environment • OpenShift v4.4.4 • Kubernetes v1.17.1 Ref: https://k9scli.io/
34. How to obtain or make a debug container?
35. Make Your Company-Wide Debug Container • Pulling the registry.redhat.io/rhel8/support-tools from Red Hat • Or get the container from netshoot: a Docker + Kubernetes network trouble-shooting swiss-army container Ref: https://github.com/pichuang/debug-container
36. References 1. GitHub - nicolaka/netshoot 2. Container Bare Metal for Networking 3. A Guide to the Kubernetes Networking Model 4. Quay - pichuang/debug-container 5. Troubleshooting from Container to Any 6. Linux Containers the Hard Way