How do I Troubleshooting on Container, more than Docker? Phil Huang 黃秉鈞 HKOSCon 2020 , Webinar, June 12, 2020 

# whois Phil Huang 黃秉鈞 ● OpenSource Community Member ○ Cloud Native Taiwan User Group (CNTUG) ○ SDNDS-TW ● Red Hat HK/TW Solution Architect ○ Ansible IT Automation ○ OpenShift Container Platform ○ Software-Defined Networking (SDN) ○ Network Function Virtualization (NFV) ● Personal Blog ○ https://blog.pichuang.com.tw Ref: https://www.linkedin.com/in/phil-huang-09b09895/ 

Cloud Native Taiwan User Group Ref: https://www.facebook.com/groups/cloudnative.tw/ More than 2700+ member in here!!! 記得要 加入 分享 一同參與 

Agenda ● Prologue ○ Container is Linux ○ Why Podman ○ Why Network Namespace is Important? ● Container? Docker? ○ Understanding of Container Networking Model/Interface ○ 3 Useful container network mode for troubleshooting ● Pod? Kubernetes? ○ Understanding of 5 Kubernetes Network Traffic ○ How to do Kubernetes network troubleshooting? ● How to obtain or get a debug container? 

Prologue 

Containers are Linux Ref: http://www.brendangregg.com/linuxperf.html https://www.redhat.com/en/blog/containers-are-linux ● A linux container is nothing more than a process that runs on Linux ○ Linux namespaces ○ cgroups ○ SELinux ○ seccomp ● It also mean you can use Linux performance tools to do some troubleshooting works 

Why Podman? ● Support multiple image formats including the OCI and Docker image formats ● 3 Benefits ○ Daemonless container engine ○ Provides a familiar command experience compatible with the Docker CLI ○ Build and run rootless containers as non-root ● How to start? ○ dnf install -y podman ○ alias docker=podman Ref: https://speakerdeck.com/pichuang/the-first-journey-from-docker-to-podman 

Why Network Namespace is Important? ● Container uses many Linux namespace technologies for isolation resource, such as user namspace / process / mnt / net ... ● For network isolation, container uses Linux network namespace technology ● Each network namespace can have its own: ○ Network interface ○ Routing tables ○ Firewall rules ○ DNS lookup ○ IP address ○ Subnets ○ ... Ref: http://redhatgov.io/workshops/containers_the_hard_way/ 

Docker ? Container ? ● Docker Registry => Container Registry ● Docker Images => Container Images ● Docker Containers => OCI Containers ● Dockerfile => Containerfile Ref: https://dwalsh.fedorapeople.org/ReplacingDockerWithPodman.pdf OCI: Open Container Initiative 

Container Networking Mode/Interface ● Why need container networking? ○ Allow containers to communicate to host machine ○ Containers need to talk to Internet ○ Containers can attach to multi networks ● Explore the nature of communication between container resource, instead of focusing on the implementation details for specific container networking standards ○ Docker use Container Network Model (CNM) ○ Podman use Container Network Interface (CNI) ● 3 Useful container network mode for troubleshooting ○ Bridge mode ○ Container mode ○ Host mode Ref: https://www.nuagenetworks.net/blog/container-networking-standards/ 

3 Useful Container Network Mode for Troubleshooting 

Bridge Mode # podman run -it quay.io/pichuang/debug-container ● Create a network namespace on the default bridge 

Container Mode # podman run -it --net container: quay.io/pichuang/debug-container ● Reuse another container’s network namespace 

Host Mode # podman run -it --net host quay.io/pichuang/debug-container ● Use host network namespace 

Kubernetes Networking Model ● Kubernetes project DOES NOT HAVE a network model default implementation ● There are multiple implementations in the world, all of which must follow CNI spec ○ The most popular CNI plugins in community: Flannel and Calico ● Explore the nature of communication between Kubernetes resource, instead of focusing on the implementation details for each of CNI plugins ● 5 Kubernetes network traffic ● 4 level debugging methods CNI: Container Network Interface 

Undestanding of 5 Kubernetes Network Traffic 

Basic Concept: 1 ~ n Containers per Pod ● Pods are the smallest deployable units in Kubernetes ● A Pod is a group of one or more containers with shared storage&network 

Traffic Model: Container to Container ● A network namespace provides a new network stack for all the containers per Pod ● Containers within a Pod share an IP Address and port space 

Traffic Model: Pod-to-Pod in the same node ● Connecting namespaces using a linux bridge ● Every Pod gets its own IP address in a flat shared networking space 

Traffic Model: Pod-to-Pod across different nodes ● Kubernetes uses iptables to handle many networking and port forwarding rules ○ iptables ○ routing tables 

Traffic Model: Pod-to-Service ● Kubernetes Service ○ Internal load balancer that routes the traffic to Pods ● When load balancing for multiple backend pods, it uses unweighted round-robin scheduling 

Traffic Model: Service-to-Internet ● Ingress ○ Routing Internet traffic to Kubernetes ● Egress ○ Routing traffic to the Internet 

How to do Kubernetes Network Troubleshooting? 

Running Container Level Debugging # Get a Shell to a Running Container kubectl exec -it pod/welcome-1-5h7z6 -- /bin/bash Ref: https://kubernetes.io/docs/tasks/debug-application-cluster/get-shell-running-container/ 

Running Container Level Debugging 

Namespace Level Debugging # Running one Pod in namespace and specific node kubectl run ocp-debug-container --image quay.io/pichuang/debug-container --restart=Never -it --attach --rm --overrides='{ "apiVersion": "v1", "spec": { "nodeSelector":{"kubernetes.io/hostname":"compute-1"}}}' Debug Pod Namespace 

Namespace Level Debugging 

Node Level Debugging # Running one Pod on specific Node kubectl run ocp-debug-container --image quay.io/pichuang/debug-container --restart=Never -it --attach --rm --overrides='{ "apiVersion": "v1", "spec": { "nodeSelector":{"kubernetes.io/hostname":"compute-1"}, "hostNetwork": true}}' Debug Pod Container OS 

Node Level Debugging 

Running Pods Level Debugging # After kubectl v1.18, you can enable “kubectl alpha debug” to use ephemeral containers kubectl alpha debug -it ephemeral-demo --image=busybox --target=ephemeral-demo # Or use kubectl-debug Ref: https://github.com/aylei/kubectl-debug 

Running Pods Level Debugging 

Environment ● OpenShift v4.4.4 ● Kubernetes v1.17.1 Ref: https://k9scli.io/ 

How to obtain or make a debug container? 

Make Your Company-Wide Debug Container ● Super easy! It just put some network diagnostic tools into a container ● Or get the container from netshoot: a Docker + Kubernetes network trouble-shooting swiss-army container Ref: https://github.com/pichuang/debug-container 

References 1. GitHub - nicolaka/netshoot 2. Container Bare Metal for Networking 3. A Guide to the Kubernetes Networking Model 4. Quay - pichuang/debug-container 5. Troubleshooting from Container to Any 6. Linux Containers the Hard Way 

Q&A Tifa: 是否加入 Cloud Native Taiwan User Group ? You: [Y/y]