Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How do I troubleshooting on container, more than docker?

How do I troubleshooting on container, more than docker?

#CNTUG #Taiwan #podman #docker #kubernetes #debug #troubleshooting

FB Cloud Native Taiwan User Group:


Phil Huang

June 12, 2020

More Decks by Phil Huang

Other Decks in Technology


  1. How do I Troubleshooting on Container, more than Docker? Phil

    Huang 黃秉鈞 <[email protected]> HKOSCon 2020 , Webinar, June 12, 2020
  2. # whois Phil Huang 黃秉鈞 • OpenSource Community Member ◦

    Cloud Native Taiwan User Group (CNTUG) ◦ SDNDS-TW • Red Hat HK/TW Solution Architect ◦ Ansible IT Automation ◦ OpenShift Container Platform ◦ Software-Defined Networking (SDN) ◦ Network Function Virtualization (NFV) • Personal Blog ◦ https://blog.pichuang.com.tw Ref: https://www.linkedin.com/in/phil-huang-09b09895/
  3. Agenda • Prologue ◦ Container is Linux ◦ Why Podman

    ◦ Why Network Namespace is Important? • Container? Docker? ◦ Understanding of Container Networking Model/Interface ◦ 3 Useful container network mode for troubleshooting • Pod? Kubernetes? ◦ Understanding of 5 Kubernetes Network Traffic ◦ How to do Kubernetes network troubleshooting? • How to obtain or get a debug container?
  4. Containers are Linux Ref: http://www.brendangregg.com/linuxperf.html https://www.redhat.com/en/blog/containers-are-linux • A linux container

    is nothing more than a process that runs on Linux ◦ Linux namespaces ◦ cgroups ◦ SELinux ◦ seccomp • It also mean you can use Linux performance tools to do some troubleshooting works
  5. Why Podman? • Support multiple image formats including the OCI

    and Docker image formats • 3 Benefits ◦ Daemonless container engine ◦ Provides a familiar command experience compatible with the Docker CLI ◦ Build and run rootless containers as non-root • How to start? ◦ dnf install -y podman ◦ alias docker=podman Ref: https://speakerdeck.com/pichuang/the-first-journey-from-docker-to-podman
  6. Why Network Namespace is Important? • Container uses many Linux

    namespace technologies for isolation resource, such as user namspace / process / mnt / net ... • For network isolation, container uses Linux network namespace technology • Each network namespace can have its own: ◦ Network interface ◦ Routing tables ◦ Firewall rules ◦ DNS lookup ◦ IP address ◦ Subnets ◦ ... Ref: http://redhatgov.io/workshops/containers_the_hard_way/
  7. Docker ? Container ? • Docker Registry => Container Registry

    • Docker Images => Container Images • Docker Containers => OCI Containers • Dockerfile => Containerfile Ref: https://dwalsh.fedorapeople.org/ReplacingDockerWithPodman.pdf OCI: Open Container Initiative
  8. Container Networking Mode/Interface • Why need container networking? ◦ Allow

    containers to communicate to host machine ◦ Containers need to talk to Internet ◦ Containers can attach to multi networks • Explore the nature of communication between container resource, instead of focusing on the implementation details for specific container networking standards ◦ Docker use Container Network Model (CNM) ◦ Podman use Container Network Interface (CNI) • 3 Useful container network mode for troubleshooting ◦ Bridge mode ◦ Container mode ◦ Host mode Ref: https://www.nuagenetworks.net/blog/container-networking-standards/
  9. Kubernetes Networking Model • Kubernetes project DOES NOT HAVE a

    network model default implementation • There are multiple implementations in the world, all of which must follow CNI spec ◦ The most popular CNI plugins in community: Flannel and Calico • Explore the nature of communication between Kubernetes resource, instead of focusing on the implementation details for each of CNI plugins • 5 Kubernetes network traffic • 4 level debugging methods CNI: Container Network Interface
  10. Basic Concept: 1 ~ n Containers per Pod • Pods

    are the smallest deployable units in Kubernetes • A Pod is a group of one or more containers with shared storage&network
  11. Traffic Model: Container to Container • A network namespace provides

    a new network stack for all the containers per Pod • Containers within a Pod share an IP Address and port space
  12. Traffic Model: Pod-to-Pod in the same node • Connecting namespaces

    using a linux bridge • Every Pod gets its own IP address in a flat shared networking space
  13. Traffic Model: Pod-to-Pod across different nodes • Kubernetes uses iptables

    to handle many networking and port forwarding rules ◦ iptables ◦ routing tables
  14. Traffic Model: Pod-to-Service • Kubernetes Service ◦ Internal load balancer

    that routes the traffic to Pods • When load balancing for multiple backend pods, it uses unweighted round-robin scheduling
  15. Traffic Model: Service-to-Internet • Ingress ◦ Routing Internet traffic to

    Kubernetes • Egress ◦ Routing traffic to the Internet
  16. Running Container Level Debugging # Get a Shell to a

    Running Container kubectl exec -it pod/welcome-1-5h7z6 -- /bin/bash Ref: https://kubernetes.io/docs/tasks/debug-application-cluster/get-shell-running-container/
  17. Namespace Level Debugging # Running one Pod in namespace and

    specific node kubectl run ocp-debug-container --image quay.io/pichuang/debug-container --restart=Never -it --attach --rm --overrides='{ "apiVersion": "v1", "spec": { "nodeSelector":{"kubernetes.io/hostname":"compute-1"}}}' Debug Pod Namespace
  18. Node Level Debugging # Running one Pod on specific Node

    kubectl run ocp-debug-container --image quay.io/pichuang/debug-container --restart=Never -it --attach --rm --overrides='{ "apiVersion": "v1", "spec": { "nodeSelector":{"kubernetes.io/hostname":"compute-1"}, "hostNetwork": true}}' Debug Pod Container OS
  19. Running Pods Level Debugging # After kubectl v1.18, you can

    enable “kubectl alpha debug” to use ephemeral containers kubectl alpha debug -it ephemeral-demo --image=busybox --target=ephemeral-demo # Or use kubectl-debug Ref: https://github.com/aylei/kubectl-debug
  20. Make Your Company-Wide Debug Container • Super easy! It just

    put some network diagnostic tools into a container • Or get the container from netshoot: a Docker + Kubernetes network trouble-shooting swiss-army container Ref: https://github.com/pichuang/debug-container
  21. References 1. GitHub - nicolaka/netshoot 2. Container Bare Metal for

    Networking 3. A Guide to the Kubernetes Networking Model 4. Quay - pichuang/debug-container 5. Troubleshooting from Container to Any 6. Linux Containers the Hard Way