How do I troubleshooting on container, more than docker?

#CNTUG #Taiwan #podman #docker #kubernetes #debug #troubleshooting

FB Cloud Native Taiwan User Group:
https://www.facebook.com/groups/cloudnative.tw/

Slide:
https://docs.google.com/presentation/d/1bz9Dh8Lcq4-0OJNwiSADKrXQuVzXJ9a6GDCW8LSyC4Q/edit#slide=id.p

Transcript

1. How do I Troubleshooting on Container,
   more than Docker?
   Phil Huang 黃秉鈞
   HKOSCon 2020 , Webinar, June 12, 2020
   

   View Slide

2. # whois Phil Huang 黃秉鈞
   ● OpenSource Community Member
   ○ Cloud Native Taiwan User Group (CNTUG)
   ○ SDNDS-TW
   ● Red Hat HK/TW Solution Architect
   ○ Ansible IT Automation
   ○ OpenShift Container Platform
   ○ Software-Defined Networking (SDN)
   ○ Network Function Virtualization (NFV)
   ● Personal Blog
   ○ https://blog.pichuang.com.tw
   Ref: https://www.linkedin.com/in/phil-huang-09b09895/
   

   View Slide

3. Cloud Native Taiwan User Group
   Ref: https://www.facebook.com/groups/cloudnative.tw/
   More than 2700+ member in here!!!
   記得要 加入 分享 一同參與
   

   View Slide

4. Agenda
   ● Prologue
   ○ Container is Linux
   ○ Why Podman
   ○ Why Network Namespace is Important?
   ● Container? Docker?
   ○ Understanding of Container Networking Model/Interface
   ○ 3 Useful container network mode for troubleshooting
   ● Pod? Kubernetes?
   ○ Understanding of 5 Kubernetes Network Traffic
   ○ How to do Kubernetes network troubleshooting?
   ● How to obtain or get a debug container?
   

   View Slide

5. Prologue
   

   View Slide

6. Containers are Linux
   Ref: http://www.brendangregg.com/linuxperf.html
   https://www.redhat.com/en/blog/containers-are-linux
   ● A linux container is nothing
   more than a process that runs
   on Linux
   ○ Linux namespaces
   ○ cgroups
   ○ SELinux
   ○ seccomp
   ● It also mean you can use Linux
   performance tools to do some
   troubleshooting works
   

   View Slide

7. Why Podman?
   ● Support multiple image formats including the
   OCI and Docker image formats
   ● 3 Benefits
   ○ Daemonless container engine
   ○ Provides a familiar command experience
   compatible with the Docker CLI
   ○ Build and run rootless containers as
   non-root
   ● How to start?
   ○ dnf install -y podman
   ○ alias docker=podman
   Ref: https://speakerdeck.com/pichuang/the-first-journey-from-docker-to-podman
   

   View Slide

8. Why Network Namespace is Important?
   ● Container uses many Linux namespace technologies for isolation resource, such as user
   namspace / process / mnt / net ...
   ● For network isolation, container uses Linux network namespace technology
   ● Each network namespace can have its own:
   ○ Network interface
   ○ Routing tables
   ○ Firewall rules
   ○ DNS lookup
   ○ IP address
   ○ Subnets
   ○ ...
   Ref: http://redhatgov.io/workshops/containers_the_hard_way/
   

   View Slide

9. Docker ? Container ?
   ● Docker Registry => Container Registry
   ● Docker Images => Container Images
   ● Docker Containers => OCI Containers
   ● Dockerfile => Containerfile
   Ref: https://dwalsh.fedorapeople.org/ReplacingDockerWithPodman.pdf
   OCI: Open Container Initiative
   

   View Slide

10. Container Networking Mode/Interface
    ● Why need container networking?
    ○ Allow containers to communicate to host machine
    ○ Containers need to talk to Internet
    ○ Containers can attach to multi networks
    ● Explore the nature of communication between container resource, instead of focusing on the
    implementation details for specific container networking standards
    ○ Docker use Container Network Model (CNM)
    ○ Podman use Container Network Interface (CNI)
    ● 3 Useful container network mode for troubleshooting
    ○ Bridge mode
    ○ Container mode
    ○ Host mode
    Ref: https://www.nuagenetworks.net/blog/container-networking-standards/
    

    View Slide

11. 3 Useful Container Network Mode for
    Troubleshooting
    

    View Slide

12. Bridge Mode
    # podman run -it quay.io/pichuang/debug-container
    ● Create a network namespace on the
    default bridge
    

    View Slide

13. Container Mode
    # podman run -it --net container: quay.io/pichuang/debug-container
    ● Reuse another container’s network
    namespace
    

    View Slide

14. Host Mode
    # podman run -it --net host quay.io/pichuang/debug-container
    ● Use host network namespace
    

    View Slide

15. Kubernetes Networking Model
    ● Kubernetes project DOES NOT HAVE a network model default implementation
    ● There are multiple implementations in the world, all of which must follow CNI spec
    ○ The most popular CNI plugins in community: Flannel and Calico
    ● Explore the nature of communication between Kubernetes resource, instead of focusing on
    the implementation details for each of CNI plugins
    ● 5 Kubernetes network traffic
    ● 4 level debugging methods
    CNI: Container Network Interface
    

    View Slide

16. Undestanding of 5 Kubernetes
    Network Traffic
    

    View Slide

17. Basic Concept: 1 ~ n Containers per Pod
    ● Pods are the smallest
    deployable units in
    Kubernetes
    ● A Pod is a group of one
    or more containers
    with shared
    storage&network
    

    View Slide

18. Traffic Model: Container to Container
    ● A network namespace
    provides a new network
    stack for all the
    containers per Pod
    ● Containers within a Pod
    share an IP Address and
    port space
    

    View Slide

19. Traffic Model: Pod-to-Pod in the same node
    ● Connecting namespaces
    using a linux bridge
    ● Every Pod gets its own IP
    address in a flat shared
    networking space
    

    View Slide

20. Traffic Model: Pod-to-Pod across different nodes
    ● Kubernetes uses iptables
    to handle many
    networking and port
    forwarding rules
    ○ iptables
    ○ routing tables
    

    View Slide

21. Traffic Model: Pod-to-Service
    ● Kubernetes Service
    ○ Internal load balancer
    that routes the traffic
    to Pods
    ● When load balancing for
    multiple backend pods, it
    uses unweighted
    round-robin scheduling
    

    View Slide

22. Traffic Model: Service-to-Internet
    ● Ingress
    ○ Routing Internet
    traffic to Kubernetes
    ● Egress
    ○ Routing traffic to the
    Internet
    

    View Slide

23. How to do Kubernetes Network
    Troubleshooting?
    

    View Slide

24. Running Container Level Debugging
    # Get a Shell to a Running Container
    kubectl exec -it pod/welcome-1-5h7z6 -- /bin/bash
    Ref: https://kubernetes.io/docs/tasks/debug-application-cluster/get-shell-running-container/
    

    View Slide

25. Running Container Level Debugging
    

    View Slide

26. Namespace Level Debugging
    # Running one Pod in namespace and specific node
    kubectl run ocp-debug-container --image quay.io/pichuang/debug-container
    --restart=Never -it --attach --rm --overrides='{ "apiVersion": "v1", "spec": {
    "nodeSelector":{"kubernetes.io/hostname":"compute-1"}}}'
    Debug Pod
    Namespace
    

    View Slide

27. Namespace Level Debugging
    

    View Slide

28. Node Level Debugging
    # Running one Pod on specific Node
    kubectl run ocp-debug-container --image quay.io/pichuang/debug-container
    --restart=Never -it --attach --rm --overrides='{ "apiVersion": "v1", "spec": {
    "nodeSelector":{"kubernetes.io/hostname":"compute-1"}, "hostNetwork": true}}'
    Debug Pod
    Container OS
    

    View Slide

29. Node Level Debugging
    

    View Slide

30. Running Pods Level Debugging
    # After kubectl v1.18, you can enable “kubectl alpha debug” to use ephemeral containers
    kubectl alpha debug -it ephemeral-demo --image=busybox --target=ephemeral-demo
    # Or use kubectl-debug
    Ref: https://github.com/aylei/kubectl-debug
    

    View Slide

31. Running Pods Level Debugging
    

    View Slide

32. Environment
    ● OpenShift v4.4.4
    ● Kubernetes v1.17.1
    Ref: https://k9scli.io/
    

    View Slide

33. How to obtain or make a debug
    container?
    

    View Slide

34. Make Your Company-Wide Debug Container
    ● Super easy! It just put some network diagnostic
    tools into a container
    ● Or get the container from netshoot: a Docker +
    Kubernetes network trouble-shooting
    swiss-army container
    Ref: https://github.com/pichuang/debug-container
    

    View Slide

35. References
    1. GitHub - nicolaka/netshoot
    2. Container Bare Metal for Networking
    3. A Guide to the Kubernetes Networking Model
    4. Quay - pichuang/debug-container
    5. Troubleshooting from Container to Any
    6. Linux Containers the Hard Way
    

    View Slide

36. Q&A
    Tifa: 是否加入 Cloud Native Taiwan User Group ?
    You: [Y/y]
    

    View Slide