How do I troubleshooting on container, more than docker?

How do I troubleshooting on container, more than docker?

#CNTUG #Taiwan #podman #docker #kubernetes #debug #troubleshooting

FB Cloud Native Taiwan User Group:
https://www.facebook.com/groups/cloudnative.tw/

Slide:
https://docs.google.com/presentation/d/1bz9Dh8Lcq4-0OJNwiSADKrXQuVzXJ9a6GDCW8LSyC4Q/edit#slide=id.p

D907136acebc72f1df878541b26f271a?s=128

Phil Huang

June 12, 2020
Tweet

Transcript

  1. How do I Troubleshooting on Container, more than Docker? Phil

    Huang 黃秉鈞 <pichuang@pichuang.com.tw> HKOSCon 2020 , Webinar, June 12, 2020
  2. # whois Phil Huang 黃秉鈞 • OpenSource Community Member ◦

    Cloud Native Taiwan User Group (CNTUG) ◦ SDNDS-TW • Red Hat HK/TW Solution Architect ◦ Ansible IT Automation ◦ OpenShift Container Platform ◦ Software-Defined Networking (SDN) ◦ Network Function Virtualization (NFV) • Personal Blog ◦ https://blog.pichuang.com.tw Ref: https://www.linkedin.com/in/phil-huang-09b09895/
  3. Cloud Native Taiwan User Group Ref: https://www.facebook.com/groups/cloudnative.tw/ More than 2700+

    member in here!!! 記得要 加入 分享 一同參與
  4. Agenda • Prologue ◦ Container is Linux ◦ Why Podman

    ◦ Why Network Namespace is Important? • Container? Docker? ◦ Understanding of Container Networking Model/Interface ◦ 3 Useful container network mode for troubleshooting • Pod? Kubernetes? ◦ Understanding of 5 Kubernetes Network Traffic ◦ How to do Kubernetes network troubleshooting? • How to obtain or get a debug container?
  5. Prologue

  6. Containers are Linux Ref: http://www.brendangregg.com/linuxperf.html https://www.redhat.com/en/blog/containers-are-linux • A linux container

    is nothing more than a process that runs on Linux ◦ Linux namespaces ◦ cgroups ◦ SELinux ◦ seccomp • It also mean you can use Linux performance tools to do some troubleshooting works
  7. Why Podman? • Support multiple image formats including the OCI

    and Docker image formats • 3 Benefits ◦ Daemonless container engine ◦ Provides a familiar command experience compatible with the Docker CLI ◦ Build and run rootless containers as non-root • How to start? ◦ dnf install -y podman ◦ alias docker=podman Ref: https://speakerdeck.com/pichuang/the-first-journey-from-docker-to-podman
  8. Why Network Namespace is Important? • Container uses many Linux

    namespace technologies for isolation resource, such as user namspace / process / mnt / net ... • For network isolation, container uses Linux network namespace technology • Each network namespace can have its own: ◦ Network interface ◦ Routing tables ◦ Firewall rules ◦ DNS lookup ◦ IP address ◦ Subnets ◦ ... Ref: http://redhatgov.io/workshops/containers_the_hard_way/
  9. Docker ? Container ? • Docker Registry => Container Registry

    • Docker Images => Container Images • Docker Containers => OCI Containers • Dockerfile => Containerfile Ref: https://dwalsh.fedorapeople.org/ReplacingDockerWithPodman.pdf OCI: Open Container Initiative
  10. Container Networking Mode/Interface • Why need container networking? ◦ Allow

    containers to communicate to host machine ◦ Containers need to talk to Internet ◦ Containers can attach to multi networks • Explore the nature of communication between container resource, instead of focusing on the implementation details for specific container networking standards ◦ Docker use Container Network Model (CNM) ◦ Podman use Container Network Interface (CNI) • 3 Useful container network mode for troubleshooting ◦ Bridge mode ◦ Container mode ◦ Host mode Ref: https://www.nuagenetworks.net/blog/container-networking-standards/
  11. 3 Useful Container Network Mode for Troubleshooting

  12. Bridge Mode # podman run -it quay.io/pichuang/debug-container • Create a

    network namespace on the default bridge
  13. Container Mode # podman run -it --net container:<container_name> quay.io/pichuang/debug-container •

    Reuse another container’s network namespace
  14. Host Mode # podman run -it --net host quay.io/pichuang/debug-container •

    Use host network namespace
  15. Kubernetes Networking Model • Kubernetes project DOES NOT HAVE a

    network model default implementation • There are multiple implementations in the world, all of which must follow CNI spec ◦ The most popular CNI plugins in community: Flannel and Calico • Explore the nature of communication between Kubernetes resource, instead of focusing on the implementation details for each of CNI plugins • 5 Kubernetes network traffic • 4 level debugging methods CNI: Container Network Interface
  16. Undestanding of 5 Kubernetes Network Traffic

  17. Basic Concept: 1 ~ n Containers per Pod • Pods

    are the smallest deployable units in Kubernetes • A Pod is a group of one or more containers with shared storage&network
  18. Traffic Model: Container to Container • A network namespace provides

    a new network stack for all the containers per Pod • Containers within a Pod share an IP Address and port space
  19. Traffic Model: Pod-to-Pod in the same node • Connecting namespaces

    using a linux bridge • Every Pod gets its own IP address in a flat shared networking space
  20. Traffic Model: Pod-to-Pod across different nodes • Kubernetes uses iptables

    to handle many networking and port forwarding rules ◦ iptables ◦ routing tables
  21. Traffic Model: Pod-to-Service • Kubernetes Service ◦ Internal load balancer

    that routes the traffic to Pods • When load balancing for multiple backend pods, it uses unweighted round-robin scheduling
  22. Traffic Model: Service-to-Internet • Ingress ◦ Routing Internet traffic to

    Kubernetes • Egress ◦ Routing traffic to the Internet
  23. How to do Kubernetes Network Troubleshooting?

  24. Running Container Level Debugging # Get a Shell to a

    Running Container kubectl exec -it pod/welcome-1-5h7z6 -- /bin/bash Ref: https://kubernetes.io/docs/tasks/debug-application-cluster/get-shell-running-container/
  25. Running Container Level Debugging

  26. Namespace Level Debugging # Running one Pod in namespace and

    specific node kubectl run ocp-debug-container --image quay.io/pichuang/debug-container --restart=Never -it --attach --rm --overrides='{ "apiVersion": "v1", "spec": { "nodeSelector":{"kubernetes.io/hostname":"compute-1"}}}' Debug Pod Namespace
  27. Namespace Level Debugging

  28. Node Level Debugging # Running one Pod on specific Node

    kubectl run ocp-debug-container --image quay.io/pichuang/debug-container --restart=Never -it --attach --rm --overrides='{ "apiVersion": "v1", "spec": { "nodeSelector":{"kubernetes.io/hostname":"compute-1"}, "hostNetwork": true}}' Debug Pod Container OS
  29. Node Level Debugging

  30. Running Pods Level Debugging # After kubectl v1.18, you can

    enable “kubectl alpha debug” to use ephemeral containers kubectl alpha debug -it ephemeral-demo --image=busybox --target=ephemeral-demo # Or use kubectl-debug Ref: https://github.com/aylei/kubectl-debug
  31. Running Pods Level Debugging

  32. Environment • OpenShift v4.4.4 • Kubernetes v1.17.1 Ref: https://k9scli.io/

  33. How to obtain or make a debug container?

  34. Make Your Company-Wide Debug Container • Super easy! It just

    put some network diagnostic tools into a container • Or get the container from netshoot: a Docker + Kubernetes network trouble-shooting swiss-army container Ref: https://github.com/pichuang/debug-container
  35. References 1. GitHub - nicolaka/netshoot 2. Container Bare Metal for

    Networking 3. A Guide to the Kubernetes Networking Model 4. Quay - pichuang/debug-container 5. Troubleshooting from Container to Any 6. Linux Containers the Hard Way
  36. Q&A Tifa: 是否加入 Cloud Native Taiwan User Group ? You:

    [Y/y]