Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How do I troubleshooting on container, more than docker [OpenShift Special Edition]

How do I troubleshooting on container, more than docker [OpenShift Special Edition]

#CNTUG #Taiwan #podman #docker #kubernetes #debug #troubleshooting

Red Hat Taiwan Facebook


Phil Huang

June 12, 2020

More Decks by Phil Huang

Other Decks in Technology


  1. How do I Troubleshooting on Container, more than Docker? Phil

    Huang <[email protected]> Solution Architect, Red Hat 2020/06/12 Special Edition
  2. # whois Phil Huang 黃秉鈞 • Red Hat HK/TW Solution

    Architect ◦ Ansible IT Automation ◦ OpenShift Container Platform ◦ Software-Defined Networking (SDN) ◦ Network Function Virtualization (NFV) • OpenSource Community Member ◦ Cloud Native Taiwan User Group (CNTUG) ◦ SDNDS-TW • Personal Blog ◦ https://blog.pichuang.com.tw Ref: https://www.linkedin.com/in/phil-huang-09b09895/
  3. Agenda • Prologue ◦ Container is Linux ◦ Why Podman

    ◦ Why Network Namespace is Important? • Container? Docker? ◦ Understanding of Container Networking Model/Interface ◦ 3 Useful container network mode for troubleshooting • Pod? Kubernetes? OpenShift? ◦ Understanding of 5 OpenShift Network Traffic ◦ How to do OpenShift network troubleshooting? • How to obtain or get a debug container?
  4. Prologue

  5. Containers are Linux Ref: http://www.brendangregg.com/linuxperf.html https://www.redhat.com/en/blog/containers-are-linux • A linux container

    is nothing more than a process that runs on Linux ◦ Linux namespaces ◦ cgroups ◦ SELinux ◦ seccomp • It also mean you can use Linux performance tools to do some troubleshooting works
  6. Why Podman? • Support multiple image formats including the OCI

    and Docker image formats • 3 Benefits ◦ Daemonless container engine ◦ Provides a familiar command experience compatible with the Docker CLI ◦ Build and run rootless containers as non-root • How to start? ◦ dnf install -y podman ◦ alias docker=podman Ref: https://speakerdeck.com/pichuang/the-first-journey-from-docker-to-podman
  7. Why Network Namespace is Important? • Container uses many Linux

    namespace technologies for isolation resource, such as user namspace / process / mnt / net ... • For network isolation, container uses Linux network namespace technology • Each network namespace can have its own: ◦ Network interface ◦ Routing tables ◦ Firewall rules ◦ DNS lookup ◦ IP address ◦ Subnets ◦ ... Ref: http://redhatgov.io/workshops/containers_the_hard_way/
  8. Docker ? Container ? • Docker Registry => Container Registry

    • Docker Images => Container Images • Docker Containers => OCI Containers • Dockerfile => Containerfile Ref: https://dwalsh.fedorapeople.org/ReplacingDockerWithPodman.pdf OCI: Open Container Initiative
  9. Container Networking Interface • Why need container networking? ◦ Allow

    containers to communicate to host machine ◦ Containers need to talk to Internet ◦ Containers can attach to multi networks • Explore the nature of communication between container resource, instead of focusing on the implementation details for specific container networking standards ◦ Docker use Container Network Model (CNM) ◦ Podman use Container Network Interface (CNI) • 3 Useful container network mode for troubleshooting ◦ Bridge mode ◦ Container mode ◦ Host mode Ref: https://www.nuagenetworks.net/blog/container-networking-standards/
  10. 3 Useful Container Network Interface for Troubleshooting

  11. $ man podman run • Implement 7 network modes •

    By default, use bridge mode
  12. Bridge Mode # podman run -it quay.io/pichuang/debug-container • Create a

    network namespace on the default bridge
  13. Container Mode # podman run -it --net container:<container_name> quay.io/pichuang/debug-container •

    Reuse another container’s network namespace
  14. Host Mode # podman run -it --net host quay.io/pichuang/debug-container •

    Use host network namespace
  15. Kubernetes Networking Model • Kubernetes project DOES NOT HAVE a

    network model default implementation • There are multiple implementations in the world, all of which must follow CNI spec ◦ The most popular CNI plugins in community: Flannel and Calico • Explore the nature of communication between Kubernetes resource, instead of focusing on the implementation details for each of CNI plugins • 5 Kubernetes network traffic • 4 level debugging methods CNI: Container Network Interface
  16. Enterprise Grade Kubernetes Platform

  17. Undestanding of 5 Kubernetes Network Traffic

  18. Basic Concept: 1 ~ n Containers per Pod • Pods

    are the smallest deployable units in Kubernetes • A Pod is a group of one or more containers with shared storage&network
  19. Traffic Model: Container to Container • A network namespace provides

    a new network stack for all the containers per Pod • Containers within a Pod share an IP Address and port space
  20. Traffic Model: Pod-to-Pod in the same node • Connecting namespaces

    using a linux bridge • Every Pod gets its own IP address in a flat shared networking space
  21. Traffic Model: Pod-to-Pod across different nodes • Kubernetes uses iptables

    to handle many networking and port forwarding rules ◦ iptables ◦ routing tables
  22. Traffic Model: Pod-to-Service • Kubernetes Service ◦ Internal load balancer

    that routes the traffic to Pods • When load balancing for multiple backend pods, it uses unweighted round-robin scheduling
  23. Traffic Model: Service-to-Internet • Ingress ◦ Routing Internet traffic to

    Kubernetes • Egress ◦ Routing traffic to the Internet
  24. How to do OpenShift Network Troubleshooting?

  25. Running Container Level Debugging # Get a Shell to a

    Running Container oc rsh welcome-1-xqbm9 -- /bin/bash Ref: https://kubernetes.io/docs/tasks/debug-application-cluster/get-shell-running-container/
  26. Running Container Level Debugging

  27. Namespace Level Debugging # Running one Pod in namespace and

    specific node oc run ocp-debug-container --image quay.io/pichuang/debug-container --restart=Never -it --attach --rm --overrides='{ "apiVersion": "v1", "spec": { "nodeSelector":{"kubernetes.io/hostname":"compute-1"}}}' Debug Pod Namespace
  28. Namespace Level Debugging

  29. Node Level Debugging # Running one Pod on specific Node

    oc run ocp-debug-container --image quay.io/pichuang/debug-container --restart=Never -it --attach --rm --overrides='{ "apiVersion": "v1", "spec": { "nodeSelector":{"kubernetes.io/hostname":"compute-1"}, "hostNetwork": true}}' Debug Pod Container OS
  30. Node Level Debugging

  31. Running Pods Level Debugging # Cloning specific Pod and exec

    it oc debug pod/productpage-v1-597b74b4c-xzf92 Ref: https://blog.pichuang.com.tw/20200521-openshift-with-coreos-part-7/
  32. Running Pods Level Debugging

  33. Environment • OpenShift v4.4.4 • Kubernetes v1.17.1 Ref: https://k9scli.io/

  34. How to obtain or make a debug container?

  35. Make Your Company-Wide Debug Container • Pulling the registry.redhat.io/rhel8/support-tools from

    Red Hat • Or get the container from netshoot: a Docker + Kubernetes network trouble-shooting swiss-army container Ref: https://github.com/pichuang/debug-container
  36. References 1. GitHub - nicolaka/netshoot 2. Container Bare Metal for

    Networking 3. A Guide to the Kubernetes Networking Model 4. Quay - pichuang/debug-container 5. Troubleshooting from Container to Any 6. Linux Containers the Hard Way
  37. None