20240425 Play and Discuss the game “K8S LAN Party”

Slide: https://docs.google.com/presentation/d/1jdLs6vGI4vy-Vq8DX5RRtebuG2OlEDVPJbW-XNsuXW0/edit?usp=sharing

Transcript

1. Play and Discuss the game “K8S LAN Party” Cloud Native Taiwan User Group 2024/04 Phil Huang <pichuang@cloudnative.tw> CNCF Ambassador / Senior Cloud Solution Architect, Microsoft 2024/04/25
2. 2 KCD TAIPEI 2024 CALL FOR PAPER 1. Push Yourself 2. Both Mandarin and English are acceptable 3. Linux Foundation will give you a credly badge for recognize https://pretalx.coscup.org/coscup-2024/ 1. Push Yourself 2. Both Mandarin and English are acceptable 3. Linux Foundation will give you a credly badge for recognize 1. Push Yourself 2. Both Mandarin and English are acceptable 3. Linux Foundation will give you a credly badge for recognize 1. Push Yourself 2. Both Mandarin and English are acceptable 3. Linux Foundation will give you a credly badge for recognize 1. Push Yourself 2. Both Mandarin and English are acceptable 3. Linux Foundation will give you a credly badge for recognize CFP DEADLINE: 2024/05/09
4. 4 Cloud Native Taiwan User Group Meetup CFP 1. Build your digital profile 2. Both Mandarin or English are acceptable 3. Both online or offline are acceptable https://sessionize.com/cntug-meetup CFP DEADLINE: Anytime
6. 6 • EVENT DATE: 2024/04/26 https://sre.ithome.com.tw/2024/workshop-page/2531 iThome SRE CONFERENCE 2024
7. 7 https://k8slanparty.com/
8. 8 You Are Here
9. Challenge 1: Recon K8S LAN PARTY 9
10. 10 0. Deploy Debug-Container into Kubernetes pichuang/debug-container $ kubectl run -n default debug-container --restart=Never --rm -i --tty --image ghcr.io/pichuang/debug-container:master -- /bin/bash You Are Here
11. 11 1. Possible Range of Service IPs Kubernetes Internal Service Discovery 20230615 Kubernetes Scalable Workloads $ env | grep KUBERNETES • Because KUBERNETES_SERVICE_HOST = 10.245.0.1 • And, k8s service ip netmask are mostly /16 or /18 • The possible service ip subnet should be 10.245.0.0/16
12. 12 [TIPS] Allocate Cluster IP to Service How can you avoid Service ClusterIP conflicts?
13. 13 2. Reverse DNS Scanning to identify live Kubernetes Kubernetes Internal Service Discovery $ nmap -oG dns_scan_svc_2 -sn -Pn -R 10.245.0.0/16
14. 14 3. Get Result…? Not exactly 為什麼我佈署的 Kubernetes 服務不會動!? 個人除錯思路分享 - 魂系架構 Phil's Workspace (pichuang.com.tw) $ curl svc-ipconfig-io.ns-ipconfig • Most service are running port 80 or 443, so it’s pretty easy to guess the port
15. 15 4. Port Scanning to each Kubernetes Service $ nmap -p- svc-ipconfig-io.ns-ipconfig.svc -p- = scan ports from 1 ~ 65535
16. 16 5. Deep Port Scanning $ nmap -p80 -sC -sV svc-ipconfig-io.ns-ipconfig.svc -sC= Use script “default” -sV= Version Detection
17. 17 6. Get Result! 為什麼我佈署的 Kubernetes 服務不會動!? 個人除錯思路分享 - 魂系架構 Phil's Workspace (pichuang.com.tw) $ curl svc-ipconfig-io.ns-ipconfig:80
18. How to Solve? CHALLENGE 1: RECON
19. 19 Use Network Policy Network Policy Editor for Kubernetes
20. 20 Choose “Limit traffic to an application” https://github.com/ahmetb/kubernetes-network-policy-recipes/blob/master/02-limit-traffic-to-an-application.md
21. 21 Control Ingress Traffic
22. Challenge 2: FINDING NEIGHBOURS K8S LAN PARTY 22
23. 23 Are Here
24. 24 Debugging with an ephemeral debug container $ kubectl debug -n kube-system -it coredns-c499c8d7d-xx52r --image=ghcr.io/pichuang/debug-container:master --target=coredns Yes. it can share the same net namespace using kubectl debug --target Debugging with an ephemeral debug container 當遇到 Distroless Container 除錯要什麼沒什麼該怎麼辦 ? 你的好朋友 kubectl debug
25. 25 Kubectl debug Architecture https://medium.com/@simardeep.oberoi/the-ephemeral-containers-in-kubernetes-31d1f1d47bcd
26. 26 Debugging using a copy of the Pod $ kubectl debug -n kube-system -it coredns-c499c8d7d-xx52r --image=ghcr.io/pichuang/debug-container:master --copy-to=new-coredns --share-processes Not applicable, as this is adding a new Pod, not utilizing an existing one Debugging using a copy of the Pod 當遇到 Distroless Container 除錯要什麼沒什麼該怎麼辦 ? 你的好朋友 kubectl debug
27. Challenge 4: Bypassing Boundaries K8S LAN PARTY 27
28. 28 0. Deploy Debug-Container into Kubernetes Node pichuang/debug-container $ kubectl debug node/aks-nodepool1-20385824-vmss000000 -it --image ghcr.io/pichuang/debug-container:master -- /bin/bash $ chroot /host /bin/bash You Are Here
29. 29 1. Get Container Pid pichuang/debug-container $ crictl ps --name httpbin $ crictl inspect --output go-template --template ‘{{.info.pid}}’
30. 30 2. Access Container Inside pichuang/debug-container $ lsns -p 32327
31. 31 3. Inspect the namespace with nsenter pichuang/debug-container $ nsenter -a -t 32327 -n ip a Operation not permitted, but do not know why
32. Thank you Body copy here is left aligned and set to 8pt. Keep body copy to a maximum of 2 lines on this page to not detract from the title and important information above