Tweet
Tweet

Slide 1

Slide 1 text

Play and Discuss the game “K8S LAN Party” Cloud Native Taiwan User Group 2024/04 Phil Huang CNCF Ambassador / Senior Cloud Solution Architect, Microsoft 2024/04/25

Slide 2

Slide 2 text

2 KCD TAIPEI 2024 CALL FOR PAPER 1. Push Yourself 2. Both Mandarin and English are acceptable 3. Linux Foundation will give you a credly badge for recognize https://pretalx.coscup.org/coscup-2024/ 1. Push Yourself 2. Both Mandarin and English are acceptable 3. Linux Foundation will give you a credly badge for recognize 1. Push Yourself 2. Both Mandarin and English are acceptable 3. Linux Foundation will give you a credly badge for recognize 1. Push Yourself 2. Both Mandarin and English are acceptable 3. Linux Foundation will give you a credly badge for recognize 1. Push Yourself 2. Both Mandarin and English are acceptable 3. Linux Foundation will give you a credly badge for recognize CFP DEADLINE: 2024/05/09

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

4 Cloud Native Taiwan User Group Meetup CFP 1. Build your digital profile 2. Both Mandarin or English are acceptable 3. Both online or offline are acceptable https://sessionize.com/cntug-meetup CFP DEADLINE: Anytime

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

6 ● EVENT DATE: 2024/04/26 https://sre.ithome.com.tw/2024/workshop-page/2531 iThome SRE CONFERENCE 2024

Slide 7

Slide 7 text

7 https://k8slanparty.com/

Slide 8

Slide 8 text

8 You Are Here

Slide 9

Slide 9 text

Challenge 1: Recon K8S LAN PARTY 9

Slide 10

Slide 10 text

10 0. Deploy Debug-Container into Kubernetes pichuang/debug-container $ kubectl run -n default debug-container --restart=Never --rm -i --tty --image ghcr.io/pichuang/debug-container:master -- /bin/bash You Are Here

Slide 11

Slide 11 text

11 1. Possible Range of Service IPs Kubernetes Internal Service Discovery 20230615 Kubernetes Scalable Workloads $ env | grep KUBERNETES ● Because KUBERNETES_SERVICE_HOST = 10.245.0.1 ● And, k8s service ip netmask are mostly /16 or /18 ● The possible service ip subnet should be 10.245.0.0/16

Slide 12

Slide 12 text

12 [TIPS] Allocate Cluster IP to Service How can you avoid Service ClusterIP conflicts?

Slide 13

Slide 13 text

13 2. Reverse DNS Scanning to identify live Kubernetes Kubernetes Internal Service Discovery $ nmap -oG dns_scan_svc_2 -sn -Pn -R 10.245.0.0/16

Slide 14

Slide 14 text

14 3. Get Result…? Not exactly 為什麼我佈署的 Kubernetes 服務不會動!? 個人除錯思路分享 - 魂系架構 Phil's Workspace (pichuang.com.tw) $ curl svc-ipconfig-io.ns-ipconfig ● Most service are running port 80 or 443, so it’s pretty easy to guess the port

Slide 15

Slide 15 text

15 4. Port Scanning to each Kubernetes Service $ nmap -p- svc-ipconfig-io.ns-ipconfig.svc -p- = scan ports from 1 ~ 65535

Slide 16

Slide 16 text

16 5. Deep Port Scanning $ nmap -p80 -sC -sV svc-ipconfig-io.ns-ipconfig.svc -sC= Use script “default” -sV= Version Detection

Slide 17

Slide 17 text

17 6. Get Result! 為什麼我佈署的 Kubernetes 服務不會動!? 個人除錯思路分享 - 魂系架構 Phil's Workspace (pichuang.com.tw) $ curl svc-ipconfig-io.ns-ipconfig:80

Slide 18

Slide 18 text

How to Solve? CHALLENGE 1: RECON

Slide 19

Slide 19 text

19 Use Network Policy Network Policy Editor for Kubernetes

Slide 20

Slide 20 text

20 Choose “Limit traffic to an application” https://github.com/ahmetb/kubernetes-network-policy-recipes/blob/master/02-limit-traffic-to-an-application.md

Slide 21

Slide 21 text

21 Control Ingress Traffic

Slide 22

Slide 22 text

Challenge 2: FINDING NEIGHBOURS K8S LAN PARTY 22

Slide 23

Slide 23 text

23 Are Here

Slide 24

Slide 24 text

24 Debugging with an ephemeral debug container $ kubectl debug -n kube-system -it coredns-c499c8d7d-xx52r --image=ghcr.io/pichuang/debug-container:master --target=coredns Yes. it can share the same net namespace using kubectl debug --target Debugging with an ephemeral debug container 當遇到 Distroless Container 除錯要什麼沒什麼該怎麼辦 ? 你的好朋友 kubectl debug

Slide 25

Slide 25 text

25 Kubectl debug Architecture https://medium.com/@simardeep.oberoi/the-ephemeral-containers-in-kubernetes-31d1f1d47bcd

Slide 26

Slide 26 text

26 Debugging using a copy of the Pod $ kubectl debug -n kube-system -it coredns-c499c8d7d-xx52r --image=ghcr.io/pichuang/debug-container:master --copy-to=new-coredns --share-processes Not applicable, as this is adding a new Pod, not utilizing an existing one Debugging using a copy of the Pod 當遇到 Distroless Container 除錯要什麼沒什麼該怎麼辦 ? 你的好朋友 kubectl debug

Slide 27

Slide 27 text

Challenge 4: Bypassing Boundaries K8S LAN PARTY 27

Slide 28

Slide 28 text

28 0. Deploy Debug-Container into Kubernetes Node pichuang/debug-container $ kubectl debug node/aks-nodepool1-20385824-vmss000000 -it --image ghcr.io/pichuang/debug-container:master -- /bin/bash $ chroot /host /bin/bash You Are Here

Slide 29

Slide 29 text

29 1. Get Container Pid pichuang/debug-container $ crictl ps --name httpbin $ crictl inspect --output go-template --template ‘{{.info.pid}}’

Slide 30

Slide 30 text

30 2. Access Container Inside pichuang/debug-container $ lsns -p 32327

Slide 31

Slide 31 text

31 3. Inspect the namespace with nsenter pichuang/debug-container $ nsenter -a -t 32327 -n ip a Operation not permitted, but do not know why

Slide 32

Slide 32 text

Thank you Body copy here is left aligned and set to 8pt. Keep body copy to a maximum of 2 lines on this page to not detract from the title and important information above