Kawaiicon: Kubernetes, it's no longer cool, but let's secure it anyway!

You work for a tech company, you must have at least one Mandatory Kuberenetes™ right? Wouldn't it be rad if you turned on a bunch of security options for it? But without it sucking and everything breaking, or having to get developers to do a bunch of work they have no real desire to? What about if we had fun on the way too? (you know, within reason, this isn't Mr. Toad's Wild Ride). This won't be "now type this SELinux policy list in to a terminal" in an impossible to read font talk, it won't be a "Buy this collection of buzzwords and tools you don't need" talk either. Learn some things about getting your security from zero to some, instead of the typical mythical industry goal of 98% to 99%.

Transcript

1. Kubernetes, it's no longer cool , * but let's secure it anyway! *"Cool" in the haha tech business sense 1 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
2. Who's this clown? 2 → Some Security Engineer at Color. → Various security at PagerDuty, Stripe, Etsy, Puppet. → Spoke at Bsides Wellington, November 2017 around the corner from here. → Two time sponsor of Wrong Island Con. 2 2 — Kawaaaaaiiiii https://twitter.com/skullmandible/status/411281851131523072 ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
3. The Shape of Things to Come™ → A Brief, 3 hour intro to Kuberninis (not really) → Rants about marketing → Things you can actually do, that's really just UNIX → Fun silly demos of derring-do 3 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
4. This talk isn't → Telling you to buy/install anything → Legal advice → "Simply deploy this Herculean 12 month project across your entire org in one command"* (service mesh, admission controller, full eBPF packet capture, a safe ingress controller.) 4 — This talk contains a chemical known to the State of California to cause...
5. Kubernetes (hence forth referred to as kuberninis) 5 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
6. Kuberninis™
7. archive.org/details/pdp-10-reference-handbook-dec-1970
8. Kuberninis Architecture (but for security) 8 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
9. So, let's start some fights! 9 — © Hbomberman 2024
12. Containers Are No Longer a Security Boundary — Unnamed AI security company graciously selling you something 12 — depthfirst.com/research/containers-are-no-longer-safe
13. 13 — Dr. Raid et al us-19-Edwards-Compendium-Of-Container-Escapes-up.pdf
14. "Guess what you get control of when you get control of the kernel?" — Hal from EM , posing as an AI security vendor C 14 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
15. "Orgs should move sensitive & untrusted workloads to Firecracker or Kata" — Someone who isn't aware companies are still moving to Kuberninis 15 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
16. Hey remember when UniKernels took over everything? — Me, laughing at something that didn't happen 16 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
17. Why do you want root/kernel/EIP? → lateral movement → persistence → better detection evading 17 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
18. 18 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
19. Why do you want root? → lateral movement if you have buckets and database access, what else do you want? → persistence nodes (not only pods) can only last a few hours → better detection evading kernel attacks might be louder than PHP 19 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
20. SecComp "short for secure computing"" > sysctl kernel.hax.pls.do.a.secure_computerising=1 20 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
21. # The format is: # <number> <abi> <name> <entry point> [<compat entry point> [noreturn]] # # The __x64_sys_*() stubs are created on-the-fly for sys_*() system calls 0 common read sys_read 1 common write sys_write 2 common open sys_open 3 common close sys_close 4 common stat sys_newstat 5 common fstat sys_newfstat 6 common lstat sys_newlstat 7 common poll sys_poll 8 common lseek sys_lseek 9 common mmap sys_mmap 10 common mprotect sys_mprotect 11 common munmap sys_munmap 12 common brk sys_brk [..SNIP..] 543 x32 io_setup compat_sys_io_setup 544 x32 io_submit compat_sys_io_submit 545 x32 execveat compat_sys_execveat 546 x32 preadv2 compat_sys_preadv64v2 547 x32 pwritev2 compat_sys_pwritev64v2 thank you to filippo.io/linux-syscall-table/
22. Case Study: IO uring IO uring, a way of doing IO much faster, using circular buffers. Interface for storage device async I/O operations. Addresses performance issues with similar interfaces provided by functions like read()/ write()... 22 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
23. Google reported 60% of the exploits to their bug bounty program where io_uring 23 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
24. Pretend to be Google > cat iouring_seccomp.json { "defaultAction": "SCMP_ACT_ALLOW", "syscalls": [ { "names": [ "io_uring_setup", "io_uring_enter", "io_uring_register" ], "action": "SCMP_ACT_ERRNO" } ] } 24 — Kawaaaaaiiiii ! infosec.exchange/@bea | > cat some_k8s_config.yaml | containers: | - name: container | securityContext: | seccompProfile: | type: Localhost | localhostProfile: iouring_seccomp.json | | | | | | | ! blog (oh no it was on php, it ded) barnbarn !
25. 25 — github.com/containerd/containerd/pull/9320
26. So if you can, just can do this securityContext: seccompProfile: type: RuntimeDefault 26 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
27. KSPM on the cheap (I had to look up what KSPM was too, don't worry) 27 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
28. CRYPTO MINERS ZOMG 28 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
29. How do we fight this? → look for containers called xmrig → look for something using all your CPU! → it's DNS, it's always DNS* (*DNS is amazing and anyone who bad mouths it is wrong) 29 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
31. > for tehcryptos in bitcoin \ zcash \ dogecoin \ monero \ bitcoincash \ litecoin \ ethereumclassic ; do curl -s \ "https://data.miningpoolstats.stream/data/${tehcryptos}.js?t=4" \ | tr -d '\\' \ | yq -P '.' >"${tehcryptos}.yaml" done 31 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
32. > for i in *.yaml ; do yq -P <"$i" '.data | .[].url' \ | gsed -E -e 's/www\.//' \ -e 's/^http[s]?:\///' \ | cut -f 2 -d '/' done ; \ | sort -u \ | tee pool_domains.txt 32 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
33. > head pool_domains.txt 0x.al 1miner.net 21pool.io 666pool.com alphablockmonero.xyz amkira.network antpool.com asicpool.space atlaspool.io ausolo.ckpool.org 33 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
34. > ; ; ; ; ; ; ; typeset -A p ; typeset -a rd ; \ d=(${(f)"$(<pool_domains.txt)"}) \ for i in ${d//:*/}; do t=$i:e ; \ e=(${(s:|:)${p[$t]:-}}) \ e+=($i:r) ; p[$t]="${(j:|:)e}" \ done ; for t in ${(k)p}; do \ rd+=("\(\(${p[$t]}\)\\\\.${t}\)") \ done ; print "${(j:|:)rd}\\\\.\$" Okay it is now 2am, I should stop writing zsh that no one can read.
36. Australian DNS (Forgive me this bad joke, I have taken leave of my senses) 36 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
37. CoreDNS 37 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
38. apiVersion: v1 kind: ConfigMap metadata: name: coredns-custom namespace: kube-system data: a-cry-for-help.override: | template IN ANY { match "<GIANT CURSED REGEX, WHY GOD WHY>" answer "{{ .Name }} 60 IN CNAME oh-no-my-apes.allgone.swagdollars" fallthrough } template IN ANY allgone.swagdollars { rcode NXDOMAIN fallthrough } log oh-no-my-apes.allgone.swagdollars \ "SKETCHY: Cryptos {proto} Request: {name} {type} from {remote}" 38 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
39. template IN ANY { match "<GIANT CURSED REGEX, WHY GOD WHY>" answer "{{ .Name }} 60 IN CNAME oh-no-my-apes.allgone.swagdollars" fallthrough } 39 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
40. template IN ANY allgone.swagdollars { rcode NXDOMAIN fallthrough } log oh-no-my-apes.allgone.swagdollars \ "SKETCHY: Cryptos {proto} Request: {name} {type} from {remote}" 40 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
41. "Bea you idiot, we can't afford DataDog Splunk <we are vendor shade neurtral>" 41 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
42. <demo> 42 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
43. 43 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
44. > ~/Code/KawaiiK8s/dns_killah2.zsh [*] A wild DNS logline appears. [INFO] SKETCHY: Cryptos udp Request: oh-no-my-apes.allgone.swagdollars. AAAA from 10.42.0.69 [!] Killing really-not-xmrig-in-a-trenchcoat in kawaiicon. pod "definitely-not-xmrig-nope" deleted from kawaiicon namespace [*] A wild DNS logline appears. [INFO] SKETCHY: Cryptos udp Request: oh-no-my-apes.allgone.swagdollars. A from 10.42.0.72 [!] Killing definitely-not-xmrig-promise in kawaiicon. pod "definitely-not-xmrig-nope" deleted from kawaiicon namespace 44 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
45. kubectl logs -l k8s-app=kube-dns -n kube-system --tail=0 --follow \ | grep --fixed --line-buffered 'SKETCHY: Cryptos' \ | while read -r line ; do echo '[*] A wild DNS logline appears.' if podns=(${(s:#:)"$( kubectl get pods -A -o wide \ | gawk "/ ${line/* /} / "'{print $2 "#" $1}')"}); then pod="${podns[1]}" ; ns="${podns[2]}" done fi echo >&2 "[!] Killing $pod in $ns." kubectl -n "$ns" delete pod --wait=false --now=true "$pod" gitlab.com/barnbarn/kawaiik8s/-/blob/main/dns_killah2.zsh
47. Container (or everything) security 1. Reduce attack surface area 2. CompartmentaliSe 3. "Principle of least Privilege™" (don't be root) 4. Pay $100,000s a year for <Vendor name deleted> 47 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
48. Everyone's (Docker|Container)file journey 1. Basically a copy of a VM/EC2 instance. 20GiB Horrorshow 2. Ubuntu image where you're still shipping gcc to prod. 3. Multi-stage build of Ubuntu. Build artefacts copied over! 4. That Alpine era, where you convince yourself musl is fine. 5. Distroless, the promised land. Sure buddy, w/e! 48 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
49. One way, remove the shells > kubectl exec -it deployments/coredns -- /bin/sh error: Internal error occurred: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "000000...000004": OCI runtime exec failed: exec failed: unable to start container process: exec: "/bin/sh": stat /bin/sh: no such file or directory 49 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
50. # syntax=docker/dockerfile:1 FROM nginx:mainline-trixie RUN apt remove \ --yes \ --allow-remove-essential \ bash \ dash \ || true 50 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
51. Perhaps too secure? > docker run -it --rm barn/nginx_sh exec /docker-entrypoint.sh: no such file or directory Secure? yes Are you getting paid? no 51 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
52. # syntax=docker/dockerfile:1 FROM nginx:mainline-trixie RUN sed -i \ 's/^\(exec "$@"\)/apt remove'\ ' --yes --allow-remove-essential'\ ' bash dash || true\n\1/' \ /docker-entrypoint.sh 52 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
53. pretend kubectl exec is some amazing 0day (please pretend, it really helps my talk) > kubectl exec -it nginx -- /usr/bin/perl print `id`; uid=0(root) gid=0(root) groups=0(root) 53 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
54. Is this security through obscurity? **Audible gasp** 54 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
55. 55 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
56. Wouldn't it be neat to automatically kill pods on exploitation? 56 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
57. Introducing hermit-sh Hermit sh (crab, because Rust) is a revolutionary new shell that doesn't run any commands. Instead, it crab walks the PID tree, killing everything it finds. 57 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
59. <demo 2> 59 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
60. 60 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
61. ! > kubectl exec -ti some-cool-application -- /bin/sh bye bye init "tini" found a pid of 7 with process of "sleep", its going too! found a pid of 8 with process of "sh", its going too! command terminated with exit code 143 > kubectl describe pods | awk '/State:/,/Ready:/' State: Running Started: Sun, 25 Oct 2026 01:00:12 -0700 Last State: Terminated Reason: Error Exit Code: 143 61 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
62. Conclusion! → We're still trying to secure UNIX from 60 years ago by tacking things on. → Kuberentes sig-security do keep adding useful powerful stuff → You don't just have to add Itsio/OPA/Rego/W*z. → DNS monitoring and attack surface reduction still apply! 62 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !
63. Thanks! Links to code and hermit-sh at gitlab.com/barnbarn Thank you to indirect for putting up with my jj and zsh horrors, ixi for "how do computers work?", Percy for the moral support, Ben for the Pi, mango for the clicker, The Crüe for a great conference, you all for coming and not throwing rotten fruit at me, my work for letting me come here to speak. 63 — Kawaaaaaiiiii ! infosec.exchange/@bea ! blog (oh no it was on php, it ded) barnbarn !