Unmask the ghouls that lurk in your cluster

Presented at Software Circus, Halloween 2020, with Chris Kranz of Sysdig.

1. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus Unmask the ghouls that lurk in your cluster, with the supernatural powers of Chris Kranz, Sysdig | Andy Randall, Kinvolk @ckranz @sysdig @falco_org @andrew_randall @kinvolkio
2. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus Double, double pods in trouble; Fire burn and cluster bubble. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus – William Shakespeare, “Macbeth”
3. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus Adoption increases attention, from all parties! IDC Tech Brief suggests security is the #1 challenge facing DevOps as you scale Kubernetes Jan 2019 Feb 2019 Apr 2019 Aug 2019 Mar 2019 Jun 2019 Oct 2019 Apr 2020 Jun 2020 Jul 2020 Jul 2020 Kubernetes dashboard vulnerability Container runtime vulnerability New vulnerabilities discovered in Envoy Severe Kubernetes HTTP/2 Vulnerability Kubernetes dashboard vulnerability kubectl cp vulnerability Kubernetes API server DoS vulnerability Vulnerability discovered in kube-proxy Kube-controller-manager vulnerable to a Server Side Request Forgery (SSRF) Vulnerability that allows man-in-the-middle (MitM) attacks kube-apiserver vulnerability that can lead to privilege escalation
4. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus Initial Access - Patient zero - The Rite of AshkEnte - Using Cloud credentials - Compromised images in registry - Kubeconfig file - Application vulnerability - Exposed dashboard Occult threat techniques of the underworld @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus Persistence - Summon Poltergeist - Exhume, then rebury upside-down - Backdoor container - Writeable hostPath mount - Kubernetes CronJob Impact - Witch burning - Ritual sacrifice - Data destruction - Resource hijacking - Denial of service Execution - Remove the head - Stake through the heart - Exec into container - bash/cmd inside container - New container - Application explore (RCE) - SSH server inside container Privilege Escalation - Demonic possession - Privileged container - Cluster-admin binding - hostPath mount - Access cloud resources Defense Evasion - Body snatching - Clear container logs - Delete events - Pod / container name similarity - Connect from proxy Credential Access - Trick or Treat - List K8s secret - Mount service - Access contain - Credentials in Lateral Movement - Infected zombie bite - Access cloud resources - Container service account - Cluster internal networking - Credentials in configuration files - Writeable volume mounts on the host - Kubernetes dashboard - Tiller endpoint Discovery - Kids entering an abandoned warehouse - Access K8s API server - Access Kubelet API - Network mapping - Kubernetes dashboard - Instance metadata API - Delete events - Pod / container name sim - Connect from proxy Access s principal er service account configuration files
5. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus What kind of ghouls would attack your cluster? Cryptomining: 95% Denial of Service: 5% (Source: Aqua 2020 Cloud Native Threat Report) @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus
6. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus The ghost of clusters past HERE LIES A. Pod “Ran as Root” Lateral movement Privilege escalation RIP M.Y. Cluster “Succumbed to a critical vulnerability” Exposed management interface @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus (* Source: Sysdig 2019 Container Usage Report) >50% of images have critical or high sev vulnerabilities* >50% of images run as root*
7. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus What ya gonna do? @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus
8. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus
9. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus Why syscalls? When you run a program you are making system calls. System calls are how a program enters the kernel to perform some tasks. • Processes • Network • File I/O • … and much more... @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus Applications Kubernetes Operating System Kernel Where the magic happens
10. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus The supernatural powers of Custom programs that run in the Linux kernel Safe virtual machine with restricted functionality (& code verification) extended Berkeley Packet Filter Hooks, functions and data structures (maps)
11. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus https://ebpf.io/
12. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus Debugging / performance analysis Application monitoring and security Fast, customizable networking Why do you care?
13. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus eBPF is hard to use directly @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus WARNING: !!! TRICKY KERNEL CODE !!!
14. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus
15. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus Falco, the syscall Wizard Purgatory (aka: kernel space) Land of the living (aka: user space) eBPF maps eBPF probe @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus eBPF maps
16. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus
17. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus Inspektor Gadget A “Swiss Army knife” collection of various BPF tools (gadgets) Integrated with Kubernetes: kubectl gadget Select pods across the cluster based on labels
18. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus Ghostbusting Gadgets profile network policy advisor traceloop tcptop tcptracer opensnoop execsnoop bindsnoop capabilities kubectl-gadget network policy advisor opensnoop execsnoop bindsnoop
19. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus
20. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus https://falco.org https://falco.org/docs/installation/ https://github.com/falcosecurity/falco https://github.com/kinvolk/inspektor-gadget Claim a haunted mystery box from Sysdig! https://go.sysdig.com/cloudstreet.html
21. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus This ghoulish fun was brought to you by Software Circus, If you liked it... please tweet about it, visit our websites and buy our products. If you didn’t... then Frank here would like a word. and