Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Unmask the ghouls that lurk in your cluster

Unmask the ghouls that lurk in your cluster

Presented at Software Circus, Halloween 2020, with Chris Kranz of Sysdig.


Andy Randall

October 29, 2020


  1. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus Unmask the ghouls that

    lurk in your cluster, with the supernatural powers of Chris Kranz, Sysdig | Andy Randall, Kinvolk @ckranz @sysdig @falco_org @andrew_randall @kinvolkio
  2. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus Double, double pods in

    trouble; Fire burn and cluster bubble. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus – William Shakespeare, “Macbeth”
  3. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus Adoption increases attention, from

    all parties! IDC Tech Brief suggests security is the #1 challenge facing DevOps as you scale Kubernetes Jan 2019 Feb 2019 Apr 2019 Aug 2019 Mar 2019 Jun 2019 Oct 2019 Apr 2020 Jun 2020 Jul 2020 Jul 2020 Kubernetes dashboard vulnerability Container runtime vulnerability New vulnerabilities discovered in Envoy Severe Kubernetes HTTP/2 Vulnerability Kubernetes dashboard vulnerability kubectl cp vulnerability Kubernetes API server DoS vulnerability Vulnerability discovered in kube-proxy Kube-controller-manager vulnerable to a Server Side Request Forgery (SSRF) Vulnerability that allows man-in-the-middle (MitM) attacks kube-apiserver vulnerability that can lead to privilege escalation
  4. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus Initial Access - Patient

    zero - The Rite of AshkEnte - Using Cloud credentials - Compromised images in registry - Kubeconfig file - Application vulnerability - Exposed dashboard Occult threat techniques of the underworld @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus Persistence - Summon Poltergeist - Exhume, then rebury upside-down - Backdoor container - Writeable hostPath mount - Kubernetes CronJob Impact - Witch burning - Ritual sacrifice - Data destruction - Resource hijacking - Denial of service Execution - Remove the head - Stake through the heart - Exec into container - bash/cmd inside container - New container - Application explore (RCE) - SSH server inside container Privilege Escalation - Demonic possession - Privileged container - Cluster-admin binding - hostPath mount - Access cloud resources Defense Evasion - Body snatching - Clear container logs - Delete events - Pod / container name similarity - Connect from proxy Credential Access - Trick or Treat - List K8s secret - Mount service - Access contain - Credentials in Lateral Movement - Infected zombie bite - Access cloud resources - Container service account - Cluster internal networking - Credentials in configuration files - Writeable volume mounts on the host - Kubernetes dashboard - Tiller endpoint Discovery - Kids entering an abandoned warehouse - Access K8s API server - Access Kubelet API - Network mapping - Kubernetes dashboard - Instance metadata API - Delete events - Pod / container name sim - Connect from proxy Access s principal er service account configuration files
  5. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus What kind of ghouls

    would attack your cluster? Cryptomining: 95% Denial of Service: 5% (Source: Aqua 2020 Cloud Native Threat Report) @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus
  6. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus The ghost of clusters

    past HERE LIES A. Pod “Ran as Root” Lateral movement Privilege escalation RIP M.Y. Cluster “Succumbed to a critical vulnerability” Exposed management interface @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus (* Source: Sysdig 2019 Container Usage Report) >50% of images have critical or high sev vulnerabilities* >50% of images run as root*
  7. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus What ya gonna do?

    @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus
  8. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus @andrew_randall @kinvolkio @ckranz @sysdig

    @falco_org @softwarecircus
  9. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus Why syscalls? When you

    run a program you are making system calls. System calls are how a program enters the kernel to perform some tasks. • Processes • Network • File I/O • … and much more... @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus Applications Kubernetes Operating System Kernel Where the magic happens
  10. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus The supernatural powers of

    Custom programs that run in the Linux kernel Safe virtual machine with restricted functionality (& code verification) extended Berkeley Packet Filter Hooks, functions and data structures (maps)
  11. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus https://ebpf.io/

  12. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus Debugging / performance analysis

    Application monitoring and security Fast, customizable networking Why do you care?
  13. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus eBPF is hard to

    use directly @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus WARNING: !!! TRICKY KERNEL CODE !!!
  14. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus @andrew_randall @kinvolkio @ckranz @sysdig

    @falco_org @softwarecircus
  15. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus Falco, the syscall Wizard

    Purgatory (aka: kernel space) Land of the living (aka: user space) eBPF maps eBPF probe @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus eBPF maps
  16. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus

  17. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus Inspektor Gadget A “Swiss

    Army knife” collection of various BPF tools (gadgets) Integrated with Kubernetes: kubectl gadget Select pods across the cluster based on labels
  18. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus Ghostbusting Gadgets profile network

    policy advisor traceloop tcptop tcptracer opensnoop execsnoop bindsnoop capabilities kubectl-gadget network policy advisor opensnoop execsnoop bindsnoop
  19. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus

  20. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus https://falco.org https://falco.org/docs/installation/ https://github.com/falcosecurity/falco https://github.com/kinvolk/inspektor-gadget

    Claim a haunted mystery box from Sysdig! https://go.sysdig.com/cloudstreet.html
  21. @andrew_randall @kinvolkio @ckranz @sysdig @falco_org @softwarecircus This ghoulish fun was

    brought to you by Software Circus, If you liked it... please tweet about it, visit our websites and buy our products. If you didn’t... then Frank here would like a word. and