Sk8ing on Thin Ice - Crash Course in Kubernetes & Security

Kubernetes is Google's answer to container orchestration and some of the tools it provides developers are indistinguishable from black magic. However, with the power that it provides it also can let you fall into some security holes that are hard to climb out of.
In this presentation we'll go through those pitfalls, along with some huge security benefits that using such a fluid containerized environment can provide you. Learn lessons from the head of security of a recently acquired financial tech startup who is all-in on Kubernetes and has the bruises and badges of honor to show for it.
For a DevOps practitioner or enthusiast this will allow you to walk a way with a better understanding of Kubernetes as a whole, and how to secure it right from the ground up. Learn to monitor your infrastructure properly for malicious behavior, how to respond to incidents, and some simple best practice security hygiene.

Transcript

1. 1 The retirement benefit that benefits everyone Matt Johansen Director of Security Honest Dollar @mattjay The retirement benefit that benefits everyone A Simple, Portable & Affordable Savings Solution May 2016 Sk8ing on Thin Ice: A Crash Course in Kubernetes & Security
2. 2 The retirement benefit that benefits everyone Honest Dollar is a company dedicated to helping people save in a society built on spending
3. 3 The retirement benefit that benefits everyone Matt Johansen - Security Guy - Wannabe Developer - SXSW, BlackHat, DEFCON, RSA, more++
4. 4 The retirement benefit that benefits everyone Agenda Security & Kubernetes What is it? | What is Kubernetes anyway? (This portion will not include how to pronounce it) Why use it? | “Because $Cool_Unicorn is using it” is not acceptable. Security Pitfalls | Trust me. There are holes to fall into. Security Benefits | You mean there are benefits to containerization outside of DevOps? Examples | No Live Demos™ - But lets walk through some cool security monitoring, logging, alerting, and other tricks. Along with some basic security hygiene.
5. 5 The retirement benefit that benefits everyone Kube Kubernetes
6. 6 The retirement benefit that benefits everyone Kube Kubernetes Source: Imesh Gunaratne - Intro to Kubernetes
7. 7 The retirement benefit that benefits everyone Kubernetes Things to keep in mind YAML Land - Config Management is hard Different Environments - We’re running a company here. Prod, Dev, QA, etc. need processes. Persistence (and not) - Pods die. Its okay (and sometimes awesome). Manage your data intelligently. Secret Sauce - Docker is not the magic. Orchestration is the magic. Learn and love your Services, RCs, and Pods. Kube Words I might say Node - Worker VMs. Pod - Group of Containers. Replication Controller - Manages Pod lifecycle & config. Service - Config for a set of Pods. Cluster - Single network & group of Nodes.
8. 8 The retirement benefit that benefits everyone Pitfalls
9. 9 The retirement benefit that benefits everyone Cluster
10. 10 The retirement benefit that benefits everyone Cluster
11. 11 The retirement benefit that benefits everyone Cluster Bad Idea Nearly identical YAMLs. Duplication of work and error prone.
12. 12 The retirement benefit that benefits everyone Cluster Separation of Concerns
13. 13 The retirement benefit that benefits everyone Environment Stability Cluster Options • Puppet (kinda) • Shell scripts + templates • rakefiles + templates (Us Currently) Lessons Learned • Launch as much as possible, with the fewest commands possible. • Track versions and automate container building if possible. • Automate testing / build pipeline to watch out for regressing over hot fixes. • Track versions (and dependencies!) between environments
14. 14 The retirement benefit that benefits everyone IP Load Balancing Watch your logs if you do this. You’ll lose origin IP. Solution: Break nginx out of cluster (I know, I know)
15. 15 The retirement benefit that benefits everyone Loophole kubectl Whats wrong with this picture?
16. 16 The retirement benefit that benefits everyone Benefits
17. 17 The retirement benefit that benefits everyone Security Benefits Security Containerization • Patching - Oh you mean `docker build`? • One process per container - Less vulnerability surface for stuff you don’t use. • Mobility - Host agnostic • Segregation - Less pivot surface • Typically short lifespan - Less old, stale, vulnerable systems. Kubernetes • Upgrade process - Bring nodes down, bring new ones up. Pods find their way to a new home. • Log granularity - Know and prioritize your log events. • Build pipeline and web hook friendly.* • Rate of change it enables is incredibly fast. • Transient by nature. *Continuous Integration is no longer a ‘nice to have’. For security it is a must.
18. 18 The retirement benefit that benefits everyone Fix Fast Continuous Integration Good luck with you two week release cycle. Source: Verizon DBIR - 2016
19. 19 The retirement benefit that benefits everyone Watch Everything Monitoring Log everything. Watch your logs. Source: Verizon DBIR - 2016
20. 20 The retirement benefit that benefits everyone Monitoring Log all the things Logging • ElasticSearch • rsyslog + RELP • auditd (go-audit) • OSQuery • nginx logs • Docker logs Alerting • ElastAlert • Slack bots • SSH Connections • Suspicious commands (curl out of a prod server?) • File watch • Anomalies (Geo, IP, data movement, spikes, etc.)
21. 21 The retirement benefit that benefits everyone Matt Johansen Director of Security Honest Dollar @mattjay The retirement benefit that benefits everyone A Simple, Portable & Affordable Savings Solution May 2016 Thank You