[2018.12 Meetup] [TALK #2] Vitali Henne - How to shoot yourself in the foot using kubernetes

One of the 6 Lightning talks of December 2018 DevOps Lisbon MeetUp.

Transcript

1. How to shoot yourself in the foot using kubernetes We’ve just opened an office in Lisbon. Why, yes, we are hiring! martin.ruderer@freiheit.com
2. k8s and freiheit. A love story? We’ve been using k8s since 2015 - just before it turned 1.0. Right now, we have 655 pods and 242 deployments. Of all the technological choices we made - this is the one I regret the least. :-) martin.ruderer@freiheit.com
3. zero downtime deployments? martin.ruderer@freiheit.com
4. zero downtime deployments? Readiness probes using health endpoints! martin.ruderer@freiheit.com
5. zero downtime deployments Readiness probes: - allow k8s to check the health of a service - are considered with rolling updates: - old services will only shut down if the freshly deployed services are healthy - new services will only receive traffic if they are healthy martin.ruderer@freiheit.com
6. zero downtime deployments If it fails for an instance, that instance won’t be served by the load balancer. - Cool during the deployment. - Cool for temporary glitches (I can’t answer right now, but I’m sure I’ll be fine in a bit) - Not so cool for real problems - your probes might just have created a down-time, because none of your services is being serviced any more. martin.ruderer@freiheit.com
7. Enter:Liveness Probes - k8s can check if a pod thinks it’s alive and kill it if necessary martin.ruderer@freiheit.com
8. CPU limits - allow k8s to schedule pods on nodes so that they are not overcommitted - Processes will be throttled if they use too much martin.ruderer@freiheit.com
9. Liveness Probes + CPU limits = <3? martin.ruderer@freiheit.com
10. Liveness Probes + CPU limits CPU limits: What happens when they are violated? -> k8s/docker throttles your service. What do services often need when they start up? -> a lot of CPU. 1. Throttled at startup 2. Can not answer the liveness probe 3. Will be killed 4. go to step 1. martin.ruderer@freiheit.com
11. Shared Resources Problems: - Nodes are overcommitted - Services are stealing resources from each other Solution: - Just set limits! New Problems: - How to set limits? martin.ruderer@freiheit.com
12. Shared Resources Need: - Good monitoring and some load tests - For JVM-services: These need limits as well. - Either use the new “Container aware” flags (1.8: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap or 1.10: -XX:+UseContainerSupport) or do it manually (don’t forget to set heap space and take into account off heap (classes, threads, buffers, Jetty)) martin.ruderer@freiheit.com
13. Things fail/pods die Solution: Readiness probes using health endpoints More problems: Services know best, right? Okay. I shouldn’t depend on other services, but at least the DB should be available? -> 3 replicas, 200 services, 1 check per second: 600 requests/sec just to answer health? martin.ruderer@freiheit.com
14. Things fail/pods die Scenario 2: Your services need about 5 minutes to start up (no problem, right, you’ve got rolling updates). The health check only depends on the availability of your DB (ok, the driver is complicated.. better restart the service). You have a weird network glitch and your DB is not available for 30 secs. -> All your services are being killed and it takes them 5 minutes to come back up. -> You’ve just earned 5 minutes of downtime. (Fail fast... but not too fast) martin.ruderer@freiheit.com
15. Multiple processes per container Example: About a year ago alpine had bad DNS - needed to have its own DNS server. Now you have two processes and only one health probe, which probably only checks one process. If the other fails, everything will fail. Can fix that! Just use something to supervise these processes. Turns out, now you have a third problem. martin.ruderer@freiheit.com
16. Multiple processes per container Example: Scheduling tasks.. I’ll just add Cron to the image! Needs to be monitored in principle.. but it’s battle proven.. guess it’s fine. Logging? Yes, we write json to STDOUT. Cron? I’ll just truncate and and prefix every line... martin.ruderer@freiheit.com
17. File systems Well.. there is the host’s file system! But I might be scheduled on a different node when I die! If that’s not a problem - who does the cleanup? If that’s a problem, Node Pinning to the rescue! And the node dies.. martin.ruderer@freiheit.com