Kubernetes for 14 months

Presentation given at DockerGrunn #10

Small note: Don't use Kube-Lego but CertManager

Transcript

1. Kubernetes for 14 months
2. KUBERNETES FOR 14 MONTHS JOSHUA PEPER - VERIFAI
3. DOCKER GRUNN PROJECT DATE CLIENT 26 02 2019 JOSHUA PEPER CTO VERIFAI
8. spin-off largest online phone retailer in the netherlands
9. kubernetes
10. Why Kubernetes Hipster tech Verifai started from scratch Experience in Docker Whole dev setup in Docker Need to be able to hyper scale
11. Why all at once No infra to replace Starting phase Downtime is not a disaster Should not be able to go down by design Help from OSSO
12. CURRENT STACK
13. Current stack (running services) Django Backend Celery workers RabbitMQ Redis Galera / MariaDB Public Docker registry Website Pootle Elasticsearch (ELK) Kibana APM Review env Public docs Sentry PostgreSQL Redis Minio Backup systems
14. Whats next Android Builds TensorFlow Jobs CEPH Gitlab Private Docker registry Verifai Webservices
15. 7 HASHTAG PRO TIPS
16. #1 LIMIT YOUR RESOURCES
19. #2 START WITH MANUAL DEPLOYMENTS
21. #3 USE KUBE-LEGO FOR SSL (AND READ THE MANUAL WHEN UPGRADING)
22. #4 FIX NODES THAT USE STORAGE
23. Fix nodes that use storage For example: Static files (NGINX) Database servers (Galera / MariaDB) Object storage (Minio (S3 compatible)) Blob storage (CEPH)
24. Fix nodes that use storage Problem: Pinning means downtime (if that node is down) Minio has options to replicate itself
25. #5 MONITOR YOUR JOBS
26. Monitor your jobs We have a few large jobs Packaging all the data required to train neural networks Lots of IO Lots of DB queries A lot of calculations 100k loops, and memory leaks It literally has taken down our entire cluster a dozen times
27. But it couldn’t go down? Yes, because of Services: Spreads load to all servers Uses all cores in the cluster Using all DB servers Writing so fast to the disk Cluster down
28. #6 POD ANTI-AFFINITY
29. Pod anti-affinity Kubernetes schedules by default to node with most spare resources Can be that all your backend pods run on 1 node Gets redistributed when node goes down Takes 2-5 seconds (for our backend) Also prevents all pods on one node issue
30. #7 MONITOR THE CONTAINER LOGS
32. THANK YOU VERY MUCH
33. DOCKER GRUNN PROJECT DATE CLIENT 26 02 2019 JOSHUA PEPER CTO VERIFAI
36. Python / Django TensorFlow / AI Kubernetes / DevOps Swift for iOS Kotlin for Android C++ PHP / Laravel Hypervisors / BGP / DevOps Swift for iOS Kotlin for Android