Scaling Fleet Clusters with gRPC

Talk given at CoreOS Fest'16, it shows how Giant Swarm managed to scale fleet CoreOS tool to manage a distributed systemd cluster at large scale.

Transcript

1. Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 1 / 29 Scaling Fleet Clusters with gRPC Giant Swarm GmbH Hugo Tavares, Hector Fernandez May 10, 2016
2. Outline 1 Fleet today 2 Why Fleet ? 3 Fleet gRPC 4 Experiment Validation Setup 5 Analysis of Results Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 2 / 29
3. About me • Infrastructure Engineer @giantswarm • Love HPC, monitoring & performance :) • Have fun with distributed systems • Slow runner • Spanish who DON’T like siesta Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 3 / 29
4. Fleet Today init distributed systemd • Manage systemd units in a cluster • Talks to systemd via dbus • Use etcd as storage and coordination system Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 4 / 29
5. Fleet Today init distributed systemd • Manage systemd units in a cluster • Talks to systemd via dbus • Use etcd as storage and coordination system Figure: fleet overview Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 4 / 29
6. Fleet Today The architecture is composed of two main building blocks: • Engine (master) • Agent (workers) Figure: fleet: Engine and Agents Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 5 / 29
7. How do ”we” use Fleet? Our services are deployed as unit chains • Service definition is done using a JSON file, ‘swarm.json‘ Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 6 / 29
8. HOWEVER :/ ... Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 7 / 29
9. Limitations of Fleet • Performance degradation over the time • actions over units taking forever • etcdctl set command can take 40ms or 1100ms • Fleet out of sync • unresponsive agents • incoherent fleet ‘list-units‘ • Limited scalability • Excessive etcd dependencies: persistent and transient storage • ”If etcd is sad. fleet is sad. We are sad.” • Fail recovery • If etcd goes down, everything falls apart Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 8 / 29
10. Why Fleet ? Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 9 / 29
11. Why Fleet ? Giant Swarm intensively use fleet as init system of our infrastructure Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 10 / 29
12. Why Fleet ? Giant Swarm intensively use fleet as init system of our infrastructure What do we love ? • Robustness • Intuitive app definition • Distributed systemd Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 10 / 29
13. Why Fleet ? How do you manage your Kubernetes systemd units? • Fleet could help you to control Kubernetes Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 11 / 29
14. Why Fleet ? How do you manage your Kubernetes systemd units? • Fleet could help you to control Kubernetes How do you manage other INFRA services? • Maybe Kubernetes is not what you want to use, e.g. cleanup tasks, monitoring Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 11 / 29
15. What’s Next ? Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 12 / 29
16. What’s next ? We started investigated how to: • Reduce the etcd dependencies • Obtain better throughput (units/sec and amount of units) • Improve fail recovery mechanisms • Mitigate the scalability problems Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 13 / 29
17. What’s next ? We started investigated how to: • Reduce the etcd dependencies • Obtain better throughput (units/sec and amount of units) • Improve fail recovery mechanisms • Mitigate the scalability problems Maybe we can use gRPC instead of etcd ! Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 13 / 29
18. Fleet gRPC • gRPC is the new communication layer between: Engine & Agents • etcd is ONLY used for PERSISTENT storage • TRANSIENT data is stored in-memory Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 14 / 29
19. Fleet gRPC • gRPC is the new communication layer between: Engine & Agents • etcd is ONLY used for PERSISTENT storage • TRANSIENT data is stored in-memory Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 14 / 29
20. Fleet gRPC Benefits Fail recovery: • Connection retry • Resistant to network hiccups • Fallback mechanism (e.g. firewall) • Agent healtchecks • If etcd goes DOWN, we are still operative (on-going) Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 15 / 29
21. Fleet gRPC Benefits Smooth Rolling Updates • Engine/Agents detect if there are etcd nodes in the cluster • Tested in our production INFRA (hybrid fleet cluster) Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 16 / 29
22. Fleet gRPC Benefits Smooth Rolling Updates • Engine/Agents detect if there are etcd nodes in the cluster • Tested in our production INFRA (hybrid fleet cluster) Other benefits: • Hybrid etcd/gRPC cluster (etcd and grpc nodes) • New leader election algorithm • TLS communication Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 16 / 29
23. Let’s validate those benefits! Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 17 / 29
24. Experimental Validation Configuration: • CoreOS cluster with 3 machines as master/engine and 4 workers/agents • systemd v219 • etcd v2.1.2 • fleet v0.11.7 • nomi v0.1.0 Tested versions: • Scenario 1: 3k units fleet-etcd vs fleet-grpc • Scenario 2: 20k units with fleet-grpc • Scenario 3: 1k units fleet-etcd with etcdv2.2 vs etcdv3 Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 18 / 29
25. systemd vs fleet How much time does it take to create 500 systemd units in 1 node? Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 19 / 29
26. systemd vs fleet How much time does it take to create 500 systemd units in 1 node? • With fleet-grpc: 58.8 secs • With fleet-etcd (current upstream version): 74.4 secs • With systemd: 162 secs (sequential) Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 19 / 29
27. systemd vs fleet How much time does it take to create 500 systemd units in 1 node? Why ? • ‘systemd linking‘ operation is very EXPENSIVE Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 19 / 29
28. Fleet gRPC - Throughput How do they perform to load/start 3k units ? • fleet-etcd: • 75 % of units started in between 1.729 - 28.14 secs • Max: 265.88 secs • fleet-grpc: • 85 % of units started in between 1.004 - 9.907 secs • Max: 90.0 secs Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 20 / 29
29. Fleet gRPC - Throughput How do they perform to load/start 3k units ? • fleet-etcd: 75 % of units started in between 1.729 - 28.14 secs • fleet-grpc: 85 % of units started in between 1.004 - 9.907 secs Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 20 / 29
30. Fleet gRPC - Threshold How many units can we handle in the cluster ? • fleet-etcd: no more than 3k • fleet-grpc: up to 20k and beyond.. Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 21 / 29
31. Fleet gRPC - Threshold How many units can we handle in the cluster ? • fleet-etcd: no more than 3k • fleet-grpc: up to 20k and beyond.. 0 5000 10000 15000 20000 0 500 1000 1500 2000 2500 Number of units Timestamp (secs) Starting Running Figure: fleet-grpc 20k Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 21 / 29
32. Fleet gRPC - CPU usage CPU usage of fleet & etcd when using fleet-etcd ? 0 20 40 60 80 100 120 140 160 180 0 200 400 600 800 1000 1200 CPU usage (per) Timestamp (secs) 0000bc285bb8c6b5 - Time/CPU 00008bf747b3b669 - Time/CPU 00002c1a00dd1eda - Time/CPU 0 50 100 150 200 250 300 0 200 400 600 800 1000 1200 CPU usage (per) Timestamp (secs) 00008bf747b3b669 - Time/CPU 00002c1a00dd1eda - Time/CPU 0000bc285bb8c6b5 - Time/CPU Figure: etcd / fleet – CPU usage Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 22 / 29
33. Fleet gRPC - CPU usage CPU usage of fleet & etcd with the fleet-gRPC ? 0 5 10 15 20 25 30 35 40 45 0 50 100 150 200 250 300 350 400 CPU usage (per) Timestamp (secs) 00002c1a00dd1eda - Time/CPU 0000bc285bb8c6b5 - Time/CPU 00008bf747b3b669 - Time/CPU 0 50 100 150 200 250 300 350 400 450 500 0 50 100 150 200 250 300 350 400 CPU usage (per) Timestamp (secs) 00002c1a00dd1eda - Time/CPU 0000bc285bb8c6b5 - Time/CPU 00008bf747b3b669 - Time/CPU Figure: etcd / fleet – CPU usage Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 23 / 29
34. Fleet-gRPC - What about fleet-etcdv3 ? Experiment: • Evaluate the DELAY when starting (load/start) 1k units 0 5 10 15 20 25 30 0 20 40 60 80 100 120 140 Delay time (secs) Number of units Completion time (secs) Start Completion/Delay secs/Units Running units 0 10 20 30 40 50 60 70 0 20 40 60 80 100 120 140 160 0 100 200 300 400 500 600 700 800 900 1000 Delay time (secs) Number of units Completion time (secs) Start Completion/Delay secs/Units Running units 0 5 10 15 20 25 30 35 40 45 0 20 40 60 80 100 120 Delay time (secs) Number of units Completion time (secs) Start Completion/Delay secs/Units Running units Figure: fleet-etcd v3 experimental / fleet-etcd v2.3 / fleet-grpc with etcdv2.2 We used the experimental etcdv3 of few months ago Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 24 / 29
35. Conclusion Fleet with gRPC offers: • Better throughput • Less etcd friction • Lower CPU usage in the fleet agents • Secure communication • Improved scalability • No need of frequent cleanup tasks Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 25 / 29
36. Future Improvements We plan to: • Use etcdv3 • Add TLS to secure the communication • Improve the unit states transmission between agents-engine, (send what it has changed) • Change the reconciliation loop mechanism • Look at the usage of systemd –user to gain in performance • systemd linking is a quite expensive operation. Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 26 / 29
37. Future Improvements On-going work (PRs/Issues) • Our fleet repo ‘github.com/giantswarm/fleet‘ • Our benchmarking tool ‘github.com/giantswarm/nomi‘ Issues • #1426 • #1418 • #3142 • #150 • #3182 • #1393 Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 27 / 29
38. WE ARE HIRING ! Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 28 / 29
39. THANKS ! Time for ... Hugo Tavares, Hector Fernandez Scaling Fleet Clusters with gRPC May 10, 2016 29 / 29