Adventures In Optimizing The Kubernetes API Server

Slides for a talk given at the Go Bangalore Meetup, Sept 3 2022.

Transcript

1. Adventures In Optimizing The Kubernetes API Server Madhav Jivrajani
2. $ whoami • Gets super excited about systems-y stuff • Work @ VMware (Kubernetes Upstream) • Within the Kubernetes community - SIG-{API Machinery, Scalability, Architecture, ContribEx}. ◦ Please reach out if you’d like to get started in the community! • Doing Go stuff for ~3 years, particularly love things around the Go runtime!
3. Agenda • 50,000 ft view of how Watches work. • Watch Cache - zooming in juuuust a little bit. • What were the problems that existed? • What did we end up doing? • A few results!
4. Overview
6. Give me all changes since resourceVersion x and stream them on the same connection!
7. Zooming In A Little Bit 🔎
8. Prelude - Initial Implementation of Watch
11. Current Design
15. Looks good! So, what’s the problem?
19. But…
23. “Copying over” Includes: • Allocating the result buffer of the desired size. ◦ “Desired size” in real world scenarios can get substantially large. • Iterating over the list of items and copying them into the buffer. • Keep in mind, all of this happens under a lock… • … while other goroutines wait for this copying over to complete. • And these waiting goroutines in turn will have their own copying to do. • Because of this, we end up with spikes in memory consumption for watches opened against the “past”. ◦ We also end up wasting a few CPU cycles.
24. The Solution
28. “Copying over” Includes: • Constructing the watchCacheInterval ◦ This includes: ▪ Calculating start and end indices - uses binary search: fast! ▪ Allocating an internal buffer of constant size for further optimization. • This means we limit the memory consumption for watches from the past to a constant amount in 99% of the cases. ◦ The remaining 1% is for special cases like resource version 0. ◦ In this case, the performance is the same as before this change - no improvements.
29. Considerations With “Async” • Considering the interval is used to serve events asynchronously, what do we need to keep in mind? • Prelude: ◦ As and when the watchCache becomes full, events are popped off - this is called “propagation”. ◦ The event to be popped off is tracked by an index internally called the startIndex. ◦ The interval tracks the event to be served also using an index called startIndex (different entities, same name!)
30. Considerations With “Async”
31. Considerations With “Async”
32. Considerations With “Async”
33. Considerations With “Async” If propagation happens faster than the interval serves events…
34. Considerations With “Async”
35. Considerations With “Async”
36. Considerations With “Async”
37. Considerations With “Async”
38. Considerations With “Async”
39. Considerations With “Async” Once the interval is invalidated, we stop the Watch.
40. A Few Results
41. Benchmark
42. Memory Usage Cumulative Total Mem In Bytes (worst case) Before 819200 After 1024
43. Memory Usage
44. Memory Profile Before:
45. Memory Profile After:
46. CPU Profile Before:
47. CPU Profile After:
48. CPU Profile After:
49. Future Work • Paginating the watchCache itself ◦ BTree based backing cache ◦ Copy on Write Semantics • Caching serialization/deserialization
50. References • Life of A Kubernetes Watch Event • PRs implementing this change: #1, #2 • Tracking issue for future work
51. Thank you! Twitter: @MadhavJivrajani K8s slack: @madhav