Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Adventures In Optimizing The Kubernetes API Server

Adventures In Optimizing The Kubernetes API Server

Slides for a talk given at the Go Bangalore Meetup, Sept 3 2022.

Madhav Jivrajani

September 06, 2022

More Decks by Madhav Jivrajani

Other Decks in Programming


  1. $ whoami • Gets super excited about systems-y stuff •

    Work @ VMware (Kubernetes Upstream) • Within the Kubernetes community - SIG-{API Machinery, Scalability, Architecture, ContribEx}. ◦ Please reach out if you’d like to get started in the community! • Doing Go stuff for ~3 years, particularly love things around the Go runtime!
  2. Agenda • 50,000 ft view of how Watches work. •

    Watch Cache - zooming in juuuust a little bit. • What were the problems that existed? • What did we end up doing? • A few results!
  3. “Copying over” Includes: • Allocating the result buffer of the

    desired size. ◦ “Desired size” in real world scenarios can get substantially large. • Iterating over the list of items and copying them into the buffer. • Keep in mind, all of this happens under a lock… • … while other goroutines wait for this copying over to complete. • And these waiting goroutines in turn will have their own copying to do. • Because of this, we end up with spikes in memory consumption for watches opened against the “past”. ◦ We also end up wasting a few CPU cycles.
  4. “Copying over” Includes: • Constructing the watchCacheInterval ◦ This includes:

    ▪ Calculating start and end indices - uses binary search: fast! ▪ Allocating an internal buffer of constant size for further optimization. • This means we limit the memory consumption for watches from the past to a constant amount in 99% of the cases. ◦ The remaining 1% is for special cases like resource version 0. ◦ In this case, the performance is the same as before this change - no improvements.
  5. Considerations With “Async” • Considering the interval is used to

    serve events asynchronously, what do we need to keep in mind? • Prelude: ◦ As and when the watchCache becomes full, events are popped off - this is called “propagation”. ◦ The event to be popped off is tracked by an index internally called the startIndex. ◦ The interval tracks the event to be served also using an index called startIndex (different entities, same name!)
  6. Future Work • Paginating the watchCache itself ◦ BTree based

    backing cache ◦ Copy on Write Semantics • Caching serialization/deserialization
  7. References • Life of A Kubernetes Watch Event • PRs

    implementing this change: #1, #2 • Tracking issue for future work