Making Your Controllers Resilient: Workqueue To The Rescue

Transcript

1. Making Your Controllers Resilient: Workqueue To The Rescue Madhav Jivrajani, Arsh Sharma - VMware
2. Queue
3. What types of faults can occur when you operate in a distributed setting?
6. In distributed systems, failure is the norm.
7. In distributed systems, failure is the norm. Which is why we have to design for failure
8. What is the cause of these failures?
13. Transient errors: “weelllll, maybe if I try again, it’ll work fine”
14. The concept of retrying is extremely useful in Distributed Systems. It can make or break your system.
15. Retry storms!
16. client-go • client-go is a library used to communicate with a k8s cluster. • https://github.com/kubernetes/client-go
18. SharedInformer • We can stay informed about when events like pod creation, node joining, etc. are triggered by using a primitive exposed by Kubernetes and the client-go called SharedInformer, inside the cache package. • Previously, each controller had its own informer cache that it would use.
20. The Big Picture https://github.com/kubernetes/sample-controller/blob/master/docs/controller-client-go.md
21. Enter Queues Now here we just had a print statement in our handler function, but most of the time your `AddFunc` will just be pushing events to a work queue.
22. What is the workqueue and why is it important?
23. workqueue package https://pkg.go.dev/k8s.io/client-go/util/workqueue
24. To try and understand this functionality provided, we’ll try and look at the following 2 things: • How does the queue itself work? • What are the different extensions to this queue that are provided?
30. How does enqueuing work with these 2 sets?
38. Due to this queue pattern, you can have: • Multiple producers producing items to be processed, and multiple consumers that pop these items out and process them. • This allows for “parallelizing” the work that needs to be done. • Next question is: what happens when an item is done processing?
44. • It is possible that we have multiple instances of the same item that are to be processed. How do we ensure that we process this item just once? • Due to multiple concurrent consumers, how do we prevent the same item being processed multiple times, concurrently? ◦ This is important because if we duplicate process an item - best case you end up doing additional work to rectify the mistake and worst case you have a thrashing effect of processing and retrying.
50. What if another item ‘1’ gets added?
55. A few things to note: • Format of the key: <resource_namespace>/<resource_name> • If no namespace, it’ll just be <resource_name>
56. What type of queues does the workqueue package have?
57. • Delaying queue ◦ Extends the queue with the ability to add an element after specified duration. • Rate limiting queue ◦ It uses the delaying queue to rate limit items being added to the queue. ◦ The default rate limiter is a simple exponential rate limiter that rate limits per key. ▪ If a key had n re-queues, it will be added after 2^n * someBaseDelay back to the queue.
58. https://engineering.bitnami.com/articles/a-deep-dive-into-kubernetes-controllers.html
59. A few practices to keep in mind: • Always handle errors outside your business logic ◦ Handling errors typically consists of requeueing a work item. • Start workers for for processing items from workqueue only after cache is in sync successfully.
60. Metrics that are exposed by the workqueue package as part of the /metrics endpoint
62. Resources and References • Kubernetes Controllers • Kubernetes client-go workqueue example • Kubernetes sample-controller • Workqueue package
63. Thank You! Arsh Madhav Twitter: @RinkiyaKeDad Twitter: @MadhavJivrajani K8s slack: @arsh K8s slack: @madhav