Life of a Kubernetes Watch Event

The watch event is essential to the kubernetes architecture. It’s the key to maintaining high availability in the kubernetes control plane. Have you ever wondered how a watch event is propagated? In this presentation, we will cover how kubernetes delivers a watch event through control plane storage, API server, and finally to clients, and what happened to a watch event across server replicas. Attendees will leave with a full understanding of the life of a kubernetes watch event, which could help you make better decisions to implement your controllers in a much more scalable and performant way.

Transcript

1. Life of a Kubernetes Watch Event Haowei Cai (roycaihw@github), Google Wenjia Zhang (wenjiaswe@github), Google
2. About us Wenjia Zhang (@wenjiaswe) Software Engineer in Google. She is an active contributor for Kubernetes SIG API Machinery and etcd open source projects. Haowei Cai (@roycaihw) Software Engineer in Google. He is an active contributor for Kubernetes SIG API Machinery and client libraries.
3. Agenda • What is a Kubernetes Watch Event? • Why is Watch Event important for Kubernetes? • How is the life of a Kubernetes Watch Event? • Key Takeaways
4. What is a Kubernetes Watch Event?
5. What is Watch? Watch is an incremental change notification feed
6. Watch vs. Poll Watch Low latency Single connection
7. Watch vs. Poll Watch Poll Extra load Extra latency Multiple connection Low latency Single connection
8. Kubelet on nodes: • Previous: periodically poll kube-apiserver for secrets and configmaps • Now: watch individual secrets • OSS PR: Kubelet watches necessary secrets/configmaps instead of periodic polling #64752 Watch vs. Poll
9. What is Event? A single change to a watched resource Watched resource runtime.Object Event Type
10. What is Event? Watched resource runtime.Object Event Type pod Added replicaSet Modified node Deleted A single change to a watched resource
11. Why is Watch Event important for Kubernetes?
12. Kubernetes core design concept Level Triggering and Soft Reconciliation
13. Declarative configuration Desired state
14. Declarative configuration Actual state Desired state
15. State is accumulation of events State Person1
16. State is accumulation of events Person1, Added Person1, Modified Person1, Modified ... Events State Person1
17. State is accumulation of events State Pod1
18. State is accumulation of events Pod1, Added Pod1, Modified Pod1, Modified ... Events State Pod1 Image version change: 0.5.0 -> 1.5.3 Adding label
19. How is the life of a K8s watch event?
20. Fingerprint of kubernetes object: resourceVersion Kubernetes object resourceVersion A resourceVersion is valid on a single kind of resource across namespaces.
21. resourceVersion created with object change/event resourceVersion Object Count pod Pod Added, 519 replicaSet node Pod Modified, 603 Pod Modified, 604 replicaSet Added, 99 replicaSet Modified, 103 replicaSet Modifed, 205 Node Added, 1 Node Modified, 3 Node Deleted, 5
22. resourceVersion created every time the resource is written Pod1, Added Pod1, Modified Pod1, Modified ... Pod1 519 603 604 Image version change: 0.5.0 -> 1.5.3 Adding label
23. resourceVersion changes every time the resource is written Pod1, Added Pod1, Modified Pod1, Modified ... Pod1 519 603 604 Image version change: 0.5.0 -> 1.5.3 Adding label
24. Pod1, Added resourceVersion: 519 Life of a K8s watch event
25. Life of a K8s watch event Schedule pods on nodes Runs controllers Business logics Pod1, Added resourceVersion: 519
26. Life of a K8s watch event Pod1, Added resourceVersion: 519 Data Store
27. Life of a K8s watch event Pod1, Added resourceVersion: 519 API server
28. Life of a K8s watch event Client-go Pod1, Added resourceVersion: 519 Go clients for talking to a kubernetes cluster.
29. Life of a K8s watch event Client-go Pod1, Added resourceVersion: 519 Schedule pods on nodes Runs controllers Business logics
30. Watch Event in etcd
31. Watch in etcd etcd watch feature provides an event-based interface for asynchronously monitoring changes to keys. Revision (etcd) == resourceVersion (apiserver)
32. Watch event in etcd Client WATCH CREATED <watch-id> EVENT <watch-id> PUT <key1> <value1> EVENT <watch-id> DELETE <key2> ... CREATE WATCH <key1>..<key2> ...
33. Watch event in etcd WATCH CREATED <watch-id> EVENT <watch-id> PUT <key1> <value1> EVENT <watch-id> DELETE <key2> ... CREATE WATCH <key1>..<key2> ...
34. Watch Event in Kube APIserver
35. Watch Event in Kube APIserver etcd watcher watch_cache cacheWatcher cacheWatcher cacheWatcher
36. etcd watcher watch_cache cacheWatcher Watch Event in Kube APIserver cacheWatcher cacheWatcher cacheWatcher Get events from etcd Sending events...
37. • Flow control… • etcd3 Event -> API server Event... etcd watcher watch_cache cacheWatcher cacheWatcher cacheWatcher cacheWatcher Get events from etcd Sending events... Watch Event in Kube APIserver
38. rv(start) -> extract rv(end) <- insert etcd watcher watch_cache cacheWatcher cacheWatcher cacheWatcher cacheWatcher Cache: circular buffer Store Watch Event in Kube APIserver
39. • Watch(): cache ◦ Limited capacity • List(): store ◦ from the "end of cache history" etcd watcher watch_cache cacheWatcher cacheWatcher cacheWatcher cacheWatcher rv(start) -> extract rv(end) <- insert Cache: circular buffer Store Watch Event in Kube APIserver
40. etcd watcher watch_cache cacheWatcher input filter result cacheWatcher cacheWatcher Serve events to clients Watch Event in Kube APIserver
41. Watch Event in Client-go
42. What is Client-go? https://github.com/kubernetes/client-go • Go clients for talking to a kubernetes cluster • Used by Kubernetes itself Clientset Dynamic Client REST Client Informer ...
43. What is Client-go? https://github.com/kubernetes/client-go • Go clients for talking to a kubernetes cluster • Used by Kubernetes itself Clientset Dynamic Client REST Client Informer ...
44. What is Client-go? https://github.com/kubernetes/client-go • Go clients for talking to a kubernetes cluster • Used by Kubernetes itself Clientset Dynamic Client REST Client Informer ...
45. What is Client-go? https://github.com/kubernetes/client-go • Go clients for talking to a kubernetes cluster • Used by Kubernetes itself Clientset Dynamic Client REST Client Informer ...
46. What is Informer? Clientset Dynamic Client REST Client Informer ... Reflector DeltaFIFO Local Cache Callback k8s.io/client-go/tools/cache k8s.io/client-go/informers • Useful component for building event-oriented controllers • Used by control plane controllers, kubelet, etc. • Reflector used by kube-apiserver watch cache
47. Kubernetes controller workflow APIServer Controller Controller Controller Controller
48. Informer Kubernetes controller workflow APIServer Controller
49. Kubernetes controller workflow APIServer Reflector DeltaFIFO Local Cache Callbacks: OnAdd OnUpdate OnDelete Workqueue Clients CRUD List/ Watch Readonly Worker client-go controller- specific courtesy of: @caesarxuchao
50. Kubernetes controller workflow APIServer Reflector DeltaFIFO Local Cache Callbacks: OnAdd OnUpdate OnDelete Workqueue Clients CRUD List/ Watch Readonly Worker client-go controller- specific Pod1, Added resourceVersion: 519
51. Kubernetes controller workflow APIServer Reflector DeltaFIFO Local Cache Callbacks: OnAdd OnUpdate OnDelete Workqueue Clients CRUD List/ Watch Readonly Worker client-go controller- specific
52. Kubernetes controller workflow APIServer Reflector DeltaFIFO Local Cache Callbacks: OnAdd OnUpdate OnDelete Workqueue Clients CRUD List/ Watch Readonly Worker client-go controller- specific
53. Kubernetes controller workflow APIServer Reflector DeltaFIFO Local Cache Callbacks: OnAdd OnUpdate OnDelete Workqueue Clients CRUD List/ Watch Readonly Worker client-go controller- specific
54. Kubernetes controller workflow APIServer Reflector DeltaFIFO Local Cache Callbacks: OnAdd OnUpdate OnDelete Workqueue Clients CRUD List/ Watch Readonly Worker client-go controller- specific
55. Kubernetes controller workflow APIServer Reflector DeltaFIFO Local Cache Callbacks: OnAdd OnUpdate OnDelete Workqueue Clients CRUD List/ Watch Readonly Worker client-go controller- specific
56. Kubernetes controller workflow APIServer Reflector DeltaFIFO Local Cache Callbacks: OnAdd OnUpdate OnDelete Workqueue Clients CRUD List/ Watch Readonly Worker client-go controller- specific
57. Kubernetes controller workflow APIServer Reflector DeltaFIFO Local Cache Callbacks: OnAdd OnUpdate OnDelete Workqueue Clients CRUD List/ Watch Readonly Worker client-go controller- specific Pod1 RV: 519 Spec: ...
58. Kubernetes controller workflow APIServer Reflector DeltaFIFO Local Cache Callbacks: OnAdd OnUpdate OnDelete Workqueue Clients CRUD List/ Watch Readonly Worker client-go controller- specific Pod1, Updated resourceVersion: 818 Pod1 RV: 519 Spec: ...
59. Kubernetes controller workflow APIServer Reflector DeltaFIFO Local Cache Callbacks: OnAdd OnUpdate OnDelete Workqueue Clients CRUD List/ Watch Readonly Worker client-go controller- specific Pod1 RV: 818 Spec: ...
60. Kubernetes controller workflow APIServer Reflector DeltaFIFO Local Cache Callbacks: OnAdd OnUpdate OnDelete Workqueue Clients CRUD List/ Watch Readonly Worker client-go controller- specific
61. Kubernetes controller workflow APIServer Reflector DeltaFIFO Local Cache Callbacks: OnAdd OnUpdate OnDelete Workqueue Clients CRUD List/ Watch Readonly Worker client-go controller- specific
62. List and Watch Pseudo code
63. List and Watch Pseudo code Extract the actual ResourceVersion
64. List and Watch Pseudo code Start watch with latest ResourceVersion
65. List and Watch Pseudo code Most likely apiserver is not responsive.
66. List and Watch Pseudo code Watch closed normally (EOF) or unexpected error
67. List and Watch Pseudo code watchHandler watches w and keeps *resourceVersion up to date
68. Fingerprint of kubernetes object: resourceVersion Kubernetes object resourceVersion A resourceVersion is valid on a single kind of resource across namespaces.
69. Recap: resourceVersion Everything has a ResourceVersion: • Changes every time when you write to the storage • Individual API object (e.g. a Pod) has ResourceVersion • For a list of API objects (e.g. a PodList) ◦ The entire list has a ResourceVersion ◦ Each API object in list items has ResourceVersion The ResourceVersion of the top-level list is what should be used when starting a watch to observe events occurring after that list was populated.
70. Recap: resourceVersion • ListOption in List Request ◦ Unspecified: etcd ◦ RV>0: the result is at least as fresh as given RV ◦ RV=0: APIServer cache (stale read: #59848)
71. Recap: resourceVersion • ListOption in Watch Request ◦ Unspecified: unspecified time point ◦ RV=0: the result is an "ADDED" event for every existing object followed by events for changes that occur after the watch was established ▪ (main reason: backwards compatibility-- #13910) ◦ Best practice: always specify last listed/watched RV
72. Watch Event on kube-scheduler, kube-controller-manager, kublet…
73. Kubernetes controller workflow APIServer Reflector DeltaFIFO Local Cache Callbacks: OnAdd OnUpdate OnDelete Workqueue Clients CRUD List/ Watch Readonly Worker client-go controller- specific
74. Mini Scheduler
75. Mini Scheduler Watches: • Node • Pod
76. Mini Scheduler “business” logic • SchedulingQueue for pods waiting to be scheduled • PodCache for scheduled pods • NodeCache for existing nodes Pseudo code
77. kube-scheduler pkg/scheduler/scheduler.go Watches: • Node • Pod • PV • PVC • RC • RS • Stateful set • Service • PDB • Storage class
78. Key Takeaways
79. • A Kubernetes Watch Event is an efficient resource change notification • Watch Event is the key to Kubernetes level triggering and soft reconciliation concept • Watch is trustworthy and efficient • Use Informer! Don’t misuse Watch!
80. Thanks! Enjoy Seattle!
81. ?