Kubernetes: What is "reconciliation"?

A very brief exploration of what we mean when we talk about reconciliation in the context of Kubernetes APIs and controllers.

This is mostly animation-style pictures, so don't be afraid of the length of it :)

Transcript

1. Google Cloud Platform Kubernetes: What is “reconciliation”? Tim Hockin <thockin@google.com> @thockin
2. Google Cloud Platform Assume there’s a cloud API to make shapes. Why shapes? It’s just concrete enough to reason about, while not getting stuck in the details.
3. Google Cloud Platform API
4. Google Cloud Platform Make shape “Foo” - type: Square - color: Red API
5. Google Cloud Platform API Foo
6. Google Cloud Platform This API is fine, but I want to wrap it into a declarative system (e.g. Kubernetes)
7. Google Cloud Platform K8s API
8. Google Cloud Platform K8s API kubectl create shape “Foo” - type: Square - color: Purple
9. Google Cloud Platform K8s API kind: Shape name: Foo type: Square color: Red
10. Google Cloud Platform K8s API Make shape “Foo” - type: Square - color: Red kind: Shape name: Foo type: Square color: Red
11. Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Red
12. Google Cloud Platform The controller will keep my Kubernetes object in sync with the underlying API
13. Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Red
14. Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Red kubectl edit shape “Foo” - type: Square - color: Purple
15. Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Purple
16. Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Purple Update shape “Foo” - color: Purple
17. Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Purple
18. Google Cloud Platform This is what we call “reconciliation”. Specifically, this is uni-directional reconciliation. What happens if...
19. Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Purple
20. Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Purple Delete shape Foo
21. Google Cloud Platform K8s API kind: Shape name: Foo type: Square color: Purple
22. Google Cloud Platform The shape I wanted has inadvertently been removed by a human or other system. If we only reconcile in one direction, we will never fix it! We need to observe that the underlying state has changed and re-assert the state we want.
23. Google Cloud Platform K8s API Make shape “Foo” - type: Square - color: Purple kind: Shape name: Foo type: Square color: Purple
24. Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Purple
25. Google Cloud Platform We usually call this bi-directional reconciliation. But it gets funkier. What if...
26. Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Purple
27. Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Purple Create shape Bar - type: Circle - color: Red
28. Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Purple Bar
29. Google Cloud Platform What should the controller do? Does it expect to have exclusive use of all shapes? If so, clean up! Does it expect to share the shapes API with other users? If so, leave it alone! Right?
30. Google Cloud Platform I said it gets funkier. What if...
31. Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Purple
32. Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Purple CRASH
33. Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Purple CRASH kubectl delete shape “Foo”
34. Google Cloud Platform K8s Foo API CRASH
35. Google Cloud Platform K8s Foo API RECOVER
36. Google Cloud Platform K8s Foo API
37. Google Cloud Platform The controller missed the deletion of the shape, but as we saw earlier, it ignored things it doesn’t know. This is a LEAK!
38. Google Cloud Platform The controller has to know which shapes it owns and which it doesn’t. HOW to do that depends on the API. Examples: • Special name prefixes • Metadata (labels, tags, description) • Controller-specific checkpoints
39. Google Cloud Platform K8s API RECOVER Foo labels: owner: k8s
40. Google Cloud Platform K8s API List shapes where owner=k8s Foo labels: owner: k8s
41. Google Cloud Platform K8s API Foo Foo labels: owner: k8s
42. Google Cloud Platform K8s API Foo labels: owner: k8s I don’t have a “Foo”
43. Google Cloud Platform K8s API Delete shape “Foo” Foo labels: owner: k8s
44. Google Cloud Platform K8s API
45. Google Cloud Platform This is sometimes called the “list-watch” pattern. Now the controller will keep things it owns in sync and ignores other things. What if...
46. Google Cloud Platform K8s API kind: Shape name: Foo type: Square color: Purple Foo labels: owner: k8s
47. Google Cloud Platform K8s API kind: Shape name: Foo type: Square color: Purple Create shape Bar - type: Circle - color: Red - labels: - owner: k8s Foo labels: owner: k8s
48. Google Cloud Platform K8s Foo labels: owner: k8s API kind: Shape name: Foo type: Square color: Purple Bar labels: owner: k8s
49. Google Cloud Platform K8s API List shapes where owner=k8s Foo labels: owner: k8s Bar labels: owner: k8s kind: Shape name: Foo type: Square color: Purple
50. Google Cloud Platform K8s API Foo labels: owner: k8s Bar labels: owner: k8s kind: Shape name: Foo type: Square color: Purple Foo Bar
51. Google Cloud Platform K8s API Foo labels: owner: k8s Bar labels: owner: k8s kind: Shape name: Foo type: Square color: Purple I don’t have a “Bar”
52. Google Cloud Platform K8s API Delete shape “Bar” Foo labels: owner: k8s Bar labels: owner: k8s kind: Shape name: Foo type: Square color: Purple
53. Google Cloud Platform K8s API Foo labels: owner: k8s kind: Shape name: Foo type: Square color: Purple
54. Google Cloud Platform Note that while doing a full reconciliation at startup is necessary, it is not sufficient. Good controllers will reconcile against underlying APIs continuously or at least periodically.
55. Google Cloud Platform How does this apply to real life? This pattern is found in almost every case where Kubernetes layers on top of some other API. Examples: • Cloud load-balancers for Services & Ingress • Cloud disks for PersistentVolumes • iptables rules for Services • Running containers for Pods
56. Google Cloud Platform Sadly, not every controller gets this right. While every controller should strive for complete reconciliation, sometimes the underlying API makes it very hard or expensive or even just impossible. :(
57. Google Cloud Platform There are some techniques that can mitigate the lack of mechanisms to denote ownership (or augment them).
58. Google Cloud Platform Finalizers
59. Google Cloud Platform K8s API
60. Google Cloud Platform kubectl create shape “Foo” - type: Square - color: Purple K8s API
61. Google Cloud Platform K8s API kind: Shape name: Foo type: Square color: Purple
62. Google Cloud Platform K8s API Patch shape “Foo” - finalizer: shapes kind: Shape name: Foo type: Square color: Purple
63. Google Cloud Platform K8s API kind: Shape name: Foo type: Square color: Purple finalizers: - shapes
64. Google Cloud Platform K8s API Make shape “Foo” - type: Square - color: Purple kind: Shape name: Foo type: Square color: Purple finalizers: - shapes
65. Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Purple finalizers: - shapes
66. Google Cloud Platform K8s Foo API CRASH kind: Shape name: Foo type: Square color: Purple finalizers: - shapes
67. Google Cloud Platform K8s Foo API CRASH kubectl delete shape “Foo” kind: Shape name: Foo type: Square color: Purple finalizers: - shapes
68. Google Cloud Platform K8s Foo API CRASH kind: Shape name: Foo type: Square color: Purple finalizers: - shapes deletionTimestamp is set, but finalizer prevents removal of the object
69. Google Cloud Platform K8s Foo API RECOVER kind: Shape name: Foo type: Square color: Purple finalizers: - shapes
70. Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Purple finalizers: - shapes
71. Google Cloud Platform The controller observes the pending deletion of the shape.
72. Google Cloud Platform K8s API Delete shape “Foo” Foo labels: owner: k8s kind: Shape name: Foo type: Square color: Purple finalizers: - shapes
73. Google Cloud Platform K8s API kind: Shape name: Foo type: Square color: Purple finalizers: - shapes
74. Google Cloud Platform K8s API Patch shape “Foo” to remove finalizer kind: Shape name: Foo type: Square color: Purple finalizers: - shapes
75. Google Cloud Platform K8s API kind: Shape name: Foo type: Square color: Purple deletion can be completed
76. Google Cloud Platform K8s API
77. Google Cloud Platform CustomResources
78. Google Cloud Platform K8s API
79. Google Cloud Platform kubectl create shape “Foo” - type: Square - color: Purple K8s API
80. Google Cloud Platform K8s API kind: Shape name: Foo type: Square color: Purple
81. Google Cloud Platform K8s API Create ShapeRef “Foo” - ownerReference: Foo kind: Shape name: Foo type: Square color: Purple
82. Google Cloud Platform K8s API kind: Shape name: Foo type: Square color: Purple kind: ShapeRef name: Foo
83. Google Cloud Platform K8s API Make shape “Foo” - type: Square - color: Purple kind: Shape name: Foo type: Square color: Purple kind: ShapeRef name: Foo
84. Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Purple kind: ShapeRef name: Foo
85. Google Cloud Platform K8s Foo API CRASH kind: Shape name: Foo type: Square color: Purple kind: ShapeRef name: Foo
86. Google Cloud Platform K8s Foo API CRASH kubectl delete shape “Foo” kind: Shape name: Foo type: Square color: Purple kind: ShapeRef name: Foo
87. Google Cloud Platform K8s Foo API CRASH kind: ShapeRef name: Foo
88. Google Cloud Platform K8s Foo API RECOVER kind: ShapeRef name: Foo
89. Google Cloud Platform K8s Foo API kind: ShapeRef name: Foo
90. Google Cloud Platform The controller did not observe the deletion of the Shape, but it does observe the dangling ShapeRef.
91. Google Cloud Platform K8s API Delete shape “Foo” Foo labels: owner: k8s kind: ShapeRef name: Foo
92. Google Cloud Platform K8s API kind: ShapeRef name: Foo
93. Google Cloud Platform K8s API Delete shaperef “Foo” kind: ShapeRef name: Foo
94. Google Cloud Platform K8s API
95. Google Cloud Platform In most of these mechanisms, there’s some amount of “you broke it, you bought it”. If a user deletes the ShapeRef or removes the finalizer or edits the underlying metadata, the linkage can be broken. You broke it, you get to keep the pieces.