Kubernetes: What is "reconciliation"?

Slide 1

Slide 1 text

Google Cloud Platform Kubernetes: What is “reconciliation”? Tim Hockin @thockin

Slide 2

Slide 2 text

Google Cloud Platform Assume there’s a cloud API to make shapes. Why shapes? It’s just concrete enough to reason about, while not getting stuck in the details.

Slide 3

Slide 3 text

Google Cloud Platform API

Slide 4

Slide 4 text

Google Cloud Platform Make shape “Foo” - type: Square - color: Red API

Slide 5

Slide 5 text

Google Cloud Platform API Foo

Slide 6

Slide 6 text

Google Cloud Platform This API is ﬁne, but I want to wrap it into a declarative system (e.g. Kubernetes)

Slide 7

Slide 7 text

Google Cloud Platform K8s API

Slide 8

Slide 8 text

Google Cloud Platform K8s API kubectl create shape “Foo” - type: Square - color: Purple

Slide 9

Slide 9 text

Google Cloud Platform K8s API kind: Shape name: Foo type: Square color: Red

Slide 10

Slide 10 text

Google Cloud Platform K8s API Make shape “Foo” - type: Square - color: Red kind: Shape name: Foo type: Square color: Red

Slide 11

Slide 11 text

Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Red

Slide 12

Slide 12 text

Google Cloud Platform The controller will keep my Kubernetes object in sync with the underlying API

Slide 13

Slide 13 text

Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Red

Slide 14

Slide 14 text

Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Red kubectl edit shape “Foo” - type: Square - color: Purple

Slide 15

Slide 15 text

Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Purple

Slide 16

Slide 16 text

Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Purple Update shape “Foo” - color: Purple

Slide 17

Slide 17 text

Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Purple

Slide 18

Slide 18 text

Google Cloud Platform This is what we call “reconciliation”. Speciﬁcally, this is uni-directional reconciliation. What happens if...

Slide 19

Slide 19 text

Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Purple

Slide 20

Slide 20 text

Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Purple Delete shape Foo

Slide 21

Slide 21 text

Google Cloud Platform K8s API kind: Shape name: Foo type: Square color: Purple

Slide 22

Slide 22 text

Google Cloud Platform The shape I wanted has inadvertently been removed by a human or other system. If we only reconcile in one direction, we will never ﬁx it! We need to observe that the underlying state has changed and re-assert the state we want.

Slide 23

Slide 23 text

Google Cloud Platform K8s API Make shape “Foo” - type: Square - color: Purple kind: Shape name: Foo type: Square color: Purple

Slide 24

Slide 24 text

Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Purple

Slide 25

Slide 25 text

Google Cloud Platform We usually call this bi-directional reconciliation. But it gets funkier. What if...

Slide 26

Slide 26 text

Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Purple

Slide 27

Slide 27 text

Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Purple Create shape Bar - type: Circle - color: Red

Slide 28

Slide 28 text

Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Purple Bar

Slide 29

Slide 29 text

Google Cloud Platform What should the controller do? Does it expect to have exclusive use of all shapes? If so, clean up! Does it expect to share the shapes API with other users? If so, leave it alone! Right?

Slide 30

Slide 30 text

Google Cloud Platform I said it gets funkier. What if...

Slide 31

Slide 31 text

Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Purple

Slide 32

Slide 32 text

Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Purple CRASH

Slide 33

Slide 33 text

Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Purple CRASH kubectl delete shape “Foo”

Slide 34

Slide 34 text

Google Cloud Platform K8s Foo API CRASH

Slide 35

Slide 35 text

Google Cloud Platform K8s Foo API RECOVER

Slide 36

Slide 36 text

Google Cloud Platform K8s Foo API

Slide 37

Slide 37 text

Google Cloud Platform The controller missed the deletion of the shape, but as we saw earlier, it ignored things it doesn’t know. This is a LEAK!

Slide 38

Slide 38 text

Google Cloud Platform The controller has to know which shapes it owns and which it doesn’t. HOW to do that depends on the API. Examples: ● Special name preﬁxes ● Metadata (labels, tags, description) ● Controller-speciﬁc checkpoints

Slide 39

Slide 39 text

Google Cloud Platform K8s API RECOVER Foo labels: owner: k8s

Slide 40

Slide 40 text

Google Cloud Platform K8s API List shapes where owner=k8s Foo labels: owner: k8s

Slide 41

Slide 41 text

Google Cloud Platform K8s API Foo Foo labels: owner: k8s

Slide 42

Slide 42 text

Google Cloud Platform K8s API Foo labels: owner: k8s I don’t have a “Foo”

Slide 43

Slide 43 text

Google Cloud Platform K8s API Delete shape “Foo” Foo labels: owner: k8s

Slide 44

Slide 44 text

Google Cloud Platform K8s API

Slide 45

Slide 45 text

Google Cloud Platform This is sometimes called the “list-watch” pattern. Now the controller will keep things it owns in sync and ignores other things. What if...

Slide 46

Slide 46 text

Google Cloud Platform K8s API kind: Shape name: Foo type: Square color: Purple Foo labels: owner: k8s

Slide 47

Slide 47 text

Google Cloud Platform K8s API kind: Shape name: Foo type: Square color: Purple Create shape Bar - type: Circle - color: Red - labels: - owner: k8s Foo labels: owner: k8s

Slide 48

Slide 48 text

Google Cloud Platform K8s Foo labels: owner: k8s API kind: Shape name: Foo type: Square color: Purple Bar labels: owner: k8s

Slide 49

Slide 49 text

Google Cloud Platform K8s API List shapes where owner=k8s Foo labels: owner: k8s Bar labels: owner: k8s kind: Shape name: Foo type: Square color: Purple

Slide 50

Slide 50 text

Google Cloud Platform K8s API Foo labels: owner: k8s Bar labels: owner: k8s kind: Shape name: Foo type: Square color: Purple Foo Bar

Slide 51

Slide 51 text

Google Cloud Platform K8s API Foo labels: owner: k8s Bar labels: owner: k8s kind: Shape name: Foo type: Square color: Purple I don’t have a “Bar”

Slide 52

Slide 52 text

Google Cloud Platform K8s API Delete shape “Bar” Foo labels: owner: k8s Bar labels: owner: k8s kind: Shape name: Foo type: Square color: Purple

Slide 53

Slide 53 text

Google Cloud Platform K8s API Foo labels: owner: k8s kind: Shape name: Foo type: Square color: Purple

Slide 54

Slide 54 text

Google Cloud Platform Note that while doing a full reconciliation at startup is necessary, it is not suﬃcient. Good controllers will reconcile against underlying APIs continuously or at least periodically.

Slide 55

Slide 55 text

Google Cloud Platform How does this apply to real life? This pattern is found in almost every case where Kubernetes layers on top of some other API. Examples: ● Cloud load-balancers for Services & Ingress ● Cloud disks for PersistentVolumes ● iptables rules for Services ● Running containers for Pods

Slide 56

Slide 56 text

Google Cloud Platform Sadly, not every controller gets this right. While every controller should strive for complete reconciliation, sometimes the underlying API makes it very hard or expensive or even just impossible. :(

Slide 57

Slide 57 text

Google Cloud Platform There are some techniques that can mitigate the lack of mechanisms to denote ownership (or augment them).

Slide 58

Slide 58 text

Google Cloud Platform Finalizers

Slide 59

Slide 59 text

Google Cloud Platform K8s API

Slide 60

Slide 60 text

Google Cloud Platform kubectl create shape “Foo” - type: Square - color: Purple K8s API

Slide 61

Slide 61 text

Google Cloud Platform K8s API kind: Shape name: Foo type: Square color: Purple

Slide 62

Slide 62 text

Google Cloud Platform K8s API Patch shape “Foo” - finalizer: shapes kind: Shape name: Foo type: Square color: Purple

Slide 63

Slide 63 text

Google Cloud Platform K8s API kind: Shape name: Foo type: Square color: Purple finalizers: - shapes

Slide 64

Slide 64 text

Google Cloud Platform K8s API Make shape “Foo” - type: Square - color: Purple kind: Shape name: Foo type: Square color: Purple finalizers: - shapes

Slide 65

Slide 65 text

Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Purple finalizers: - shapes

Slide 66

Slide 66 text

Google Cloud Platform K8s Foo API CRASH kind: Shape name: Foo type: Square color: Purple finalizers: - shapes

Slide 67

Slide 67 text

Google Cloud Platform K8s Foo API CRASH kubectl delete shape “Foo” kind: Shape name: Foo type: Square color: Purple finalizers: - shapes

Slide 68

Slide 68 text

Google Cloud Platform K8s Foo API CRASH kind: Shape name: Foo type: Square color: Purple finalizers: - shapes deletionTimestamp is set, but finalizer prevents removal of the object

Slide 69

Slide 69 text

Google Cloud Platform K8s Foo API RECOVER kind: Shape name: Foo type: Square color: Purple finalizers: - shapes

Slide 70

Slide 70 text

Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Purple finalizers: - shapes

Slide 71

Slide 71 text

Google Cloud Platform The controller observes the pending deletion of the shape.

Slide 72

Slide 72 text

Google Cloud Platform K8s API Delete shape “Foo” Foo labels: owner: k8s kind: Shape name: Foo type: Square color: Purple finalizers: - shapes

Slide 73

Slide 73 text

Google Cloud Platform K8s API kind: Shape name: Foo type: Square color: Purple finalizers: - shapes

Slide 74

Slide 74 text

Google Cloud Platform K8s API Patch shape “Foo” to remove finalizer kind: Shape name: Foo type: Square color: Purple finalizers: - shapes

Slide 75

Slide 75 text

Google Cloud Platform K8s API kind: Shape name: Foo type: Square color: Purple deletion can be completed

Slide 76

Slide 76 text

Google Cloud Platform K8s API

Slide 77

Slide 77 text

Google Cloud Platform CustomResources

Slide 78

Slide 78 text

Google Cloud Platform K8s API

Slide 79

Slide 79 text

Google Cloud Platform kubectl create shape “Foo” - type: Square - color: Purple K8s API

Slide 80

Slide 80 text

Google Cloud Platform K8s API kind: Shape name: Foo type: Square color: Purple

Slide 81

Slide 81 text

Google Cloud Platform K8s API Create ShapeRef “Foo” - ownerReference: Foo kind: Shape name: Foo type: Square color: Purple

Slide 82

Slide 82 text

Google Cloud Platform K8s API kind: Shape name: Foo type: Square color: Purple kind: ShapeRef name: Foo

Slide 83

Slide 83 text

Google Cloud Platform K8s API Make shape “Foo” - type: Square - color: Purple kind: Shape name: Foo type: Square color: Purple kind: ShapeRef name: Foo

Slide 84

Slide 84 text

Google Cloud Platform K8s Foo API kind: Shape name: Foo type: Square color: Purple kind: ShapeRef name: Foo

Slide 85

Slide 85 text

Google Cloud Platform K8s Foo API CRASH kind: Shape name: Foo type: Square color: Purple kind: ShapeRef name: Foo

Slide 86

Slide 86 text

Google Cloud Platform K8s Foo API CRASH kubectl delete shape “Foo” kind: Shape name: Foo type: Square color: Purple kind: ShapeRef name: Foo

Slide 87

Slide 87 text

Google Cloud Platform K8s Foo API CRASH kind: ShapeRef name: Foo

Slide 88

Slide 88 text

Google Cloud Platform K8s Foo API RECOVER kind: ShapeRef name: Foo

Slide 89

Slide 89 text

Google Cloud Platform K8s Foo API kind: ShapeRef name: Foo

Slide 90

Slide 90 text

Google Cloud Platform The controller did not observe the deletion of the Shape, but it does observe the dangling ShapeRef.

Slide 91

Slide 91 text

Google Cloud Platform K8s API Delete shape “Foo” Foo labels: owner: k8s kind: ShapeRef name: Foo

Slide 92

Slide 92 text

Google Cloud Platform K8s API kind: ShapeRef name: Foo

Slide 93

Slide 93 text

Google Cloud Platform K8s API Delete shaperef “Foo” kind: ShapeRef name: Foo

Slide 94

Slide 94 text

Google Cloud Platform K8s API

Slide 95

Slide 95 text

Google Cloud Platform In most of these mechanisms, there’s some amount of “you broke it, you bought it”. If a user deletes the ShapeRef or removes the ﬁnalizer or edits the underlying metadata, the linkage can be broken. You broke it, you get to keep the pieces.