Kubernetes: What is "reconciliation"?

Kubernetes: What is "reconciliation"?

A very brief exploration of what we mean when we talk about reconciliation in the context of Kubernetes APIs and controllers.

This is mostly animation-style pictures, so don't be afraid of the length of it :)

569f10721398d92f5033097ac6d9132c?s=128

Tim Hockin

July 28, 2019
Tweet

Transcript

  1. Google Cloud Platform Kubernetes: What is “reconciliation”? Tim Hockin <thockin@google.com>

    @thockin
  2. Google Cloud Platform Assume there’s a cloud API to make

    shapes. Why shapes? It’s just concrete enough to reason about, while not getting stuck in the details.
  3. Google Cloud Platform API

  4. Google Cloud Platform Make shape “Foo” - type: Square -

    color: Red API
  5. Google Cloud Platform API Foo

  6. Google Cloud Platform This API is fine, but I want

    to wrap it into a declarative system (e.g. Kubernetes)
  7. Google Cloud Platform K8s API

  8. Google Cloud Platform K8s API kubectl create shape “Foo” -

    type: Square - color: Purple
  9. Google Cloud Platform K8s API kind: Shape name: Foo type:

    Square color: Red
  10. Google Cloud Platform K8s API Make shape “Foo” - type:

    Square - color: Red kind: Shape name: Foo type: Square color: Red
  11. Google Cloud Platform K8s Foo API kind: Shape name: Foo

    type: Square color: Red
  12. Google Cloud Platform The controller will keep my Kubernetes object

    in sync with the underlying API
  13. Google Cloud Platform K8s Foo API kind: Shape name: Foo

    type: Square color: Red
  14. Google Cloud Platform K8s Foo API kind: Shape name: Foo

    type: Square color: Red kubectl edit shape “Foo” - type: Square - color: Purple
  15. Google Cloud Platform K8s Foo API kind: Shape name: Foo

    type: Square color: Purple
  16. Google Cloud Platform K8s Foo API kind: Shape name: Foo

    type: Square color: Purple Update shape “Foo” - color: Purple
  17. Google Cloud Platform K8s Foo API kind: Shape name: Foo

    type: Square color: Purple
  18. Google Cloud Platform This is what we call “reconciliation”. Specifically,

    this is uni-directional reconciliation. What happens if...
  19. Google Cloud Platform K8s Foo API kind: Shape name: Foo

    type: Square color: Purple
  20. Google Cloud Platform K8s Foo API kind: Shape name: Foo

    type: Square color: Purple Delete shape Foo
  21. Google Cloud Platform K8s API kind: Shape name: Foo type:

    Square color: Purple
  22. Google Cloud Platform The shape I wanted has inadvertently been

    removed by a human or other system. If we only reconcile in one direction, we will never fix it! We need to observe that the underlying state has changed and re-assert the state we want.
  23. Google Cloud Platform K8s API Make shape “Foo” - type:

    Square - color: Purple kind: Shape name: Foo type: Square color: Purple
  24. Google Cloud Platform K8s Foo API kind: Shape name: Foo

    type: Square color: Purple
  25. Google Cloud Platform We usually call this bi-directional reconciliation. But

    it gets funkier. What if...
  26. Google Cloud Platform K8s Foo API kind: Shape name: Foo

    type: Square color: Purple
  27. Google Cloud Platform K8s Foo API kind: Shape name: Foo

    type: Square color: Purple Create shape Bar - type: Circle - color: Red
  28. Google Cloud Platform K8s Foo API kind: Shape name: Foo

    type: Square color: Purple Bar
  29. Google Cloud Platform What should the controller do? Does it

    expect to have exclusive use of all shapes? If so, clean up! Does it expect to share the shapes API with other users? If so, leave it alone! Right?
  30. Google Cloud Platform I said it gets funkier. What if...

  31. Google Cloud Platform K8s Foo API kind: Shape name: Foo

    type: Square color: Purple
  32. Google Cloud Platform K8s Foo API kind: Shape name: Foo

    type: Square color: Purple CRASH
  33. Google Cloud Platform K8s Foo API kind: Shape name: Foo

    type: Square color: Purple CRASH kubectl delete shape “Foo”
  34. Google Cloud Platform K8s Foo API CRASH

  35. Google Cloud Platform K8s Foo API RECOVER

  36. Google Cloud Platform K8s Foo API

  37. Google Cloud Platform The controller missed the deletion of the

    shape, but as we saw earlier, it ignored things it doesn’t know. This is a LEAK!
  38. Google Cloud Platform The controller has to know which shapes

    it owns and which it doesn’t. HOW to do that depends on the API. Examples: • Special name prefixes • Metadata (labels, tags, description) • Controller-specific checkpoints
  39. Google Cloud Platform K8s API RECOVER Foo labels: owner: k8s

  40. Google Cloud Platform K8s API List shapes where owner=k8s Foo

    labels: owner: k8s
  41. Google Cloud Platform K8s API Foo Foo labels: owner: k8s

  42. Google Cloud Platform K8s API Foo labels: owner: k8s I

    don’t have a “Foo”
  43. Google Cloud Platform K8s API Delete shape “Foo” Foo labels:

    owner: k8s
  44. Google Cloud Platform K8s API

  45. Google Cloud Platform This is sometimes called the “list-watch” pattern.

    Now the controller will keep things it owns in sync and ignores other things. What if...
  46. Google Cloud Platform K8s API kind: Shape name: Foo type:

    Square color: Purple Foo labels: owner: k8s
  47. Google Cloud Platform K8s API kind: Shape name: Foo type:

    Square color: Purple Create shape Bar - type: Circle - color: Red - labels: - owner: k8s Foo labels: owner: k8s
  48. Google Cloud Platform K8s Foo labels: owner: k8s API kind:

    Shape name: Foo type: Square color: Purple Bar labels: owner: k8s
  49. Google Cloud Platform K8s API List shapes where owner=k8s Foo

    labels: owner: k8s Bar labels: owner: k8s kind: Shape name: Foo type: Square color: Purple
  50. Google Cloud Platform K8s API Foo labels: owner: k8s Bar

    labels: owner: k8s kind: Shape name: Foo type: Square color: Purple Foo Bar
  51. Google Cloud Platform K8s API Foo labels: owner: k8s Bar

    labels: owner: k8s kind: Shape name: Foo type: Square color: Purple I don’t have a “Bar”
  52. Google Cloud Platform K8s API Delete shape “Bar” Foo labels:

    owner: k8s Bar labels: owner: k8s kind: Shape name: Foo type: Square color: Purple
  53. Google Cloud Platform K8s API Foo labels: owner: k8s kind:

    Shape name: Foo type: Square color: Purple
  54. Google Cloud Platform Note that while doing a full reconciliation

    at startup is necessary, it is not sufficient. Good controllers will reconcile against underlying APIs continuously or at least periodically.
  55. Google Cloud Platform How does this apply to real life?

    This pattern is found in almost every case where Kubernetes layers on top of some other API. Examples: • Cloud load-balancers for Services & Ingress • Cloud disks for PersistentVolumes • iptables rules for Services • Running containers for Pods
  56. Google Cloud Platform Sadly, not every controller gets this right.

    While every controller should strive for complete reconciliation, sometimes the underlying API makes it very hard or expensive or even just impossible. :(
  57. Google Cloud Platform There are some techniques that can mitigate

    the lack of mechanisms to denote ownership (or augment them).
  58. Google Cloud Platform Finalizers

  59. Google Cloud Platform K8s API

  60. Google Cloud Platform kubectl create shape “Foo” - type: Square

    - color: Purple K8s API
  61. Google Cloud Platform K8s API kind: Shape name: Foo type:

    Square color: Purple
  62. Google Cloud Platform K8s API Patch shape “Foo” - finalizer:

    shapes kind: Shape name: Foo type: Square color: Purple
  63. Google Cloud Platform K8s API kind: Shape name: Foo type:

    Square color: Purple finalizers: - shapes
  64. Google Cloud Platform K8s API Make shape “Foo” - type:

    Square - color: Purple kind: Shape name: Foo type: Square color: Purple finalizers: - shapes
  65. Google Cloud Platform K8s Foo API kind: Shape name: Foo

    type: Square color: Purple finalizers: - shapes
  66. Google Cloud Platform K8s Foo API CRASH kind: Shape name:

    Foo type: Square color: Purple finalizers: - shapes
  67. Google Cloud Platform K8s Foo API CRASH kubectl delete shape

    “Foo” kind: Shape name: Foo type: Square color: Purple finalizers: - shapes
  68. Google Cloud Platform K8s Foo API CRASH kind: Shape name:

    Foo type: Square color: Purple finalizers: - shapes deletionTimestamp is set, but finalizer prevents removal of the object
  69. Google Cloud Platform K8s Foo API RECOVER kind: Shape name:

    Foo type: Square color: Purple finalizers: - shapes
  70. Google Cloud Platform K8s Foo API kind: Shape name: Foo

    type: Square color: Purple finalizers: - shapes
  71. Google Cloud Platform The controller observes the pending deletion of

    the shape.
  72. Google Cloud Platform K8s API Delete shape “Foo” Foo labels:

    owner: k8s kind: Shape name: Foo type: Square color: Purple finalizers: - shapes
  73. Google Cloud Platform K8s API kind: Shape name: Foo type:

    Square color: Purple finalizers: - shapes
  74. Google Cloud Platform K8s API Patch shape “Foo” to remove

    finalizer kind: Shape name: Foo type: Square color: Purple finalizers: - shapes
  75. Google Cloud Platform K8s API kind: Shape name: Foo type:

    Square color: Purple deletion can be completed
  76. Google Cloud Platform K8s API

  77. Google Cloud Platform CustomResources

  78. Google Cloud Platform K8s API

  79. Google Cloud Platform kubectl create shape “Foo” - type: Square

    - color: Purple K8s API
  80. Google Cloud Platform K8s API kind: Shape name: Foo type:

    Square color: Purple
  81. Google Cloud Platform K8s API Create ShapeRef “Foo” - ownerReference:

    Foo kind: Shape name: Foo type: Square color: Purple
  82. Google Cloud Platform K8s API kind: Shape name: Foo type:

    Square color: Purple kind: ShapeRef name: Foo
  83. Google Cloud Platform K8s API Make shape “Foo” - type:

    Square - color: Purple kind: Shape name: Foo type: Square color: Purple kind: ShapeRef name: Foo
  84. Google Cloud Platform K8s Foo API kind: Shape name: Foo

    type: Square color: Purple kind: ShapeRef name: Foo
  85. Google Cloud Platform K8s Foo API CRASH kind: Shape name:

    Foo type: Square color: Purple kind: ShapeRef name: Foo
  86. Google Cloud Platform K8s Foo API CRASH kubectl delete shape

    “Foo” kind: Shape name: Foo type: Square color: Purple kind: ShapeRef name: Foo
  87. Google Cloud Platform K8s Foo API CRASH kind: ShapeRef name:

    Foo
  88. Google Cloud Platform K8s Foo API RECOVER kind: ShapeRef name:

    Foo
  89. Google Cloud Platform K8s Foo API kind: ShapeRef name: Foo

  90. Google Cloud Platform The controller did not observe the deletion

    of the Shape, but it does observe the dangling ShapeRef.
  91. Google Cloud Platform K8s API Delete shape “Foo” Foo labels:

    owner: k8s kind: ShapeRef name: Foo
  92. Google Cloud Platform K8s API kind: ShapeRef name: Foo

  93. Google Cloud Platform K8s API Delete shaperef “Foo” kind: ShapeRef

    name: Foo
  94. Google Cloud Platform K8s API

  95. Google Cloud Platform In most of these mechanisms, there’s some

    amount of “you broke it, you bought it”. If a user deletes the ShapeRef or removes the finalizer or edits the underlying metadata, the linkage can be broken. You broke it, you get to keep the pieces.