$30 off During Our Annual Pro Sale. View Details »

Kubernetes: What is "reconciliation"?

Kubernetes: What is "reconciliation"?

A very brief exploration of what we mean when we talk about reconciliation in the context of Kubernetes APIs and controllers.

This is mostly animation-style pictures, so don't be afraid of the length of it :)

Tim Hockin

July 28, 2019
Tweet

More Decks by Tim Hockin

Other Decks in Technology

Transcript

  1. Google Cloud Platform
    Kubernetes: What is
    “reconciliation”?
    Tim Hockin
    @thockin

    View Slide

  2. Google Cloud Platform
    Assume there’s a cloud API to make shapes.
    Why shapes? It’s just concrete enough to reason about,
    while not getting stuck in the details.

    View Slide

  3. Google Cloud Platform
    API

    View Slide

  4. Google Cloud Platform
    Make shape “Foo”
    - type: Square
    - color: Red
    API

    View Slide

  5. Google Cloud Platform
    API
    Foo

    View Slide

  6. Google Cloud Platform
    This API is fine, but I want to wrap it into a declarative
    system (e.g. Kubernetes)

    View Slide

  7. Google Cloud Platform
    K8s API

    View Slide

  8. Google Cloud Platform
    K8s API
    kubectl create shape “Foo”
    - type: Square
    - color: Purple

    View Slide

  9. Google Cloud Platform
    K8s API
    kind: Shape
    name: Foo
    type: Square
    color: Red

    View Slide

  10. Google Cloud Platform
    K8s API
    Make shape “Foo”
    - type: Square
    - color: Red
    kind: Shape
    name: Foo
    type: Square
    color: Red

    View Slide

  11. Google Cloud Platform
    K8s
    Foo
    API
    kind: Shape
    name: Foo
    type: Square
    color: Red

    View Slide

  12. Google Cloud Platform
    The controller will keep my Kubernetes object in sync with
    the underlying API

    View Slide

  13. Google Cloud Platform
    K8s
    Foo
    API
    kind: Shape
    name: Foo
    type: Square
    color: Red

    View Slide

  14. Google Cloud Platform
    K8s
    Foo
    API
    kind: Shape
    name: Foo
    type: Square
    color: Red
    kubectl edit shape “Foo”
    - type: Square
    - color: Purple

    View Slide

  15. Google Cloud Platform
    K8s
    Foo
    API
    kind: Shape
    name: Foo
    type: Square
    color: Purple

    View Slide

  16. Google Cloud Platform
    K8s
    Foo
    API
    kind: Shape
    name: Foo
    type: Square
    color: Purple
    Update shape “Foo”
    - color: Purple

    View Slide

  17. Google Cloud Platform
    K8s
    Foo
    API
    kind: Shape
    name: Foo
    type: Square
    color: Purple

    View Slide

  18. Google Cloud Platform
    This is what we call “reconciliation”. Specifically, this is
    uni-directional reconciliation.
    What happens if...

    View Slide

  19. Google Cloud Platform
    K8s
    Foo
    API
    kind: Shape
    name: Foo
    type: Square
    color: Purple

    View Slide

  20. Google Cloud Platform
    K8s
    Foo
    API
    kind: Shape
    name: Foo
    type: Square
    color: Purple
    Delete shape Foo

    View Slide

  21. Google Cloud Platform
    K8s API
    kind: Shape
    name: Foo
    type: Square
    color: Purple

    View Slide

  22. Google Cloud Platform
    The shape I wanted has inadvertently been removed by a
    human or other system.
    If we only reconcile in one direction, we will never fix it!
    We need to observe that the underlying state has changed
    and re-assert the state we want.

    View Slide

  23. Google Cloud Platform
    K8s API
    Make shape “Foo”
    - type: Square
    - color: Purple
    kind: Shape
    name: Foo
    type: Square
    color: Purple

    View Slide

  24. Google Cloud Platform
    K8s
    Foo
    API
    kind: Shape
    name: Foo
    type: Square
    color: Purple

    View Slide

  25. Google Cloud Platform
    We usually call this bi-directional reconciliation.
    But it gets funkier. What if...

    View Slide

  26. Google Cloud Platform
    K8s
    Foo
    API
    kind: Shape
    name: Foo
    type: Square
    color: Purple

    View Slide

  27. Google Cloud Platform
    K8s
    Foo
    API
    kind: Shape
    name: Foo
    type: Square
    color: Purple
    Create shape Bar
    - type: Circle
    - color: Red

    View Slide

  28. Google Cloud Platform
    K8s
    Foo
    API
    kind: Shape
    name: Foo
    type: Square
    color: Purple
    Bar

    View Slide

  29. Google Cloud Platform
    What should the controller do?
    Does it expect to have exclusive use of all shapes? If so,
    clean up!
    Does it expect to share the shapes API with other users?
    If so, leave it alone!
    Right?

    View Slide

  30. Google Cloud Platform
    I said it gets funkier. What if...

    View Slide

  31. Google Cloud Platform
    K8s
    Foo
    API
    kind: Shape
    name: Foo
    type: Square
    color: Purple

    View Slide

  32. Google Cloud Platform
    K8s
    Foo
    API
    kind: Shape
    name: Foo
    type: Square
    color: Purple
    CRASH

    View Slide

  33. Google Cloud Platform
    K8s
    Foo
    API
    kind: Shape
    name: Foo
    type: Square
    color: Purple
    CRASH
    kubectl delete shape
    “Foo”

    View Slide

  34. Google Cloud Platform
    K8s
    Foo
    API
    CRASH

    View Slide

  35. Google Cloud Platform
    K8s
    Foo
    API
    RECOVER

    View Slide

  36. Google Cloud Platform
    K8s
    Foo
    API

    View Slide

  37. Google Cloud Platform
    The controller missed the deletion of the shape, but as we
    saw earlier, it ignored things it doesn’t know.
    This is a LEAK!

    View Slide

  38. Google Cloud Platform
    The controller has to know which shapes it owns and
    which it doesn’t.
    HOW to do that depends on the API. Examples:
    ● Special name prefixes
    ● Metadata (labels, tags, description)
    ● Controller-specific checkpoints

    View Slide

  39. Google Cloud Platform
    K8s API
    RECOVER
    Foo
    labels:
    owner: k8s

    View Slide

  40. Google Cloud Platform
    K8s API
    List shapes where
    owner=k8s
    Foo
    labels:
    owner: k8s

    View Slide

  41. Google Cloud Platform
    K8s API
    Foo
    Foo
    labels:
    owner: k8s

    View Slide

  42. Google Cloud Platform
    K8s API
    Foo
    labels:
    owner: k8s
    I don’t have a
    “Foo”

    View Slide

  43. Google Cloud Platform
    K8s API
    Delete shape “Foo”
    Foo
    labels:
    owner: k8s

    View Slide

  44. Google Cloud Platform
    K8s API

    View Slide

  45. Google Cloud Platform
    This is sometimes called the “list-watch” pattern.
    Now the controller will keep things it owns in sync and
    ignores other things.
    What if...

    View Slide

  46. Google Cloud Platform
    K8s API
    kind: Shape
    name: Foo
    type: Square
    color: Purple
    Foo
    labels:
    owner: k8s

    View Slide

  47. Google Cloud Platform
    K8s API
    kind: Shape
    name: Foo
    type: Square
    color: Purple
    Create shape Bar
    - type: Circle
    - color: Red
    - labels:
    - owner: k8s
    Foo
    labels:
    owner: k8s

    View Slide

  48. Google Cloud Platform
    K8s
    Foo
    labels:
    owner: k8s
    API
    kind: Shape
    name: Foo
    type: Square
    color: Purple
    Bar
    labels:
    owner: k8s

    View Slide

  49. Google Cloud Platform
    K8s API
    List shapes where
    owner=k8s
    Foo
    labels:
    owner: k8s
    Bar
    labels:
    owner: k8s
    kind: Shape
    name: Foo
    type: Square
    color: Purple

    View Slide

  50. Google Cloud Platform
    K8s API
    Foo
    labels:
    owner: k8s
    Bar
    labels:
    owner: k8s
    kind: Shape
    name: Foo
    type: Square
    color: Purple
    Foo
    Bar

    View Slide

  51. Google Cloud Platform
    K8s API
    Foo
    labels:
    owner: k8s
    Bar
    labels:
    owner: k8s
    kind: Shape
    name: Foo
    type: Square
    color: Purple
    I don’t have a
    “Bar”

    View Slide

  52. Google Cloud Platform
    K8s API
    Delete shape “Bar”
    Foo
    labels:
    owner: k8s
    Bar
    labels:
    owner: k8s
    kind: Shape
    name: Foo
    type: Square
    color: Purple

    View Slide

  53. Google Cloud Platform
    K8s API
    Foo
    labels:
    owner: k8s
    kind: Shape
    name: Foo
    type: Square
    color: Purple

    View Slide

  54. Google Cloud Platform
    Note that while doing a full reconciliation at startup is
    necessary, it is not sufficient.
    Good controllers will reconcile against underlying APIs
    continuously or at least periodically.

    View Slide

  55. Google Cloud Platform
    How does this apply to real life?
    This pattern is found in almost every case where
    Kubernetes layers on top of some other API. Examples:
    ● Cloud load-balancers for Services & Ingress
    ● Cloud disks for PersistentVolumes
    ● iptables rules for Services
    ● Running containers for Pods

    View Slide

  56. Google Cloud Platform
    Sadly, not every controller gets this right.
    While every controller should strive for complete
    reconciliation, sometimes the underlying API makes it very
    hard or expensive or even just impossible.
    :(

    View Slide

  57. Google Cloud Platform
    There are some techniques that can mitigate the lack of
    mechanisms to denote ownership (or augment them).

    View Slide

  58. Google Cloud Platform
    Finalizers

    View Slide

  59. Google Cloud Platform
    K8s API

    View Slide

  60. Google Cloud Platform
    kubectl create shape “Foo”
    - type: Square
    - color: Purple
    K8s API

    View Slide

  61. Google Cloud Platform
    K8s API
    kind: Shape
    name: Foo
    type: Square
    color: Purple

    View Slide

  62. Google Cloud Platform
    K8s API
    Patch shape “Foo”
    - finalizer: shapes
    kind: Shape
    name: Foo
    type: Square
    color: Purple

    View Slide

  63. Google Cloud Platform
    K8s API
    kind: Shape
    name: Foo
    type: Square
    color: Purple
    finalizers:
    - shapes

    View Slide

  64. Google Cloud Platform
    K8s API
    Make shape “Foo”
    - type: Square
    - color: Purple
    kind: Shape
    name: Foo
    type: Square
    color: Purple
    finalizers:
    - shapes

    View Slide

  65. Google Cloud Platform
    K8s
    Foo
    API
    kind: Shape
    name: Foo
    type: Square
    color: Purple
    finalizers:
    - shapes

    View Slide

  66. Google Cloud Platform
    K8s
    Foo
    API
    CRASH
    kind: Shape
    name: Foo
    type: Square
    color: Purple
    finalizers:
    - shapes

    View Slide

  67. Google Cloud Platform
    K8s
    Foo
    API
    CRASH
    kubectl delete shape
    “Foo”
    kind: Shape
    name: Foo
    type: Square
    color: Purple
    finalizers:
    - shapes

    View Slide

  68. Google Cloud Platform
    K8s
    Foo
    API
    CRASH
    kind: Shape
    name: Foo
    type: Square
    color: Purple
    finalizers:
    - shapes
    deletionTimestamp
    is set, but finalizer
    prevents removal of
    the object

    View Slide

  69. Google Cloud Platform
    K8s
    Foo
    API
    RECOVER
    kind: Shape
    name: Foo
    type: Square
    color: Purple
    finalizers:
    - shapes

    View Slide

  70. Google Cloud Platform
    K8s
    Foo
    API
    kind: Shape
    name: Foo
    type: Square
    color: Purple
    finalizers:
    - shapes

    View Slide

  71. Google Cloud Platform
    The controller observes the pending deletion of the shape.

    View Slide

  72. Google Cloud Platform
    K8s API
    Delete shape “Foo”
    Foo
    labels:
    owner: k8s
    kind: Shape
    name: Foo
    type: Square
    color: Purple
    finalizers:
    - shapes

    View Slide

  73. Google Cloud Platform
    K8s API
    kind: Shape
    name: Foo
    type: Square
    color: Purple
    finalizers:
    - shapes

    View Slide

  74. Google Cloud Platform
    K8s API
    Patch shape “Foo” to
    remove finalizer
    kind: Shape
    name: Foo
    type: Square
    color: Purple
    finalizers:
    - shapes

    View Slide

  75. Google Cloud Platform
    K8s API
    kind: Shape
    name: Foo
    type: Square
    color: Purple
    deletion can be
    completed

    View Slide

  76. Google Cloud Platform
    K8s API

    View Slide

  77. Google Cloud Platform
    CustomResources

    View Slide

  78. Google Cloud Platform
    K8s API

    View Slide

  79. Google Cloud Platform
    kubectl create shape “Foo”
    - type: Square
    - color: Purple
    K8s API

    View Slide

  80. Google Cloud Platform
    K8s API
    kind: Shape
    name: Foo
    type: Square
    color: Purple

    View Slide

  81. Google Cloud Platform
    K8s API
    Create ShapeRef “Foo”
    - ownerReference: Foo
    kind: Shape
    name: Foo
    type: Square
    color: Purple

    View Slide

  82. Google Cloud Platform
    K8s API
    kind: Shape
    name: Foo
    type: Square
    color: Purple
    kind: ShapeRef
    name: Foo

    View Slide

  83. Google Cloud Platform
    K8s API
    Make shape “Foo”
    - type: Square
    - color: Purple
    kind: Shape
    name: Foo
    type: Square
    color: Purple
    kind: ShapeRef
    name: Foo

    View Slide

  84. Google Cloud Platform
    K8s
    Foo
    API
    kind: Shape
    name: Foo
    type: Square
    color: Purple
    kind: ShapeRef
    name: Foo

    View Slide

  85. Google Cloud Platform
    K8s
    Foo
    API
    CRASH
    kind: Shape
    name: Foo
    type: Square
    color: Purple
    kind: ShapeRef
    name: Foo

    View Slide

  86. Google Cloud Platform
    K8s
    Foo
    API
    CRASH
    kubectl delete shape
    “Foo”
    kind: Shape
    name: Foo
    type: Square
    color: Purple
    kind: ShapeRef
    name: Foo

    View Slide

  87. Google Cloud Platform
    K8s
    Foo
    API
    CRASH
    kind: ShapeRef
    name: Foo

    View Slide

  88. Google Cloud Platform
    K8s
    Foo
    API
    RECOVER
    kind: ShapeRef
    name: Foo

    View Slide

  89. Google Cloud Platform
    K8s
    Foo
    API
    kind: ShapeRef
    name: Foo

    View Slide

  90. Google Cloud Platform
    The controller did not observe the deletion of the Shape,
    but it does observe the dangling ShapeRef.

    View Slide

  91. Google Cloud Platform
    K8s API
    Delete shape “Foo”
    Foo
    labels:
    owner: k8s
    kind: ShapeRef
    name: Foo

    View Slide

  92. Google Cloud Platform
    K8s API
    kind: ShapeRef
    name: Foo

    View Slide

  93. Google Cloud Platform
    K8s API
    Delete shaperef “Foo”
    kind: ShapeRef
    name: Foo

    View Slide

  94. Google Cloud Platform
    K8s API

    View Slide

  95. Google Cloud Platform
    In most of these mechanisms, there’s some amount of
    “you broke it, you bought it”.
    If a user deletes the ShapeRef or removes the finalizer or
    edits the underlying metadata, the linkage can be broken.
    You broke it, you get to keep the pieces.

    View Slide