Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kubernetes Controllers - are they loops or events?

Tim Hockin
February 20, 2021

Kubernetes Controllers - are they loops or events?

Tim Hockin

February 20, 2021
Tweet

More Decks by Tim Hockin

Other Decks in Technology

Transcript

  1. Kubernetes Controllers
    Are they loops or events?
    Tim Hockin
    @thockin
    v1

    View full-size slide

  2. Background on “reconciliation”:
    https://speakerdeck.com/thockin/kubernetes-what-is-reconciliation

    View full-size slide

  3. Background on “edge vs. level”:
    https://speakerdeck.com/thockin/edge-vs-level-triggered-logic

    View full-size slide

  4. Usually when we talk about
    controllers we refer to them
    as a “loop”

    View full-size slide

  5. Imagine a controller for Pods
    (aka kubelet). It has 2 jobs:
    1) Actuate the pod API
    2) Report status on pods

    View full-size slide

  6. What you’d expect looks
    something like:

    View full-size slide

  7. Node Kubernetes API
    a
    kubelet
    b
    c
    Get all pods

    View full-size slide

  8. Node Kubernetes API
    a
    kubelet
    b
    c
    { name: a, ... }
    { name: b, ... }
    { name: c, ... }

    View full-size slide

  9. Node Kubernetes API
    a
    kubelet
    b
    c
    for each pod p {
    if p is running {
    verify p config
    } else {
    start p
    }
    gather status
    }

    View full-size slide

  10. Node Kubernetes API
    a
    kubelet
    b
    c
    Set status
    c
    a
    b

    View full-size slide

  11. ...then repeat
    (aka “a poll loop”)

    View full-size slide

  12. Here’s where it matters

    View full-size slide

  13. Node Kubernetes API
    a
    kubelet
    b
    c
    c
    a
    b
    kubectl
    delete pod b

    View full-size slide

  14. Node Kubernetes API
    a
    kubelet
    c
    c
    a
    b
    kubectl
    delete pod b

    View full-size slide

  15. Node Kubernetes API
    a
    kubelet
    c
    Get all pods
    c
    a
    b

    View full-size slide

  16. Node Kubernetes API
    a
    kubelet
    c
    { name: a, ... }
    { name: c, ... }
    c
    a
    b

    View full-size slide

  17. Node Kubernetes API
    a
    kubelet
    c
    I have “b” but API
    doesn’t - delete it!
    c
    a
    b

    View full-size slide

  18. Node Kubernetes API
    a
    kubelet
    c
    Set status
    c
    a

    View full-size slide

  19. This is correct level-triggered
    reconciliation
    Read desired state, make it so

    View full-size slide

  20. Some controllers are
    implemented this way, but it’s
    inefficient at scale

    View full-size slide

  21. Imagine thousands of
    controllers (kubelet,
    kube-proxy, dns, ingress,
    storage...) polling
    continuously

    View full-size slide

  22. We need to achieve the same
    behavior more efficiently

    View full-size slide

  23. We could poll less often, but
    then it takes a long (and
    variable) time to react - not a
    great UX

    View full-size slide

  24. Enter the “list-watch” model

    View full-size slide

  25. Node Kubernetes API
    a
    kubelet
    b
    c
    Get all pods

    View full-size slide

  26. Node Kubernetes API
    a
    kubelet
    b
    c
    { name: a, ... }
    { name: b, ... }
    { name: c, ... }

    View full-size slide

  27. Node Kubernetes API
    a
    kubelet
    b
    c
    Cache:
    { name: a, ... }
    { name: b, ... }
    { name: c, ... }

    View full-size slide

  28. Node Kubernetes API
    a
    kubelet
    b
    c
    Watch all
    pods
    Cache:
    { name: a, ... }
    { name: b, ... }
    { name: c, ... }

    View full-size slide

  29. Node Kubernetes API
    a
    kubelet
    b
    c
    Cache:
    { name: a, ... }
    { name: b, ... }
    { name: c, ... }
    for each pod p {
    if p is running {
    verify p config
    } else {
    start p
    }
    gather status
    }

    View full-size slide

  30. Node Kubernetes API
    a
    kubelet
    b
    c
    Set status
    c
    a
    b
    Cache:
    { name: a, ... }
    { name: b, ... }
    { name: c, ... }

    View full-size slide

  31. We trade memory (the cache)
    for other resources (API
    server CPU in particular)

    View full-size slide

  32. There’s no point in polling my
    own cache, so what happens
    next?

    View full-size slide

  33. Remember that watch we did
    earlier? That’s an open
    stream for events.

    View full-size slide

  34. Node Kubernetes API
    a
    kubelet
    b
    c
    c
    a
    b
    kubectl
    delete pod b
    Cache:
    { name: a, ... }
    { name: b, ... }
    { name: c, ... }

    View full-size slide

  35. Node Kubernetes API
    a
    kubelet
    c
    c
    a
    b
    kubectl
    delete pod b
    Cache:
    { name: a, ... }
    { name: b, ... }
    { name: c, ... }

    View full-size slide

  36. Node Kubernetes API
    a
    kubelet
    c
    Delete:
    { name: b, ... }
    c
    a
    b
    Cache:
    { name: a, ... }
    { name: b, ... }
    { name: c, ... }

    View full-size slide

  37. Node Kubernetes API
    a
    kubelet
    c
    Delete:
    { name: b, ... }
    c
    a
    b
    Cache:
    { name: a, ... }
    { name: c, ... }

    View full-size slide

  38. Node Kubernetes API
    a
    kubelet
    c
    Cache:
    { name: a, ... }
    { name: c, ... }
    c
    a
    b
    API said to delete
    pod “b”.

    View full-size slide

  39. Node Kubernetes API
    a
    kubelet
    c
    Cache:
    { name: a, ... }
    { name: c, ... }
    c
    a
    API said to delete
    pod “b”.

    View full-size slide

  40. “But you said edge-triggered
    is bad!”

    View full-size slide

  41. It is! But this isn’t
    edge-triggered.

    View full-size slide

  42. The cache is updated by
    events (edges) but we are still
    reconciling state

    View full-size slide

  43. The controller can be
    restarted at any time and the
    cache will be reconstructed -
    we can’t “miss an edge*”
    * modulo bugs, read on

    View full-size slide

  44. Even if you miss an event, you
    can still recover the state

    View full-size slide

  45. Ultimately it’s all just
    software, and software has
    bugs. Controllers should
    re-list periodically to get full
    state...

    View full-size slide

  46. ...but we’ve put a lot of energy
    into making sure that our
    list-watch is reliable.

    View full-size slide