$30 off During Our Annual Pro Sale. View Details »

Kubernetes Controllers - are they loops or events?

Tim Hockin
February 20, 2021

Kubernetes Controllers - are they loops or events?

Tim Hockin

February 20, 2021
Tweet

More Decks by Tim Hockin

Other Decks in Technology

Transcript

  1. Kubernetes Controllers
    Are they loops or events?
    Tim Hockin
    @thockin
    v1

    View Slide

  2. Background on “reconciliation”:
    https://speakerdeck.com/thockin/kubernetes-what-is-reconciliation

    View Slide

  3. Background on “edge vs. level”:
    https://speakerdeck.com/thockin/edge-vs-level-triggered-logic

    View Slide

  4. Usually when we talk about
    controllers we refer to them
    as a “loop”

    View Slide

  5. Imagine a controller for Pods
    (aka kubelet). It has 2 jobs:
    1) Actuate the pod API
    2) Report status on pods

    View Slide

  6. What you’d expect looks
    something like:

    View Slide

  7. Node Kubernetes API
    a
    kubelet
    b
    c
    Get all pods

    View Slide

  8. Node Kubernetes API
    a
    kubelet
    b
    c
    { name: a, ... }
    { name: b, ... }
    { name: c, ... }

    View Slide

  9. Node Kubernetes API
    a
    kubelet
    b
    c
    for each pod p {
    if p is running {
    verify p config
    } else {
    start p
    }
    gather status
    }

    View Slide

  10. Node Kubernetes API
    a
    kubelet
    b
    c
    Set status
    c
    a
    b

    View Slide

  11. ...then repeat
    (aka “a poll loop”)

    View Slide

  12. Here’s where it matters

    View Slide

  13. Node Kubernetes API
    a
    kubelet
    b
    c
    c
    a
    b
    kubectl
    delete pod b

    View Slide

  14. Node Kubernetes API
    a
    kubelet
    c
    c
    a
    b
    kubectl
    delete pod b

    View Slide

  15. Node Kubernetes API
    a
    kubelet
    c
    Get all pods
    c
    a
    b

    View Slide

  16. Node Kubernetes API
    a
    kubelet
    c
    { name: a, ... }
    { name: c, ... }
    c
    a
    b

    View Slide

  17. Node Kubernetes API
    a
    kubelet
    c
    I have “b” but API
    doesn’t - delete it!
    c
    a
    b

    View Slide

  18. Node Kubernetes API
    a
    kubelet
    c
    Set status
    c
    a

    View Slide

  19. This is correct level-triggered
    reconciliation
    Read desired state, make it so

    View Slide

  20. Some controllers are
    implemented this way, but it’s
    inefficient at scale

    View Slide

  21. Imagine thousands of
    controllers (kubelet,
    kube-proxy, dns, ingress,
    storage...) polling
    continuously

    View Slide

  22. We need to achieve the same
    behavior more efficiently

    View Slide

  23. We could poll less often, but
    then it takes a long (and
    variable) time to react - not a
    great UX

    View Slide

  24. Enter the “list-watch” model

    View Slide

  25. Node Kubernetes API
    a
    kubelet
    b
    c
    Get all pods

    View Slide

  26. Node Kubernetes API
    a
    kubelet
    b
    c
    { name: a, ... }
    { name: b, ... }
    { name: c, ... }

    View Slide

  27. Node Kubernetes API
    a
    kubelet
    b
    c
    Cache:
    { name: a, ... }
    { name: b, ... }
    { name: c, ... }

    View Slide

  28. Node Kubernetes API
    a
    kubelet
    b
    c
    Watch all
    pods
    Cache:
    { name: a, ... }
    { name: b, ... }
    { name: c, ... }

    View Slide

  29. Node Kubernetes API
    a
    kubelet
    b
    c
    Cache:
    { name: a, ... }
    { name: b, ... }
    { name: c, ... }
    for each pod p {
    if p is running {
    verify p config
    } else {
    start p
    }
    gather status
    }

    View Slide

  30. Node Kubernetes API
    a
    kubelet
    b
    c
    Set status
    c
    a
    b
    Cache:
    { name: a, ... }
    { name: b, ... }
    { name: c, ... }

    View Slide

  31. We trade memory (the cache)
    for other resources (API
    server CPU in particular)

    View Slide

  32. There’s no point in polling my
    own cache, so what happens
    next?

    View Slide

  33. Remember that watch we did
    earlier? That’s an open
    stream for events.

    View Slide

  34. Node Kubernetes API
    a
    kubelet
    b
    c
    c
    a
    b
    kubectl
    delete pod b
    Cache:
    { name: a, ... }
    { name: b, ... }
    { name: c, ... }

    View Slide

  35. Node Kubernetes API
    a
    kubelet
    c
    c
    a
    b
    kubectl
    delete pod b
    Cache:
    { name: a, ... }
    { name: b, ... }
    { name: c, ... }

    View Slide

  36. Node Kubernetes API
    a
    kubelet
    c
    Delete:
    { name: b, ... }
    c
    a
    b
    Cache:
    { name: a, ... }
    { name: b, ... }
    { name: c, ... }

    View Slide

  37. Node Kubernetes API
    a
    kubelet
    c
    Delete:
    { name: b, ... }
    c
    a
    b
    Cache:
    { name: a, ... }
    { name: c, ... }

    View Slide

  38. Node Kubernetes API
    a
    kubelet
    c
    Cache:
    { name: a, ... }
    { name: c, ... }
    c
    a
    b
    API said to delete
    pod “b”.

    View Slide

  39. Node Kubernetes API
    a
    kubelet
    c
    Cache:
    { name: a, ... }
    { name: c, ... }
    c
    a
    API said to delete
    pod “b”.

    View Slide

  40. “But you said edge-triggered
    is bad!”

    View Slide

  41. It is! But this isn’t
    edge-triggered.

    View Slide

  42. The cache is updated by
    events (edges) but we are still
    reconciling state

    View Slide

  43. “???”

    View Slide

  44. The controller can be
    restarted at any time and the
    cache will be reconstructed -
    we can’t “miss an edge*”
    * modulo bugs, read on

    View Slide

  45. Even if you miss an event, you
    can still recover the state

    View Slide

  46. Ultimately it’s all just
    software, and software has
    bugs. Controllers should
    re-list periodically to get full
    state...

    View Slide

  47. ...but we’ve put a lot of energy
    into making sure that our
    list-watch is reliable.

    View Slide