Kubernetes Controllers - are they loops or events?

Transcript

1. Kubernetes Controllers
   Are they loops or events?
   Tim Hockin
   @thockin
   v1
   

   View Slide

2. Background on “reconciliation”:
   https://speakerdeck.com/thockin/kubernetes-what-is-reconciliation
   

   View Slide

3. Background on “edge vs. level”:
   https://speakerdeck.com/thockin/edge-vs-level-triggered-logic
   

   View Slide

4. Usually when we talk about
   controllers we refer to them
   as a “loop”
   

   View Slide

5. Imagine a controller for Pods
   (aka kubelet). It has 2 jobs:
   1) Actuate the pod API
   2) Report status on pods
   

   View Slide

6. What you’d expect looks
   something like:
   

   View Slide

7. Node Kubernetes API
   a
   kubelet
   b
   c
   Get all pods
   

   View Slide

8. Node Kubernetes API
   a
   kubelet
   b
   c
   { name: a, ... }
   { name: b, ... }
   { name: c, ... }
   

   View Slide

9. Node Kubernetes API
   a
   kubelet
   b
   c
   for each pod p {
   if p is running {
   verify p config
   } else {
   start p
   }
   gather status
   }
   

   View Slide

10. Node Kubernetes API
    a
    kubelet
    b
    c
    Set status
    c
    a
    b
    

    View Slide

11. ...then repeat
    (aka “a poll loop”)
    

    View Slide

12. Here’s where it matters
    

    View Slide

13. Node Kubernetes API
    a
    kubelet
    b
    c
    c
    a
    b
    kubectl
    delete pod b
    

    View Slide

14. Node Kubernetes API
    a
    kubelet
    c
    c
    a
    b
    kubectl
    delete pod b
    

    View Slide

15. Node Kubernetes API
    a
    kubelet
    c
    Get all pods
    c
    a
    b
    

    View Slide

16. Node Kubernetes API
    a
    kubelet
    c
    { name: a, ... }
    { name: c, ... }
    c
    a
    b
    

    View Slide

17. Node Kubernetes API
    a
    kubelet
    c
    I have “b” but API
    doesn’t - delete it!
    c
    a
    b
    

    View Slide

18. Node Kubernetes API
    a
    kubelet
    c
    Set status
    c
    a
    

    View Slide

19. This is correct level-triggered
    reconciliation
    Read desired state, make it so
    

    View Slide

20. Some controllers are
    implemented this way, but it’s
    inefficient at scale
    

    View Slide

21. Imagine thousands of
    controllers (kubelet,
    kube-proxy, dns, ingress,
    storage...) polling
    continuously
    

    View Slide

22. We need to achieve the same
    behavior more efficiently
    

    View Slide

23. We could poll less often, but
    then it takes a long (and
    variable) time to react - not a
    great UX
    

    View Slide

24. Enter the “list-watch” model
    

    View Slide

25. Node Kubernetes API
    a
    kubelet
    b
    c
    Get all pods
    

    View Slide

26. Node Kubernetes API
    a
    kubelet
    b
    c
    { name: a, ... }
    { name: b, ... }
    { name: c, ... }
    

    View Slide

27. Node Kubernetes API
    a
    kubelet
    b
    c
    Cache:
    { name: a, ... }
    { name: b, ... }
    { name: c, ... }
    

    View Slide

28. Node Kubernetes API
    a
    kubelet
    b
    c
    Watch all
    pods
    Cache:
    { name: a, ... }
    { name: b, ... }
    { name: c, ... }
    

    View Slide

29. Node Kubernetes API
    a
    kubelet
    b
    c
    Cache:
    { name: a, ... }
    { name: b, ... }
    { name: c, ... }
    for each pod p {
    if p is running {
    verify p config
    } else {
    start p
    }
    gather status
    }
    

    View Slide

30. Node Kubernetes API
    a
    kubelet
    b
    c
    Set status
    c
    a
    b
    Cache:
    { name: a, ... }
    { name: b, ... }
    { name: c, ... }
    

    View Slide

31. We trade memory (the cache)
    for other resources (API
    server CPU in particular)
    

    View Slide

32. There’s no point in polling my
    own cache, so what happens
    next?
    

    View Slide

33. Remember that watch we did
    earlier? That’s an open
    stream for events.
    

    View Slide

34. Node Kubernetes API
    a
    kubelet
    b
    c
    c
    a
    b
    kubectl
    delete pod b
    Cache:
    { name: a, ... }
    { name: b, ... }
    { name: c, ... }
    

    View Slide

35. Node Kubernetes API
    a
    kubelet
    c
    c
    a
    b
    kubectl
    delete pod b
    Cache:
    { name: a, ... }
    { name: b, ... }
    { name: c, ... }
    

    View Slide

36. Node Kubernetes API
    a
    kubelet
    c
    Delete:
    { name: b, ... }
    c
    a
    b
    Cache:
    { name: a, ... }
    { name: b, ... }
    { name: c, ... }
    

    View Slide

37. Node Kubernetes API
    a
    kubelet
    c
    Delete:
    { name: b, ... }
    c
    a
    b
    Cache:
    { name: a, ... }
    { name: c, ... }
    

    View Slide

38. Node Kubernetes API
    a
    kubelet
    c
    Cache:
    { name: a, ... }
    { name: c, ... }
    c
    a
    b
    API said to delete
    pod “b”.
    

    View Slide

39. Node Kubernetes API
    a
    kubelet
    c
    Cache:
    { name: a, ... }
    { name: c, ... }
    c
    a
    API said to delete
    pod “b”.
    

    View Slide

40. “But you said edge-triggered
    is bad!”
    

    View Slide

41. It is! But this isn’t
    edge-triggered.
    

    View Slide

42. The cache is updated by
    events (edges) but we are still
    reconciling state
    

    View Slide

43. “???”
    

    View Slide

44. The controller can be
    restarted at any time and the
    cache will be reconstructed -
    we can’t “miss an edge*”
    * modulo bugs, read on
    

    View Slide

45. Even if you miss an event, you
    can still recover the state
    

    View Slide

46. Ultimately it’s all just
    software, and software has
    bugs. Controllers should
    re-list periodically to get full
    state...
    

    View Slide

47. ...but we’ve put a lot of energy
    into making sure that our
    list-watch is reliable.
    

    View Slide