Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Kubernetes Controllers - are they loops or events?
Search
Tim Hockin
February 20, 2021
Technology
11
3.8k
Kubernetes Controllers - are they loops or events?
Tim Hockin
February 20, 2021
Tweet
Share
More Decks by Tim Hockin
See All by Tim Hockin
Kubernetes in the 2nd Decade
thockin
0
330
Why Service is the worst API in Kubernetes, and what we can do about it
thockin
2
880
Kubernetes Pod Probes
thockin
6
4.3k
Go Workspaces for Kubernetes
thockin
2
1k
Code Review in Kubernetes
thockin
2
1.7k
Multi-cluster: past, present, future
thockin
0
490
Kubernetes Network Models (why is this so dang hard?)
thockin
9
1.9k
KubeCon EU 2020: SIG-Network Intro and Deep-Dive
thockin
8
1.3k
A Non-Technical Kubernetes Talk (KubeCon EU 2020)
thockin
3
600
Other Decks in Technology
See All in Technology
計測による継続的なCI/CDの改善
sansantech
PRO
1
550
Pythonデータ分析実践試験 出題傾向や学習のポイントとテクニカルハイライト
terapyon
1
150
激動の一年を通じて見えてきた「技術でリードする」ということ
ktr_0731
7
5.1k
LangfuseではじめるAIアプリのLLMトレーシング
codenote
0
160
地に足の付いた現実的な技術選定から魔力のある体験を得る『AIレシート読み取り機能』のケーススタディ / From Grounded Tech Choices to Magical UX: A Case Study of AI Receipt Scanning
moznion
4
1.5k
Как мы автоматизировали интеграционное тестирование с Gonkey и не пожалели. Паша Егорычев, Кирилл Поляков
lamodatech
0
2.1k
Cursorをチョッパヤインタビューライターにチューニングする方法 / how to tuning cursor for interview write
shuzon
2
230
Datadog のトライアルを成功に導く技術 / Techniques for a successful Datadog trial
nulabinc
PRO
0
150
Amplifyとゼロからはじめた AIコーディング。失敗と気づき
mkdev10
1
100
事業と組織から目を逸らずに技術でリードする
ogugu9
7
1k
製造業向けIoTソリューション提案資料.pdf
haruki_uiru
0
260
MagicPodが描くAIエージェント戦略とソフトウェアテストの未来
magicpod
0
230
Featured
See All Featured
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
14
1.5k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
53k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
41
2.3k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
47
2.7k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
32
5.6k
Build The Right Thing And Hit Your Dates
maggiecrowley
35
2.7k
Automating Front-end Workflow
addyosmani
1370
200k
Six Lessons from altMBA
skipperchong
28
3.8k
RailsConf 2023
tenderlove
30
1.1k
The MySQL Ecosystem @ GitHub 2015
samlambert
251
12k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
52
2.5k
How STYLIGHT went responsive
nonsquared
100
5.5k
Transcript
Kubernetes Controllers Are they loops or events? Tim Hockin @thockin
v1
Background on “reconciliation”: https://speakerdeck.com/thockin/kubernetes-what-is-reconciliation
Background on “edge vs. level”: https://speakerdeck.com/thockin/edge-vs-level-triggered-logic
Usually when we talk about controllers we refer to them
as a “loop”
Imagine a controller for Pods (aka kubelet). It has 2
jobs: 1) Actuate the pod API 2) Report status on pods
What you’d expect looks something like:
Node Kubernetes API a kubelet b c Get all pods
Node Kubernetes API a kubelet b c { name: a,
... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet b c for each pod
p { if p is running { verify p config } else { start p } gather status }
Node Kubernetes API a kubelet b c Set status c
a b
...then repeat (aka “a poll loop”)
Here’s where it matters
Node Kubernetes API a kubelet b c c a b
kubectl delete pod b
Node Kubernetes API a kubelet c c a b kubectl
delete pod b
Node Kubernetes API a kubelet c Get all pods c
a b
Node Kubernetes API a kubelet c { name: a, ...
} { name: c, ... } c a b
Node Kubernetes API a kubelet c I have “b” but
API doesn’t - delete it! c a b
Node Kubernetes API a kubelet c Set status c a
This is correct level-triggered reconciliation Read desired state, make it
so
Some controllers are implemented this way, but it’s inefficient at
scale
Imagine thousands of controllers (kubelet, kube-proxy, dns, ingress, storage...) polling
continuously
We need to achieve the same behavior more efficiently
We could poll less often, but then it takes a
long (and variable) time to react - not a great UX
Enter the “list-watch” model
Node Kubernetes API a kubelet b c Get all pods
Node Kubernetes API a kubelet b c { name: a,
... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet b c Cache: { name:
a, ... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet b c Watch all pods
Cache: { name: a, ... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet b c Cache: { name:
a, ... } { name: b, ... } { name: c, ... } for each pod p { if p is running { verify p config } else { start p } gather status }
Node Kubernetes API a kubelet b c Set status c
a b Cache: { name: a, ... } { name: b, ... } { name: c, ... }
We trade memory (the cache) for other resources (API server
CPU in particular)
There’s no point in polling my own cache, so what
happens next?
Remember that watch we did earlier? That’s an open stream
for events.
Node Kubernetes API a kubelet b c c a b
kubectl delete pod b Cache: { name: a, ... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet c c a b kubectl
delete pod b Cache: { name: a, ... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet c Delete: { name: b,
... } c a b Cache: { name: a, ... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet c Delete: { name: b,
... } c a b Cache: { name: a, ... } { name: c, ... }
Node Kubernetes API a kubelet c Cache: { name: a,
... } { name: c, ... } c a b API said to delete pod “b”.
Node Kubernetes API a kubelet c Cache: { name: a,
... } { name: c, ... } c a API said to delete pod “b”.
“But you said edge-triggered is bad!”
It is! But this isn’t edge-triggered.
The cache is updated by events (edges) but we are
still reconciling state
“???”
The controller can be restarted at any time and the
cache will be reconstructed - we can’t “miss an edge*” * modulo bugs, read on
Even if you miss an event, you can still recover
the state
Ultimately it’s all just software, and software has bugs. Controllers
should re-list periodically to get full state...
...but we’ve put a lot of energy into making sure
that our list-watch is reliable.