Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Kubernetes Controllers - are they loops or events?
Search
Tim Hockin
February 20, 2021
Technology
11
3.7k
Kubernetes Controllers - are they loops or events?
Tim Hockin
February 20, 2021
Tweet
Share
More Decks by Tim Hockin
See All by Tim Hockin
Kubernetes in the 2nd Decade
thockin
0
290
Why Service is the worst API in Kubernetes, and what we can do about it
thockin
2
770
Kubernetes Pod Probes
thockin
6
4.2k
Go Workspaces for Kubernetes
thockin
2
980
Code Review in Kubernetes
thockin
2
1.7k
Multi-cluster: past, present, future
thockin
0
460
Kubernetes Network Models (why is this so dang hard?)
thockin
9
1.8k
KubeCon EU 2020: SIG-Network Intro and Deep-Dive
thockin
8
1.2k
A Non-Technical Kubernetes Talk (KubeCon EU 2020)
thockin
3
580
Other Decks in Technology
See All in Technology
能動的ドメイン名ライフサイクル管理のすゝめ / Practice on Active Domain Name Lifecycle Management
nttcom
0
340
2025年のARグラスの潮流
kotauchisunsun
0
730
「完全に理解したTalk」完全に理解した
segavvy
1
280
LangGraphとFlaskを用いた社内資料検索ボットの実装②Retriever構築編
aoikumadaki
0
100
あなたの人生も変わるかも?AWS認定2つで始まったウソみたいな話
iwamot
1
610
20240513 - 框裡框外_文學院學生如何在AI世代安身立命 @ 淡江大學
dpys
0
640
12 Days of OpenAIから読み解く、生成AI 2025年のトレンド
shunsukeono_am
0
1.1k
チームが毎日小さな変化と適応を続けたら1年間でスケール可能なアジャイルチームができた話 / Building a Scalable Agile Team
kakehashi
1
190
AI×医用画像の現状と可能性_2024年版/AI×medical_imaging_in_japan_2024
tdys13
1
1.3k
OCI技術資料 : ファイル・ストレージ 概要
ocise
3
12k
KMP with Crashlytics
sansantech
PRO
0
140
20240522 - 躍遷創作理念 @ PicCollage Workshop
dpys
0
310
Featured
See All Featured
Bash Introduction
62gerente
610
210k
Stop Working from a Prison Cell
hatefulcrawdad
267
20k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
8
1.2k
Into the Great Unknown - MozCon
thekraken
34
1.6k
Build your cross-platform service in a week with App Engine
jlugia
229
18k
RailsConf 2023
tenderlove
29
970
Learning to Love Humans: Emotional Interface Design
aarron
274
40k
Making the Leap to Tech Lead
cromwellryan
133
9k
The Straight Up "How To Draw Better" Workshop
denniskardys
232
140k
How to Think Like a Performance Engineer
csswizardry
22
1.3k
GraphQLとの向き合い方2022年版
quramy
44
13k
Building a Modern Day E-commerce SEO Strategy
aleyda
38
7k
Transcript
Kubernetes Controllers Are they loops or events? Tim Hockin @thockin
v1
Background on “reconciliation”: https://speakerdeck.com/thockin/kubernetes-what-is-reconciliation
Background on “edge vs. level”: https://speakerdeck.com/thockin/edge-vs-level-triggered-logic
Usually when we talk about controllers we refer to them
as a “loop”
Imagine a controller for Pods (aka kubelet). It has 2
jobs: 1) Actuate the pod API 2) Report status on pods
What you’d expect looks something like:
Node Kubernetes API a kubelet b c Get all pods
Node Kubernetes API a kubelet b c { name: a,
... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet b c for each pod
p { if p is running { verify p config } else { start p } gather status }
Node Kubernetes API a kubelet b c Set status c
a b
...then repeat (aka “a poll loop”)
Here’s where it matters
Node Kubernetes API a kubelet b c c a b
kubectl delete pod b
Node Kubernetes API a kubelet c c a b kubectl
delete pod b
Node Kubernetes API a kubelet c Get all pods c
a b
Node Kubernetes API a kubelet c { name: a, ...
} { name: c, ... } c a b
Node Kubernetes API a kubelet c I have “b” but
API doesn’t - delete it! c a b
Node Kubernetes API a kubelet c Set status c a
This is correct level-triggered reconciliation Read desired state, make it
so
Some controllers are implemented this way, but it’s inefficient at
scale
Imagine thousands of controllers (kubelet, kube-proxy, dns, ingress, storage...) polling
continuously
We need to achieve the same behavior more efficiently
We could poll less often, but then it takes a
long (and variable) time to react - not a great UX
Enter the “list-watch” model
Node Kubernetes API a kubelet b c Get all pods
Node Kubernetes API a kubelet b c { name: a,
... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet b c Cache: { name:
a, ... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet b c Watch all pods
Cache: { name: a, ... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet b c Cache: { name:
a, ... } { name: b, ... } { name: c, ... } for each pod p { if p is running { verify p config } else { start p } gather status }
Node Kubernetes API a kubelet b c Set status c
a b Cache: { name: a, ... } { name: b, ... } { name: c, ... }
We trade memory (the cache) for other resources (API server
CPU in particular)
There’s no point in polling my own cache, so what
happens next?
Remember that watch we did earlier? That’s an open stream
for events.
Node Kubernetes API a kubelet b c c a b
kubectl delete pod b Cache: { name: a, ... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet c c a b kubectl
delete pod b Cache: { name: a, ... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet c Delete: { name: b,
... } c a b Cache: { name: a, ... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet c Delete: { name: b,
... } c a b Cache: { name: a, ... } { name: c, ... }
Node Kubernetes API a kubelet c Cache: { name: a,
... } { name: c, ... } c a b API said to delete pod “b”.
Node Kubernetes API a kubelet c Cache: { name: a,
... } { name: c, ... } c a API said to delete pod “b”.
“But you said edge-triggered is bad!”
It is! But this isn’t edge-triggered.
The cache is updated by events (edges) but we are
still reconciling state
“???”
The controller can be restarted at any time and the
cache will be reconstructed - we can’t “miss an edge*” * modulo bugs, read on
Even if you miss an event, you can still recover
the state
Ultimately it’s all just software, and software has bugs. Controllers
should re-list periodically to get full state...
...but we’ve put a lot of energy into making sure
that our list-watch is reliable.