Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Kubernetes Controllers - are they loops or events?
Search
Tim Hockin
February 20, 2021
Technology
11
3.6k
Kubernetes Controllers - are they loops or events?
Tim Hockin
February 20, 2021
Tweet
Share
More Decks by Tim Hockin
See All by Tim Hockin
Kubernetes in the 2nd Decade
thockin
0
240
Why Service is the worst API in Kubernetes, and what we can do about it
thockin
2
700
Kubernetes Pod Probes
thockin
6
4.1k
Go Workspaces for Kubernetes
thockin
2
950
Code Review in Kubernetes
thockin
2
1.6k
Multi-cluster: past, present, future
thockin
0
430
Kubernetes Network Models (why is this so dang hard?)
thockin
9
1.7k
KubeCon EU 2020: SIG-Network Intro and Deep-Dive
thockin
8
1.2k
A Non-Technical Kubernetes Talk (KubeCon EU 2020)
thockin
3
570
Other Decks in Technology
See All in Technology
XSS攻撃から考察するAWS設定不備の恐怖/20241012 Hironobu Otaki
shift_evolve
0
180
VPoE Meetup Vol.1 VPoEとして実践してきたことと反省点
coconala_engineer
2
200
本番のトラフィック量でHudiを検証して見えてきた課題
joker1007
2
290
WebRTC と AI の組み合わせ
tnoho
0
390
なぜ Rack を理解すべきかプレトーク / Why should you understand Rack - Pre-talk
hogelog
0
250
自動テストの信頼性を高めるミューテーションテストの活用に向けて
tarappo
2
110
SwiftSyntaxでUIKitとSwiftUIの使用率を完璧に計測できちゃう件について
ldf_tech
0
150
開発健全性の可視化と開発者体験の改善 ~ Compassでエンジニアに活力と生産性を ~
atlassianjapan
0
100
複数の外部サービスデータの統合と変換を実現する Railsのインポートアーキテクチャ / Rails import architecture for integration and transformation of multiple external service data
aiandrox
0
390
JBoss EAPによるクラウドネイティブのススメ
chiroito
0
150
Amplify Gen 2ではじめる 生成AIアプリ開発入門
tsukuboshi
0
410
今日から始める技術的負債の解消
leveragestech
3
460
Featured
See All Featured
Six Lessons from altMBA
skipperchong
26
3.4k
YesSQL, Process and Tooling at Scale
rocio
167
14k
Gamification - CAS2011
davidbonilla
80
5k
Building a Scalable Design System with Sketch
lauravandoore
459
33k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
159
15k
Measuring & Analyzing Core Web Vitals
bluesmoon
0
9
jQuery: Nuts, Bolts and Bling
dougneiner
61
7.5k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
191
16k
How to Think Like a Performance Engineer
csswizardry
18
1k
StorybookのUI Testing Handbookを読んだ
zakiyama
26
5.2k
No one is an island. Learnings from fostering a developers community.
thoeni
19
2.9k
Git: the NoSQL Database
bkeepers
PRO
425
64k
Transcript
Kubernetes Controllers Are they loops or events? Tim Hockin @thockin
v1
Background on “reconciliation”: https://speakerdeck.com/thockin/kubernetes-what-is-reconciliation
Background on “edge vs. level”: https://speakerdeck.com/thockin/edge-vs-level-triggered-logic
Usually when we talk about controllers we refer to them
as a “loop”
Imagine a controller for Pods (aka kubelet). It has 2
jobs: 1) Actuate the pod API 2) Report status on pods
What you’d expect looks something like:
Node Kubernetes API a kubelet b c Get all pods
Node Kubernetes API a kubelet b c { name: a,
... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet b c for each pod
p { if p is running { verify p config } else { start p } gather status }
Node Kubernetes API a kubelet b c Set status c
a b
...then repeat (aka “a poll loop”)
Here’s where it matters
Node Kubernetes API a kubelet b c c a b
kubectl delete pod b
Node Kubernetes API a kubelet c c a b kubectl
delete pod b
Node Kubernetes API a kubelet c Get all pods c
a b
Node Kubernetes API a kubelet c { name: a, ...
} { name: c, ... } c a b
Node Kubernetes API a kubelet c I have “b” but
API doesn’t - delete it! c a b
Node Kubernetes API a kubelet c Set status c a
This is correct level-triggered reconciliation Read desired state, make it
so
Some controllers are implemented this way, but it’s inefficient at
scale
Imagine thousands of controllers (kubelet, kube-proxy, dns, ingress, storage...) polling
continuously
We need to achieve the same behavior more efficiently
We could poll less often, but then it takes a
long (and variable) time to react - not a great UX
Enter the “list-watch” model
Node Kubernetes API a kubelet b c Get all pods
Node Kubernetes API a kubelet b c { name: a,
... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet b c Cache: { name:
a, ... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet b c Watch all pods
Cache: { name: a, ... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet b c Cache: { name:
a, ... } { name: b, ... } { name: c, ... } for each pod p { if p is running { verify p config } else { start p } gather status }
Node Kubernetes API a kubelet b c Set status c
a b Cache: { name: a, ... } { name: b, ... } { name: c, ... }
We trade memory (the cache) for other resources (API server
CPU in particular)
There’s no point in polling my own cache, so what
happens next?
Remember that watch we did earlier? That’s an open stream
for events.
Node Kubernetes API a kubelet b c c a b
kubectl delete pod b Cache: { name: a, ... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet c c a b kubectl
delete pod b Cache: { name: a, ... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet c Delete: { name: b,
... } c a b Cache: { name: a, ... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet c Delete: { name: b,
... } c a b Cache: { name: a, ... } { name: c, ... }
Node Kubernetes API a kubelet c Cache: { name: a,
... } { name: c, ... } c a b API said to delete pod “b”.
Node Kubernetes API a kubelet c Cache: { name: a,
... } { name: c, ... } c a API said to delete pod “b”.
“But you said edge-triggered is bad!”
It is! But this isn’t edge-triggered.
The cache is updated by events (edges) but we are
still reconciling state
“???”
The controller can be restarted at any time and the
cache will be reconstructed - we can’t “miss an edge*” * modulo bugs, read on
Even if you miss an event, you can still recover
the state
Ultimately it’s all just software, and software has bugs. Controllers
should re-list periodically to get full state...
...but we’ve put a lot of energy into making sure
that our list-watch is reliable.