Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Kubernetes Controllers - are they loops or events?
Tim Hockin
February 20, 2021
Technology
10
2.4k
Kubernetes Controllers - are they loops or events?
Tim Hockin
February 20, 2021
Tweet
Share
More Decks by Tim Hockin
See All by Tim Hockin
Kubernetes Network Models (why is this so dang hard?)
thockin
9
1.2k
KubeCon EU 2020: SIG-Network Intro and Deep-Dive
thockin
8
980
A Non-Technical Kubernetes Talk (KubeCon EU 2020)
thockin
3
440
Bringing Traffic Into Your Kubernetes Cluster
thockin
44
9.8k
Kubernetes and Networks - why is this so dang hard?
thockin
62
52k
How to convert strings (and how not to). AKA Escaping is harder than it seems.
thockin
25
1.6k
Why Community Matters
thockin
0
190
Bridging a Multi-Cloud Environment
thockin
3
470
The Long Road to Dual-Stack IPv6 in Kubernetes
thockin
1
220
Other Decks in Technology
See All in Technology
疎ベクトル検索と密ベクトル検索: 第68回 Machine Learning 15minutes! Broadcast
keyakkie
1
250
私のAWS愛を聞け!ここが好きだよAmazon FSx for NetApp ONTAP
non97
0
740
Amplifyで Webアプリケーションの 堅固な土台をサクッと構築する方法
kawasakiteruo
0
210
DBRE 活動と information_schema
_awache
0
250
EKS AnywhereとIAM Anywhereを組み合わせてみた
regmarmcem
0
260
JAWS-UG 朝会 #36 登壇資料
takakuni
1
540
AutoMLを利用した機械学習モデル構築時に意識すること
sbtechnight
0
150
IBM Cloud Festa Online 2022 Summer
1ftseabass
PRO
0
190
大声で伝えたい!定時に帰る方法
sbtechnight
0
220
Red Hat Partner Training Portal のご紹介 / Red Hat Partner Training Portal Introduction
rhpej
0
110
20220803投資先CXO候補者向け 会社紹介資料_合同会社BLUEPRINT
hik
0
250
Power BI のうらがわ
hanaseleb
1
130
Featured
See All Featured
Atom: Resistance is Futile
akmur
255
20k
Imperfection Machines: The Place of Print at Facebook
scottboms
253
12k
Creatively Recalculating Your Daily Design Routine
revolveconf
207
10k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
269
12k
Intergalactic Javascript Robots from Outer Space
tanoku
260
25k
GraphQLとの向き合い方2022年版
quramy
16
8.4k
The MySQL Ecosystem @ GitHub 2015
samlambert
239
11k
The Illustrated Children's Guide to Kubernetes
chrisshort
18
40k
The Web Native Designer (August 2011)
paulrobertlloyd
75
2k
ParisWeb 2013: Learning to Love: Crash Course in Emotional UX Design
dotmariusz
100
5.9k
How STYLIGHT went responsive
nonsquared
85
4k
The Brand Is Dead. Long Live the Brand.
mthomps
46
2.7k
Transcript
Kubernetes Controllers Are they loops or events? Tim Hockin @thockin
v1
Background on “reconciliation”: https://speakerdeck.com/thockin/kubernetes-what-is-reconciliation
Background on “edge vs. level”: https://speakerdeck.com/thockin/edge-vs-level-triggered-logic
Usually when we talk about controllers we refer to them
as a “loop”
Imagine a controller for Pods (aka kubelet). It has 2
jobs: 1) Actuate the pod API 2) Report status on pods
What you’d expect looks something like:
Node Kubernetes API a kubelet b c Get all pods
Node Kubernetes API a kubelet b c { name: a,
... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet b c for each pod
p { if p is running { verify p config } else { start p } gather status }
Node Kubernetes API a kubelet b c Set status c
a b
...then repeat (aka “a poll loop”)
Here’s where it matters
Node Kubernetes API a kubelet b c c a b
kubectl delete pod b
Node Kubernetes API a kubelet c c a b kubectl
delete pod b
Node Kubernetes API a kubelet c Get all pods c
a b
Node Kubernetes API a kubelet c { name: a, ...
} { name: c, ... } c a b
Node Kubernetes API a kubelet c I have “b” but
API doesn’t - delete it! c a b
Node Kubernetes API a kubelet c Set status c a
This is correct level-triggered reconciliation Read desired state, make it
so
Some controllers are implemented this way, but it’s inefficient at
scale
Imagine thousands of controllers (kubelet, kube-proxy, dns, ingress, storage...) polling
continuously
We need to achieve the same behavior more efficiently
We could poll less often, but then it takes a
long (and variable) time to react - not a great UX
Enter the “list-watch” model
Node Kubernetes API a kubelet b c Get all pods
Node Kubernetes API a kubelet b c { name: a,
... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet b c Cache: { name:
a, ... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet b c Watch all pods
Cache: { name: a, ... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet b c Cache: { name:
a, ... } { name: b, ... } { name: c, ... } for each pod p { if p is running { verify p config } else { start p } gather status }
Node Kubernetes API a kubelet b c Set status c
a b Cache: { name: a, ... } { name: b, ... } { name: c, ... }
We trade memory (the cache) for other resources (API server
CPU in particular)
There’s no point in polling my own cache, so what
happens next?
Remember that watch we did earlier? That’s an open stream
for events.
Node Kubernetes API a kubelet b c c a b
kubectl delete pod b Cache: { name: a, ... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet c c a b kubectl
delete pod b Cache: { name: a, ... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet c Delete: { name: b,
... } c a b Cache: { name: a, ... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet c Delete: { name: b,
... } c a b Cache: { name: a, ... } { name: c, ... }
Node Kubernetes API a kubelet c Cache: { name: a,
... } { name: c, ... } c a b API said to delete pod “b”.
Node Kubernetes API a kubelet c Cache: { name: a,
... } { name: c, ... } c a API said to delete pod “b”.
“But you said edge-triggered is bad!”
It is! But this isn’t edge-triggered.
The cache is updated by events (edges) but we are
still reconciling state
“???”
The controller can be restarted at any time and the
cache will be reconstructed - we can’t “miss an edge*” * modulo bugs, read on
Even if you miss an event, you can still recover
the state
Ultimately it’s all just software, and software has bugs. Controllers
should re-list periodically to get full state...
...but we’ve put a lot of energy into making sure
that our list-watch is reliable.