Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Kubernetes Controllers - are they loops or events?
Search
Tim Hockin
February 20, 2021
Technology
4.3k
11
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Kubernetes Controllers - are they loops or events?
Tim Hockin
February 20, 2021
Transcript
47 slides
Kubernetes Controllers Are they loops or events? Tim Hockin @thockin
v1
Background on “reconciliation”: https://speakerdeck.com/thockin/kubernetes-what-is-reconciliation
Background on “edge vs. level”: https://speakerdeck.com/thockin/edge-vs-level-triggered-logic
Usually when we talk about controllers we refer to them
as a “loop”
Imagine a controller for Pods (aka kubelet). It has 2
jobs: 1) Actuate the pod API 2) Report status on pods
What you’d expect looks something like:
Node Kubernetes API a kubelet b c Get all pods
Node Kubernetes API a kubelet b c { name: a,
... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet b c for each pod
p { if p is running { verify p config } else { start p } gather status }
Node Kubernetes API a kubelet b c Set status c
a b
...then repeat (aka “a poll loop”)
Here’s where it matters
Node Kubernetes API a kubelet b c c a b
kubectl delete pod b
Node Kubernetes API a kubelet c c a b kubectl
delete pod b
Node Kubernetes API a kubelet c Get all pods c
a b
Node Kubernetes API a kubelet c { name: a, ...
} { name: c, ... } c a b
Node Kubernetes API a kubelet c I have “b” but
API doesn’t - delete it! c a b
Node Kubernetes API a kubelet c Set status c a
This is correct level-triggered reconciliation Read desired state, make it
so
Some controllers are implemented this way, but it’s inefficient at
scale
Imagine thousands of controllers (kubelet, kube-proxy, dns, ingress, storage...) polling
continuously
We need to achieve the same behavior more efficiently
We could poll less often, but then it takes a
long (and variable) time to react - not a great UX
Enter the “list-watch” model
Node Kubernetes API a kubelet b c Get all pods
Node Kubernetes API a kubelet b c { name: a,
... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet b c Cache: { name:
a, ... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet b c Watch all pods
Cache: { name: a, ... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet b c Cache: { name:
a, ... } { name: b, ... } { name: c, ... } for each pod p { if p is running { verify p config } else { start p } gather status }
Node Kubernetes API a kubelet b c Set status c
a b Cache: { name: a, ... } { name: b, ... } { name: c, ... }
We trade memory (the cache) for other resources (API server
CPU in particular)
There’s no point in polling my own cache, so what
happens next?
Remember that watch we did earlier? That’s an open stream
for events.
Node Kubernetes API a kubelet b c c a b
kubectl delete pod b Cache: { name: a, ... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet c c a b kubectl
delete pod b Cache: { name: a, ... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet c Delete: { name: b,
... } c a b Cache: { name: a, ... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet c Delete: { name: b,
... } c a b Cache: { name: a, ... } { name: c, ... }
Node Kubernetes API a kubelet c Cache: { name: a,
... } { name: c, ... } c a b API said to delete pod “b”.
Node Kubernetes API a kubelet c Cache: { name: a,
... } { name: c, ... } c a API said to delete pod “b”.
“But you said edge-triggered is bad!”
It is! But this isn’t edge-triggered.
The cache is updated by events (edges) but we are
still reconciling state
“???”
The controller can be restarted at any time and the
cache will be reconstructed - we can’t “miss an edge*” * modulo bugs, read on
Even if you miss an event, you can still recover
the state
Ultimately it’s all just software, and software has bugs. Controllers
should re-list periodically to get full state...
...but we’ve put a lot of energy into making sure
that our list-watch is reliable.
More Decks by Tim Hockin
See All by Tim Hockin
Kubernetes in the 2nd Decade
thockin
0
600
Why Service is the worst API in Kubernetes, and what we can do about it
thockin
2
1.2k
Kubernetes Pod Probes
thockin
6
5.1k
Go Workspaces for Kubernetes
thockin
2
1.2k
Code Review in Kubernetes
thockin
2
2k
Multi-cluster: past, present, future
thockin
0
630
Kubernetes Network Models (why is this so dang hard?)
thockin
9
2.1k
KubeCon EU 2020: SIG-Network Intro and Deep-Dive
thockin
8
1.4k
A Non-Technical Kubernetes Talk (KubeCon EU 2020)
thockin
3
690
Other Decks in Technology
See All in Technology
E2E Test というハーネス — AI時代の自動テスト民主化
daishukojima
10
6k
個人ASを取得して、自宅Kubernetes上のVMをグローバルIPで公開した話
munenick
0
220
ブース設計、 お前もプロダクトだったのか / So Booth Design Was a Product All Along
akihiro0038
0
140
NutanixへのSSHアクセスの今後について
shadowhat
0
150
おうち環境でも「なるべく止めたくない!」 ― おうち環境の機器冗長化の理想と現実
yuutosi
1
640
コーディングできない僕にBobが舞い降りた
shumpei3
0
180
こんなに変わった! AWSネットワークの常識
shotashiratori
16
11k
マネージャーをエスパーにしない マネジメントされる側から始める、働きやすい現場づくり
comi190327
1
850
異夢同船〜異なる夢でも同じ船でなんとかする〜
fortegp05
1
580
成長するID管理基盤における運用もコストも犠牲にしないオブザーバビリティ向上の実践 / Improving Observability for a Growing Identity Platform Without Adding Operational Burden or Cost
kaminashi
1
110
Del vibe coding a un Desarrollo (agéntico) fiable
jcabot
0
200
20261113_JAWS_Osaka
takuyay0ne
1
600
Featured
See All Featured
Skip the Path - Find Your Career Trail
mkilby
1
300
Dominate Local Search Results - an insider guide to GBP, reviews, and Local SEO
greggifford
PRO
0
410
The Curious Case for Waylosing
cassininazir
1
620
Bootstrapping a Software Product
garrettdimon
PRO
306
120k
The untapped power of vector embeddings
frankvandijk
2
2k
Faster Mobile Websites
deanohume
310
32k
How GitHub (no longer) Works
holman
316
150k
Google's AI Overviews - The New Search
badams
0
1.7k
Embracing the Ebb and Flow
colly
88
5.2k
What’s in a name? Adding method to the madness
productmarketing
PRO
24
4.2k
A designer walks into a library…
pauljervisheath
211
25k
DBのスキルで生き残る技術 - AI時代におけるテーブル設計の勘所
soudai
PRO
68
61k