Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
etcd - mission-critical key-value store - OSCON...
Search
Brandon Philips
May 20, 2016
Programming
0
170
etcd - mission-critical key-value store - OSCON 2016
Brandon Philips
May 20, 2016
Tweet
Share
More Decks by Brandon Philips
See All by Brandon Philips
Node.js Workflow with Minikube and Skaffold
philips
0
250
Manage the App on Kubernetes
philips
0
330
Production Backbone Monitoring Containerized Apps
philips
0
160
KubeCon EU 2017: Dancing on the Edge of a Volcano
philips
1
690
rkt - KubeCon EU keynote - 2017
philips
1
250
FOSDEM_Keynote_2017-_.pdf
philips
0
120
Tectonic Summit Day 2 Keynote
philips
0
330
Kubernetes: Simple to Manage Anywhere (self-hosted, Tectonic upgrade demo)
philips
0
370
KubeCon Keynote 2016- Distributed Systems Simplified on Kubernetes
philips
2
540
Other Decks in Programming
See All in Programming
ワープロって実は計算機で
pepepper
2
1.4k
20250808_AIAgent勉強会_ClaudeCodeデータ分析の実運用〜競馬を題材に回収率100%の先を目指すメソッドとは〜
kkakeru
0
200
CSC305 Summer Lecture 05
javiergs
PRO
0
110
Google I/O recap web編 大分Web祭り2025
kponda
0
2.9k
実践!App Intents対応
yuukiw00w
1
350
『リコリス・リコイル』に学ぶ!! 〜キャリア戦略における計画的偶発性理論と変わる勇気の重要性〜
wanko_it
1
590
一人でAIプロダクトを作るための工夫 〜技術選定・開発プロセス編〜 / I want AI to work harder
rkaga
13
2.8k
Dart 参戦!!静的型付き言語界の隠れた実力者
kno3a87
0
210
未来を拓くAI技術〜エージェント開発とAI駆動開発〜
leveragestech
2
180
令和最新版手のひらコンピュータ
koba789
14
8k
MCPで実現するAIエージェント駆動のNext.jsアプリデバッグ手法
nyatinte
6
840
AI時代のドメイン駆動設計-DDD実践におけるAI活用のあり方 / ddd-in-ai-era
minodriven
23
9k
Featured
See All Featured
How GitHub (no longer) Works
holman
315
140k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
36
2.5k
Building Adaptive Systems
keathley
43
2.7k
4 Signs Your Business is Dying
shpigford
184
22k
Testing 201, or: Great Expectations
jmmastey
45
7.6k
Why You Should Never Use an ORM
jnunemaker
PRO
59
9.5k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
4k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
48
9.6k
Measuring & Analyzing Core Web Vitals
bluesmoon
9
570
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
44
2.4k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
139
34k
The World Runs on Bad Software
bkeepers
PRO
70
11k
Transcript
Brandon Philips @BrandonPhilips |
[email protected]
etcd - mission-critical key-value store
Demos https://github.com/philips/2016-OSCON-etcd
Uncoordinated Upgrades
... ... ... ... ... ... Unavailable Uncoordinated Upgrades
Motivation CoreOS cluster reboot lock - Decrement a semaphore key
atomically - Reboot and wait... - After reboot increment the semaphore key
3 CoreOS updates coordination
CoreOS updates coordination 3
... CoreOS updates coordination 2
... ... ... CoreOS updates coordination 0
... ... ... CoreOS updates coordination 0
... ... CoreOS updates coordination 0
... ... CoreOS updates coordination 0
... ... CoreOS updates coordination 1
... ... ... CoreOS updates coordination 0
CoreOS updates coordination
Store Application Configuration config
config Start / Restart Start / Restart Store Application Configuration
config Update Store Application Configuration
config Unavailable Store Application Configuration
Requirements Strong Consistency - mutual exclusive at any time for
locking purpose Highly Available - resilient to single points of failure & network partitions Watchable - push configuration updates to application
Requirements CAP - We want CP - We want something
like Paxos
Common problem GFS Paxos Big Table Spanner CFS Chubby Google
- “All” infrastructure relies on Paxos
Common problem Amazon - Replicated log powers ec2 Microsoft -
Boxwood powers storage infrastructure Hadoop - ZooKeeper is the heart of the ecosystem
COMMON PROBLEM #GIFEE and Cloud Native Solution
10,000 Stars on Github 250 contributors Google, Red Hat, EMC,
Cisco, Huawei, Baidu, Alibaba...
THE HEART OF CLOUD NATIVE Kubernetes, Cloud Foundry Diego, Project
Calico, many others
ETCD KEY VALUE STORE Fully Replicated, Highly Available, Consistent
PUT(foo, bar), GET(foo), DELETE(foo) Watch(foo) CAS(foo, bar, bar1) Key-value Operations
DEMO play.etcd.io
Runtime Reconfiguration Point-in-time Backup Extensive Metrics etcd Operationality
ETCD v3 Successor of etcd v2
ETCD v3 Better Performance
ETCD v3 More Efficient APIs
Multi-Version Put(foo, bar) Put(foo, bar1) Put(foo, bar2) Get(foo) -> bar2
Multi-Version Put(foo, bar) Put(foo, bar1) Put(foo, bar2) Get(foo, 1) ->
bar
Tx.If( Compare(Value("foo"), ">", "bar"), Compare(Version("foo"), "=", 2), ... ).Then( Put("ok","true")...
).Else( Put("ok","false")... ).Commit() Mini-Transactions
l = CreateLease(15 * second) Put(foo, bar, l) l.KeepAlive() l.Revoke()
Leases
w = Watch(foo) for { r = w.Recv() print(r.Event) //
PUT print(r.KV) // foo,bar } Streaming Watch
Synchronization LoC
ETCD v2 machine coordination -> O(10k)
ETCD v3 app/container coordination -> O(1M)
Performance 1K keys
Performance Snapshot caused performance degradation etcd2 - 600K keys
Performance etcd2 - 600K keys Snapshot triggered elections
ZooKeeper Performance Non-blocking full snapshot Efficient memory management
Performance ZooKeeper default
Performance Snapshot triggered election ZooKeeper default
Performance Snapshot ZooKeeper default
Performance GC ZooKeeper snapshot disabled
Reliable Performance - Similar to ZooKeeper with snapshot disabled -
Incremental snapshot - No Garbage Collection Pauses - Off-heap storage
Performance etcd3 /ZooKeeper snapshot disabled
Performance etcd3 /ZooKeeper snapshot disabled
Memory 10GB 2.4GB 0.8GB 512MB data - 2M 256B keys
Reliability 99% at small scale is easy - Failure is
infrequent and human manageable 99% at large scale is not enough - Not manageable by humans 99.99% at large scale - Reliable systems at bottom layer
HOW DO WE ACHIEVE RELIABILITY WAL, Snapshots, Testing
Write Ahead Log Append only - Simple is good Rolling
CRC protected - Storage & OSes can be unreliable
Snapshots Torturing DBs for Fun and Profit (OSDI2014) - The
simpler database is safer - LMDB was the winner Boltdb an append only B+Tree - A simpler LMDB written in Go
Testing Clusters Failure Inject failures into running clusters White box
runtime checking - Hash state of the system - Progress of the system
Testing Cluster Health with Failures Issue lock operations across cluster
Ensure the correctness of client library
TESTING CLUSTER dash.etcd.io
etcd/raft Reliability Designed for testability and flexibility Used by large
scale db systems and others - Cockroachdb, TiKV, Dgraph
etcd vs others Do one thing
etcd vs others Only do the One Thing
etcd vs others Do it Really Well
etcd Reliability Do it Really Well
ETCD v3.0 BETA Efficient and Scalable
BETA AVAILABLE TODAY github.com/coreos/etcd
FUTURE WORK Proxy, Caching, Watch Coalescing, Secondary Index
GET INVOLVED github.com/coreos/etcd
The smartest way to run your container infrastructure. tectonic.com @tectonic
QUAY Secure hosting for private Docker repositories quay.io @quayio
Brandon Philips @BrandonPhilips |
[email protected]
etcd - mission-critical key-value store
Thank you!