Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
etcd - mission-critical key-value store - CoreO...
Search
Brandon Philips
May 09, 2016
Programming
430
3
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
etcd - mission-critical key-value store - CoreOS Fest 2016
Brandon Philips
May 09, 2016
Transcript
69 slides
Brandon Philips @BrandonPhilips |
[email protected]
etcd - mission-critical key-value store
Uncoordinated Upgrades
... ... ... ... ... ... Unavailable Uncoordinated Upgrades
Motivation CoreOS cluster reboot lock - Decrement a semaphore key
atomically - Reboot and wait... - After reboot increment the semaphore key
3 CoreOS updates coordination
CoreOS updates coordination 3
... CoreOS updates coordination 2
... ... ... CoreOS updates coordination 0
... ... ... CoreOS updates coordination 0
... ... CoreOS updates coordination 0
... ... CoreOS updates coordination 0
... ... CoreOS updates coordination 0
... ... ... CoreOS updates coordination 0
CoreOS updates coordination
Store Application Configuration config
config Start / Restart Start / Restart Store Application Configuration
config Update Store Application Configuration
config Unavailable Store Application Configuration
Requirements Strong Consistency - mutual exclusive at any time for
locking purpose Highly Available - resilient to single points of failure & network partitions Watchable - push configuration updates to application
Requirements CAP - We want CP - We want something
like Paxos
Common problem GFS Paxos Big Table Spanner CFS Chubby Google
- “All” infrastructure relies on Paxos
Common problem Amazon - Replicated log powers ec2 Microsoft -
Boxwood powers storage infrastructure Hadoop - ZooKeeper is the heart of the ecosystem
COMMON PROBLEM #GIFEE and Cloud Native Solution
10,000 Stars on Github 250 contributors Google, Red Hat, EMC,
Cisco, Huawei, Baidu, Alibaba...
THE HEART OF CLOUD NATIVE Kubernetes, Cloud Foundry Diego, Project
Calico, many others
ETCD KEY VALUE STORE Fully Replicated, Highly Available, Consistent
PUT(foo, bar), GET(foo), DELETE(foo) Watch(foo) CAS(foo, bar, bar1) Key-value Operations
DEMO play.etcd.io
Runtime Reconfiguration Point-in-time Backup Extensive Metrics etcd Operationality
ETCD v3 Successor of etcd v2
ETCD v3 Better Performance
ETCD v3 More Efficient APIs
Multi-Version Put(foo, bar) Put(foo, bar1) Put(foo, bar2) Get(foo) -> bar2
Multi-Version Put(foo, bar) Put(foo, bar1) Put(foo, bar2) Get(foo, 1) ->
bar
Tx.If( Compare(Value("foo"), ">", "bar"), Compare(Version("foo"), "=", 2), ... ).Then( Put("ok","true")...
).Else( Put("ok","false")... ).Commit() Mini-Transactions
l = CreateLease(15 * second) Put(foo, bar, l) l.KeepAlive() l.Revoke()
Leases
w = Watch(foo) for { r = w.Recv() print(r.Event) //
PUT print(r.KV) // foo,bar } Streaming Watch
Synchronization LoC
ETCD v2 machine coordination -> O(10k)
ETCD v3 app/container coordination -> O(1M)
Performance 1K keys
Performance Snapshot caused performance degradation etcd2 - 600K keys
Performance etcd2 - 600K keys Snapshot triggered elections
ZooKeeper Performance Non-blocking full snapshot Efficient memory management
Performance ZooKeeper default
Performance Snapshot triggered election ZooKeeper default
Performance Snapshot ZooKeeper default
Performance GC ZooKeeper snapshot disabled
Reliable Performance - Similar to ZooKeeper with snapshot disabled -
Incremental snapshot - No Garbage Collection Pauses - Off-heap storage
Performance etcd3 /ZooKeeper snapshot disabled
Performance etcd3 /ZooKeeper snapshot disabled
Memory 10GB 2.4GB 0.8GB 512MB data - 2M 256B keys
Reliability 99% at small scale is easy - Failure is
infrequent and human manageable 99% at large scale is not enough - Not manageable by humans 99.99% at large scale - Reliable systems at bottom layer
HOW DO WE ACHIEVE RELIABILITY WAL, Snapshots, Testing
Write Ahead Log Append only - Simple is good Rolling
CRC protected - Storage & OSes can be unreliable
Snapshots Torturing DBs for Fun and Profit (OSDI2014) - The
simpler database is safer - LMDB was the winner Boltdb an append only B+Tree - A simpler LMDB written in Go
Testing Clusters Failure Inject failures into running clusters White box
runtime checking - Hash state of the system - Progress of the system
Testing Cluster Health with Failures Issue lock operations across cluster
Ensure the correctness of client library
TESTING CLUSTER dash.etcd.io
etcd/raft Reliability Designed for testability and flexibility Used by large
scale db systems and others - Cockroachdb, TiKV, Dgraph
etcd vs others Do one thing
etcd vs others Only do the One Thing
etcd vs others Do it Really Well
etcd Reliability Do it Really Well
ETCD v3.0 BETA Efficient and Scalable
BETA AVAILABLE TODAY github.com/coreos/etcd
FUTURE WORK Proxy, Caching, Watch Coalescing, Secondary Index
GET INVOLVED github.com/coreos/etcd
Brandon Philips @BrandonPhilips |
[email protected]
etcd - mission-critical key-value store
Thank you!
More Decks by Brandon Philips
See All by Brandon Philips
Node.js Workflow with Minikube and Skaffold
philips
0
310
Manage the App on Kubernetes
philips
0
390
Production Backbone Monitoring Containerized Apps
philips
0
260
KubeCon EU 2017: Dancing on the Edge of a Volcano
philips
1
900
rkt - KubeCon EU keynote - 2017
philips
1
330
FOSDEM_Keynote_2017-_.pdf
philips
0
190
Tectonic Summit Day 2 Keynote
philips
0
440
Kubernetes: Simple to Manage Anywhere (self-hosted, Tectonic upgrade demo)
philips
0
470
KubeCon Keynote 2016- Distributed Systems Simplified on Kubernetes
philips
2
610
Other Decks in Programming
See All in Programming
Python Lambdaにおけるテストしやすい設計とAI支援の活用 / Designing Testable Python Lambda Functions with AI
kitano_yuichi
1
330
Getting Started with Vibe Coding
miccheng
0
230
Kotlin Fest 2026: Kotlin コンパイラに Contribute してみよう! - Kotlin/Wasm で深い再帰を書けるようにした話
ternbusty
1
230
AIが実装を担う時代の「技術設計」と「チームマネジメント」
kamina_zzz
2
270
LLM 101
dalinaum
0
500
Lambda durable functionsでHuman in the Loopを楽に…楽に……
har1101
2
330
エラーチャンネルのボールを拾い続けて鍛えるエンジニアの地力 / phpcon-niigata2026
meihei3
1
230
선언형 햅틱 라이브러리 진동을 소개합니다
l2hyunwoo
1
320
Engineering Biometric UX Across Android Versions | DroidKnights 2026
ryunen344
1
210
Claude Code ハーネスDeep Dive – 君はどこまで制御する? -
takateeen_knj
6
5.9k
DroidKnights 2026 Android A2UI Jetpack Compose 기반의 안전한 AI UI 렌더링 - 박제창
itsmedreamwalker
1
240
疎結合とは何か?ソフトウェア設計における結合バランスについて考える #phpcon_niigata
kajitack
1
230
Featured
See All Featured
Chasing Engaging Ingredients in Design
codingconduct
0
400
How To Stay Up To Date on Web Technology
chriscoyier
790
250k
Documentation Writing (for coders)
carmenintech
77
5.6k
How to Align SEO within the Product Triangle To Get Buy-In & Support - #RIMC
aleyda
2
1.9k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
333
23k
AI: The stuff that nobody shows you
jnunemaker
PRO
12
1.3k
How to train your dragon (web standard)
notwaldorf
96
6.9k
The SEO Collaboration Effect
kristinabergwall1
1
620
Agile that works and the tools we love
rasmusluckow
331
22k
Effective software design: The role of men in debugging patriarchy in IT @ Voxxed Days AMS
baasie
1
610
Jamie Indigo - Trashchat’s Guide to Black Boxes: Technical SEO Tactics for LLMs
techseoconnect
PRO
0
750
The Pragmatic Product Professional
lauravandoore
37
7.5k