Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Building systems with etcd @ SCALE x14
Search
Brandon Philips
January 24, 2016
Technology
2
490
Building systems with etcd @ SCALE x14
SCALE x14
Brandon Philips
January 24, 2016
Tweet
Share
More Decks by Brandon Philips
See All by Brandon Philips
Node.js Workflow with Minikube and Skaffold
philips
0
250
Manage the App on Kubernetes
philips
0
330
Production Backbone Monitoring Containerized Apps
philips
0
150
KubeCon EU 2017: Dancing on the Edge of a Volcano
philips
1
680
rkt - KubeCon EU keynote - 2017
philips
1
250
FOSDEM_Keynote_2017-_.pdf
philips
0
110
Tectonic Summit Day 2 Keynote
philips
0
320
Kubernetes: Simple to Manage Anywhere (self-hosted, Tectonic upgrade demo)
philips
0
360
KubeCon Keynote 2016- Distributed Systems Simplified on Kubernetes
philips
2
540
Other Decks in Technology
See All in Technology
MCPに潜むセキュリティリスクを考えてみる
milix_m
1
880
新規事業におけるAIリサーチの活用例
ranxxx
0
170
大規模組織にAIエージェントを迅速に導入するためのセキュリティの勘所 / AI agents for large-scale organizations
i35_267
6
330
AI人生苦節10年で会得したAIがやること_人間がやること.pdf
shibuiwilliam
1
210
Turn Your Community into a Fundraising Catalyst for Black Philanthropy Month
auctria
PRO
0
200
地域コミュニティへの「感謝」と「恩返し」 / 20250726jawsug-tochigi
kasacchiful
0
100
「手を動かした者だけが世界を変える」ソフトウェア開発だけではない開発者人生
onishi
15
7.6k
Tiptapで実現する堅牢で柔軟なエディター開発
kirik
1
150
モバイルゲームの開発を支える基盤の歩み ~再現性のある開発ラインを量産する秘訣~
qualiarts
0
630
Ktor + Google Cloud Tasks/PubSub におけるOTel Messaging計装の実践
sansantech
PRO
1
330
ML Pipelineの開発と運用を OpenTelemetryで繋ぐ @ OpenTelemetry Meetup 2025-07
getty708
0
320
クマ×共生 HACKATHON - 熊対策を『特別な行動」から「生活の一部」に -
pharaohkj
0
200
Featured
See All Featured
Unsuck your backbone
ammeep
671
58k
Intergalactic Javascript Robots from Outer Space
tanoku
271
27k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
8
720
jQuery: Nuts, Bolts and Bling
dougneiner
63
7.8k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
283
13k
GitHub's CSS Performance
jonrohan
1031
460k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
PRO
181
54k
Making the Leap to Tech Lead
cromwellryan
134
9.4k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
48
2.9k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
2.9k
4 Signs Your Business is Dying
shpigford
184
22k
We Have a Design System, Now What?
morganepeng
53
7.7k
Transcript
Fault Tolerant Infrastructure Building Systems with etcd @coreoslinux @brandonphilips
Brandon Philips CTO, CoreOS github.com/philips
What is CoreOS?
What is CoreOS?
None
None
What is CoreOS?
None
The smartest way to run your container infrastructure. tectonic.com @tectonic
QUAY Secure hosting for private Docker repositories quay.io @quayio
Why build CoreOS?
you
you as a sw engineer
your with Ada.Text_IO; procedure Hello_World is use Ada.Text_IO; begin Put_Line("Hello,
world!"); end; #include <stdio.h> int main() { printf("Hello, world!\n"); } package main import "fmt" func main() { fmt.Println("Hello, world!") }
your container image
your /bin/java /opt/app.jar /lib/libc
your /bin/python /opt/app.py /lib/libc
your com.example.app d474e8c57737625c
your d474e8c57737625c Signed By: Alice
you as an ops engineer
your
your com.example.webapp x3
your com.example.webapp x3
your ??? com.example.webapp x3
How do we do it?
architecture in practice cluster operations
worker kubelet worker kubelet worker kubelet scheduler & API worker
kubelet w ku t worker kubelet
machine configuration OS operations
distributed configuration cluster operations
github.com/philips/hacks/tree/master/etcd- demos
etcd
/etc distributed
open source software failure tolerant durable watchable exposed via HTTP
runtime reconfigurable
Data Store API -X GET Get Wait -X PUT Put
Create CAS -X DELETE Delete CAD
clusters etcd basics
Typical Cluster Leader Follower
API etcd basics
fault tolerance etcd basics
Available Leader Follower
Available Leader Follower
Available Leader Follower
Unavailable Leader Follower
leader fault tolerance etcd basics
Available Leader Follower
Available Leader Follower
Temporarily Unavailable Leader Follower
Available Leader Follower
Unavailable Leader Follower
wal, snapshots, backups etcd durability
discovery, static etcd bootstrap
$ curl discovery.etcd.io/new?size=5 discovery.etcd.io/6eadeac2
discovery
discovery
discovery
Leader Follower discovery
live addition and removal etcd reconfig
Leader Follower
Leader Follower
Leader Follower
etcd apps
reboot locksmith etcd apps
None
None
Cluster Wide Reboot Lock • Need to reboot? Decrement the
semaphore key atomically with etcd. • manager.Reboot() and wait... • After reboot increment the semaphore key in etcd atomically.
skydns etcd apps
vulcand etcd apps
None
confd etcd apps
pulling it together kubernetes
k8s/mesos/etc scheduler scheduling
getting work to servers scheduling
$ scp app host:/opt $ ssh host systemd-run /opt/app
$ scp app host:/opt $ ssh host systemd-run /opt/app
$ fab deploy:app
$ fab deploy:app
$ fab deploy:app
$ fab deploy:collector-app
$ fab deploy:collector-app
$ fab deploy:collector-app
$ fab deploy deploy:collector-app
$ fab lowest-loadaverage
$ fab lowest-loadaverage host1
$ fab lowest-loadaverage host1 $ fab -H host1 deploy:job
You Scheduler API Scheduler Machine(s)
while true { todo = diff(desState, curState) schedule(todo) }
while true { todo = diff(desState, curState) schedule(todo) }
while true { todo = diff(desState, curState) schedule(todo) }
while true { todo = diff(desState, curState) schedule(todo) }
dns, LBs, k8s labels services
flexible service discovery k8s labels
pod env=dev app=web pod env=test app=web pod env=prod app=web
pod env=dev app=web pod env=test app=web pod env=prod app=web service
test.example.com select(env=dev,app=web) service beta.example.com select(env=test,app=web) OR select(env=prod,app=web) service example.com select(env=prod,app=web)
github.com/coreos/coreos-kubernetes
etcd.ngrok.io
worker kubelet worker kubelet scheduler & API
worker & API works on 1 node too
coreos.com/careers work with us
@coreoslinux @tectonicstack @brandonphilips thank you