Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Building systems with etcd @ SCALE x14
Search
Brandon Philips
January 24, 2016
Technology
2
490
Building systems with etcd @ SCALE x14
SCALE x14
Brandon Philips
January 24, 2016
Tweet
Share
More Decks by Brandon Philips
See All by Brandon Philips
Node.js Workflow with Minikube and Skaffold
philips
0
250
Manage the App on Kubernetes
philips
0
330
Production Backbone Monitoring Containerized Apps
philips
0
140
KubeCon EU 2017: Dancing on the Edge of a Volcano
philips
1
670
rkt - KubeCon EU keynote - 2017
philips
1
240
FOSDEM_Keynote_2017-_.pdf
philips
0
110
Tectonic Summit Day 2 Keynote
philips
0
320
Kubernetes: Simple to Manage Anywhere (self-hosted, Tectonic upgrade demo)
philips
0
340
KubeCon Keynote 2016- Distributed Systems Simplified on Kubernetes
philips
2
540
Other Decks in Technology
See All in Technology
Navigation3でViewModelにデータを渡す方法
mikanichinose
0
220
PHP開発者のためのSOLID原則再入門 #phpcon / PHP Conference Japan 2025
shogogg
4
750
Javaで作る RAGを活用した Q&Aアプリケーション
recruitengineers
PRO
1
110
Welcome to the LLM Club
koic
0
170
Oracle Audit Vault and Database Firewall 20 概要
oracle4engineer
PRO
3
1.7k
登壇ネタの見つけ方 / How to find talk topics
pinkumohikan
5
480
変化する開発、進化する体系時代に適応するソフトウェアエンジニアの知識と考え方(JaSST'25 Kansai)
mizunori
1
220
製造業からパッケージ製品まで、あらゆる領域をカバー!生成AIを利用したテストシナリオ生成 / 20250627 Suguru Ishii
shift_evolve
PRO
1
140
Claude Code Actionを使ったコード品質改善の取り組み
potix2
PRO
6
2.3k
Witchcraft for Memory
pocke
1
350
Snowflake Summit 2025 データエンジニアリング関連新機能紹介 / Snowflake Summit 2025 What's New about Data Engineering
tiltmax3
0
310
2年でここまで成長!AWSで育てたAI Slack botの軌跡
iwamot
PRO
4
710
Featured
See All Featured
Designing for humans not robots
tammielis
253
25k
Statistics for Hackers
jakevdp
799
220k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
34
5.9k
jQuery: Nuts, Bolts and Bling
dougneiner
63
7.8k
Build your cross-platform service in a week with App Engine
jlugia
231
18k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
8
790
Building a Modern Day E-commerce SEO Strategy
aleyda
42
7.3k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
667
120k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
2.9k
Done Done
chrislema
184
16k
BBQ
matthewcrist
89
9.7k
Side Projects
sachag
455
42k
Transcript
Fault Tolerant Infrastructure Building Systems with etcd @coreoslinux @brandonphilips
Brandon Philips CTO, CoreOS github.com/philips
What is CoreOS?
What is CoreOS?
None
None
What is CoreOS?
None
The smartest way to run your container infrastructure. tectonic.com @tectonic
QUAY Secure hosting for private Docker repositories quay.io @quayio
Why build CoreOS?
you
you as a sw engineer
your with Ada.Text_IO; procedure Hello_World is use Ada.Text_IO; begin Put_Line("Hello,
world!"); end; #include <stdio.h> int main() { printf("Hello, world!\n"); } package main import "fmt" func main() { fmt.Println("Hello, world!") }
your container image
your /bin/java /opt/app.jar /lib/libc
your /bin/python /opt/app.py /lib/libc
your com.example.app d474e8c57737625c
your d474e8c57737625c Signed By: Alice
you as an ops engineer
your
your com.example.webapp x3
your com.example.webapp x3
your ??? com.example.webapp x3
How do we do it?
architecture in practice cluster operations
worker kubelet worker kubelet worker kubelet scheduler & API worker
kubelet w ku t worker kubelet
machine configuration OS operations
distributed configuration cluster operations
github.com/philips/hacks/tree/master/etcd- demos
etcd
/etc distributed
open source software failure tolerant durable watchable exposed via HTTP
runtime reconfigurable
Data Store API -X GET Get Wait -X PUT Put
Create CAS -X DELETE Delete CAD
clusters etcd basics
Typical Cluster Leader Follower
API etcd basics
fault tolerance etcd basics
Available Leader Follower
Available Leader Follower
Available Leader Follower
Unavailable Leader Follower
leader fault tolerance etcd basics
Available Leader Follower
Available Leader Follower
Temporarily Unavailable Leader Follower
Available Leader Follower
Unavailable Leader Follower
wal, snapshots, backups etcd durability
discovery, static etcd bootstrap
$ curl discovery.etcd.io/new?size=5 discovery.etcd.io/6eadeac2
discovery
discovery
discovery
Leader Follower discovery
live addition and removal etcd reconfig
Leader Follower
Leader Follower
Leader Follower
etcd apps
reboot locksmith etcd apps
None
None
Cluster Wide Reboot Lock • Need to reboot? Decrement the
semaphore key atomically with etcd. • manager.Reboot() and wait... • After reboot increment the semaphore key in etcd atomically.
skydns etcd apps
vulcand etcd apps
None
confd etcd apps
pulling it together kubernetes
k8s/mesos/etc scheduler scheduling
getting work to servers scheduling
$ scp app host:/opt $ ssh host systemd-run /opt/app
$ scp app host:/opt $ ssh host systemd-run /opt/app
$ fab deploy:app
$ fab deploy:app
$ fab deploy:app
$ fab deploy:collector-app
$ fab deploy:collector-app
$ fab deploy:collector-app
$ fab deploy deploy:collector-app
$ fab lowest-loadaverage
$ fab lowest-loadaverage host1
$ fab lowest-loadaverage host1 $ fab -H host1 deploy:job
You Scheduler API Scheduler Machine(s)
while true { todo = diff(desState, curState) schedule(todo) }
while true { todo = diff(desState, curState) schedule(todo) }
while true { todo = diff(desState, curState) schedule(todo) }
while true { todo = diff(desState, curState) schedule(todo) }
dns, LBs, k8s labels services
flexible service discovery k8s labels
pod env=dev app=web pod env=test app=web pod env=prod app=web
pod env=dev app=web pod env=test app=web pod env=prod app=web service
test.example.com select(env=dev,app=web) service beta.example.com select(env=test,app=web) OR select(env=prod,app=web) service example.com select(env=prod,app=web)
github.com/coreos/coreos-kubernetes
etcd.ngrok.io
worker kubelet worker kubelet scheduler & API
worker & API works on 1 node too
coreos.com/careers work with us
@coreoslinux @tectonicstack @brandonphilips thank you