Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Building systems with etcd @ SCALE x14
Search
Brandon Philips
January 24, 2016
Technology
2
480
Building systems with etcd @ SCALE x14
SCALE x14
Brandon Philips
January 24, 2016
Tweet
Share
More Decks by Brandon Philips
See All by Brandon Philips
Node.js Workflow with Minikube and Skaffold
philips
0
230
Manage the App on Kubernetes
philips
0
320
Production Backbone Monitoring Containerized Apps
philips
0
130
KubeCon EU 2017: Dancing on the Edge of a Volcano
philips
1
640
rkt - KubeCon EU keynote - 2017
philips
1
240
FOSDEM_Keynote_2017-_.pdf
philips
0
97
Tectonic Summit Day 2 Keynote
philips
0
310
Kubernetes: Simple to Manage Anywhere (self-hosted, Tectonic upgrade demo)
philips
0
320
KubeCon Keynote 2016- Distributed Systems Simplified on Kubernetes
philips
2
530
Other Decks in Technology
See All in Technology
データエンジニアリング領域におけるDuckDBのユースケース
chanyou0311
9
2.2k
あなたが人生で成功するための5つの普遍的法則 #jawsug #jawsdays2025 / 20250301 HEROZ
yoshidashingo
2
310
偏光画像処理ライブラリを作った話
elerac
1
170
JAWS DAYS 2025 アーキテクチャ道場 事前説明会 / JAWS DAYS 2025 briefing document
naospon
0
140
急成長する企業で作った、エンジニアが輝ける制度/ 20250227 Rinto Ikenoue
shift_evolve
0
140
Change Managerを活用して本番環境へのセキュアなGUIアクセスを統制する / Control Secure GUI Access to the Production Environment with Change Manager
yuj1osm
0
100
役員・マネージャー・著者・エンジニアそれぞれの立場から見たAWS認定資格
nrinetcom
PRO
4
6.2k
エンジニアリング価値を黒字化する バリューベース戦略を用いた 技術戦略策定の道のり
kzkmaeda
6
2.9k
アジャイルな開発チームでテスト戦略の話は誰がする? / Who Talks About Test Strategy?
ak1210
1
630
Fraxinus00tw assembly manual
fukumay
0
110
OPENLOGI Company Profile for engineer
hr01
1
20k
30→150人のエンジニア組織拡大に伴うアジャイル文化を醸成する役割と取り組みの変化
nagata03
0
190
Featured
See All Featured
Docker and Python
trallard
44
3.3k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
366
25k
The Power of CSS Pseudo Elements
geoffreycrofte
75
5.5k
GraphQLの誤解/rethinking-graphql
sonatard
68
10k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
6
570
How To Stay Up To Date on Web Technology
chriscoyier
790
250k
Building Applications with DynamoDB
mza
93
6.2k
How to Ace a Technical Interview
jacobian
276
23k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
175
52k
Code Reviewing Like a Champion
maltzj
521
39k
For a Future-Friendly Web
brad_frost
176
9.6k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
12
990
Transcript
Fault Tolerant Infrastructure Building Systems with etcd @coreoslinux @brandonphilips
Brandon Philips CTO, CoreOS github.com/philips
What is CoreOS?
What is CoreOS?
None
None
What is CoreOS?
None
The smartest way to run your container infrastructure. tectonic.com @tectonic
QUAY Secure hosting for private Docker repositories quay.io @quayio
Why build CoreOS?
you
you as a sw engineer
your with Ada.Text_IO; procedure Hello_World is use Ada.Text_IO; begin Put_Line("Hello,
world!"); end; #include <stdio.h> int main() { printf("Hello, world!\n"); } package main import "fmt" func main() { fmt.Println("Hello, world!") }
your container image
your /bin/java /opt/app.jar /lib/libc
your /bin/python /opt/app.py /lib/libc
your com.example.app d474e8c57737625c
your d474e8c57737625c Signed By: Alice
you as an ops engineer
your
your com.example.webapp x3
your com.example.webapp x3
your ??? com.example.webapp x3
How do we do it?
architecture in practice cluster operations
worker kubelet worker kubelet worker kubelet scheduler & API worker
kubelet w ku t worker kubelet
machine configuration OS operations
distributed configuration cluster operations
github.com/philips/hacks/tree/master/etcd- demos
etcd
/etc distributed
open source software failure tolerant durable watchable exposed via HTTP
runtime reconfigurable
Data Store API -X GET Get Wait -X PUT Put
Create CAS -X DELETE Delete CAD
clusters etcd basics
Typical Cluster Leader Follower
API etcd basics
fault tolerance etcd basics
Available Leader Follower
Available Leader Follower
Available Leader Follower
Unavailable Leader Follower
leader fault tolerance etcd basics
Available Leader Follower
Available Leader Follower
Temporarily Unavailable Leader Follower
Available Leader Follower
Unavailable Leader Follower
wal, snapshots, backups etcd durability
discovery, static etcd bootstrap
$ curl discovery.etcd.io/new?size=5 discovery.etcd.io/6eadeac2
discovery
discovery
discovery
Leader Follower discovery
live addition and removal etcd reconfig
Leader Follower
Leader Follower
Leader Follower
etcd apps
reboot locksmith etcd apps
None
None
Cluster Wide Reboot Lock • Need to reboot? Decrement the
semaphore key atomically with etcd. • manager.Reboot() and wait... • After reboot increment the semaphore key in etcd atomically.
skydns etcd apps
vulcand etcd apps
None
confd etcd apps
pulling it together kubernetes
k8s/mesos/etc scheduler scheduling
getting work to servers scheduling
$ scp app host:/opt $ ssh host systemd-run /opt/app
$ scp app host:/opt $ ssh host systemd-run /opt/app
$ fab deploy:app
$ fab deploy:app
$ fab deploy:app
$ fab deploy:collector-app
$ fab deploy:collector-app
$ fab deploy:collector-app
$ fab deploy deploy:collector-app
$ fab lowest-loadaverage
$ fab lowest-loadaverage host1
$ fab lowest-loadaverage host1 $ fab -H host1 deploy:job
You Scheduler API Scheduler Machine(s)
while true { todo = diff(desState, curState) schedule(todo) }
while true { todo = diff(desState, curState) schedule(todo) }
while true { todo = diff(desState, curState) schedule(todo) }
while true { todo = diff(desState, curState) schedule(todo) }
dns, LBs, k8s labels services
flexible service discovery k8s labels
pod env=dev app=web pod env=test app=web pod env=prod app=web
pod env=dev app=web pod env=test app=web pod env=prod app=web service
test.example.com select(env=dev,app=web) service beta.example.com select(env=test,app=web) OR select(env=prod,app=web) service example.com select(env=prod,app=web)
github.com/coreos/coreos-kubernetes
etcd.ngrok.io
worker kubelet worker kubelet scheduler & API
worker & API works on 1 node too
coreos.com/careers work with us
@coreoslinux @tectonicstack @brandonphilips thank you