Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Building systems with etcd @ SCALE x14
Search
Brandon Philips
January 24, 2016
Technology
2
480
Building systems with etcd @ SCALE x14
SCALE x14
Brandon Philips
January 24, 2016
Tweet
Share
More Decks by Brandon Philips
See All by Brandon Philips
Node.js Workflow with Minikube and Skaffold
philips
0
230
Manage the App on Kubernetes
philips
0
310
Production Backbone Monitoring Containerized Apps
philips
0
130
KubeCon EU 2017: Dancing on the Edge of a Volcano
philips
1
640
rkt - KubeCon EU keynote - 2017
philips
1
240
FOSDEM_Keynote_2017-_.pdf
philips
0
97
Tectonic Summit Day 2 Keynote
philips
0
310
Kubernetes: Simple to Manage Anywhere (self-hosted, Tectonic upgrade demo)
philips
0
320
KubeCon Keynote 2016- Distributed Systems Simplified on Kubernetes
philips
2
530
Other Decks in Technology
See All in Technology
RemoveだらけのPHPUnit 12に備えよう
cocoeyes02
0
260
Iceberg Meetup Japan #1 : Iceberg and Databricks
databricksjapan
0
320
大規模アジャイルフレームワークから学ぶエンジニアマネジメントの本質
staka121
PRO
3
990
開発組織を進化させる!AWSで実践するチームトポロジー
iwamot
1
170
CDKでカスタムランタイムを作成して、Lambdaをnode.js23+TypeScriptで動かしてみた
smt7174
2
100
設計を積み重ねてシステムを刷新する
sansantech
PRO
0
160
Share my, our lessons from the road to re:Invent
naospon
0
140
Exadata Database Service on Cloud@Customer セキュリティ、ネットワーク、および管理について
oracle4engineer
PRO
2
1.5k
AWSを活用したIoTにおけるセキュリティ対策のご紹介
kwskyk
0
330
AIエージェント時代のエンジニアになろう #jawsug #jawsdays2025 / 20250301 Agentic AI Engineering
yoshidashingo
8
3.4k
サイト信頼性エンジニアリングとAmazon Web Services / SRE and AWS
ymotongpoo
7
1.4k
AWSではじめる Web APIテスト実践ガイド / A practical guide to testing Web APIs on AWS
yokawasa
7
620
Featured
See All Featured
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
30
2.2k
A Philosophy of Restraint
colly
203
16k
Reflections from 52 weeks, 52 projects
jeffersonlam
348
20k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
44
7k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
10
1.3k
YesSQL, Process and Tooling at Scale
rocio
172
14k
Imperfection Machines: The Place of Print at Facebook
scottboms
267
13k
For a Future-Friendly Web
brad_frost
176
9.6k
A Modern Web Designer's Workflow
chriscoyier
693
190k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
45
9.4k
Building Adaptive Systems
keathley
40
2.4k
Testing 201, or: Great Expectations
jmmastey
42
7.2k
Transcript
Fault Tolerant Infrastructure Building Systems with etcd @coreoslinux @brandonphilips
Brandon Philips CTO, CoreOS github.com/philips
What is CoreOS?
What is CoreOS?
None
None
What is CoreOS?
None
The smartest way to run your container infrastructure. tectonic.com @tectonic
QUAY Secure hosting for private Docker repositories quay.io @quayio
Why build CoreOS?
you
you as a sw engineer
your with Ada.Text_IO; procedure Hello_World is use Ada.Text_IO; begin Put_Line("Hello,
world!"); end; #include <stdio.h> int main() { printf("Hello, world!\n"); } package main import "fmt" func main() { fmt.Println("Hello, world!") }
your container image
your /bin/java /opt/app.jar /lib/libc
your /bin/python /opt/app.py /lib/libc
your com.example.app d474e8c57737625c
your d474e8c57737625c Signed By: Alice
you as an ops engineer
your
your com.example.webapp x3
your com.example.webapp x3
your ??? com.example.webapp x3
How do we do it?
architecture in practice cluster operations
worker kubelet worker kubelet worker kubelet scheduler & API worker
kubelet w ku t worker kubelet
machine configuration OS operations
distributed configuration cluster operations
github.com/philips/hacks/tree/master/etcd- demos
etcd
/etc distributed
open source software failure tolerant durable watchable exposed via HTTP
runtime reconfigurable
Data Store API -X GET Get Wait -X PUT Put
Create CAS -X DELETE Delete CAD
clusters etcd basics
Typical Cluster Leader Follower
API etcd basics
fault tolerance etcd basics
Available Leader Follower
Available Leader Follower
Available Leader Follower
Unavailable Leader Follower
leader fault tolerance etcd basics
Available Leader Follower
Available Leader Follower
Temporarily Unavailable Leader Follower
Available Leader Follower
Unavailable Leader Follower
wal, snapshots, backups etcd durability
discovery, static etcd bootstrap
$ curl discovery.etcd.io/new?size=5 discovery.etcd.io/6eadeac2
discovery
discovery
discovery
Leader Follower discovery
live addition and removal etcd reconfig
Leader Follower
Leader Follower
Leader Follower
etcd apps
reboot locksmith etcd apps
None
None
Cluster Wide Reboot Lock • Need to reboot? Decrement the
semaphore key atomically with etcd. • manager.Reboot() and wait... • After reboot increment the semaphore key in etcd atomically.
skydns etcd apps
vulcand etcd apps
None
confd etcd apps
pulling it together kubernetes
k8s/mesos/etc scheduler scheduling
getting work to servers scheduling
$ scp app host:/opt $ ssh host systemd-run /opt/app
$ scp app host:/opt $ ssh host systemd-run /opt/app
$ fab deploy:app
$ fab deploy:app
$ fab deploy:app
$ fab deploy:collector-app
$ fab deploy:collector-app
$ fab deploy:collector-app
$ fab deploy deploy:collector-app
$ fab lowest-loadaverage
$ fab lowest-loadaverage host1
$ fab lowest-loadaverage host1 $ fab -H host1 deploy:job
You Scheduler API Scheduler Machine(s)
while true { todo = diff(desState, curState) schedule(todo) }
while true { todo = diff(desState, curState) schedule(todo) }
while true { todo = diff(desState, curState) schedule(todo) }
while true { todo = diff(desState, curState) schedule(todo) }
dns, LBs, k8s labels services
flexible service discovery k8s labels
pod env=dev app=web pod env=test app=web pod env=prod app=web
pod env=dev app=web pod env=test app=web pod env=prod app=web service
test.example.com select(env=dev,app=web) service beta.example.com select(env=test,app=web) OR select(env=prod,app=web) service example.com select(env=prod,app=web)
github.com/coreos/coreos-kubernetes
etcd.ngrok.io
worker kubelet worker kubelet scheduler & API
worker & API works on 1 node too
coreos.com/careers work with us
@coreoslinux @tectonicstack @brandonphilips thank you