Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Configuration Management 101 @ Scale12x
Search
someara
February 28, 2014
Technology
2
740
Configuration Management 101 @ Scale12x
someara
February 28, 2014
Tweet
Share
More Decks by someara
See All by someara
Docker Docker Docker Chef
someara
0
340
Hand Crafted Artisinal Chef Resources
someara
0
450
Configuration Management Camp 2015
someara
1
420
Cookbook Reusability
someara
0
410
TDI with ChefDK 0.0.1
someara
2
480
Configuration Management 101
someara
3
630
Introduction to Chef - Scale 10x
someara
2
380
Introduction to Chef - NYLUG Jan 2012
someara
2
250
Introduction to Chef - LISA11
someara
10
6k
Other Decks in Technology
See All in Technology
OpenTelemetry を使ったトレースエグザンプラーの活用 / otel-trace-exemplar
k6s4i53rx
2
640
カオナビの利用実績をアウトカムへつなげる旅 / example-of-data-management-startup-in-kaonavi
kaonavi
0
120
Garoon 開発チーム / Garoon development team
cybozuinsideout
PRO
2
2.9k
Janus
bkuhlmann
1
490
[PlatformCon 24] Platform Orchestrators: The Missing Middle of Internal Developer Platforms?
danielbryantuk
1
180
TransitGatewayの基礎
toru_kubota
0
230
株式会社EventHub・エンジニア採用資料
eventhub
0
1.9k
オブザーバビリティの Primary Signals
onk
PRO
0
550
少数チームで挑む: SwiftUI, TCA, KMPを用いた 新規動画配信アプリ 「ABEMA Live」の開発について
tomu28
0
540
o11y入門_外形監視を利用したWebアプリケーションへの最適なモニタリング_TechBrew
k5k
3
100
アプリがつくるNOT A HOTELブランド
hokuts
1
450
Databricks における 『MLOps』
databricksjapan
2
140
Featured
See All Featured
WebSockets: Embracing the real-time Web
robhawkes
59
7k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
356
22k
How To Stay Up To Date on Web Technology
chriscoyier
782
250k
Agile that works and the tools we love
rasmusluckow
324
20k
Building a Scalable Design System with Sketch
lauravandoore
455
32k
Making the Leap to Tech Lead
cromwellryan
123
8.5k
Designing for Performance
lara
601
67k
Design by the Numbers
sachag
274
18k
Scaling GitHub
holman
457
140k
Designing for humans not robots
tammielis
247
25k
Bootstrapping a Software Product
garrettdimon
PRO
301
110k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
501
140k
Transcript
Configuration Management 101! Scale 12x
AKA
Sean Drops the Fucking Science
Sean OMeara!
[email protected]
! @someara
Sean OMeara!
[email protected]
! @someara
Sean OMeara!
[email protected]
! @someara
whoami
Part 1
The Dawn of Configuration Management
• What is configuration management?! • Strategies and techniques for
managing configuration and its complexity! • The art of change management
Manual Configuration
• Intuitive! • How we all start out! • Log
into machine, manipulate with fingers! • Make with the clicky clicky! • Long tradition
• Somehow feels the “safest"! • First instinct in emergencies!
• This is an illusion! • Do not do this
• Labor intensive! • Error prone! • Difficult to reproduce!
• Obviously unsustainable
Scripting
• setup.sh! • setup.pl! • setup.py! • setup.rb
• doit.sh! • doit.pl! • doit.py! • doit.rb
• Ad-hoc in nature! • Loss of history! • Lacks
testing methodology! • A step in the right direction
File Distribution
• NFS! • SMB! • AFS! • SSHFS! • GlusterFS
• uucp! • rcp! • ftp! • http! • scp
•Distributed systems! •Shares often managed manually or with scripts! •Package
repositories! •Pull is better than push! •Scp on a cron *
Execution Management
• Image management! • Snapshots and cloning! • Containers
• SSH on a for loop! • Func! • Commands
on message queues! • ISConf
• Loss of history! • Image sprawl! • Easy to
order change across nodes
Convergent Operators! (promises)
None
Mark Burgess
The rest of us
Tools
• CFEngine! • Bcfg2! • Puppet! • Chef! • Salt!
• Ansible
None
Part 2
Policy http://www.flickr.com/photos/sfllaw/222795669/
• /etc/passwd should be mode 0644! • /etc/shadow should be
mode 0600
• user ‘kermit’ should exist! • user ‘fonzi’ should exist!
• group ‘muppets’ should exist! • group ‘muppets’ should contain kermit and fonzi
• package ‘ntpd’ should be installed! • ntpd should sync
with our AD service! • service ‘ntpd’ should be running
• package ‘httpd’ should be installed! • httpd should be
expose /mnt/software/java! • service ‘httpd’ should be running
• The Java JDK, version 7u45, found on an internally
hosted web server, should be installed into /usr/local/jdk-7u45/
Polices are declarations about the state of things in a
system
Polices are applied repeatedly and repair the system when needed
Policies often change
• package ‘widget-factory’ should be installed at version 1.2.3
• package ‘widget-factory’ should be installed at version 1.3.0
http://www.flickr.com/photos/jakepjohnson/4937767595 Repeatability
Repeatable -> Idempotent -> Convergent
• Scripts are not generally repeatable
None
None
• But they can be!
None
! Idempotent operations can be applied infinite times and will
yield the same result every time
None
Idempotent http://www.flickr.com/photos/ian_munroe/4758240536/
http://www.flickr.com/photos/ian_munroe/4758240536/ Idempotent NOT GOOD ENOUGH
! Convergent operations test state and repair if needed
None
None
! A control loop keeps the system stable and allows
for change when policy is updated
Autonomous agent Policy: The box should be closed
Convergence
None
None
None
Converging with Bash
git clone
[email protected]
:someara/ cbash.git
None
None
None
None
None
None
None
None
None
None
None
None
None
Convergence and Iteration
None
None
None
None
None
None
Does order matter?
YES
None
Promises http://www.flickr.com/photos/nazzen9009/6809694353/
• Agents are autonomous! • A promise is a signal
or message perceived by an observer.! • Promises may or may not be kept.! • Agents can observe other agents! • Agents only have local information *! • Inner workings of agents are assumed to be unknown http://markburgess.org/BookOfPromises.pdf
• Agents have intentions (possible behaviors)! • Agents can make
assessments about other agents http://markburgess.org/BookOfPromises.pdf
• Configuration Management tools embody tenants of Promise Theory intentionally
or not
Domain Specific Languages
! DSLs restrict machine instructions to convergent operations
! DSLs manage ordering
None
type subject intentions
None
type subject intentions
signal
None
type subject intention
observation
None
type subject intentions
None
type intention subject
signal
Intermission
None
Part 3
Composition
None
Recipes
resource one resource two resource three
{ testable intent
recipe[http::server]
recipe[http::server]
recipes supporting files
Types
None
interface implementation
None
intentions parameters
None
new scope intention implementation
new scope intention implementation
Artifacts
metadata
None
metadata
None
http v0.1.0 chef-server api yum v3.0.0
Delivery
• nodes request their own initial run_list
recipe[httpd::server] chef-server api run_list: http v0.1.0
recipe[httpd::server] chef-server api run_list: http v0.1.0 recipe[openssh::server] openssh v3.2.1
recipe[ntp::client] chef-server api run_list: http v0.1.0 recipe[openssh::server] openssh v3.2.1 recipe[httpd::server]
ntp v1.0.0
• Push vs Pull! • Networking considerations! • Machines down
for maintenance! • Machines that don’t exist yet
Dependencies
None
None
None
None
None
None
recipe[widgetfactory] chef-server api run_list: http v0.1.0 yum v3.0.0 widgetfactory v1.0.0
Integration testing
• Test that a set of agents has achieved their
combined goal
• lsof -i :80! • ps -ef | grep httpd!
• curl localhost 2>&1 > /dev/null
• Berkshelf! • Vagrant! • Kitchen.ci! • Bats! • Serverspec
Environments
• Environments constrain cookbook versions! • Environments can set data
None
None
• Environments can be used to test branches! • Environments
can be used to segregate machines! • Environments can be manipulated programatically
None
None
http v0.1.0 chef-server api http v0.2.0 openssh v1.2.3 postgresql v3.2.1
recipe[widgetfactory] run_list: http v0.1.0 yum v3.0.0 widgetfactory v1.0.0 chef_environment: production
recipe[widgetfactory] run_list: http v0.2.0 yum v3.0.0 widgetfactory v1.0.0 chef_environment: staging
Part 4
Clusters http://www.flickr.com/photos/youraccount/5938852370/
Typical Cluster
loadbalancer application db-slave db-master
Production httpd 0.1.0
Production Staging httpd 0.1.0 httpd 0.1.0
Production Staging UUID httpd 0.1.0 httpd 0.1.0 httpd 0.2.0
Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
Production Staging httpd 0.2.0 httpd 0.2.0
Production httpd 0.2.0
An Ordering Problem
loadbalancer application
• Take a machine out of the pool! • Drain
the connections! • Modify configuration! • Insert it back into the pool
loadbalancer application
loadbalancer application
loadbalancer application
loadbalancer application
loadbalancer application
loadbalancer application
loadbalancer application
loadbalancer application
loadbalancer application
loadbalancer application
Orchestration
• Conductor showing signals to autonomous agents (creative policy manipulation)!
• External actor controlling sequencing (execution management)! • Application level sequencing (vector clocks, etc)
• Infrastructures are snowflakes! • Solutions are unique to applications
by nature! • Configuration Management 201
• There is no separation between ‘infrastructure’ and ‘application’! •
Distributed systems are hard! • Specialists need to work together
Devops
• Study Promise Theory! • Study distributed systems! • Develop
high quality primitives! • Be excellent to each other
Fin