Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Configuration Management 101 @ Scale12x
someara
February 28, 2014
Technology
2
640
Configuration Management 101 @ Scale12x
someara
February 28, 2014
Tweet
Share
More Decks by someara
See All by someara
Docker Docker Docker Chef
someara
0
240
Hand Crafted Artisinal Chef Resources
someara
0
330
Configuration Management Camp 2015
someara
1
320
Cookbook Reusability
someara
0
290
TDI with ChefDK 0.0.1
someara
2
370
Configuration Management 101
someara
3
550
Introduction to Chef - Scale 10x
someara
2
350
Introduction to Chef - NYLUG Jan 2012
someara
2
240
Introduction to Chef - LISA11
someara
10
6k
Other Decks in Technology
See All in Technology
AWS Cloud Forensics & Incident Response
e11i0t_4lders0n
0
420
Deep dive in Reserved Instance ~脳死推奨量購入からの脱却~
kzkmaeda
0
550
AWS re:Invent 2022で発表された新機能を試してみた ~Cloud OperationとSecurity~ / New Cloud Operation and Security Features Announced at AWS reInvent 2022
yuj1osm
1
220
AI Builderについて
miyakemito
1
980
NGINXENG JP#2 - 2-NGINXの動作の詳細
hiropo20
1
140
OVN-Kubernetes-Introduction-ja-2023-01-27.pdf
orimanabu
1
460
Dockerに疲れた人のためのLXDではじめるシステムコンテナ入門
devops_vtj
0
130
NGINXENG JP#2 - 4-NGINX-エンジニアリング勉強会
hiropo20
0
130
ユーザーテストガイドライン VERSION 2.0
kouzoukaikaku
0
1.6k
UEでPLATEAU触ってみた
41h0_shiho
0
240
インフラ技術基礎勉強会 開催概要
toru_kubota
0
180
JAWS-UG 横浜 #54 資料
takakuni
0
220
Featured
See All Featured
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
239
19k
How To Stay Up To Date on Web Technology
chriscoyier
779
250k
Pencils Down: Stop Designing & Start Developing
hursman
114
10k
GitHub's CSS Performance
jonrohan
1020
430k
Build The Right Thing And Hit Your Dates
maggiecrowley
22
1.4k
Build your cross-platform service in a week with App Engine
jlugia
221
17k
Designing for Performance
lara
600
65k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
29
8k
Producing Creativity
orderedlist
PRO
335
38k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
7
590
Web development in the modern age
philhawksworth
197
9.6k
Optimizing for Happiness
mojombo
365
64k
Transcript
Configuration Management 101! Scale 12x
AKA
Sean Drops the Fucking Science
Sean OMeara!
[email protected]
! @someara
Sean OMeara!
[email protected]
! @someara
Sean OMeara!
[email protected]
! @someara
whoami
Part 1
The Dawn of Configuration Management
• What is configuration management?! • Strategies and techniques for
managing configuration and its complexity! • The art of change management
Manual Configuration
• Intuitive! • How we all start out! • Log
into machine, manipulate with fingers! • Make with the clicky clicky! • Long tradition
• Somehow feels the “safest"! • First instinct in emergencies!
• This is an illusion! • Do not do this
• Labor intensive! • Error prone! • Difficult to reproduce!
• Obviously unsustainable
Scripting
• setup.sh! • setup.pl! • setup.py! • setup.rb
• doit.sh! • doit.pl! • doit.py! • doit.rb
• Ad-hoc in nature! • Loss of history! • Lacks
testing methodology! • A step in the right direction
File Distribution
• NFS! • SMB! • AFS! • SSHFS! • GlusterFS
• uucp! • rcp! • ftp! • http! • scp
•Distributed systems! •Shares often managed manually or with scripts! •Package
repositories! •Pull is better than push! •Scp on a cron *
Execution Management
• Image management! • Snapshots and cloning! • Containers
• SSH on a for loop! • Func! • Commands
on message queues! • ISConf
• Loss of history! • Image sprawl! • Easy to
order change across nodes
Convergent Operators! (promises)
None
Mark Burgess
The rest of us
Tools
• CFEngine! • Bcfg2! • Puppet! • Chef! • Salt!
• Ansible
None
Part 2
Policy http://www.flickr.com/photos/sfllaw/222795669/
• /etc/passwd should be mode 0644! • /etc/shadow should be
mode 0600
• user ‘kermit’ should exist! • user ‘fonzi’ should exist!
• group ‘muppets’ should exist! • group ‘muppets’ should contain kermit and fonzi
• package ‘ntpd’ should be installed! • ntpd should sync
with our AD service! • service ‘ntpd’ should be running
• package ‘httpd’ should be installed! • httpd should be
expose /mnt/software/java! • service ‘httpd’ should be running
• The Java JDK, version 7u45, found on an internally
hosted web server, should be installed into /usr/local/jdk-7u45/
Polices are declarations about the state of things in a
system
Polices are applied repeatedly and repair the system when needed
Policies often change
• package ‘widget-factory’ should be installed at version 1.2.3
• package ‘widget-factory’ should be installed at version 1.3.0
http://www.flickr.com/photos/jakepjohnson/4937767595 Repeatability
Repeatable -> Idempotent -> Convergent
• Scripts are not generally repeatable
None
None
• But they can be!
None
! Idempotent operations can be applied infinite times and will
yield the same result every time
None
Idempotent http://www.flickr.com/photos/ian_munroe/4758240536/
http://www.flickr.com/photos/ian_munroe/4758240536/ Idempotent NOT GOOD ENOUGH
! Convergent operations test state and repair if needed
None
None
! A control loop keeps the system stable and allows
for change when policy is updated
Autonomous agent Policy: The box should be closed
Convergence
None
None
None
Converging with Bash
git clone
[email protected]
:someara/ cbash.git
None
None
None
None
None
None
None
None
None
None
None
None
None
Convergence and Iteration
None
None
None
None
None
None
Does order matter?
YES
None
Promises http://www.flickr.com/photos/nazzen9009/6809694353/
• Agents are autonomous! • A promise is a signal
or message perceived by an observer.! • Promises may or may not be kept.! • Agents can observe other agents! • Agents only have local information *! • Inner workings of agents are assumed to be unknown http://markburgess.org/BookOfPromises.pdf
• Agents have intentions (possible behaviors)! • Agents can make
assessments about other agents http://markburgess.org/BookOfPromises.pdf
• Configuration Management tools embody tenants of Promise Theory intentionally
or not
Domain Specific Languages
! DSLs restrict machine instructions to convergent operations
! DSLs manage ordering
None
type subject intentions
None
type subject intentions
signal
None
type subject intention
observation
None
type subject intentions
None
type intention subject
signal
Intermission
None
Part 3
Composition
None
Recipes
resource one resource two resource three
{ testable intent
recipe[http::server]
recipe[http::server]
recipes supporting files
Types
None
interface implementation
None
intentions parameters
None
new scope intention implementation
new scope intention implementation
Artifacts
metadata
None
metadata
None
http v0.1.0 chef-server api yum v3.0.0
Delivery
• nodes request their own initial run_list
recipe[httpd::server] chef-server api run_list: http v0.1.0
recipe[httpd::server] chef-server api run_list: http v0.1.0 recipe[openssh::server] openssh v3.2.1
recipe[ntp::client] chef-server api run_list: http v0.1.0 recipe[openssh::server] openssh v3.2.1 recipe[httpd::server]
ntp v1.0.0
• Push vs Pull! • Networking considerations! • Machines down
for maintenance! • Machines that don’t exist yet
Dependencies
None
None
None
None
None
None
recipe[widgetfactory] chef-server api run_list: http v0.1.0 yum v3.0.0 widgetfactory v1.0.0
Integration testing
• Test that a set of agents has achieved their
combined goal
• lsof -i :80! • ps -ef | grep httpd!
• curl localhost 2>&1 > /dev/null
• Berkshelf! • Vagrant! • Kitchen.ci! • Bats! • Serverspec
Environments
• Environments constrain cookbook versions! • Environments can set data
None
None
• Environments can be used to test branches! • Environments
can be used to segregate machines! • Environments can be manipulated programatically
None
None
http v0.1.0 chef-server api http v0.2.0 openssh v1.2.3 postgresql v3.2.1
recipe[widgetfactory] run_list: http v0.1.0 yum v3.0.0 widgetfactory v1.0.0 chef_environment: production
recipe[widgetfactory] run_list: http v0.2.0 yum v3.0.0 widgetfactory v1.0.0 chef_environment: staging
Part 4
Clusters http://www.flickr.com/photos/youraccount/5938852370/
Typical Cluster
loadbalancer application db-slave db-master
Production httpd 0.1.0
Production Staging httpd 0.1.0 httpd 0.1.0
Production Staging UUID httpd 0.1.0 httpd 0.1.0 httpd 0.2.0
Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
Production Staging httpd 0.2.0 httpd 0.2.0
Production httpd 0.2.0
An Ordering Problem
loadbalancer application
• Take a machine out of the pool! • Drain
the connections! • Modify configuration! • Insert it back into the pool
loadbalancer application
loadbalancer application
loadbalancer application
loadbalancer application
loadbalancer application
loadbalancer application
loadbalancer application
loadbalancer application
loadbalancer application
loadbalancer application
Orchestration
• Conductor showing signals to autonomous agents (creative policy manipulation)!
• External actor controlling sequencing (execution management)! • Application level sequencing (vector clocks, etc)
• Infrastructures are snowflakes! • Solutions are unique to applications
by nature! • Configuration Management 201
• There is no separation between ‘infrastructure’ and ‘application’! •
Distributed systems are hard! • Specialists need to work together
Devops
• Study Promise Theory! • Study distributed systems! • Develop
high quality primitives! • Be excellent to each other
Fin