Slide 1
Slide 1 text
ConfigMgmtCamp 2020
Ryn Daniels they/them @rynchantress
untitled config game
Slide 2
Slide 2 text
It's a lovely morning in the
data center, and you are a
horrible goose devop.
@rynchantress ConfigMgmtCamp
Slide 3
Slide 3 text
•Professional computerer
•Knows several things about Terraform
•Very good at puns
Me now...
@rynchantress ConfigMgmtCamp
Slide 4
Slide 4 text
• 200 beauBful and unique snowflake servers
• 0 automaBon
• 1 very sad new sysadmin
Back in the day...
@rynchantress ConfigMgmtCamp
Slide 5
Slide 5 text
Computers were a mistake
Computers are bad, actually
@rynchantress ConfigMgmtCamp
Slide 6
Slide 6 text
• ApplicaBon state
• System state
Let's talk about state
@rynchantress ConfigMgmtCamp
Slide 7
Slide 7 text
• ApplicaBon state
• System state
Let's talk about state
@rynchantress ConfigMgmtCamp
Slide 8
Slide 8 text
• What is the system doing?
• What is the system supposed to be doing?
• What versions of things exist in the system?
• What is and is not installed?
• What things changed and when and by whom?
System State
@rynchantress ConfigMgmtCamp
Slide 9
Slide 9 text
Me watching all the other
sysadmins with their cool
configura
Slide 10
Slide 10 text
Bash scripts don't manage system state well.
@rynchantress ConfigMgmtCamp
Slide 11
Slide 11 text
@rynchantress ConfigMgmtCamp
Me ge>ng ready to do a
configura
Slide 12
Slide 12 text
@rynchantress ConfigMgmtCamp
Slide 13
Slide 13 text
@rynchantress ConfigMgmtCamp
Slide 14
Slide 14 text
known system state + known config change = new known system state
@rynchantress ConfigMgmtCamp
Slide 15
Slide 15 text
@rynchantress ConfigMgmtCamp
Slide 16
Slide 16 text
• "Known" state wasn't actually known
• Configuring system state didn't account for
unknown applicaBon state
• Undo in configuraBon state couldn't undo changes
to applicaBon state
Wat.
@rynchantress ConfigMgmtCamp
Slide 17
Slide 17 text
• ApplicaBon state
• System state
Let's talk about state
@rynchantress ConfigMgmtCamp
Slide 18
Slide 18 text
Application state
@rynchantress ConfigMgmtCamp
• What is the applicaBon doing?
• What does the applicaBon care about?
• What things changed and when and by whom?
• It's always a caching problem.
Slide 19
Slide 19 text
@rynchantress ConfigMgmtCamp
Slide 20
Slide 20 text
1.Don't forget about applicaBon state
2.Knowns aren't always known
horrible lessons learned
@rynchantress ConfigMgmtCamp
Slide 21
Slide 21 text
@rynchantress ConfigMgmtCamp
Slide 22
Slide 22 text
@rynchantress ConfigMgmtCamp
Me with my several years of
experience and smart
colleagues and tested tooling,
ge>ng ready to do a no-op
Slide 23
Slide 23 text
@rynchantress ConfigMgmtCamp
Slide 24
Slide 24 text
known config state + no-op = same known config state
@rynchantress ConfigMgmtCamp
Slide 25
Slide 25 text
@rynchantress ConfigMgmtCamp
Slide 26
Slide 26 text
• System state interacted with configuraBon
change to cause new system state
• Unknown and inconsistent configuraBon state
across infrastructure
• Uncontrolled source state changes combined
with inconsistent configuraBon state
Wat.
@rynchantress ConfigMgmtCamp
Slide 27
Slide 27 text
• ApplicaBon state
• System state
• ConfiguraBon state
• Source state
Let's talk about state
@rynchantress ConfigMgmtCamp
Slide 28
Slide 28 text
Configuration State
@rynchantress ConfigMgmtCamp
• What did humans tell the config management to do?
• What does the config management system know about the world?
• How is the config management tool running?
Slide 29
Slide 29 text
Source State
@rynchantress ConfigMgmtCamp
• Sources of packages
• Sources of applicaBon code
• Sources of config management code
• Whatever else you can put in source control, have fun!
Slide 30
Slide 30 text
@rynchantress ConfigMgmtCamp
Slide 31
Slide 31 text
1.Don't forget about applicaBon state
2.Knowns aren't always known
3.Monitor your configuraBon state
4.Don't leave your states unsupervised
horrible lessons learned
@rynchantress ConfigMgmtCamp
Slide 32
Slide 32 text
@rynchantress ConfigMgmtCamp
Slide 33
Slide 33 text
@rynchantress ConfigMgmtCamp
Me with my several more years
of experience and other smart
colleagues and tested tooling,
ge>ng ready to do a devops
Slide 34
Slide 34 text
@rynchantress ConfigMgmtCamp
Slide 35
Slide 35 text
known system state + known config change = new known system state
@rynchantress ConfigMgmtCamp
Slide 36
Slide 36 text
@rynchantress ConfigMgmtCamp
Slide 37
Slide 37 text
• Wildly and unknowably different configuraBon
states
• Current configuraBon state not fully captured in
source state
• Wat even was in the terraform plan output
again?!
Wat.
@rynchantress ConfigMgmtCamp
Slide 38
Slide 38 text
• ApplicaBon state
• System state
• ConfiguraBon state
• Source state
• Terraform state
Let's talk about state
@rynchantress ConfigMgmtCamp
Slide 39
Slide 39 text
• ApplicaBon state
• System state
• ConfiguraBon state
• Source state
• Terraform state
Let's talk about state
@rynchantress ConfigMgmtCamp
Slide 40
Slide 40 text
@rynchantress ConfigMgmtCamp
Slide 41
Slide 41 text
1.Don't forget about applicaBon state
2.Knowns aren't always known
3.Monitor your configuraBon state
4.Don't leave your states unsupervised
5.Enforce source state completeness and config state consistency
horrible lessons learned
@rynchantress ConfigMgmtCamp
Slide 42
Slide 42 text
@rynchantress ConfigMgmtCamp
Slide 43
Slide 43 text
@rynchantress ConfigMgmtCamp
Slide 44
Slide 44 text
• ApplicaBon state
• System state
• ConfiguraBon state
• Source state
• Terraform state
• Human state
Let's talk about state
@rynchantress ConfigMgmtCamp
Slide 45
Slide 45 text
Let's talk about human factors
@rynchantress ConfigMgmtCamp
Slide 46
Slide 46 text
@rynchantress ConfigMgmtCamp
Slide 47
Slide 47 text
Computers were a mistake
@rynchantress ConfigMgmtCamp
fewer sources of state
Slide 48
Slide 48 text
@rynchantress ConfigMgmtCamp
Me with so many shiny new
monitoring and observability
tools, ssh-ing into a server like
a professional
Slide 49
Slide 49 text
Computers were a mistake
@rynchantress ConfigMgmtCamp
surface incorrect state
Slide 50
Slide 50 text
Computers were a mistake
@rynchantress ConfigMgmtCamp
make things human-readable
Slide 51
Slide 51 text
@rynchantress ConfigMgmtCamp
Slide 52
Slide 52 text
@rynchantress ConfigMgmtCamp
Slide 53
Slide 53 text
1.Don't forget about applicaBon state
2.Knowns aren't always known
3.Monitor your configuraBon state
4.Don't leave your states unsupervised
5.Enforce source state completeness and config state consistency
6.Get rid of unreliable tools
7.Make sure your tooling is usable by humans
horrible lessons learned
@rynchantress ConfigMgmtCamp
Slide 54
Slide 54 text
@rynchantress ConfigMgmtCamp
THANK YOU!
Special thanks to House House for creating Untitled Goose Game and Samuel Fine for creating the Untitled Goose Game Generator