untitled config game

Keynote presented at Config Management Camp 2020

Transcript

1. ConfigMgmtCamp 2020 Ryn Daniels they/them @rynchantress untitled config game
2. It's a lovely morning in the data center, and you are a horrible goose devop. @rynchantress ConfigMgmtCamp
3. •Professional computerer •Knows several things about Terraform •Very good at puns Me now... @rynchantress ConfigMgmtCamp
4. • 200 beauBful and unique snowflake servers • 0 automaBon • 1 very sad new sysadmin Back in the day... @rynchantress ConfigMgmtCamp
5. Computers were a mistake Computers are bad, actually @rynchantress ConfigMgmtCamp
6. • ApplicaBon state • System state Let's talk about state @rynchantress ConfigMgmtCamp
7. • ApplicaBon state • System state Let's talk about state @rynchantress ConfigMgmtCamp
8. • What is the system doing? • What is the system supposed to be doing? • What versions of things exist in the system? • What is and is not installed? • What things changed and when and by whom? System State @rynchantress ConfigMgmtCamp
9. Me watching all the other sysadmins with their cool configura<on management while I sit in the data center with a bash script @rynchantress ConfigMgmtCamp
10. Bash scripts don't manage system state well. @rynchantress ConfigMgmtCamp
11. @rynchantress ConfigMgmtCamp Me ge>ng ready to do a configura<on management
12. @rynchantress ConfigMgmtCamp
13. @rynchantress ConfigMgmtCamp
14. known system state + known config change = new known system state @rynchantress ConfigMgmtCamp
15. @rynchantress ConfigMgmtCamp
16. • "Known" state wasn't actually known • Configuring system state didn't account for unknown applicaBon state • Undo in configuraBon state couldn't undo changes to applicaBon state Wat. @rynchantress ConfigMgmtCamp
17. • ApplicaBon state • System state Let's talk about state @rynchantress ConfigMgmtCamp
18. Application state @rynchantress ConfigMgmtCamp • What is the applicaBon doing? • What does the applicaBon care about? • What things changed and when and by whom? • It's always a caching problem.
19. @rynchantress ConfigMgmtCamp
20. 1.Don't forget about applicaBon state 2.Knowns aren't always known horrible lessons learned @rynchantress ConfigMgmtCamp
21. @rynchantress ConfigMgmtCamp
22. @rynchantress ConfigMgmtCamp Me with my several years of experience and smart colleagues and tested tooling, ge>ng ready to do a no-op
23. @rynchantress ConfigMgmtCamp
24. known config state + no-op = same known config state @rynchantress ConfigMgmtCamp
25. @rynchantress ConfigMgmtCamp
26. • System state interacted with configuraBon change to cause new system state • Unknown and inconsistent configuraBon state across infrastructure • Uncontrolled source state changes combined with inconsistent configuraBon state Wat. @rynchantress ConfigMgmtCamp
27. • ApplicaBon state • System state • ConfiguraBon state • Source state Let's talk about state @rynchantress ConfigMgmtCamp
28. Configuration State @rynchantress ConfigMgmtCamp • What did humans tell the config management to do? • What does the config management system know about the world? • How is the config management tool running?
29. Source State @rynchantress ConfigMgmtCamp • Sources of packages • Sources of applicaBon code • Sources of config management code • Whatever else you can put in source control, have fun!
30. @rynchantress ConfigMgmtCamp
31. 1.Don't forget about applicaBon state 2.Knowns aren't always known 3.Monitor your configuraBon state 4.Don't leave your states unsupervised horrible lessons learned @rynchantress ConfigMgmtCamp
32. @rynchantress ConfigMgmtCamp
33. @rynchantress ConfigMgmtCamp Me with my several more years of experience and other smart colleagues and tested tooling, ge>ng ready to do a devops
34. @rynchantress ConfigMgmtCamp
35. known system state + known config change = new known system state @rynchantress ConfigMgmtCamp
36. @rynchantress ConfigMgmtCamp
37. • Wildly and unknowably different configuraBon states • Current configuraBon state not fully captured in source state • Wat even was in the terraform plan output again?! Wat. @rynchantress ConfigMgmtCamp
38. • ApplicaBon state • System state • ConfiguraBon state • Source state • Terraform state Let's talk about state @rynchantress ConfigMgmtCamp
39. • ApplicaBon state • System state • ConfiguraBon state • Source state • Terraform state Let's talk about state @rynchantress ConfigMgmtCamp
40. @rynchantress ConfigMgmtCamp
41. 1.Don't forget about applicaBon state 2.Knowns aren't always known 3.Monitor your configuraBon state 4.Don't leave your states unsupervised 5.Enforce source state completeness and config state consistency horrible lessons learned @rynchantress ConfigMgmtCamp
42. @rynchantress ConfigMgmtCamp
43. @rynchantress ConfigMgmtCamp
44. • ApplicaBon state • System state • ConfiguraBon state • Source state • Terraform state • Human state Let's talk about state @rynchantress ConfigMgmtCamp
45. Let's talk about human factors @rynchantress ConfigMgmtCamp
46. @rynchantress ConfigMgmtCamp
47. Computers were a mistake @rynchantress ConfigMgmtCamp fewer sources of state
48. @rynchantress ConfigMgmtCamp Me with so many shiny new monitoring and observability tools, ssh-ing into a server like a professional
49. Computers were a mistake @rynchantress ConfigMgmtCamp surface incorrect state
50. Computers were a mistake @rynchantress ConfigMgmtCamp make things human-readable
51. @rynchantress ConfigMgmtCamp
52. @rynchantress ConfigMgmtCamp
53. 1.Don't forget about applicaBon state 2.Knowns aren't always known 3.Monitor your configuraBon state 4.Don't leave your states unsupervised 5.Enforce source state completeness and config state consistency 6.Get rid of unreliable tools 7.Make sure your tooling is usable by humans horrible lessons learned @rynchantress ConfigMgmtCamp
54. @rynchantress ConfigMgmtCamp THANK YOU! Special thanks to House House for creating Untitled Goose Game and Samuel Fine for creating the Untitled Goose Game Generator