untitled config game

Keynote presented at Config Management Camp 2020

Transcript

1. ConfigMgmtCamp 2020
   Ryn Daniels they/them @rynchantress
   untitled config game
   

   View Slide

2. It's a lovely morning in the
   data center, and you are a
   horrible goose devop.
   @rynchantress ConfigMgmtCamp
   

   View Slide

3. •Professional computerer
   •Knows several things about Terraform
   •Very good at puns
   Me now...
   @rynchantress ConfigMgmtCamp
   

   View Slide

4. • 200 beauBful and unique snowflake servers
   • 0 automaBon
   • 1 very sad new sysadmin
   Back in the day...
   @rynchantress ConfigMgmtCamp
   

   View Slide

5. Computers were a mistake
   Computers are bad, actually
   @rynchantress ConfigMgmtCamp
   

   View Slide

6. • ApplicaBon state
   • System state
   Let's talk about state
   @rynchantress ConfigMgmtCamp
   

   View Slide

7. • ApplicaBon state
   • System state
   Let's talk about state
   @rynchantress ConfigMgmtCamp
   

   View Slide

8. • What is the system doing?
   • What is the system supposed to be doing?
   • What versions of things exist in the system?
   • What is and is not installed?
   • What things changed and when and by whom?
   System State
   @rynchantress ConfigMgmtCamp
   

   View Slide

9. Me watching all the other
   sysadmins with their cool
   configurawhile I sit in the data center
   with a bash script
   @rynchantress ConfigMgmtCamp
   

   View Slide

10. Bash scripts don't manage system state well.
    @rynchantress ConfigMgmtCamp
    

    View Slide

11. @rynchantress ConfigMgmtCamp
    Me ge>ng ready to do a
    configura

    View Slide

12. @rynchantress ConfigMgmtCamp
    

    View Slide

13. @rynchantress ConfigMgmtCamp
    

    View Slide

14. known system state + known config change = new known system state
    @rynchantress ConfigMgmtCamp
    

    View Slide

15. @rynchantress ConfigMgmtCamp
    

    View Slide

16. • "Known" state wasn't actually known
    • Configuring system state didn't account for
    unknown applicaBon state
    • Undo in configuraBon state couldn't undo changes
    to applicaBon state
    Wat.
    @rynchantress ConfigMgmtCamp
    

    View Slide

17. • ApplicaBon state
    • System state
    Let's talk about state
    @rynchantress ConfigMgmtCamp
    

    View Slide

18. Application state
    @rynchantress ConfigMgmtCamp
    • What is the applicaBon doing?
    • What does the applicaBon care about?
    • What things changed and when and by whom?
    • It's always a caching problem.
    

    View Slide

19. @rynchantress ConfigMgmtCamp
    

    View Slide

20. 1.Don't forget about applicaBon state
    2.Knowns aren't always known
    horrible lessons learned
    @rynchantress ConfigMgmtCamp
    

    View Slide

21. @rynchantress ConfigMgmtCamp
    

    View Slide

22. @rynchantress ConfigMgmtCamp
    Me with my several years of
    experience and smart
    colleagues and tested tooling,
    ge>ng ready to do a no-op
    

    View Slide

23. @rynchantress ConfigMgmtCamp
    

    View Slide

24. known config state + no-op = same known config state
    @rynchantress ConfigMgmtCamp
    

    View Slide

25. @rynchantress ConfigMgmtCamp
    

    View Slide

26. • System state interacted with configuraBon
    change to cause new system state
    • Unknown and inconsistent configuraBon state
    across infrastructure
    • Uncontrolled source state changes combined
    with inconsistent configuraBon state
    Wat.
    @rynchantress ConfigMgmtCamp
    

    View Slide

27. • ApplicaBon state
    • System state
    • ConfiguraBon state
    • Source state
    Let's talk about state
    @rynchantress ConfigMgmtCamp
    

    View Slide

28. Configuration State
    @rynchantress ConfigMgmtCamp
    • What did humans tell the config management to do?
    • What does the config management system know about the world?
    • How is the config management tool running?
    

    View Slide

29. Source State
    @rynchantress ConfigMgmtCamp
    • Sources of packages
    • Sources of applicaBon code
    • Sources of config management code
    • Whatever else you can put in source control, have fun!
    

    View Slide

30. @rynchantress ConfigMgmtCamp
    

    View Slide

31. 1.Don't forget about applicaBon state
    2.Knowns aren't always known
    3.Monitor your configuraBon state
    4.Don't leave your states unsupervised
    horrible lessons learned
    @rynchantress ConfigMgmtCamp
    

    View Slide

32. @rynchantress ConfigMgmtCamp
    

    View Slide

33. @rynchantress ConfigMgmtCamp
    Me with my several more years
    of experience and other smart
    colleagues and tested tooling,
    ge>ng ready to do a devops
    

    View Slide

34. @rynchantress ConfigMgmtCamp
    

    View Slide

35. known system state + known config change = new known system state
    @rynchantress ConfigMgmtCamp
    

    View Slide

36. @rynchantress ConfigMgmtCamp
    

    View Slide

37. • Wildly and unknowably different configuraBon
    states
    • Current configuraBon state not fully captured in
    source state
    • Wat even was in the terraform plan output
    again?!
    Wat.
    @rynchantress ConfigMgmtCamp
    

    View Slide

38. • ApplicaBon state
    • System state
    • ConfiguraBon state
    • Source state
    • Terraform state
    Let's talk about state
    @rynchantress ConfigMgmtCamp
    

    View Slide

39. • ApplicaBon state
    • System state
    • ConfiguraBon state
    • Source state
    • Terraform state
    Let's talk about state
    @rynchantress ConfigMgmtCamp
    

    View Slide

40. @rynchantress ConfigMgmtCamp
    

    View Slide

41. 1.Don't forget about applicaBon state
    2.Knowns aren't always known
    3.Monitor your configuraBon state
    4.Don't leave your states unsupervised
    5.Enforce source state completeness and config state consistency
    horrible lessons learned
    @rynchantress ConfigMgmtCamp
    

    View Slide

42. @rynchantress ConfigMgmtCamp
    

    View Slide

43. @rynchantress ConfigMgmtCamp
    

    View Slide

44. • ApplicaBon state
    • System state
    • ConfiguraBon state
    • Source state
    • Terraform state
    • Human state
    Let's talk about state
    @rynchantress ConfigMgmtCamp
    

    View Slide

45. Let's talk about human factors
    @rynchantress ConfigMgmtCamp
    

    View Slide

46. @rynchantress ConfigMgmtCamp
    

    View Slide

47. Computers were a mistake
    @rynchantress ConfigMgmtCamp
    fewer sources of state
    

    View Slide

48. @rynchantress ConfigMgmtCamp
    Me with so many shiny new
    monitoring and observability
    tools, ssh-ing into a server like
    a professional
    

    View Slide

49. Computers were a mistake
    @rynchantress ConfigMgmtCamp
    surface incorrect state
    

    View Slide

50. Computers were a mistake
    @rynchantress ConfigMgmtCamp
    make things human-readable
    

    View Slide

51. @rynchantress ConfigMgmtCamp
    

    View Slide

52. @rynchantress ConfigMgmtCamp
    

    View Slide

53. 1.Don't forget about applicaBon state
    2.Knowns aren't always known
    3.Monitor your configuraBon state
    4.Don't leave your states unsupervised
    5.Enforce source state completeness and config state consistency
    6.Get rid of unreliable tools
    7.Make sure your tooling is usable by humans
    horrible lessons learned
    @rynchantress ConfigMgmtCamp
    

    View Slide

54. @rynchantress ConfigMgmtCamp
    THANK YOU!
    Special thanks to House House for creating Untitled Goose Game and Samuel Fine for creating the Untitled Goose Game Generator
    

    View Slide