Upgrade to Pro — share decks privately, control downloads, hide ads and more …

untitled config game

Ryn Daniels
February 03, 2020

untitled config game

Keynote presented at Config Management Camp 2020

Ryn Daniels

February 03, 2020
Tweet

More Decks by Ryn Daniels

Other Decks in Technology

Transcript

  1. ConfigMgmtCamp 2020
    Ryn Daniels they/them @rynchantress
    untitled config game

    View Slide

  2. It's a lovely morning in the
    data center, and you are a
    horrible goose devop.
    @rynchantress ConfigMgmtCamp

    View Slide

  3. •Professional computerer
    •Knows several things about Terraform
    •Very good at puns
    Me now...
    @rynchantress ConfigMgmtCamp

    View Slide

  4. • 200 beauBful and unique snowflake servers
    • 0 automaBon
    • 1 very sad new sysadmin
    Back in the day...
    @rynchantress ConfigMgmtCamp

    View Slide

  5. Computers were a mistake
    Computers are bad, actually
    @rynchantress ConfigMgmtCamp

    View Slide

  6. • ApplicaBon state
    • System state
    Let's talk about state
    @rynchantress ConfigMgmtCamp

    View Slide

  7. • ApplicaBon state
    • System state
    Let's talk about state
    @rynchantress ConfigMgmtCamp

    View Slide

  8. • What is the system doing?
    • What is the system supposed to be doing?
    • What versions of things exist in the system?
    • What is and is not installed?
    • What things changed and when and by whom?
    System State
    @rynchantress ConfigMgmtCamp

    View Slide

  9. Me watching all the other
    sysadmins with their cool
    configurawhile I sit in the data center
    with a bash script
    @rynchantress ConfigMgmtCamp

    View Slide

  10. Bash scripts don't manage system state well.
    @rynchantress ConfigMgmtCamp

    View Slide

  11. @rynchantress ConfigMgmtCamp
    Me ge>ng ready to do a
    configura

    View Slide

  12. @rynchantress ConfigMgmtCamp

    View Slide

  13. @rynchantress ConfigMgmtCamp

    View Slide

  14. known system state + known config change = new known system state
    @rynchantress ConfigMgmtCamp

    View Slide

  15. @rynchantress ConfigMgmtCamp

    View Slide

  16. • "Known" state wasn't actually known
    • Configuring system state didn't account for
    unknown applicaBon state
    • Undo in configuraBon state couldn't undo changes
    to applicaBon state
    Wat.
    @rynchantress ConfigMgmtCamp

    View Slide

  17. • ApplicaBon state
    • System state
    Let's talk about state
    @rynchantress ConfigMgmtCamp

    View Slide

  18. Application state
    @rynchantress ConfigMgmtCamp
    • What is the applicaBon doing?
    • What does the applicaBon care about?
    • What things changed and when and by whom?
    • It's always a caching problem.

    View Slide

  19. @rynchantress ConfigMgmtCamp

    View Slide

  20. 1.Don't forget about applicaBon state
    2.Knowns aren't always known
    horrible lessons learned
    @rynchantress ConfigMgmtCamp

    View Slide

  21. @rynchantress ConfigMgmtCamp

    View Slide

  22. @rynchantress ConfigMgmtCamp
    Me with my several years of
    experience and smart
    colleagues and tested tooling,
    ge>ng ready to do a no-op

    View Slide

  23. @rynchantress ConfigMgmtCamp

    View Slide

  24. known config state + no-op = same known config state
    @rynchantress ConfigMgmtCamp

    View Slide

  25. @rynchantress ConfigMgmtCamp

    View Slide

  26. • System state interacted with configuraBon
    change to cause new system state
    • Unknown and inconsistent configuraBon state
    across infrastructure
    • Uncontrolled source state changes combined
    with inconsistent configuraBon state
    Wat.
    @rynchantress ConfigMgmtCamp

    View Slide

  27. • ApplicaBon state
    • System state
    • ConfiguraBon state
    • Source state
    Let's talk about state
    @rynchantress ConfigMgmtCamp

    View Slide

  28. Configuration State
    @rynchantress ConfigMgmtCamp
    • What did humans tell the config management to do?
    • What does the config management system know about the world?
    • How is the config management tool running?

    View Slide

  29. Source State
    @rynchantress ConfigMgmtCamp
    • Sources of packages
    • Sources of applicaBon code
    • Sources of config management code
    • Whatever else you can put in source control, have fun!

    View Slide

  30. @rynchantress ConfigMgmtCamp

    View Slide

  31. 1.Don't forget about applicaBon state
    2.Knowns aren't always known
    3.Monitor your configuraBon state
    4.Don't leave your states unsupervised
    horrible lessons learned
    @rynchantress ConfigMgmtCamp

    View Slide

  32. @rynchantress ConfigMgmtCamp

    View Slide

  33. @rynchantress ConfigMgmtCamp
    Me with my several more years
    of experience and other smart
    colleagues and tested tooling,
    ge>ng ready to do a devops

    View Slide

  34. @rynchantress ConfigMgmtCamp

    View Slide

  35. known system state + known config change = new known system state
    @rynchantress ConfigMgmtCamp

    View Slide

  36. @rynchantress ConfigMgmtCamp

    View Slide

  37. • Wildly and unknowably different configuraBon
    states
    • Current configuraBon state not fully captured in
    source state
    • Wat even was in the terraform plan output
    again?!
    Wat.
    @rynchantress ConfigMgmtCamp

    View Slide

  38. • ApplicaBon state
    • System state
    • ConfiguraBon state
    • Source state
    • Terraform state
    Let's talk about state
    @rynchantress ConfigMgmtCamp

    View Slide

  39. • ApplicaBon state
    • System state
    • ConfiguraBon state
    • Source state
    • Terraform state
    Let's talk about state
    @rynchantress ConfigMgmtCamp

    View Slide

  40. @rynchantress ConfigMgmtCamp

    View Slide

  41. 1.Don't forget about applicaBon state
    2.Knowns aren't always known
    3.Monitor your configuraBon state
    4.Don't leave your states unsupervised
    5.Enforce source state completeness and config state consistency
    horrible lessons learned
    @rynchantress ConfigMgmtCamp

    View Slide

  42. @rynchantress ConfigMgmtCamp

    View Slide

  43. @rynchantress ConfigMgmtCamp

    View Slide

  44. • ApplicaBon state
    • System state
    • ConfiguraBon state
    • Source state
    • Terraform state
    • Human state
    Let's talk about state
    @rynchantress ConfigMgmtCamp

    View Slide

  45. Let's talk about human factors
    @rynchantress ConfigMgmtCamp

    View Slide

  46. @rynchantress ConfigMgmtCamp

    View Slide

  47. Computers were a mistake
    @rynchantress ConfigMgmtCamp
    fewer sources of state

    View Slide

  48. @rynchantress ConfigMgmtCamp
    Me with so many shiny new
    monitoring and observability
    tools, ssh-ing into a server like
    a professional

    View Slide

  49. Computers were a mistake
    @rynchantress ConfigMgmtCamp
    surface incorrect state

    View Slide

  50. Computers were a mistake
    @rynchantress ConfigMgmtCamp
    make things human-readable

    View Slide

  51. @rynchantress ConfigMgmtCamp

    View Slide

  52. @rynchantress ConfigMgmtCamp

    View Slide

  53. 1.Don't forget about applicaBon state
    2.Knowns aren't always known
    3.Monitor your configuraBon state
    4.Don't leave your states unsupervised
    5.Enforce source state completeness and config state consistency
    6.Get rid of unreliable tools
    7.Make sure your tooling is usable by humans
    horrible lessons learned
    @rynchantress ConfigMgmtCamp

    View Slide

  54. @rynchantress ConfigMgmtCamp
    THANK YOU!
    Special thanks to House House for creating Untitled Goose Game and Samuel Fine for creating the Untitled Goose Game Generator

    View Slide