Put Some Dev in Your Devops

Put Some Dev in Your Devops

Presented at Uptime 2017

2396958133b7324fc7afe505dfa02572?s=128

Ryn Daniels

August 26, 2017
Tweet

Transcript

  1. Put Some Dev in your devops Ryn Daniels
 Uptime 2017

  2. @rynchantress UPtime 2017 Hurray, devops!

  3. @rynchantress Uptime 2017 Ryn Daniels
 they/them - @rynchantress Infrastructure Ops

    Witch, Travis CI
  4. @rynchantress Uptime 2017 (the devops) agenda STORYTIME: INFRASTRUCTURE PROVISIONING THE

    PROBLEM: WHEN OPS WRITE CODE ADDING SOME DEV: IT WASN’T THAT BAD! RESULTS: I HOPE WE ALL LEARNED SOMETHING HERE TAKEAWAYS: YOU CAN DEV THE DEVOPS TOO! MORE STORYTIME: CONFIGURATION MANAGEMENT
  5. @rynchantress Uptime 2017 disclaimer

  6. @rynchantress 1 Uptime 2017 (the devops) agenda STORYTIME: INFRASTRUCTURE PROVISIONING

    THE PROBLEM: WHEN OPS WRITE CODE ADDING SOME DEV: IT WASN’T THAT BAD! RESULTS: I HOPE WE ALL LEARNED SOMETHING HERE TAKEAWAYS: YOU CAN DEV THE DEVOPS TOO! MORE STORYTIME: CONFIGURATION MANAGEMENT
  7. @rynchantress UPtime 2017 Infrastructure Provisioning

  8. None
  9. @rynchantress UPtime 2017 Inventory Management Disk Partitioning IPMI Configuration Initial

    Network Setup Infrastructure Provisioning OS Installation DNS Setup Final Network Setup Configuration Management
  10. @rynchantress UPtime 2017 Header

  11. @rynchantress UPtime 2017 + “we’re just ops people, these are

    just ops scripts!”
  12. @rynchantress UPtime 2017 Inventory Management Disk Partitioning IPMI Configuration Initial

    Network Setup Self-Service Infrastructure Provisioning OS Installation DNS Setup Final Network Setup Configuration Management Bulk Build Capabilities
  13. @rynchantress UPtime 2017 The indigo tool suite •gabriel: OS installation,

    final network setup, DNS, configuration management •zaar: decommissioning boxes (undoing all of the above)
  14. @rynchantress UPtime 2017 •sledgehammer: disk partitioning •unattended sledgehammer: inventory management,

    initial network setup, IPMI configuration The indigo tool suite (Continued)
  15. @rynchantress UPtime 2017 •sledgehammer executor: bootable disk image that installs

    and loads the… •sledgehammer payload: rpm that sets up and runs all of the unattended sledgehammer tasks The indigo tool suite (Continued Again)
  16. @rynchantress UPtime 2017 •Indigo API: Mostly providing a way to

    avoid handing out duplicate IPMI IPs •Indigo Sweeper: Queuing up Cobbler syncs instead of DDOSing the Cobbler servers •Indigo Tailer: Hey, having logs to look at would be cool! The indigo tool suite (this is the last one i promise)
  17. @rynchantress UPtime 2017 Inventory Management Disk Partitioning IPMI Configuration Initial

    Network Setup Self-Service Infrastructure Provisioning OS Installation DNS Setup Final Network Setup Configuration Management Bulk Build Capabilities
  18. @rynchantress 2 Uptime 2017 (the devops) agenda STORYTIME: INFRASTRUCTURE PROVISIONING

    THE PROBLEM: WHEN OPS WRITE CODE ADDING SOME DEV: IT WASN’T THAT BAD! RESULTS: I HOPE WE ALL LEARNED SOMETHING HERE TAKEAWAYS: YOU CAN DEV THE DEVOPS TOO! MORE STORYTIME: CONFIGURATION MANAGEMENT
  19. lolwhoops.

  20. @rynchantress UPtime 2017 •No Planning •Low Readability •Duplicated Code •(Human)

    Single Points of Failure unmaintainable code
  21. @rynchantress UPtime 2017 untested code •No Tests at All •Untestable

    Code •No Deploy Process •“F*** It, We’ll Do It Live”
  22. @rynchantress UPtime 2017 critical code •Critical Path for Infrastructure Provisioning

    •Important for Data Center Team •Critical Code Should Be Tested!
  23. @rynchantress 3 Uptime 2017 (the devops) agenda STORYTIME: INFRASTRUCTURE PROVISIONING

    THE PROBLEM: WHEN OPS WRITE CODE ADDING SOME DEV: IT WASN’T THAT BAD! RESULTS: I HOPE WE ALL LEARNED SOMETHING HERE TAKEAWAYS: YOU CAN DEV THE DEVOPS TOO! MORE STORYTIME: CONFIGURATION MANAGEMENT
  24. @rynchantress UPtime 2017 “Oh, I have a CS Degree! I

    can add some OO, make use of inheritance to cut down on copy pasta, do some refactoring, add some tests…”
  25. @rynchantress UPtime 2017 LOL NOPE.

  26. @rynchantress UPtime 2017 Don’t “test” code in production.

  27. @rynchantress Uptime 2017 “What if we had literally any way

    to test this at all?” LOL.
  28. @rynchantress UPtime 2017 Testing unattended mode

  29. @rynchantress uptime 2017 “Uhhhhhh hey data center team, let me

    know if this new deploy broke anything maybe?” The old way
  30. @rynchantress uptime 2017 > sledgehammer.rb -h test-host 
 --unattended The

    new way
  31. @rynchantress UPtime 2017 adding versioning

  32. @rynchantress uptime 2017 The old way (replace payload version 0.1

    with new version 0.1) “Uhhhhhh hey data center team, let me know if this new deploy broke anything maybe?”
  33. @rynchantress uptime 2017 (build payload version 0.2) sledgehammer.rb -h test-host

    
 --unattended -p 0.2 The new way
  34. @rynchantress UPtime 2017 a separate test iso!

  35. @rynchantress uptime 2017 The old way (replace sledgehammer ISO with

    new sledgehammer ISO) “Uhhhhhh hey data center team, let me know if this new deploy broke anything maybe?”
  36. @rynchantress uptime 2017 The new way

  37. @rynchantress UPtime 2017 creating a test api

  38. @rynchantress uptime 2017 The old way (replace old indigo API

    with new indigo API) “Uhhhhhh hey data center team, let me know if this new deploy broke anything maybe?”
  39. @rynchantress uptime 2017 > rackup -olocalhost -p8765 > sledgehammer.rb -h

    test-host 
 --unattended -p 0.2 -l 8765 The new way
  40. @rynchantress UPtime 2017 •NOT hard-coding one version everywhere! •Local/test versions

    of different components •Testing the code paths that our customers use most often •Notably less sadness several yaks later…
  41. @rynchantress UPtime 2017 deployinator!

  42. @rynchantress Uptime 2017

  43. @rynchantress UPtime 2017 unit tests! (and refactoring!) ((Finally!!))

  44. @rynchantress UPtime 2017 unit testing and refactoring goals • Using

    inheritance instead of duplicating methods all over • Using optional parameters for mocking/stubbing and creating shorter/simpler methods for more testability • Using tests as additional documentation for what the code is supposed to do • Giving people even more confidence when making changes
  45. @rynchantress UPtime 2017 example time! yay!

  46. @rynchantress Uptime 2017 def find_switchport(hostname, host_interface) unless ipmi_mac = @assetmgmt_helper.get_mac(hostname,

    'ilo') @formatter.log_and_raise("Racktables has no MAC for the IPMI interface. Game over.") end ... end
  47. @rynchantress Uptime 2017 def find_switchport(hostname, host_interface, assetmgmt_helper=@assetmgmt_helper) unless ipmi_mac =

    assetmgmt_helper.get_mac(hostname, 'ilo') @formatter.log_and_raise("Racktables has no MAC for the IPMI interface. Game over.") end ... end
  48. @rynchantress Uptime 2017 def test_find_switchport assetmgmt_helper = Indigo::Helpers::RacktablesHelper.new(@config, @formatter) assetmgmt_helper.stubs(:get_mac).returns("00AABBCCDDEE")

    assetmgmt_helper.stubs(:get_switchname).returns("switch-123.e tsy.com") ipmi_helper = Indigo::Helpers::IpmiHelper.new(@config, @formatter, assetmgmt_helper=assetmgmt_helper) switchport = ipmi_helper.find_switchport("beerops- test.etsy.com", 'ilo', snmp_helper) assert_equal(["switch-123.etsy.com", "gi0/42"], switchport) end
  49. None
  50. @rynchantress UPtime 2017 the end?!

  51. @rynchantress 4 Uptime 2017 (the devops) agenda STORYTIME: INFRASTRUCTURE PROVISIONING

    THE PROBLEM: WHEN OPS WRITE CODE ADDING SOME DEV: IT WASN’T THAT BAD! RESULTS: I HOPE WE ALL LEARNED SOMETHING HERE TAKEAWAYS: YOU CAN DEV THE DEVOPS TOO! MORE STORYTIME: CONFIGURATION MANAGEMENT
  52. @rynchantress UPtime 2017 the knife-spork dance

  53. @rynchantress UPtime 2017 the knife-spork dance knife spork check cookbook_name

    knife spork upload cookbook_name knife spork promote cookbook_name --remote git commit git push
  54. @rynchantress UPtime 2017 the knife-spork dance knife spork check cookbook_name

    knife spork upload cookbook_name knife spork promote cookbook_name --remote (Jump into slack to figure out who promoted but forgot to commit their changes) (Wait for them to commit and push) git pull knife spork promote cookbook_name --remote (Maybe forget to commit and push, perpetuating the cycle)
  55. @rynchantress UPtime 2017 Deployinator!

  56. @rynchantress Uptime 2017

  57. @rynchantress Uptime 2017

  58. None
  59. @rynchantress UPtime 2017 the end?!?!

  60. @rynchantress 5 Uptime 2017 (the devops) agenda STORYTIME: INFRASTRUCTURE PROVISIONING

    THE PROBLEM: WHEN OPS WRITE CODE ADDING SOME DEV: IT WASN’T THAT BAD! RESULTS: I HOPE WE ALL LEARNED SOMETHING HERE TAKEAWAYS: YOU CAN DEV THE DEVOPS TOO! MORE STORYTIME: CONFIGURATION MANAGEMENT
  61. happier users

  62. @rynchantress UPtime 2017 Fewer (human) SPOFS

  63. @rynchantress UPtime 2017 maintainable tools

  64. @rynchantress 6 Uptime 2017 (the devops) agenda STORYTIME: INFRASTRUCTURE PROVISIONING

    THE PROBLEM: WHEN OPS WRITE CODE ADDING SOME DEV: IT WASN’T THAT BAD! RESULTS: I HOPE WE ALL LEARNED SOMETHING HERE TAKEAWAYS: YOU CAN DEV THE DEVOPS TOO! MORE STORYTIME: CONFIGURATION MANAGEMENT
  65. @rynchantress UPtime 2017 •Ops code is still code •Start with

    the biggest pain points first •Maintainability > cleverness •You CAN teach old ops new tricks! •Leave things better than you found them you can dev the devops too <3
  66. @rynchantress Uptime 2017 Thank you!