Put Some Dev in Your Devops

Presented at Uptime 2017

Transcript

1. Put Some Dev in your devops Ryn Daniels
 Uptime 2017
2. @rynchantress UPtime 2017 Hurray, devops!
3. @rynchantress Uptime 2017 Ryn Daniels
 they/them - @rynchantress Infrastructure Ops Witch, Travis CI
4. @rynchantress Uptime 2017 (the devops) agenda STORYTIME: INFRASTRUCTURE PROVISIONING THE PROBLEM: WHEN OPS WRITE CODE ADDING SOME DEV: IT WASN’T THAT BAD! RESULTS: I HOPE WE ALL LEARNED SOMETHING HERE TAKEAWAYS: YOU CAN DEV THE DEVOPS TOO! MORE STORYTIME: CONFIGURATION MANAGEMENT
5. @rynchantress Uptime 2017 disclaimer
6. @rynchantress 1 Uptime 2017 (the devops) agenda STORYTIME: INFRASTRUCTURE PROVISIONING THE PROBLEM: WHEN OPS WRITE CODE ADDING SOME DEV: IT WASN’T THAT BAD! RESULTS: I HOPE WE ALL LEARNED SOMETHING HERE TAKEAWAYS: YOU CAN DEV THE DEVOPS TOO! MORE STORYTIME: CONFIGURATION MANAGEMENT
7. @rynchantress UPtime 2017 Infrastructure Provisioning
9. @rynchantress UPtime 2017 Inventory Management Disk Partitioning IPMI Configuration Initial Network Setup Infrastructure Provisioning OS Installation DNS Setup Final Network Setup Configuration Management
10. @rynchantress UPtime 2017 Header
11. @rynchantress UPtime 2017 + “we’re just ops people, these are just ops scripts!”
12. @rynchantress UPtime 2017 Inventory Management Disk Partitioning IPMI Configuration Initial Network Setup Self-Service Infrastructure Provisioning OS Installation DNS Setup Final Network Setup Configuration Management Bulk Build Capabilities
13. @rynchantress UPtime 2017 The indigo tool suite •gabriel: OS installation, final network setup, DNS, configuration management •zaar: decommissioning boxes (undoing all of the above)
14. @rynchantress UPtime 2017 •sledgehammer: disk partitioning •unattended sledgehammer: inventory management, initial network setup, IPMI configuration The indigo tool suite (Continued)
15. @rynchantress UPtime 2017 •sledgehammer executor: bootable disk image that installs and loads the… •sledgehammer payload: rpm that sets up and runs all of the unattended sledgehammer tasks The indigo tool suite (Continued Again)
16. @rynchantress UPtime 2017 •Indigo API: Mostly providing a way to avoid handing out duplicate IPMI IPs •Indigo Sweeper: Queuing up Cobbler syncs instead of DDOSing the Cobbler servers •Indigo Tailer: Hey, having logs to look at would be cool! The indigo tool suite (this is the last one i promise)
17. @rynchantress UPtime 2017 Inventory Management Disk Partitioning IPMI Configuration Initial Network Setup Self-Service Infrastructure Provisioning OS Installation DNS Setup Final Network Setup Configuration Management Bulk Build Capabilities
18. @rynchantress 2 Uptime 2017 (the devops) agenda STORYTIME: INFRASTRUCTURE PROVISIONING THE PROBLEM: WHEN OPS WRITE CODE ADDING SOME DEV: IT WASN’T THAT BAD! RESULTS: I HOPE WE ALL LEARNED SOMETHING HERE TAKEAWAYS: YOU CAN DEV THE DEVOPS TOO! MORE STORYTIME: CONFIGURATION MANAGEMENT
19. lolwhoops.
20. @rynchantress UPtime 2017 •No Planning •Low Readability •Duplicated Code •(Human) Single Points of Failure unmaintainable code
21. @rynchantress UPtime 2017 untested code •No Tests at All •Untestable Code •No Deploy Process •“F*** It, We’ll Do It Live”
22. @rynchantress UPtime 2017 critical code •Critical Path for Infrastructure Provisioning •Important for Data Center Team •Critical Code Should Be Tested!
23. @rynchantress 3 Uptime 2017 (the devops) agenda STORYTIME: INFRASTRUCTURE PROVISIONING THE PROBLEM: WHEN OPS WRITE CODE ADDING SOME DEV: IT WASN’T THAT BAD! RESULTS: I HOPE WE ALL LEARNED SOMETHING HERE TAKEAWAYS: YOU CAN DEV THE DEVOPS TOO! MORE STORYTIME: CONFIGURATION MANAGEMENT
24. @rynchantress UPtime 2017 “Oh, I have a CS Degree! I can add some OO, make use of inheritance to cut down on copy pasta, do some refactoring, add some tests…”
25. @rynchantress UPtime 2017 LOL NOPE.
26. @rynchantress UPtime 2017 Don’t “test” code in production.
27. @rynchantress Uptime 2017 “What if we had literally any way to test this at all?” LOL.
28. @rynchantress UPtime 2017 Testing unattended mode
29. @rynchantress uptime 2017 “Uhhhhhh hey data center team, let me know if this new deploy broke anything maybe?” The old way
30. @rynchantress uptime 2017 > sledgehammer.rb -h test-host 
 --unattended The new way
31. @rynchantress UPtime 2017 adding versioning
32. @rynchantress uptime 2017 The old way (replace payload version 0.1 with new version 0.1) “Uhhhhhh hey data center team, let me know if this new deploy broke anything maybe?”
33. @rynchantress uptime 2017 (build payload version 0.2) sledgehammer.rb -h test-host 
 --unattended -p 0.2 The new way
34. @rynchantress UPtime 2017 a separate test iso!
35. @rynchantress uptime 2017 The old way (replace sledgehammer ISO with new sledgehammer ISO) “Uhhhhhh hey data center team, let me know if this new deploy broke anything maybe?”
36. @rynchantress uptime 2017 The new way
37. @rynchantress UPtime 2017 creating a test api
38. @rynchantress uptime 2017 The old way (replace old indigo API with new indigo API) “Uhhhhhh hey data center team, let me know if this new deploy broke anything maybe?”
39. @rynchantress uptime 2017 > rackup -olocalhost -p8765 > sledgehammer.rb -h test-host 
 --unattended -p 0.2 -l 8765 The new way
40. @rynchantress UPtime 2017 •NOT hard-coding one version everywhere! •Local/test versions of different components •Testing the code paths that our customers use most often •Notably less sadness several yaks later…
41. @rynchantress UPtime 2017 deployinator!
42. @rynchantress Uptime 2017
43. @rynchantress UPtime 2017 unit tests! (and refactoring!) ((Finally!!))
44. @rynchantress UPtime 2017 unit testing and refactoring goals • Using inheritance instead of duplicating methods all over • Using optional parameters for mocking/stubbing and creating shorter/simpler methods for more testability • Using tests as additional documentation for what the code is supposed to do • Giving people even more confidence when making changes
45. @rynchantress UPtime 2017 example time! yay!
46. @rynchantress Uptime 2017 def find_switchport(hostname, host_interface) unless ipmi_mac = @assetmgmt_helper.get_mac(hostname, 'ilo') @formatter.log_and_raise("Racktables has no MAC for the IPMI interface. Game over.") end ... end
47. @rynchantress Uptime 2017 def find_switchport(hostname, host_interface, assetmgmt_helper=@assetmgmt_helper) unless ipmi_mac = assetmgmt_helper.get_mac(hostname, 'ilo') @formatter.log_and_raise("Racktables has no MAC for the IPMI interface. Game over.") end ... end
48. @rynchantress Uptime 2017 def test_find_switchport assetmgmt_helper = Indigo::Helpers::RacktablesHelper.new(@config, @formatter) assetmgmt_helper.stubs(:get_mac).returns("00AABBCCDDEE") assetmgmt_helper.stubs(:get_switchname).returns("switch-123.e tsy.com") ipmi_helper = Indigo::Helpers::IpmiHelper.new(@config, @formatter, assetmgmt_helper=assetmgmt_helper) switchport = ipmi_helper.find_switchport("beerops- test.etsy.com", 'ilo', snmp_helper) assert_equal(["switch-123.etsy.com", "gi0/42"], switchport) end
50. @rynchantress UPtime 2017 the end?!
51. @rynchantress 4 Uptime 2017 (the devops) agenda STORYTIME: INFRASTRUCTURE PROVISIONING THE PROBLEM: WHEN OPS WRITE CODE ADDING SOME DEV: IT WASN’T THAT BAD! RESULTS: I HOPE WE ALL LEARNED SOMETHING HERE TAKEAWAYS: YOU CAN DEV THE DEVOPS TOO! MORE STORYTIME: CONFIGURATION MANAGEMENT
52. @rynchantress UPtime 2017 the knife-spork dance
53. @rynchantress UPtime 2017 the knife-spork dance knife spork check cookbook_name knife spork upload cookbook_name knife spork promote cookbook_name --remote git commit git push
54. @rynchantress UPtime 2017 the knife-spork dance knife spork check cookbook_name knife spork upload cookbook_name knife spork promote cookbook_name --remote (Jump into slack to figure out who promoted but forgot to commit their changes) (Wait for them to commit and push) git pull knife spork promote cookbook_name --remote (Maybe forget to commit and push, perpetuating the cycle)
55. @rynchantress UPtime 2017 Deployinator!
56. @rynchantress Uptime 2017
57. @rynchantress Uptime 2017
59. @rynchantress UPtime 2017 the end?!?!
60. @rynchantress 5 Uptime 2017 (the devops) agenda STORYTIME: INFRASTRUCTURE PROVISIONING THE PROBLEM: WHEN OPS WRITE CODE ADDING SOME DEV: IT WASN’T THAT BAD! RESULTS: I HOPE WE ALL LEARNED SOMETHING HERE TAKEAWAYS: YOU CAN DEV THE DEVOPS TOO! MORE STORYTIME: CONFIGURATION MANAGEMENT
61. happier users
62. @rynchantress UPtime 2017 Fewer (human) SPOFS
63. @rynchantress UPtime 2017 maintainable tools
64. @rynchantress 6 Uptime 2017 (the devops) agenda STORYTIME: INFRASTRUCTURE PROVISIONING THE PROBLEM: WHEN OPS WRITE CODE ADDING SOME DEV: IT WASN’T THAT BAD! RESULTS: I HOPE WE ALL LEARNED SOMETHING HERE TAKEAWAYS: YOU CAN DEV THE DEVOPS TOO! MORE STORYTIME: CONFIGURATION MANAGEMENT
65. @rynchantress UPtime 2017 •Ops code is still code •Start with the biggest pain points first •Maintainability > cleverness •You CAN teach old ops new tricks! •Leave things better than you found them you can dev the devops too <3
66. @rynchantress Uptime 2017 Thank you!