Put Some Dev in Your Devops

Put Some Dev in your devops Ryn Daniels  Uptime 2017

@rynchantress UPtime 2017 Hurray, devops!

@rynchantress Uptime 2017 Ryn Daniels  they/them - @rynchantress Infrastructure Ops
Witch, Travis CI

@rynchantress Uptime 2017 (the devops) agenda STORYTIME: INFRASTRUCTURE PROVISIONING THE
PROBLEM: WHEN OPS WRITE CODE ADDING SOME DEV: IT WASN’T THAT BAD! RESULTS: I HOPE WE ALL LEARNED SOMETHING HERE TAKEAWAYS: YOU CAN DEV THE DEVOPS TOO! MORE STORYTIME: CONFIGURATION MANAGEMENT

@rynchantress Uptime 2017 disclaimer

@rynchantress 1 Uptime 2017 (the devops) agenda STORYTIME: INFRASTRUCTURE PROVISIONING
THE PROBLEM: WHEN OPS WRITE CODE ADDING SOME DEV: IT WASN’T THAT BAD! RESULTS: I HOPE WE ALL LEARNED SOMETHING HERE TAKEAWAYS: YOU CAN DEV THE DEVOPS TOO! MORE STORYTIME: CONFIGURATION MANAGEMENT

@rynchantress UPtime 2017 Infrastructure Provisioning

@rynchantress UPtime 2017 Inventory Management Disk Partitioning IPMI Configuration Initial
Network Setup Infrastructure Provisioning OS Installation DNS Setup Final Network Setup Configuration Management

@rynchantress UPtime 2017 Header

@rynchantress UPtime 2017 + “we’re just ops people, these are
just ops scripts!”

Network Setup Self-Service Infrastructure Provisioning OS Installation DNS Setup Final Network Setup Configuration Management Bulk Build Capabilities

@rynchantress UPtime 2017 The indigo tool suite •gabriel: OS installation,
final network setup, DNS, configuration management •zaar: decommissioning boxes (undoing all of the above)

@rynchantress UPtime 2017 •sledgehammer: disk partitioning •unattended sledgehammer: inventory management,
initial network setup, IPMI configuration The indigo tool suite (Continued)

@rynchantress UPtime 2017 •sledgehammer executor: bootable disk image that installs
and loads the… •sledgehammer payload: rpm that sets up and runs all of the unattended sledgehammer tasks The indigo tool suite (Continued Again)

@rynchantress UPtime 2017 •Indigo API: Mostly providing a way to
avoid handing out duplicate IPMI IPs •Indigo Sweeper: Queuing up Cobbler syncs instead of DDOSing the Cobbler servers •Indigo Tailer: Hey, having logs to look at would be cool! The indigo tool suite (this is the last one i promise)

Network Setup Self-Service Infrastructure Provisioning OS Installation DNS Setup Final Network Setup Configuration Management Bulk Build Capabilities

lolwhoops.

@rynchantress UPtime 2017 •No Planning •Low Readability •Duplicated Code •(Human)
Single Points of Failure unmaintainable code

@rynchantress UPtime 2017 untested code •No Tests at All •Untestable
Code •No Deploy Process •“F*** It, We’ll Do It Live”

@rynchantress UPtime 2017 critical code •Critical Path for Infrastructure Provisioning
•Important for Data Center Team •Critical Code Should Be Tested!

@rynchantress UPtime 2017 “Oh, I have a CS Degree! I
can add some OO, make use of inheritance to cut down on copy pasta, do some refactoring, add some tests…”

@rynchantress UPtime 2017 LOL NOPE.

@rynchantress UPtime 2017 Don’t “test” code in production.

@rynchantress Uptime 2017 “What if we had literally any way
to test this at all?” LOL.

@rynchantress UPtime 2017 Testing unattended mode

@rynchantress uptime 2017 “Uhhhhhh hey data center team, let me
know if this new deploy broke anything maybe?” The old way

@rynchantress uptime 2017 > sledgehammer.rb -h test-host   --unattended The
new way

@rynchantress UPtime 2017 adding versioning

@rynchantress uptime 2017 The old way (replace payload version 0.1
with new version 0.1) “Uhhhhhh hey data center team, let me know if this new deploy broke anything maybe?”

@rynchantress uptime 2017 (build payload version 0.2) sledgehammer.rb -h test-host
  --unattended -p 0.2 The new way

@rynchantress UPtime 2017 a separate test iso!

@rynchantress uptime 2017 The old way (replace sledgehammer ISO with
new sledgehammer ISO) “Uhhhhhh hey data center team, let me know if this new deploy broke anything maybe?”

@rynchantress uptime 2017 The new way

@rynchantress UPtime 2017 creating a test api

@rynchantress uptime 2017 The old way (replace old indigo API
with new indigo API) “Uhhhhhh hey data center team, let me know if this new deploy broke anything maybe?”

@rynchantress uptime 2017 > rackup -olocalhost -p8765 > sledgehammer.rb -h
test-host   --unattended -p 0.2 -l 8765 The new way

@rynchantress UPtime 2017 •NOT hard-coding one version everywhere! •Local/test versions
of different components •Testing the code paths that our customers use most often •Notably less sadness several yaks later…

@rynchantress UPtime 2017 deployinator!

@rynchantress Uptime 2017

@rynchantress UPtime 2017 unit tests! (and refactoring!) ((Finally!!))

@rynchantress UPtime 2017 unit testing and refactoring goals • Using
inheritance instead of duplicating methods all over • Using optional parameters for mocking/stubbing and creating shorter/simpler methods for more testability • Using tests as additional documentation for what the code is supposed to do • Giving people even more confidence when making changes

@rynchantress UPtime 2017 example time! yay!

@rynchantress Uptime 2017 def find_switchport(hostname, host_interface) unless ipmi_mac = @assetmgmt_helper.get_mac(hostname,
'ilo') @formatter.log_and_raise("Racktables has no MAC for the IPMI interface. Game over.") end ... end

@rynchantress Uptime 2017 def find_switchport(hostname, host_interface, assetmgmt_helper=@assetmgmt_helper) unless ipmi_mac =
assetmgmt_helper.get_mac(hostname, 'ilo') @formatter.log_and_raise("Racktables has no MAC for the IPMI interface. Game over.") end ... end

@rynchantress Uptime 2017 def test_find_switchport assetmgmt_helper = Indigo::Helpers::RacktablesHelper.new(@config, @formatter) assetmgmt_helper.stubs(:get_mac).returns("00AABBCCDDEE")
assetmgmt_helper.stubs(:get_switchname).returns("switch-123.e tsy.com") ipmi_helper = Indigo::Helpers::IpmiHelper.new(@config, @formatter, assetmgmt_helper=assetmgmt_helper) switchport = ipmi_helper.find_switchport("beerops- test.etsy.com", 'ilo', snmp_helper) assert_equal(["switch-123.etsy.com", "gi0/42"], switchport) end

@rynchantress UPtime 2017 the end?!

@rynchantress UPtime 2017 the knife-spork dance

@rynchantress UPtime 2017 the knife-spork dance knife spork check cookbook_name
knife spork upload cookbook_name knife spork promote cookbook_name --remote git commit git push

@rynchantress UPtime 2017 the knife-spork dance knife spork check cookbook_name
knife spork upload cookbook_name knife spork promote cookbook_name --remote (Jump into slack to figure out who promoted but forgot to commit their changes) (Wait for them to commit and push) git pull knife spork promote cookbook_name --remote (Maybe forget to commit and push, perpetuating the cycle)

@rynchantress UPtime 2017 Deployinator!

@rynchantress Uptime 2017

@rynchantress UPtime 2017 the end?!?!

happier users

@rynchantress UPtime 2017 Fewer (human) SPOFS

@rynchantress UPtime 2017 maintainable tools

@rynchantress UPtime 2017 •Ops code is still code •Start with
the biggest pain points first •Maintainability > cleverness •You CAN teach old ops new tricks! •Leave things better than you found them you can dev the devops too <3

@rynchantress Uptime 2017 Thank you!

Put Some Dev in Your Devops

Put Some Dev in Your Devops

More Decks by Ryn Daniels

Other Decks in Technology

Featured

Transcript