Put Some Dev in Your Devops

Slide 1

Slide 1 text

Put Some Dev in your devops Ryn Daniels  Uptime 2017

Slide 2

Slide 2 text

@rynchantress UPtime 2017 Hurray, devops!

Slide 3

Slide 3 text

@rynchantress Uptime 2017 Ryn Daniels  they/them - @rynchantress Infrastructure Ops Witch, Travis CI

Slide 4

Slide 4 text

@rynchantress Uptime 2017 (the devops) agenda STORYTIME: INFRASTRUCTURE PROVISIONING THE PROBLEM: WHEN OPS WRITE CODE ADDING SOME DEV: IT WASN’T THAT BAD! RESULTS: I HOPE WE ALL LEARNED SOMETHING HERE TAKEAWAYS: YOU CAN DEV THE DEVOPS TOO! MORE STORYTIME: CONFIGURATION MANAGEMENT

Slide 5

Slide 5 text

@rynchantress Uptime 2017 disclaimer

Slide 6

Slide 6 text

@rynchantress 1 Uptime 2017 (the devops) agenda STORYTIME: INFRASTRUCTURE PROVISIONING THE PROBLEM: WHEN OPS WRITE CODE ADDING SOME DEV: IT WASN’T THAT BAD! RESULTS: I HOPE WE ALL LEARNED SOMETHING HERE TAKEAWAYS: YOU CAN DEV THE DEVOPS TOO! MORE STORYTIME: CONFIGURATION MANAGEMENT

Slide 7

Slide 7 text

@rynchantress UPtime 2017 Infrastructure Provisioning

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

@rynchantress UPtime 2017 Inventory Management Disk Partitioning IPMI Configuration Initial Network Setup Infrastructure Provisioning OS Installation DNS Setup Final Network Setup Configuration Management

Slide 10

Slide 10 text

@rynchantress UPtime 2017 Header

Slide 11

Slide 11 text

@rynchantress UPtime 2017 + “we’re just ops people, these are just ops scripts!”

Slide 12

Slide 12 text

@rynchantress UPtime 2017 Inventory Management Disk Partitioning IPMI Configuration Initial Network Setup Self-Service Infrastructure Provisioning OS Installation DNS Setup Final Network Setup Configuration Management Bulk Build Capabilities

Slide 13

Slide 13 text

@rynchantress UPtime 2017 The indigo tool suite •gabriel: OS installation, final network setup, DNS, configuration management •zaar: decommissioning boxes (undoing all of the above)

Slide 14

Slide 14 text

@rynchantress UPtime 2017 •sledgehammer: disk partitioning •unattended sledgehammer: inventory management, initial network setup, IPMI configuration The indigo tool suite (Continued)

Slide 15

Slide 15 text

@rynchantress UPtime 2017 •sledgehammer executor: bootable disk image that installs and loads the… •sledgehammer payload: rpm that sets up and runs all of the unattended sledgehammer tasks The indigo tool suite (Continued Again)

Slide 16

Slide 16 text

@rynchantress UPtime 2017 •Indigo API: Mostly providing a way to avoid handing out duplicate IPMI IPs •Indigo Sweeper: Queuing up Cobbler syncs instead of DDOSing the Cobbler servers •Indigo Tailer: Hey, having logs to look at would be cool! The indigo tool suite (this is the last one i promise)

Slide 17

Slide 17 text

Slide 18

Slide 18 text

@rynchantress 2 Uptime 2017 (the devops) agenda STORYTIME: INFRASTRUCTURE PROVISIONING THE PROBLEM: WHEN OPS WRITE CODE ADDING SOME DEV: IT WASN’T THAT BAD! RESULTS: I HOPE WE ALL LEARNED SOMETHING HERE TAKEAWAYS: YOU CAN DEV THE DEVOPS TOO! MORE STORYTIME: CONFIGURATION MANAGEMENT

Slide 19

Slide 19 text

lolwhoops.

Slide 20

Slide 20 text

@rynchantress UPtime 2017 •No Planning •Low Readability •Duplicated Code •(Human) Single Points of Failure unmaintainable code

Slide 21

Slide 21 text

@rynchantress UPtime 2017 untested code •No Tests at All •Untestable Code •No Deploy Process •“F*** It, We’ll Do It Live”

Slide 22

Slide 22 text

@rynchantress UPtime 2017 critical code •Critical Path for Infrastructure Provisioning •Important for Data Center Team •Critical Code Should Be Tested!

Slide 23

Slide 23 text

@rynchantress 3 Uptime 2017 (the devops) agenda STORYTIME: INFRASTRUCTURE PROVISIONING THE PROBLEM: WHEN OPS WRITE CODE ADDING SOME DEV: IT WASN’T THAT BAD! RESULTS: I HOPE WE ALL LEARNED SOMETHING HERE TAKEAWAYS: YOU CAN DEV THE DEVOPS TOO! MORE STORYTIME: CONFIGURATION MANAGEMENT

Slide 24

Slide 24 text

@rynchantress UPtime 2017 “Oh, I have a CS Degree! I can add some OO, make use of inheritance to cut down on copy pasta, do some refactoring, add some tests…”

Slide 25

Slide 25 text

@rynchantress UPtime 2017 LOL NOPE.

Slide 26

Slide 26 text

@rynchantress UPtime 2017 Don’t “test” code in production.

Slide 27

Slide 27 text

@rynchantress Uptime 2017 “What if we had literally any way to test this at all?” LOL.

Slide 28

Slide 28 text

@rynchantress UPtime 2017 Testing unattended mode

Slide 29

Slide 29 text

@rynchantress uptime 2017 “Uhhhhhh hey data center team, let me know if this new deploy broke anything maybe?” The old way

Slide 30

Slide 30 text

@rynchantress uptime 2017 > sledgehammer.rb -h test-host   --unattended The new way

Slide 31

Slide 31 text

@rynchantress UPtime 2017 adding versioning

Slide 32

Slide 32 text

@rynchantress uptime 2017 The old way (replace payload version 0.1 with new version 0.1) “Uhhhhhh hey data center team, let me know if this new deploy broke anything maybe?”

Slide 33

Slide 33 text

@rynchantress uptime 2017 (build payload version 0.2) sledgehammer.rb -h test-host   --unattended -p 0.2 The new way

Slide 34

Slide 34 text

@rynchantress UPtime 2017 a separate test iso!

Slide 35

Slide 35 text

@rynchantress uptime 2017 The old way (replace sledgehammer ISO with new sledgehammer ISO) “Uhhhhhh hey data center team, let me know if this new deploy broke anything maybe?”

Slide 36

Slide 36 text

@rynchantress uptime 2017 The new way

Slide 37

Slide 37 text

@rynchantress UPtime 2017 creating a test api

Slide 38

Slide 38 text

@rynchantress uptime 2017 The old way (replace old indigo API with new indigo API) “Uhhhhhh hey data center team, let me know if this new deploy broke anything maybe?”

Slide 39

Slide 39 text

@rynchantress uptime 2017 > rackup -olocalhost -p8765 > sledgehammer.rb -h test-host   --unattended -p 0.2 -l 8765 The new way

Slide 40

Slide 40 text

@rynchantress UPtime 2017 •NOT hard-coding one version everywhere! •Local/test versions of different components •Testing the code paths that our customers use most often •Notably less sadness several yaks later…

Slide 41

Slide 41 text

@rynchantress UPtime 2017 deployinator!

Slide 42

Slide 42 text

@rynchantress Uptime 2017

Slide 43

Slide 43 text

@rynchantress UPtime 2017 unit tests! (and refactoring!) ((Finally!!))

Slide 44

Slide 44 text

@rynchantress UPtime 2017 unit testing and refactoring goals • Using inheritance instead of duplicating methods all over • Using optional parameters for mocking/stubbing and creating shorter/simpler methods for more testability • Using tests as additional documentation for what the code is supposed to do • Giving people even more confidence when making changes

Slide 45

Slide 45 text

@rynchantress UPtime 2017 example time! yay!

Slide 46

Slide 46 text

@rynchantress Uptime 2017 def find_switchport(hostname, host_interface) unless ipmi_mac = @assetmgmt_helper.get_mac(hostname, 'ilo') @formatter.log_and_raise("Racktables has no MAC for the IPMI interface. Game over.") end ... end

Slide 47

Slide 47 text

@rynchantress Uptime 2017 def find_switchport(hostname, host_interface, assetmgmt_helper=@assetmgmt_helper) unless ipmi_mac = assetmgmt_helper.get_mac(hostname, 'ilo') @formatter.log_and_raise("Racktables has no MAC for the IPMI interface. Game over.") end ... end

Slide 48

Slide 48 text

@rynchantress Uptime 2017 def test_find_switchport assetmgmt_helper = Indigo::Helpers::RacktablesHelper.new(@config, @formatter) assetmgmt_helper.stubs(:get_mac).returns("00AABBCCDDEE") assetmgmt_helper.stubs(:get_switchname).returns("switch-123.e tsy.com") ipmi_helper = Indigo::Helpers::IpmiHelper.new(@config, @formatter, assetmgmt_helper=assetmgmt_helper) switchport = ipmi_helper.find_switchport("beerops- test.etsy.com", 'ilo', snmp_helper) assert_equal(["switch-123.etsy.com", "gi0/42"], switchport) end

Slide 49

Slide 49 text

No content

Slide 50

Slide 50 text

@rynchantress UPtime 2017 the end?!

Slide 51

Slide 51 text

@rynchantress 4 Uptime 2017 (the devops) agenda STORYTIME: INFRASTRUCTURE PROVISIONING THE PROBLEM: WHEN OPS WRITE CODE ADDING SOME DEV: IT WASN’T THAT BAD! RESULTS: I HOPE WE ALL LEARNED SOMETHING HERE TAKEAWAYS: YOU CAN DEV THE DEVOPS TOO! MORE STORYTIME: CONFIGURATION MANAGEMENT

Slide 52

Slide 52 text

@rynchantress UPtime 2017 the knife-spork dance

Slide 53

Slide 53 text

@rynchantress UPtime 2017 the knife-spork dance knife spork check cookbook_name knife spork upload cookbook_name knife spork promote cookbook_name --remote git commit git push

Slide 54

Slide 54 text

@rynchantress UPtime 2017 the knife-spork dance knife spork check cookbook_name knife spork upload cookbook_name knife spork promote cookbook_name --remote (Jump into slack to figure out who promoted but forgot to commit their changes) (Wait for them to commit and push) git pull knife spork promote cookbook_name --remote (Maybe forget to commit and push, perpetuating the cycle)

Slide 55

Slide 55 text

@rynchantress UPtime 2017 Deployinator!

Slide 56

Slide 56 text

@rynchantress Uptime 2017

Slide 57

Slide 57 text

@rynchantress Uptime 2017

Slide 58

Slide 58 text

No content

Slide 59

Slide 59 text

@rynchantress UPtime 2017 the end?!?!

Slide 60

Slide 60 text

@rynchantress 5 Uptime 2017 (the devops) agenda STORYTIME: INFRASTRUCTURE PROVISIONING THE PROBLEM: WHEN OPS WRITE CODE ADDING SOME DEV: IT WASN’T THAT BAD! RESULTS: I HOPE WE ALL LEARNED SOMETHING HERE TAKEAWAYS: YOU CAN DEV THE DEVOPS TOO! MORE STORYTIME: CONFIGURATION MANAGEMENT

Slide 61

Slide 61 text

happier users

Slide 62

Slide 62 text

@rynchantress UPtime 2017 Fewer (human) SPOFS

Slide 63

Slide 63 text

@rynchantress UPtime 2017 maintainable tools

Slide 64

Slide 64 text

@rynchantress 6 Uptime 2017 (the devops) agenda STORYTIME: INFRASTRUCTURE PROVISIONING THE PROBLEM: WHEN OPS WRITE CODE ADDING SOME DEV: IT WASN’T THAT BAD! RESULTS: I HOPE WE ALL LEARNED SOMETHING HERE TAKEAWAYS: YOU CAN DEV THE DEVOPS TOO! MORE STORYTIME: CONFIGURATION MANAGEMENT

Slide 65

Slide 65 text

@rynchantress UPtime 2017 •Ops code is still code •Start with the biggest pain points first •Maintainability > cleverness •You CAN teach old ops new tricks! •Leave things better than you found them you can dev the devops too <3

Slide 66

Slide 66 text

@rynchantress Uptime 2017 Thank you!