Fire It Up: How Empowered People Automated Provisioning in 6 Datacenters Across 4 Continents

This talk is a case study of a group of Rackers who automated network device provisioning by leveraging new and existing services.

Transcript

1. March 1, 2013
   How Empowered People Automated
   Provisioning in 6 Datacenters Across 4
   Continents
   Fire It Up
   Friday, March 1, 13
   

   View Slide

2. Me
   Friday, March 1, 13
   

   View Slide

3. Friday, March 1, 13
   

   View Slide

4. Friday, March 1, 13
   

   View Slide

5. Friday, March 1, 13
   

   View Slide

6. Friday, March 1, 13
   

   View Slide

7. even the cloud runs on hardware
   Friday, March 1, 13
   

   View Slide

8. trade-offs
   Friday, March 1, 13
   

   View Slide

9. SOME MEN JUST WANT TO WATCH
   THE WORLD BURN
   Friday, March 1, 13
   

   View Slide

10. Firewalls and Load Balancers
    =
    Devices
    Friday, March 1, 13
    

    View Slide

11. OS
    customization
    files = configs
    Friday, March 1, 13
    

    View Slide

12. Friday, March 1, 13
    

    View Slide

13. Friday, March 1, 13
    

    View Slide

14. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
    Provisioning Through Time
    14
    Friday, March 1, 13
    

    View Slide

15. Every
    technology
    goes
    through
    three
    stages:
    Friday, March 1, 13
    

    View Slide

16. first a crudely simple and
    quite unsatisfactory
    gadget;
    Friday, March 1, 13
    

    View Slide

17. second, an enormously
    complicated group of gadgets
    designed to overcome the
    shortcomings of the original
    and achieving thereby
    somewhat satisfactory
    performance through extremely
    complex compromise;
    Friday, March 1, 13
    

    View Slide

18. third, a final proper
    design therefrom.
    Friday, March 1, 13
    

    View Slide

19. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
    19
    Friday, March 1, 13
    

    View Slide

20. Friday, March 1, 13
    

    View Slide

21. what about that web app thingy people
    used to manually generate configs of
    dubious quality?
    Friday, March 1, 13
    

    View Slide

22. Rule of
    Separation
    Friday, March 1, 13
    

    View Slide

23. mechanism
    Friday, March 1, 13
    

    View Slide

24. policy
    Friday, March 1, 13
    

    View Slide

25. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
    Extracting A Service
    25
    Friday, March 1, 13
    

    View Slide

26. be your own client first
    Friday, March 1, 13
    

    View Slide

27. remain framework agnostic at the
    domain level
    Friday, March 1, 13
    

    View Slide

28. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
    Lack of Ubiquitous Language
    28
    Friday, March 1, 13
    

    View Slide

29. Friday, March 1, 13
    

    View Slide

30. Automating a process is just as much
    about organizational change as
    technical change
    Friday, March 1, 13
    

    View Slide

31. Friday, March 1, 13
    

    View Slide

32. Friday, March 1, 13
    

    View Slide

33. /swimlanes
    !"" deployment.rb
    !"" discovery.rb
    !"" network.rb
    #"" preparation.rb
    Friday, March 1, 13
    

    View Slide

34. class Swimlanes::Discovery < Swimlane
    # ... omitted ...
    def perform
    DeviceDiscovery.run(device)
    rescue => error
    @errors << error
    raise error
    ensure
    Orchestration.notify(device, self)
    end
    end
    Friday, March 1, 13
    

    View Slide

35. •learning together
    •easier troubleshooting
    Friday, March 1, 13
    

    View Slide

36. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
    Inflexible Consumers
    36
    Friday, March 1, 13
    

    View Slide

37. some API consumers make their own
    tradeoffs which limit their flexibility
    Friday, March 1, 13
    

    View Slide

38. POST /devices/12345/swimlanes/discovery
    Friday, March 1, 13
    

    View Slide

39. POST /devices/123456/kicks
    { “step”: “1 - Discovery” }
    Friday, March 1, 13
    

    View Slide

40. class KicksController < ApplicationController
    # ... omitted ...
    def create
    @kick_process = KickProcess.start(@device,
    params[:step])
    respond_with(@kick_process)
    end
    end
    Friday, March 1, 13
    

    View Slide

41. class KickProcess
    STEPS = {
    ‘1 - Discovery’ => [ Swimlanes::Discovery ]
    }
    def self.start(device, step)
    kick_process = new(device, step)
    kick_process.start
    end
    def start
    STEPS[step].each do |action|
    action.perform(device)
    end
    self
    end
    end
    Friday, March 1, 13
    

    View Slide

42. •maintains flexibility
    •isolates inflexibility
    Friday, March 1, 13
    

    View Slide

43. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
    Coding Bottlenecks
    43
    Friday, March 1, 13
    

    View Slide

44. service objects
    Friday, March 1, 13
    

    View Slide

45. increased surface area
    Friday, March 1, 13
    

    View Slide

46. class DeviceDiscovery
    def self.run(device)
    device_discovery = new(device)
    device_discovery.run
    end
    def run
    switch = FE::Switch.discover(device)
    mac = switch.mac_address.
    select { |m|
    m == device.mac_address
    }.first
    FE::MacAddress.update(mac, device)
    end
    end
    Friday, March 1, 13
    

    View Slide

47. class DeviceDiscovery
    def self.run(device)
    device_discovery = new(device)
    device_discovery.run
    end
    def run
    switch = Switch.discover(device)
    mac = MacAddress.discover(switch)
    MacAddressCorrelation.
    create(device, mac)
    end
    end
    Friday, March 1, 13
    

    View Slide

48. easier troubleshooting
    Friday, March 1, 13
    

    View Slide

49. •increased surface area
    •easier composition
    •troubleshooting speed
    Friday, March 1, 13
    

    View Slide

50. trade-offs
    Friday, March 1, 13
    

    View Slide

51. https://twitter.com/plibin/status/106078180976238592
    Friday, March 1, 13
    

    View Slide

52. BOOM. Value added.
    Friday, March 1, 13
    

    View Slide

53. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
    THANK YOU
    @jschairb
    joshua.schairbaum@gmail.com
    53
    Friday, March 1, 13
    

    View Slide