Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Infra as Code: The Organizational Bottleneck and the Shared Service Anti-Pattern

C8a8889a30543fdb8cf2841a19d43834?s=47 Evan Gilman
September 29, 2016

Infra as Code: The Organizational Bottleneck and the Shared Service Anti-Pattern

As your company's Engineering team grows, and the rate of infrastructure evolution increases, it becomes increasingly apparent that it is not only your Ops team that needs to manipulate the infrastructure codebase - it's everyone! So tools are built, skills are taught, and before you know it, the infrastructure codebase is the most contentious in the whole organization. This talk chronicles PagerDuty's journey from Ops-driven evolution to distributed evolution, the pains involved and the hard lessons learned.

C8a8889a30543fdb8cf2841a19d43834?s=128

Evan Gilman

September 29, 2016
Tweet

Transcript

  1. 9/25/16 @evan2645 Infra as Code: The Organizational Bottleneck and the

    Shared Service Anti-Pattern EVAN GILMAN
  2. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN About Me
  3. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN
  4. 9/25/16 @evan2645 This Talk is NOT RESILIENT INFRASTRUCTURE ORCHESTRATION WITH

    SERF
  5. 9/25/16 @evan2645 This Talk IS RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF

  6. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN Infra as Code at PagerDuty Chef Evolution at PagerDuty Human Implication of Shared Services What We’re Trying and How to Avoid Agenda
  7. 9/25/16 @evan2645 Chef RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF Nomad +

    Mesos
  8. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN Chef at PagerDuty
  9. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN • Per-App Cookbooks Chef at PagerDuty
  10. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN • Per-App Cookbooks • Many Common Cookbooks Chef at PagerDuty
  11. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN • Per-App Cookbooks • Many Common Cookbooks • Mono-Repo Chef at PagerDuty
  12. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN • Per-App Cookbooks • Many Common Cookbooks • Mono-Repo • No Versioning Chef at PagerDuty
  13. 9/25/16 @evan2645 Through the Ages RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF

  14. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN Baby Chef
  15. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN Baby Chef
  16. 9/25/16 @evan2645 PD-Specific Tooling RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF

  17. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN • Lots of Chef Search PD-Specific Tooling
  18. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN • Lots of Chef Search • Lots of External Calls PD-Specific Tooling
  19. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN PD-Specific Tooling • Lots of Chef Search • Lots of External Calls • Linear Scaling == Bad
  20. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN PD-Specific Tooling • Lots of Chef Search • Lots of External Calls • Linear Scaling == Bad • Chef as Shared Resource
  21. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN • Lots of Chef Search • Lots of External Calls • Linear Scaling == Bad • Chef as Shared Resource • APIs as Shared Resources PD-Specific Tooling
  22. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN • Instrument Everything • Concerted Effort • Some Specialize • Convention Emerges Optimization
  23. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN Optimization • Instrument Everything • Concerted Effort • Some Specialize • Convention Emerges
  24. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN Continued Growth
  25. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN • Copy/Paste! Continued Growth
  26. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN • Copy/Paste! • Refactors! Continued Growth
  27. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN • Copy/Paste! • Refactors! • Services rarely identical in practice Continued Growth
  28. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN • Copy/Paste! • Refactors! • Services rarely identical in practice • Limited Ops bandwidth Continued Growth
  29. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN • Copy/Paste! • Refactors! • Services rarely identical in practice • Limited Ops bandwidth • Service creation time skyrockets Continued Growth
  30. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN • Copy/Paste! • Refactors! • Services rarely identical in practice • Limited Ops bandwidth • Service creation time skyrockets Continued Growth
  31. 9/25/16 @evan2645 Chef For All! RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF

  32. 9/25/16 @evan2645 RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF Strong Convention

  33. 9/25/16 @evan2645 CONTRIBUTING.md RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF

  34. 9/25/16 @evan2645 CONTRIBUTING.md RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF

  35. 9/25/16 @evan2645 But… I just copy/pasted! RESILIENT INFRASTRUCTURE ORCHESTRATION WITH

    SERF
  36. 9/25/16 @evan2645 But… It’s just one query… RESILIENT INFRASTRUCTURE ORCHESTRATION

    WITH SERF
  37. 9/25/16 @evan2645 Hey… That’s not in CONTRIBUTING.md! RESILIENT INFRASTRUCTURE ORCHESTRATION

    WITH SERF
  38. 9/25/16 @evan2645 C’mon… I just wanna get stuff done RESILIENT

    INFRASTRUCTURE ORCHESTRATION WITH SERF
  39. 9/25/16 @evan2645 Shared Services Considered Harmful RESILIENT INFRASTRUCTURE ORCHESTRATION WITH

    SERF
  40. 9/25/16 @evan2645 Shared Services Considered Harmful RESILIENT INFRASTRUCTURE ORCHESTRATION WITH

    SERF
  41. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN Solution?
  42. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN Solution? Dev needs != Ops needs
  43. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN Solution? Dev needs != Ops needs Chef interface complex and powerful
  44. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN Solution? Dev needs != Ops needs Chef interface complex and powerful Like an ATV instead of a Golf Cart
  45. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN Dev needs != Ops needs Chef interface complex and powerful Like an ATV instead of a Golf Cart Focus on Dev experience Solution?
  46. 9/25/16 @evan2645 Data as a Service Definition RESILIENT INFRASTRUCTURE ORCHESTRATION

    WITH SERF
  47. 9/25/16 @evan2645 Artifact + Service Definition RESILIENT INFRASTRUCTURE ORCHESTRATION WITH

    SERF
  48. 9/25/16 @evan2645 CI/CD Pipeline for Workloads RESILIENT INFRASTRUCTURE ORCHESTRATION WITH

    SERF
  49. 9/25/16 @evan2645 CI/CD Pipeline for Workloads RESILIENT INFRASTRUCTURE ORCHESTRATION WITH

    SERF (with linters)
  50. 9/25/16 @evan2645 Work is Ongoing… RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF

  51. 9/25/16 @evan2645 Work is Ongoing… RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF

    (we need help!)
  52. 9/25/16 @evan2645 Avoidance? RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF

  53. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN Always Know Your Customer Focus on the Infrastructure Interface/API Don’t Fight Conway’s Law Be Product-Focused
  54. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN Always Know Your Customer Focus on the Infrastructure Interface/API Don’t Fight Conway’s Law Be Product-Focused
  55. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN Always Know Your Customer Focus on the Infrastructure Interface/API Don’t Fight Conway’s Law Be Product-Focused
  56. 9/25/16 @evan2645 Thank You! RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF

  57. 9/25/16 @evan2645 Infra as Code: The Organizational Bottleneck and the

    Shared Service Anti-Pattern EVAN GILMAN