Scalable Meatfrastructure: Building Stable DevOps Teams

Presented at LISA15 on November 11, 2015

When we think of scalable infrastructure, we think of technologies such as AWS, Kafka, or Mesos. But to operate these shiny things, you need people. Organized and resilient people.

You wouldn’t rollout containerization while your app has a memory leak, so you shouldn’t expand your DevOps team without addressing the leaks in your day-to-day communication and processes. Come learn some of the steps necessary to make your most important platform--your engineers--stable and successful.

Transcript

1. Scalable Meatfrastructure
   Building stable DevOps teams
   Alice Goldfuss
   @alicegoldfuss
   

   View Slide

2. I’m
   Alice
   SRE @
   

   View Slide

3. View Slide

4. View Slide

5. View Slide

6. View Slide

7. View Slide

8. What is
   Meatfrastructure?
   

   View Slide

9. View Slide

10. Enable Logging
    PIPE STDOUT
    Default configuration
    Stable distribution
    Reliable under high load
    

    View Slide

11. Enable Logging
    PIPE STDOUT
    Default configuration
    Stable distribution
    Reliable under high load
    >
    

    View Slide

12. The
    bus
    Factor
    

    View Slide

13. Oral to Written
    

    View Slide

14. Story time
    

    View Slide

15. Write it all down
    • Your team’s mission
    • Your team’s chosen tools and
    methodologies
    • Standard practices and procedures
    

    View Slide

16. View Slide

17. Ship it
    

    View Slide

18. View Slide

19. Enable Logging
    PIPE STDOUT
    Default configuration
    Stable distribution
    Reliable under high load
    >
    

    View Slide

20. Success!
    

    View Slide

21. View Slide

22. View Slide

23. View Slide

24. Single Source of Truth
    • Everyone knows where it is
    • Everyone can contribute to it
    • It’s searchable
    • It’s archivable
    

    View Slide

25. • GitHub pages for company
    processes
    • Gists for configuration changes
    • Repos for runbooks
    

    View Slide

26. View Slide

27. Hold
    up
    

    View Slide

28. View Slide

29. View Slide

30. View Slide

31. View Slide

32. View Slide

33. View Slide

34. View Slide

35. View Slide

36. View Slide

37. View Slide

38. View Slide

39. View Slide

40. View Slide

41. View Slide

42. View Slide

43. Ship it
    

    View Slide

44. What’s on
    fire today?
    

    View Slide

45. What’s on fire today?
    

    View Slide

46. What’s on fire today?
    

    View Slide

47. What’s on fire today?
    Coffee doesn’t
    make processors
    run faster :(
    

    View Slide

48. Enable Logging
    PIPE STDOUT
    Default configuration
    Stable distribution
    Reliable under high load
    >
    

    View Slide

49. View Slide

50. View Slide

51. View Slide

52. View Slide

53. be a
    hero
    

    View Slide

54. the
    sysadmin
    will see
    you now
    

    View Slide

55. View Slide

56. View Slide

57. “It’s just a
    proof of concept.”
    - someone you
    hate now
    

    View Slide

58. View Slide

59. What’s approved?
    • Languages
    • Frameworks
    • Tools
    

    View Slide

60. A Real World Example!
    

    View Slide

61. ship it
    

    View Slide

62. View Slide

63. View Slide

64. Enable Logging
    PIPE STDOUT
    Default configuration
    Stable distribution
    Reliable under high load
    >
    

    View Slide

65. View Slide

66. View Slide

67. View Slide

68. me
    

    View Slide

69. Left out
    

    View Slide

70. Learning
    slowed
    

    View Slide

71. Out of
    sight
    

    View Slide

72. View Slide

73. View Slide

74. no
    

    View Slide

75. Level the playing field
    • Take notes during stand-ups
    • Record meetings
    • Fly them in
    

    View Slide

76. Matt Headroom
    

    View Slide

77. swag drop
    

    View Slide

78. View Slide

79. Ship it
    

    View Slide

80. View Slide

81. Enable Logging
    PIPE STDOUT
    Default configuration
    Stable distribution
    Reliable under high load
    >
    

    View Slide

82. View Slide

83. View Slide

84. View Slide

85. View Slide

86. View Slide

87. Where to start?
    • Big shop? Get invested
    • Small shop? Get creative
    

    View Slide

88. View Slide

89. ship it
    

    View Slide

90. View Slide

91. Enable Logging
    PIPE STDOUT
    Default configuration
    Stable distribution
    Reliable under high load
    

    View Slide

92. share info
    in one place
    help other teams help you
    make remote workers equal
    make onboarding smooth
    

    View Slide

93. share info
    in one place
    help other teams help you
    make remote workers equal
    make onboarding smooth
    >
    

    View Slide

94. View Slide

95. share info
    in one place
    help other teams help you
    make remote workers equal
    make onboarding smooth
    >
    

    View Slide

96. What’s on fire today?
    Coffee doesn’t
    make processors
    run faster :(
    

    View Slide

97. share info
    in one place
    help other teams help you
    make remote workers equal
    make onboarding smooth
    >
    

    View Slide

98. View Slide

99. share info
    in one place
    help other teams help you
    make remote workers equal
    make onboarding smooth
    >
    

    View Slide

100. View Slide

101. share info
     in one place
     help other teams help you
     make remote workers equal
     make onboarding smooth
     >
     

     View Slide

102. View Slide

103. almost
     done
     

     View Slide

104. Thanks!
     @alicegoldfuss
     

     View Slide

105. Safe Harbor
     This document and the information herein (including any information that may be incorporated by reference) is
     provided for informational purposes only and should not be construed as an offer, commitment, promise or
     obligation on behalf of New Relic, Inc. (“New Relic”) to sell securities or deliver any product, material, code,
     functionality, or other feature. Any information provided hereby is proprietary to New Relic and may not be
     replicated or disclosed without New Relic’s express written permission.
     Such information may contain forward-looking statements within the meaning of federal securities laws. Any
     statement that is not a historical fact or refers to expectations, projections, future plans, objectives, estimates,
     goals, or other characterizations of future events is a forward-looking statement. These forward-looking
     statements can often be identified as such because the context of the statement will include words such as
     “believes,” “anticipates,” “expects” or words of similar import.
     Actual results may differ materially from those expressed in these forward-looking statements, which speak only
     as of the date hereof, and are subject to change at any time without notice. Existing and prospective investors,
     customers and other third parties transacting business with New Relic are cautioned not to place undue reliance
     on this forward-looking information. The achievement or success of the matters covered by such forward-looking
     statements are based on New Relic’s current assumptions, expectations, and beliefs and are subject to
     substantial risks, uncertainties, assumptions, and changes in circumstances that may cause the actual results,
     performance, or achievements to differ materially from those expressed or implied in any forward-looking
     statement. Further information on factors that could affect such forward-looking statements is included in the
     filings we make with the SEC from time to time. Copies of these documents may be obtained by visiting New
     Relic’s Investor Relations website at ir.newrelic.com or the SEC’s website at www.sec.gov.
     New Relic assumes no obligation and does not intend to update these forward-looking statements, except as
     required by law. New Relic makes no warranties, expressed or implied, in this document or otherwise, with
     respect to the information provided.
     

     View Slide

106. Images (Slide #, Source)
     1 National Cancer Institute https://visualsonline.cancer.gov/details.cfm?
     imageid=2402
     2 http://www.flickr.com/photos/24365773@N03/6296336021
     8 https://www.flickr.com/photos/boston_public_library/10312241466
     9 https://www.flickr.com/photos/restlessglobetrotter/2661016046
     12 https://www.flickr.com/photos/126433814@N04/15144220507/
     13 https://www.flickr.com/photos/seattlemunicipalarchives/3974145507/
     14 https://www.flickr.com/photos/paurian/3577372039
     16 https://www.flickr.com/photos/42931449@N07/5397530925/
     17 https://www.flickr.com/photos/usoceangov/5369581593/
     18 https://www.flickr.com/photos/kkoshy/7364348244
     22 https://www.flickr.com/photos/yaketyyakyak/7001670664/
     

     View Slide

107. 27 https://www.flickr.com/photos/oceanyamaha/186146223/
     28 https://www.flickr.com/photos/26010466@N07/4073794760/
     31 https://www.flickr.com/photos/iicd/5349200426/
     32 Bletchley Park Trust/Science & Society Picture Library https://www.flickr.com/
     photos/101251639@N02/9669449367/
     33 https://www.flickr.com/photos/leszekleszczynski/9325662885/
     35 https://www.flickr.com/photos/expertinfantry/5467513914/
     36 https://www.flickr.com/photos/mdalmuld/7180902804/
     37 https://www.flickr.com/photos/usdagov/13472498405
     42 https://www.flickr.com/photos/shropshiretelfordtsb/16685300160/
     43 https://www.flickr.com/photos/portofsandiego/6105051324/
     44 openclipart.org
     45 openclipart.org
     50 https://www.flickr.com/photos/dccentralkitchen/16166140493/
     51 https://www.flickr.com/photos/spine/419733435/
     

     View Slide

108. 53 https://www.flickr.com/photos/wedhyherman/16733805724
     54 https://www.flickr.com/photos/95072945@N05/8667775349/
     55 https://www.flickr.com/photos/evelynishere/3687971867/
     56 https://www.flickr.com/photos/notionscapital/7417864934/
     57 https://www.flickr.com/photos/95072945@N05/11282868043/
     58 https://www.flickr.com/photos/buzzymelibee/8690774332/
     61 https://www.flickr.com/photos/dmertl/5949203837/
     65 TaxRebate.org.uk https://www.flickr.com/photos/59937401@N07/5857483483/
     66 https://www.flickr.com/photos/wocintechchat/21910907593/
     69 https://www.flickr.com/photos/evanblaser/6123759194
     71 https://www.flickr.com/photos/pokerbrit/9137670399/
     72 https://www.flickr.com/photos/mcbarnicle/4030630210
     73 https://www.flickr.com/photos/nostri-imago/3279464108/
     77 https://www.flickr.com/photos/108291893@N02/10788841805/
     

     View Slide

109. 80 https://www.flickr.com/photos/marilynjane/5877663279
     82 https://www.flickr.com/photos/jayneandd/4450623309
     84 StockMonkeys.com https://www.flickr.com/photos/86530412@N02/8194661881
     85 https://www.flickr.com/photos/michaelduxbury/5824469025
     86 https://www.flickr.com/photos/riebart/4466482623
     88 https://www.flickr.com/photos/90412460@N00/12115039876
     89 https://www.flickr.com/photos/bvi4092/12026957146
     103 https://www.flickr.com/photos/london/902817172
     All other images are in the public domain. Try pexels.com for a great selection.
     

     View Slide