A Pragmatic Approach to Rails Deployments and Operations

Lessons learned from deploying and managing hundreds of Rails applications

Transcript

1. Lessons learned from deploying and managing
   hundreds of Rails applications
   A Pragmatic
   Approach to Rails
   Deployments and
   Operations
   Monday, May 28, 12
   

   View Slide

2. Lessons learned from deploying and managing
   hundreds of Rails applications
   Rails Deployments
   and Operations
   Monday, May 28, 12
   

   View Slide

3. o hai, im josh
   Monday, May 28, 12
   

   View Slide

4. •@techpickles
   •http://github.com/technicalpickles
   •http://technicalpickles.com
   Monday, May 28, 12
   

   View Slide

5. Monday, May 28, 12
   

   View Slide

6. Awesomeness
   Engineer of
   Supreme Versatility
   III
   Monday, May 28, 12
   

   View Slide

7. Managed hosting
   and operations
   Monday, May 28, 12
   

   View Slide

8. I’m talking about
   Rails Operations
   Monday, May 28, 12
   

   View Slide

9. WTF is
   Rails Operations?
   Monday, May 28, 12
   

   View Slide

10. Development and maintenance
    of a production Rails application
    Monday, May 28, 12
    

    View Slide

11. Before we dig in too
    deep...
    Monday, May 28, 12
    

    View Slide

12. Monday, May 28, 12
    

    View Slide

13. Does development generate revenue?
    Question
    Monday, May 28, 12
    

    View Slide

14. Monday, May 28, 12
    

    View Slide

15. •Takes place on laptops, desktop
    machines, staging servers
    Monday, May 28, 12
    

    View Slide

16. •Takes place on laptops, desktop
    machines, staging servers
    •No real users
    Monday, May 28, 12
    

    View Slide

17. •Takes place on laptops, desktop
    machines, staging servers
    •No real users
    •Unknown if it truly works
    Monday, May 28, 12
    

    View Slide

18. •Takes place on laptops, desktop
    machines, staging servers
    •No real users
    •Unknown if it truly works
    •Tests are green, but...
    Monday, May 28, 12
    

    View Slide

19. Monday, May 28, 12
    

    View Slide

20. •Step 1: Development
    Monday, May 28, 12
    

    View Slide

21. •Step 1: Development
    •Step 2: Operations
    Monday, May 28, 12
    

    View Slide

22. •Step 1: Development
    •Step 2: Operations
    •Step 3: ......
    Monday, May 28, 12
    

    View Slide

23. •Step 1: Development
    •Step 2: Operations
    •Step 3: ......
    •Step 4: PROFIT!
    Monday, May 28, 12
    

    View Slide

24. NO
    Monday, May 28, 12
    

    View Slide

25. but it CREATES
    potential revenue
    Monday, May 28, 12
    

    View Slide

26. Monday, May 28, 12
    

    View Slide

27. •Step 1: Development
    Monday, May 28, 12
    

    View Slide

28. •Step 1: Development
    •Step 2: .......
    Monday, May 28, 12
    

    View Slide

29. •Step 1: Development
    •Step 2: .......
    •Step 3: PROFIT
    Monday, May 28, 12
    

    View Slide

30. Does operations generate revenue?
    Question
    Monday, May 28, 12
    

    View Slide

31. Monday, May 28, 12
    

    View Slide

32. •Lives on servers located in data
    centers and clouds
    Monday, May 28, 12
    

    View Slide

33. •Lives on servers located in data
    centers and clouds
    •Real users
    Monday, May 28, 12
    

    View Slide

34. •Lives on servers located in data
    centers and clouds
    •Real users
    •Either code works, or it doesn’t
    Monday, May 28, 12
    

    View Slide

35. •Lives on servers located in data
    centers and clouds
    •Real users
    •Either code works, or it doesn’t
    •Either the application is available or
    not
    Monday, May 28, 12
    

    View Slide

36. NO
    Monday, May 28, 12
    

    View Slide

37. but it PRESERVES
    potential revenue
    Monday, May 28, 12
    

    View Slide

38. Uh, what generates revenue?
    Question
    Monday, May 28, 12
    

    View Slide

39. Million Dollar
    Question
    Monday, May 28, 12
    

    View Slide

40. Monday, May 28, 12
    

    View Slide

41. •Working features (or at least that
    work enough)
    Monday, May 28, 12
    

    View Slide

42. •Working features (or at least that
    work enough)
    •Infrastructure to keep the application
    up and running (or at least up
    enough)
    Monday, May 28, 12
    

    View Slide

43. •Working features (or at least that
    work enough)
    •Infrastructure to keep the application
    up and running (or at least up
    enough)
    •A business model that works (or at
    least works enough)
    Monday, May 28, 12
    

    View Slide

44. •Working features (or at least that
    work enough)
    •Infrastructure to keep the application
    up and running (or at least up
    enough)
    •A business model that works (or at
    least works enough)
    •???????????????????????
    Monday, May 28, 12
    

    View Slide

45. Lessons learned
    Monday, May 28, 12
    

    View Slide

46. Common threads
    Monday, May 28, 12
    

    View Slide

47. Monday, May 28, 12
    

    View Slide

48. Monday, May 28, 12
    

    View Slide

49. Monday, May 28, 12
    

    View Slide

50. Pattern
    Configuration
    Management
    Monday, May 28, 12
    

    View Slide

51. duh.
    It’s about managing
    configuration.
    Monday, May 28, 12
    

    View Slide

52. You write code that
    manages your
    servers’
    configuration
    Monday, May 28, 12
    

    View Slide

53. Monday, May 28, 12
    

    View Slide

54. •apache package is installed
    Monday, May 28, 12
    

    View Slide

55. •apache package is installed
    •apache service is running
    Monday, May 28, 12
    

    View Slide

56. •apache package is installed
    •apache service is running
    •deploy user exists
    Monday, May 28, 12
    

    View Slide

57. •apache package is installed
    •apache service is running
    •deploy user exists
    •cron jobs
    Monday, May 28, 12
    

    View Slide

58. •apache package is installed
    •apache service is running
    •deploy user exists
    •cron jobs
    •etc
    Monday, May 28, 12
    

    View Slide

59. •Puppet
    •Chef
    •Moonshine
    Monday, May 28, 12
    

    View Slide

60. Automation
    Monday, May 28, 12
    

    View Slide

61. Monday, May 28, 12
    

    View Slide

62. Monday, May 28, 12
    

    View Slide

63. Pattern
    Staging
    Environments
    Monday, May 28, 12
    

    View Slide

64. Preproduction
    servers
    Monday, May 28, 12
    

    View Slide

65. Gives you a place to
    ensure correctness
    of deploy
    Monday, May 28, 12
    

    View Slide

66. configuration
    management
    +
    staging servers
    =
    VERY YES
    Monday, May 28, 12
    

    View Slide

67. Monday, May 28, 12
    

    View Slide

68. Monday, May 28, 12
    

    View Slide

69. Repeat after me
    Monday, May 28, 12
    

    View Slide

70. Repeat after me
    • development
    Monday, May 28, 12
    

    View Slide

71. Repeat after me
    • development
    • staging
    Monday, May 28, 12
    

    View Slide

72. Repeat after me
    • development
    • staging
    • production
    Monday, May 28, 12
    

    View Slide

73. capistrano-gitflow
    Monday, May 28, 12
    

    View Slide

74. Pattern
    Deploy early,
    deploy often
    Monday, May 28, 12
    

    View Slide

75. Although technically, I guess it’s the same
    A play on release
    early, release often.
    Monday, May 28, 12
    

    View Slide

76. Monday, May 28, 12
    

    View Slide

77. Monday, May 28, 12
    

    View Slide

78. Antipattern
    Feeling Driven
    Development
    Monday, May 28, 12
    

    View Slide

79. Monday, May 28, 12
    

    View Slide

80. The front page feels
    slow
    Monday, May 28, 12
    

    View Slide

81. The primary key
    seems like it’s
    increasing rapidly
    Monday, May 28, 12
    

    View Slide

82. IO seems high
    Monday, May 28, 12
    

    View Slide

83. What does it even
    mean?
    Monday, May 28, 12
    

    View Slide

84. Counterpattern
    Science Driven
    Development
    Monday, May 28, 12
    

    View Slide

85. Monday, May 28, 12
    

    View Slide

86. Metrics everywhere!
    Monday, May 28, 12
    

    View Slide

87. •New Relic - http://newrelic.com
    •Scout - http://scoutapp.com
    Monday, May 28, 12
    

    View Slide

88. The front page is taking 10 seconds to load, but
    we really need it to be loading in under 1
    second
    The front page feels
    slow
    Monday, May 28, 12
    

    View Slide

89. The primary key is at 90% of it’s maximum, up
    from 80% yesterday, and looks like it’ll run out
    overnight.
    The primary key
    seems like it’s
    increasing rapidly
    Monday, May 28, 12
    

    View Slide

90. IO fluctuates up to 90% sometimes, but doesn’t
    appear to have a negative effect
    IO seems high
    Monday, May 28, 12
    

    View Slide

91. Monday, May 28, 12
    

    View Slide

92. Topic
    Monitoring
    Monday, May 28, 12
    

    View Slide

93. How do you know
    when everything is
    awful?
    Monday, May 28, 12
    

    View Slide

94. How would you
    prefer to know?
    Monday, May 28, 12
    

    View Slide

95. How would you
    prefer to know?
    •A late night phone call?
    Monday, May 28, 12
    

    View Slide

96. How would you
    prefer to know?
    •A late night phone call?
    •Angry tweets that you are down?
    Monday, May 28, 12
    

    View Slide

97. How would you
    prefer to know?
    •A late night phone call?
    •Angry tweets that you are down?
    •You personally checking everything
    all the time?
    Monday, May 28, 12
    

    View Slide

98. How would you
    prefer to know?
    •A late night phone call?
    •Angry tweets that you are down?
    •You personally checking everything
    all the time?
    •An automated system to let you
    know?
    Monday, May 28, 12
    

    View Slide

99. •Nagios
    •Scout
    •New Relic
    •Pingdom
    Monday, May 28, 12
    

    View Slide

100. What to monitor
     Monday, May 28, 12
     

     View Slide

101. Define urgency
     Monday, May 28, 12
     

     View Slide

102. Must be actionable
     Monday, May 28, 12
     

     View Slide

103. Single point of
     contact
     Monday, May 28, 12
     

     View Slide

104. Monday, May 28, 12
     

     View Slide

105. Pattern
     Vertical scaling
     Monday, May 28, 12
     

     View Slide

106. Now what?
     Your app is slow
     Monday, May 28, 12
     

     View Slide

107. Monday, May 28, 12
     

     View Slide

108. Resources are
     (relatively) cheap
     Monday, May 28, 12
     

     View Slide

109. Developers are
     (relatively)
     expensive
     Monday, May 28, 12
     

     View Slide

110. Monday, May 28, 12
     

     View Slide

111. Monday, May 28, 12
     

     View Slide

112. Antipattern
     Hipster Stack
     Monday, May 28, 12
     

     View Slide

113. Monday, May 28, 12
     

     View Slide

114. “I read a blog post
     about how mongo is
     totally web scale”
     Monday, May 28, 12
     

     View Slide

115. Cargo cult
     operations
     Monday, May 28, 12
     

     View Slide

116. Monday, May 28, 12
     

     View Slide

117. Monday, May 28, 12
     

     View Slide

118. Monday, May 28, 12
     

     View Slide

119. If you’re still going
     to go hipster...
     •experiment in branches
     •understand operational impact
     •Staging!
     Monday, May 28, 12
     

     View Slide

120. .... wait, what?
     Test in production
     Monday, May 28, 12
     

     View Slide

121. Fin.
     Monday, May 28, 12
     

     View Slide

122. Further Reading
     •Web Operations - John Allspaw and
     Jesse Robins
     •Continuous Delivery - Jez Humble
     and David Farley
     •“Web Operations for Developers
     101”
     Monday, May 28, 12
     

     View Slide

123. find me here
     josh@railsmachine.com
     @techpickles
     Want to talk ops?
     Monday, May 28, 12
     

     View Slide