Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A Pragmatic Approach to Rails Deployments and Operations

A Pragmatic Approach to Rails Deployments and Operations

Lessons learned from deploying and managing hundreds of Rails applications

technicalpickles

May 28, 2012
Tweet

Other Decks in Technology

Transcript

  1. Lessons learned from deploying and managing
    hundreds of Rails applications
    A Pragmatic
    Approach to Rails
    Deployments and
    Operations
    Monday, May 28, 12

    View full-size slide

  2. Lessons learned from deploying and managing
    hundreds of Rails applications
    Rails Deployments
    and Operations
    Monday, May 28, 12

    View full-size slide

  3. o hai, im josh
    Monday, May 28, 12

    View full-size slide

  4. •@techpickles
    •http://github.com/technicalpickles
    •http://technicalpickles.com
    Monday, May 28, 12

    View full-size slide

  5. Monday, May 28, 12

    View full-size slide

  6. Awesomeness
    Engineer of
    Supreme Versatility
    III
    Monday, May 28, 12

    View full-size slide

  7. Managed hosting
    and operations
    Monday, May 28, 12

    View full-size slide

  8. I’m talking about
    Rails Operations
    Monday, May 28, 12

    View full-size slide

  9. WTF is
    Rails Operations?
    Monday, May 28, 12

    View full-size slide

  10. Development and maintenance
    of a production Rails application
    Monday, May 28, 12

    View full-size slide

  11. Before we dig in too
    deep...
    Monday, May 28, 12

    View full-size slide

  12. Monday, May 28, 12

    View full-size slide

  13. Does development generate revenue?
    Question
    Monday, May 28, 12

    View full-size slide

  14. Monday, May 28, 12

    View full-size slide

  15. •Takes place on laptops, desktop
    machines, staging servers
    Monday, May 28, 12

    View full-size slide

  16. •Takes place on laptops, desktop
    machines, staging servers
    •No real users
    Monday, May 28, 12

    View full-size slide

  17. •Takes place on laptops, desktop
    machines, staging servers
    •No real users
    •Unknown if it truly works
    Monday, May 28, 12

    View full-size slide

  18. •Takes place on laptops, desktop
    machines, staging servers
    •No real users
    •Unknown if it truly works
    •Tests are green, but...
    Monday, May 28, 12

    View full-size slide

  19. Monday, May 28, 12

    View full-size slide

  20. •Step 1: Development
    Monday, May 28, 12

    View full-size slide

  21. •Step 1: Development
    •Step 2: Operations
    Monday, May 28, 12

    View full-size slide

  22. •Step 1: Development
    •Step 2: Operations
    •Step 3: ......
    Monday, May 28, 12

    View full-size slide

  23. •Step 1: Development
    •Step 2: Operations
    •Step 3: ......
    •Step 4: PROFIT!
    Monday, May 28, 12

    View full-size slide

  24. NO
    Monday, May 28, 12

    View full-size slide

  25. but it CREATES
    potential revenue
    Monday, May 28, 12

    View full-size slide

  26. Monday, May 28, 12

    View full-size slide

  27. •Step 1: Development
    Monday, May 28, 12

    View full-size slide

  28. •Step 1: Development
    •Step 2: .......
    Monday, May 28, 12

    View full-size slide

  29. •Step 1: Development
    •Step 2: .......
    •Step 3: PROFIT
    Monday, May 28, 12

    View full-size slide

  30. Does operations generate revenue?
    Question
    Monday, May 28, 12

    View full-size slide

  31. Monday, May 28, 12

    View full-size slide

  32. •Lives on servers located in data
    centers and clouds
    Monday, May 28, 12

    View full-size slide

  33. •Lives on servers located in data
    centers and clouds
    •Real users
    Monday, May 28, 12

    View full-size slide

  34. •Lives on servers located in data
    centers and clouds
    •Real users
    •Either code works, or it doesn’t
    Monday, May 28, 12

    View full-size slide

  35. •Lives on servers located in data
    centers and clouds
    •Real users
    •Either code works, or it doesn’t
    •Either the application is available or
    not
    Monday, May 28, 12

    View full-size slide

  36. NO
    Monday, May 28, 12

    View full-size slide

  37. but it PRESERVES
    potential revenue
    Monday, May 28, 12

    View full-size slide

  38. Uh, what generates revenue?
    Question
    Monday, May 28, 12

    View full-size slide

  39. Million Dollar
    Question
    Monday, May 28, 12

    View full-size slide

  40. Monday, May 28, 12

    View full-size slide

  41. •Working features (or at least that
    work enough)
    Monday, May 28, 12

    View full-size slide

  42. •Working features (or at least that
    work enough)
    •Infrastructure to keep the application
    up and running (or at least up
    enough)
    Monday, May 28, 12

    View full-size slide

  43. •Working features (or at least that
    work enough)
    •Infrastructure to keep the application
    up and running (or at least up
    enough)
    •A business model that works (or at
    least works enough)
    Monday, May 28, 12

    View full-size slide

  44. •Working features (or at least that
    work enough)
    •Infrastructure to keep the application
    up and running (or at least up
    enough)
    •A business model that works (or at
    least works enough)
    •???????????????????????
    Monday, May 28, 12

    View full-size slide

  45. Lessons learned
    Monday, May 28, 12

    View full-size slide

  46. Common threads
    Monday, May 28, 12

    View full-size slide

  47. Monday, May 28, 12

    View full-size slide

  48. Monday, May 28, 12

    View full-size slide

  49. Monday, May 28, 12

    View full-size slide

  50. Pattern
    Configuration
    Management
    Monday, May 28, 12

    View full-size slide

  51. duh.
    It’s about managing
    configuration.
    Monday, May 28, 12

    View full-size slide

  52. You write code that
    manages your
    servers’
    configuration
    Monday, May 28, 12

    View full-size slide

  53. Monday, May 28, 12

    View full-size slide

  54. •apache package is installed
    Monday, May 28, 12

    View full-size slide

  55. •apache package is installed
    •apache service is running
    Monday, May 28, 12

    View full-size slide

  56. •apache package is installed
    •apache service is running
    •deploy user exists
    Monday, May 28, 12

    View full-size slide

  57. •apache package is installed
    •apache service is running
    •deploy user exists
    •cron jobs
    Monday, May 28, 12

    View full-size slide

  58. •apache package is installed
    •apache service is running
    •deploy user exists
    •cron jobs
    •etc
    Monday, May 28, 12

    View full-size slide

  59. •Puppet
    •Chef
    •Moonshine
    Monday, May 28, 12

    View full-size slide

  60. Automation
    Monday, May 28, 12

    View full-size slide

  61. Monday, May 28, 12

    View full-size slide

  62. Monday, May 28, 12

    View full-size slide

  63. Pattern
    Staging
    Environments
    Monday, May 28, 12

    View full-size slide

  64. Preproduction
    servers
    Monday, May 28, 12

    View full-size slide

  65. Gives you a place to
    ensure correctness
    of deploy
    Monday, May 28, 12

    View full-size slide

  66. configuration
    management
    +
    staging servers
    =
    VERY YES
    Monday, May 28, 12

    View full-size slide

  67. Monday, May 28, 12

    View full-size slide

  68. Monday, May 28, 12

    View full-size slide

  69. Repeat after me
    Monday, May 28, 12

    View full-size slide

  70. Repeat after me
    • development
    Monday, May 28, 12

    View full-size slide

  71. Repeat after me
    • development
    • staging
    Monday, May 28, 12

    View full-size slide

  72. Repeat after me
    • development
    • staging
    • production
    Monday, May 28, 12

    View full-size slide

  73. capistrano-gitflow
    Monday, May 28, 12

    View full-size slide

  74. Pattern
    Deploy early,
    deploy often
    Monday, May 28, 12

    View full-size slide

  75. Although technically, I guess it’s the same
    A play on release
    early, release often.
    Monday, May 28, 12

    View full-size slide

  76. Monday, May 28, 12

    View full-size slide

  77. Monday, May 28, 12

    View full-size slide

  78. Antipattern
    Feeling Driven
    Development
    Monday, May 28, 12

    View full-size slide

  79. Monday, May 28, 12

    View full-size slide

  80. The front page feels
    slow
    Monday, May 28, 12

    View full-size slide

  81. The primary key
    seems like it’s
    increasing rapidly
    Monday, May 28, 12

    View full-size slide

  82. IO seems high
    Monday, May 28, 12

    View full-size slide

  83. What does it even
    mean?
    Monday, May 28, 12

    View full-size slide

  84. Counterpattern
    Science Driven
    Development
    Monday, May 28, 12

    View full-size slide

  85. Monday, May 28, 12

    View full-size slide

  86. Metrics everywhere!
    Monday, May 28, 12

    View full-size slide

  87. •New Relic - http://newrelic.com
    •Scout - http://scoutapp.com
    Monday, May 28, 12

    View full-size slide

  88. The front page is taking 10 seconds to load, but
    we really need it to be loading in under 1
    second
    The front page feels
    slow
    Monday, May 28, 12

    View full-size slide

  89. The primary key is at 90% of it’s maximum, up
    from 80% yesterday, and looks like it’ll run out
    overnight.
    The primary key
    seems like it’s
    increasing rapidly
    Monday, May 28, 12

    View full-size slide

  90. IO fluctuates up to 90% sometimes, but doesn’t
    appear to have a negative effect
    IO seems high
    Monday, May 28, 12

    View full-size slide

  91. Monday, May 28, 12

    View full-size slide

  92. Topic
    Monitoring
    Monday, May 28, 12

    View full-size slide

  93. How do you know
    when everything is
    awful?
    Monday, May 28, 12

    View full-size slide

  94. How would you
    prefer to know?
    Monday, May 28, 12

    View full-size slide

  95. How would you
    prefer to know?
    •A late night phone call?
    Monday, May 28, 12

    View full-size slide

  96. How would you
    prefer to know?
    •A late night phone call?
    •Angry tweets that you are down?
    Monday, May 28, 12

    View full-size slide

  97. How would you
    prefer to know?
    •A late night phone call?
    •Angry tweets that you are down?
    •You personally checking everything
    all the time?
    Monday, May 28, 12

    View full-size slide

  98. How would you
    prefer to know?
    •A late night phone call?
    •Angry tweets that you are down?
    •You personally checking everything
    all the time?
    •An automated system to let you
    know?
    Monday, May 28, 12

    View full-size slide

  99. •Nagios
    •Scout
    •New Relic
    •Pingdom
    Monday, May 28, 12

    View full-size slide

  100. What to monitor
    Monday, May 28, 12

    View full-size slide

  101. Define urgency
    Monday, May 28, 12

    View full-size slide

  102. Must be actionable
    Monday, May 28, 12

    View full-size slide

  103. Single point of
    contact
    Monday, May 28, 12

    View full-size slide

  104. Monday, May 28, 12

    View full-size slide

  105. Pattern
    Vertical scaling
    Monday, May 28, 12

    View full-size slide

  106. Now what?
    Your app is slow
    Monday, May 28, 12

    View full-size slide

  107. Monday, May 28, 12

    View full-size slide

  108. Resources are
    (relatively) cheap
    Monday, May 28, 12

    View full-size slide

  109. Developers are
    (relatively)
    expensive
    Monday, May 28, 12

    View full-size slide

  110. Monday, May 28, 12

    View full-size slide

  111. Monday, May 28, 12

    View full-size slide

  112. Antipattern
    Hipster Stack
    Monday, May 28, 12

    View full-size slide

  113. Monday, May 28, 12

    View full-size slide

  114. “I read a blog post
    about how mongo is
    totally web scale”
    Monday, May 28, 12

    View full-size slide

  115. Cargo cult
    operations
    Monday, May 28, 12

    View full-size slide

  116. Monday, May 28, 12

    View full-size slide

  117. Monday, May 28, 12

    View full-size slide

  118. Monday, May 28, 12

    View full-size slide

  119. If you’re still going
    to go hipster...
    •experiment in branches
    •understand operational impact
    •Staging!
    Monday, May 28, 12

    View full-size slide

  120. .... wait, what?
    Test in production
    Monday, May 28, 12

    View full-size slide

  121. Fin.
    Monday, May 28, 12

    View full-size slide

  122. Further Reading
    •Web Operations - John Allspaw and
    Jesse Robins
    •Continuous Delivery - Jez Humble
    and David Farley
    •“Web Operations for Developers
    101”
    Monday, May 28, 12

    View full-size slide

  123. find me here
    [email protected]
    @techpickles
    Want to talk ops?
    Monday, May 28, 12

    View full-size slide