A Pragmatic Approach to Rails Deployments and Operations

Lessons learned from deploying and managing hundreds of Rails applications

Transcript

1. Lessons learned from deploying and managing hundreds of Rails applications A Pragmatic Approach to Rails Deployments and Operations Monday, May 28, 12
2. Lessons learned from deploying and managing hundreds of Rails applications Rails Deployments and Operations Monday, May 28, 12
3. o hai, im josh Monday, May 28, 12
4. •@techpickles •http://github.com/technicalpickles •http://technicalpickles.com Monday, May 28, 12
5. Monday, May 28, 12
6. Awesomeness Engineer of Supreme Versatility III Monday, May 28, 12
7. Managed hosting and operations Monday, May 28, 12
8. I’m talking about Rails Operations Monday, May 28, 12
9. WTF is Rails Operations? Monday, May 28, 12
10. Development and maintenance of a production Rails application Monday, May 28, 12
11. Before we dig in too deep... Monday, May 28, 12
12. Monday, May 28, 12
13. Does development generate revenue? Question Monday, May 28, 12
14. Monday, May 28, 12
15. •Takes place on laptops, desktop machines, staging servers Monday, May 28, 12
16. •Takes place on laptops, desktop machines, staging servers •No real users Monday, May 28, 12
17. •Takes place on laptops, desktop machines, staging servers •No real users •Unknown if it truly works Monday, May 28, 12
18. •Takes place on laptops, desktop machines, staging servers •No real users •Unknown if it truly works •Tests are green, but... Monday, May 28, 12
19. Monday, May 28, 12
20. •Step 1: Development Monday, May 28, 12
21. •Step 1: Development •Step 2: Operations Monday, May 28, 12
22. •Step 1: Development •Step 2: Operations •Step 3: ...... Monday, May 28, 12
23. •Step 1: Development •Step 2: Operations •Step 3: ...... •Step 4: PROFIT! Monday, May 28, 12
24. NO Monday, May 28, 12
25. but it CREATES potential revenue Monday, May 28, 12
26. Monday, May 28, 12
27. •Step 1: Development Monday, May 28, 12
28. •Step 1: Development •Step 2: ....... Monday, May 28, 12
29. •Step 1: Development •Step 2: ....... •Step 3: PROFIT Monday, May 28, 12
30. Does operations generate revenue? Question Monday, May 28, 12
31. Monday, May 28, 12
32. •Lives on servers located in data centers and clouds Monday, May 28, 12
33. •Lives on servers located in data centers and clouds •Real users Monday, May 28, 12
34. •Lives on servers located in data centers and clouds •Real users •Either code works, or it doesn’t Monday, May 28, 12
35. •Lives on servers located in data centers and clouds •Real users •Either code works, or it doesn’t •Either the application is available or not Monday, May 28, 12
36. NO Monday, May 28, 12
37. but it PRESERVES potential revenue Monday, May 28, 12
38. Uh, what generates revenue? Question Monday, May 28, 12
39. Million Dollar Question Monday, May 28, 12
40. Monday, May 28, 12
41. •Working features (or at least that work enough) Monday, May 28, 12
42. •Working features (or at least that work enough) •Infrastructure to keep the application up and running (or at least up enough) Monday, May 28, 12
43. •Working features (or at least that work enough) •Infrastructure to keep the application up and running (or at least up enough) •A business model that works (or at least works enough) Monday, May 28, 12
44. •Working features (or at least that work enough) •Infrastructure to keep the application up and running (or at least up enough) •A business model that works (or at least works enough) •??????????????????????? Monday, May 28, 12
45. Lessons learned Monday, May 28, 12
46. Common threads Monday, May 28, 12
47. Monday, May 28, 12
48. Monday, May 28, 12
49. Monday, May 28, 12
50. Pattern Configuration Management Monday, May 28, 12
51. duh. It’s about managing configuration. Monday, May 28, 12
52. You write code that manages your servers’ configuration Monday, May 28, 12
53. Monday, May 28, 12
54. •apache package is installed Monday, May 28, 12
55. •apache package is installed •apache service is running Monday, May 28, 12
56. •apache package is installed •apache service is running •deploy user exists Monday, May 28, 12
57. •apache package is installed •apache service is running •deploy user exists •cron jobs Monday, May 28, 12
58. •apache package is installed •apache service is running •deploy user exists •cron jobs •etc Monday, May 28, 12
59. •Puppet •Chef •Moonshine Monday, May 28, 12
60. Automation Monday, May 28, 12
61. Monday, May 28, 12
62. Monday, May 28, 12
63. Pattern Staging Environments Monday, May 28, 12
64. Preproduction servers Monday, May 28, 12
65. Gives you a place to ensure correctness of deploy Monday, May 28, 12
66. configuration management + staging servers = VERY YES Monday, May 28, 12
67. Monday, May 28, 12
68. Monday, May 28, 12
69. Repeat after me Monday, May 28, 12
70. Repeat after me • development Monday, May 28, 12
71. Repeat after me • development • staging Monday, May 28, 12
72. Repeat after me • development • staging • production Monday, May 28, 12
73. capistrano-gitflow Monday, May 28, 12
74. Pattern Deploy early, deploy often Monday, May 28, 12
75. Although technically, I guess it’s the same A play on release early, release often. Monday, May 28, 12
76. Monday, May 28, 12
77. Monday, May 28, 12
78. Antipattern Feeling Driven Development Monday, May 28, 12
79. Monday, May 28, 12
80. The front page feels slow Monday, May 28, 12
81. The primary key seems like it’s increasing rapidly Monday, May 28, 12
82. IO seems high Monday, May 28, 12
83. What does it even mean? Monday, May 28, 12
84. Counterpattern Science Driven Development Monday, May 28, 12
85. Monday, May 28, 12
86. Metrics everywhere! Monday, May 28, 12
87. •New Relic - http://newrelic.com •Scout - http://scoutapp.com Monday, May 28, 12
88. The front page is taking 10 seconds to load, but we really need it to be loading in under 1 second The front page feels slow Monday, May 28, 12
89. The primary key is at 90% of it’s maximum, up from 80% yesterday, and looks like it’ll run out overnight. The primary key seems like it’s increasing rapidly Monday, May 28, 12
90. IO fluctuates up to 90% sometimes, but doesn’t appear to have a negative effect IO seems high Monday, May 28, 12
91. Monday, May 28, 12
92. Topic Monitoring Monday, May 28, 12
93. How do you know when everything is awful? Monday, May 28, 12
94. How would you prefer to know? Monday, May 28, 12
95. How would you prefer to know? •A late night phone call? Monday, May 28, 12
96. How would you prefer to know? •A late night phone call? •Angry tweets that you are down? Monday, May 28, 12
97. How would you prefer to know? •A late night phone call? •Angry tweets that you are down? •You personally checking everything all the time? Monday, May 28, 12
98. How would you prefer to know? •A late night phone call? •Angry tweets that you are down? •You personally checking everything all the time? •An automated system to let you know? Monday, May 28, 12
99. •Nagios •Scout •New Relic •Pingdom Monday, May 28, 12
100. What to monitor Monday, May 28, 12
101. Define urgency Monday, May 28, 12
102. Must be actionable Monday, May 28, 12
103. Single point of contact Monday, May 28, 12
104. Monday, May 28, 12
105. Pattern Vertical scaling Monday, May 28, 12
106. Now what? Your app is slow Monday, May 28, 12
107. Monday, May 28, 12
108. Resources are (relatively) cheap Monday, May 28, 12
109. Developers are (relatively) expensive Monday, May 28, 12
110. Monday, May 28, 12
111. Monday, May 28, 12
112. Antipattern Hipster Stack Monday, May 28, 12
113. Monday, May 28, 12
114. “I read a blog post about how mongo is totally web scale” Monday, May 28, 12
115. Cargo cult operations Monday, May 28, 12
116. Monday, May 28, 12
117. Monday, May 28, 12
118. Monday, May 28, 12
119. If you’re still going to go hipster... •experiment in branches •understand operational impact •Staging! Monday, May 28, 12
120. .... wait, what? Test in production Monday, May 28, 12
121. Fin. Monday, May 28, 12
122. Further Reading •Web Operations - John Allspaw and Jesse Robins •Continuous Delivery - Jez Humble and David Farley •“Web Operations for Developers 101” Monday, May 28, 12
123. find me here josh@railsmachine.com @techpickles Want to talk ops? Monday, May 28, 12