The road to Continuous Deployment

Deploying code changes continuously to your production environment has many benefits. New features get to your users much faster which enables very short and quick iterations on them. And due to the small change sets you tend to encounter less bugs and regressions. In this talk I’ll present what Continuous Deployment means, and what you have to do, step-by-step, to make it work successfully for your project. We will cover topics like trunk based development, feature flags, partial roll outs, continuous integration, deployment automation and application monitoring.

Transcript

1. The road to Continuous Deployment @BastianHofmann
2. What?
3. Why?
4. How?
5. A few words about me
13. Questions? Ask
14. http://speakerdeck.com/u/bastianhofmann
15. ?
16. Company?
17. Freelancing?
18. Single product?
19. Agency?
20. Do you know … / Do you do …
21. Continuous Integration?
22. Continuous Deployment?
23. Feature flags?
24. Central log management?
25. KPIs?
26. If not, you will after the talk.
27. So
28. Continuous Deployment
29. Deploying code changes to production continuously
30. Classic approach
31. Waterfall
32. http://en.wikipedia.org/wiki/File:Waterfall_model.svg
33. Web! Hello?
34. Agile
35. http://en.wikipedia.org/wiki/File:Scrum_process.svg
36. Going further
37. Continuous Deployment
38. Benefits
39. Smaller change sets
40. Less Bugs
41. Finding the bug
42. Shortest „time to bugfix“ possible
43. Better performance
44. Better conversion
45. Faster iterations
46. Awesome, I’m sold
47. How can I introduce it?
48. The whole team has to live it
49. Not only doing more releases!
50. Step by step
51. is very important
52. Manual QA
53. does not scale
54. Automated Tests
55. Unit Tests
56. Integration Tests
57. Acceptance Tests
58. Smoke Tests
59. JS Tests
60. Regression Tests
61. Continuous Integration
62. Jenkins http://jenkins-ci.org/
64. Development process
65. Feature branches are bad
68. Merge Hell
69. -> Big changes
70. Trunk/master based development
71. Every commit gets integrated
72. Every commit gets deployed
73. No commits that are not stable
74. Frequent release branches with cherry pics
75. http://paulhammant.com/2013/03/04/facebook-tbd/
76. http://paulhammant.com/2013/05/06/googles-scaled-trunk-based-development/
77. http://paulhammant.com/2013/03/04/facebook-tbd/
78. But there is stuff that takes longer then an hour to develop
79. Feature Flags/ Toggles
80. Release !== Deployment
81. public function hasAccess( $accountId, array $roleNames ) { return featureFlag()->isActive( FeatureFlag::TEST_ONE ); }
84. Partial rollout
85. Experiments (A/B Testing)
86. other also possible: goal early integration
87. Finally: Getting your code live
88. Automatic builds
89. Composer
90. Bower
91. NPM
92. Browserify
93. JS
94. CSS
95. HTML
96. Static caches
97. Ant
98. Phing
99. Grunt
101. Custom scripts
102. Fast builds
103. Hook it into Jenkins
104. Automatic deployment
105. One-click- deployment
106. Fast deployment
107. Fast rollbacks
108. http://www.ansible.com/
109. http://capistranorb.com/
110. Etsy Deploynator http://codeascraft.com/2010/05/20/quantum-of-deployment/
111. Custom scripts
112. Cache warmup
113. Asset invalidation
114. https://www.npmjs.org/package/grunt-cache-invalidate https://www.npmjs.org/package/gulp-rev https://www.npmjs.org/package/grunt-static-ctrl
115. Staging System(s)
116. Central DEV Production Near Production Integration
117. Build/Test/Release pipeline
119. Schema migrations
121. Non BC breaking
122. Only safe operations
123. Migration in steps
124. No software is bug free
125. Finding bugs as early as possible
126. Detecting errors
128. Central log management
129. Logstash, Elasticsearch, Kibana
130. Detecting regressions
131. Bugs
132. Performance
133. KPIs
134. Plotting everything
135. Graphite http://graphite.wikidot.com/
137. Finding the cause
138. Anomaly detection
139. http://codeascraft.com/2013/06/11/introducing-kale/
140. Profiling
141. XHProf https://github.com/facebook/xhprof
142. Use it in production for a subset of requests
143. XHGUI https://github.com/perftools/xhgui
146. Qafoo Profiler https://profiler.qafoolabs.com/
148. New Relic http://newrelic.com/
150. Symfony Debug Toolbar
153. Extend it http://symfony.com/doc/current/cookbook/profiler/ data_collector.html
156. Aggregate it
158. Synopsis
159. Invest in your processes
160. Invest in quality
161. Invest in performance
162. Invest in agility
163. Continuous Deployment
164. http://twitter.com/BastianHofmann http://lanyrd.com/people/BastianHofmann http://speakerdeck.com/u/bastianhofmann mail@bastianhofmann.de
166. Aggregate the logs in a central place
167. Make them easily full-text searchable
168. Make them aggregate-able
169. Always Log to file
170. Seriously...
171. Always Log to file
172. Directly to a database
173. webserver webserver webserver DB
174. Disadvantages
175. Database is down?
176. Database is slow?
177. Database is full?
178. How to integrate access logs?
179. Influences application performance
180. Frontend?
181. Better solutions?
182. Logstash + Elasticsearch + Kibana
183. Full text search
184. Structured Messages
185. Always Log to file
186. Logstash elasticsearch webserver webserver webserver AMQP log log log logstash logstash logstash
187. logstash http://logstash.net/
188. input filter output
189. Very rich plugin system
190. Logstash elasticsearch webserver webserver webserver AMQP log log log logstash logstash logstash
191. input { file { type => "error" path => [ "/var/logs/php/*.log" ] add_field => [ "severity", "error" ] } file { type => "access" path => [ "/var/logs/apache/*_access.log" ] add_field => [ "severity", "info" ] }
192. filter{ grok { match => ["@source", "\/%{USERNAME:facility}\.log$"] } grok { type => "access" pattern => "^%{IP:OriginalIp} \s[a-zA-Z0-9_-]+\s[a-zA-Z0-9_-]+\s\[.*? \]\s\"%{DATA:Request}..." } }
193. output { amqp { host => "amqp.host" exchange_type => "fanout" name => "logs" } }
194. Logstash elasticsearch webserver webserver webserver AMQP log log log logstash logstash logstash
195. input {
 rabbitmq {
 queue => "logs"
 host => "amqp.host"
 exchange => "ls_exchange"
 exclusive => false
 }
 }
196. filter { grok {
 match => [
 'Type','EntityNotFoundException| TimeoutException'
 ]
 break_on_match => true
 add_tag => ['not_serious']
 tag_on_failure => ['serious']
 } }
197. output {
 elasticsearch {
 embedded => false
 bind_host => "localhost"
 bind_port => "9305"
 host => "localhost"
 cluster => "logs"
 }
 }
198. Kibana http://www.elasticsearch.org/overview/kibana/
200. Bonus material
201. Remember: Response Times
202. Easiest approach:
     Get it out of the access log
203. 192.168.56.1 - - [09/Jul/2012:19:18:19 +0200] "GET /rg_trunk/webroot/c/af10c/ images/template/rg_logo_default.png HTTP/ 1.1" 200 882 "http://devm/rg_trunk/webroot/ directory/publications/"
204. Is this actually what we want to measure?
205. External Resources CSS JavaScript Images Slow connections Slow Browsers SSL Handshakes DNS resolving
206. That is what we want to measure!
207. https://github.com/lognormal/boomerang
208. browser JS: boomerang logstash trackingServer access log requests tracking image with timing information as query parameters graphite statsd