Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The road to Continuous Deployment

The road to Continuous Deployment

Deploying code changes continuously to your production environment has many benefits. New features get to your users much faster which enables very short and quick iterations on them. And due to the small change sets you tend to encounter less bugs and regressions. In this talk I’ll present what Continuous Deployment means, and what you have to do, step-by-step, to make it work successfully for your project. We will cover topics like trunk based development, feature flags, partial roll outs, continuous integration, deployment automation and application monitoring.

8e82eb7e128a14a16d642ae55227339b?s=128

Bastian Hofmann

October 30, 2014
Tweet

Transcript

  1. The road to Continuous Deployment @BastianHofmann

  2. What?

  3. Why?

  4. How?

  5. A few words about me

  6. None
  7. None
  8. None
  9. None
  10. None
  11. None
  12. None
  13. Questions? Ask

  14. http://speakerdeck.com/u/bastianhofmann

  15. ?

  16. Company?

  17. Freelancing?

  18. Single product?

  19. Agency?

  20. Do you know … / Do you do …

  21. Continuous Integration?

  22. Continuous Deployment?

  23. Feature flags?

  24. Central log management?

  25. KPIs?

  26. If not, you will after the talk.

  27. So

  28. Continuous Deployment

  29. Deploying code changes to production continuously

  30. Classic approach

  31. Waterfall

  32. http://en.wikipedia.org/wiki/File:Waterfall_model.svg

  33. Web! Hello?

  34. Agile

  35. http://en.wikipedia.org/wiki/File:Scrum_process.svg

  36. Going further

  37. Continuous Deployment

  38. Benefits

  39. Smaller change sets

  40. Less Bugs

  41. Finding the bug

  42. Shortest „time to bugfix“ possible

  43. Better performance

  44. Better conversion

  45. Faster iterations

  46. Awesome, I’m sold

  47. How can I introduce it?

  48. The whole team has to live it

  49. Not only doing more releases!

  50. Step by step

  51. is very important

  52. Manual QA

  53. does not scale

  54. Automated Tests

  55. Unit Tests

  56. Integration Tests

  57. Acceptance Tests

  58. Smoke Tests

  59. JS Tests

  60. Regression Tests

  61. Continuous Integration

  62. Jenkins http://jenkins-ci.org/

  63. None
  64. Development process

  65. Feature branches are bad

  66. None
  67. None
  68. Merge Hell

  69. -> Big changes

  70. Trunk/master based development

  71. Every commit gets integrated

  72. Every commit gets deployed

  73. No commits that are not stable

  74. Frequent release branches with cherry pics

  75. http://paulhammant.com/2013/03/04/facebook-tbd/

  76. http://paulhammant.com/2013/05/06/googles-scaled-trunk-based-development/

  77. http://paulhammant.com/2013/03/04/facebook-tbd/

  78. But there is stuff that takes longer then an hour

    to develop
  79. Feature Flags/ Toggles

  80. Release !== Deployment

  81. public function hasAccess( $accountId, array $roleNames ) { return featureFlag()->isActive(

    FeatureFlag::TEST_ONE ); }
  82. None
  83. None
  84. Partial rollout

  85. Experiments (A/B Testing)

  86. other also possible: goal early integration

  87. Finally: Getting your code live

  88. Automatic builds

  89. Composer

  90. Bower

  91. NPM

  92. Browserify

  93. JS

  94. CSS

  95. HTML

  96. Static caches

  97. Ant

  98. Phing

  99. Grunt

  100. None
  101. Custom scripts

  102. Fast builds

  103. Hook it into Jenkins

  104. Automatic deployment

  105. One-click- deployment

  106. Fast deployment

  107. Fast rollbacks

  108. http://www.ansible.com/

  109. http://capistranorb.com/

  110. Etsy Deploynator http://codeascraft.com/2010/05/20/quantum-of-deployment/

  111. Custom scripts

  112. Cache warmup

  113. Asset invalidation

  114. https://www.npmjs.org/package/grunt-cache-invalidate https://www.npmjs.org/package/gulp-rev https://www.npmjs.org/package/grunt-static-ctrl

  115. Staging System(s)

  116. Central DEV Production Near Production Integration

  117. Build/Test/Release pipeline

  118. None
  119. Schema migrations

  120. None
  121. Non BC breaking

  122. Only safe operations

  123. Migration in steps

  124. No software is bug free

  125. Finding bugs as early as possible

  126. Detecting errors

  127. None
  128. Central log management

  129. Logstash, Elasticsearch, Kibana

  130. Detecting regressions

  131. Bugs

  132. Performance

  133. KPIs

  134. Plotting everything

  135. Graphite http://graphite.wikidot.com/

  136. None
  137. Finding the cause

  138. Anomaly detection

  139. http://codeascraft.com/2013/06/11/introducing-kale/

  140. Profiling

  141. XHProf https://github.com/facebook/xhprof

  142. Use it in production for a subset of requests

  143. XHGUI https://github.com/perftools/xhgui

  144. None
  145. None
  146. Qafoo Profiler https://profiler.qafoolabs.com/

  147. None
  148. New Relic http://newrelic.com/

  149. None
  150. Symfony Debug Toolbar

  151. None
  152. None
  153. Extend it http://symfony.com/doc/current/cookbook/profiler/ data_collector.html

  154. None
  155. None
  156. Aggregate it

  157. None
  158. Synopsis

  159. Invest in your processes

  160. Invest in quality

  161. Invest in performance

  162. Invest in agility

  163. Continuous Deployment

  164. http://twitter.com/BastianHofmann http://lanyrd.com/people/BastianHofmann http://speakerdeck.com/u/bastianhofmann mail@bastianhofmann.de

  165. None
  166. Aggregate the logs in a central place

  167. Make them easily full-text searchable

  168. Make them aggregate-able

  169. Always Log to file

  170. Seriously...

  171. Always Log to file

  172. Directly to a database

  173. webserver webserver webserver DB

  174. Disadvantages

  175. Database is down?

  176. Database is slow?

  177. Database is full?

  178. How to integrate access logs?

  179. Influences application performance

  180. Frontend?

  181. Better solutions?

  182. Logstash + Elasticsearch + Kibana

  183. Full text search

  184. Structured Messages

  185. Always Log to file

  186. Logstash elasticsearch webserver webserver webserver AMQP log log log logstash

    logstash logstash
  187. logstash http://logstash.net/

  188. input filter output

  189. Very rich plugin system

  190. Logstash elasticsearch webserver webserver webserver AMQP log log log logstash

    logstash logstash
  191. input { file { type => "error" path => [

    "/var/logs/php/*.log" ] add_field => [ "severity", "error" ] } file { type => "access" path => [ "/var/logs/apache/*_access.log" ] add_field => [ "severity", "info" ] }
  192. filter{ grok { match => ["@source", "\/%{USERNAME:facility}\.log$"] } grok {

    type => "access" pattern => "^%{IP:OriginalIp} \s[a-zA-Z0-9_-]+\s[a-zA-Z0-9_-]+\s\[.*? \]\s\"%{DATA:Request}..." } }
  193. output { amqp { host => "amqp.host" exchange_type => "fanout"

    name => "logs" } }
  194. Logstash elasticsearch webserver webserver webserver AMQP log log log logstash

    logstash logstash
  195. input {
 rabbitmq {
 queue => "logs"
 host => "amqp.host"


    exchange => "ls_exchange"
 exclusive => false
 }
 }
  196. filter { grok {
 match => [
 'Type','EntityNotFoundException| TimeoutException'
 ]


    break_on_match => true
 add_tag => ['not_serious']
 tag_on_failure => ['serious']
 } }
  197. output {
 elasticsearch {
 embedded => false
 bind_host => "localhost"


    bind_port => "9305"
 host => "localhost"
 cluster => "logs"
 }
 }
  198. Kibana http://www.elasticsearch.org/overview/kibana/

  199. None
  200. Bonus material

  201. Remember: Response Times

  202. Easiest approach: Get it out of the access log

  203. 192.168.56.1 - - [09/Jul/2012:19:18:19 +0200] "GET /rg_trunk/webroot/c/af10c/ images/template/rg_logo_default.png HTTP/ 1.1"

    200 882 "http://devm/rg_trunk/webroot/ directory/publications/"
  204. Is this actually what we want to measure?

  205. External Resources CSS JavaScript Images Slow connections Slow Browsers SSL

    Handshakes DNS resolving
  206. That is what we want to measure!

  207. https://github.com/lognormal/boomerang

  208. browser JS: boomerang logstash trackingServer access log requests tracking image

    with timing information as query parameters graphite statsd