Continuous Delivery: a journey not a destination

All too often we hear teams talk about Continuous Delivery as a destination. There are questions asked along the lines of what are the things a team needs to do in order to qualify as doing Continuous Delivery.

We believe this is a flawed line of questioning. Continuous Delivery is not a destination that is arrived at by doing a fixed set of steps.

We posit that Continuous Delivery is a value system adopted by teams that informs their choice of architectural principles, the space of available designs, the selection of development practices, the choice of tools and most importantly, the culture of collaboration that they embrace.

Thought of in this way, we believe Continuous Delivery can have as much of an impact on our software delivery process as testability and TDD have on our development processes.

In this talk we will share our experiences on how Continuous Delivery helped us evolve our architecture and choice of tools on our current project SnapCI, a hosted Continuous Delivery tool. We will present how our early design of many well defined Microservices, highly customised but unreliable monitoring and configuration management evolved into a much simpler architecture and a more reliable product.

We hope that using examples such as feature toggles over branches, starting as a Monolith over Microservices and others, our audience takes away how they can start practising CD on their projects and use it to drive their architecture design and choice of tools and technologies.

Transcript

1. CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION Akshay Karle & Fernando Junior @snap_ci | https://snap-ci.com 1
2. 2 The essence of my philosophy to software delivery is to build software so that it is always in a state where it could be put into production. We call this Continuous Delivery because we are continuously running a deployment pipeline that tests if this software is in a state to be delivered. - Jez Humble http://martinfowler.com/delivery.html
3. 3 High-performing IT organizations deploy 30x more frequently with 200x shorter lead times; they have 60x fewer failures and recover 168x faster. https://puppetlabs.com/sites/default/files/2015-state-of-devops-report.pdf
4. DO WANT!!! 4
5. 5 frequent, reliable releases of high quality valuable software
6. ??? 6
7. SNAP, BACK IN 2012… Built on 9 components (microservices, anyone?) A message-queue for fast background job processing Chef-server for infrastructure provisioning Relational & NoSQL databases Extensive Nagios monitoring Completely automated infrastructure creation Build and deployment pipelines for each component So clearly - we had it nailed. Right? 7
8. Well, actually… 8
9. CONSEQUENCES Likelihood of build setup passing 50% Likelihood of changes actually triggering a build 50% Likelihood that someone knew why 0% Likelihood of 4 SSH sessions firing up debug 100% Likelihood that the infrastructure was busted 50% Frequency of production upgrades Twice a month Consequence of upgrades Site unavailable! Feedback to users about what was going on Minimal 9
10. 10 frequent, reliable releases of high quality valuable software
11. FAIL! 11
12. Our journey so far..
13. Queues per build Messages lost, redelivery problems Single, centralized queue No need to manage queues Delayed::Job
14. one Nagios to rule them all too many components too many metrics and alerts ignored alerts when things actually did go wrong separated application monitoring basic server monitoring aggregated logs we don’t need to manage anything +
15. Server became cognizant of what state would go into chef reliable and faster chef runs nothing to manage user build specific configuration data dependency on the chef server for deployments had to manage the server Solo
16. Single mutable environments Too many components READONLY MODE Long running deployments Loosing build requests Loosing customers Fewer components Blue-green deployments
17. COUPLED MODULES TO MONOLITH TO REAL MODULES Highly coupled initial state Collapsed moving parts to a monolith Deploying frequently highlights rates of change This was used to make new modules Much more decoupled state today 17
18. BLUE-GREEN DEPLOYMENTS IN SNAP 18 VZHOST Build Server web server VZHOST Build Server web server L B DATABASE
19. SO, WHAT’S THE CURRENT STATE? Build setups almost always pass… …but if there are failures, the team gets alerted Deployments happen every other day New releases happen once every 2~3 weeks Near zero downtime 19
20. YAGNI 20
21. How do we work now? 21
22. SEQUENCING OF STORIES 22 short (idealized) cycle time, more technical risk time time longer (idealized) cycle time, less technical risk
23. FEATURE TOGGLES Hide unfinished UI elements Control backend behaviour Test with feature toggles Avoid multi-component feature toggles 23 http://martinfowler.com/bliki/FeatureToggle.html
24. FRONTEND 24 <% if feature_enabled?(:parallel_stage) %>
 <!-- show new html -->
 <% else %>
 <!-- show old html -->
 <% end %>
25. BACKEND 25 if feature_enabled?(:parallel_stage)
 # new logic
 else
 # old logic
 end
26. TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do
 describe "feature enabled" do
 before(:each) do
 with_feature_enabled(:parallel_stage)
 end
 
 it 'should not show the job tabs when there is only one job' do
 end
 end
 
 describe "feature disabled" do
 before(:each) do
 with_feature_disabled(:parallel_stage)
 end
 
 it 'should not show the job tabs but should show the logs' do
 end
 end
 end
27. MIGRATION OF DATA Consider existing schema as well as data Incremental changes Rollbacks Zero downtime releases 27
28. INTRODUCE JOBS 28 jobs id
29. POPULATE ALL EXISTING STAGES 29 stages id started_at completed_at result … jobs id stage_id …
30. COPY ATTRIBUTES TO JOB 30 stages … started_at completed_at result … jobs … started_at completed_at result …
31. SWITCH THE APPLICATION TO START USING THE JOB MODEL 31 # After switch
 class Stage
 def result
 results = jobs.collect { |job| job.result }
 return :failed if results.any?(:failed)
 :passed
 end
 end # Before switch
 class Stage
 attr_reader :result
 end # in transition
 class Stage
 def result
 if feature_enabled?(:parallel_stage)
 results = jobs.collect { |job| job.result }
 return :failed if results.any?(:failed)
 :passed
 else
 result
 end
 end
 end
32. REMOVE UNUSED ATTRIBUTES 32 stages id started_at completed_at result … jobs … started_at completed_at result …
33. IDENTICAL ENVIRONMENTS/AUTOMATION 33 staging build n+1 upgrade smoke tests
    ver. n To production
34. 34 build n+1 upgrade smoke tests
    ver. n+1 To production staging
35. 35 build n+1 upgrade smoke tests
    ver. n+1 staging To production
36. 36
    ver. n+1 staging build n+1 upgrade smoke tests build n+2 upgrade smoke tests
    To production
37. 37
    ver. n+2 staging build n+1 upgrade smoke tests build n+2 upgrade smoke tests
    To production
38. 38
    ver. n+2 staging build n+1 upgrade smoke tests build n+2 upgrade smoke tests
    To production
39. SUMMARY Start with getting working software into hands of users Layer on qualities of reliability, automation, frequent releases etc Kick the can down the road Don’t be afraid of “re-design”/”re-work” Let CD guide your architecture and design - not the other way around 39
40. Akshay Karle, @akshay_karle THANK YOU Fernando Junior, @nandopaf @snap_ci | https://snap-ci.com