[2018.04 Meetup][TALK] Dave Farley - Optimising Continuous Delivery

Talk: Optimising Continuous Delivery
Speaker: Dave Farley (@davefarley77)

The deployment pipeline is a central idea in Continuous Delivery. It represents the channel through which ideas inside your business flow to the outside world so that they can be delivered to your users in the form of working software. First steps in creating a working deployment pipeline are very rewarding. They help to improve the quality of the software that you create and usually lead to happier users.

As teams mature in their adoption of Continuous Delivery though, the pipeline often begins to slow. We have more tests, we are testing more complex things, our software becomes more complex and more difficult to deploy.

How do we address these growing pains? What are the next steps, beyond the CD beginner’s basic automation? How do we optimise our pipelines to meet our needs as these needs grow?

This talk will address the growing pains in Continuous Delivery and describe some strategies to cope with them.

Dave Farley is the co-author of the Jolt-award winning book ‘Continuous Delivery’ a regular conference speaker and blogger and a contributor to the Reactive Manifesto. Dave is an independent software developer and consultant, and founder and director of Continuous Delivery Ltd.

Transcript

1. Dave Farley http://www.davefarley.net @davefarley77 http://www.continuous-delivery.co.uk Optimising Continuous Delivery Faster is Better
2. What Do We Really Want?
3. What Do We Really Want? Idea
4. What Do We Really Want? Idea Customer
5. What Do We Really Want? Feedback Idea Quickly Cheaply Reliably Customer
6. What Really Works? Smart Automation - a repeatable, reliable process for releasing software Unit Test Code Idea Executable spec. Build Release
7. What Really Works? Smart Automation - a repeatable, reliable process for releasing software Unit Test Code Idea Executable spec. Build Release
8. What Really Works? Smart Automation - a repeatable, reliable process for releasing software Unit Test Code Idea Executable spec. Build Release “It doesn’t matter how intelligent you are, if you guess and that guess cannot be backed up by experimental evidence – then it is still a guess!” - Richard Feynman
9. Cycle Time 103 days Typical Traditional Cycle Time 10 days 64 days
10. Cycle Time Commit Stage Compile Unit test Analysis Build Installers Automated acceptance testing Automated performance testing Manual testing Release 57 mins 3 mins 20 mins 20 mins 30 mins 4 mins Typical CD Cycle Time 103 days Typical Traditional Cycle Time 10 days 64 days
11. The Principles of Continuous Delivery
12. The Principles of Continuous Delivery Create a repeatable, reliable process for releasing software.
13. The Principles of Continuous Delivery Create a repeatable, reliable process for releasing software. Automate almost everything.
14. The Principles of Continuous Delivery Create a repeatable, reliable process for releasing software. Automate almost everything. Keep everything under version control.
15. The Principles of Continuous Delivery Create a repeatable, reliable process for releasing software. Automate almost everything. Keep everything under version control. If it hurts, do it more often – bring the pain forward.
16. The Principles of Continuous Delivery Create a repeatable, reliable process for releasing software. Automate almost everything. Keep everything under version control. If it hurts, do it more often – bring the pain forward. Build quality in.
17. The Principles of Continuous Delivery Create a repeatable, reliable process for releasing software. Automate almost everything. Keep everything under version control. If it hurts, do it more often – bring the pain forward. Build quality in. Done means released.
18. The Principles of Continuous Delivery Create a repeatable, reliable process for releasing software. Automate almost everything. Keep everything under version control. If it hurts, do it more often – bring the pain forward. Build quality in. Done means released. Everybody is responsible for the release process.
19. The Principles of Continuous Delivery Create a repeatable, reliable process for releasing software. Automate almost everything. Keep everything under version control. If it hurts, do it more often – bring the pain forward. Build quality in. Done means released. Everybody is responsible for the release process. Improve continuously.
20. Basic Continuous Delivery Process Local Dev. Env. Source Repository
21. Basic Continuous Delivery Process Local Dev. Env. Source Repository
22. Basic Continuous Delivery Process Local Dev. Env. Commit Source Repository
23. Basic Continuous Delivery Process Local Dev. Env. Commit Source Repository
24. Basic Continuous Delivery Process Artifact Repository Local Dev. Env. Commit Source Repository
25. Basic Continuous Delivery Process Artifact Repository Local Dev. Env. Commit Commit Source Repository
26. Basic Continuous Delivery Process Artifact Repository Local Dev. Env. Acceptance Commit Commit Source Repository
27. Basic Continuous Delivery Process Artifact Repository Local Dev. Env. Acceptance Commit Commit Source Repository
28. Basic Continuous Delivery Process Artifact Repository Local Dev. Env. Acceptance Commit Commit Acceptance Source Repository
29. Basic Continuous Delivery Process Artifact Repository Local Dev. Env. Acceptance Commit Production Env. Deployment App. Commit Acceptance Source Repository
30. Basic Continuous Delivery Process Artifact Repository Local Dev. Env. Acceptance Commit Production Env. Deployment App. Commit Acceptance Source Repository
31. Basic Continuous Delivery Process Artifact Repository Local Dev. Env. Acceptance Commit Production Env. Deployment App. Commit Acceptance Production Source Repository
32. Basic Continuous Delivery Process Artifact Repository Local Dev. Env. Deployment Pipeline Acceptance Commit Production Env. Deployment App. Commit Acceptance Production Source Repository
33. Example Continuous Delivery Process Artifact Repository Local Dev. Env. Acceptance Commit Commit Acceptance Source Repository
34. Example Continuous Delivery Process Artifact Repository Local Dev. Env. Acceptance Commit Commit Acceptance Manual Source Repository Manual Test Env. Deployment App.
35. Example Continuous Delivery Process Artifact Repository Local Dev. Env. Acceptance Commit Component Performance Commit Acceptance Manual Perf1 Source Repository Manual Test Env. Deployment App.
36. Example Continuous Delivery Process Artifact Repository Local Dev. Env. Acceptance Commit Component Performance System Performance Commit Acceptance Manual Perf1 Perf2 Source Repository Manual Test Env. Deployment App.
37. Example Continuous Delivery Process Artifact Repository Local Dev. Env. Acceptance Commit Component Performance System Performance Commit Acceptance Manual Perf1 Perf2 Staged Source Repository Manual Test Env. Deployment App. Data Migration
38. Example Continuous Delivery Process Artifact Repository Local Dev. Env. Deployment Pipeline Acceptance Commit Component Performance System Performance Production Env. Deployment App. Commit Acceptance Manual Perf1 Perf2 Staged Production Source Repository Manual Test Env. Deployment App. Data Migration
39. Scope of a Deployment Pipeline
40. Scope of a Deployment Pipeline
41. Scope of a Deployment Pipeline ? ? ? ? ?
42. Scope of a Deployment Pipeline
43. Scope of a Deployment Pipeline
44. Scope of a Deployment Pipeline An Independently Deployable Unit!
45. Scope of a Deployment Pipeline
46. Scope of a Deployment Pipeline
47. Scope of a Deployment Pipeline
48. Scope of a Deployment Pipeline
49. Scope of a Deployment Pipeline
50. Scope of a Deployment Pipeline Independently Deployable Units!
51. Where To Test - Branching
52. Idea Production Where To Test - Branching
53. Idea Production Where To Test - Branching
54. Idea Production Where To Test - Branching
55. Idea Production Where To Test - Branching
56. Idea Production ? Where To Test - Branching
57. Idea Production ? Where To Test - Branching
58. Idea Production ? Where To Test - Branching
59. Idea Production ? ? ? Where To Test - Branching
60. Idea Production Where To Test - Branching
61. Idea Production Where To Test - Branching
62. Idea Production Where To Test - Branching
63. 1 2 3 4 5 6 7
64. Idea Production 1 2 3 4 5 6 7
65. Idea Production 1 2 3 4 5 6 7
66. Idea Production 1 2 3 4 5 6 7
67. Idea Production 1 2 3 4 5 6 7
68. Idea Production 1 2 3 4 5 6 7
69. (C)opyright Dave Farley 2015 Branching Strategies
70. (C)opyright Dave Farley 2015 Branching Strategies • Don’t Branch!
71. (C)opyright Dave Farley 2015 Branching Strategies • Don’t Branch! • Don’t Branch!
72. (C)opyright Dave Farley 2015 Branching Strategies • Don’t Branch! • Don’t Branch! •Don’t Branch!
73. The Pipeline is a Strategic Resource • It is the only channel to production • All change in production flows through it • So if it breaks or goes slow, the impact is enormous
74. Pipeline Problems • Slow Commit Stage • Slow Acceptance Stage • Failing Tests • Intermittent Tests • Pipeline Changes
75. Pipeline Problems • Slow Commit Stage • Slow Acceptance Stage • Failing Tests • Intermittent Tests • Pipeline Changes
76. (C)opyright Dave Farley 2015 Local Dev. Env. Slow Commit Stage Artifact Repository Deployment Pipeline Acceptance Component Performance System Performance Production Env. Deployment App. Commit Acceptance Manual Perf1 Perf2 Staged Production Manual Test Env. Deployment App. Data Migration Source Repository Commit
77. (C)opyright Dave Farley 2015 Local Dev. Env. Slow Commit Stage Artifact Repository Deployment Pipeline Acceptance Component Performance System Performance Production Env. Deployment App. Commit Acceptance Manual Perf1 Perf2 Staged Production Manual Test Env. Deployment App. Data Migration Source Repository Commit
78. (C)opyright Dave Farley 2015 Local Dev. Env. Slow Commit Stage Artifact Repository Deployment Pipeline Acceptance Component Performance System Performance Production Env. Deployment App. Commit Acceptance Manual Perf1 Perf2 Staged Production Manual Test Env. Deployment App. Data Migration Source Repository Commit Commit Stage Compile Unit Test Analysis Build Installers
79. (C)opyright Dave Farley 2015 Local Dev. Env. Slow Commit Stage Artifact Repository Deployment Pipeline Acceptance Component Performance System Performance Production Env. Deployment App. Commit Acceptance Manual Perf1 Perf2 Staged Production Manual Test Env. Deployment App. Data Migration Source Repository Commit Commit Stage Compile Unit Test Analysis Build Installers Ideal Feedback Cycle for this stage: < 5min
80. (C)opyright Dave Farley 2015 Local Dev. Env. Slow Commit Stage Artifact Repository Deployment Pipeline Acceptance Component Performance System Performance Production Env. Deployment App. Commit Acceptance Manual Perf1 Perf2 Staged Production Manual Test Env. Deployment App. Data Migration Source Repository Commit
81. (C)opyright Dave Farley 2015 Local Dev. Env. Slow Commit Stage Artifact Repository Deployment Pipeline Acceptance Component Performance System Performance Production Env. Deployment App. Commit Acceptance Manual Perf1 Perf2 Staged Production Manual Test Env. Deployment App. Data Migration Source Repository Commit This is the most valuable Feedback-Cycle in the Pipeline!
82. (C)opyright Dave Farley 2015 Local Dev. Env. Slow Commit Stage Artifact Repository Deployment Pipeline Acceptance Component Performance System Performance Production Env. Deployment App. Commit Acceptance Manual Perf1 Perf2 Staged Production Manual Test Env. Deployment App. Data Migration Source Repository Commit This is the most valuable Feedback-Cycle in the Pipeline! Invest Time, Effort & Money
83. Slow Commit Stage - Compilation
84. Slow Commit Stage - Compilation MAKE YOUR BUILDS EFFICIENT!!! 
 Try a re-write.
85. Slow Commit Stage - Compilation
86. Slow Commit Stage - Compilation Buy Faster Hardware
87. Slow Commit Stage - Compilation
88. Slow Commit Stage - Compilation Parallelise the build
89. Slow Commit Stage - Compilation
90. Slow Commit Stage - Compilation Decompose your system into independent modules or services
 Each with it’s own pipeline
91. Slow Commit Stage - Compilation Decompose your system into independent modules or services
 Each with it’s own pipeline
92. Slow Commit Stage - Compilation Decompose your system into independent modules or services
 Each with it’s own pipeline WARNING!!! This is not as easy as it first seems Dependency Management is one of the really problems in Computer Science! HARD Microservices is no “Silver Bullet” Remember: “Independently Deployable Units”
93. Slow Commit Stage - Compilation Decompose your system into independent modules or services
 Each with it’s own pipeline
94. Slow Commit Stage - Compilation
95. Slow Commit Stage - Compilation Look to a modern build system, Gradle, Bazel, Buck, …
96. Slow Commit Stage - Compilation
97. Slow Commit Stage - Compilation Try running the build from RAM Disk
 Good for I/O bound compilation cycles (e.g. C, C++)
98. Slow Commit Stage - Testing
99. Slow Commit Stage - Testing Look to your tests!
100. Slow Commit Stage - Testing
101. Slow Commit Stage - Testing Commit tests should be mostly pure unit tests Don’t Start the app! Don’t Write to Disk! Don’t Write to the DB! Don’t Use the Network! Fake External Dependencies!
102. Slow Commit Stage - Testing
103. Slow Commit Stage - Testing Accept slow tests grudgingly
104. Slow Commit Stage - Testing
105. Slow Commit Stage - Testing Always think about efficiency when writing tests • Use Chicken Counting (0, 1, Many) • Use Mocks & Stubs
106. Slow Commit Stage - Testing
107. Slow Commit Stage - Testing Periodically: Sort your tests by duration and speed-up the slowest 10%
108. Slow Commit Stage - Testing
109. Slow Commit Stage - Testing Run Tests in Parallel • If this is difficult, your test isolation is poor - fix it!
110. Slow Commit Stage - Testing
111. Slow Commit Stage - Testing Last Resort: 
 Move slow, problematic but high-value tests to a later stage in the pipeline
112. Slow Commit Stage - Analysis
113. Slow Commit Stage - Analysis Static Analysis can give valuable feedback
114. Slow Commit Stage - Analysis
115. Slow Commit Stage - Analysis It needs to pay for itself though. If the analysis doesn’t catch bugs, drop it!
116. Slow Commit Stage - Analysis
117. Slow Commit Stage - Analysis Last Resort: 
 Move analysis out of commit stage
118. Slow Commit Stage - Analysis
119. Slow Commit Stage - Analysis Last Last Resort: 
 Give Up! Move analysis out of pipeline
120. Slow Commit Stage - Installers
121. Slow Commit Stage - Installers We want to test Release Candidates that may make it to production
122. Slow Commit Stage - Installers
123. Slow Commit Stage - Installers The output of a successful commit stage is production deployable software
124. Slow Commit Stage - Installers
125. Slow Commit Stage - Installers For some tech. This is difficult to achieve
126. Slow Commit Stage - Installers
127. Slow Commit Stage - Installers If creating installers is slow, move it out of Commit Stage
 This is fine because creating installers is not usually a source of errors
128. (C)opyright Dave Farley 2016 Human CI
129. (C)opyright Dave Farley 2016 Human CI • Always run Commit Tests locally before committing
130. (C)opyright Dave Farley 2016 Human CI • Always run Commit Tests locally before committing • Wait for result of Commit Build
131. (C)opyright Dave Farley 2016 Human CI • Always run Commit Tests locally before committing • Wait for result of Commit Build • Don’t Go Home, To Lunch, To a Meeting, To Make Coffee…
132. (C)opyright Dave Farley 2016 Human CI • Always run Commit Tests locally before committing • Wait for result of Commit Build • Don’t Go Home, To Lunch, To a Meeting, To Make Coffee… • Fix or Revert Failures within 10 minutes
133. (C)opyright Dave Farley 2016 Human CI • Always run Commit Tests locally before committing • Wait for result of Commit Build • Don’t Go Home, To Lunch, To a Meeting, To Make Coffee… • Fix or Revert Failures within 10 minutes • If a Team Mate as “Gone Home”, “Gone to Lunch” etc. Revert their change
134. (C)opyright Dave Farley 2016 Human CI • Always run Commit Tests locally before committing • Wait for result of Commit Build • Don’t Go Home, To Lunch, To a Meeting, To Make Coffee… • Fix or Revert Failures within 10 minutes • If a Team Mate as “Gone Home”, “Gone to Lunch” etc. Revert their change • If someone else notices that you broke the build before you do, it is a sin!
135. (C)opyright Dave Farley 2016 Human CI • Always run Commit Tests locally before committing • Wait for result of Commit Build • Don’t Go Home, To Lunch, To a Meeting, To Make Coffee… • Fix or Revert Failures within 10 minutes • If a Team Mate as “Gone Home”, “Gone to Lunch” etc. Revert their change • If someone else notices that you broke the build before you do, it is a sin! • Once the Commit Stage passes, move on to your next task
136. (C)opyright Dave Farley 2016 Human CI • Always run Commit Tests locally before committing • Wait for result of Commit Build • Don’t Go Home, To Lunch, To a Meeting, To Make Coffee… • Fix or Revert Failures within 10 minutes • If a Team Mate as “Gone Home”, “Gone to Lunch” etc. Revert their change • If someone else notices that you broke the build before you do, it is a sin! • Once the Commit Stage passes, move on to your next task • Monitor the progress of your change through the pipeline.
137. (C)opyright Dave Farley 2016 Human CI • Always run Commit Tests locally before committing • Wait for result of Commit Build • Don’t Go Home, To Lunch, To a Meeting, To Make Coffee… • Fix or Revert Failures within 10 minutes • If a Team Mate as “Gone Home”, “Gone to Lunch” etc. Revert their change • If someone else notices that you broke the build before you do, it is a sin! • Once the Commit Stage passes, move on to your next task • Monitor the progress of your change through the pipeline. • If a later stage fails, address the failure immediately
138. (C)opyright Dave Farley 2016 Human CI • Always run Commit Tests locally before committing • Wait for result of Commit Build • Don’t Go Home, To Lunch, To a Meeting, To Make Coffee… • Fix or Revert Failures within 10 minutes • If a Team Mate as “Gone Home”, “Gone to Lunch” etc. Revert their change • If someone else notices that you broke the build before you do, it is a sin! • Once the Commit Stage passes, move on to your next task • Monitor the progress of your change through the pipeline. • If a later stage fails, address the failure immediately CD is about optimising your process to keep the software in a releasable state That means pipeline failures are top priority!
139. (C)opyright Dave Farley 2015 Commit Strategies Artifact Repository Commit Source Repository Commit Local Dev. Env.
140. (C)opyright Dave Farley 2015 Commit Strategies Artifact Repository Commit Source Repository Commit Local Dev. Env. Human CI
141. (C)opyright Dave Farley 2015 Commit Strategies Artifact Repository Commit Source Repository Commit Local Dev. Env. Artifact Repository Source Repository Commit Local Dev. Env. Commit Pass? Human CI
142. (C)opyright Dave Farley 2015 Commit Strategies Artifact Repository Commit Source Repository Commit Local Dev. Env. Artifact Repository Source Repository Commit Local Dev. Env. Commit Pass? Human CI Gated/Pre-Commit CI
143. Slow Commit Stage - Summary • Performance of the Commit Stage Matters • Teams with a fast Commit Stage keep the build green • Commit Stage performance as an important thing. Invest in it!
144. Pipeline Problems • Slow Commit Stage • Slow Acceptance Stage • Failing Tests • Intermittent Tests • Pipeline Changes
145. Pipeline Problems • Slow Commit Stage • Slow Acceptance Stage • Failing Tests • Intermittent Tests • Pipeline Changes
146. (C)opyright Dave Farley 2015 Local Dev. Env. Artifact Repository Deployment Pipeline Component Performance System Performance Production Env. Deployment App. Commit Acceptance Manual Perf1 Perf2 Staged Production Manual Test Env. Deployment App. Data Migration Source Repository Commit Acceptance Slow Acceptance Stage
147. (C)opyright Dave Farley 2015 Local Dev. Env. Artifact Repository Deployment Pipeline Component Performance System Performance Production Env. Deployment App. Commit Acceptance Manual Perf1 Perf2 Staged Production Manual Test Env. Deployment App. Data Migration Source Repository Commit Acceptance Slow Acceptance Stage
148. (C)opyright Dave Farley 2015 Local Dev. Env. Artifact Repository Deployment Pipeline Component Performance System Performance Production Env. Deployment App. Commit Acceptance Manual Perf1 Perf2 Staged Production Manual Test Env. Deployment App. Data Migration Source Repository Commit Acceptance Acceptance Stage Configure Environment Deploy Release Candidate Smoke Test/Health Check Acceptance Test Slow Acceptance Stage
149. (C)opyright Dave Farley 2015 Local Dev. Env. Artifact Repository Deployment Pipeline Component Performance System Performance Production Env. Deployment App. Commit Acceptance Manual Perf1 Perf2 Staged Production Manual Test Env. Deployment App. Data Migration Source Repository Commit Acceptance Acceptance Stage Configure Environment Deploy Release Candidate Smoke Test/Health Check Acceptance Test Slow Acceptance Stage Ideal Feedback Cycle for this stage: < 45min
150. Slow Acceptance Stage - Configure Env.
151. Slow Acceptance Stage - Configure Env. Consider rates of change of different layers
 (e.g. OS changes less often than the app)
152. Slow Acceptance Stage - Configure Env.
153. Slow Acceptance Stage - Configure Env. Layer your configuration appropriately
154. Slow Acceptance Stage - Configure Env.
155. Slow Acceptance Stage - Configure Env. Consider versioning pre-baked “Image-Templates”
156. Slow Acceptance Stage - Configure Env.
157. Slow Acceptance Stage - Configure Env. • Consider the use of Containers, like Docker
158. Slow Acceptance Stage - Deploy
159. Slow Acceptance Stage - Deploy • Successful Deployment ends with a running, working system
160. Slow Acceptance Stage - Deploy • Successful Deployment ends with a running, working system • So for efficiency we need to consider:
161. Slow Acceptance Stage - Deploy • Successful Deployment ends with a running, working system • So for efficiency we need to consider: • Time to Deploy
162. Slow Acceptance Stage - Deploy • Successful Deployment ends with a running, working system • So for efficiency we need to consider: • Time to Deploy • Time to Migrate Data
163. Slow Acceptance Stage - Deploy • Successful Deployment ends with a running, working system • So for efficiency we need to consider: • Time to Deploy • Time to Migrate Data • Time to Start the System
164. Slow Acceptance Stage - Deploy • Successful Deployment ends with a running, working system • So for efficiency we need to consider: • Time to Deploy • Time to Migrate Data • Time to Start the System • Time to Validate the System
165. Slow Acceptance Stage - Deploy
166. Slow Acceptance Stage - Deploy Time To Deploy: • Keep deployable artefacts lean
167. Slow Acceptance Stage - Deploy
168. Slow Acceptance Stage - Deploy Time To Deploy: • Keep deployable artefacts lean • Packaging strategy can make a big difference
169. Slow Acceptance Stage - Deploy
170. Slow Acceptance Stage - Deploy Time To Deploy: • Keep deployable artefacts lean • Packaging strategy can make a big difference • Think about network/infrastructure constraints at deployment time
171. Slow Acceptance Stage - Deploy
172. Slow Acceptance Stage - Deploy Time To Deploy: • Keep deployable artefacts lean • Packaging strategy can make a big difference • Think about network/infrastructure constraints at deployment time • Minimise data-sets, where you can
173. Slow Acceptance Stage - Deploy
174. Slow Acceptance Stage - Deploy Time To Deploy: • Keep deployable artefacts lean • Packaging strategy can make a big difference • Think about network/infrastructure constraints at deployment time • Minimise data-sets, where you can • Consider better modularisation - Independently deployable services
175. Slow Acceptance Stage - Deploy
176. Slow Acceptance Stage - Deploy Time To Start the System: • Minimise Start-up dependencies & ordering
177. Slow Acceptance Stage - Deploy
178. Slow Acceptance Stage - Deploy Time To Start the System: • Minimise Start-up dependencies & ordering • Invest in optimising the code executed during startup - Make it efficient!
179. Slow Acceptance Stage - Deploy
180. Slow Acceptance Stage - Deploy Time To Start the System: • Minimise Start-up dependencies & ordering • Invest in optimising the code executed during startup - Make it efficient! • Consider better modularisation - Independently deployable services
181. Slow Acceptance Stage - Testing
182. Slow Acceptance Stage - Testing • There is a common cycle: • Teams start writing acceptance tests and run them all sequentially
183. Slow Acceptance Stage - Testing • There is a common cycle: • Teams start writing acceptance tests and run them all sequentially • As number of Acceptance Tests increases the time they take to deliver feedback lengthens
184. Slow Acceptance Stage - Testing • There is a common cycle: • Teams start writing acceptance tests and run them all sequentially • As number of Acceptance Tests increases the time they take to deliver feedback lengthens • Teams parallelise Acceptance Tests tactically
 Typically teams start by running half the tests on one machine and half on another.
185. Slow Acceptance Stage - Testing • There is a common cycle: • Teams start writing acceptance tests and run them all sequentially • As number of Acceptance Tests increases the time they take to deliver feedback lengthens • Teams parallelise Acceptance Tests tactically
 Typically teams start by running half the tests on one machine and half on another. • Teams move to more strategic (automated) parallelisation strategy
186. Slow Acceptance Stage - Testing • There is a common cycle: • Teams start writing acceptance tests and run them all sequentially • As number of Acceptance Tests increases the time they take to deliver feedback lengthens • Teams parallelise Acceptance Tests tactically
 Typically teams start by running half the tests on one machine and half on another. • Teams move to more strategic (automated) parallelisation strategy Treat Acceptance Test Isolation Seriously From Day One!
187. Slow Acceptance Stage - Testing • Test Avoidance is a great idea, but difficult to do well • Acceptance Tests find things that you don’t expect • Be careful that you don’t spend too much time calculating which tests to avoid
188. Scaling-Up Artifact Repository Deployment Pipeline Acceptance Commit Component Performance System Performance Staging Env. Deployment App. Production Env. Deployment App. Source Repository Manual Test Env. Deployment App.
189. Scaling-Up Artifact Repository Deployment Pipeline Acceptance Commit Component Performance System Performance Staging Env. Deployment App. Production Env. Deployment App. Source Repository Manual Test Env. Deployment App. Deployment Pipeline Commit Manual Test Env. Deployment App. Artifact Repository Acceptance Acceptance Test Environment
190. Scaling-Up Artifact Repository Deployment Pipeline Acceptance Commit Component Performance System Performance Staging Env. Deployment App. Production Env. Deployment App. Source Repository Manual Test Env. Deployment App. Deployment Pipeline Commit Manual Test Env. Deployment App. Artifact Repository Acceptance Acceptance Test Environment A A
191. Scaling-Up Artifact Repository Deployment Pipeline Acceptance Commit Component Performance System Performance Staging Env. Deployment App. Production Env. Deployment App. Source Repository Manual Test Env. Deployment App. Deployment Pipeline Commit Manual Test Env. Deployment App. Artifact Repository Acceptance Acceptance Test Environment A A
192. Scaling-Up Artifact Repository Deployment Pipeline Acceptance Commit Component Performance System Performance Staging Env. Deployment App. Production Env. Deployment App. Source Repository Manual Test Env. Deployment App. Deployment Pipeline Commit Manual Test Env. Deployment App. Artifact Repository Acceptance Acceptance Test Environment Test Host Test Host Test Host Test Host Test Host A A
193. Scaling-Up Artifact Repository Deployment Pipeline Acceptance Commit Component Performance System Performance Staging Env. Deployment App. Production Env. Deployment App. Source Repository Manual Test Env. Deployment App. Deployment Pipeline Commit Manual Test Env. Deployment App. Artifact Repository Acceptance Acceptance Acceptance Test Environment Test Host Test Host Test Host Test Host Test Host A A
194. (C)opyright Dave Farley 2016 Dynamic Acceptance Test Allocation
195. (C)opyright Dave Farley 2016 Dynamic Acceptance Test Allocation
196. Pipeline Problems • Slow Commit Stage • Slow Acceptance Stage • Failing Tests • Intermittent Tests • Pipeline Changes
197. Pipeline Problems • Slow Commit Stage • Slow Acceptance Stage • Failing Tests • Intermittent Tests • Pipeline Changes
198. Failing Tests • The Efficiency of the Feedback Cycle is central to the ability of a team to keep tests green!
199. Artifact Repository Local Dev. Env. Deployment Pipeline Acceptance Commit Component Performance System Performance Production Env. Deployment Commit Acceptance Manual Perf1 Perf2 Staged Production Source Repository Manual Test Deployment Data Migration
200. Artifact Repository Local Dev. Env. Deployment Pipeline Acceptance Commit Component Performance System Performance Production Env. Deployment Commit Acceptance Manual Perf1 Perf2 Staged Production Source Repository Manual Test Deployment Data Migration < 5 Minutes
201. Artifact Repository Local Dev. Env. Deployment Pipeline Acceptance Commit Component Performance System Performance Production Env. Deployment Commit Acceptance Manual Perf1 Perf2 Staged Production Source Repository Manual Test Deployment Data Migration < 5 Minutes < 60 Minutes
202. Failing Tests • Continuous Delivery is about: 
 Keeping the build in a releasable state
203. Failing Tests • Continuous Delivery is about: 
 Keeping the build in a releasable state Prioritise Fixing Failing Tests!
204. Pipeline Problems • Slow Commit Stage • Slow Acceptance Stage • Failing Tests • Intermittent Tests • Pipeline Changes
205. Pipeline Problems • Slow Commit Stage • Slow Acceptance Stage • Failing Tests • Intermittent Tests • Pipeline Changes
206. Intermittent Tests
207. Intermittent Tests Intermittency is a pain!
208. Intermittent Tests
209. Intermittent Tests Too many teams live with intermittent tests
210. Intermittent Tests
211. Intermittent Tests Some CI systems even support running failing tests several times!! Why? • If the test passes once and fails once how do you know which result is correct?
212. Intermittent Tests
213. Intermittent Tests Don’t live with intermittent tests - solve the problem
214. Intermittent Tests
215. Intermittent Tests Treat intermittent tests as failing tests
216. Intermittent Tests
217. Intermittent Tests Last Resort: If you can’t fix it, is it really worth keeping?
218. Intermittent Tests Common Causes: • Race Conditions • Poor Test Isolation • Poorly Designed Test Cases • Sometimes it is something serious!
219. Intermittent Tests Common Causes: • Race Conditions • Poor Test Isolation • Poorly Designed Test Cases • Sometimes it is something serious! Prioritise Fixing Intermittent Tests!
220. Pipeline Problems • Slow Commit Stage • Slow Acceptance Stage • Failing Tests • Intermittent Tests • Pipeline Changes
221. Pipeline Problems • Slow Commit Stage • Slow Acceptance Stage • Failing Tests • Intermittent Tests • Pipeline Changes
222. Pipeline Changes • The Deployment Pipeline is a Strategic Resource • Consider SLAs for your pipeline • Redundancy, Clustering, HLA?
223. Pipeline Changes
224. Pipeline Changes The Pipeline is a complex system in its own right
225. Pipeline Changes
226. Pipeline Changes Consider versioning it and your tools
227. Pipeline Changes
228. Pipeline Changes Consider writing test cases for some pipeline behaviours
229. Pipeline Changes
230. Pipeline Changes Consider Blue/Green deployment strategies for pipeline changes
231. Pipeline Changes
232. Pipeline Changes Use an “Infrastructure as code” approach for all pipeline hosts
233. Make Your Pipeline the ONLY Route to Production
234. Make Your Pipeline the ONLY Route to Production Artefact Repository Local Dev. Env. Deployment Pipeline Acceptance Commit Component Performance System Performance Production Env. Deployment App. Commit Acceptance Manual Perf1 Perf2 Staged Production Source Repository Manual Test Env. Deployment App. Data Migration
235. Continuous Delivery As Competitive Advantage
236. Continuous Delivery As Competitive Advantage Continuous Delivery Experiment Experiment Experiment Experiment Experiment Experiment
237. Q&A http://www.continuous-delivery.co.uk Dave Farley http://www.davefarley.net @davefarley77