Journey to painless releases: Continuous delivery for Philips Hue Android

Video: https://www.youtube.com/watch?v=VkFcJZWtufA

Wind back one year ago:
- releasing took us 10 weeks
- our app was plagued by regression
- developers felt completely detached from our users

Clearly our traditional strategy of heavily testing each release (up-front quality) wasn't working. So we decided to flip things around: why not try to reduce the cost of errors instead? Hence we decided to move our Philips Hue Android app to continuous deployment.

In this talk you will learn that if releasing is painful, you should actually release even more often. Step by step I'll explain how we moved the Philips Hue Android app to continuous deployment, what our current release process/CI looks like, what changes we made to our overall testing approach, how we empowered developers and testers, and how we made our release process completely painless.

Transcript

1. @MOLSJEROEN JOURNEY TO PAINLESS RELEASES CONTINUOUS DELIVERY FOR PHILIPS HUE
2. @MOLSJEROEN @MOLSJEROEN
3. @MOLSJEROEN CHALLENGES CONTINUOUS DELIVERY MIGRATE YOUR PROJECT RESULTS
4. CHALLENGES
5. @MOLSJEROEN RELEASE PROCESS DEVELOPMENT
6. @MOLSJEROEN RELEASE PROCESS DEVELOPMENT TESTING
7. @MOLSJEROEN RELEASE PROCESS DEVELOPMENT TESTING TESTING
8. @MOLSJEROEN RELEASE PROCESS DEVELOPMENT TESTING TESTING BETA
9. @MOLSJEROEN RELEASE PROCESS DEVELOPMENT TESTING TESTING BETA 10 weeks
10. @MOLSJEROEN RELEASE PROCESS DEVELOPMENT TESTING TESTING BETA DEVELOPMENT TESTING TESTING BETA 10 weeks
11. @MOLSJEROEN RELEASE PROCESS DEVELOPMENT TESTING TESTING BETA DEVELOPMENT TESTING TESTING BETA DEVELOPMENT TESTING TESTIN 10 weeks
12. @MOLSJEROEN RELEASE PROCESS DEVELOPMENT TESTING TESTING BETA DEVELOPMENT TESTING TESTING BETA DEVELOPMENT TESTING TESTIN 10 weeks 3 releases
13. @MOLSJEROEN RELEASE PROCESS DEVELOPMENT TESTING TESTING BETA 10 weeks
14. @MOLSJEROEN RELEASE PROCESS DEVELOPMENT TESTING TESTING BETA 10 weeks
15. @MOLSJEROEN RELEASE PROCESS DEVELOPMENT TESTING TESTING BETA 10 weeks
16. @MOLSJEROEN FRUSTRATIONS Issues were found extremely late Developers felt detached from our users Constant context switches between releases Releasing very painful Friction between developers and testers
17. @MOLSJEROEN BRANCHING MODEL develop master release feature bugfix feature hotfix
18. @MOLSJEROEN hotfix feature bugfix BRANCHING MODEL master release feature develop
19. @MOLSJEROEN BRANCHING MODEL develop master release feature bugfix feature hotfix
20. @MOLSJEROEN BRANCHING MODEL develop master release feature bugfix feature hotfix
21. @MOLSJEROEN BRANCHING MODEL develop master release feature bugfix feature hotfix
22. @MOLSJEROEN FRUSTRATIONS Unstable development branch Merging nightmares Poor code reviews Confusion about what fix is where Development was slow
23. @MOLSJEROEN RESULTS
24. STRATEGY OF UPFRONT TESTING NOT WORKING
25. @MOLSJEROEN GOALS Speed up development Get a grip on quality Feedback between developers and users
28. CONTINUOUS DELIVERY
29. IF IT HURTS, DO IT MORE OFTEN Jez Humble
30. @MOLSJEROEN WHAT DO WE WANT TO ACHIEVE? DEVELOPMENT TESTING TESTING BETA DEVELOPMENT TESTING TESTING BETA DEVELOPMENT TESTING TESTIN
31. @MOLSJEROEN WHAT DO WE WANT TO ACHIEVE? DEVELOPMENT TESTING TESTING BETA DEVELOPMENT TESTING TESTING BETA DEVELOPMENT TESTING TESTING BETA
32. @MOLSJEROEN WHAT DO WE WANT TO ACHIEVE? DEVELOPMENT TESTING BETA TESTING DEVELOPMENT TESTING BETA TESTING DEVELOPMENT TESTING BETA TESTING
33. @MOLSJEROEN WHAT DO WE WANT TO ACHIEVE? BETA DEVELOPMENT TEST TEST BETA DEVELOPMENT TEST TEST BETA DEVELOPMENT TEST TEST
34. @MOLSJEROEN WHAT DO WE WANT TO ACHIEVE? DEV & TEST BETA DEV & TEST BETA DEV & TEST BETA
35. IF IT HURTS, DO IT MORE OFTEN AND BRING THE PAIN FORWARD Jez Humble
36. @MOLSJEROEN TERMINOLOGY Continuous integration Continuous delivery Continuous deployment
37. DEVELOPMENT PROCESS
38. @MOLSJEROEN DEVELOPMENT PROCESS develop master release
39. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH develop master release feature
40. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST develop master release feature
41. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY develop master release feature
42. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY develop master release feature VERIFY
43. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY MERGE VERIFY develop master release feature
44. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY MERGE VERIFY develop master release feature 1-5 weeks
45. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY MERGE PROMOTE MASTER VERIFY develop master release feature 1-5 weeks
46. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY MERGE PROMOTE MASTER VERIFY develop master release feature STABILIZE 1-5 weeks
47. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY MERGE PROMOTE MASTER VERIFY develop master release feature STABILIZE 3 weeks 1-5 weeks
48. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY develop master release feature 3 weeks 1-5 weeks
49. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST QUALIFY MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY develop master release feature 3 weeks 1-5 weeks
50. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST QUALIFY MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY develop master release feature 3 weeks 1-5 weeks SHIP
51. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST QUALIFY MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY SHIP develop master release feature MERGE BACK 1-5 weeks 3 weeks
52. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST QUALIFY MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY SHIP develop master release feature MERGE BACK 1-5 weeks 3 weeks 3 weeks
53. @MOLSJEROEN MERGING/INTEGRATING FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY SHIP develop master release feature MERGE BACK 1-5 weeks 3 weeks 3 weeks QUALIFY
54. @MOLSJEROEN MERGING/INTEGRATING FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY SHIP develop master release feature MERGE BACK 1-5 weeks 3 weeks 3 weeks QUALIFY
55. SPEED UP INTEGRATION
56. @MOLSJEROEN
57. @MOLSJEROEN
58. @MOLSJEROEN Large pull request
59. @MOLSJEROEN Large pull request Hard to
 review
60. @MOLSJEROEN Large pull request Hard to
 review Hard to
 verify
61. @MOLSJEROEN Large pull request Hard to
 review Long lived
 pull request Hard to
 verify
62. @MOLSJEROEN Large pull request Hard to
 review Long lived
 pull request Merge
 conflicts Hard to
 verify
63. @MOLSJEROEN Large pull request Hard to
 review Long lived
 pull request Merge
 conflicts Overhead: 
 regression, 
 dependencies,… Hard to
 verify
64. @MOLSJEROEN MERGING/INTEGRATING FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY SHIP MERGE BACK 1-5 weeks 3 weeks 3 weeks QUALIFY FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY SHIP develop master release feature MERGE BACK 1-5 weeks 3 weeks 3 weeks QUALIFY
65. @MOLSJEROEN PULL REQUEST QUALIFICATION Develop feature BUILD
 TEST REVIEW
66. @MOLSJEROEN QUALIFICATION: SPEED UP BUILDS Update all dependencies to latest version Follow best practices Modularize Butterknife reflect Gradle build cache and build scans Monitor and keep on experimenting
67. @MOLSJEROEN QUALIFICATION: SPEED UP BUILDS 3m 45s 1m 26s
68. @MOLSJEROEN QUALIFICATION: SPEED UP CI Git hook instead of polling Git cache Preload gradle wrapper in docker container Unit tests over instrumentation tests Run subset of tests Parallel tasks
69. HOW MUCH TIME ARE YOU WILLING TO INVEST IN QUALIFYING PULL REQUESTS?
70. @MOLSJEROEN QUALIFICATION: SPEED UP TESTS Legacy “unit tests” written as instrumentation Converted 1272 tests Build speed decreased by 3m 18s -9% code coverage Removed CI bottleneck Make room for new Espresso tests
71. @MOLSJEROEN QUALIFICATION: SPEED UP BUILDS ~20 min 5m 10s
72. @MOLSJEROEN QUALIFICATION: SPEED UP TESTS Instrumentation tests Unit tests
73. @MOLSJEROEN QUALIFICATION: SPEED UP TESTS Instrumentation tests Unit tests
74. @MOLSJEROEN QUALIFICATION: SPEED UP TESTS … Instrumentation tests Unit tests …
75. @MOLSJEROEN QUALIFICATION: SPEED UP BUILDS (10 PRS) ~1 hour 5m 10s
76. @MOLSJEROEN VERIFICATION Develop feature
77. @MOLSJEROEN VERIFICATION: MOVE TO MASTER Develop feature
78. @MOLSJEROEN VERIFICATION: MOVE TO MASTER 1-3 days 0 days
79. @MOLSJEROEN DEVELOPMENT Develop feature
80. @MOLSJEROEN DEVELOPMENT: SMALLER BRANCHES feature Develop
81. @MOLSJEROEN DEVELOPMENT: SMALLER BRANCHES 1-5 weeks 0,5 - 2 days
82. "IF YOU MERGE EVERY DAY, SUDDENLY YOU NEVER GET TO THE POINT WHERE YOU HAVE HUGE CONFLICTS THAT ARE HARD TO RESOLVE." Linus Torvalds
83. @MOLSJEROEN RECAP: SPEEDING UP INTEGRATION Blazing fast builds A state of the art CI Do verification on develop Many small branches => Integration speed up from 1 - 5 weeks to 0,5 - 2days
84. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST QUALIFY MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY SHIP develop master release feature MERGE BACK 0,5-2 days 3 weeks 3 weeks
85. REDUCE COMPLEXITY
86. @MOLSJEROEN FEATURE BRANCHES develop master release feature bugfix feature hotfix
87. @MOLSJEROEN FEATURE BRANCHES: ONLY FROM DEVELOP develop master release feature bugfix hotfix
88. @MOLSJEROEN DEVELOP BRANCH develop master release feature bugfix hotfix
89. @MOLSJEROEN DEVELOP BRANCH master release feature bugfix hotfix
90. ACCIDENTAL VS ESSENTIAL COMPLEXITY
91. @MOLSJEROEN BUILD VARIANTS buildTypes { debug { signingConfig signingConfigs.debug ext.enableCrashlytics = false } release { minifyEnabled true proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro' signingConfig signingConfigs.release } uitest { // provide implementation for idling resources } } flavorDimensions "MODE" productFlavors { daily { applicationId 'com.philips.lighting.hue2.daily' dimension "MODE" buildConfigField "String", "HOCKEY_APP_ID", Keys.hockeyAppDaily buildConfigField "String", "AMPLITUDE_APP_ID", Keys.amplitudeDaily } store { dimension "MODE" buildConfigField "String", "HOCKEY_APP_ID", Keys.hockeyApp buildConfigField "String", "AMPLITUDE_APP_ID", Keys.amplitudeStore } } variantFilter { variant -> def names = variant.flavors*.name if (names.contains("store") && variant.buildType.name == 'debug') { variant.setIgnore(true) } if (names.contains("store") && variant.buildType.name == 'uitest') { variant.setIgnore(true) } }
92. @MOLSJEROEN BUILD VARIANTS Prod API keys Proguard Test settings Idling resource s Leak canary Prod firmware Develop
 Portal Release Release test Debug Espresso
93. @MOLSJEROEN IDLING RESOURCES: UI TEST VARIANT object EspressoIdlingResource { private val idling = BaseIdlingResource() fun increment(code: String, reason: String) = idling.increment(code, reason) fun decrement(code: String, reason: String) = idling.decrement(code, reason) }
94. @MOLSJEROEN IDLING RESOURCES: DEBUG AND RELEASE VARIANT object EspressoIdlingResource { fun increment(code: String, reason: String) { /* NOP */ } fun decrement(code: String, reason: String) { /* NOP */ } }
95. @MOLSJEROEN IDLING RESOURCES: UNIFIED object EspressoIdlingResource { private val idling = BaseIdlingResource() fun increment(code: String, reason: String) = if (RuntimeBehavior.isFeatureEnabled(IDLING_RESOURCES)) { idling.increment(code, reason) } fun decrement(code: String, reason: String) = if (RuntimeBehavior.isFeatureEnabled(IDLING_RESOURCES)) { mCountingIdlingResource.decrement(code reason) } }
96. @MOLSJEROEN IDLING RESOURCES: UNIFIED object EspressoIdlingResource { private val idling = BaseIdlingResource() fun increment(code: String, reason: String) = if (RuntimeBehavior.isFeatureEnabled(IDLING_RESOURCES)) { idling.increment(code, reason) } fun decrement(code: String, reason: String) = if (RuntimeBehavior.isFeatureEnabled(IDLING_RESOURCES)) { mCountingIdlingResource.decrement(code reason) } }
97. @MOLSJEROEN BUILD VARIANTS Prod API keys Proguard Test settings Idling resource s Leak canary Prod firmware Develop
 Portal Release Release test Debug Espresso
98. @MOLSJEROEN BUILD VARIANTS Prod API keys Proguard Test settings Idling resource s Leak canary Prod firmware Develop
 Portal Release Release test Debug Espresso
99. @MOLSJEROEN RELEASE AND DEBUG
100. @MOLSJEROEN SOURCE FOLDERS
101. @MOLSJEROEN GRADLE CONFIGURATION buildTypes { debug { signingConfig signingConfigs.debug applicationIdSuffix '.daily' ext.enableCrashlytics = false buildConfigField "String", "HOCKEY_APP_ID", Keys.hockeyAppDebug buildConfigField "String", "AMPLITUDE_APP_ID", Keys.amplitudeDebug } release { signingConfig signingConfigs.release minifyEnabled true proguardFiles 'proguard-rules.pro' buildConfigField "String", "HOCKEY_APP_ID", Keys.hockeyAppRelease buildConfigField "String", "AMPLITUDE_APP_ID", Keys.amplitudeRelease }
102. SIMPLICITY IS THE SOUL OF EFFICIENCY Austin Freeman
103. @MOLSJEROEN REDUCE COMPLEXITY: SUMMARY Drop unnecessary branches No branches from branches Minimize build variants Feature flags for dynamic configuration => Remove all accidental complexity
104. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST QUALIFY MERGE PROMOTE RELEASE STABILIZE VERIFY SHIP master release feature MERGE BACK 0,5-2 days 6 weeks*
105. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST QUALIFY MERGE PROMOTE RELEASE STABILIZE VERIFY RELEASE master release feature MERGE BACK 0,5-2 days ~4 weeks
106. STABILIZE MASTER
107. @MOLSJEROEN FEATURE FLAGS: NEW FEATURES override fun onViewCreated(view: View) { ... if (RuntimeBehavior.isFeatureEnabled(FeatureFlag.POWER_ON_BEHAVIOR)) { power_on_behavior.visibility = View.VISIBLE } else { power_on_behavior.visibility = View.GONE } ... }
108. @MOLSJEROEN FEATURE FLAGS: NEW FEATURES override fun onViewCreated(view: View) { ... if (RuntimeBehavior.isFeatureEnabled(FeatureFlag.POWER_ON_BEHAVIOR)) { power_on_behavior.visibility = View.VISIBLE } else { power_on_behavior.visibility = View.GONE } ... }
109. @MOLSJEROEN FEATURE FLAGS: EXISTING CODE fun withLegacy() { var code = LegacyLogic() code.with(OutUnitTests()) // ... code.doMagic() }
110. @MOLSJEROEN FEATURE FLAGS: EXISTING CODE fun withLegacy() { var code = LegacyLogic() code.with(OutUnitTests()) // ... code.doMagic() }
111. @MOLSJEROEN FEATURE FLAGS: EXISTING CODE fun withLegacy() { var code = NewLogic() code.with(UnitTests()) // ... code.doMagic() }
112. @MOLSJEROEN FEATURE FLAGS: EXISTING CODE fun withLegacy() { if (RuntimeBehavior.isFeatureEnabled(FeatureFlag.LOGIC_REFACTOR)) { var code = NewLogic() code.with(UnitTests()) // ... code.doMagic() } else { var code = LegacyLogic() code.with(OutUnitTests()) // ... code.doMagic() } }
113. @MOLSJEROEN
114. @MOLSJEROEN QUALIFY PULL REQUESTS master feature
115. @MOLSJEROEN QUALIFY PULL REQUESTS master feature 
 BUILD
 REVIEW

116. @MOLSJEROEN QUALIFY PULL REQUESTS master feature 
 BUILD
 REVIEW ANALYZE

117. @MOLSJEROEN QUALIFY PULL REQUESTS master feature 
 BUILD
 REVIEW
 ANALYZE
 TEST
118. @MOLSJEROEN AUTOMATE TESTING UNIT INTEGRATION E2E
119. @MOLSJEROEN AUTOMATE TESTING Library Feature Feature Feature Feature Feature Feature Feature Feature Feature App Library Library Lib Lib Library Library Lib
120. @MOLSJEROEN
122. @MOLSJEROEN TEST STRATEGY Pull requests: Run all automated level1 app Espresso tests Run all automated tests of the modules that are impacted Manual verification if high impact Release: Run all automated Espresso tests Run all automated feature and library tests Deploy: Manual exploratory testing
123. @MOLSJEROEN TEST STRATEGY Pull requests: Run all automated level1 app Espresso tests Run all automated tests of the modules that are impacted Manual verification if high impact Release: Run all automated Espresso tests Run all automated feature and library tests Deploy: Manual exploratory testing
124. @MOLSJEROEN SIMULATE ACTUAL MERGE master feature BUILD
 REVIEW
 ANALYZE
 TEST
125. @MOLSJEROEN SIMULATE ACTUAL MERGE master feature 
 MERGE BUILD
 REVIEW
 ANALYZE
 TEST
126. @MOLSJEROEN master feature SIMULATE ACTUAL MERGE
127. @MOLSJEROEN master feature SIMULATE ACTUAL MERGE
128. @MOLSJEROEN master feature feature feature SIMULATE ACTUAL MERGE
129. @MOLSJEROEN master feature feature feature SIMULATE ACTUAL MERGE
130. @MOLSJEROEN master feature feature feature SIMULATE ACTUAL MERGE
131. @MOLSJEROEN MERGE PULL REQUESTS master feature 
 MERGE BUILD
 REVIEW
 ANALYZE
 TEST
132. @MOLSJEROEN MERGE PULL REQUESTS master feature 
 MERGE BUILD
 REVIEW
 ANALYZE
 TEST IMPACT

133. @MOLSJEROEN IMPACT ANALYSIS SIDE EFFECTS POSSIBLE IN 
 OTHER PARTS OF CODE? (REGRESSION) ALL SIDE EFFECTS CONTAINED BEHIND A FEATURE FLAG? LOW IMPACT LOW IMPACT HIGH IMPACT Yes Yes No Not sure Not sure No
134. @MOLSJEROEN IMPACT ANALYSIS
135. @MOLSJEROEN STABILIZE MASTER: SUMMARY Use feature toggles Static code analysis Heavily automate testing Merge master before checks Requalify pull requests after each merge Impact assessment => Able to drop release branch
136. @MOLSJEROEN DEVELOPMENT PROCESS 0,5-2 days 2 weeks FEATURE BRANCH FEATURE FLAG DEVELOP PULL REQUEST MERGE QUALIFY
 ++ VERIFY IMPACT master feature BETA MANUAL RELEASE SHIP
137. @MOLSJEROEN DEVELOPMENT PROCESS 0,5-2 days 2 weeks FEATURE BRANCH FEATURE FLAG DEVELOP PULL REQUEST MERGE QUALIFY
 ++ VERIFY IMPACT master feature BETA MANUAL RELEASE SHIP
138. AUTOMATE DELIVERY
139. @MOLSJEROEN UPDATE TRANSLATIONS master feature
140. @MOLSJEROEN UPDATE TRANSLATIONS master feature translations
141. @MOLSJEROEN UPDATE TRANSLATIONS master feature translations
142. @MOLSJEROEN UPDATE SDK AND ASSETS master feature translations SDK assets
143. @MOLSJEROEN RELEASE APP (DELIVERY) stage('Deploy to Google Play Internal Test Track') { steps { script { def releaseNotes = readFile 'app/build/release_notes.txt' androidApkUpload( googleCredentialsId: 'Google Play Android Developer', trackName: 'internal', apkFilesPattern: '**/*-release.apk', deobfuscationFilesPattern: '**/mapping.txt', recentChangeList: [[language: 'en-US', text: releaseNotes]] ) } } }
144. @MOLSJEROEN RELEASE APP (DELIVERY) stage('Deploy to HockeyApp') { steps { sh "pipelines/scripts/hockeyapp/deploy_app_debug.sh" } }
145. @MOLSJEROEN IDEAL SCENARIO master feature bugfix
146. @MOLSJEROEN IDEAL SCENARIO master feature bugfix hotfix
147. @MOLSJEROEN IDEAL SCENARIO master feature bugfix hotfix
148. @MOLSJEROEN IDEAL SCENARIO master feature bugfix hotfix
149. @MOLSJEROEN ACTUAL SCENARIO master feature bugfix hotfix
150. @MOLSJEROEN ACTUAL SCENARIO master feature bugfix hotfix
151. @MOLSJEROEN ACTUAL SCENARIO master feature bugfix hotfix
152. @MOLSJEROEN ACTUAL SCENARIO master feature bugfix hotfix
153. @MOLSJEROEN RELEASING master internal test beta production
154. @MOLSJEROEN RELEASING master internal test beta production
155. @MOLSJEROEN AUTOMATE DELIVERY: SUMMARY Integrate dependencies automatically Automate upload to Google play/hockey app Release directly from master Hotfixes from separate branch
156. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH FEATURE FLAG DEVELOP PULL REQUEST MERGE QUALIFY
 ++ master RELEASE VERIFY feature IMPACT 0,5-2 days 2 hours BETA SHIP 1 week*
157. LESSONS LEARNED
158. @MOLSJEROEN 1. BI-WEEKLY RELEASES DEVELOPMENT TESTING TESTING BETA DEVELOPMENT TESTING TESTING BETA DEVELOPMENT TESTING TESTIN 10 weeks
159. @MOLSJEROEN 1. BI-WEEKLY RELEASES DEV & TEST BETA DEV & TEST BETA DEV & TEST BETA 2 weeks
160. @MOLSJEROEN 1. BI-WEEKLY RELEASES BETA DEVELOPMENT TEST TEST BETA DEVELOPMENT TEST TEST BETA DEVELOPMENT TEST TEST
161. @MOLSJEROEN 1. BI-WEEKLY RELEASES BETA DEVELOPMENT TEST TEST BETA DEVELOPMENT TEST TEST BETA DEVELOPMENT TEST TEST
162. @MOLSJEROEN 1. BI-WEEKLY RELEASES DEV & TEST BETA DEV & TEST BETA DEV & TEST BETA 2 weeks DEV & TEST BETA
163. @MOLSJEROEN 2. MERGING WITHOUT VERIFICATION master feature 
 MERGE BUILD
 REVIEW
 ANALYZE
 TEST IMPACT

164. @MOLSJEROEN 2. MERGING WITHOUT VERIFICATION
165. @MOLSJEROEN 2. MERGING WITHOUT VERIFICATION
166. TRUST TAKES YEARS TO BUILD, SECONDS TO BREAK, AND FOREVER TO REPAIR. Unknown
167. @MOLSJEROEN 2. MERGING WITHOUT VERIFICATION
168. @MOLSJEROEN 2. MERGING WITHOUT VERIFICATION
169. @MOLSJEROEN 3. YOU WILL MESS UP
170. @MOLSJEROEN SUMMARY Focus on whitebox testing Gain trust and give trust back Don’t blow your trust! Rely on automated testing Closer collaboration between testers and developers Root cause analyses
171. RESULTS
172. @MOLSJEROEN RESULTS 29
173. @MOLSJEROEN TIME TO RELEASE Build time (min) 0 10 20 30 40 50 Build number
174. @MOLSJEROEN
175. @MOLSJEROEN
176. @MOLSJEROEN INTEGRATION SPEED 620 ~1/dev/day
177. @MOLSJEROEN APP RATING
178. @MOLSJEROEN PR INFLOW 0 500 1000 1500 2000 JAN 2018 APRIL 2018 JULY 2018 OCT 2018
179. @MOLSJEROEN PRODUCTIVITY 0 10 20 30 40 50 NOV 2018 DEC 2018 JAN 2019 MARCH 2019
180. @MOLSJEROEN ONE. SINGLE. BRANCH. master feature bugfix
181. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH FEATURE FLAG DEVELOP PULL REQUEST MERGE QUALIFY
 ++ master RELEASE VERIFY feature IMPACT 0,5-2 days 2 hours BETA SHIP 1 week*
182. RELEASING HAS BECOME SO EASY, WE EVEN SHIP FRIDAY EVENINGS Hue android team
183. STRATEGY OF REDUCING COST OF FAILURE IS WORKING
184. WRAP UP
185. “THE MOST POWERFUL TOOL WE HAVE AS DEVELOPERS IS AUTOMATION.” Scott Hanselman
186. @MOLSJEROEN SPEED UP INTEGRATION REMOVE ACCIDENTAL COMPLEXITY STABILIZE MASTER AUTOMATE TESTING AND DELIVERY PREFER TRUST OVER PROCESS
187. @MOLSJEROEN HTTPS://JEROENMOLS.COM/BLOG
188. @MOLSJEROEN IMAGE CREDITS Welcome image by Clement127 https://www.flickr.com/photos/clement127/35754378573/in/photolist-Wtux5Z Material design icons by Google
 https://material.io/tools/icons
189. MOLSJEROEN