Pro Yearly is on sale from $80 to $50! »

Journey to painless releases: Continuous delivery for Philips Hue Android

Journey to painless releases: Continuous delivery for Philips Hue Android

Video: https://www.youtube.com/watch?v=VkFcJZWtufA

Wind back one year ago:
- releasing took us 10 weeks
- our app was plagued by regression
- developers felt completely detached from our users

Clearly our traditional strategy of heavily testing each release (up-front quality) wasn't working. So we decided to flip things around: why not try to reduce the cost of errors instead? Hence we decided to move our Philips Hue Android app to continuous deployment.

In this talk you will learn that if releasing is painful, you should actually release even more often. Step by step I'll explain how we moved the Philips Hue Android app to continuous deployment, what our current release process/CI looks like, what changes we made to our overall testing approach, how we empowered developers and testers, and how we made our release process completely painless.

5f57d2d205e77e185986459c1b89a874?s=128

Jeroen Mols

April 23, 2019
Tweet

Transcript

  1. @MOLSJEROEN JOURNEY TO PAINLESS RELEASES CONTINUOUS DELIVERY FOR PHILIPS HUE

  2. @MOLSJEROEN @MOLSJEROEN

  3. @MOLSJEROEN CHALLENGES CONTINUOUS DELIVERY MIGRATE YOUR PROJECT RESULTS

  4. CHALLENGES

  5. @MOLSJEROEN RELEASE PROCESS DEVELOPMENT

  6. @MOLSJEROEN RELEASE PROCESS DEVELOPMENT TESTING

  7. @MOLSJEROEN RELEASE PROCESS DEVELOPMENT TESTING TESTING

  8. @MOLSJEROEN RELEASE PROCESS DEVELOPMENT TESTING TESTING BETA

  9. @MOLSJEROEN RELEASE PROCESS DEVELOPMENT TESTING TESTING BETA 10 weeks

  10. @MOLSJEROEN RELEASE PROCESS DEVELOPMENT TESTING TESTING BETA DEVELOPMENT TESTING TESTING

    BETA 10 weeks
  11. @MOLSJEROEN RELEASE PROCESS DEVELOPMENT TESTING TESTING BETA DEVELOPMENT TESTING TESTING

    BETA DEVELOPMENT TESTING TESTIN 10 weeks
  12. @MOLSJEROEN RELEASE PROCESS DEVELOPMENT TESTING TESTING BETA DEVELOPMENT TESTING TESTING

    BETA DEVELOPMENT TESTING TESTIN 10 weeks 3 releases
  13. @MOLSJEROEN RELEASE PROCESS DEVELOPMENT TESTING TESTING BETA 10 weeks

  14. @MOLSJEROEN RELEASE PROCESS DEVELOPMENT TESTING TESTING BETA 10 weeks

  15. @MOLSJEROEN RELEASE PROCESS DEVELOPMENT TESTING TESTING BETA 10 weeks

  16. @MOLSJEROEN FRUSTRATIONS Issues were found extremely late Developers felt detached

    from our users Constant context switches between releases Releasing very painful Friction between developers and testers
  17. @MOLSJEROEN BRANCHING MODEL develop master release feature bugfix feature hotfix

  18. @MOLSJEROEN hotfix feature bugfix BRANCHING MODEL master release feature develop

  19. @MOLSJEROEN BRANCHING MODEL develop master release feature bugfix feature hotfix

  20. @MOLSJEROEN BRANCHING MODEL develop master release feature bugfix feature hotfix

  21. @MOLSJEROEN BRANCHING MODEL develop master release feature bugfix feature hotfix

  22. @MOLSJEROEN FRUSTRATIONS Unstable development branch Merging nightmares Poor code reviews

    Confusion about what fix is where Development was slow
  23. @MOLSJEROEN RESULTS

  24. STRATEGY OF UPFRONT TESTING NOT WORKING

  25. @MOLSJEROEN GOALS Speed up development Get a grip on quality

    Feedback between developers and users
  26. None
  27. None
  28. CONTINUOUS DELIVERY

  29. IF IT HURTS, DO IT MORE OFTEN Jez Humble

  30. @MOLSJEROEN WHAT DO WE WANT TO ACHIEVE? DEVELOPMENT TESTING TESTING

    BETA DEVELOPMENT TESTING TESTING BETA DEVELOPMENT TESTING TESTIN
  31. @MOLSJEROEN WHAT DO WE WANT TO ACHIEVE? DEVELOPMENT TESTING TESTING

    BETA DEVELOPMENT TESTING TESTING BETA DEVELOPMENT TESTING TESTING BETA
  32. @MOLSJEROEN WHAT DO WE WANT TO ACHIEVE? DEVELOPMENT TESTING BETA

    TESTING DEVELOPMENT TESTING BETA TESTING DEVELOPMENT TESTING BETA TESTING
  33. @MOLSJEROEN WHAT DO WE WANT TO ACHIEVE? BETA DEVELOPMENT TEST

    TEST BETA DEVELOPMENT TEST TEST BETA DEVELOPMENT TEST TEST
  34. @MOLSJEROEN WHAT DO WE WANT TO ACHIEVE? DEV & TEST

    BETA DEV & TEST BETA DEV & TEST BETA
  35. IF IT HURTS, DO IT MORE OFTEN AND BRING THE

    PAIN FORWARD Jez Humble
  36. @MOLSJEROEN TERMINOLOGY Continuous integration Continuous delivery Continuous deployment

  37. DEVELOPMENT PROCESS

  38. @MOLSJEROEN DEVELOPMENT PROCESS develop master release

  39. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH develop master release feature

  40. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST develop

    master release feature
  41. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY

    develop master release feature
  42. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY

    develop master release feature VERIFY
  43. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY

    MERGE VERIFY develop master release feature
  44. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY

    MERGE VERIFY develop master release feature 1-5 weeks
  45. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY

    MERGE PROMOTE MASTER VERIFY develop master release feature 1-5 weeks
  46. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY

    MERGE PROMOTE MASTER VERIFY develop master release feature STABILIZE 1-5 weeks
  47. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY

    MERGE PROMOTE MASTER VERIFY develop master release feature STABILIZE 3 weeks 1-5 weeks
  48. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY

    MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY develop master release feature 3 weeks 1-5 weeks
  49. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST

    QUALIFY MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY develop master release feature 3 weeks 1-5 weeks
  50. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST

    QUALIFY MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY develop master release feature 3 weeks 1-5 weeks SHIP
  51. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST

    QUALIFY MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY SHIP develop master release feature MERGE BACK 1-5 weeks 3 weeks
  52. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST

    QUALIFY MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY SHIP develop master release feature MERGE BACK 1-5 weeks 3 weeks 3 weeks
  53. @MOLSJEROEN MERGING/INTEGRATING FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST MERGE

    PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY SHIP develop master release feature MERGE BACK 1-5 weeks 3 weeks 3 weeks QUALIFY
  54. @MOLSJEROEN MERGING/INTEGRATING FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST MERGE

    PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY SHIP develop master release feature MERGE BACK 1-5 weeks 3 weeks 3 weeks QUALIFY
  55. SPEED UP INTEGRATION

  56. @MOLSJEROEN

  57. @MOLSJEROEN

  58. @MOLSJEROEN Large pull request

  59. @MOLSJEROEN Large pull request Hard to
 review

  60. @MOLSJEROEN Large pull request Hard to
 review Hard to
 verify

  61. @MOLSJEROEN Large pull request Hard to
 review Long lived
 pull

    request Hard to
 verify
  62. @MOLSJEROEN Large pull request Hard to
 review Long lived
 pull

    request Merge
 conflicts Hard to
 verify
  63. @MOLSJEROEN Large pull request Hard to
 review Long lived
 pull

    request Merge
 conflicts Overhead: 
 regression, 
 dependencies,… Hard to
 verify
  64. @MOLSJEROEN MERGING/INTEGRATING FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST MERGE

    PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY SHIP MERGE BACK 1-5 weeks 3 weeks 3 weeks QUALIFY FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY SHIP develop master release feature MERGE BACK 1-5 weeks 3 weeks 3 weeks QUALIFY
  65. @MOLSJEROEN PULL REQUEST QUALIFICATION Develop feature BUILD
 TEST REVIEW

  66. @MOLSJEROEN QUALIFICATION: SPEED UP BUILDS Update all dependencies to latest

    version Follow best practices Modularize Butterknife reflect Gradle build cache and build scans Monitor and keep on experimenting
  67. @MOLSJEROEN QUALIFICATION: SPEED UP BUILDS 3m 45s 1m 26s

  68. @MOLSJEROEN QUALIFICATION: SPEED UP CI Git hook instead of polling

    Git cache Preload gradle wrapper in docker container Unit tests over instrumentation tests Run subset of tests Parallel tasks
  69. HOW MUCH TIME ARE YOU WILLING TO INVEST IN QUALIFYING

    PULL REQUESTS?
  70. @MOLSJEROEN QUALIFICATION: SPEED UP TESTS Legacy “unit tests” written as

    instrumentation Converted 1272 tests Build speed decreased by 3m 18s -9% code coverage Removed CI bottleneck Make room for new Espresso tests
  71. @MOLSJEROEN QUALIFICATION: SPEED UP BUILDS ~20 min 5m 10s

  72. @MOLSJEROEN QUALIFICATION: SPEED UP TESTS Instrumentation tests Unit tests

  73. @MOLSJEROEN QUALIFICATION: SPEED UP TESTS Instrumentation tests Unit tests

  74. @MOLSJEROEN QUALIFICATION: SPEED UP TESTS … Instrumentation tests Unit tests

  75. @MOLSJEROEN QUALIFICATION: SPEED UP BUILDS (10 PRS) ~1 hour 5m

    10s
  76. @MOLSJEROEN VERIFICATION Develop feature

  77. @MOLSJEROEN VERIFICATION: MOVE TO MASTER Develop feature

  78. @MOLSJEROEN VERIFICATION: MOVE TO MASTER 1-3 days 0 days

  79. @MOLSJEROEN DEVELOPMENT Develop feature

  80. @MOLSJEROEN DEVELOPMENT: SMALLER BRANCHES feature Develop

  81. @MOLSJEROEN DEVELOPMENT: SMALLER BRANCHES 1-5 weeks 0,5 - 2 days

  82. "IF YOU MERGE EVERY DAY, SUDDENLY YOU NEVER GET TO

    THE POINT WHERE YOU HAVE HUGE CONFLICTS THAT ARE HARD TO RESOLVE." Linus Torvalds
  83. @MOLSJEROEN RECAP: SPEEDING UP INTEGRATION Blazing fast builds A state

    of the art CI Do verification on develop Many small branches => Integration speed up from 1 - 5 weeks to 0,5 - 2days
  84. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST

    QUALIFY MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY SHIP develop master release feature MERGE BACK 0,5-2 days 3 weeks 3 weeks
  85. REDUCE COMPLEXITY

  86. @MOLSJEROEN FEATURE BRANCHES develop master release feature bugfix feature hotfix

  87. @MOLSJEROEN FEATURE BRANCHES: ONLY FROM DEVELOP develop master release feature

    bugfix hotfix
  88. @MOLSJEROEN DEVELOP BRANCH develop master release feature bugfix hotfix

  89. @MOLSJEROEN DEVELOP BRANCH master release feature bugfix hotfix

  90. ACCIDENTAL VS ESSENTIAL COMPLEXITY

  91. @MOLSJEROEN BUILD VARIANTS buildTypes { debug { signingConfig signingConfigs.debug ext.enableCrashlytics

    = false } release { minifyEnabled true proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro' signingConfig signingConfigs.release } uitest { // provide implementation for idling resources } } flavorDimensions "MODE" productFlavors { daily { applicationId 'com.philips.lighting.hue2.daily' dimension "MODE" buildConfigField "String", "HOCKEY_APP_ID", Keys.hockeyAppDaily buildConfigField "String", "AMPLITUDE_APP_ID", Keys.amplitudeDaily } store { dimension "MODE" buildConfigField "String", "HOCKEY_APP_ID", Keys.hockeyApp buildConfigField "String", "AMPLITUDE_APP_ID", Keys.amplitudeStore } } variantFilter { variant -> def names = variant.flavors*.name if (names.contains("store") && variant.buildType.name == 'debug') { variant.setIgnore(true) } if (names.contains("store") && variant.buildType.name == 'uitest') { variant.setIgnore(true) } }
  92. @MOLSJEROEN BUILD VARIANTS Prod API keys Proguard Test settings Idling

    resource s Leak canary Prod firmware Develop
 Portal Release Release test Debug Espresso
  93. @MOLSJEROEN IDLING RESOURCES: UI TEST VARIANT object EspressoIdlingResource { private

    val idling = BaseIdlingResource() fun increment(code: String, reason: String) = idling.increment(code, reason) fun decrement(code: String, reason: String) = idling.decrement(code, reason) }
  94. @MOLSJEROEN IDLING RESOURCES: DEBUG AND RELEASE VARIANT object EspressoIdlingResource {

    fun increment(code: String, reason: String) { /* NOP */ } fun decrement(code: String, reason: String) { /* NOP */ } }
  95. @MOLSJEROEN IDLING RESOURCES: UNIFIED object EspressoIdlingResource { private val idling

    = BaseIdlingResource() fun increment(code: String, reason: String) = if (RuntimeBehavior.isFeatureEnabled(IDLING_RESOURCES)) { idling.increment(code, reason) } fun decrement(code: String, reason: String) = if (RuntimeBehavior.isFeatureEnabled(IDLING_RESOURCES)) { mCountingIdlingResource.decrement(code reason) } }
  96. @MOLSJEROEN IDLING RESOURCES: UNIFIED object EspressoIdlingResource { private val idling

    = BaseIdlingResource() fun increment(code: String, reason: String) = if (RuntimeBehavior.isFeatureEnabled(IDLING_RESOURCES)) { idling.increment(code, reason) } fun decrement(code: String, reason: String) = if (RuntimeBehavior.isFeatureEnabled(IDLING_RESOURCES)) { mCountingIdlingResource.decrement(code reason) } }
  97. @MOLSJEROEN BUILD VARIANTS Prod API keys Proguard Test settings Idling

    resource s Leak canary Prod firmware Develop
 Portal Release Release test Debug Espresso
  98. @MOLSJEROEN BUILD VARIANTS Prod API keys Proguard Test settings Idling

    resource s Leak canary Prod firmware Develop
 Portal Release Release test Debug Espresso
  99. @MOLSJEROEN RELEASE AND DEBUG

  100. @MOLSJEROEN SOURCE FOLDERS

  101. @MOLSJEROEN GRADLE CONFIGURATION buildTypes { debug { signingConfig signingConfigs.debug applicationIdSuffix

    '.daily' ext.enableCrashlytics = false buildConfigField "String", "HOCKEY_APP_ID", Keys.hockeyAppDebug buildConfigField "String", "AMPLITUDE_APP_ID", Keys.amplitudeDebug } release { signingConfig signingConfigs.release minifyEnabled true proguardFiles 'proguard-rules.pro' buildConfigField "String", "HOCKEY_APP_ID", Keys.hockeyAppRelease buildConfigField "String", "AMPLITUDE_APP_ID", Keys.amplitudeRelease }
  102. SIMPLICITY IS THE SOUL OF EFFICIENCY Austin Freeman

  103. @MOLSJEROEN REDUCE COMPLEXITY: SUMMARY Drop unnecessary branches No branches from

    branches Minimize build variants Feature flags for dynamic configuration => Remove all accidental complexity
  104. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST

    QUALIFY MERGE PROMOTE RELEASE STABILIZE VERIFY SHIP master release feature MERGE BACK 0,5-2 days 6 weeks*
  105. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST

    QUALIFY MERGE PROMOTE RELEASE STABILIZE VERIFY RELEASE master release feature MERGE BACK 0,5-2 days ~4 weeks
  106. STABILIZE MASTER

  107. @MOLSJEROEN FEATURE FLAGS: NEW FEATURES override fun onViewCreated(view: View) {

    ... if (RuntimeBehavior.isFeatureEnabled(FeatureFlag.POWER_ON_BEHAVIOR)) { power_on_behavior.visibility = View.VISIBLE } else { power_on_behavior.visibility = View.GONE } ... }
  108. @MOLSJEROEN FEATURE FLAGS: NEW FEATURES override fun onViewCreated(view: View) {

    ... if (RuntimeBehavior.isFeatureEnabled(FeatureFlag.POWER_ON_BEHAVIOR)) { power_on_behavior.visibility = View.VISIBLE } else { power_on_behavior.visibility = View.GONE } ... }
  109. @MOLSJEROEN FEATURE FLAGS: EXISTING CODE fun withLegacy() { var code

    = LegacyLogic() code.with(OutUnitTests()) // ... code.doMagic() }
  110. @MOLSJEROEN FEATURE FLAGS: EXISTING CODE fun withLegacy() { var code

    = LegacyLogic() code.with(OutUnitTests()) // ... code.doMagic() }
  111. @MOLSJEROEN FEATURE FLAGS: EXISTING CODE fun withLegacy() { var code

    = NewLogic() code.with(UnitTests()) // ... code.doMagic() }
  112. @MOLSJEROEN FEATURE FLAGS: EXISTING CODE fun withLegacy() { if (RuntimeBehavior.isFeatureEnabled(FeatureFlag.LOGIC_REFACTOR))

    { var code = NewLogic() code.with(UnitTests()) // ... code.doMagic() } else { var code = LegacyLogic() code.with(OutUnitTests()) // ... code.doMagic() } }
  113. @MOLSJEROEN

  114. @MOLSJEROEN QUALIFY PULL REQUESTS master feature

  115. @MOLSJEROEN QUALIFY PULL REQUESTS master feature 
 BUILD
 REVIEW


  116. @MOLSJEROEN QUALIFY PULL REQUESTS master feature 
 BUILD
 REVIEW ANALYZE


  117. @MOLSJEROEN QUALIFY PULL REQUESTS master feature 
 BUILD
 REVIEW
 ANALYZE


    TEST
  118. @MOLSJEROEN AUTOMATE TESTING UNIT INTEGRATION E2E

  119. @MOLSJEROEN AUTOMATE TESTING Library Feature Feature Feature Feature Feature Feature

    Feature Feature Feature App Library Library Lib Lib Library Library Lib
  120. @MOLSJEROEN

  121. None
  122. @MOLSJEROEN TEST STRATEGY Pull requests: Run all automated level1 app

    Espresso tests Run all automated tests of the modules that are impacted Manual verification if high impact Release: Run all automated Espresso tests Run all automated feature and library tests Deploy: Manual exploratory testing
  123. @MOLSJEROEN TEST STRATEGY Pull requests: Run all automated level1 app

    Espresso tests Run all automated tests of the modules that are impacted Manual verification if high impact Release: Run all automated Espresso tests Run all automated feature and library tests Deploy: Manual exploratory testing
  124. @MOLSJEROEN SIMULATE ACTUAL MERGE master feature BUILD
 REVIEW
 ANALYZE
 TEST

  125. @MOLSJEROEN SIMULATE ACTUAL MERGE master feature 
 MERGE BUILD
 REVIEW


    ANALYZE
 TEST
  126. @MOLSJEROEN master feature SIMULATE ACTUAL MERGE

  127. @MOLSJEROEN master feature SIMULATE ACTUAL MERGE

  128. @MOLSJEROEN master feature feature feature SIMULATE ACTUAL MERGE

  129. @MOLSJEROEN master feature feature feature SIMULATE ACTUAL MERGE

  130. @MOLSJEROEN master feature feature feature SIMULATE ACTUAL MERGE

  131. @MOLSJEROEN MERGE PULL REQUESTS master feature 
 MERGE BUILD
 REVIEW


    ANALYZE
 TEST
  132. @MOLSJEROEN MERGE PULL REQUESTS master feature 
 MERGE BUILD
 REVIEW


    ANALYZE
 TEST IMPACT

  133. @MOLSJEROEN IMPACT ANALYSIS SIDE EFFECTS POSSIBLE IN 
 OTHER PARTS

    OF CODE? (REGRESSION) ALL SIDE EFFECTS CONTAINED BEHIND A FEATURE FLAG? LOW IMPACT LOW IMPACT HIGH IMPACT Yes Yes No Not sure Not sure No
  134. @MOLSJEROEN IMPACT ANALYSIS

  135. @MOLSJEROEN STABILIZE MASTER: SUMMARY Use feature toggles Static code analysis

    Heavily automate testing Merge master before checks Requalify pull requests after each merge Impact assessment => Able to drop release branch
  136. @MOLSJEROEN DEVELOPMENT PROCESS 0,5-2 days 2 weeks FEATURE BRANCH FEATURE

    FLAG DEVELOP PULL REQUEST MERGE QUALIFY
 ++ VERIFY IMPACT master feature BETA MANUAL RELEASE SHIP
  137. @MOLSJEROEN DEVELOPMENT PROCESS 0,5-2 days 2 weeks FEATURE BRANCH FEATURE

    FLAG DEVELOP PULL REQUEST MERGE QUALIFY
 ++ VERIFY IMPACT master feature BETA MANUAL RELEASE SHIP
  138. AUTOMATE DELIVERY

  139. @MOLSJEROEN UPDATE TRANSLATIONS master feature

  140. @MOLSJEROEN UPDATE TRANSLATIONS master feature translations

  141. @MOLSJEROEN UPDATE TRANSLATIONS master feature translations

  142. @MOLSJEROEN UPDATE SDK AND ASSETS master feature translations SDK assets

  143. @MOLSJEROEN RELEASE APP (DELIVERY) stage('Deploy to Google Play Internal Test

    Track') { steps { script { def releaseNotes = readFile 'app/build/release_notes.txt' androidApkUpload( googleCredentialsId: 'Google Play Android Developer', trackName: 'internal', apkFilesPattern: '**/*-release.apk', deobfuscationFilesPattern: '**/mapping.txt', recentChangeList: [[language: 'en-US', text: releaseNotes]] ) } } }
  144. @MOLSJEROEN RELEASE APP (DELIVERY) stage('Deploy to HockeyApp') { steps {

    sh "pipelines/scripts/hockeyapp/deploy_app_debug.sh" } }
  145. @MOLSJEROEN IDEAL SCENARIO master feature bugfix

  146. @MOLSJEROEN IDEAL SCENARIO master feature bugfix hotfix

  147. @MOLSJEROEN IDEAL SCENARIO master feature bugfix hotfix

  148. @MOLSJEROEN IDEAL SCENARIO master feature bugfix hotfix

  149. @MOLSJEROEN ACTUAL SCENARIO master feature bugfix hotfix

  150. @MOLSJEROEN ACTUAL SCENARIO master feature bugfix hotfix

  151. @MOLSJEROEN ACTUAL SCENARIO master feature bugfix hotfix

  152. @MOLSJEROEN ACTUAL SCENARIO master feature bugfix hotfix

  153. @MOLSJEROEN RELEASING master internal test beta production

  154. @MOLSJEROEN RELEASING master internal test beta production

  155. @MOLSJEROEN AUTOMATE DELIVERY: SUMMARY Integrate dependencies automatically Automate upload to

    Google play/hockey app Release directly from master Hotfixes from separate branch
  156. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH FEATURE FLAG DEVELOP PULL REQUEST

    MERGE QUALIFY
 ++ master RELEASE VERIFY feature IMPACT 0,5-2 days 2 hours BETA SHIP 1 week*
  157. LESSONS LEARNED

  158. @MOLSJEROEN 1. BI-WEEKLY RELEASES DEVELOPMENT TESTING TESTING BETA DEVELOPMENT TESTING

    TESTING BETA DEVELOPMENT TESTING TESTIN 10 weeks
  159. @MOLSJEROEN 1. BI-WEEKLY RELEASES DEV & TEST BETA DEV &

    TEST BETA DEV & TEST BETA 2 weeks
  160. @MOLSJEROEN 1. BI-WEEKLY RELEASES BETA DEVELOPMENT TEST TEST BETA DEVELOPMENT

    TEST TEST BETA DEVELOPMENT TEST TEST
  161. @MOLSJEROEN 1. BI-WEEKLY RELEASES BETA DEVELOPMENT TEST TEST BETA DEVELOPMENT

    TEST TEST BETA DEVELOPMENT TEST TEST
  162. @MOLSJEROEN 1. BI-WEEKLY RELEASES DEV & TEST BETA DEV &

    TEST BETA DEV & TEST BETA 2 weeks DEV & TEST BETA
  163. @MOLSJEROEN 2. MERGING WITHOUT VERIFICATION master feature 
 MERGE BUILD


    REVIEW
 ANALYZE
 TEST IMPACT

  164. @MOLSJEROEN 2. MERGING WITHOUT VERIFICATION

  165. @MOLSJEROEN 2. MERGING WITHOUT VERIFICATION

  166. TRUST TAKES YEARS TO BUILD, SECONDS TO BREAK, AND FOREVER

    TO REPAIR. Unknown
  167. @MOLSJEROEN 2. MERGING WITHOUT VERIFICATION

  168. @MOLSJEROEN 2. MERGING WITHOUT VERIFICATION

  169. @MOLSJEROEN 3. YOU WILL MESS UP

  170. @MOLSJEROEN SUMMARY Focus on whitebox testing Gain trust and give

    trust back Don’t blow your trust! Rely on automated testing Closer collaboration between testers and developers Root cause analyses
  171. RESULTS

  172. @MOLSJEROEN RESULTS 29

  173. @MOLSJEROEN TIME TO RELEASE Build time (min) 0 10 20

    30 40 50 Build number
  174. @MOLSJEROEN

  175. @MOLSJEROEN

  176. @MOLSJEROEN INTEGRATION SPEED 620 ~1/dev/day

  177. @MOLSJEROEN APP RATING

  178. @MOLSJEROEN PR INFLOW 0 500 1000 1500 2000 JAN 2018

    APRIL 2018 JULY 2018 OCT 2018
  179. @MOLSJEROEN PRODUCTIVITY 0 10 20 30 40 50 NOV 2018

    DEC 2018 JAN 2019 MARCH 2019
  180. @MOLSJEROEN ONE. SINGLE. BRANCH. master feature bugfix

  181. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH FEATURE FLAG DEVELOP PULL REQUEST

    MERGE QUALIFY
 ++ master RELEASE VERIFY feature IMPACT 0,5-2 days 2 hours BETA SHIP 1 week*
  182. RELEASING HAS BECOME SO EASY, WE EVEN SHIP FRIDAY EVENINGS

    Hue android team
  183. STRATEGY OF REDUCING COST OF FAILURE IS WORKING

  184. WRAP UP

  185. “THE MOST POWERFUL TOOL WE HAVE AS DEVELOPERS IS AUTOMATION.”

    Scott Hanselman
  186. @MOLSJEROEN SPEED UP INTEGRATION REMOVE ACCIDENTAL COMPLEXITY STABILIZE MASTER AUTOMATE

    TESTING AND DELIVERY PREFER TRUST OVER PROCESS
  187. @MOLSJEROEN HTTPS://JEROENMOLS.COM/BLOG

  188. @MOLSJEROEN IMAGE CREDITS Welcome image by Clement127 https://www.flickr.com/photos/clement127/35754378573/in/photolist-Wtux5Z Material design

    icons by Google
 https://material.io/tools/icons
  189. MOLSJEROEN