Journey to painless releases: Continuous delivery for Philips Hue Android

Journey to painless releases: Continuous delivery for Philips Hue Android

Video: https://www.youtube.com/watch?v=VkFcJZWtufA

Wind back one year ago:
- releasing took us 10 weeks
- our app was plagued by regression
- developers felt completely detached from our users

Clearly our traditional strategy of heavily testing each release (up-front quality) wasn't working. So we decided to flip things around: why not try to reduce the cost of errors instead? Hence we decided to move our Philips Hue Android app to continuous deployment.

In this talk you will learn that if releasing is painful, you should actually release even more often. Step by step I'll explain how we moved the Philips Hue Android app to continuous deployment, what our current release process/CI looks like, what changes we made to our overall testing approach, how we empowered developers and testers, and how we made our release process completely painless.

5f57d2d205e77e185986459c1b89a874?s=128

Jeroen Mols

April 23, 2019
Tweet

Transcript

  1. 16.

    @MOLSJEROEN FRUSTRATIONS Issues were found extremely late Developers felt detached

    from our users Constant context switches between releases Releasing very painful Friction between developers and testers
  2. 22.
  3. 25.

    @MOLSJEROEN GOALS Speed up development Get a grip on quality

    Feedback between developers and users
  4. 26.
  5. 27.
  6. 30.

    @MOLSJEROEN WHAT DO WE WANT TO ACHIEVE? DEVELOPMENT TESTING TESTING

    BETA DEVELOPMENT TESTING TESTING BETA DEVELOPMENT TESTING TESTIN
  7. 31.

    @MOLSJEROEN WHAT DO WE WANT TO ACHIEVE? DEVELOPMENT TESTING TESTING

    BETA DEVELOPMENT TESTING TESTING BETA DEVELOPMENT TESTING TESTING BETA
  8. 32.

    @MOLSJEROEN WHAT DO WE WANT TO ACHIEVE? DEVELOPMENT TESTING BETA

    TESTING DEVELOPMENT TESTING BETA TESTING DEVELOPMENT TESTING BETA TESTING
  9. 33.

    @MOLSJEROEN WHAT DO WE WANT TO ACHIEVE? BETA DEVELOPMENT TEST

    TEST BETA DEVELOPMENT TEST TEST BETA DEVELOPMENT TEST TEST
  10. 34.

    @MOLSJEROEN WHAT DO WE WANT TO ACHIEVE? DEV & TEST

    BETA DEV & TEST BETA DEV & TEST BETA
  11. 35.
  12. 44.

    @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY

    MERGE VERIFY develop master release feature 1-5 weeks
  13. 45.

    @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY

    MERGE PROMOTE MASTER VERIFY develop master release feature 1-5 weeks
  14. 46.

    @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY

    MERGE PROMOTE MASTER VERIFY develop master release feature STABILIZE 1-5 weeks
  15. 47.

    @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY

    MERGE PROMOTE MASTER VERIFY develop master release feature STABILIZE 3 weeks 1-5 weeks
  16. 48.

    @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY

    MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY develop master release feature 3 weeks 1-5 weeks
  17. 49.

    @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST

    QUALIFY MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY develop master release feature 3 weeks 1-5 weeks
  18. 50.

    @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST

    QUALIFY MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY develop master release feature 3 weeks 1-5 weeks SHIP
  19. 51.

    @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST

    QUALIFY MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY SHIP develop master release feature MERGE BACK 1-5 weeks 3 weeks
  20. 52.

    @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST

    QUALIFY MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY SHIP develop master release feature MERGE BACK 1-5 weeks 3 weeks 3 weeks
  21. 53.

    @MOLSJEROEN MERGING/INTEGRATING FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST MERGE

    PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY SHIP develop master release feature MERGE BACK 1-5 weeks 3 weeks 3 weeks QUALIFY
  22. 54.

    @MOLSJEROEN MERGING/INTEGRATING FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST MERGE

    PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY SHIP develop master release feature MERGE BACK 1-5 weeks 3 weeks 3 weeks QUALIFY
  23. 62.

    @MOLSJEROEN Large pull request Hard to
 review Long lived
 pull

    request Merge
 conflicts Hard to
 verify
  24. 63.

    @MOLSJEROEN Large pull request Hard to
 review Long lived
 pull

    request Merge
 conflicts Overhead: 
 regression, 
 dependencies,… Hard to
 verify
  25. 64.

    @MOLSJEROEN MERGING/INTEGRATING FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST MERGE

    PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY SHIP MERGE BACK 1-5 weeks 3 weeks 3 weeks QUALIFY FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY SHIP develop master release feature MERGE BACK 1-5 weeks 3 weeks 3 weeks QUALIFY
  26. 66.

    @MOLSJEROEN QUALIFICATION: SPEED UP BUILDS Update all dependencies to latest

    version Follow best practices Modularize Butterknife reflect Gradle build cache and build scans Monitor and keep on experimenting
  27. 68.

    @MOLSJEROEN QUALIFICATION: SPEED UP CI Git hook instead of polling

    Git cache Preload gradle wrapper in docker container Unit tests over instrumentation tests Run subset of tests Parallel tasks
  28. 70.

    @MOLSJEROEN QUALIFICATION: SPEED UP TESTS Legacy “unit tests” written as

    instrumentation Converted 1272 tests Build speed decreased by 3m 18s -9% code coverage Removed CI bottleneck Make room for new Espresso tests
  29. 82.

    "IF YOU MERGE EVERY DAY, SUDDENLY YOU NEVER GET TO

    THE POINT WHERE YOU HAVE HUGE CONFLICTS THAT ARE HARD TO RESOLVE." Linus Torvalds
  30. 83.

    @MOLSJEROEN RECAP: SPEEDING UP INTEGRATION Blazing fast builds A state

    of the art CI Do verification on develop Many small branches => Integration speed up from 1 - 5 weeks to 0,5 - 2days
  31. 84.

    @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST

    QUALIFY MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY SHIP develop master release feature MERGE BACK 0,5-2 days 3 weeks 3 weeks
  32. 91.

    @MOLSJEROEN BUILD VARIANTS buildTypes { debug { signingConfig signingConfigs.debug ext.enableCrashlytics

    = false } release { minifyEnabled true proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro' signingConfig signingConfigs.release } uitest { // provide implementation for idling resources } } flavorDimensions "MODE" productFlavors { daily { applicationId 'com.philips.lighting.hue2.daily' dimension "MODE" buildConfigField "String", "HOCKEY_APP_ID", Keys.hockeyAppDaily buildConfigField "String", "AMPLITUDE_APP_ID", Keys.amplitudeDaily } store { dimension "MODE" buildConfigField "String", "HOCKEY_APP_ID", Keys.hockeyApp buildConfigField "String", "AMPLITUDE_APP_ID", Keys.amplitudeStore } } variantFilter { variant -> def names = variant.flavors*.name if (names.contains("store") && variant.buildType.name == 'debug') { variant.setIgnore(true) } if (names.contains("store") && variant.buildType.name == 'uitest') { variant.setIgnore(true) } }
  33. 92.

    @MOLSJEROEN BUILD VARIANTS Prod API keys Proguard Test settings Idling

    resource s Leak canary Prod firmware Develop
 Portal Release Release test Debug Espresso
  34. 93.

    @MOLSJEROEN IDLING RESOURCES: UI TEST VARIANT object EspressoIdlingResource { private

    val idling = BaseIdlingResource() fun increment(code: String, reason: String) = idling.increment(code, reason) fun decrement(code: String, reason: String) = idling.decrement(code, reason) }
  35. 94.

    @MOLSJEROEN IDLING RESOURCES: DEBUG AND RELEASE VARIANT object EspressoIdlingResource {

    fun increment(code: String, reason: String) { /* NOP */ } fun decrement(code: String, reason: String) { /* NOP */ } }
  36. 95.

    @MOLSJEROEN IDLING RESOURCES: UNIFIED object EspressoIdlingResource { private val idling

    = BaseIdlingResource() fun increment(code: String, reason: String) = if (RuntimeBehavior.isFeatureEnabled(IDLING_RESOURCES)) { idling.increment(code, reason) } fun decrement(code: String, reason: String) = if (RuntimeBehavior.isFeatureEnabled(IDLING_RESOURCES)) { mCountingIdlingResource.decrement(code reason) } }
  37. 96.

    @MOLSJEROEN IDLING RESOURCES: UNIFIED object EspressoIdlingResource { private val idling

    = BaseIdlingResource() fun increment(code: String, reason: String) = if (RuntimeBehavior.isFeatureEnabled(IDLING_RESOURCES)) { idling.increment(code, reason) } fun decrement(code: String, reason: String) = if (RuntimeBehavior.isFeatureEnabled(IDLING_RESOURCES)) { mCountingIdlingResource.decrement(code reason) } }
  38. 97.

    @MOLSJEROEN BUILD VARIANTS Prod API keys Proguard Test settings Idling

    resource s Leak canary Prod firmware Develop
 Portal Release Release test Debug Espresso
  39. 98.

    @MOLSJEROEN BUILD VARIANTS Prod API keys Proguard Test settings Idling

    resource s Leak canary Prod firmware Develop
 Portal Release Release test Debug Espresso
  40. 101.

    @MOLSJEROEN GRADLE CONFIGURATION buildTypes { debug { signingConfig signingConfigs.debug applicationIdSuffix

    '.daily' ext.enableCrashlytics = false buildConfigField "String", "HOCKEY_APP_ID", Keys.hockeyAppDebug buildConfigField "String", "AMPLITUDE_APP_ID", Keys.amplitudeDebug } release { signingConfig signingConfigs.release minifyEnabled true proguardFiles 'proguard-rules.pro' buildConfigField "String", "HOCKEY_APP_ID", Keys.hockeyAppRelease buildConfigField "String", "AMPLITUDE_APP_ID", Keys.amplitudeRelease }
  41. 103.

    @MOLSJEROEN REDUCE COMPLEXITY: SUMMARY Drop unnecessary branches No branches from

    branches Minimize build variants Feature flags for dynamic configuration => Remove all accidental complexity
  42. 104.

    @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST

    QUALIFY MERGE PROMOTE RELEASE STABILIZE VERIFY SHIP master release feature MERGE BACK 0,5-2 days 6 weeks*
  43. 105.

    @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST

    QUALIFY MERGE PROMOTE RELEASE STABILIZE VERIFY RELEASE master release feature MERGE BACK 0,5-2 days ~4 weeks
  44. 107.

    @MOLSJEROEN FEATURE FLAGS: NEW FEATURES override fun onViewCreated(view: View) {

    ... if (RuntimeBehavior.isFeatureEnabled(FeatureFlag.POWER_ON_BEHAVIOR)) { power_on_behavior.visibility = View.VISIBLE } else { power_on_behavior.visibility = View.GONE } ... }
  45. 108.

    @MOLSJEROEN FEATURE FLAGS: NEW FEATURES override fun onViewCreated(view: View) {

    ... if (RuntimeBehavior.isFeatureEnabled(FeatureFlag.POWER_ON_BEHAVIOR)) { power_on_behavior.visibility = View.VISIBLE } else { power_on_behavior.visibility = View.GONE } ... }
  46. 109.

    @MOLSJEROEN FEATURE FLAGS: EXISTING CODE fun withLegacy() { var code

    = LegacyLogic() code.with(OutUnitTests()) // ... code.doMagic() }
  47. 110.

    @MOLSJEROEN FEATURE FLAGS: EXISTING CODE fun withLegacy() { var code

    = LegacyLogic() code.with(OutUnitTests()) // ... code.doMagic() }
  48. 111.

    @MOLSJEROEN FEATURE FLAGS: EXISTING CODE fun withLegacy() { var code

    = NewLogic() code.with(UnitTests()) // ... code.doMagic() }
  49. 112.

    @MOLSJEROEN FEATURE FLAGS: EXISTING CODE fun withLegacy() { if (RuntimeBehavior.isFeatureEnabled(FeatureFlag.LOGIC_REFACTOR))

    { var code = NewLogic() code.with(UnitTests()) // ... code.doMagic() } else { var code = LegacyLogic() code.with(OutUnitTests()) // ... code.doMagic() } }
  50. 119.

    @MOLSJEROEN AUTOMATE TESTING Library Feature Feature Feature Feature Feature Feature

    Feature Feature Feature App Library Library Lib Lib Library Library Lib
  51. 121.
  52. 122.

    @MOLSJEROEN TEST STRATEGY Pull requests: Run all automated level1 app

    Espresso tests Run all automated tests of the modules that are impacted Manual verification if high impact Release: Run all automated Espresso tests Run all automated feature and library tests Deploy: Manual exploratory testing
  53. 123.

    @MOLSJEROEN TEST STRATEGY Pull requests: Run all automated level1 app

    Espresso tests Run all automated tests of the modules that are impacted Manual verification if high impact Release: Run all automated Espresso tests Run all automated feature and library tests Deploy: Manual exploratory testing
  54. 133.

    @MOLSJEROEN IMPACT ANALYSIS SIDE EFFECTS POSSIBLE IN 
 OTHER PARTS

    OF CODE? (REGRESSION) ALL SIDE EFFECTS CONTAINED BEHIND A FEATURE FLAG? LOW IMPACT LOW IMPACT HIGH IMPACT Yes Yes No Not sure Not sure No
  55. 135.

    @MOLSJEROEN STABILIZE MASTER: SUMMARY Use feature toggles Static code analysis

    Heavily automate testing Merge master before checks Requalify pull requests after each merge Impact assessment => Able to drop release branch
  56. 136.

    @MOLSJEROEN DEVELOPMENT PROCESS 0,5-2 days 2 weeks FEATURE BRANCH FEATURE

    FLAG DEVELOP PULL REQUEST MERGE QUALIFY
 ++ VERIFY IMPACT master feature BETA MANUAL RELEASE SHIP
  57. 137.

    @MOLSJEROEN DEVELOPMENT PROCESS 0,5-2 days 2 weeks FEATURE BRANCH FEATURE

    FLAG DEVELOP PULL REQUEST MERGE QUALIFY
 ++ VERIFY IMPACT master feature BETA MANUAL RELEASE SHIP
  58. 143.

    @MOLSJEROEN RELEASE APP (DELIVERY) stage('Deploy to Google Play Internal Test

    Track') { steps { script { def releaseNotes = readFile 'app/build/release_notes.txt' androidApkUpload( googleCredentialsId: 'Google Play Android Developer', trackName: 'internal', apkFilesPattern: '**/*-release.apk', deobfuscationFilesPattern: '**/mapping.txt', recentChangeList: [[language: 'en-US', text: releaseNotes]] ) } } }
  59. 144.

    @MOLSJEROEN RELEASE APP (DELIVERY) stage('Deploy to HockeyApp') { steps {

    sh "pipelines/scripts/hockeyapp/deploy_app_debug.sh" } }
  60. 155.

    @MOLSJEROEN AUTOMATE DELIVERY: SUMMARY Integrate dependencies automatically Automate upload to

    Google play/hockey app Release directly from master Hotfixes from separate branch
  61. 156.

    @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH FEATURE FLAG DEVELOP PULL REQUEST

    MERGE QUALIFY
 ++ master RELEASE VERIFY feature IMPACT 0,5-2 days 2 hours BETA SHIP 1 week*
  62. 159.
  63. 162.

    @MOLSJEROEN 1. BI-WEEKLY RELEASES DEV & TEST BETA DEV &

    TEST BETA DEV & TEST BETA 2 weeks DEV & TEST BETA
  64. 170.

    @MOLSJEROEN SUMMARY Focus on whitebox testing Gain trust and give

    trust back Don’t blow your trust! Rely on automated testing Closer collaboration between testers and developers Root cause analyses
  65. 171.
  66. 178.
  67. 179.
  68. 181.

    @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH FEATURE FLAG DEVELOP PULL REQUEST

    MERGE QUALIFY
 ++ master RELEASE VERIFY feature IMPACT 0,5-2 days 2 hours BETA SHIP 1 week*
  69. 184.
  70. 189.