Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Journey to painless releases: Continuous delivery for Philips Hue Android

Journey to painless releases: Continuous delivery for Philips Hue Android

Video: https://www.youtube.com/watch?v=VkFcJZWtufA

Wind back one year ago:
- releasing took us 10 weeks
- our app was plagued by regression
- developers felt completely detached from our users

Clearly our traditional strategy of heavily testing each release (up-front quality) wasn't working. So we decided to flip things around: why not try to reduce the cost of errors instead? Hence we decided to move our Philips Hue Android app to continuous deployment.

In this talk you will learn that if releasing is painful, you should actually release even more often. Step by step I'll explain how we moved the Philips Hue Android app to continuous deployment, what our current release process/CI looks like, what changes we made to our overall testing approach, how we empowered developers and testers, and how we made our release process completely painless.

Jeroen Mols

April 23, 2019
Tweet

More Decks by Jeroen Mols

Other Decks in Programming

Transcript

  1. @MOLSJEROEN
    JOURNEY TO
    PAINLESS RELEASES
    CONTINUOUS DELIVERY FOR PHILIPS HUE

    View full-size slide

  2. @MOLSJEROEN
    @MOLSJEROEN

    View full-size slide

  3. @MOLSJEROEN
    CHALLENGES
    CONTINUOUS DELIVERY
    MIGRATE YOUR PROJECT
    RESULTS

    View full-size slide

  4. @MOLSJEROEN
    RELEASE PROCESS
    DEVELOPMENT

    View full-size slide

  5. @MOLSJEROEN
    RELEASE PROCESS
    DEVELOPMENT TESTING

    View full-size slide

  6. @MOLSJEROEN
    RELEASE PROCESS
    DEVELOPMENT TESTING TESTING

    View full-size slide

  7. @MOLSJEROEN
    RELEASE PROCESS
    DEVELOPMENT TESTING TESTING BETA

    View full-size slide

  8. @MOLSJEROEN
    RELEASE PROCESS
    DEVELOPMENT TESTING TESTING BETA
    10 weeks

    View full-size slide

  9. @MOLSJEROEN
    RELEASE PROCESS
    DEVELOPMENT TESTING TESTING BETA
    DEVELOPMENT TESTING TESTING BETA
    10 weeks

    View full-size slide

  10. @MOLSJEROEN
    RELEASE PROCESS
    DEVELOPMENT TESTING TESTING BETA
    DEVELOPMENT TESTING TESTING BETA
    DEVELOPMENT TESTING TESTIN
    10 weeks

    View full-size slide

  11. @MOLSJEROEN
    RELEASE PROCESS
    DEVELOPMENT TESTING TESTING BETA
    DEVELOPMENT TESTING TESTING BETA
    DEVELOPMENT TESTING TESTIN
    10 weeks
    3 releases

    View full-size slide

  12. @MOLSJEROEN
    RELEASE PROCESS
    DEVELOPMENT TESTING TESTING BETA
    10 weeks

    View full-size slide

  13. @MOLSJEROEN
    RELEASE PROCESS
    DEVELOPMENT TESTING TESTING BETA
    10 weeks

    View full-size slide

  14. @MOLSJEROEN
    RELEASE PROCESS
    DEVELOPMENT TESTING TESTING BETA
    10 weeks

    View full-size slide

  15. @MOLSJEROEN
    FRUSTRATIONS
    Issues were found extremely late
    Developers felt detached from our users
    Constant context switches between releases
    Releasing very painful
    Friction between developers and testers

    View full-size slide

  16. @MOLSJEROEN
    BRANCHING MODEL
    develop
    master
    release
    feature bugfix
    feature
    hotfix

    View full-size slide

  17. @MOLSJEROEN
    hotfix
    feature bugfix
    BRANCHING MODEL
    master
    release
    feature
    develop

    View full-size slide

  18. @MOLSJEROEN
    BRANCHING MODEL
    develop
    master
    release
    feature bugfix
    feature
    hotfix

    View full-size slide

  19. @MOLSJEROEN
    BRANCHING MODEL
    develop
    master
    release
    feature bugfix
    feature
    hotfix

    View full-size slide

  20. @MOLSJEROEN
    BRANCHING MODEL
    develop
    master
    release
    feature bugfix
    feature
    hotfix

    View full-size slide

  21. @MOLSJEROEN
    FRUSTRATIONS
    Unstable development branch
    Merging nightmares
    Poor code reviews
    Confusion about what fix is where
    Development was slow

    View full-size slide

  22. @MOLSJEROEN
    RESULTS

    View full-size slide

  23. STRATEGY OF UPFRONT TESTING
    NOT WORKING

    View full-size slide

  24. @MOLSJEROEN
    GOALS
    Speed up development
    Get a grip on quality
    Feedback between developers and users

    View full-size slide

  25. CONTINUOUS DELIVERY

    View full-size slide

  26. IF IT HURTS, DO IT MORE OFTEN
    Jez Humble

    View full-size slide

  27. @MOLSJEROEN
    WHAT DO WE WANT TO ACHIEVE?
    DEVELOPMENT TESTING TESTING BETA
    DEVELOPMENT TESTING TESTING BETA
    DEVELOPMENT TESTING TESTIN

    View full-size slide

  28. @MOLSJEROEN
    WHAT DO WE WANT TO ACHIEVE?
    DEVELOPMENT TESTING TESTING BETA
    DEVELOPMENT TESTING TESTING BETA
    DEVELOPMENT TESTING TESTING BETA

    View full-size slide

  29. @MOLSJEROEN
    WHAT DO WE WANT TO ACHIEVE?
    DEVELOPMENT TESTING BETA
    TESTING
    DEVELOPMENT TESTING BETA
    TESTING
    DEVELOPMENT TESTING BETA
    TESTING

    View full-size slide

  30. @MOLSJEROEN
    WHAT DO WE WANT TO ACHIEVE?
    BETA
    DEVELOPMENT TEST TEST
    BETA
    DEVELOPMENT TEST TEST
    BETA
    DEVELOPMENT TEST TEST

    View full-size slide

  31. @MOLSJEROEN
    WHAT DO WE WANT TO ACHIEVE?
    DEV & TEST BETA
    DEV & TEST BETA
    DEV & TEST BETA

    View full-size slide

  32. IF IT HURTS, DO IT MORE OFTEN
    AND BRING THE PAIN FORWARD
    Jez Humble

    View full-size slide

  33. @MOLSJEROEN
    TERMINOLOGY
    Continuous integration
    Continuous delivery
    Continuous deployment

    View full-size slide

  34. DEVELOPMENT PROCESS

    View full-size slide

  35. @MOLSJEROEN
    DEVELOPMENT PROCESS
    develop
    master
    release

    View full-size slide

  36. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    develop
    master
    release
    feature

    View full-size slide

  37. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    DEVELOP
    MENT
    PULL
    REQUEST
    develop
    master
    release
    feature

    View full-size slide

  38. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    DEVELOP
    MENT
    PULL
    REQUEST QUALIFY
    develop
    master
    release
    feature

    View full-size slide

  39. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    DEVELOP
    MENT
    PULL
    REQUEST QUALIFY
    develop
    master
    release
    feature
    VERIFY

    View full-size slide

  40. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    DEVELOP
    MENT
    PULL
    REQUEST QUALIFY MERGE
    VERIFY
    develop
    master
    release
    feature

    View full-size slide

  41. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    DEVELOP
    MENT
    PULL
    REQUEST QUALIFY MERGE
    VERIFY
    develop
    master
    release
    feature
    1-5 weeks

    View full-size slide

  42. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    DEVELOP
    MENT
    PULL
    REQUEST QUALIFY MERGE PROMOTE
    MASTER
    VERIFY
    develop
    master
    release
    feature
    1-5 weeks

    View full-size slide

  43. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    DEVELOP
    MENT
    PULL
    REQUEST QUALIFY MERGE PROMOTE
    MASTER
    VERIFY
    develop
    master
    release
    feature
    STABILIZE
    1-5 weeks

    View full-size slide

  44. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    DEVELOP
    MENT
    PULL
    REQUEST QUALIFY MERGE PROMOTE
    MASTER
    VERIFY
    develop
    master
    release
    feature
    STABILIZE
    3 weeks
    1-5 weeks

    View full-size slide

  45. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    DEVELOP
    MENT
    PULL
    REQUEST QUALIFY MERGE PROMOTE
    RELEASE
    PROMOTE
    MASTER STABILIZE
    VERIFY
    develop
    master
    release
    feature
    3 weeks
    1-5 weeks

    View full-size slide

  46. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    DEVELOP
    MENT BETA
    PULL
    REQUEST QUALIFY MERGE PROMOTE
    RELEASE
    PROMOTE
    MASTER STABILIZE
    VERIFY
    develop
    master
    release
    feature
    3 weeks
    1-5 weeks

    View full-size slide

  47. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    DEVELOP
    MENT BETA
    PULL
    REQUEST QUALIFY MERGE PROMOTE
    RELEASE
    PROMOTE
    MASTER STABILIZE
    VERIFY
    develop
    master
    release
    feature
    3 weeks
    1-5 weeks
    SHIP

    View full-size slide

  48. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    DEVELOP
    MENT BETA
    PULL
    REQUEST QUALIFY MERGE PROMOTE
    RELEASE
    PROMOTE
    MASTER STABILIZE
    VERIFY SHIP
    develop
    master
    release
    feature
    MERGE
    BACK
    1-5 weeks 3 weeks

    View full-size slide

  49. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    DEVELOP
    MENT BETA
    PULL
    REQUEST QUALIFY MERGE PROMOTE
    RELEASE
    PROMOTE
    MASTER STABILIZE
    VERIFY SHIP
    develop
    master
    release
    feature
    MERGE
    BACK
    1-5 weeks 3 weeks 3 weeks

    View full-size slide

  50. @MOLSJEROEN
    MERGING/INTEGRATING
    FEATURE
    BRANCH
    DEVELOP
    MENT BETA
    PULL
    REQUEST MERGE PROMOTE
    RELEASE
    PROMOTE
    MASTER STABILIZE
    VERIFY SHIP
    develop
    master
    release
    feature
    MERGE
    BACK
    1-5 weeks 3 weeks 3 weeks
    QUALIFY

    View full-size slide

  51. @MOLSJEROEN
    MERGING/INTEGRATING
    FEATURE
    BRANCH
    DEVELOP
    MENT BETA
    PULL
    REQUEST MERGE PROMOTE
    RELEASE
    PROMOTE
    MASTER STABILIZE
    VERIFY SHIP
    develop
    master
    release
    feature
    MERGE
    BACK
    1-5 weeks 3 weeks 3 weeks
    QUALIFY

    View full-size slide

  52. SPEED UP INTEGRATION

    View full-size slide

  53. @MOLSJEROEN
    Large pull request

    View full-size slide

  54. @MOLSJEROEN
    Large pull request
    Hard to

    review

    View full-size slide

  55. @MOLSJEROEN
    Large pull request
    Hard to

    review
    Hard to

    verify

    View full-size slide

  56. @MOLSJEROEN
    Large pull request
    Hard to

    review
    Long lived

    pull request
    Hard to

    verify

    View full-size slide

  57. @MOLSJEROEN
    Large pull request
    Hard to

    review
    Long lived

    pull request
    Merge

    conflicts
    Hard to

    verify

    View full-size slide

  58. @MOLSJEROEN
    Large pull request
    Hard to

    review
    Long lived

    pull request
    Merge

    conflicts
    Overhead: 

    regression, 

    dependencies,…
    Hard to

    verify

    View full-size slide

  59. @MOLSJEROEN
    MERGING/INTEGRATING
    FEATURE
    BRANCH
    DEVELOP
    MENT BETA
    PULL
    REQUEST MERGE PROMOTE
    RELEASE
    PROMOTE
    MASTER STABILIZE
    VERIFY SHIP MERGE
    BACK
    1-5 weeks 3 weeks 3 weeks
    QUALIFY
    FEATURE
    BRANCH
    DEVELOP
    MENT BETA
    PULL
    REQUEST MERGE PROMOTE
    RELEASE
    PROMOTE
    MASTER STABILIZE
    VERIFY SHIP
    develop
    master
    release
    feature
    MERGE
    BACK
    1-5 weeks 3 weeks 3 weeks
    QUALIFY

    View full-size slide

  60. @MOLSJEROEN
    PULL REQUEST QUALIFICATION
    Develop
    feature BUILD

    TEST
    REVIEW

    View full-size slide

  61. @MOLSJEROEN
    QUALIFICATION: SPEED UP BUILDS
    Update all dependencies to latest version
    Follow best practices
    Modularize
    Butterknife reflect
    Gradle build cache and build scans
    Monitor and keep on experimenting

    View full-size slide

  62. @MOLSJEROEN
    QUALIFICATION: SPEED UP BUILDS
    3m 45s
    1m 26s

    View full-size slide

  63. @MOLSJEROEN
    QUALIFICATION: SPEED UP CI
    Git hook instead of polling
    Git cache
    Preload gradle wrapper in docker container
    Unit tests over instrumentation tests
    Run subset of tests
    Parallel tasks

    View full-size slide

  64. HOW MUCH TIME ARE YOU WILLING TO
    INVEST IN QUALIFYING PULL REQUESTS?

    View full-size slide

  65. @MOLSJEROEN
    QUALIFICATION: SPEED UP TESTS
    Legacy “unit tests” written as instrumentation
    Converted 1272 tests
    Build speed decreased by 3m 18s
    -9% code coverage
    Removed CI bottleneck
    Make room for new Espresso tests

    View full-size slide

  66. @MOLSJEROEN
    QUALIFICATION: SPEED UP BUILDS
    ~20 min
    5m 10s

    View full-size slide

  67. @MOLSJEROEN
    QUALIFICATION: SPEED UP TESTS
    Instrumentation tests
    Unit tests

    View full-size slide

  68. @MOLSJEROEN
    QUALIFICATION: SPEED UP TESTS
    Instrumentation tests
    Unit tests

    View full-size slide

  69. @MOLSJEROEN
    QUALIFICATION: SPEED UP TESTS

    Instrumentation tests
    Unit tests

    View full-size slide

  70. @MOLSJEROEN
    QUALIFICATION: SPEED UP BUILDS (10 PRS)
    ~1 hour
    5m 10s

    View full-size slide

  71. @MOLSJEROEN
    VERIFICATION
    Develop
    feature

    View full-size slide

  72. @MOLSJEROEN
    VERIFICATION: MOVE TO MASTER
    Develop
    feature

    View full-size slide

  73. @MOLSJEROEN
    VERIFICATION: MOVE TO MASTER
    1-3 days
    0 days

    View full-size slide

  74. @MOLSJEROEN
    DEVELOPMENT
    Develop
    feature

    View full-size slide

  75. @MOLSJEROEN
    DEVELOPMENT: SMALLER BRANCHES
    feature
    Develop

    View full-size slide

  76. @MOLSJEROEN
    DEVELOPMENT: SMALLER BRANCHES
    1-5 weeks
    0,5 - 2 days

    View full-size slide

  77. "IF YOU MERGE EVERY DAY, SUDDENLY
    YOU NEVER GET TO THE POINT WHERE
    YOU HAVE HUGE CONFLICTS THAT ARE
    HARD TO RESOLVE."
    Linus Torvalds

    View full-size slide

  78. @MOLSJEROEN
    RECAP: SPEEDING UP INTEGRATION
    Blazing fast builds
    A state of the art CI
    Do verification on develop
    Many small branches
    => Integration speed up from 1 - 5 weeks to 0,5 - 2days

    View full-size slide

  79. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    DEVELOP
    MENT BETA
    PULL
    REQUEST QUALIFY MERGE PROMOTE
    RELEASE
    PROMOTE
    MASTER STABILIZE
    VERIFY SHIP
    develop
    master
    release
    feature
    MERGE
    BACK
    0,5-2 days 3 weeks 3 weeks

    View full-size slide

  80. REDUCE COMPLEXITY

    View full-size slide

  81. @MOLSJEROEN
    FEATURE BRANCHES
    develop
    master
    release
    feature bugfix
    feature
    hotfix

    View full-size slide

  82. @MOLSJEROEN
    FEATURE BRANCHES: ONLY FROM DEVELOP
    develop
    master
    release
    feature bugfix
    hotfix

    View full-size slide

  83. @MOLSJEROEN
    DEVELOP BRANCH
    develop
    master
    release
    feature bugfix
    hotfix

    View full-size slide

  84. @MOLSJEROEN
    DEVELOP BRANCH
    master
    release
    feature bugfix
    hotfix

    View full-size slide

  85. ACCIDENTAL VS ESSENTIAL COMPLEXITY

    View full-size slide

  86. @MOLSJEROEN
    BUILD VARIANTS
    buildTypes {
    debug {
    signingConfig signingConfigs.debug
    ext.enableCrashlytics = false
    }
    release {
    minifyEnabled true
    proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
    signingConfig signingConfigs.release
    }
    uitest {
    // provide implementation for idling resources
    }
    }
    flavorDimensions "MODE"
    productFlavors {
    daily {
    applicationId 'com.philips.lighting.hue2.daily'
    dimension "MODE"
    buildConfigField "String", "HOCKEY_APP_ID", Keys.hockeyAppDaily
    buildConfigField "String", "AMPLITUDE_APP_ID", Keys.amplitudeDaily
    }
    store {
    dimension "MODE"
    buildConfigField "String", "HOCKEY_APP_ID", Keys.hockeyApp
    buildConfigField "String", "AMPLITUDE_APP_ID", Keys.amplitudeStore
    }
    }
    variantFilter { variant ->
    def names = variant.flavors*.name
    if (names.contains("store") && variant.buildType.name == 'debug') {
    variant.setIgnore(true)
    }
    if (names.contains("store") && variant.buildType.name == 'uitest') {
    variant.setIgnore(true)
    }
    }

    View full-size slide

  87. @MOLSJEROEN
    BUILD VARIANTS
    Prod API
    keys
    Proguard Test
    settings
    Idling
    resource
    s
    Leak
    canary
    Prod
    firmware
    Develop

    Portal
    Release
    Release test
    Debug
    Espresso

    View full-size slide

  88. @MOLSJEROEN
    IDLING RESOURCES: UI TEST VARIANT
    object EspressoIdlingResource {
    private val idling = BaseIdlingResource()
    fun increment(code: String, reason: String) = idling.increment(code, reason)
    fun decrement(code: String, reason: String) = idling.decrement(code, reason)
    }

    View full-size slide

  89. @MOLSJEROEN
    IDLING RESOURCES: DEBUG AND RELEASE VARIANT
    object EspressoIdlingResource {
    fun increment(code: String, reason: String) { /* NOP */ }
    fun decrement(code: String, reason: String) { /* NOP */ }
    }

    View full-size slide

  90. @MOLSJEROEN
    IDLING RESOURCES: UNIFIED
    object EspressoIdlingResource {
    private val idling = BaseIdlingResource()
    fun increment(code: String, reason: String) =
    if (RuntimeBehavior.isFeatureEnabled(IDLING_RESOURCES)) {
    idling.increment(code, reason)
    }
    fun decrement(code: String, reason: String) =
    if (RuntimeBehavior.isFeatureEnabled(IDLING_RESOURCES)) {
    mCountingIdlingResource.decrement(code reason)
    }
    }

    View full-size slide

  91. @MOLSJEROEN
    IDLING RESOURCES: UNIFIED
    object EspressoIdlingResource {
    private val idling = BaseIdlingResource()
    fun increment(code: String, reason: String) =
    if (RuntimeBehavior.isFeatureEnabled(IDLING_RESOURCES)) {
    idling.increment(code, reason)
    }
    fun decrement(code: String, reason: String) =
    if (RuntimeBehavior.isFeatureEnabled(IDLING_RESOURCES)) {
    mCountingIdlingResource.decrement(code reason)
    }
    }

    View full-size slide

  92. @MOLSJEROEN
    BUILD VARIANTS
    Prod API
    keys
    Proguard Test
    settings
    Idling
    resource
    s
    Leak
    canary
    Prod
    firmware
    Develop

    Portal
    Release
    Release test
    Debug
    Espresso

    View full-size slide

  93. @MOLSJEROEN
    BUILD VARIANTS
    Prod API
    keys
    Proguard Test
    settings
    Idling
    resource
    s
    Leak
    canary
    Prod
    firmware
    Develop

    Portal
    Release
    Release test
    Debug
    Espresso

    View full-size slide

  94. @MOLSJEROEN
    RELEASE AND DEBUG

    View full-size slide

  95. @MOLSJEROEN
    SOURCE FOLDERS

    View full-size slide

  96. @MOLSJEROEN
    GRADLE CONFIGURATION
    buildTypes {
    debug {
    signingConfig signingConfigs.debug
    applicationIdSuffix '.daily'
    ext.enableCrashlytics = false
    buildConfigField "String", "HOCKEY_APP_ID", Keys.hockeyAppDebug
    buildConfigField "String", "AMPLITUDE_APP_ID", Keys.amplitudeDebug
    }
    release {
    signingConfig signingConfigs.release
    minifyEnabled true
    proguardFiles 'proguard-rules.pro'
    buildConfigField "String", "HOCKEY_APP_ID", Keys.hockeyAppRelease
    buildConfigField "String", "AMPLITUDE_APP_ID", Keys.amplitudeRelease
    }

    View full-size slide

  97. SIMPLICITY IS THE SOUL OF EFFICIENCY
    Austin Freeman

    View full-size slide

  98. @MOLSJEROEN
    REDUCE COMPLEXITY: SUMMARY
    Drop unnecessary branches
    No branches from branches
    Minimize build variants
    Feature flags for dynamic configuration
    => Remove all accidental complexity

    View full-size slide

  99. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    DEVELOP
    MENT BETA
    PULL
    REQUEST QUALIFY MERGE PROMOTE
    RELEASE STABILIZE
    VERIFY SHIP
    master
    release
    feature
    MERGE
    BACK
    0,5-2 days 6 weeks*

    View full-size slide

  100. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    DEVELOP
    MENT BETA
    PULL
    REQUEST QUALIFY MERGE PROMOTE
    RELEASE STABILIZE
    VERIFY RELEASE
    master
    release
    feature
    MERGE
    BACK
    0,5-2 days ~4 weeks

    View full-size slide

  101. STABILIZE MASTER

    View full-size slide

  102. @MOLSJEROEN
    FEATURE FLAGS: NEW FEATURES
    override fun onViewCreated(view: View) {
    ...
    if (RuntimeBehavior.isFeatureEnabled(FeatureFlag.POWER_ON_BEHAVIOR)) {
    power_on_behavior.visibility = View.VISIBLE
    } else {
    power_on_behavior.visibility = View.GONE
    }
    ...
    }

    View full-size slide

  103. @MOLSJEROEN
    FEATURE FLAGS: NEW FEATURES
    override fun onViewCreated(view: View) {
    ...
    if (RuntimeBehavior.isFeatureEnabled(FeatureFlag.POWER_ON_BEHAVIOR)) {
    power_on_behavior.visibility = View.VISIBLE
    } else {
    power_on_behavior.visibility = View.GONE
    }
    ...
    }

    View full-size slide

  104. @MOLSJEROEN
    FEATURE FLAGS: EXISTING CODE
    fun withLegacy() {
    var code = LegacyLogic()
    code.with(OutUnitTests())
    // ...
    code.doMagic()
    }

    View full-size slide

  105. @MOLSJEROEN
    FEATURE FLAGS: EXISTING CODE
    fun withLegacy() {
    var code = LegacyLogic()
    code.with(OutUnitTests())
    // ...
    code.doMagic()
    }

    View full-size slide

  106. @MOLSJEROEN
    FEATURE FLAGS: EXISTING CODE
    fun withLegacy() {
    var code = NewLogic()
    code.with(UnitTests())
    // ...
    code.doMagic()
    }

    View full-size slide

  107. @MOLSJEROEN
    FEATURE FLAGS: EXISTING CODE
    fun withLegacy() {
    if (RuntimeBehavior.isFeatureEnabled(FeatureFlag.LOGIC_REFACTOR)) {
    var code = NewLogic()
    code.with(UnitTests())
    // ...
    code.doMagic()
    } else {
    var code = LegacyLogic()
    code.with(OutUnitTests())
    // ...
    code.doMagic()
    }
    }

    View full-size slide

  108. @MOLSJEROEN
    QUALIFY PULL REQUESTS
    master
    feature

    View full-size slide

  109. @MOLSJEROEN
    QUALIFY PULL REQUESTS
    master
    feature

    BUILD

    REVIEW


    View full-size slide

  110. @MOLSJEROEN
    QUALIFY PULL REQUESTS
    master
    feature

    BUILD

    REVIEW
    ANALYZE


    View full-size slide

  111. @MOLSJEROEN
    QUALIFY PULL REQUESTS
    master
    feature

    BUILD

    REVIEW

    ANALYZE

    TEST

    View full-size slide

  112. @MOLSJEROEN
    AUTOMATE TESTING
    UNIT
    INTEGRATION
    E2E

    View full-size slide

  113. @MOLSJEROEN
    AUTOMATE TESTING
    Library
    Feature
    Feature
    Feature
    Feature
    Feature
    Feature
    Feature
    Feature
    Feature
    App
    Library
    Library
    Lib Lib Library
    Library
    Lib

    View full-size slide

  114. @MOLSJEROEN
    TEST STRATEGY
    Pull requests:
    Run all automated level1 app Espresso tests
    Run all automated tests of the modules that are impacted
    Manual verification if high impact
    Release:
    Run all automated Espresso tests
    Run all automated feature and library tests
    Deploy: Manual exploratory testing

    View full-size slide

  115. @MOLSJEROEN
    TEST STRATEGY
    Pull requests:
    Run all automated level1 app Espresso tests
    Run all automated tests of the modules that are impacted
    Manual verification if high impact
    Release:
    Run all automated Espresso tests
    Run all automated feature and library tests
    Deploy: Manual exploratory testing

    View full-size slide

  116. @MOLSJEROEN
    SIMULATE ACTUAL MERGE
    master
    feature BUILD

    REVIEW

    ANALYZE

    TEST

    View full-size slide

  117. @MOLSJEROEN
    SIMULATE ACTUAL MERGE
    master
    feature

    MERGE
    BUILD

    REVIEW

    ANALYZE

    TEST

    View full-size slide

  118. @MOLSJEROEN
    master
    feature
    SIMULATE ACTUAL MERGE

    View full-size slide

  119. @MOLSJEROEN
    master
    feature
    SIMULATE ACTUAL MERGE

    View full-size slide

  120. @MOLSJEROEN
    master
    feature
    feature
    feature
    SIMULATE ACTUAL MERGE

    View full-size slide

  121. @MOLSJEROEN
    master
    feature
    feature
    feature
    SIMULATE ACTUAL MERGE

    View full-size slide

  122. @MOLSJEROEN
    master
    feature
    feature
    feature
    SIMULATE ACTUAL MERGE

    View full-size slide

  123. @MOLSJEROEN
    MERGE PULL REQUESTS
    master
    feature

    MERGE
    BUILD

    REVIEW

    ANALYZE

    TEST

    View full-size slide

  124. @MOLSJEROEN
    MERGE PULL REQUESTS
    master
    feature

    MERGE
    BUILD

    REVIEW

    ANALYZE

    TEST
    IMPACT


    View full-size slide

  125. @MOLSJEROEN
    IMPACT ANALYSIS
    SIDE EFFECTS POSSIBLE IN 

    OTHER PARTS OF CODE? (REGRESSION)
    ALL SIDE EFFECTS CONTAINED
    BEHIND A FEATURE FLAG?
    LOW IMPACT
    LOW IMPACT
    HIGH IMPACT
    Yes
    Yes
    No
    Not sure
    Not sure
    No

    View full-size slide

  126. @MOLSJEROEN
    IMPACT ANALYSIS

    View full-size slide

  127. @MOLSJEROEN
    STABILIZE MASTER: SUMMARY
    Use feature toggles
    Static code analysis
    Heavily automate testing
    Merge master before checks
    Requalify pull requests after each merge
    Impact assessment
    => Able to drop release branch

    View full-size slide

  128. @MOLSJEROEN
    DEVELOPMENT PROCESS
    0,5-2 days 2 weeks
    FEATURE
    BRANCH
    FEATURE
    FLAG DEVELOP PULL
    REQUEST MERGE
    QUALIFY

    ++ VERIFY
    IMPACT
    master
    feature
    BETA
    MANUAL
    RELEASE SHIP

    View full-size slide

  129. @MOLSJEROEN
    DEVELOPMENT PROCESS
    0,5-2 days 2 weeks
    FEATURE
    BRANCH
    FEATURE
    FLAG DEVELOP PULL
    REQUEST MERGE
    QUALIFY

    ++ VERIFY
    IMPACT
    master
    feature
    BETA
    MANUAL
    RELEASE SHIP

    View full-size slide

  130. AUTOMATE DELIVERY

    View full-size slide

  131. @MOLSJEROEN
    UPDATE TRANSLATIONS
    master
    feature

    View full-size slide

  132. @MOLSJEROEN
    UPDATE TRANSLATIONS
    master
    feature
    translations

    View full-size slide

  133. @MOLSJEROEN
    UPDATE TRANSLATIONS
    master
    feature
    translations

    View full-size slide

  134. @MOLSJEROEN
    UPDATE SDK AND ASSETS
    master
    feature
    translations SDK assets

    View full-size slide

  135. @MOLSJEROEN
    RELEASE APP (DELIVERY)
    stage('Deploy to Google Play Internal Test Track') {
    steps {
    script {
    def releaseNotes = readFile 'app/build/release_notes.txt'
    androidApkUpload(
    googleCredentialsId: 'Google Play Android Developer',
    trackName: 'internal',
    apkFilesPattern: '**/*-release.apk',
    deobfuscationFilesPattern: '**/mapping.txt',
    recentChangeList: [[language: 'en-US', text: releaseNotes]]
    )
    }
    }
    }

    View full-size slide

  136. @MOLSJEROEN
    RELEASE APP (DELIVERY)
    stage('Deploy to HockeyApp') {
    steps {
    sh "pipelines/scripts/hockeyapp/deploy_app_debug.sh"
    }
    }

    View full-size slide

  137. @MOLSJEROEN
    IDEAL SCENARIO
    master
    feature bugfix

    View full-size slide

  138. @MOLSJEROEN
    IDEAL SCENARIO
    master
    feature bugfix
    hotfix

    View full-size slide

  139. @MOLSJEROEN
    IDEAL SCENARIO
    master
    feature bugfix
    hotfix

    View full-size slide

  140. @MOLSJEROEN
    IDEAL SCENARIO
    master
    feature bugfix
    hotfix

    View full-size slide

  141. @MOLSJEROEN
    ACTUAL SCENARIO
    master
    feature bugfix
    hotfix

    View full-size slide

  142. @MOLSJEROEN
    ACTUAL SCENARIO
    master
    feature bugfix
    hotfix

    View full-size slide

  143. @MOLSJEROEN
    ACTUAL SCENARIO
    master
    feature bugfix
    hotfix

    View full-size slide

  144. @MOLSJEROEN
    ACTUAL SCENARIO
    master
    feature bugfix
    hotfix

    View full-size slide

  145. @MOLSJEROEN
    RELEASING
    master
    internal test
    beta
    production

    View full-size slide

  146. @MOLSJEROEN
    RELEASING
    master
    internal test
    beta
    production

    View full-size slide

  147. @MOLSJEROEN
    AUTOMATE DELIVERY: SUMMARY
    Integrate dependencies automatically
    Automate upload to Google play/hockey app
    Release directly from master
    Hotfixes from separate branch

    View full-size slide

  148. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    FEATURE
    FLAG DEVELOP PULL
    REQUEST MERGE
    QUALIFY

    ++
    master
    RELEASE VERIFY
    feature
    IMPACT
    0,5-2 days 2 hours
    BETA SHIP
    1 week*

    View full-size slide

  149. LESSONS LEARNED

    View full-size slide

  150. @MOLSJEROEN
    1. BI-WEEKLY RELEASES
    DEVELOPMENT TESTING TESTING BETA
    DEVELOPMENT TESTING TESTING BETA
    DEVELOPMENT TESTING TESTIN
    10 weeks

    View full-size slide

  151. @MOLSJEROEN
    1. BI-WEEKLY RELEASES
    DEV & TEST BETA
    DEV & TEST BETA
    DEV & TEST BETA
    2 weeks

    View full-size slide

  152. @MOLSJEROEN
    1. BI-WEEKLY RELEASES
    BETA
    DEVELOPMENT TEST TEST
    BETA
    DEVELOPMENT TEST TEST
    BETA
    DEVELOPMENT TEST TEST

    View full-size slide

  153. @MOLSJEROEN
    1. BI-WEEKLY RELEASES
    BETA
    DEVELOPMENT TEST
    TEST
    BETA
    DEVELOPMENT TEST
    TEST
    BETA
    DEVELOPMENT TEST
    TEST

    View full-size slide

  154. @MOLSJEROEN
    1. BI-WEEKLY RELEASES
    DEV & TEST BETA
    DEV & TEST BETA
    DEV & TEST BETA
    2 weeks
    DEV & TEST BETA

    View full-size slide

  155. @MOLSJEROEN
    2. MERGING WITHOUT VERIFICATION
    master
    feature

    MERGE
    BUILD

    REVIEW

    ANALYZE

    TEST
    IMPACT


    View full-size slide

  156. @MOLSJEROEN
    2. MERGING WITHOUT VERIFICATION

    View full-size slide

  157. @MOLSJEROEN
    2. MERGING WITHOUT VERIFICATION

    View full-size slide

  158. TRUST TAKES YEARS TO BUILD,
    SECONDS TO BREAK,
    AND FOREVER TO REPAIR.
    Unknown

    View full-size slide

  159. @MOLSJEROEN
    2. MERGING WITHOUT VERIFICATION

    View full-size slide

  160. @MOLSJEROEN
    2. MERGING WITHOUT VERIFICATION

    View full-size slide

  161. @MOLSJEROEN
    3. YOU WILL MESS UP

    View full-size slide

  162. @MOLSJEROEN
    SUMMARY
    Focus on whitebox testing
    Gain trust and give trust back
    Don’t blow your trust!
    Rely on automated testing
    Closer collaboration between testers and developers
    Root cause analyses

    View full-size slide

  163. @MOLSJEROEN
    RESULTS
    29

    View full-size slide

  164. @MOLSJEROEN
    TIME TO RELEASE
    Build time (min)
    0
    10
    20
    30
    40
    50
    Build number

    View full-size slide

  165. @MOLSJEROEN
    INTEGRATION SPEED
    620 ~1/dev/day

    View full-size slide

  166. @MOLSJEROEN
    APP RATING

    View full-size slide

  167. @MOLSJEROEN
    PR INFLOW
    0
    500
    1000
    1500
    2000
    JAN 2018 APRIL 2018 JULY 2018 OCT 2018

    View full-size slide

  168. @MOLSJEROEN
    PRODUCTIVITY
    0
    10
    20
    30
    40
    50
    NOV 2018 DEC 2018 JAN 2019 MARCH 2019

    View full-size slide

  169. @MOLSJEROEN
    ONE. SINGLE. BRANCH.
    master
    feature bugfix

    View full-size slide

  170. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    FEATURE
    FLAG DEVELOP PULL
    REQUEST MERGE
    QUALIFY

    ++
    master
    RELEASE VERIFY
    feature
    IMPACT
    0,5-2 days 2 hours
    BETA SHIP
    1 week*

    View full-size slide

  171. RELEASING HAS BECOME SO EASY,
    WE EVEN SHIP FRIDAY EVENINGS
    Hue android team

    View full-size slide

  172. STRATEGY OF REDUCING COST OF
    FAILURE IS WORKING

    View full-size slide

  173. “THE MOST POWERFUL TOOL WE HAVE
    AS DEVELOPERS IS AUTOMATION.”
    Scott Hanselman

    View full-size slide

  174. @MOLSJEROEN
    SPEED UP INTEGRATION
    REMOVE ACCIDENTAL COMPLEXITY
    STABILIZE MASTER
    AUTOMATE TESTING AND DELIVERY
    PREFER TRUST OVER PROCESS

    View full-size slide

  175. @MOLSJEROEN
    HTTPS://JEROENMOLS.COM/BLOG

    View full-size slide

  176. @MOLSJEROEN
    IMAGE CREDITS
    Welcome image by Clement127
    https://www.flickr.com/photos/clement127/35754378573/in/photolist-Wtux5Z
    Material design icons by Google

    https://material.io/tools/icons

    View full-size slide