Journey to painless releases: Continuous delivery for Philips Hue Android

Video: https://www.youtube.com/watch?v=VkFcJZWtufA

Wind back one year ago:
- releasing took us 10 weeks
- our app was plagued by regression
- developers felt completely detached from our users

Clearly our traditional strategy of heavily testing each release (up-front quality) wasn't working. So we decided to flip things around: why not try to reduce the cost of errors instead? Hence we decided to move our Philips Hue Android app to continuous deployment.

In this talk you will learn that if releasing is painful, you should actually release even more often. Step by step I'll explain how we moved the Philips Hue Android app to continuous deployment, what our current release process/CI looks like, what changes we made to our overall testing approach, how we empowered developers and testers, and how we made our release process completely painless.

Transcript

1. @MOLSJEROEN
   JOURNEY TO
   PAINLESS RELEASES
   CONTINUOUS DELIVERY FOR PHILIPS HUE
   

   View Slide

2. @MOLSJEROEN
   @MOLSJEROEN
   

   View Slide

3. @MOLSJEROEN
   CHALLENGES
   CONTINUOUS DELIVERY
   MIGRATE YOUR PROJECT
   RESULTS
   

   View Slide

4. CHALLENGES
   

   View Slide

5. @MOLSJEROEN
   RELEASE PROCESS
   DEVELOPMENT
   

   View Slide

6. @MOLSJEROEN
   RELEASE PROCESS
   DEVELOPMENT TESTING
   

   View Slide

7. @MOLSJEROEN
   RELEASE PROCESS
   DEVELOPMENT TESTING TESTING
   

   View Slide

8. @MOLSJEROEN
   RELEASE PROCESS
   DEVELOPMENT TESTING TESTING BETA
   

   View Slide

9. @MOLSJEROEN
   RELEASE PROCESS
   DEVELOPMENT TESTING TESTING BETA
   10 weeks
   

   View Slide

10. @MOLSJEROEN
    RELEASE PROCESS
    DEVELOPMENT TESTING TESTING BETA
    DEVELOPMENT TESTING TESTING BETA
    10 weeks
    

    View Slide

11. @MOLSJEROEN
    RELEASE PROCESS
    DEVELOPMENT TESTING TESTING BETA
    DEVELOPMENT TESTING TESTING BETA
    DEVELOPMENT TESTING TESTIN
    10 weeks
    

    View Slide

12. @MOLSJEROEN
    RELEASE PROCESS
    DEVELOPMENT TESTING TESTING BETA
    DEVELOPMENT TESTING TESTING BETA
    DEVELOPMENT TESTING TESTIN
    10 weeks
    3 releases
    

    View Slide

13. @MOLSJEROEN
    RELEASE PROCESS
    DEVELOPMENT TESTING TESTING BETA
    10 weeks
    

    View Slide

14. @MOLSJEROEN
    RELEASE PROCESS
    DEVELOPMENT TESTING TESTING BETA
    10 weeks
    

    View Slide

15. @MOLSJEROEN
    RELEASE PROCESS
    DEVELOPMENT TESTING TESTING BETA
    10 weeks
    

    View Slide

16. @MOLSJEROEN
    FRUSTRATIONS
    Issues were found extremely late
    Developers felt detached from our users
    Constant context switches between releases
    Releasing very painful
    Friction between developers and testers
    

    View Slide

17. @MOLSJEROEN
    BRANCHING MODEL
    develop
    master
    release
    feature bugfix
    feature
    hotfix
    

    View Slide

18. @MOLSJEROEN
    hotfix
    feature bugfix
    BRANCHING MODEL
    master
    release
    feature
    develop
    

    View Slide

19. @MOLSJEROEN
    BRANCHING MODEL
    develop
    master
    release
    feature bugfix
    feature
    hotfix
    

    View Slide

20. @MOLSJEROEN
    BRANCHING MODEL
    develop
    master
    release
    feature bugfix
    feature
    hotfix
    

    View Slide

21. @MOLSJEROEN
    BRANCHING MODEL
    develop
    master
    release
    feature bugfix
    feature
    hotfix
    

    View Slide

22. @MOLSJEROEN
    FRUSTRATIONS
    Unstable development branch
    Merging nightmares
    Poor code reviews
    Confusion about what fix is where
    Development was slow
    

    View Slide

23. @MOLSJEROEN
    RESULTS
    

    View Slide

24. STRATEGY OF UPFRONT TESTING
    NOT WORKING
    

    View Slide

25. @MOLSJEROEN
    GOALS
    Speed up development
    Get a grip on quality
    Feedback between developers and users
    

    View Slide

26. View Slide

27. View Slide

28. CONTINUOUS DELIVERY
    

    View Slide

29. IF IT HURTS, DO IT MORE OFTEN
    Jez Humble
    

    View Slide

30. @MOLSJEROEN
    WHAT DO WE WANT TO ACHIEVE?
    DEVELOPMENT TESTING TESTING BETA
    DEVELOPMENT TESTING TESTING BETA
    DEVELOPMENT TESTING TESTIN
    

    View Slide

31. @MOLSJEROEN
    WHAT DO WE WANT TO ACHIEVE?
    DEVELOPMENT TESTING TESTING BETA
    DEVELOPMENT TESTING TESTING BETA
    DEVELOPMENT TESTING TESTING BETA
    

    View Slide

32. @MOLSJEROEN
    WHAT DO WE WANT TO ACHIEVE?
    DEVELOPMENT TESTING BETA
    TESTING
    DEVELOPMENT TESTING BETA
    TESTING
    DEVELOPMENT TESTING BETA
    TESTING
    

    View Slide

33. @MOLSJEROEN
    WHAT DO WE WANT TO ACHIEVE?
    BETA
    DEVELOPMENT TEST TEST
    BETA
    DEVELOPMENT TEST TEST
    BETA
    DEVELOPMENT TEST TEST
    

    View Slide

34. @MOLSJEROEN
    WHAT DO WE WANT TO ACHIEVE?
    DEV & TEST BETA
    DEV & TEST BETA
    DEV & TEST BETA
    

    View Slide

35. IF IT HURTS, DO IT MORE OFTEN
    AND BRING THE PAIN FORWARD
    Jez Humble
    

    View Slide

36. @MOLSJEROEN
    TERMINOLOGY
    Continuous integration
    Continuous delivery
    Continuous deployment
    

    View Slide

37. DEVELOPMENT PROCESS
    

    View Slide

38. @MOLSJEROEN
    DEVELOPMENT PROCESS
    develop
    master
    release
    

    View Slide

39. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    develop
    master
    release
    feature
    

    View Slide

40. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    DEVELOP
    MENT
    PULL
    REQUEST
    develop
    master
    release
    feature
    

    View Slide

41. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    DEVELOP
    MENT
    PULL
    REQUEST QUALIFY
    develop
    master
    release
    feature
    

    View Slide

42. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    DEVELOP
    MENT
    PULL
    REQUEST QUALIFY
    develop
    master
    release
    feature
    VERIFY
    

    View Slide

43. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    DEVELOP
    MENT
    PULL
    REQUEST QUALIFY MERGE
    VERIFY
    develop
    master
    release
    feature
    

    View Slide

44. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    DEVELOP
    MENT
    PULL
    REQUEST QUALIFY MERGE
    VERIFY
    develop
    master
    release
    feature
    1-5 weeks
    

    View Slide

45. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    DEVELOP
    MENT
    PULL
    REQUEST QUALIFY MERGE PROMOTE
    MASTER
    VERIFY
    develop
    master
    release
    feature
    1-5 weeks
    

    View Slide

46. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    DEVELOP
    MENT
    PULL
    REQUEST QUALIFY MERGE PROMOTE
    MASTER
    VERIFY
    develop
    master
    release
    feature
    STABILIZE
    1-5 weeks
    

    View Slide

47. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    DEVELOP
    MENT
    PULL
    REQUEST QUALIFY MERGE PROMOTE
    MASTER
    VERIFY
    develop
    master
    release
    feature
    STABILIZE
    3 weeks
    1-5 weeks
    

    View Slide

48. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    DEVELOP
    MENT
    PULL
    REQUEST QUALIFY MERGE PROMOTE
    RELEASE
    PROMOTE
    MASTER STABILIZE
    VERIFY
    develop
    master
    release
    feature
    3 weeks
    1-5 weeks
    

    View Slide

49. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    DEVELOP
    MENT BETA
    PULL
    REQUEST QUALIFY MERGE PROMOTE
    RELEASE
    PROMOTE
    MASTER STABILIZE
    VERIFY
    develop
    master
    release
    feature
    3 weeks
    1-5 weeks
    

    View Slide

50. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    DEVELOP
    MENT BETA
    PULL
    REQUEST QUALIFY MERGE PROMOTE
    RELEASE
    PROMOTE
    MASTER STABILIZE
    VERIFY
    develop
    master
    release
    feature
    3 weeks
    1-5 weeks
    SHIP
    

    View Slide

51. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    DEVELOP
    MENT BETA
    PULL
    REQUEST QUALIFY MERGE PROMOTE
    RELEASE
    PROMOTE
    MASTER STABILIZE
    VERIFY SHIP
    develop
    master
    release
    feature
    MERGE
    BACK
    1-5 weeks 3 weeks
    

    View Slide

52. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    DEVELOP
    MENT BETA
    PULL
    REQUEST QUALIFY MERGE PROMOTE
    RELEASE
    PROMOTE
    MASTER STABILIZE
    VERIFY SHIP
    develop
    master
    release
    feature
    MERGE
    BACK
    1-5 weeks 3 weeks 3 weeks
    

    View Slide

53. @MOLSJEROEN
    MERGING/INTEGRATING
    FEATURE
    BRANCH
    DEVELOP
    MENT BETA
    PULL
    REQUEST MERGE PROMOTE
    RELEASE
    PROMOTE
    MASTER STABILIZE
    VERIFY SHIP
    develop
    master
    release
    feature
    MERGE
    BACK
    1-5 weeks 3 weeks 3 weeks
    QUALIFY
    

    View Slide

54. @MOLSJEROEN
    MERGING/INTEGRATING
    FEATURE
    BRANCH
    DEVELOP
    MENT BETA
    PULL
    REQUEST MERGE PROMOTE
    RELEASE
    PROMOTE
    MASTER STABILIZE
    VERIFY SHIP
    develop
    master
    release
    feature
    MERGE
    BACK
    1-5 weeks 3 weeks 3 weeks
    QUALIFY
    

    View Slide

55. SPEED UP INTEGRATION
    

    View Slide

56. @MOLSJEROEN
    

    View Slide

57. @MOLSJEROEN
    

    View Slide

58. @MOLSJEROEN
    Large pull request
    

    View Slide

59. @MOLSJEROEN
    Large pull request
    Hard to

    review
    

    View Slide

60. @MOLSJEROEN
    Large pull request
    Hard to

    review
    Hard to

    verify
    

    View Slide

61. @MOLSJEROEN
    Large pull request
    Hard to

    review
    Long lived

    pull request
    Hard to

    verify
    

    View Slide

62. @MOLSJEROEN
    Large pull request
    Hard to

    review
    Long lived

    pull request
    Merge

    conflicts
    Hard to

    verify
    

    View Slide

63. @MOLSJEROEN
    Large pull request
    Hard to

    review
    Long lived

    pull request
    Merge

    conflicts
    Overhead: 

    regression, 

    dependencies,…
    Hard to

    verify
    

    View Slide

64. @MOLSJEROEN
    MERGING/INTEGRATING
    FEATURE
    BRANCH
    DEVELOP
    MENT BETA
    PULL
    REQUEST MERGE PROMOTE
    RELEASE
    PROMOTE
    MASTER STABILIZE
    VERIFY SHIP MERGE
    BACK
    1-5 weeks 3 weeks 3 weeks
    QUALIFY
    FEATURE
    BRANCH
    DEVELOP
    MENT BETA
    PULL
    REQUEST MERGE PROMOTE
    RELEASE
    PROMOTE
    MASTER STABILIZE
    VERIFY SHIP
    develop
    master
    release
    feature
    MERGE
    BACK
    1-5 weeks 3 weeks 3 weeks
    QUALIFY
    

    View Slide

65. @MOLSJEROEN
    PULL REQUEST QUALIFICATION
    Develop
    feature BUILD

    TEST
    REVIEW
    

    View Slide

66. @MOLSJEROEN
    QUALIFICATION: SPEED UP BUILDS
    Update all dependencies to latest version
    Follow best practices
    Modularize
    Butterknife reflect
    Gradle build cache and build scans
    Monitor and keep on experimenting
    

    View Slide

67. @MOLSJEROEN
    QUALIFICATION: SPEED UP BUILDS
    3m 45s
    1m 26s
    

    View Slide

68. @MOLSJEROEN
    QUALIFICATION: SPEED UP CI
    Git hook instead of polling
    Git cache
    Preload gradle wrapper in docker container
    Unit tests over instrumentation tests
    Run subset of tests
    Parallel tasks
    

    View Slide

69. HOW MUCH TIME ARE YOU WILLING TO
    INVEST IN QUALIFYING PULL REQUESTS?
    

    View Slide

70. @MOLSJEROEN
    QUALIFICATION: SPEED UP TESTS
    Legacy “unit tests” written as instrumentation
    Converted 1272 tests
    Build speed decreased by 3m 18s
    -9% code coverage
    Removed CI bottleneck
    Make room for new Espresso tests
    

    View Slide

71. @MOLSJEROEN
    QUALIFICATION: SPEED UP BUILDS
    ~20 min
    5m 10s
    

    View Slide

72. @MOLSJEROEN
    QUALIFICATION: SPEED UP TESTS
    Instrumentation tests
    Unit tests
    

    View Slide

73. @MOLSJEROEN
    QUALIFICATION: SPEED UP TESTS
    Instrumentation tests
    Unit tests
    

    View Slide

74. @MOLSJEROEN
    QUALIFICATION: SPEED UP TESTS
    …
    Instrumentation tests
    Unit tests
    …
    

    View Slide

75. @MOLSJEROEN
    QUALIFICATION: SPEED UP BUILDS (10 PRS)
    ~1 hour
    5m 10s
    

    View Slide

76. @MOLSJEROEN
    VERIFICATION
    Develop
    feature
    

    View Slide

77. @MOLSJEROEN
    VERIFICATION: MOVE TO MASTER
    Develop
    feature
    

    View Slide

78. @MOLSJEROEN
    VERIFICATION: MOVE TO MASTER
    1-3 days
    0 days
    

    View Slide

79. @MOLSJEROEN
    DEVELOPMENT
    Develop
    feature
    

    View Slide

80. @MOLSJEROEN
    DEVELOPMENT: SMALLER BRANCHES
    feature
    Develop
    

    View Slide

81. @MOLSJEROEN
    DEVELOPMENT: SMALLER BRANCHES
    1-5 weeks
    0,5 - 2 days
    

    View Slide

82. "IF YOU MERGE EVERY DAY, SUDDENLY
    YOU NEVER GET TO THE POINT WHERE
    YOU HAVE HUGE CONFLICTS THAT ARE
    HARD TO RESOLVE."
    Linus Torvalds
    

    View Slide

83. @MOLSJEROEN
    RECAP: SPEEDING UP INTEGRATION
    Blazing fast builds
    A state of the art CI
    Do verification on develop
    Many small branches
    => Integration speed up from 1 - 5 weeks to 0,5 - 2days
    

    View Slide

84. @MOLSJEROEN
    DEVELOPMENT PROCESS
    FEATURE
    BRANCH
    DEVELOP
    MENT BETA
    PULL
    REQUEST QUALIFY MERGE PROMOTE
    RELEASE
    PROMOTE
    MASTER STABILIZE
    VERIFY SHIP
    develop
    master
    release
    feature
    MERGE
    BACK
    0,5-2 days 3 weeks 3 weeks
    

    View Slide

85. REDUCE COMPLEXITY
    

    View Slide

86. @MOLSJEROEN
    FEATURE BRANCHES
    develop
    master
    release
    feature bugfix
    feature
    hotfix
    

    View Slide

87. @MOLSJEROEN
    FEATURE BRANCHES: ONLY FROM DEVELOP
    develop
    master
    release
    feature bugfix
    hotfix
    

    View Slide

88. @MOLSJEROEN
    DEVELOP BRANCH
    develop
    master
    release
    feature bugfix
    hotfix
    

    View Slide

89. @MOLSJEROEN
    DEVELOP BRANCH
    master
    release
    feature bugfix
    hotfix
    

    View Slide

90. ACCIDENTAL VS ESSENTIAL COMPLEXITY
    

    View Slide

91. @MOLSJEROEN
    BUILD VARIANTS
    buildTypes {
    debug {
    signingConfig signingConfigs.debug
    ext.enableCrashlytics = false
    }
    release {
    minifyEnabled true
    proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
    signingConfig signingConfigs.release
    }
    uitest {
    // provide implementation for idling resources
    }
    }
    flavorDimensions "MODE"
    productFlavors {
    daily {
    applicationId 'com.philips.lighting.hue2.daily'
    dimension "MODE"
    buildConfigField "String", "HOCKEY_APP_ID", Keys.hockeyAppDaily
    buildConfigField "String", "AMPLITUDE_APP_ID", Keys.amplitudeDaily
    }
    store {
    dimension "MODE"
    buildConfigField "String", "HOCKEY_APP_ID", Keys.hockeyApp
    buildConfigField "String", "AMPLITUDE_APP_ID", Keys.amplitudeStore
    }
    }
    variantFilter { variant ->
    def names = variant.flavors*.name
    if (names.contains("store") && variant.buildType.name == 'debug') {
    variant.setIgnore(true)
    }
    if (names.contains("store") && variant.buildType.name == 'uitest') {
    variant.setIgnore(true)
    }
    }
    

    View Slide

92. @MOLSJEROEN
    BUILD VARIANTS
    Prod API
    keys
    Proguard Test
    settings
    Idling
    resource
    s
    Leak
    canary
    Prod
    firmware
    Develop

    Portal
    Release
    Release test
    Debug
    Espresso
    

    View Slide

93. @MOLSJEROEN
    IDLING RESOURCES: UI TEST VARIANT
    object EspressoIdlingResource {
    private val idling = BaseIdlingResource()
    fun increment(code: String, reason: String) = idling.increment(code, reason)
    fun decrement(code: String, reason: String) = idling.decrement(code, reason)
    }
    

    View Slide

94. @MOLSJEROEN
    IDLING RESOURCES: DEBUG AND RELEASE VARIANT
    object EspressoIdlingResource {
    fun increment(code: String, reason: String) { /* NOP */ }
    fun decrement(code: String, reason: String) { /* NOP */ }
    }
    

    View Slide

95. @MOLSJEROEN
    IDLING RESOURCES: UNIFIED
    object EspressoIdlingResource {
    private val idling = BaseIdlingResource()
    fun increment(code: String, reason: String) =
    if (RuntimeBehavior.isFeatureEnabled(IDLING_RESOURCES)) {
    idling.increment(code, reason)
    }
    fun decrement(code: String, reason: String) =
    if (RuntimeBehavior.isFeatureEnabled(IDLING_RESOURCES)) {
    mCountingIdlingResource.decrement(code reason)
    }
    }
    

    View Slide

96. @MOLSJEROEN
    IDLING RESOURCES: UNIFIED
    object EspressoIdlingResource {
    private val idling = BaseIdlingResource()
    fun increment(code: String, reason: String) =
    if (RuntimeBehavior.isFeatureEnabled(IDLING_RESOURCES)) {
    idling.increment(code, reason)
    }
    fun decrement(code: String, reason: String) =
    if (RuntimeBehavior.isFeatureEnabled(IDLING_RESOURCES)) {
    mCountingIdlingResource.decrement(code reason)
    }
    }
    

    View Slide

97. @MOLSJEROEN
    BUILD VARIANTS
    Prod API
    keys
    Proguard Test
    settings
    Idling
    resource
    s
    Leak
    canary
    Prod
    firmware
    Develop

    Portal
    Release
    Release test
    Debug
    Espresso
    

    View Slide

98. @MOLSJEROEN
    BUILD VARIANTS
    Prod API
    keys
    Proguard Test
    settings
    Idling
    resource
    s
    Leak
    canary
    Prod
    firmware
    Develop

    Portal
    Release
    Release test
    Debug
    Espresso
    

    View Slide

99. @MOLSJEROEN
    RELEASE AND DEBUG
    

    View Slide

100. @MOLSJEROEN
     SOURCE FOLDERS
     

     View Slide

101. @MOLSJEROEN
     GRADLE CONFIGURATION
     buildTypes {
     debug {
     signingConfig signingConfigs.debug
     applicationIdSuffix '.daily'
     ext.enableCrashlytics = false
     buildConfigField "String", "HOCKEY_APP_ID", Keys.hockeyAppDebug
     buildConfigField "String", "AMPLITUDE_APP_ID", Keys.amplitudeDebug
     }
     release {
     signingConfig signingConfigs.release
     minifyEnabled true
     proguardFiles 'proguard-rules.pro'
     buildConfigField "String", "HOCKEY_APP_ID", Keys.hockeyAppRelease
     buildConfigField "String", "AMPLITUDE_APP_ID", Keys.amplitudeRelease
     }
     

     View Slide

102. SIMPLICITY IS THE SOUL OF EFFICIENCY
     Austin Freeman
     

     View Slide

103. @MOLSJEROEN
     REDUCE COMPLEXITY: SUMMARY
     Drop unnecessary branches
     No branches from branches
     Minimize build variants
     Feature flags for dynamic configuration
     => Remove all accidental complexity
     

     View Slide

104. @MOLSJEROEN
     DEVELOPMENT PROCESS
     FEATURE
     BRANCH
     DEVELOP
     MENT BETA
     PULL
     REQUEST QUALIFY MERGE PROMOTE
     RELEASE STABILIZE
     VERIFY SHIP
     master
     release
     feature
     MERGE
     BACK
     0,5-2 days 6 weeks*
     

     View Slide

105. @MOLSJEROEN
     DEVELOPMENT PROCESS
     FEATURE
     BRANCH
     DEVELOP
     MENT BETA
     PULL
     REQUEST QUALIFY MERGE PROMOTE
     RELEASE STABILIZE
     VERIFY RELEASE
     master
     release
     feature
     MERGE
     BACK
     0,5-2 days ~4 weeks
     

     View Slide

106. STABILIZE MASTER
     

     View Slide

107. @MOLSJEROEN
     FEATURE FLAGS: NEW FEATURES
     override fun onViewCreated(view: View) {
     ...
     if (RuntimeBehavior.isFeatureEnabled(FeatureFlag.POWER_ON_BEHAVIOR)) {
     power_on_behavior.visibility = View.VISIBLE
     } else {
     power_on_behavior.visibility = View.GONE
     }
     ...
     }
     

     View Slide

108. @MOLSJEROEN
     FEATURE FLAGS: NEW FEATURES
     override fun onViewCreated(view: View) {
     ...
     if (RuntimeBehavior.isFeatureEnabled(FeatureFlag.POWER_ON_BEHAVIOR)) {
     power_on_behavior.visibility = View.VISIBLE
     } else {
     power_on_behavior.visibility = View.GONE
     }
     ...
     }
     

     View Slide

109. @MOLSJEROEN
     FEATURE FLAGS: EXISTING CODE
     fun withLegacy() {
     var code = LegacyLogic()
     code.with(OutUnitTests())
     // ...
     code.doMagic()
     }
     

     View Slide

110. @MOLSJEROEN
     FEATURE FLAGS: EXISTING CODE
     fun withLegacy() {
     var code = LegacyLogic()
     code.with(OutUnitTests())
     // ...
     code.doMagic()
     }
     

     View Slide

111. @MOLSJEROEN
     FEATURE FLAGS: EXISTING CODE
     fun withLegacy() {
     var code = NewLogic()
     code.with(UnitTests())
     // ...
     code.doMagic()
     }
     

     View Slide

112. @MOLSJEROEN
     FEATURE FLAGS: EXISTING CODE
     fun withLegacy() {
     if (RuntimeBehavior.isFeatureEnabled(FeatureFlag.LOGIC_REFACTOR)) {
     var code = NewLogic()
     code.with(UnitTests())
     // ...
     code.doMagic()
     } else {
     var code = LegacyLogic()
     code.with(OutUnitTests())
     // ...
     code.doMagic()
     }
     }
     

     View Slide

113. @MOLSJEROEN
     

     View Slide

114. @MOLSJEROEN
     QUALIFY PULL REQUESTS
     master
     feature
     

     View Slide

115. @MOLSJEROEN
     QUALIFY PULL REQUESTS
     master
     feature
     

     BUILD

     REVIEW

     

     View Slide

116. @MOLSJEROEN
     QUALIFY PULL REQUESTS
     master
     feature
     

     BUILD

     REVIEW
     ANALYZE

     

     View Slide

117. @MOLSJEROEN
     QUALIFY PULL REQUESTS
     master
     feature
     

     BUILD

     REVIEW

     ANALYZE

     TEST
     

     View Slide

118. @MOLSJEROEN
     AUTOMATE TESTING
     UNIT
     INTEGRATION
     E2E
     

     View Slide

119. @MOLSJEROEN
     AUTOMATE TESTING
     Library
     Feature
     Feature
     Feature
     Feature
     Feature
     Feature
     Feature
     Feature
     Feature
     App
     Library
     Library
     Lib Lib Library
     Library
     Lib
     

     View Slide

120. @MOLSJEROEN
     

     View Slide

121. View Slide

122. @MOLSJEROEN
     TEST STRATEGY
     Pull requests:
     Run all automated level1 app Espresso tests
     Run all automated tests of the modules that are impacted
     Manual verification if high impact
     Release:
     Run all automated Espresso tests
     Run all automated feature and library tests
     Deploy: Manual exploratory testing
     

     View Slide

123. @MOLSJEROEN
     TEST STRATEGY
     Pull requests:
     Run all automated level1 app Espresso tests
     Run all automated tests of the modules that are impacted
     Manual verification if high impact
     Release:
     Run all automated Espresso tests
     Run all automated feature and library tests
     Deploy: Manual exploratory testing
     

     View Slide

124. @MOLSJEROEN
     SIMULATE ACTUAL MERGE
     master
     feature BUILD

     REVIEW

     ANALYZE

     TEST
     

     View Slide

125. @MOLSJEROEN
     SIMULATE ACTUAL MERGE
     master
     feature
     

     MERGE
     BUILD

     REVIEW

     ANALYZE

     TEST
     

     View Slide

126. @MOLSJEROEN
     master
     feature
     SIMULATE ACTUAL MERGE
     

     View Slide

127. @MOLSJEROEN
     master
     feature
     SIMULATE ACTUAL MERGE
     

     View Slide

128. @MOLSJEROEN
     master
     feature
     feature
     feature
     SIMULATE ACTUAL MERGE
     

     View Slide

129. @MOLSJEROEN
     master
     feature
     feature
     feature
     SIMULATE ACTUAL MERGE
     

     View Slide

130. @MOLSJEROEN
     master
     feature
     feature
     feature
     SIMULATE ACTUAL MERGE
     

     View Slide

131. @MOLSJEROEN
     MERGE PULL REQUESTS
     master
     feature
     

     MERGE
     BUILD

     REVIEW

     ANALYZE

     TEST
     

     View Slide

132. @MOLSJEROEN
     MERGE PULL REQUESTS
     master
     feature
     

     MERGE
     BUILD

     REVIEW

     ANALYZE

     TEST
     IMPACT

     

     View Slide

133. @MOLSJEROEN
     IMPACT ANALYSIS
     SIDE EFFECTS POSSIBLE IN 

     OTHER PARTS OF CODE? (REGRESSION)
     ALL SIDE EFFECTS CONTAINED
     BEHIND A FEATURE FLAG?
     LOW IMPACT
     LOW IMPACT
     HIGH IMPACT
     Yes
     Yes
     No
     Not sure
     Not sure
     No
     

     View Slide

134. @MOLSJEROEN
     IMPACT ANALYSIS
     

     View Slide

135. @MOLSJEROEN
     STABILIZE MASTER: SUMMARY
     Use feature toggles
     Static code analysis
     Heavily automate testing
     Merge master before checks
     Requalify pull requests after each merge
     Impact assessment
     => Able to drop release branch
     

     View Slide

136. @MOLSJEROEN
     DEVELOPMENT PROCESS
     0,5-2 days 2 weeks
     FEATURE
     BRANCH
     FEATURE
     FLAG DEVELOP PULL
     REQUEST MERGE
     QUALIFY

     ++ VERIFY
     IMPACT
     master
     feature
     BETA
     MANUAL
     RELEASE SHIP
     

     View Slide

137. @MOLSJEROEN
     DEVELOPMENT PROCESS
     0,5-2 days 2 weeks
     FEATURE
     BRANCH
     FEATURE
     FLAG DEVELOP PULL
     REQUEST MERGE
     QUALIFY

     ++ VERIFY
     IMPACT
     master
     feature
     BETA
     MANUAL
     RELEASE SHIP
     

     View Slide

138. AUTOMATE DELIVERY
     

     View Slide

139. @MOLSJEROEN
     UPDATE TRANSLATIONS
     master
     feature
     

     View Slide

140. @MOLSJEROEN
     UPDATE TRANSLATIONS
     master
     feature
     translations
     

     View Slide

141. @MOLSJEROEN
     UPDATE TRANSLATIONS
     master
     feature
     translations
     

     View Slide

142. @MOLSJEROEN
     UPDATE SDK AND ASSETS
     master
     feature
     translations SDK assets
     

     View Slide

143. @MOLSJEROEN
     RELEASE APP (DELIVERY)
     stage('Deploy to Google Play Internal Test Track') {
     steps {
     script {
     def releaseNotes = readFile 'app/build/release_notes.txt'
     androidApkUpload(
     googleCredentialsId: 'Google Play Android Developer',
     trackName: 'internal',
     apkFilesPattern: '**/*-release.apk',
     deobfuscationFilesPattern: '**/mapping.txt',
     recentChangeList: [[language: 'en-US', text: releaseNotes]]
     )
     }
     }
     }
     

     View Slide

144. @MOLSJEROEN
     RELEASE APP (DELIVERY)
     stage('Deploy to HockeyApp') {
     steps {
     sh "pipelines/scripts/hockeyapp/deploy_app_debug.sh"
     }
     }
     

     View Slide

145. @MOLSJEROEN
     IDEAL SCENARIO
     master
     feature bugfix
     

     View Slide

146. @MOLSJEROEN
     IDEAL SCENARIO
     master
     feature bugfix
     hotfix
     

     View Slide

147. @MOLSJEROEN
     IDEAL SCENARIO
     master
     feature bugfix
     hotfix
     

     View Slide

148. @MOLSJEROEN
     IDEAL SCENARIO
     master
     feature bugfix
     hotfix
     

     View Slide

149. @MOLSJEROEN
     ACTUAL SCENARIO
     master
     feature bugfix
     hotfix
     

     View Slide

150. @MOLSJEROEN
     ACTUAL SCENARIO
     master
     feature bugfix
     hotfix
     

     View Slide

151. @MOLSJEROEN
     ACTUAL SCENARIO
     master
     feature bugfix
     hotfix
     

     View Slide

152. @MOLSJEROEN
     ACTUAL SCENARIO
     master
     feature bugfix
     hotfix
     

     View Slide

153. @MOLSJEROEN
     RELEASING
     master
     internal test
     beta
     production
     

     View Slide

154. @MOLSJEROEN
     RELEASING
     master
     internal test
     beta
     production
     

     View Slide

155. @MOLSJEROEN
     AUTOMATE DELIVERY: SUMMARY
     Integrate dependencies automatically
     Automate upload to Google play/hockey app
     Release directly from master
     Hotfixes from separate branch
     

     View Slide

156. @MOLSJEROEN
     DEVELOPMENT PROCESS
     FEATURE
     BRANCH
     FEATURE
     FLAG DEVELOP PULL
     REQUEST MERGE
     QUALIFY

     ++
     master
     RELEASE VERIFY
     feature
     IMPACT
     0,5-2 days 2 hours
     BETA SHIP
     1 week*
     

     View Slide

157. LESSONS LEARNED
     

     View Slide

158. @MOLSJEROEN
     1. BI-WEEKLY RELEASES
     DEVELOPMENT TESTING TESTING BETA
     DEVELOPMENT TESTING TESTING BETA
     DEVELOPMENT TESTING TESTIN
     10 weeks
     

     View Slide

159. @MOLSJEROEN
     1. BI-WEEKLY RELEASES
     DEV & TEST BETA
     DEV & TEST BETA
     DEV & TEST BETA
     2 weeks
     

     View Slide

160. @MOLSJEROEN
     1. BI-WEEKLY RELEASES
     BETA
     DEVELOPMENT TEST TEST
     BETA
     DEVELOPMENT TEST TEST
     BETA
     DEVELOPMENT TEST TEST
     

     View Slide

161. @MOLSJEROEN
     1. BI-WEEKLY RELEASES
     BETA
     DEVELOPMENT TEST
     TEST
     BETA
     DEVELOPMENT TEST
     TEST
     BETA
     DEVELOPMENT TEST
     TEST
     

     View Slide

162. @MOLSJEROEN
     1. BI-WEEKLY RELEASES
     DEV & TEST BETA
     DEV & TEST BETA
     DEV & TEST BETA
     2 weeks
     DEV & TEST BETA
     

     View Slide

163. @MOLSJEROEN
     2. MERGING WITHOUT VERIFICATION
     master
     feature
     

     MERGE
     BUILD

     REVIEW

     ANALYZE

     TEST
     IMPACT

     

     View Slide

164. @MOLSJEROEN
     2. MERGING WITHOUT VERIFICATION
     

     View Slide

165. @MOLSJEROEN
     2. MERGING WITHOUT VERIFICATION
     

     View Slide

166. TRUST TAKES YEARS TO BUILD,
     SECONDS TO BREAK,
     AND FOREVER TO REPAIR.
     Unknown
     

     View Slide

167. @MOLSJEROEN
     2. MERGING WITHOUT VERIFICATION
     

     View Slide

168. @MOLSJEROEN
     2. MERGING WITHOUT VERIFICATION
     

     View Slide

169. @MOLSJEROEN
     3. YOU WILL MESS UP
     

     View Slide

170. @MOLSJEROEN
     SUMMARY
     Focus on whitebox testing
     Gain trust and give trust back
     Don’t blow your trust!
     Rely on automated testing
     Closer collaboration between testers and developers
     Root cause analyses
     

     View Slide

171. RESULTS
     

     View Slide

172. @MOLSJEROEN
     RESULTS
     29
     

     View Slide

173. @MOLSJEROEN
     TIME TO RELEASE
     Build time (min)
     0
     10
     20
     30
     40
     50
     Build number
     

     View Slide

174. @MOLSJEROEN
     

     View Slide

175. @MOLSJEROEN
     

     View Slide

176. @MOLSJEROEN
     INTEGRATION SPEED
     620 ~1/dev/day
     

     View Slide

177. @MOLSJEROEN
     APP RATING
     

     View Slide

178. @MOLSJEROEN
     PR INFLOW
     0
     500
     1000
     1500
     2000
     JAN 2018 APRIL 2018 JULY 2018 OCT 2018
     

     View Slide

179. @MOLSJEROEN
     PRODUCTIVITY
     0
     10
     20
     30
     40
     50
     NOV 2018 DEC 2018 JAN 2019 MARCH 2019
     

     View Slide

180. @MOLSJEROEN
     ONE. SINGLE. BRANCH.
     master
     feature bugfix
     

     View Slide

181. @MOLSJEROEN
     DEVELOPMENT PROCESS
     FEATURE
     BRANCH
     FEATURE
     FLAG DEVELOP PULL
     REQUEST MERGE
     QUALIFY

     ++
     master
     RELEASE VERIFY
     feature
     IMPACT
     0,5-2 days 2 hours
     BETA SHIP
     1 week*
     

     View Slide

182. RELEASING HAS BECOME SO EASY,
     WE EVEN SHIP FRIDAY EVENINGS
     Hue android team
     

     View Slide

183. STRATEGY OF REDUCING COST OF
     FAILURE IS WORKING
     

     View Slide

184. WRAP UP
     

     View Slide

185. “THE MOST POWERFUL TOOL WE HAVE
     AS DEVELOPERS IS AUTOMATION.”
     Scott Hanselman
     

     View Slide

186. @MOLSJEROEN
     SPEED UP INTEGRATION
     REMOVE ACCIDENTAL COMPLEXITY
     STABILIZE MASTER
     AUTOMATE TESTING AND DELIVERY
     PREFER TRUST OVER PROCESS
     

     View Slide

187. @MOLSJEROEN
     HTTPS://JEROENMOLS.COM/BLOG
     

     View Slide

188. @MOLSJEROEN
     IMAGE CREDITS
     Welcome image by Clement127
     https://www.flickr.com/photos/clement127/35754378573/in/photolist-Wtux5Z
     Material design icons by Google

     https://material.io/tools/icons
     

     View Slide

189. MOLSJEROEN
     

     View Slide