Journey to painless releases: Continuous delivery for Philips Hue Android

Slide 1

Slide 1 text

@MOLSJEROEN JOURNEY TO PAINLESS RELEASES CONTINUOUS DELIVERY FOR PHILIPS HUE

Slide 2

Slide 2 text

@MOLSJEROEN @MOLSJEROEN

Slide 3

Slide 3 text

@MOLSJEROEN CHALLENGES CONTINUOUS DELIVERY MIGRATE YOUR PROJECT RESULTS

Slide 4

Slide 4 text

CHALLENGES

Slide 5

Slide 5 text

@MOLSJEROEN RELEASE PROCESS DEVELOPMENT

Slide 6

Slide 6 text

@MOLSJEROEN RELEASE PROCESS DEVELOPMENT TESTING

Slide 7

Slide 7 text

@MOLSJEROEN RELEASE PROCESS DEVELOPMENT TESTING TESTING

Slide 8

Slide 8 text

@MOLSJEROEN RELEASE PROCESS DEVELOPMENT TESTING TESTING BETA

Slide 9

Slide 9 text

@MOLSJEROEN RELEASE PROCESS DEVELOPMENT TESTING TESTING BETA 10 weeks

Slide 10

Slide 10 text

@MOLSJEROEN RELEASE PROCESS DEVELOPMENT TESTING TESTING BETA DEVELOPMENT TESTING TESTING BETA 10 weeks

Slide 11

Slide 11 text

@MOLSJEROEN RELEASE PROCESS DEVELOPMENT TESTING TESTING BETA DEVELOPMENT TESTING TESTING BETA DEVELOPMENT TESTING TESTIN 10 weeks

Slide 12

Slide 12 text

@MOLSJEROEN RELEASE PROCESS DEVELOPMENT TESTING TESTING BETA DEVELOPMENT TESTING TESTING BETA DEVELOPMENT TESTING TESTIN 10 weeks 3 releases

Slide 13

Slide 13 text

@MOLSJEROEN RELEASE PROCESS DEVELOPMENT TESTING TESTING BETA 10 weeks

Slide 14

Slide 14 text

@MOLSJEROEN RELEASE PROCESS DEVELOPMENT TESTING TESTING BETA 10 weeks

Slide 15

Slide 15 text

@MOLSJEROEN RELEASE PROCESS DEVELOPMENT TESTING TESTING BETA 10 weeks

Slide 16

Slide 16 text

@MOLSJEROEN FRUSTRATIONS Issues were found extremely late Developers felt detached from our users Constant context switches between releases Releasing very painful Friction between developers and testers

Slide 17

Slide 17 text

@MOLSJEROEN BRANCHING MODEL develop master release feature bugﬁx feature hotﬁx

Slide 18

Slide 18 text

@MOLSJEROEN hotﬁx feature bugﬁx BRANCHING MODEL master release feature develop

Slide 19

Slide 19 text

@MOLSJEROEN BRANCHING MODEL develop master release feature bugﬁx feature hotﬁx

Slide 20

Slide 20 text

@MOLSJEROEN BRANCHING MODEL develop master release feature bugﬁx feature hotﬁx

Slide 21

Slide 21 text

@MOLSJEROEN BRANCHING MODEL develop master release feature bugﬁx feature hotﬁx

Slide 22

Slide 22 text

@MOLSJEROEN FRUSTRATIONS Unstable development branch Merging nightmares Poor code reviews Confusion about what ﬁx is where Development was slow

Slide 23

Slide 23 text

@MOLSJEROEN RESULTS

Slide 24

Slide 24 text

STRATEGY OF UPFRONT TESTING NOT WORKING

Slide 25

Slide 25 text

@MOLSJEROEN GOALS Speed up development Get a grip on quality Feedback between developers and users

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

CONTINUOUS DELIVERY

Slide 29

Slide 29 text

IF IT HURTS, DO IT MORE OFTEN Jez Humble

Slide 30

Slide 30 text

@MOLSJEROEN WHAT DO WE WANT TO ACHIEVE? DEVELOPMENT TESTING TESTING BETA DEVELOPMENT TESTING TESTING BETA DEVELOPMENT TESTING TESTIN

Slide 31

Slide 31 text

@MOLSJEROEN WHAT DO WE WANT TO ACHIEVE? DEVELOPMENT TESTING TESTING BETA DEVELOPMENT TESTING TESTING BETA DEVELOPMENT TESTING TESTING BETA

Slide 32

Slide 32 text

@MOLSJEROEN WHAT DO WE WANT TO ACHIEVE? DEVELOPMENT TESTING BETA TESTING DEVELOPMENT TESTING BETA TESTING DEVELOPMENT TESTING BETA TESTING

Slide 33

Slide 33 text

@MOLSJEROEN WHAT DO WE WANT TO ACHIEVE? BETA DEVELOPMENT TEST TEST BETA DEVELOPMENT TEST TEST BETA DEVELOPMENT TEST TEST

Slide 34

Slide 34 text

@MOLSJEROEN WHAT DO WE WANT TO ACHIEVE? DEV & TEST BETA DEV & TEST BETA DEV & TEST BETA

Slide 35

Slide 35 text

IF IT HURTS, DO IT MORE OFTEN AND BRING THE PAIN FORWARD Jez Humble

Slide 36

Slide 36 text

@MOLSJEROEN TERMINOLOGY Continuous integration Continuous delivery Continuous deployment

Slide 37

Slide 37 text

DEVELOPMENT PROCESS

Slide 38

Slide 38 text

@MOLSJEROEN DEVELOPMENT PROCESS develop master release

Slide 39

Slide 39 text

@MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH develop master release feature

Slide 40

Slide 40 text

@MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST develop master release feature

Slide 41

Slide 41 text

@MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY develop master release feature

Slide 42

Slide 42 text

@MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY develop master release feature VERIFY

Slide 43

Slide 43 text

@MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY MERGE VERIFY develop master release feature

Slide 44

Slide 44 text

@MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY MERGE VERIFY develop master release feature 1-5 weeks

Slide 45

Slide 45 text

@MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY MERGE PROMOTE MASTER VERIFY develop master release feature 1-5 weeks

Slide 46

Slide 46 text

@MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY MERGE PROMOTE MASTER VERIFY develop master release feature STABILIZE 1-5 weeks

Slide 47

Slide 47 text

@MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY MERGE PROMOTE MASTER VERIFY develop master release feature STABILIZE 3 weeks 1-5 weeks

Slide 48

Slide 48 text

@MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY develop master release feature 3 weeks 1-5 weeks

Slide 49

Slide 49 text

@MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST QUALIFY MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY develop master release feature 3 weeks 1-5 weeks

Slide 50

Slide 50 text

@MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST QUALIFY MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY develop master release feature 3 weeks 1-5 weeks SHIP

Slide 51

Slide 51 text

@MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST QUALIFY MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY SHIP develop master release feature MERGE BACK 1-5 weeks 3 weeks

Slide 52

Slide 52 text

Slide 53

Slide 53 text

@MOLSJEROEN MERGING/INTEGRATING FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY SHIP develop master release feature MERGE BACK 1-5 weeks 3 weeks 3 weeks QUALIFY

Slide 54

Slide 54 text

Slide 55

Slide 55 text

SPEED UP INTEGRATION

Slide 56

Slide 56 text

@MOLSJEROEN

Slide 57

Slide 57 text

@MOLSJEROEN

Slide 58

Slide 58 text

@MOLSJEROEN Large pull request

Slide 59

Slide 59 text

@MOLSJEROEN Large pull request Hard to  review

Slide 60

Slide 60 text

@MOLSJEROEN Large pull request Hard to  review Hard to  verify

Slide 61

Slide 61 text

@MOLSJEROEN Large pull request Hard to  review Long lived  pull request Hard to  verify

Slide 62

Slide 62 text

@MOLSJEROEN Large pull request Hard to  review Long lived  pull request Merge  conﬂicts Hard to  verify

Slide 63

Slide 63 text

@MOLSJEROEN Large pull request Hard to  review Long lived  pull request Merge  conﬂicts Overhead:   regression,   dependencies,… Hard to  verify

Slide 64

Slide 64 text

@MOLSJEROEN MERGING/INTEGRATING FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY SHIP MERGE BACK 1-5 weeks 3 weeks 3 weeks QUALIFY FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY SHIP develop master release feature MERGE BACK 1-5 weeks 3 weeks 3 weeks QUALIFY

Slide 65

Slide 65 text

@MOLSJEROEN PULL REQUEST QUALIFICATION Develop feature BUILD  TEST REVIEW

Slide 66

Slide 66 text

@MOLSJEROEN QUALIFICATION: SPEED UP BUILDS Update all dependencies to latest version Follow best practices Modularize Butterknife reﬂect Gradle build cache and build scans Monitor and keep on experimenting

Slide 67

Slide 67 text

@MOLSJEROEN QUALIFICATION: SPEED UP BUILDS 3m 45s 1m 26s

Slide 68

Slide 68 text

@MOLSJEROEN QUALIFICATION: SPEED UP CI Git hook instead of polling Git cache Preload gradle wrapper in docker container Unit tests over instrumentation tests Run subset of tests Parallel tasks

Slide 69

Slide 69 text

HOW MUCH TIME ARE YOU WILLING TO INVEST IN QUALIFYING PULL REQUESTS?

Slide 70

Slide 70 text

@MOLSJEROEN QUALIFICATION: SPEED UP TESTS Legacy “unit tests” written as instrumentation Converted 1272 tests Build speed decreased by 3m 18s -9% code coverage Removed CI bottleneck Make room for new Espresso tests

Slide 71

Slide 71 text

@MOLSJEROEN QUALIFICATION: SPEED UP BUILDS ~20 min 5m 10s

Slide 72

Slide 72 text

@MOLSJEROEN QUALIFICATION: SPEED UP TESTS Instrumentation tests Unit tests

Slide 73

Slide 73 text

@MOLSJEROEN QUALIFICATION: SPEED UP TESTS Instrumentation tests Unit tests

Slide 74

Slide 74 text

@MOLSJEROEN QUALIFICATION: SPEED UP TESTS … Instrumentation tests Unit tests …

Slide 75

Slide 75 text

@MOLSJEROEN QUALIFICATION: SPEED UP BUILDS (10 PRS) ~1 hour 5m 10s

Slide 76

Slide 76 text

@MOLSJEROEN VERIFICATION Develop feature

Slide 77

Slide 77 text

@MOLSJEROEN VERIFICATION: MOVE TO MASTER Develop feature

Slide 78

Slide 78 text

@MOLSJEROEN VERIFICATION: MOVE TO MASTER 1-3 days 0 days

Slide 79

Slide 79 text

@MOLSJEROEN DEVELOPMENT Develop feature

Slide 80

Slide 80 text

@MOLSJEROEN DEVELOPMENT: SMALLER BRANCHES feature Develop

Slide 81

Slide 81 text

@MOLSJEROEN DEVELOPMENT: SMALLER BRANCHES 1-5 weeks 0,5 - 2 days

Slide 82

Slide 82 text

"IF YOU MERGE EVERY DAY, SUDDENLY YOU NEVER GET TO THE POINT WHERE YOU HAVE HUGE CONFLICTS THAT ARE HARD TO RESOLVE." Linus Torvalds

Slide 83

Slide 83 text

@MOLSJEROEN RECAP: SPEEDING UP INTEGRATION Blazing fast builds A state of the art CI Do veriﬁcation on develop Many small branches => Integration speed up from 1 - 5 weeks to 0,5 - 2days

Slide 84

Slide 84 text

Slide 85

Slide 85 text

REDUCE COMPLEXITY

Slide 86

Slide 86 text

@MOLSJEROEN FEATURE BRANCHES develop master release feature bugﬁx feature hotﬁx

Slide 87

Slide 87 text

@MOLSJEROEN FEATURE BRANCHES: ONLY FROM DEVELOP develop master release feature bugﬁx hotﬁx

Slide 88

Slide 88 text

@MOLSJEROEN DEVELOP BRANCH develop master release feature bugﬁx hotﬁx

Slide 89

Slide 89 text

@MOLSJEROEN DEVELOP BRANCH master release feature bugﬁx hotﬁx

Slide 90

Slide 90 text

ACCIDENTAL VS ESSENTIAL COMPLEXITY

Slide 91

Slide 91 text

@MOLSJEROEN BUILD VARIANTS buildTypes { debug { signingConfig signingConfigs.debug ext.enableCrashlytics = false } release { minifyEnabled true proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro' signingConfig signingConfigs.release } uitest { // provide implementation for idling resources } } flavorDimensions "MODE" productFlavors { daily { applicationId 'com.philips.lighting.hue2.daily' dimension "MODE" buildConfigField "String", "HOCKEY_APP_ID", Keys.hockeyAppDaily buildConfigField "String", "AMPLITUDE_APP_ID", Keys.amplitudeDaily } store { dimension "MODE" buildConfigField "String", "HOCKEY_APP_ID", Keys.hockeyApp buildConfigField "String", "AMPLITUDE_APP_ID", Keys.amplitudeStore } } variantFilter { variant -> def names = variant.flavors*.name if (names.contains("store") && variant.buildType.name == 'debug') { variant.setIgnore(true) } if (names.contains("store") && variant.buildType.name == 'uitest') { variant.setIgnore(true) } }

Slide 92

Slide 92 text

@MOLSJEROEN BUILD VARIANTS Prod API keys Proguard Test settings Idling resource s Leak canary Prod ﬁrmware Develop  Portal Release Release test Debug Espresso

Slide 93

Slide 93 text

@MOLSJEROEN IDLING RESOURCES: UI TEST VARIANT object EspressoIdlingResource { private val idling = BaseIdlingResource() fun increment(code: String, reason: String) = idling.increment(code, reason) fun decrement(code: String, reason: String) = idling.decrement(code, reason) }

Slide 94

Slide 94 text

@MOLSJEROEN IDLING RESOURCES: DEBUG AND RELEASE VARIANT object EspressoIdlingResource { fun increment(code: String, reason: String) { /* NOP */ } fun decrement(code: String, reason: String) { /* NOP */ } }

Slide 95

Slide 95 text

@MOLSJEROEN IDLING RESOURCES: UNIFIED object EspressoIdlingResource { private val idling = BaseIdlingResource() fun increment(code: String, reason: String) = if (RuntimeBehavior.isFeatureEnabled(IDLING_RESOURCES)) { idling.increment(code, reason) } fun decrement(code: String, reason: String) = if (RuntimeBehavior.isFeatureEnabled(IDLING_RESOURCES)) { mCountingIdlingResource.decrement(code reason) } }

Slide 96

Slide 96 text

Slide 97

Slide 97 text

@MOLSJEROEN BUILD VARIANTS Prod API keys Proguard Test settings Idling resource s Leak canary Prod ﬁrmware Develop  Portal Release Release test Debug Espresso

Slide 98

Slide 98 text

@MOLSJEROEN BUILD VARIANTS Prod API keys Proguard Test settings Idling resource s Leak canary Prod ﬁrmware Develop  Portal Release Release test Debug Espresso

Slide 99

Slide 99 text

@MOLSJEROEN RELEASE AND DEBUG

Slide 100

Slide 100 text

@MOLSJEROEN SOURCE FOLDERS

Slide 101

Slide 101 text

@MOLSJEROEN GRADLE CONFIGURATION buildTypes { debug { signingConfig signingConfigs.debug applicationIdSuffix '.daily' ext.enableCrashlytics = false buildConfigField "String", "HOCKEY_APP_ID", Keys.hockeyAppDebug buildConfigField "String", "AMPLITUDE_APP_ID", Keys.amplitudeDebug } release { signingConfig signingConfigs.release minifyEnabled true proguardFiles 'proguard-rules.pro' buildConfigField "String", "HOCKEY_APP_ID", Keys.hockeyAppRelease buildConfigField "String", "AMPLITUDE_APP_ID", Keys.amplitudeRelease }

Slide 102

Slide 102 text

SIMPLICITY IS THE SOUL OF EFFICIENCY Austin Freeman

Slide 103

Slide 103 text

@MOLSJEROEN REDUCE COMPLEXITY: SUMMARY Drop unnecessary branches No branches from branches Minimize build variants Feature ﬂags for dynamic conﬁguration => Remove all accidental complexity

Slide 104

Slide 104 text

@MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST QUALIFY MERGE PROMOTE RELEASE STABILIZE VERIFY SHIP master release feature MERGE BACK 0,5-2 days 6 weeks*

Slide 105

Slide 105 text

@MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST QUALIFY MERGE PROMOTE RELEASE STABILIZE VERIFY RELEASE master release feature MERGE BACK 0,5-2 days ~4 weeks

Slide 106

Slide 106 text

STABILIZE MASTER

Slide 107

Slide 107 text

@MOLSJEROEN FEATURE FLAGS: NEW FEATURES override fun onViewCreated(view: View) { ... if (RuntimeBehavior.isFeatureEnabled(FeatureFlag.POWER_ON_BEHAVIOR)) { power_on_behavior.visibility = View.VISIBLE } else { power_on_behavior.visibility = View.GONE } ... }

Slide 108

Slide 108 text

Slide 109

Slide 109 text

@MOLSJEROEN FEATURE FLAGS: EXISTING CODE fun withLegacy() { var code = LegacyLogic() code.with(OutUnitTests()) // ... code.doMagic() }

Slide 110

Slide 110 text

@MOLSJEROEN FEATURE FLAGS: EXISTING CODE fun withLegacy() { var code = LegacyLogic() code.with(OutUnitTests()) // ... code.doMagic() }

Slide 111

Slide 111 text

@MOLSJEROEN FEATURE FLAGS: EXISTING CODE fun withLegacy() { var code = NewLogic() code.with(UnitTests()) // ... code.doMagic() }

Slide 112

Slide 112 text

@MOLSJEROEN FEATURE FLAGS: EXISTING CODE fun withLegacy() { if (RuntimeBehavior.isFeatureEnabled(FeatureFlag.LOGIC_REFACTOR)) { var code = NewLogic() code.with(UnitTests()) // ... code.doMagic() } else { var code = LegacyLogic() code.with(OutUnitTests()) // ... code.doMagic() } }

Slide 113

Slide 113 text

@MOLSJEROEN

Slide 114

Slide 114 text

@MOLSJEROEN QUALIFY PULL REQUESTS master feature

Slide 115

Slide 115 text

@MOLSJEROEN QUALIFY PULL REQUESTS master feature   BUILD  REVIEW 

Slide 116

Slide 116 text

@MOLSJEROEN QUALIFY PULL REQUESTS master feature   BUILD  REVIEW ANALYZE 

Slide 117

Slide 117 text

@MOLSJEROEN QUALIFY PULL REQUESTS master feature   BUILD  REVIEW  ANALYZE  TEST

Slide 118

Slide 118 text

@MOLSJEROEN AUTOMATE TESTING UNIT INTEGRATION E2E

Slide 119

Slide 119 text

@MOLSJEROEN AUTOMATE TESTING Library Feature Feature Feature Feature Feature Feature Feature Feature Feature App Library Library Lib Lib Library Library Lib

Slide 120

Slide 120 text

@MOLSJEROEN

Slide 121

Slide 121 text

No content

Slide 122

Slide 122 text

@MOLSJEROEN TEST STRATEGY Pull requests: Run all automated level1 app Espresso tests Run all automated tests of the modules that are impacted Manual veriﬁcation if high impact Release: Run all automated Espresso tests Run all automated feature and library tests Deploy: Manual exploratory testing

Slide 123

Slide 123 text

Slide 124

Slide 124 text

@MOLSJEROEN SIMULATE ACTUAL MERGE master feature BUILD  REVIEW  ANALYZE  TEST

Slide 125

Slide 125 text

@MOLSJEROEN SIMULATE ACTUAL MERGE master feature   MERGE BUILD  REVIEW  ANALYZE  TEST

Slide 126

Slide 126 text

@MOLSJEROEN master feature SIMULATE ACTUAL MERGE

Slide 127

Slide 127 text

@MOLSJEROEN master feature SIMULATE ACTUAL MERGE

Slide 128

Slide 128 text

@MOLSJEROEN master feature feature feature SIMULATE ACTUAL MERGE

Slide 129

Slide 129 text

@MOLSJEROEN master feature feature feature SIMULATE ACTUAL MERGE

Slide 130

Slide 130 text

@MOLSJEROEN master feature feature feature SIMULATE ACTUAL MERGE

Slide 131

Slide 131 text

@MOLSJEROEN MERGE PULL REQUESTS master feature   MERGE BUILD  REVIEW  ANALYZE  TEST

Slide 132

Slide 132 text

@MOLSJEROEN MERGE PULL REQUESTS master feature   MERGE BUILD  REVIEW  ANALYZE  TEST IMPACT 

Slide 133

Slide 133 text

@MOLSJEROEN IMPACT ANALYSIS SIDE EFFECTS POSSIBLE IN   OTHER PARTS OF CODE? (REGRESSION) ALL SIDE EFFECTS CONTAINED BEHIND A FEATURE FLAG? LOW IMPACT LOW IMPACT HIGH IMPACT Yes Yes No Not sure Not sure No

Slide 134

Slide 134 text

@MOLSJEROEN IMPACT ANALYSIS

Slide 135

Slide 135 text

@MOLSJEROEN STABILIZE MASTER: SUMMARY Use feature toggles Static code analysis Heavily automate testing Merge master before checks Requalify pull requests after each merge Impact assessment => Able to drop release branch

Slide 136

Slide 136 text

@MOLSJEROEN DEVELOPMENT PROCESS 0,5-2 days 2 weeks FEATURE BRANCH FEATURE FLAG DEVELOP PULL REQUEST MERGE QUALIFY  ++ VERIFY IMPACT master feature BETA MANUAL RELEASE SHIP

Slide 137

Slide 137 text

@MOLSJEROEN DEVELOPMENT PROCESS 0,5-2 days 2 weeks FEATURE BRANCH FEATURE FLAG DEVELOP PULL REQUEST MERGE QUALIFY  ++ VERIFY IMPACT master feature BETA MANUAL RELEASE SHIP

Slide 138

Slide 138 text

AUTOMATE DELIVERY

Slide 139

Slide 139 text

@MOLSJEROEN UPDATE TRANSLATIONS master feature

Slide 140

Slide 140 text

@MOLSJEROEN UPDATE TRANSLATIONS master feature translations

Slide 141

Slide 141 text

@MOLSJEROEN UPDATE TRANSLATIONS master feature translations

Slide 142

Slide 142 text

@MOLSJEROEN UPDATE SDK AND ASSETS master feature translations SDK assets

Slide 143

Slide 143 text

@MOLSJEROEN RELEASE APP (DELIVERY) stage('Deploy to Google Play Internal Test Track') { steps { script { def releaseNotes = readFile 'app/build/release_notes.txt' androidApkUpload( googleCredentialsId: 'Google Play Android Developer', trackName: 'internal', apkFilesPattern: '**/*-release.apk', deobfuscationFilesPattern: '**/mapping.txt', recentChangeList: [[language: 'en-US', text: releaseNotes]] ) } } }

Slide 144

Slide 144 text

@MOLSJEROEN RELEASE APP (DELIVERY) stage('Deploy to HockeyApp') { steps { sh "pipelines/scripts/hockeyapp/deploy_app_debug.sh" } }

Slide 145

Slide 145 text

@MOLSJEROEN IDEAL SCENARIO master feature bugﬁx

Slide 146

Slide 146 text

@MOLSJEROEN IDEAL SCENARIO master feature bugﬁx hotﬁx

Slide 147

Slide 147 text

@MOLSJEROEN IDEAL SCENARIO master feature bugﬁx hotﬁx

Slide 148

Slide 148 text

@MOLSJEROEN IDEAL SCENARIO master feature bugﬁx hotﬁx

Slide 149

Slide 149 text

@MOLSJEROEN ACTUAL SCENARIO master feature bugﬁx hotﬁx

Slide 150

Slide 150 text

@MOLSJEROEN ACTUAL SCENARIO master feature bugﬁx hotﬁx

Slide 151

Slide 151 text

@MOLSJEROEN ACTUAL SCENARIO master feature bugﬁx hotﬁx

Slide 152

Slide 152 text

@MOLSJEROEN ACTUAL SCENARIO master feature bugﬁx hotﬁx

Slide 153

Slide 153 text

@MOLSJEROEN RELEASING master internal test beta production

Slide 154

Slide 154 text

@MOLSJEROEN RELEASING master internal test beta production

Slide 155

Slide 155 text

@MOLSJEROEN AUTOMATE DELIVERY: SUMMARY Integrate dependencies automatically Automate upload to Google play/hockey app Release directly from master Hotﬁxes from separate branch

Slide 156

Slide 156 text

@MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH FEATURE FLAG DEVELOP PULL REQUEST MERGE QUALIFY  ++ master RELEASE VERIFY feature IMPACT 0,5-2 days 2 hours BETA SHIP 1 week*

Slide 157

Slide 157 text

LESSONS LEARNED

Slide 158

Slide 158 text

@MOLSJEROEN 1. BI-WEEKLY RELEASES DEVELOPMENT TESTING TESTING BETA DEVELOPMENT TESTING TESTING BETA DEVELOPMENT TESTING TESTIN 10 weeks

Slide 159

Slide 159 text

@MOLSJEROEN 1. BI-WEEKLY RELEASES DEV & TEST BETA DEV & TEST BETA DEV & TEST BETA 2 weeks

Slide 160

Slide 160 text

@MOLSJEROEN 1. BI-WEEKLY RELEASES BETA DEVELOPMENT TEST TEST BETA DEVELOPMENT TEST TEST BETA DEVELOPMENT TEST TEST

Slide 161

Slide 161 text

@MOLSJEROEN 1. BI-WEEKLY RELEASES BETA DEVELOPMENT TEST TEST BETA DEVELOPMENT TEST TEST BETA DEVELOPMENT TEST TEST

Slide 162

Slide 162 text

@MOLSJEROEN 1. BI-WEEKLY RELEASES DEV & TEST BETA DEV & TEST BETA DEV & TEST BETA 2 weeks DEV & TEST BETA

Slide 163

Slide 163 text

@MOLSJEROEN 2. MERGING WITHOUT VERIFICATION master feature   MERGE BUILD  REVIEW  ANALYZE  TEST IMPACT 

Slide 164

Slide 164 text

@MOLSJEROEN 2. MERGING WITHOUT VERIFICATION

Slide 165

Slide 165 text

@MOLSJEROEN 2. MERGING WITHOUT VERIFICATION

Slide 166

Slide 166 text

TRUST TAKES YEARS TO BUILD, SECONDS TO BREAK, AND FOREVER TO REPAIR. Unknown

Slide 167

Slide 167 text

@MOLSJEROEN 2. MERGING WITHOUT VERIFICATION

Slide 168

Slide 168 text

@MOLSJEROEN 2. MERGING WITHOUT VERIFICATION

Slide 169

Slide 169 text

@MOLSJEROEN 3. YOU WILL MESS UP

Slide 170

Slide 170 text

@MOLSJEROEN SUMMARY Focus on whitebox testing Gain trust and give trust back Don’t blow your trust! Rely on automated testing Closer collaboration between testers and developers Root cause analyses

Slide 171

Slide 171 text

RESULTS

Slide 172

Slide 172 text

@MOLSJEROEN RESULTS 29

Slide 173

Slide 173 text

@MOLSJEROEN TIME TO RELEASE Build time (min) 0 10 20 30 40 50 Build number

Slide 174

Slide 174 text

@MOLSJEROEN

Slide 175

Slide 175 text

@MOLSJEROEN

Slide 176

Slide 176 text

@MOLSJEROEN INTEGRATION SPEED 620 ~1/dev/day

Slide 177

Slide 177 text

@MOLSJEROEN APP RATING

Slide 178

Slide 178 text

@MOLSJEROEN PR INFLOW 0 500 1000 1500 2000 JAN 2018 APRIL 2018 JULY 2018 OCT 2018

Slide 179

Slide 179 text

@MOLSJEROEN PRODUCTIVITY 0 10 20 30 40 50 NOV 2018 DEC 2018 JAN 2019 MARCH 2019

Slide 180

Slide 180 text

@MOLSJEROEN ONE. SINGLE. BRANCH. master feature bugﬁx

Slide 181

Slide 181 text

@MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH FEATURE FLAG DEVELOP PULL REQUEST MERGE QUALIFY  ++ master RELEASE VERIFY feature IMPACT 0,5-2 days 2 hours BETA SHIP 1 week*

Slide 182

Slide 182 text

RELEASING HAS BECOME SO EASY, WE EVEN SHIP FRIDAY EVENINGS Hue android team

Slide 183

Slide 183 text

STRATEGY OF REDUCING COST OF FAILURE IS WORKING

Slide 184

Slide 184 text

WRAP UP

Slide 185

Slide 185 text

“THE MOST POWERFUL TOOL WE HAVE AS DEVELOPERS IS AUTOMATION.” Scott Hanselman

Slide 186

Slide 186 text

@MOLSJEROEN SPEED UP INTEGRATION REMOVE ACCIDENTAL COMPLEXITY STABILIZE MASTER AUTOMATE TESTING AND DELIVERY PREFER TRUST OVER PROCESS

Slide 187

Slide 187 text

@MOLSJEROEN HTTPS://JEROENMOLS.COM/BLOG

Slide 188

Slide 188 text

@MOLSJEROEN IMAGE CREDITS Welcome image by Clement127 https://www.ﬂickr.com/photos/clement127/35754378573/in/photolist-Wtux5Z Material design icons by Google  https://material.io/tools/icons

Slide 189

Slide 189 text

MOLSJEROEN