Developer Productivity On a Budget

Transcript

1. Jordan Terry — Senior Android Engineer at Marks & Spencer - @jordanterry Developer Productivity On a Budget Adam Ahmed - Senior Android Engineer at TravelPerk — @oheyadam
2. Background
3. Background •We both work on Platform/Foundations teams
4. Background •We both work on Platform/Foundations teams •Mid-sized mobile orgs (30+ Engineers)
5. Background •We both work on Platform/Foundations teams •Mid-sized mobile orgs (30+ Engineers) •Our teams’ focus is improving developer productivity
6. Background •We both work on Platform/Foundations teams •Mid-sized mobile orgs (30+ Engineers) •Our teams’ focus is improving developer productivity •The things we’ll talk about were born of necessity to our organizations
7. What We’ll Cover
8. What We’ll Cover •Automating bug reporting and triaging
9. What We’ll Cover •Automating bug reporting and triaging •Rolling out convention plugins
10. What We’ll Cover •Automating bug reporting and triaging •Rolling out convention plugins •Automating our release trains
11. What We’ll Cover •Automating bug reporting and triaging •Rolling out convention plugins •Automating our release trains •Reporting build times
12. What We’ll Cover •Automating bug reporting and triaging •Rolling out convention plugins •Automating our release trains •Reporting build times •Improving our CI workflows
13. What We’ll Cover •Automating bug reporting and triaging •Rolling out convention plugins •Automating our release trains •Reporting build times •Improving our CI workflows •Linting!
14. Automating bug reporting and triaging @oheyadam
15. What was the problem here? @oheyadam
16. What was the problem here? •Different squads own different parts of the code-base @oheyadam
17. What was the problem here? •Different squads own different parts of the code-base •We wanted a way to immediately find out when a crash happens @oheyadam
18. What was the problem here? •Different squads own different parts of the code-base •We wanted a way to immediately find out when a crash happens •Each squad has their own Jira board @oheyadam
19. What did we try? @oheyadam
20. What did we try? •Crashlytics Jira integration @oheyadam
21. What did we try? •Crashlytics Jira integration •Slack notifications @oheyadam
22. What did we try? •Crashlytics Jira integration •Slack notifications •Assigning fire-fighters @oheyadam
23. What did we try? •Crashlytics Jira integration •Slack notifications •Assigning fire-fighters •Rolling out our own tool! @oheyadam
24. What does it do? @oheyadam
25. What does it do? •Runs as a cron job @oheyadam
26. What does it do? •Runs as a cron job •Checks if there are any new crashes @oheyadam
27. What does it do? •Runs as a cron job •Checks if there are any new crashes •Parses stack-traces and figures out who the owner is @oheyadam
28. What does it do? •Runs as a cron job •Checks if there are any new crashes •Parses stack-traces and figures out who the owner is •Talks to OpenAI to try to suggest a solution @oheyadam
29. What does it do? •Runs as a cron job •Checks if there are any new crashes •Parses stack-traces and figures out who the owner is •Talks to OpenAI to try to suggest a solution •Opens a detailed Jira ticket on their board @oheyadam
30. What does it do? •Sends them a Slack notification @oheyadam
31. What does it do? •Sends them a Slack notification •Allows us to customize bug reporting logic for teams @oheyadam
32. What does it do? •Sends them a Slack notification •Allows us to customize bug reporting logic for teams •Allows us to do the same for other tools like Datadog @oheyadam
33. What does it do? •Sends them a Slack notification •Allows us to customize bug reporting logic for teams •Allows us to do the same for other tools like Datadog •Works for both platforms @oheyadam
34. @oheyadam
35. Convention plugins @jordanterry
36. Problem Build configuration contained large amounts of copy and pasted duplication. As we modularised more, this would become a maintenance problem. @jordanterry
37. Solution The platform team wanted to ensure we could scale our build configuration to achieve our modularisation goals. To achieve this, we adopted Gradle convention plugins. @jordanterry
38. State of the build files @jordanterry
39. State of the build files @jordanterry compileOptions { sourceCompatibility = JavaVersion.VERSION_1_8 targetCompatibility = JavaVersion.VERSION_1_8 }
40. State of the build files @jordanterry compileOptions { sourceCompatibility '1.8' targetCompatibility '1.8' }
41. State of the build files @jordanterry plugins { id 'com.android.library' id 'org.jetbrains.kotlin.android' }
42. State of the build files @jordanterry android { compileSdkVersion rootProject.compileSdkVersion defaultConfig { minSdkVersion rootProject.minSdkVersion targetSdkVersion rootProject.targetSdkVersion } }
43. State of the build files @jordanterry • Duplications
44. State of the build files @jordanterry • Duplications • Inconsistencies
45. State of the build files @jordanterry • Duplications • Inconsistencies • Crucial applied scripts
46. State of the build files @jordanterry • Duplications • Inconsistencies • Crucial applied scripts • Maintenance headaches
47. The Goal plugins { alias(libs.plugins.mns.android.application) } @jordanterry
48. The Goal plugins { alias(libs.plugins.mns.android.application) } android { namespace = “com.mns.some.module” } @jordanterry
49. The Goal plugins { alias(libs.plugins.mns.android.application) } android { namespace = “com.mns.some.module” } dependencies { implementation(libs.some.dependency) } @jordanterry
50. The two types of convention - Script import com.android.build.gradle.LibraryExtension plugins { id("com.android.library") kotlin("android") } // Configure Android configuration, including extensions.getByType<LibraryExtension>().apply { compileSdk = 33 defaultConfig { setMinSdkVersion(23) setTargetSdkVersion("33") testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner" } compileOptions { targetCompatibility = JavaVersion.VERSION_1_8 sourceCompatibility = JavaVersion.VERSION_1_8 } kotlinOptions { jvmTarget = JavaVersion.VERSION_1_8.toString() } } @jordanterry
51. The two types of convention - A plugin class KotlinAndroidLibraryPlugin : Plugin<Project> { override fun apply(target: Project) { with(target) { pluginManager.apply("com.android.library") pluginManager.apply("org.jetbrains.kotlin.android") // Configure Android configuration. extensions.getByType<LibraryExtension>().apply { compileSdk = ANDROID_COMPILE_SDK lint { @Suppress("UnstableApiUsage") baseline = file("baseline-lint.xml") } configureAndroidExtension(ANDROID_MIN_SDK, ANDROID_TARGET_SDK) } configureJavaJvm(JAVA_LANGUAGE_VERSION) configureKotlinJvm(JAVA_LANGUAGE_VERSION) configureJunit() configureSpotless() configureJacocoAndroid(extensions.getByType<LibraryAndroidComponentsExtension>()) } } } @jordanterry
52. What were the benefits? @jordanterry
53. What were the benefits? • Removed 1,000s of lines of build configuration from the project @jordanterry
54. What were the benefits? @jordanterry
55. What were the benefits? • Removed 1,000s of lines of build configuration from the project • We can roll out new build features with low effort @jordanterry
56. What were the benefits? @jordanterry
57. What were the benefits? • Removed 1,000s of lines of build configuration from the project • We can roll out new build features with low effort • We have unit tests, functional and integration tests for our builds
58. What were the benefits? @jordanterry
59. State of a build file now plugins { alias(libs.plugins.mns.android.library) } android { namespace = "com.marksandspencer.androidfeatureflaglib" testOptions { unitTests { isReturnDefaultValues = true } } } dependencies { implementation(project(":data:logging:api")) implementation(libs.coroutines) implementation(libs.lifecycle.common.java8) testImplementation(project(":testutils")) testImplementation(libs.bundles.junit5.kotlin) testImplementation(libs.arch.core.testing) testImplementation(libs.mockk) testImplementation(libs.truth) testImplementation(libs.coroutines.test) testRuntimeOnly(libs.bundles.junit5.runtime) }
60. Step by step? - Move to a script import com.android.build.gradle.LibraryExtension plugins { id("com.android.library") kotlin("android") } // Configure Android configuration, including extensions.getByType<LibraryExtension>().apply { compileSdk = 33 defaultConfig { setMinSdkVersion(23) setTargetSdkVersion("33") testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner" } compileOptions { targetCompatibility = JavaVersion.VERSION_1_8 sourceCompatibility = JavaVersion.VERSION_1_8 } kotlinOptions { jvmTarget = JavaVersion.VERSION_1_8.toString() } } @jordanterry
61. Step by step? - Move to a script import com.android.build.gradle.LibraryExtension plugins { id("com.android.library") kotlin("android") } // Configure Android configuration, including extensions.getByType<LibraryExtension>().apply { compileSdk = 33 defaultConfig { setMinSdkVersion(23) setTargetSdkVersion("33") testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner" } compileOptions { targetCompatibility = JavaVersion.VERSION_1_8 sourceCompatibility = JavaVersion.VERSION_1_8 } kotlinOptions { jvmTarget = JavaVersion.VERSION_1_8.toString() } } @jordanterry
62. Lessons learned @jordanterry
63. Lessons learned • Evangelise for idiomatic Gradle concepts @jordanterry
64. Lessons learned • Evangelise for idiomatic Gradle concepts • Don’t keep your conventions in the build-logic module forever @jordanterry
65. Lessons learned • Evangelise for idiomatic Gradle concepts • Don’t keep your conventions in the build-logic module forever • Think carefully about what you include in your conventions @jordanterry
66. Who and why would benefit? @jordanterry
67. Who and why would benefit? @jordanterry •The smaller development team • Ease modularisation growing pains
68. Who and why would benefit? @jordanterry •The medium sized development team • More devs can do their work without worrying about build configuration • Helps the devs help the business
69. Who and why would benefit? @jordanterry •The larger team • Why don’t you have a platform team?
70. Other resources
71. Other resources •Herding Elephants - https:// developer.squareup.com/blog/herding- elephants/
72. Other resources •Herding Elephants - https:// developer.squareup.com/blog/herding- elephants/ •Stampeding Elephants - https:// developer.squareup.com/blog/stampeding- elephants/
73. Other resources •Herding Elephants - https:// developer.squareup.com/blog/herding- elephants/ •Stampeding Elephants - https:// developer.squareup.com/blog/stampeding- elephants/ •Now in Android - https://github.com/ android/nowinandroid
74. Other resources •Herding Elephants - https:// developer.squareup.com/blog/herding-elephants/ •Stampeding Elephants - https:// developer.squareup.com/blog/stampeding-elephants/ •Now in Android - https://github.com/android/ nowinandroid •Idiomatic Android - https://github.com/jjohannes/ idiomatic-gradle
75. Other resources •Herding Elephants - https://developer.squareup.com/blog/ herding-elephants/ •Stampeding Elephants - https://developer.squareup.com/ blog/stampeding-elephants/ •Now in Android - https://github.com/android/nowinandroid •Idiomatic Android - https://github.com/jjohannes/ idiomatic-gradle •Gradle Docs - https://docs.gradle.org/current/userguide/ userguide.html
76. Automating our release trains @oheyadam
77. What was the problem? @oheyadam
78. What was the problem? Preparing for a new release is a lengthy manual process @oheyadam
79. Preparing for a new release is a lengthy manual process • Announce a code freeze @oheyadam
80. Preparing for a new release is a lengthy manual process • Announce a code freeze • Find and notify the release captain @oheyadam
81. Preparing for a new release is a lengthy manual process • Announce a code freeze • Find and notify the release captain • Notify everyone else in the mobile org @oheyadam
82. Preparing for a new release is a lengthy manual process • Announce a code freeze • Find and notify the release captain • Notify everyone else in the mobile org • Create a release branch, update version codes, download latest strings @oheyadam
83. Preparing for a new release is a lengthy manual process • Create a release branch, update version codes, download latest strings • Document the release version and notes somewhere @oheyadam
84. Preparing for a new release is a lengthy manual process • Create a release branch, update version codes, download latest strings • Document the release version and notes somewhere • Ensure tests pass and QA is done @oheyadam
85. Preparing for a new release is a lengthy manual process • Ensure tests pass and QA is done • Upload to Play Store @oheyadam
86. Preparing for a new release is a lengthy manual process • Ensure tests pass and QA is done • Upload to Play Store • Potentially have to repeat some of these steps if someone pushes to the release branch again, or wants to create a hot-fix @oheyadam
87. What did we do? @oheyadam
88. What did we do? Automate it! @oheyadam
89. Automate it! We drive the release stages using Calendar events, Zapier, Fastlane, and our CI provider @oheyadam
90. We drive the release stages using Calendar events, Zapier, Fastlane, and our CI provider @oheyadam
91. We drive the release stages using Calendar events, Zapier, Fastlane, and our CI provider @oheyadam
92. We drive the release stages using Calendar events, Zapier, Fastlane, and our CI provider @oheyadam
93. We drive the release stages using Calendar events, Zapier, Fastlane, and our CI provider @oheyadam
94. We drive the release stages using Calendar events, Zapier, Fastlane, and our CI provider @oheyadam
95. We drive the release stages using Calendar events, Zapier, Fastlane, and our CI provider @oheyadam
96. We drive the release stages using Calendar events, Zapier, Fastlane, and our CI provider @oheyadam
97. We drive the release stages using Calendar events, Zapier, Fastlane, and our CI provider @oheyadam
98. We drive the release stages using Calendar events, Zapier, Fastlane, and our CI provider @oheyadam
99. We drive the release stages using Calendar events, Zapier, Fastlane, and our CI provider @oheyadam
100. Reporting build times @jordanterry
101. Problem How long do our builds take? @jordanterry
102. Problem How long do our builds take? ..err, I don’t know @jordanterry
103. Solution Build a plugin to report build data to our data engine. This data can now be used to build dashboards, or be directly used in our OKRs. @jordanterry
104. How? @jordanterry
105. How? @jordanterry
106. How? @jordanterry
107. How? - Gluing it together class ObservabilityPlugin : Plugin<Project> { override fun apply(target: Project) { val plugin = TalaiotBasePlugin() plugin.apply(target) target.extensions.configure(BaseExtension::class.java) { publishers { customPublishers(NewRelicBuildPublisher(…)) } } } } @jordanterry
108. Dashboarding! - Total build time @jordanterry
109. Dashboarding! - Time Spent @jordanterry
110. Dashboarding! - Successful Builds @jordanterry
111. Does this help developer productivity? @jordanterry
112. Does this help developer productivity? • Not directly @jordanterry
113. Does this help developer productivity? • Not directly • Dashboarding and alerting can act as a canary for bad changes, or celebrations for good changes @jordanterry
114. Does this help developer productivity? • Not directly • Dashboarding and alerting can act as a canary for bad changes, or celebrations for good changes • Data helps build a case for contributing time towards platform initiatives @jordanterry
115. The elephant in the room @jordanterry
116. The elephant in the room • Gradle Enterprise (Devlocity) is the best tool for this @jordanterry
117. The elephant in the room • Gradle Enterprise (Devlocity) is the best tool for this • Incredible insights into your builds @jordanterry
118. The elephant in the room • Gradle Enterprise (Devlocity) is the best tool for this • Incredible insights into your builds • https://gradle.com/develocity/ @jordanterry
119. Improving our CI workflows @oheyadam
120. What was the problem here? @oheyadam
121. What was the problem here? •CI builds take too long @oheyadam
122. What was the problem here? •CI builds take too long •PRs stay up for a long time @oheyadam
123. What was the problem here? •CI builds take too long •PRs stay up for a long time •Product iterations are slow @oheyadam
124. What was the problem here? •CI builds take too long •PRs stay up for a long time •Product iterations are slow •Mobile org doesn’t use the same CI provider as the rest of the company @oheyadam
125. What did we do? @oheyadam
126. What did we do? •Investigated our current CI setup @oheyadam
127. What did we do? •Investigated our current CI setup •Migrated to a different provider @oheyadam
128. What did we do? •Investigated our current CI setup •Migrated to a different provider @oheyadam
129. What did we do? •Investigated our current CI setup •Migrated to a different provider •CCI allowed us to tap into existing knowledge and infrastructure @oheyadam
130. What did we do? •Investigated our current CI setup •Migrated to a different provider •CCI allowed us to tap into existing knowledge and infrastructure •Utilized remote build cache on our CI @oheyadam
131. What did we do? •Utilized remote build cache on our CI @oheyadam
132. What did we do? •Utilized remote build cache on our CI •Utilized CI parallelism @oheyadam
133. What did we do? •Utilized CI parallelism @oheyadam
134. What did we do? •Utilized CI parallelism •Fan-out @oheyadam
135. What did we do? •Fan-out @oheyadam
136. What did we do? •Utilized CI parallelism •Fan-out •Removed any unnecessary steps / downloads @oheyadam
137. What did we do? •Utilized CI parallelism •Fan-out •Removed any unnecessary steps / downloads •Reduced the execution time of our PR checks by up to 80% @oheyadam
138. What did we do? •Reduced the execution time of our PR checks by up to 80% •Reduced the execution time of our nightly / E2E tests / release jobs by up to 60% @oheyadam
139. What did we do? •Reduced the execution time of our PR checks by up to 80% •Reduced the execution time of our nightly / E2E tests / release jobs by up to 60% •Happy, productive developers! @oheyadam
140. Linting! @jordanterry
141. Problem How can we make our code consistent? Whilst keeping the developer experience consistent? @jordanterry
142. Solution Lint everything! Adopt Spotless + KtLint, Android Linter, other tooling. @jordanterry
143. The goal @jordanterry
144. The goal • Reduce bike shedding in pull requests @jordanterry
145. The goal • Reduce bike shedding in pull requests • Keep the feedback loop as low as possible @jordanterry
146. The goal • Reduce bike shedding in pull requests • Keep the feedback loop as low as possible •Pick up as many issues as early as possible @jordanterry
147. Spotless + KtLint https://github.com/di ff plug/spotless/tree/main/plugin-gradle • Adopt Spotless Gradle plugin • Adopt KtLint • Highlights issues • Autofixes issues @jordanterry
148. Spotless + KtLint https://github.com/di ff plug/spotless/tree/main/plugin-gradle • Adopt Spotless Gradle plugin @jordanterry
149. Spotless + KtLint https://github.com/di ff plug/spotless/tree/main/plugin-gradle • Adopt Spotless Gradle plugin • Adopt KtLint • Highlights issues • Autofixes issues @jordanterry
150. How we rolled it out @jordanterry
151. How we rolled it out • Integrate Spotless into our conventions plugins @jordanterry
152. How we rolled it out • Integrate Spotless into our conventions plugins • Write some tests! @jordanterry
153. How we rolled it out • Integrate Spotless into our conventions plugins • Write some tests! • Roll out throughout codebase with a version bump @jordanterry
154. Android Linting @jordanterry
155. Android Linting http://googlesamples.github.io/android-custom-lint-rules/book.md.html • Baseline your project @jordanterry
156. Android Linting http://googlesamples.github.io/android-custom-lint-rules/book.md.html • Baseline your project • Upload the lint reports to the CI @jordanterry
157. Android Linting http://googlesamples.github.io/android-custom-lint-rules/book.md.html • Baseline your project • Upload the lint reports to the CI • Write your own lint rules @jordanterry
158. Linting Gradle? @jordanterry
159. Linting Gradle? • Lint your dependencies • Removing unused dependencies helps your project • Be aware of dependency changes @jordanterry
160. Linting Gradle? • Lint your dependencies • Removing unused dependencies helps your project • Be aware of dependency changes • Lint your Gradle plugins @jordanterry
161. Move to the IDE! @jordanterry
162. Move to the IDE! • Use IntelliJ’s built in functionality @jordanterry
163. Move to the IDE! • Use IntelliJ’s built in functionality • Find tooling that provides plugins @jordanterry
164. Move to the IDE! • Use IntelliJ’s built in functionality • Find tooling that provides plugins • IDEs have the quickest feedback loop @jordanterry
165. Move to the IDE! • Use IntelliJ’s built in functionality • Find tooling that provides plugins • IDEs have the quickest feedback loop • Change your CI to a validation tool @jordanterry
166. Why should you do this? @jordanterry
167. Who should do this? Everyone @jordanterry
168. We’re Hiring!
169. Thank you!