Rolling-out like a Rock-Star

Slides for the JFokus 2021 Talk - Rolling-out like a Rock-Star

Video:
https://www.youtube.com/watch?v=FgFSJ1uBGUQ

Abstract:
Rolling-out a feature is a fine art.

Users are becoming more and more demanding. A single crash could entice them away from your app (and land a shameful 1-star review).

In this context, the possibility of remote controlling your app is a key tool. First, it can protect you against crashes and incidents. Moreover, it can help you experiment on the user experience to fit your users’ tastes.

At Spotify, feature rollout and experimentation are at the foundation of our development. We deliver daily more than a thousand feature flags on a variety of different apps (Android, iOS, and more).

To achieve this, we built an in-house solution to support our experimentation needs. Throughout the years we collected a series of learnings, success stories and pitfalls. In this talk, I will share some of them. Afterwards, you will be able to set the stage for a flawless rollout.

Transcript

1. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 1 Spotify Rolling out like a Rock Star Nicola Corti @cortinico
2. 2 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify Nicola Corti Android Infrastructure Engineer twitter.com/cortinico github.com/cortinico
3. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 3 Spotify This is… ABBA!
4. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 4 Spotify This is… AB-BA!
5. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 5 Spotify AB-BA Our first experimentation framework. Unfortunately it was: • Hard to Scale • All was Global • Hard to Maintain • Not Flexible enough
6. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 6 Spotify Introducing Remote Configuration
7. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 7 Spotify Disclaimer • A/B Testing & Data Science • User Allocation • Analytics • Event Reporting • User Behaviour
8. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 8 Spotify
9. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 9 Spotify if (lyrics.isEnabled())
10. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 10 Spotify if (lyrics.isEnabled()) { // Show Lyrics experience }
11. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 11 Spotify if (lyrics.isEnabled()) { // Show Lyrics experience } else { // Show Default experience }
12. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 12 Spotify if (lyrics.isEnabled()) { // Show Lyrics experience } else { // Show Default experience } …not that simple
13. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 13 Spotify Principles
14. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 14 Spotify Safety • Always deliver a Consistent Configuration • Fault Tolerant • Offline • Default values • Developer safety
15. 15 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify Ubiquity • Client SDKs for Android, iOS, Web and C++ • Multiple platform (Android, iOS) • Multiple devices (Phone, Wear, Tablet, Auto) • Multiple apps (Music, Stations, Lite, …)
16. 16 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify Ubiquity • Client SDKs for Android, iOS, Web and C++ • Multiple platform (Android, iOS) • Multiple devices (Phone, Wear, Tablet, Auto) • Multiple apps (Music, Stations, Lite, …)
17. 17 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify Simplicity • “It should just work” • API that is easy to use • Code Generation as a rule of thumb • Developer Tools to support: • Daily work • Testing
18. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 18 Spotify Architecture
19. 19 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify Publish Build Remote Config Admin Remote Config Resolver Resolve properties.yaml Experiment/Rollout
20. 20 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify Publish Build Remote Config Admin Remote Config Resolver Resolve properties.yaml Experiment/Rollout
21. 21 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify backstage.io
22. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 22 Spotify Build
23. 23 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify Publish Build Remote Config Admin Remote Config Resolver Resolve properties.yaml Experiment/Rollout
24. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 24 Spotify A wild properties.yaml appears!
25. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 25 Spotify A wild properties.yaml appears! properties:
26. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 26 Spotify A wild properties.yaml appears! properties: - id: player-module.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false
27. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 27 Spotify A wild properties.yaml appears! properties: - id: player-module.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false
28. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 28 Spotify A wild properties.yaml appears! properties: - id: player-module.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false
29. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 29 Spotify A wild properties.yaml appears! properties: - id: player-module.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false - id: player-module.hello_message desc: What message the user should see enumSpec: values: [ hello, ciao, hej, hallo ] defaultValue: hello
30. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 30 Spotify A wild properties.yaml appears! properties: - id: player-module.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false - id: player-module.hello_message desc: What message the user should see enumSpec: values: [ hello, ciao, hej, hallo ] defaultValue: hello
31. properties: - id: player-module.lyrics_enabled desc: Whether realtime lyrics are enabled boolSpec: defaultValue: false - id: player-module.hello_message desc: What message the user should see enumSpec: values: [ hello, ciao, hej, hallo ] defaultValue: hello - id: player-module.message_size desc: The text size for the hello message integerSpec: lower: 0 upper: 100 defaultValue: 42 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 31 Spotify A wild properties.yaml appears!
32. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 32 Spotify A wild properties.yaml appears! properties: - id: player-module.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false - id: player-module.hello_message desc: What message the user should see enumSpec: values: [ hello, ciao, hej, hallo ] defaultValue: hello - id: player-module.message_size desc: The text size for the hello message integerSpec: lower: 0 upper: 100 defaultValue: 42
33. 33 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify Building properties.yaml PlayerProperties.java Business Logic PlayerProperties Module.java • Gradle Plugin • Code Generation for • Properties Classes • Dagger Modules • Immutability • Type Safety • No String values • Strong Ownership • Thanks to Backstage generate Properties player module
34. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 34 Spotify remoteConfig { generateDaggerModule = true } build.gradle properties.yaml properties: - id: player.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false @Module object PlayerModule { #// ##... } PlayerModule.kt
35. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 35 Spotify remoteConfig { generateDaggerModule = true } build.gradle properties.yaml @Module( includes = [] ) object PlayerModule { #// ##... } PlayerModule.kt properties: - id: player.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false
36. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 36 Spotify remoteConfig { generateDaggerModule = true } build.gradle properties.yaml @Module( includes = [ PlayerPropertiesModule#::class, ] ) object PlayerModule { #// ##... } PlayerModule.kt properties: - id: player.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false
37. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 37 Spotify remoteConfig { generateDaggerModule = true } build.gradle properties.yaml @Module( includes = [ PlayerPropertiesModule#::class, ] ) object PlayerModule { #// ##... } PlayerModule.kt properties: - id: player.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false
38. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 38 Spotify PlayerPresenter.kt
39. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 39 Spotify PlayerPresenter.kt class PlayerPresenter {
40. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 40 Spotify PlayerPresenter.kt class PlayerPresenter { var properties: PlayerProperties
41. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 41 Spotify PlayerPresenter.kt class PlayerPresenter { @Inject lateinit var properties: PlayerProperties
42. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 42 Spotify PlayerPresenter.kt class PlayerPresenter { @Inject lateinit var properties: PlayerProperties fun showPlayer() { }
43. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 43 Spotify PlayerPresenter.kt class PlayerPresenter { @Inject lateinit var properties: PlayerProperties fun showPlayer() { if (properties) }
44. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 44 Spotify PlayerPresenter.kt class PlayerPresenter { @Inject lateinit var properties: PlayerProperties fun showPlayer() { if (properties.lyricsEnabled()) { } properties.yaml properties: - id: player.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false
45. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 45 Spotify PlayerPresenter.kt class PlayerPresenter { @Inject lateinit var properties: PlayerProperties fun showPlayer() { if (properties.lyricsEnabled()) { #// Show Lyrics Experience } } properties.yaml properties: - id: player.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false
46. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 46 Spotify PlayerPresenter.kt class PlayerPresenter { @Inject lateinit var properties: PlayerProperties fun showPlayer() { if (properties.lyricsEnabled()) { #// Show Lyrics Experience } else { #// Show Default Experience } } properties.yaml properties: - id: player.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false
47. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 47 Spotify Publish
48. 48 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify Publish Build Remote Config Admin Remote Config Resolver Resolve properties.yaml Experiment/Rollout
49. • Happening on CI • No need to define properties twice • CI checks to validate • Strict Versioning • Single source of truth 49 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify Publish player module podcast module search module playlist module properties.yaml properties.yaml properties.yaml properties.yaml publish Properties Remote Config Admin
50. • Happening on CI • No need to define properties twice • CI checks to validate • Strict Versioning • Single source of truth 50 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify Publish player module podcast module search module playlist module properties.yaml properties.yaml properties.yaml properties.yaml publish Properties Remote Config Admin C++ Layer properties.yaml properties.yaml …
51. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 51 Spotify Resolve
52. 52 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify Publish Build Remote Config Admin Remote Config Resolver Resolve properties.yaml Experiment/Rollout
53. 53 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify Resolve Client SDK player module podcast module search module playlist module C++ Layer Remote Config Resolver PlayerProperties.java CoreProperties.m PlaylistProperties.java PodcastProperties.java SearchProperties.java • Fetch Strategy • Activate and Fetch • Plus recurrent fetch in background
54. 54 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify Fetch Strategy Open the App User Logged in? Fetch & Activate Configuration Load Remote Configuration Cached Config? Activate Config Activate Default Config Schedule Delayed Fetch Remote Config Resolver yes no yes no First User Session Logged In Users
55. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 55 Spotify Lessons Learned
56. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 56 Spotify Correctness • Always deliver a Consistent Configuration • Make sure you cover every entry-point • Deeplink • Push notifications • Avoid Missbucketing
57. 57 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify Experiment vs Rollout • Experiment • Evaluate the user behaviour • Collect insights • Make informed decision • Rollout • Test new libraries • Gradually release new features • Kill switches
58. 58 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify Propagation Time Build the Feature Setup the Experiment New Version hits the Play Store Enable the Experiment Users experience the Experiment Time between you coding the feature and your users experiencing it (Days, Weeks) Up to 7 days From mins to days
59. 59 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify Lifecycle & Experiments • Startup Experiments: happening early on during the app lifecycle • Support for clearing Faulty Configurations • Push mechanism • Clear configs • Speedup Propagation
60. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 60 Spotify Buy vs Build • Buy • Firebase Remote Config, Rollout.io, Facebook Planout, etc. • Cost • Speed • Build • Scalability • Flexibility • Safety • Developer Happiness
61. 61 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify Thanks @cortinico
62. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 62 Spotify