Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Rolling-out like a Rock-Star

Rolling-out like a Rock-Star

Slides for the JFokus 2021 Talk - Rolling-out like a Rock-Star

Video:
https://www.youtube.com/watch?v=FgFSJ1uBGUQ

Abstract:
Rolling-out a feature is a fine art.

Users are becoming more and more demanding. A single crash could entice them away from your app (and land a shameful 1-star review).

In this context, the possibility of remote controlling your app is a key tool. First, it can protect you against crashes and incidents. Moreover, it can help you experiment on the user experience to fit your users’ tastes.

At Spotify, feature rollout and experimentation are at the foundation of our development. We deliver daily more than a thousand feature flags on a variety of different apps (Android, iOS, and more).

To achieve this, we built an in-house solution to support our experimentation needs. Throughout the years we collected a series of learnings, success stories and pitfalls. In this talk, I will share some of them. Afterwards, you will be able to set the stage for a flawless rollout.

3dc29e8cfc6ef333e2b41a1b0e826b57?s=128

Nicola Corti

June 11, 2021
Tweet

More Decks by Nicola Corti

Other Decks in Technology

Transcript

  1. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 1 Spotify

    Rolling out like a Rock Star Nicola Corti @cortinico
  2. 2 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify

    Nicola Corti Android Infrastructure Engineer twitter.com/cortinico github.com/cortinico
  3. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 3 Spotify

    This is… ABBA!
  4. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 4 Spotify

    This is… AB-BA!
  5. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 5 Spotify

    AB-BA Our first experimentation framework. Unfortunately it was: • Hard to Scale • All was Global • Hard to Maintain • Not Flexible enough
  6. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 6 Spotify

    Introducing Remote Configuration
  7. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 7 Spotify

    Disclaimer • A/B Testing & Data Science • User Allocation • Analytics • Event Reporting • User Behaviour
  8. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 8 Spotify

  9. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 9 Spotify

    if (lyrics.isEnabled())
  10. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 10 Spotify

    if (lyrics.isEnabled()) { // Show Lyrics experience }
  11. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 11 Spotify

    if (lyrics.isEnabled()) { // Show Lyrics experience } else { // Show Default experience }
  12. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 12 Spotify

    if (lyrics.isEnabled()) { // Show Lyrics experience } else { // Show Default experience } …not that simple
  13. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 13 Spotify

    Principles
  14. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 14 Spotify

    Safety • Always deliver a Consistent Configuration • Fault Tolerant • Offline • Default values • Developer safety
  15. 15 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify

    Ubiquity • Client SDKs for Android, iOS, Web and C++ • Multiple platform (Android, iOS) • Multiple devices (Phone, Wear, Tablet, Auto) • Multiple apps (Music, Stations, Lite, …)
  16. 16 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify

    Ubiquity • Client SDKs for Android, iOS, Web and C++ • Multiple platform (Android, iOS) • Multiple devices (Phone, Wear, Tablet, Auto) • Multiple apps (Music, Stations, Lite, …)
  17. 17 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify

    Simplicity • “It should just work” • API that is easy to use • Code Generation as a rule of thumb • Developer Tools to support: • Daily work • Testing
  18. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 18 Spotify

    Architecture
  19. 19 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify

    Publish Build Remote Config Admin Remote Config Resolver Resolve properties.yaml Experiment/Rollout
  20. 20 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify

    Publish Build Remote Config Admin Remote Config Resolver Resolve properties.yaml Experiment/Rollout
  21. 21 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify

    backstage.io
  22. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 22 Spotify

    Build
  23. 23 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify

    Publish Build Remote Config Admin Remote Config Resolver Resolve properties.yaml Experiment/Rollout
  24. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 24 Spotify

    A wild properties.yaml appears!
  25. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 25 Spotify

    A wild properties.yaml appears! properties:
  26. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 26 Spotify

    A wild properties.yaml appears! properties: - id: player-module.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false
  27. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 27 Spotify

    A wild properties.yaml appears! properties: - id: player-module.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false
  28. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 28 Spotify

    A wild properties.yaml appears! properties: - id: player-module.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false
  29. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 29 Spotify

    A wild properties.yaml appears! properties: - id: player-module.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false - id: player-module.hello_message desc: What message the user should see enumSpec: values: [ hello, ciao, hej, hallo ] defaultValue: hello
  30. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 30 Spotify

    A wild properties.yaml appears! properties: - id: player-module.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false - id: player-module.hello_message desc: What message the user should see enumSpec: values: [ hello, ciao, hej, hallo ] defaultValue: hello
  31. properties: - id: player-module.lyrics_enabled desc: Whether realtime lyrics are enabled

    boolSpec: defaultValue: false - id: player-module.hello_message desc: What message the user should see enumSpec: values: [ hello, ciao, hej, hallo ] defaultValue: hello - id: player-module.message_size desc: The text size for the hello message integerSpec: lower: 0 upper: 100 defaultValue: 42 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 31 Spotify A wild properties.yaml appears!
  32. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 32 Spotify

    A wild properties.yaml appears! properties: - id: player-module.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false - id: player-module.hello_message desc: What message the user should see enumSpec: values: [ hello, ciao, hej, hallo ] defaultValue: hello - id: player-module.message_size desc: The text size for the hello message integerSpec: lower: 0 upper: 100 defaultValue: 42
  33. 33 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify

    Building properties.yaml PlayerProperties.java Business Logic PlayerProperties Module.java • Gradle Plugin • Code Generation for • Properties Classes • Dagger Modules • Immutability • Type Safety • No String values • Strong Ownership • Thanks to Backstage generate Properties player module
  34. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 34 Spotify

    remoteConfig { generateDaggerModule = true } build.gradle properties.yaml properties: - id: player.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false @Module object PlayerModule { #// ##... } PlayerModule.kt
  35. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 35 Spotify

    remoteConfig { generateDaggerModule = true } build.gradle properties.yaml @Module( includes = [] ) object PlayerModule { #// ##... } PlayerModule.kt properties: - id: player.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false
  36. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 36 Spotify

    remoteConfig { generateDaggerModule = true } build.gradle properties.yaml @Module( includes = [ PlayerPropertiesModule#::class, ] ) object PlayerModule { #// ##... } PlayerModule.kt properties: - id: player.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false
  37. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 37 Spotify

    remoteConfig { generateDaggerModule = true } build.gradle properties.yaml @Module( includes = [ PlayerPropertiesModule#::class, ] ) object PlayerModule { #// ##... } PlayerModule.kt properties: - id: player.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false
  38. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 38 Spotify

    PlayerPresenter.kt
  39. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 39 Spotify

    PlayerPresenter.kt class PlayerPresenter {
  40. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 40 Spotify

    PlayerPresenter.kt class PlayerPresenter { var properties: PlayerProperties
  41. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 41 Spotify

    PlayerPresenter.kt class PlayerPresenter { @Inject lateinit var properties: PlayerProperties
  42. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 42 Spotify

    PlayerPresenter.kt class PlayerPresenter { @Inject lateinit var properties: PlayerProperties fun showPlayer() { }
  43. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 43 Spotify

    PlayerPresenter.kt class PlayerPresenter { @Inject lateinit var properties: PlayerProperties fun showPlayer() { if (properties) }
  44. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 44 Spotify

    PlayerPresenter.kt class PlayerPresenter { @Inject lateinit var properties: PlayerProperties fun showPlayer() { if (properties.lyricsEnabled()) { } properties.yaml properties: - id: player.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false
  45. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 45 Spotify

    PlayerPresenter.kt class PlayerPresenter { @Inject lateinit var properties: PlayerProperties fun showPlayer() { if (properties.lyricsEnabled()) { #// Show Lyrics Experience } } properties.yaml properties: - id: player.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false
  46. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 46 Spotify

    PlayerPresenter.kt class PlayerPresenter { @Inject lateinit var properties: PlayerProperties fun showPlayer() { if (properties.lyricsEnabled()) { #// Show Lyrics Experience } else { #// Show Default Experience } } properties.yaml properties: - id: player.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false
  47. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 47 Spotify

    Publish
  48. 48 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify

    Publish Build Remote Config Admin Remote Config Resolver Resolve properties.yaml Experiment/Rollout
  49. • Happening on CI • No need to define properties

    twice • CI checks to validate • Strict Versioning • Single source of truth 49 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify Publish player module podcast module search module playlist module properties.yaml properties.yaml properties.yaml properties.yaml publish Properties Remote Config Admin
  50. • Happening on CI • No need to define properties

    twice • CI checks to validate • Strict Versioning • Single source of truth 50 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify Publish player module podcast module search module playlist module properties.yaml properties.yaml properties.yaml properties.yaml publish Properties Remote Config Admin C++ Layer properties.yaml properties.yaml …
  51. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 51 Spotify

    Resolve
  52. 52 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify

    Publish Build Remote Config Admin Remote Config Resolver Resolve properties.yaml Experiment/Rollout
  53. 53 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify

    Resolve Client SDK player module podcast module search module playlist module C++ Layer Remote Config Resolver PlayerProperties.java CoreProperties.m PlaylistProperties.java PodcastProperties.java SearchProperties.java • Fetch Strategy • Activate and Fetch • Plus recurrent fetch in background
  54. 54 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify

    Fetch Strategy Open the App User Logged in? Fetch & Activate Configuration Load Remote Configuration Cached Config? Activate Config Activate Default Config Schedule Delayed Fetch Remote Config Resolver yes no yes no First User Session Logged In Users
  55. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 55 Spotify

    Lessons Learned
  56. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 56 Spotify

    Correctness • Always deliver a Consistent Configuration • Make sure you cover every entry-point • Deeplink • Push notifications • Avoid Missbucketing
  57. 57 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify

    Experiment vs Rollout • Experiment • Evaluate the user behaviour • Collect insights • Make informed decision • Rollout • Test new libraries • Gradually release new features • Kill switches
  58. 58 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify

    Propagation Time Build the Feature Setup the Experiment New Version hits the Play Store Enable the Experiment Users experience the Experiment Time between you coding the feature and your users experiencing it (Days, Weeks) Up to 7 days From mins to days
  59. 59 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify

    Lifecycle & Experiments • Startup Experiments: happening early on during the app lifecycle • Support for clearing Faulty Configurations • Push mechanism • Clear configs • Speedup Propagation
  60. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 60 Spotify

    Buy vs Build • Buy • Firebase Remote Config, Rollout.io, Facebook Planout, etc. • Cost • Speed • Build • Scalability • Flexibility • Safety • Developer Happiness
  61. 61 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify

    Thanks @cortinico
  62. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 62 Spotify