Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Rolling-out like a Rock-Star

Rolling-out like a Rock-Star

Slides for the JFokus 2021 Talk - Rolling-out like a Rock-Star

Video:
https://www.youtube.com/watch?v=FgFSJ1uBGUQ

Abstract:
Rolling-out a feature is a fine art.

Users are becoming more and more demanding. A single crash could entice them away from your app (and land a shameful 1-star review).

In this context, the possibility of remote controlling your app is a key tool. First, it can protect you against crashes and incidents. Moreover, it can help you experiment on the user experience to fit your users’ tastes.

At Spotify, feature rollout and experimentation are at the foundation of our development. We deliver daily more than a thousand feature flags on a variety of different apps (Android, iOS, and more).

To achieve this, we built an in-house solution to support our experimentation needs. Throughout the years we collected a series of learnings, success stories and pitfalls. In this talk, I will share some of them. Afterwards, you will be able to set the stage for a flawless rollout.

Nicola Corti

June 11, 2021
Tweet

More Decks by Nicola Corti

Other Decks in Technology

Transcript

  1. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 1
    Spotify
    Rolling out like a
    Rock Star
    Nicola Corti
    @cortinico

    View full-size slide

  2. 2
    Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential
    Spotify
    Nicola Corti
    Android Infrastructure Engineer
    twitter.com/cortinico
    github.com/cortinico

    View full-size slide

  3. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 3
    Spotify
    This is… ABBA!

    View full-size slide

  4. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 4
    Spotify
    This is… AB-BA!

    View full-size slide

  5. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 5
    Spotify
    AB-BA
    Our first experimentation framework.
    Unfortunately it was:
    • Hard to Scale
    • All was Global
    • Hard to Maintain
    • Not Flexible enough

    View full-size slide

  6. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 6
    Spotify
    Introducing
    Remote
    Configuration

    View full-size slide

  7. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 7
    Spotify
    Disclaimer
    • A/B Testing & Data Science
    • User Allocation
    • Analytics
    • Event Reporting
    • User Behaviour

    View full-size slide

  8. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 8
    Spotify

    View full-size slide

  9. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 9
    Spotify
    if (lyrics.isEnabled())

    View full-size slide

  10. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 10
    Spotify
    if (lyrics.isEnabled()) {
    // Show Lyrics experience
    }

    View full-size slide

  11. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 11
    Spotify
    if (lyrics.isEnabled()) {
    // Show Lyrics experience
    } else {
    // Show Default experience
    }

    View full-size slide

  12. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 12
    Spotify
    if (lyrics.isEnabled()) {
    // Show Lyrics experience
    } else {
    // Show Default experience
    }
    …not that simple

    View full-size slide

  13. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 13
    Spotify
    Principles

    View full-size slide

  14. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 14
    Spotify
    Safety
    • Always deliver a Consistent
    Configuration
    • Fault Tolerant
    • Offline
    • Default values
    • Developer safety

    View full-size slide

  15. 15
    Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential
    Spotify
    Ubiquity
    • Client SDKs for Android, iOS, Web and C++
    • Multiple platform (Android, iOS)
    • Multiple devices (Phone, Wear, Tablet, Auto)
    • Multiple apps (Music, Stations, Lite, …)

    View full-size slide

  16. 16
    Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential
    Spotify
    Ubiquity
    • Client SDKs for Android, iOS, Web and C++
    • Multiple platform (Android, iOS)
    • Multiple devices (Phone, Wear, Tablet, Auto)
    • Multiple apps (Music, Stations, Lite, …)

    View full-size slide

  17. 17
    Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential
    Spotify
    Simplicity
    • “It should just work”
    • API that is easy to use
    • Code Generation as a rule of thumb
    • Developer Tools to support:
    • Daily work
    • Testing

    View full-size slide

  18. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 18
    Spotify
    Architecture

    View full-size slide

  19. 19
    Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential
    Spotify
    Publish
    Build
    Remote Config Admin Remote Config Resolver
    Resolve
    properties.yaml Experiment/Rollout

    View full-size slide

  20. 20
    Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential
    Spotify
    Publish
    Build
    Remote Config Admin Remote Config Resolver
    Resolve
    properties.yaml Experiment/Rollout

    View full-size slide

  21. 21
    Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential
    Spotify
    backstage.io

    View full-size slide

  22. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 22
    Spotify
    Build

    View full-size slide

  23. 23
    Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential
    Spotify
    Publish
    Build
    Remote Config Admin Remote Config Resolver
    Resolve
    properties.yaml Experiment/Rollout

    View full-size slide

  24. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 24
    Spotify
    A wild properties.yaml appears!

    View full-size slide

  25. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 25
    Spotify
    A wild properties.yaml appears!
    properties:

    View full-size slide

  26. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 26
    Spotify
    A wild properties.yaml appears!
    properties:

    - id: player-module.lyrics_enabled

    desc: Whether Realtime Lyrics are enabled

    boolSpec:

    defaultValue: false

    View full-size slide

  27. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 27
    Spotify
    A wild properties.yaml appears!
    properties:

    - id: player-module.lyrics_enabled

    desc: Whether Realtime Lyrics are enabled

    boolSpec:

    defaultValue: false

    View full-size slide

  28. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 28
    Spotify
    A wild properties.yaml appears!
    properties:

    - id: player-module.lyrics_enabled

    desc: Whether Realtime Lyrics are enabled

    boolSpec:

    defaultValue: false

    View full-size slide

  29. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 29
    Spotify
    A wild properties.yaml appears!
    properties:

    - id: player-module.lyrics_enabled

    desc: Whether Realtime Lyrics are enabled

    boolSpec:

    defaultValue: false

    - id: player-module.hello_message

    desc: What message the user should see

    enumSpec:

    values: [ hello, ciao, hej, hallo ]

    defaultValue: hello

    View full-size slide

  30. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 30
    Spotify
    A wild properties.yaml appears!
    properties:

    - id: player-module.lyrics_enabled

    desc: Whether Realtime Lyrics are enabled

    boolSpec:

    defaultValue: false

    - id: player-module.hello_message

    desc: What message the user should see

    enumSpec:

    values: [ hello, ciao, hej, hallo ]

    defaultValue: hello

    View full-size slide

  31. properties:

    - id: player-module.lyrics_enabled

    desc: Whether realtime lyrics are enabled

    boolSpec:

    defaultValue: false

    - id: player-module.hello_message

    desc: What message the user should see

    enumSpec:

    values: [ hello, ciao, hej, hallo ]

    defaultValue: hello

    - id: player-module.message_size

    desc: The text size for the hello message

    integerSpec:

    lower: 0

    upper: 100

    defaultValue: 42

    Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 31
    Spotify
    A wild properties.yaml appears!

    View full-size slide

  32. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 32
    Spotify
    A wild properties.yaml appears!
    properties:

    - id: player-module.lyrics_enabled

    desc: Whether Realtime Lyrics are enabled

    boolSpec:

    defaultValue: false

    - id: player-module.hello_message

    desc: What message the user should see

    enumSpec:

    values: [ hello, ciao, hej, hallo ]

    defaultValue: hello

    - id: player-module.message_size

    desc: The text size for the hello message

    integerSpec:

    lower: 0

    upper: 100

    defaultValue: 42

    View full-size slide

  33. 33
    Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential
    Spotify
    Building
    properties.yaml
    PlayerProperties.java
    Business Logic
    PlayerProperties
    Module.java
    • Gradle Plugin
    • Code Generation for
    • Properties Classes
    • Dagger Modules
    • Immutability
    • Type Safety
    • No String values
    • Strong Ownership
    • Thanks to Backstage
    generate
    Properties
    player module

    View full-size slide

  34. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 34
    Spotify
    remoteConfig {

    generateDaggerModule = true

    }
    build.gradle
    properties.yaml
    properties:

    - id: player.lyrics_enabled

    desc: Whether Realtime Lyrics are enabled

    boolSpec:

    defaultValue: false

    @Module

    object PlayerModule {

    #// ##...

    }

    PlayerModule.kt

    View full-size slide

  35. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 35
    Spotify
    remoteConfig {

    generateDaggerModule = true

    }
    build.gradle
    properties.yaml
    @Module(

    includes = []

    )

    object PlayerModule {

    #// ##...

    }

    PlayerModule.kt
    properties:

    - id: player.lyrics_enabled

    desc: Whether Realtime Lyrics are enabled

    boolSpec:

    defaultValue: false

    View full-size slide

  36. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 36
    Spotify
    remoteConfig {

    generateDaggerModule = true

    }
    build.gradle
    properties.yaml
    @Module(

    includes = [

    PlayerPropertiesModule#::class,

    ]

    )

    object PlayerModule {

    #// ##...

    }

    PlayerModule.kt
    properties:

    - id: player.lyrics_enabled

    desc: Whether Realtime Lyrics are enabled

    boolSpec:

    defaultValue: false

    View full-size slide

  37. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 37
    Spotify
    remoteConfig {

    generateDaggerModule = true

    }
    build.gradle
    properties.yaml
    @Module(

    includes = [

    PlayerPropertiesModule#::class,

    ]

    )

    object PlayerModule {

    #// ##...

    }

    PlayerModule.kt
    properties:

    - id: player.lyrics_enabled

    desc: Whether Realtime Lyrics are enabled

    boolSpec:

    defaultValue: false

    View full-size slide

  38. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 38
    Spotify
    PlayerPresenter.kt

    View full-size slide

  39. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 39
    Spotify
    PlayerPresenter.kt
    class PlayerPresenter {

    View full-size slide

  40. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 40
    Spotify
    PlayerPresenter.kt
    class PlayerPresenter {

    var properties: PlayerProperties

    View full-size slide

  41. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 41
    Spotify
    PlayerPresenter.kt
    class PlayerPresenter {

    @Inject

    lateinit var properties: PlayerProperties

    View full-size slide

  42. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 42
    Spotify
    PlayerPresenter.kt
    class PlayerPresenter {

    @Inject

    lateinit var properties: PlayerProperties

    fun showPlayer() {

    }

    View full-size slide

  43. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 43
    Spotify
    PlayerPresenter.kt
    class PlayerPresenter {

    @Inject

    lateinit var properties: PlayerProperties

    fun showPlayer() {

    if (properties)

    }

    View full-size slide

  44. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 44
    Spotify
    PlayerPresenter.kt
    class PlayerPresenter {

    @Inject

    lateinit var properties: PlayerProperties

    fun showPlayer() {

    if (properties.lyricsEnabled()) {

    }

    properties.yaml
    properties:

    - id: player.lyrics_enabled

    desc: Whether Realtime Lyrics are enabled

    boolSpec:

    defaultValue: false

    View full-size slide

  45. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 45
    Spotify
    PlayerPresenter.kt
    class PlayerPresenter {

    @Inject

    lateinit var properties: PlayerProperties

    fun showPlayer() {

    if (properties.lyricsEnabled()) {

    #// Show Lyrics Experience

    }

    }

    properties.yaml
    properties:

    - id: player.lyrics_enabled

    desc: Whether Realtime Lyrics are enabled

    boolSpec:

    defaultValue: false

    View full-size slide

  46. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 46
    Spotify
    PlayerPresenter.kt
    class PlayerPresenter {

    @Inject

    lateinit var properties: PlayerProperties

    fun showPlayer() {

    if (properties.lyricsEnabled()) {

    #// Show Lyrics Experience

    } else {

    #// Show Default Experience

    }

    }

    properties.yaml
    properties:

    - id: player.lyrics_enabled

    desc: Whether Realtime Lyrics are enabled

    boolSpec:

    defaultValue: false

    View full-size slide

  47. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 47
    Spotify
    Publish

    View full-size slide

  48. 48
    Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential
    Spotify
    Publish
    Build
    Remote Config Admin Remote Config Resolver
    Resolve
    properties.yaml Experiment/Rollout

    View full-size slide

  49. • Happening on CI
    • No need to define properties twice
    • CI checks to validate
    • Strict Versioning
    • Single source of truth
    49
    Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential
    Spotify
    Publish
    player module
    podcast module
    search module playlist module
    properties.yaml
    properties.yaml
    properties.yaml properties.yaml
    publish
    Properties
    Remote Config Admin

    View full-size slide

  50. • Happening on CI
    • No need to define properties twice
    • CI checks to validate
    • Strict Versioning
    • Single source of truth
    50
    Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential
    Spotify
    Publish
    player module
    podcast module
    search module playlist module
    properties.yaml
    properties.yaml
    properties.yaml properties.yaml
    publish
    Properties
    Remote Config Admin
    C++ Layer
    properties.yaml
    properties.yaml

    View full-size slide

  51. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 51
    Spotify
    Resolve

    View full-size slide

  52. 52
    Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential
    Spotify
    Publish
    Build
    Remote Config Admin Remote Config Resolver
    Resolve
    properties.yaml Experiment/Rollout

    View full-size slide

  53. 53
    Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential
    Spotify
    Resolve
    Client SDK
    player module
    podcast module
    search module playlist module
    C++ Layer
    Remote Config Resolver
    PlayerProperties.java
    CoreProperties.m
    PlaylistProperties.java
    PodcastProperties.java
    SearchProperties.java
    • Fetch Strategy
    • Activate and Fetch
    • Plus recurrent fetch in background

    View full-size slide

  54. 54
    Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential
    Spotify
    Fetch Strategy
    Open the App
    User
    Logged in?
    Fetch & Activate
    Configuration
    Load Remote
    Configuration
    Cached
    Config?
    Activate
    Config
    Activate
    Default Config
    Schedule
    Delayed Fetch
    Remote Config Resolver
    yes
    no
    yes
    no
    First User Session
    Logged In Users

    View full-size slide

  55. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 55
    Spotify
    Lessons Learned

    View full-size slide

  56. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 56
    Spotify
    Correctness
    • Always deliver a Consistent Configuration
    • Make sure you cover every entry-point
    • Deeplink
    • Push notifications
    • Avoid Missbucketing

    View full-size slide

  57. 57
    Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential
    Spotify
    Experiment vs Rollout
    • Experiment
    • Evaluate the user behaviour
    • Collect insights
    • Make informed decision
    • Rollout
    • Test new libraries
    • Gradually release new features
    • Kill switches

    View full-size slide

  58. 58
    Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential
    Spotify
    Propagation Time
    Build the Feature
    Setup the
    Experiment
    New Version
    hits the Play Store
    Enable the
    Experiment
    Users experience
    the Experiment
    Time between you coding the feature and your users experiencing it
    (Days, Weeks)
    Up to 7 days From mins to days

    View full-size slide

  59. 59
    Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential
    Spotify
    Lifecycle & Experiments
    • Startup Experiments: happening early on during the app lifecycle
    • Support for clearing Faulty Configurations
    • Push mechanism
    • Clear configs
    • Speedup Propagation

    View full-size slide

  60. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 60
    Spotify
    Buy vs Build
    • Buy
    • Firebase Remote Config, Rollout.io, Facebook Planout, etc.
    • Cost
    • Speed
    • Build
    • Scalability
    • Flexibility
    • Safety
    • Developer Happiness

    View full-size slide

  61. 61
    Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential
    Spotify
    Thanks
    @cortinico

    View full-size slide

  62. Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 62
    Spotify

    View full-size slide