[DevDojo] Experiments and feature flags - 2025

This DevDojo presentation introduces experiments as a method to define and compare user experiences using control and treatments within a parameterized app configuration.
It elaborates on how Experiments, Launches, and Remote Configs work together to manage user traffic and dynamically deliver various app experiences to users.
The session concludes by demonstrating how feature flags integrate with the experimentation platform to enable continuous integration, flexible rollouts, and instant releases or rollbacks.

Transcript

1. Experiments and feature flags Igor @iov
2. Experiment lifecycle Hypothesis Design Trial Launch New default Steps you normally go through when running a trial Experiment Launch Remote config Experimentation platform concepts
3. Experiment lifecycle Experiment Experimentation platform concepts • Has control • Has one or more treatments Launch Remote config
4. Experiment lifecycle Experiment Experimentation platform concepts • Has control • Has one or more treatments Control and treatments define user experience. It allows us to try and compare different experiences. That is to experiment with it. Launch Remote config
5. Experiment lifecycle Experiment Experimentation platform concepts • Has control • Has one or more treatments Launch Remote config Fix samples of users in our experiment.
6. Experiment lifecycle Experiment Launch Remote config Experimentation platform concepts • Has control • Has one or more treatments We use control to fix a sample of users who will get the current experience. It is the base to compare against.
7. Experiment lifecycle Experiment Launch Remote config Experimentation platform concepts • Has control • Has one or more treatments We use treatments to fix samples of users who will get new experience. There could be a few variations of the new experience.
8. Experiment: How do we change user experience? Your app configuration defines what experience the user will get. Control and treatments can change that configuration. It is possible because your app is configurable. Sometimes we also say your app is parameterized. Hence the name parameter we use in the platform.
9. Config + Your app = User experience button.setBackgroundColor(parameters .list_button_color) button.setIcon(parameters .list_button_icon) button.setPosition(parameters .list_button_position) Your app Config ... list_button_color = "red", list_button_position = "bottom_left", list_button_icon = "camera", ... User experience
10. Experiment lifecycle Experiment Experimentation platform concepts • Makes the successful treatment available to your users Launch Remote config
11. Experiment lifecycle Experiment Experimentation platform concepts • Defines current user experience Launch Remote config Sometimes we call it the “default” experience. It is how your users experience your app, unless they are in another experiment.
12. Traffic: Experiments, Launches, Remote configs Experiments, Launches and Remote configs work with traffic, that is your app users. They do it slightly differently.
13. Traffic: Experiments, Launches, Remote configs C 50 Experiment Experiment traffic T1 T2 All traffic Control and 2 treatments (C, T1, T2) Treatment ratio is 33%/33%/33% Runs on 20% of all traffic Is rolled out to 50% 50 50 50 20 X Launch Rolls out a single treatment Runs on all traffic but holdout* Is rolled out to 50% All traffic X Launch traffic * Holdouts allow to keep a fixed sample of users out of an experiment Remote config traffic Remote config Runs on all traffic All traffic
14. Your app config Your app config for the purpose of experimentation • Defines user experience • Is defined in terms of parameters
15. Your app config per user With app config and experimentation platform you can change experience per user. Your app default experience
16. Your app config per user Let’s see how your app config is created per user Base Remote config Launches Experiments Final config
17. Your app config per user Let’s see how your app config is created per user Base Remote config Launches Experiments Final config • Comes with your app • 100% reliable • Complete • Often in binary • Not easy to update
18. Your app config per user Let’s see how your app config is created per user Base Remote config Launches Experiments Final config • Is downloaded from the experimentation platform • Defines current “default” user experience • Easy to change
19. Your app config per user Let’s see how your app config is created per user Base Remote config Launches Experiments Final config • New feature is rollout
20. Your app config per user Let’s see how your app config is created per user Base Remote config Launches Experiments Final config • Running experiments
21. Your app config per user Let’s see how your app config is created per user Base Remote config Launches Experiments Final config All of the above are merged to create final user configuration. This is what the end user will experience. merged
22. Experiments, Launches and Remote configs Let’s review the concepts. And look how it all works in real life. RC L E Experiments Launches Remote configs
23. Experiments, Launches and Remote configs Experiments • Run experiments on a subset of all traffic • Run experiments in parallel • Allow multiple treatments • Allow control which experiments users will see at the same time Good fit for running experiments E
24. Experiments, Launches and Remote configs • Apply to all users • Gradually roll out a single treatment • May not overlap • Can be overwritten by experiments Good fit for safely rolling out new features Launches L
25. Experiments, Launches and Remote configs • Apply to all users • No gradual rollout (or on/off rollout) • Configs may overlap • No logs • Can be overwritten by launches and experiments Good fit to define default experience of your app Remote configs RC
26. Experiments, Launches and Remote configs How do they work together? A B C Launch B Config B B D Launch D Config E D E RC L E t Experiments Launches Remote Configs
27. Experiments, Launches and Remote configs How do they work together? A B C Launch B Config B B D Launch D Config E D E RC L E t Experiment E1 starts. It has control A and two treatments B and C. Experiments Launches Remote Configs
28. Experiment E1 ends. The treatment B was successful. We start launching B. Experiments, Launches and Remote configs How do they work together? A B C Launch B Config B B D Launch D Config E D E RC L E t Experiments Launches Remote Configs
29. Experiment E2 starts. We try an improvement in the same area in the app. Experiments, Launches and Remote configs How do they work together? A B C Launch B Config B B D Launch D Config E D E RC L E t Experiments Launches Remote Configs
30. The B feature is launched. We make B the default experience. Experiments, Launches and Remote configs How do they work together? A B C Launch B Config B B D Launch D Config E D E RC L E t Experiments Launches Remote Configs
31. Experiment E2 ends. Treatment D was successful. We start launching D. Experiments, Launches and Remote configs How do they work together? A B C Launch B Config B B D Launch D Config E D E RC L E t Experiments Launches Remote Configs
32. Experiment E3 starts. We try an improvement in the same area in the app. Experiments, Launches and Remote configs How do they work together? A B C Launch B Config B B D Launch D Config E D E RC L E t Experiments Launches Remote Configs
33. The D feature is launched. We make D the default experience. Experiments, Launches and Remote configs How do they work together? A B C Launch B Config B B D Launch D Config E D E RC L E t Experiments Launches Remote Configs
34. Experiments, Launches and Remote configs t’ A B C Launch B Config B B D Launch D Config E D E RC L E t How do they work together? All traffic A B C A B C - Experiment User gets Experiment E1 runs on 80% of all traffic. It is rolled out 100%. 80% Experiments Launches Remote Configs ? ? ? ?
35. How do they work together? A B C Launch B Config B B D Launch D Config E D E RC L E t All traffic B D B D B - Experiment E2 User gets Experiment E2 runs on 50% of all traffic. It is rolled out 100%. Launch B is rolled out to 80%. Launch B 80% 50% t’ Experiments, Launches and Remote configs Experiments Launches Remote Configs ? ? ? ?
36. Experiments, Launches and Remote configs A B C Launch B B D Launch D Config E D E RC L E Experiments Launches Remote Configs t Config B All traffic D E D E D B Experiment E3 User gets Experiment E3 runs on 50% of all traffic. It is rolled out 100%. Launch D is rolled out to 80%. Launch D 80% 50% t’ Config B ? ? ? ?
37. One more way to segment users So far we have segmented users by traffic. We can also segment users by their attributes. We call it targeting. Config B All traffic B - B - User gets iOS, version >= 5 Android, version >= 113 OR Target specific users Config B All traffic B B B B User gets Target all users * * Targeting all users will be not allowed.
38. Experimentation spellbook There is much more to learn about experimentation to become real experimentation wizard. I would like to introduce you to a few more magic spells from the experimentation spellbook. - SRM - Confounding variable - Twyman's law - SUTVA
39. A story Imagine you and your friends are writing a book together. Each person is writing their own chapter, but it's supposed to be one coherent story. t 1 month John is a police officer John is a firefighter John is a chef
40. A story Imagine you and your friends are writing a book together. Each person is writing their own chapter, but it's supposed to be one coherent story. t 1 month John is a police officer John is a firefighter John is a chef
41. Writing a story in isolation leads to … • Stay late trying to merge conflicts • Rewrite large portions of the story • Figure out whose changes should “win” • Fix dozens of characters • And possibly even start over in some areas Integration hell The same happens when we make software code features code
42. A story Authors met EVERY DAY to read their latest changes to each other John wants to be in police
43. A story 1 day 1 day 1 day 1 day t Authors met EVERY DAY to read their latest changes to each other John wants to be in police
44. A story 1 day 1 day 1 day 1 day t "Oh, you made John a police officer? Let's decide right now what his job should be, before we write more." That's much easier than trying to fix it after writing entire chapters! John wants to be in police
45. A story The pain of a small conflict today is much better than the agony of massive conflicts next month.
46. A story Integrate early. Integrate often.
47. In software we call it Continuous Integration or CI You will need tools to do CI properly. But remember, CI is more about discipline.
48. A story of CI, remember … 1 day 1 day 1 day 1 day t 1 month 1 week t Integrate daily Or be ready for hard time
49. Integrate daily you say? It can take a few weeks to finish a feature. - What do we do? If we merge unfinished code - We cannot deploy/release - Our codebase can be broken Right?
50. Not really. We can use feature flags.
51. With feature flags… … the code is integrated, but the feature is off t IF false THEN Your feature in development 40% Day 4
52. 63% With feature flags… … we can integrate continuously, daily, several times a day! IF false THEN Your feature in development t Day 7
53. 76% With feature flags… … we resolve conflicts early and often IF false THEN Your feature in development t Day 11
54. 91% With feature flags… … we can deploy our system and release other features IF false THEN Your feature in development t Day 17
55. Feature flags + experimentation platform That is a good software development practice. But why I am telling you about it? This presentation is about experimentation platform, right?
56. Feature flags + experimentation platform There are more benefits to gain from your new CI discipline if you combine it with the experimentation platform. =
57. Feature flags + experimentation platform With experimentation platform you can to toggle your feature flags in an instant. = Wow! You will become a wizard!
58. Feature flags + experimentation platform It means we can separate deploying code from releasing features. Deploy code and release features Deploy Deploy Deploy Deploy Release Release =
59. One click releases Release in seconds. ON OFF OFF ON Literally just flip the switch.
60. One click releases Release in seconds. ON Literally just flip the switch. OFF ON OFF
61. 13% Gradual rollout Make your feature available gradually (roll it out). ON OFF 91% Feature rollout
62. Gradual rollout Slowly show your feature to the users. Watch metrics, logs, customer inquiries. Should there be any issue, fewer users will be affected. Deploy code and release features Deploy Deploy Deploy Deploy Release Release VS
63. One click rollback Roll back in seconds. Just flip the switch back. ON OFF 91% Feature rollout
64. Feature flags + experimentation platform Happy, calm engineers =
65. Feature flags + experimentation platform Happy, calm engineers Satisfied users =
66. Working on a feature using feature flags Develop the feature Release the feature Remove the feature flag from the code End the feature flag in the platform t Typical process of working on a feature using feature flags and experimentation platform
67. Feature flags. Summary. • Integrate early. Integrate often. • Small conflicts today over massive conflicts next month. • Rollout gradually. • Separate deploy from release. • Release in seconds.
68. Experiments and Feature flags It was a lot! …there is still so much to learn! Exciting adventures await! Onwards! Wow! You are a wizard now!