Have Your Cake and Eat It Too with AB Testing

Presented at Droidcon NYC 2017.

Abstract:
Nobody wants to waste time building features that won’t be successful. Unfortunately, it’s often difficult to anticipate users’ reactions and once your feature is out in the wild, it’s hard to iterate and gauge the right direction to take a failing feature — are users finding it difficult to use? Maybe they just can’t find it within your app? Or maybe the UI doesn’t invoke engagement? Enter A/B testing: try out variations on the same feature with different segments of your user base, make changes on-the-fly without a new release, and have the data to back up what works best for your users. Enable your team to make confident, data-driven decisions to bring about the success of your features. The Capital One Wallet teams have actively been integrating A/B tests into new features and we’ve had successes even with seemly-insignificant A/B tests -- for example, slightly tweaking push notification text through A/B tests has led to a 5-10% increase in our notification clickthru rates! In this talk, you’ll learn the different types of A/B tests which can be conducted and some best practices around A/B testing -- how should you be testing? And when is it appropriate? You’ll learn some of the considerations of A/B testing that come specifically with Android and you’ll also see some examples of using third party libraries -- including Firebase Remote Config and Swrve. Get the best tips on how to choose a third party library to eliminate "gotchas" and even gain insight towards building an in-house solution. Embrace your indecisiveness, try all the possibilities, and maximize the potential of your future features.

Transcript

1. Have Your Cake and Eat It Too with A/B Testing Nick Capurso, @nickcapurso Capital One
2. What am I going to learn? ä What is A/B testing? ä Why should I A/B test? ä How do I A/B test? ⬥ Evaluating third-party libraries ⬦  Firebase Remote Config ⬦  Swrve
4. Cake for developers We want…: ä Scale ä Engaged users ä To build great features ä To use best practices …but also: ä Customized user experiences ä Drive user engagement ä Successful features ä Iterate on features ä Rated highly
5. We want…: ä Scale ä Engaged users ä To build great features ä To use best practices …but also: ä Customized user experiences ä Drive user engagement ä Successful features ä Iterate on features ä Rated highly Cake for developers
6. Answer: A/B Testing We want…: ä Scale ä Engaged users ä To build great features ä To use best practices Enables: ä Customized user experiences ä Drive user engagement ä Successful features ä Iterate on features ä Rated highly
7. What is A/B Testing? Email Password Sign In Sign Up Forgot?
8. What is A/B Testing? Sign Up Email Password Sign In Sign Up Forgot? Email Password Sign In Forgot?
9. What is A/B Testing? Group A Group B Sign Up Email Password Sign In Sign Up Forgot? Email Password Sign In Forgot?
10. What is A/B Testing? Group A Group B 40% 58% Sign Up Email Password Sign In Sign Up Forgot? Email Password Sign In Forgot?
11. What is A/B Testing? Group A Group B 40% 58% Sign Up Email Password Sign In Sign Up Forgot? Email Password Sign In Forgot?
12. What is A/B Testing? Group A Group B Four steps: ä Define a goal ä Segment users into groups ä Deliver different content ä Monitor goal progress
13. What is A/B Testing? Control Variant 1 Four steps: ä Define a goal ä Segment users into groups ä Deliver different content ä Monitor goal progress Variant 2
14. What is A/B Testing? Winning Variant Applied Four steps: ä Define a goal ä Segment users into groups ä Deliver different content ä Monitor goal progress
15. What is A/B Testing? Usually, some extra steps: ä Define a goal ä Segment users into groups ä Deliver different content ä Monitor goal progress ä Iterate your testing ä Hardcode the winner Control Variant 1 Variant 2
16. Example: Notification Permission Group A Group B ä Goal: User opts in to pushes ä Segment: Random percentage ä Deliver: Different titles ä Monitor: Catchier titles == more opt ins?
17. Third Party Support Evaluation & Usage
18. (Remote Config)
19. (Remote Config)
20. Basic Interaction Model Server Content
21. Basic Interaction Model Server Test 1 Test 2 Content
22. Basic Interaction Model SDK init(…)
23. Basic Interaction Model SDK Server Test 1 Test 2 Content getContent (“User A”) init(…)
24. Basic Interaction Model SDK Server Test 1 Test 2 Content
25. Basic Interaction Model SDK GET
26. Basic Interaction Model Server Test 1 SDK “Yes” clicked “Yes” clicked 20% 50% 30% User clicked “Yes”
27. Considerations when Choosing a Third-Party Library A/B Test Preparation ä  SDK Initialization ä  Content Management ä  Analytics ä  User Segmentation ä  A/B Test Setup & Iteration Platform Features ä  User Engagement Mechanisms ä  Manual data upload ä  Data export Other ä  End-User Identity ä  QA Testing
28. Considerations when Choosing a Third-Party Library A/B Test Preparation ä  SDK Initialization ä  Content Management ä  Analytics ä  User Segmentation ä  A/B Test Setup & Iteration Platform Features ä  User Engagement Mechanisms ä  Manual data upload ä  Data export Other ä  End-User Identity ä  QA Testing
29. SDK Initialization ä  When & how is content downloaded?
30. SDK Initialization ä  When & how is content downloaded? FirebaseRemoteConfig.getInstance().fetch(0).addOnCompleteListener(task -> { if (task.isSuccessful()) { firebaseRemoteConfig.activateFetched(); } } );
31. SDK Initialization ä  When & how is content downloaded? SwrveSDK.createInstance(this, 12345, "api_key"); SwrveSDK.setResourcesListener(() -> { // Resources updated });
32. SDK Initialization ä  Initialization during push notifications ⬥  Customize message based on user properties. ⬥  Download times? ⬥  Swrve: Difficult ⬥  Firebase Remote Config: Yes ä  Caching downloads
33. Content Management ä  Location Permission: ⬥  Title ⬥  Image ⬥  Body ⬥  Button
34. Content Organization – Remote Config (Parameters)
35. Content Organization – Remote Config (Parameters)
36. Content Organization -- Swrve (Resources & Attributes)
37. Content Organization -- Swrve (Resources & Attributes)
38. Content Retrieval // Firebase -- supply parameter name FirebaseRemoteConfig.getInstance().getString("permission_location_title"); // Swrve -- supply resource ID, attribute name, default value SwrveSDK.getResourceManager().getAttributeAsString( "permission.location", "title", "We Need Your Location");
39. Content Retrieval // Firebase -- supply parameter name FirebaseRemoteConfig.getInstance().getString("permission_location_title"); // Swrve -- supply resource ID, attribute name, default value SwrveSDK.getResourceManager().getAttributeAsString( "permission.location", "title", "We Need Your Location");
40. Content Retrieval FirebaseRemoteConfig.getInstance().setDefaults(R.xml.config_defaults);
41. Content Management ä Organization ⬥  Remote Config: Flat hierarchy (Parameters) ⬥  Swrve: 2-level hierarchy (Resource -> Attributes)
42. Content Management ä Defaults ⬥  Remote Config: XML or Java Map ⬥  Swrve: Specify at retrieval
43. Content Management ä How to A/B test images? ⬥  Filenames ⬦  No loading needed ⬦  Embed in client ⬥  URLs ⬦  Dynamic ⬦  Need prefetching / loading ⬩  Picasso
44. Content Management ä Text formatting ⬥  Line breaks, bold / italics, etc. ⬦  Nether Swrve nor Remote Config support ⬥  Html.fromHtml(...)
45. Content Management ä Localization ⬥  Remote Config: use the Conditions system (next section) ⬥  Swrve: separate attributes for different languages
46. Content Management ä Character limits ⬥  Swrve: 256 chars per attribute ⬥  Firebase: Project total can’t exceed 500,000.
47. Analytics ä Remote Config – Firebase Analytics ä Swrve – Built-in FirebaseAnalytics.getInstance(this).logEvent("permission_location_granted", null); SwrveSDK.event("permission_location_granted");
48. Analytics ä Maintaining internal analytics ⬥  “Spikes” from A/B tests ⬥  You should supply “A/B test” contextual information Time Count
49. User Segmentation ä Analytics triggered ⬥  “All users who did not log in this week” ä User properties ⬥  “All users who live on the East coast” ä Random percentage ⬥  “Shuffle the userbase into N groups” ⬥  Stability
50. User Segmentation ä Analytics triggered ⬥  “All users who did not log in this week” ä User properties ⬥  “All users who live on the East coast” ä Random percentage ⬥  “Shuffle the userbase into N groups” ⬥  Stability
51. User Segmentation – User Properties // Firebase (note, keys limited to 24 chars) FirebaseAnalytics.getInstance(this).setUserProperty( "favorite_show", "Game of Thrones"); // Swrve Map<String, String> map = new HashMap<>(); map.put("favorite_show", "Game of Thrones"); SwrveSDK.userUpdate(map);
52. User Segmentation – Remote Config Conditions
53. User Segmentation – Analytics (Remote Config)
54. User Segmentation – Remote Config Conditions
55. User Segmentation – Analytics (Swrve)
56. User Segmentation – Swrve User Segments
57. User Segmentation ä Analytics triggered ⬥  “All users who did not log in this week” ä User properties ⬥  “All users who live on the East coast” ä Random percentage ⬥  “Shuffle the userbase into N groups” ⬥  Stability
58. User Segmentation – Random (Remote Config) ä Randomize the userbase and take the first 25%.
59. User Segmentation – Random (Remote Config) ä “Randomization key” acts as the “seed” for the randomization.
60. User Segmentation – Random (Remote Config) ä “Randomization key” acts as the “seed” for the randomization.
61. User Segmentation – Random (Swrve) ä Swrve – built into A/B testing
62. A/B Testing Group A Group B Sign Up Email Password Sign In Sign Up Forgot? Email Password Sign In Forgot?
63. A/B Testing – Swrve
64. A/B Testing – Swrve
65. A/B Testing – Swrve User should contribute at most once
66. A/B Testing – Swrve User can contribute any number of times
67. A/B Test Setup – Swrve
68. A/B Test Setup – Swrve
69. A/B Test Setup – Swrve
70. A/B Test Setup – Swrve
71. A/B Test Setup – Swrve
72. A/B Testing – Remote Config Define Variants with Conditions
73. A/B Testing – Remote Config Apply Conditions to Parameters
74. A/B Testing – Remote Config Apply Conditions to Parameters
75. A/B Testing – Remote Config Problem: Analytics doesn’t know about your A/B test Events, user properties Parameters, conditions, A/B tests
76. A/B Testing – Remote Config Solution: Add a “Test Variant” Parameter -> User Property analytics.setUserProperty("location_test_variant", firebaseRemoteConfig.getString("permission_location_test_variant"));
77. A/B Testing – Remote Config Event View with Filters
78. A/B Testing – Iterate! Time Count
79. A/B Testing – Iterate! Time Count A/B Test v1 Running Low # of users exposed
80. A/B Testing – Iterate! Time Count A/B Test v2 Running More users exposed
81. A/B Testing – Iterate! Time Count A/B Test v2 Running More users exposed Engagement rises accordingly
82. A/B Testing ä Scalability for a large number of tests ⬥  How easy is it to run & organize tests? ⬥  Remote Config: A/B tests are tied into parameters ⬥  Swrve: Dedicated A/B test section
83. A/B Testing ä Flexibility with user segments ⬥  Ability to construct multifaceted user segments ⬥  Remote Config: Conditions ⬥  Swrve: User Segments
84. A/B Testing ä Conversion vs. Engagement test ⬥  Remote Config: Difficult ⬥  Swrve: Built-in
85. A/B Testing ä Choosing & applying a winner ⬥  Swrve: Built-in to A/B testing ⬥  Remote Config: Removing variant conditions & setting default ⬥  Don’t forget to iterate! ⬦  Re-run, but with a different split – is your winning variant still winning? ⬥  Hardcode the winner in your next release ⬦  Remove A/B test code
86. A/B Testing ä Choosing & applying a winner ⬥  Swrve: Built-in to A/B testing ⬥  Remote Config: Removing variant conditions & setting default ⬥  Don’t forget to iterate! ⬦  Re-run, but with a different split – is your winning variant still winning? ⬥  Hardcode the winner in your next release ⬦  Remove A/B test code
87. A/B Testing ä Choosing & applying a winner ⬥  Swrve: Built-in to A/B testing ⬥  Remote Config: Removing variant conditions & setting default ⬥  Don’t forget to iterate! ⬦  Re-run, but with a different split – is your winning variant still winning? ⬥  Hardcode the winner in your next release ⬦  Remove A/B test code
88. Considerations when Choosing a Third-Party Library A/B Test Preparation ä  SDK Initialization ä  Item Management ä  Analytics ä  User Segmentation ä  A/B Test Setup & Iteration Platform Features ä  User Engagement Mechanisms ä  Manual data upload ä  Data export Other ä  End-User Identity ä  QA Testing
89. User Engagement Mechanisms ä Push Notifications ⬥  A/B tested pushes ⬥  Remote Config – tricky, separate campaigns per variant.
90. User Engagement Mechanisms ä Push Notifications ⬥  A/B tested pushes ⬥  Swrve - built into push campaigns.
91. User Engagement Mechanisms ä Push Notifications ⬥  Respect user opt-outs firebaseAnalytics.setUserProperty(“push_opt_out", “TRUE");
92. Manual Data Upload ä Problem: User segmentation for a one-off test. ⬥ i.e. “Target all users who made a purchase at Starbucks last week.”
93. Manual Data Upload ä Problem: User segmentation for a one-off test. ⬥ i.e. “Target all users who made a purchase at Starbucks last week.” ⬦ Devs have to bloat the code ⬦ Confidential data ⬦ Data may not be all in the platform yet
94. Manual Data Upload ä Problem: User segmentation for a one-off test. ⬥ i.e. “Target all users who made a purchase at Starbucks last week.” ⬦ Devs have to bloat the code ⬦ Confidential data ⬦ Data may not be all in the platform yet
95. Manual Data Upload ä Problem: User segmentation for a one-off test. ⬥ i.e. “Target all users who made a purchase at Starbucks last week.” ⬦ Devs have to bloat the code ⬦ Confidential data ⬦ Data may not be all in the platform yet
96. Manual Data Upload ä Problem: User segmentation for a one-off test. ⬥ i.e. “Target all users who made a purchase at Starbucks last week.” ⬦ Devs have to bloat the code ⬦ Confidential data ⬦ Data may not be all in the platform yet
97. Manual Data Upload ä Solution 1: One-off “eligibility” user property firebaseAnalytics.setUserProperty(“special_campaign_1", “TRUE");
98. Manual Data Upload ä Solution 1: One-off “eligibility” user property ä Solution 2: batch upload CSV: ⬥  Supported by Swrve’s REST API firebaseAnalytics.setUserProperty(“special_campaign_1", “TRUE"); PLATFORM USER ID SPECIAL CAMPAIGN 1 ELIGIBLE 18 TRUE 53 TRUE 1408 TRUE
99. Data Export ä Drawing compex conclusions from A/B tested data ⬥  Remote Config: BigQuery ⬥  Both: CSV Exports ä Useful when multiple factors determine “success” ⬥  Feature engagement + feedback + increased login ?
100. Considerations when Choosing a Third-Party Library A/B Test Preparation ä  SDK Initialization ä  Item Management ä  Analytics ä  User Segmentation ä  A/B Test Setup & Iteration Platform Features ä  User Engagement Mechanisms ä  Manual data upload ä  Data export Other ä  End-User Identity ä  QA Testing
101. End-User Identity ä Lots of user-specific data needed. ä Mapping to/from your users.
102. End-User Identity ä Lots of user-specific data needed. ä Mapping to/from your users. Internal APIs ”Nick” A/B Platform “User 12AB”
103. End-User Identity ä Lots of user-specific data needed. ä Mapping to/from your users. ä How do you protect your users’ private information, but also run powerful A/B Tests?
104. End-User Identity ä Leak-safe, correlation ID, used only for third- parties ⬥  Internal ID ßà Correlation ID à Third Party ä Override platform’s ID or use a user property // Firebase FirebaseAnalytics.getInstance(this).setUserId("12345"); // Swrve SwrveSDK.getConfig().setUserId("12345");
105. End-User Identity ä Leak-safe, correlation ID, used only for third- parties ⬥  Internal ID ßà Correlation ID à Third Party ä Override platform’s ID or use a user property // Firebase FirebaseAnalytics.getInstance(this).setUserId("12345"); // Swrve SwrveSDK.getConfig().setUserId("12345");
106. QA Testing ä Easy place device in any A/B test variant ä Verify events & user properties sent
107. QA Testing -- Remote Config ä No easy way to “force” a variant.
108. QA Testing -- Swrve
109. Considerations when Choosing a Third-Party Library A/B Test Preparation ä  SDK Initialization ä  Item Management ä  Analytics ä  User Segmentation ä  A/B Test Setup & Iteration Platform Features ä  User Engagement Mechanisms ä  Manual data upload ä  Data export Other ä  End-User Identity ä  QA Testing
110. Thanks! Any questions? Nick Capurso Capital One @nickcapurso
111. Considerations when Choosing a Third-Party Library A/B Test Preparation ä  SDK Initialization ä  Item Management ä  Analytics ä  User Segmentation ä  A/B Test Setup & Iteration Platform Features ä  User Engagement Mechanisms ä  Manual data upload ä  Data export Other ä  End-User Identity ä  QA Testing Nick Capurso Capital One @nickcapurso
112. Extra – Dynamic Configurations – Remote Config ä The “Remote Config” aspect is great on Firebase!
113. Extra – Dynamic Configurations -- Swrve ä Equivalent for resources not in A/B tests. ä Delivering unique content to segments is tricky.
114. Extra - User Engagement Mechanisms ä Interstitial Screens
115. Thanks! Nick Capurso Capital One @nickcapurso Presentation template by SlidesCarnival