A/B Testing Evolution

Experimentation and learning are at the heart of how we work at Vinted. This talk tells the story of how we improved our A/B Testing solution.

Transcript

1. A/B TESTING EVOLUTION
2. MMOZURAS mmozuras CODINGFEARLESSLY.COM
4. Make SECOND-HAND THE FIRST CHOICE WORLDWIDE
5. A/B TEST
10. EXPERIMENT ALL THE THINGS
11. EXPERIMENT ALL THE THINGS MOST OF THE THINGS
13. In THE OLDEN DAYS
15. WHO NEEDS DOCUMENTATION?
16. TESTS = { pill: { lt: 1.0, us: 0.5 }, upld: { lt: 0.5, de: 0.2 } }
17. TESTS = { pill: { us: 1.0 }, pills: { us: 1.0 }, } if pill? && pills?
18. CONTROL TEST GROUPS? ISOLATION?
19. CONTROL TEST GROUPS? ISOLATION? NOPE
20. STORING AB TEST ASSIGNMENT?
21. STORING AB TEST ASSIGNMENT? MYSQL
22. AB_TESTS ID: 1 VALUE: PILL_1,UPLD_0
23. USERS.FIRST_AB_TEST_ID USERS.AB_TEST_ID LIKES.AB_TEST_ID
24. AB: PILL_1,UPLD_0
25. STORING EVENTS?
26. STORING EVENTS? MYSQL
27. EVENTS uSER_ID: 1 TYPE: …
28. ANALYTICS? REPLICATE MYSQL
29. EVERYTHING?
30. EVERYTHING?! MYSQL
31. BROUGHT TO YOU BY:
32. EASY FAMILIAR
33. MORE MORE MORE MORE MORE
34. MORE USERS
35. MORE DEVELOPERS
37. KNOWLEDGE SCALING
38. MORE AB TESTS
39. PILL_1,THPS_0,UPLD_1
40. PILL_1,THPS_0,UPLD_1, FFIX_0, FSHIP_0,MTG_1, WAY_1,OF_1,KINGS_1
41. PILL_1,THPS_0,UPLD_1, FFIX_0, FSHIP_0,MTG_1, WAY_1,OF_1,KINGS_1, WORD_1,SOF_0,RAD_1,IAN_1,CE_1, STORM_0,LIGHT_1,ARCH_1,IVE_0
42. MORE EVENTS
43. 1.21 GIGAWATTS THOUSANDS OF GIGABYTES
45. BACK TO THE FUTURE
46. RESEARCH DOZENS OF WHITEPAPERS AND PRESENTATIONS
47. DOCUMENT IMPLICITLY GITHUB ISSUE WITH 70+ COMMENTS
48. STORING AB TEST ASSIGNMENT?
50. STORING AB TEST ASSIGNMENT? NOPE
51. SHA256(USERS.ANON_ID)
52. VINTED/AB
54. bucket_id = SHA256(salt + id) % bucket_count return if not (test.buckets.include?(bucket_id)) return if not Time.now.between?(test.start_at, test.end_at) weight_id = SHA256(test.seed + id) % test.weight_sum test.variants.find do |v| v.accumulated_weight > weight_id end
55. bucket_id = SHA256(salt + id) % bucket_count return if not (test.buckets.include?(bucket_id)) return if not Time.now.between?(test.start_at, test.end_at) weight_id = SHA256(test.seed + id) % test.weight_sum test.variants.find do |v| v.accumulated_weight > weight_id end
56. bucket_id = SHA256(salt + id) % bucket_count return if not (test.buckets.include?(bucket_id)) return if not Time.now.between?(test.start_at, test.end_at) weight_id = SHA256(test.seed + id) % test.weight_sum test.variants.find do |v| v.accumulated_weight > weight_id end
57. bucket_id = SHA256(salt + id) % bucket_count return if not (test.buckets.include?(bucket_id)) return if not Time.now.between?(test.start_at, test.end_at) weight_id = SHA256(test.seed + id) % test.weight_sum test.variants.find do |v| v.accumulated_weight > weight_id end
58. CONTROL TEST GROUPS? ISOLATION?
59. CONTROL TEST GROUPS? ISOLATION? YES
60. ab = Ab::Tests.new(config, id) ab.pill.red? case ab.pill.variant when 'red' ‘real world’ when 'blue' ‘fabricated reality’ end
61. VINTED/AB-JAVA VINTED/AB VINTED/AB-iOS
62. { "id": 42, "name": "pill", "start_at": "2014-05-21T11:06:30+0300", "end_at": "2014-05-28T11:06:30+0300", "variants": [ { "name": "red", "chance_weight": 5 }, { "name": "blue", "chance_weight": 2 } ] },
63. AB CONFIG SERVICE
64. ADMIN.vinted.NET/ EXPERIMENTS
66. STORING EVENTS?
67. STORING EVENTS? BIG DATA stack
68. SIMPLE SCALE
69. PERFECT SOLUTION?
71. NO PERFECT SOLUTION
72. NO PERFECT ARCHITECTURE
73. NO PERFECT LANGUAGE
74. EVOLVE
75. THANKS!