A/B Testing Evolution

A/B Testing Evolution

Experimentation and learning are at the heart of how we work at Vinted. This talk tells the story of how we improved our A/B Testing solution.

B5009f255a6f0c386d40d6143948df8e?s=128

Mindaugas Mozūras

March 12, 2015
Tweet

Transcript

  1. A/B TESTING EVOLUTION

  2. MMOZURAS mmozuras CODINGFEARLESSLY.COM

  3. None
  4. Make SECOND-HAND THE FIRST CHOICE WORLDWIDE

  5. A/B TEST

  6. None
  7. None
  8. None
  9. None
  10. EXPERIMENT ALL THE THINGS

  11. EXPERIMENT ALL THE THINGS MOST OF THE THINGS

  12. None
  13. In THE OLDEN DAYS

  14. None
  15. WHO NEEDS DOCUMENTATION?

  16. TESTS = { pill: { lt: 1.0, us: 0.5 },

    upld: { lt: 0.5, de: 0.2 } }
  17. TESTS = { pill: { us: 1.0 }, pills: {

    us: 1.0 }, } if pill? && pills?
  18. CONTROL TEST GROUPS? ISOLATION?

  19. CONTROL TEST GROUPS? ISOLATION? NOPE

  20. STORING AB TEST ASSIGNMENT?

  21. STORING AB TEST ASSIGNMENT? MYSQL

  22. AB_TESTS ID: 1 VALUE: PILL_1,UPLD_0

  23. USERS.FIRST_AB_TEST_ID USERS.AB_TEST_ID LIKES.AB_TEST_ID

  24. AB: PILL_1,UPLD_0

  25. STORING EVENTS?

  26. STORING EVENTS? MYSQL

  27. EVENTS uSER_ID: 1 TYPE: …

  28. ANALYTICS? REPLICATE MYSQL

  29. EVERYTHING?

  30. EVERYTHING?! MYSQL

  31. BROUGHT TO YOU BY:

  32. EASY FAMILIAR

  33. MORE MORE MORE MORE MORE

  34. MORE USERS

  35. MORE DEVELOPERS

  36. None
  37. KNOWLEDGE SCALING

  38. MORE AB TESTS

  39. PILL_1,THPS_0,UPLD_1

  40. PILL_1,THPS_0,UPLD_1, FFIX_0, FSHIP_0,MTG_1, WAY_1,OF_1,KINGS_1

  41. PILL_1,THPS_0,UPLD_1, FFIX_0, FSHIP_0,MTG_1, WAY_1,OF_1,KINGS_1, WORD_1,SOF_0,RAD_1,IAN_1,CE_1, STORM_0,LIGHT_1,ARCH_1,IVE_0

  42. MORE EVENTS

  43. 1.21 GIGAWATTS THOUSANDS OF GIGABYTES

  44. None
  45. BACK TO THE FUTURE

  46. RESEARCH DOZENS OF WHITEPAPERS AND PRESENTATIONS

  47. DOCUMENT IMPLICITLY GITHUB ISSUE WITH 70+ COMMENTS

  48. STORING AB TEST ASSIGNMENT?

  49. None
  50. STORING AB TEST ASSIGNMENT? NOPE

  51. SHA256(USERS.ANON_ID)

  52. VINTED/AB

  53. None
  54. bucket_id = SHA256(salt + id) % bucket_count return if not

    (test.buckets.include?(bucket_id)) return if not Time.now.between?(test.start_at, test.end_at) weight_id = SHA256(test.seed + id) % test.weight_sum test.variants.find do |v| v.accumulated_weight > weight_id end
  55. bucket_id = SHA256(salt + id) % bucket_count return if not

    (test.buckets.include?(bucket_id)) return if not Time.now.between?(test.start_at, test.end_at) weight_id = SHA256(test.seed + id) % test.weight_sum test.variants.find do |v| v.accumulated_weight > weight_id end
  56. bucket_id = SHA256(salt + id) % bucket_count return if not

    (test.buckets.include?(bucket_id)) return if not Time.now.between?(test.start_at, test.end_at) weight_id = SHA256(test.seed + id) % test.weight_sum test.variants.find do |v| v.accumulated_weight > weight_id end
  57. bucket_id = SHA256(salt + id) % bucket_count return if not

    (test.buckets.include?(bucket_id)) return if not Time.now.between?(test.start_at, test.end_at) weight_id = SHA256(test.seed + id) % test.weight_sum test.variants.find do |v| v.accumulated_weight > weight_id end
  58. CONTROL TEST GROUPS? ISOLATION?

  59. CONTROL TEST GROUPS? ISOLATION? YES

  60. ab = Ab::Tests.new(config, id) ab.pill.red? case ab.pill.variant when 'red' ‘real

    world’ when 'blue' ‘fabricated reality’ end
  61. VINTED/AB-JAVA VINTED/AB VINTED/AB-iOS

  62. { "id": 42, "name": "pill", "start_at": "2014-05-21T11:06:30+0300", "end_at": "2014-05-28T11:06:30+0300", "variants":

    [ { "name": "red", "chance_weight": 5 }, { "name": "blue", "chance_weight": 2 } ] },
  63. AB CONFIG SERVICE

  64. ADMIN.vinted.NET/ EXPERIMENTS

  65. None
  66. STORING EVENTS?

  67. STORING EVENTS? BIG DATA stack

  68. SIMPLE SCALE

  69. PERFECT SOLUTION?

  70. None
  71. NO PERFECT SOLUTION

  72. NO PERFECT ARCHITECTURE

  73. NO PERFECT LANGUAGE

  74. EVOLVE

  75. THANKS!