A/B Testing Evolution

Experimentation and learning are at the heart of how we work at Vinted. This talk tells the story of how we improved our A/B Testing solution.

Transcript

1. A/B TESTING
   EVOLUTION
   

   View Slide

2. MMOZURAS
   mmozuras
   CODINGFEARLESSLY.COM
   

   View Slide

3. View Slide

4. Make SECOND-HAND THE
   FIRST CHOICE WORLDWIDE
   

   View Slide

5. A/B
   TEST
   

   View Slide

6. View Slide

7. View Slide

8. View Slide

9. View Slide

10. EXPERIMENT
    ALL THE THINGS
    

    View Slide

11. EXPERIMENT
    ALL THE THINGS
    MOST OF THE THINGS
    

    View Slide

12. View Slide

13. In THE OLDEN
    DAYS
    

    View Slide

14. View Slide

15. WHO NEEDS
    DOCUMENTATION?
    

    View Slide

16. TESTS = {
    pill: {
    lt: 1.0, us: 0.5
    },
    upld: {
    lt: 0.5, de: 0.2
    }
    }
    

    View Slide

17. TESTS = {
    pill: { us: 1.0 },
    pills: { us: 1.0 },
    }
    if pill? && pills?
    

    View Slide

18. CONTROL TEST GROUPS?
    ISOLATION?
    

    View Slide

19. CONTROL TEST GROUPS?
    ISOLATION? NOPE
    

    View Slide

20. STORING AB TEST
    ASSIGNMENT?
    

    View Slide

21. STORING AB TEST
    ASSIGNMENT? MYSQL
    

    View Slide

22. AB_TESTS
    ID: 1
    VALUE: PILL_1,UPLD_0
    

    View Slide

23. USERS.FIRST_AB_TEST_ID
    USERS.AB_TEST_ID
    LIKES.AB_TEST_ID
    

    View Slide

24. AB: PILL_1,UPLD_0
    

    View Slide

25. STORING EVENTS?
    

    View Slide

26. STORING EVENTS?
    MYSQL
    

    View Slide

27. EVENTS
    uSER_ID: 1
    TYPE: …
    

    View Slide

28. ANALYTICS?
    REPLICATE MYSQL
    

    View Slide

29. EVERYTHING?
    

    View Slide

30. EVERYTHING?!
    MYSQL
    

    View Slide

31. BROUGHT TO YOU BY:
    

    View Slide

32. EASY
    FAMILIAR
    

    View Slide

33. MORE
    MORE
    MORE
    MORE
    MORE
    

    View Slide

34. MORE USERS
    

    View Slide

35. MORE DEVELOPERS
    

    View Slide

36. View Slide

37. KNOWLEDGE SCALING
    

    View Slide

38. MORE AB TESTS
    

    View Slide

39. PILL_1,THPS_0,UPLD_1
    

    View Slide

40. PILL_1,THPS_0,UPLD_1,
    FFIX_0, FSHIP_0,MTG_1,
    WAY_1,OF_1,KINGS_1
    

    View Slide

41. PILL_1,THPS_0,UPLD_1,
    FFIX_0, FSHIP_0,MTG_1,
    WAY_1,OF_1,KINGS_1,
    WORD_1,SOF_0,RAD_1,IAN_1,CE_1,
    STORM_0,LIGHT_1,ARCH_1,IVE_0
    

    View Slide

42. MORE EVENTS
    

    View Slide

43. 1.21 GIGAWATTS
    THOUSANDS OF GIGABYTES
    

    View Slide

44. View Slide

45. BACK TO THE
    FUTURE
    

    View Slide

46. RESEARCH
    DOZENS OF WHITEPAPERS AND PRESENTATIONS
    

    View Slide

47. DOCUMENT IMPLICITLY
    GITHUB ISSUE WITH 70+ COMMENTS
    

    View Slide

48. STORING AB TEST
    ASSIGNMENT?
    

    View Slide

49. View Slide

50. STORING AB TEST
    ASSIGNMENT? NOPE
    

    View Slide

51. SHA256(USERS.ANON_ID)
    

    View Slide

52. VINTED/AB
    

    View Slide

53. View Slide

54. bucket_id =
    SHA256(salt + id) % bucket_count
    return if not
    (test.buckets.include?(bucket_id))
    return if not
    Time.now.between?(test.start_at, test.end_at)
    weight_id =
    SHA256(test.seed + id) % test.weight_sum
    test.variants.find do |v|
    v.accumulated_weight > weight_id
    end
    

    View Slide

55. bucket_id =
    SHA256(salt + id) % bucket_count
    return if not
    (test.buckets.include?(bucket_id))
    return if not
    Time.now.between?(test.start_at, test.end_at)
    weight_id =
    SHA256(test.seed + id) % test.weight_sum
    test.variants.find do |v|
    v.accumulated_weight > weight_id
    end
    

    View Slide

56. bucket_id =
    SHA256(salt + id) % bucket_count
    return if not
    (test.buckets.include?(bucket_id))
    return if not
    Time.now.between?(test.start_at, test.end_at)
    weight_id =
    SHA256(test.seed + id) % test.weight_sum
    test.variants.find do |v|
    v.accumulated_weight > weight_id
    end
    

    View Slide

57. bucket_id =
    SHA256(salt + id) % bucket_count
    return if not
    (test.buckets.include?(bucket_id))
    return if not
    Time.now.between?(test.start_at, test.end_at)
    weight_id =
    SHA256(test.seed + id) % test.weight_sum
    test.variants.find do |v|
    v.accumulated_weight > weight_id
    end
    

    View Slide

58. CONTROL TEST GROUPS?
    ISOLATION?
    

    View Slide

59. CONTROL TEST GROUPS?
    ISOLATION? YES
    

    View Slide

60. ab = Ab::Tests.new(config, id)
    ab.pill.red?
    case ab.pill.variant
    when 'red'
    ‘real world’
    when 'blue'
    ‘fabricated reality’
    end
    

    View Slide

61. VINTED/AB-JAVA
    VINTED/AB
    VINTED/AB-iOS
    

    View Slide

62. {
    "id": 42,
    "name": "pill",
    "start_at": "2014-05-21T11:06:30+0300",
    "end_at": "2014-05-28T11:06:30+0300",
    "variants": [
    {
    "name": "red",
    "chance_weight": 5
    },
    {
    "name": "blue",
    "chance_weight": 2
    }
    ]
    },
    

    View Slide

63. AB CONFIG SERVICE
    

    View Slide

64. ADMIN.vinted.NET/
    EXPERIMENTS
    

    View Slide

65. View Slide

66. STORING EVENTS?
    

    View Slide

67. STORING EVENTS?
    BIG DATA stack
    

    View Slide

68. SIMPLE
    SCALE
    

    View Slide

69. PERFECT
    SOLUTION?
    

    View Slide

70. View Slide

71. NO PERFECT
    SOLUTION
    

    View Slide

72. NO PERFECT
    ARCHITECTURE
    

    View Slide

73. NO PERFECT
    LANGUAGE
    

    View Slide

74. EVOLVE
    

    View Slide

75. THANKS!
    

    View Slide