Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A/B Testing Evolution

A/B Testing Evolution

Experimentation and learning are at the heart of how we work at Vinted. This talk tells the story of how we improved our A/B Testing solution.

Mindaugas Mozūras

March 12, 2015
Tweet

More Decks by Mindaugas Mozūras

Other Decks in Programming

Transcript

  1. A/B TESTING
    EVOLUTION

    View Slide

  2. MMOZURAS
    mmozuras
    CODINGFEARLESSLY.COM

    View Slide

  3. View Slide

  4. Make SECOND-HAND THE
    FIRST CHOICE WORLDWIDE

    View Slide

  5. A/B
    TEST

    View Slide

  6. View Slide

  7. View Slide

  8. View Slide

  9. View Slide

  10. EXPERIMENT
    ALL THE THINGS

    View Slide

  11. EXPERIMENT
    ALL THE THINGS
    MOST OF THE THINGS

    View Slide

  12. View Slide

  13. In THE OLDEN
    DAYS

    View Slide

  14. View Slide

  15. WHO NEEDS
    DOCUMENTATION?

    View Slide

  16. TESTS = {
    pill: {
    lt: 1.0, us: 0.5
    },
    upld: {
    lt: 0.5, de: 0.2
    }
    }

    View Slide

  17. TESTS = {
    pill: { us: 1.0 },
    pills: { us: 1.0 },
    }
    if pill? && pills?

    View Slide

  18. CONTROL TEST GROUPS?
    ISOLATION?

    View Slide

  19. CONTROL TEST GROUPS?
    ISOLATION? NOPE

    View Slide

  20. STORING AB TEST
    ASSIGNMENT?

    View Slide

  21. STORING AB TEST
    ASSIGNMENT? MYSQL

    View Slide

  22. AB_TESTS
    ID: 1
    VALUE: PILL_1,UPLD_0

    View Slide

  23. USERS.FIRST_AB_TEST_ID
    USERS.AB_TEST_ID
    LIKES.AB_TEST_ID

    View Slide

  24. AB: PILL_1,UPLD_0

    View Slide

  25. STORING EVENTS?

    View Slide

  26. STORING EVENTS?
    MYSQL

    View Slide

  27. EVENTS
    uSER_ID: 1
    TYPE: …

    View Slide

  28. ANALYTICS?
    REPLICATE MYSQL

    View Slide

  29. EVERYTHING?

    View Slide

  30. EVERYTHING?!
    MYSQL

    View Slide

  31. BROUGHT TO YOU BY:

    View Slide

  32. EASY
    FAMILIAR

    View Slide

  33. MORE
    MORE
    MORE
    MORE
    MORE

    View Slide

  34. MORE USERS

    View Slide

  35. MORE DEVELOPERS

    View Slide

  36. View Slide

  37. KNOWLEDGE SCALING

    View Slide

  38. MORE AB TESTS

    View Slide

  39. PILL_1,THPS_0,UPLD_1

    View Slide

  40. PILL_1,THPS_0,UPLD_1,
    FFIX_0, FSHIP_0,MTG_1,
    WAY_1,OF_1,KINGS_1

    View Slide

  41. PILL_1,THPS_0,UPLD_1,
    FFIX_0, FSHIP_0,MTG_1,
    WAY_1,OF_1,KINGS_1,
    WORD_1,SOF_0,RAD_1,IAN_1,CE_1,
    STORM_0,LIGHT_1,ARCH_1,IVE_0

    View Slide

  42. MORE EVENTS

    View Slide

  43. 1.21 GIGAWATTS
    THOUSANDS OF GIGABYTES

    View Slide

  44. View Slide

  45. BACK TO THE
    FUTURE

    View Slide

  46. RESEARCH
    DOZENS OF WHITEPAPERS AND PRESENTATIONS

    View Slide

  47. DOCUMENT IMPLICITLY
    GITHUB ISSUE WITH 70+ COMMENTS

    View Slide

  48. STORING AB TEST
    ASSIGNMENT?

    View Slide

  49. View Slide

  50. STORING AB TEST
    ASSIGNMENT? NOPE

    View Slide

  51. SHA256(USERS.ANON_ID)

    View Slide

  52. VINTED/AB

    View Slide

  53. View Slide

  54. bucket_id =
    SHA256(salt + id) % bucket_count
    return if not
    (test.buckets.include?(bucket_id))
    return if not
    Time.now.between?(test.start_at, test.end_at)
    weight_id =
    SHA256(test.seed + id) % test.weight_sum
    test.variants.find do |v|
    v.accumulated_weight > weight_id
    end

    View Slide

  55. bucket_id =
    SHA256(salt + id) % bucket_count
    return if not
    (test.buckets.include?(bucket_id))
    return if not
    Time.now.between?(test.start_at, test.end_at)
    weight_id =
    SHA256(test.seed + id) % test.weight_sum
    test.variants.find do |v|
    v.accumulated_weight > weight_id
    end

    View Slide

  56. bucket_id =
    SHA256(salt + id) % bucket_count
    return if not
    (test.buckets.include?(bucket_id))
    return if not
    Time.now.between?(test.start_at, test.end_at)
    weight_id =
    SHA256(test.seed + id) % test.weight_sum
    test.variants.find do |v|
    v.accumulated_weight > weight_id
    end

    View Slide

  57. bucket_id =
    SHA256(salt + id) % bucket_count
    return if not
    (test.buckets.include?(bucket_id))
    return if not
    Time.now.between?(test.start_at, test.end_at)
    weight_id =
    SHA256(test.seed + id) % test.weight_sum
    test.variants.find do |v|
    v.accumulated_weight > weight_id
    end

    View Slide

  58. CONTROL TEST GROUPS?
    ISOLATION?

    View Slide

  59. CONTROL TEST GROUPS?
    ISOLATION? YES

    View Slide

  60. ab = Ab::Tests.new(config, id)
    ab.pill.red?
    case ab.pill.variant
    when 'red'
    ‘real world’
    when 'blue'
    ‘fabricated reality’
    end

    View Slide

  61. VINTED/AB-JAVA
    VINTED/AB
    VINTED/AB-iOS

    View Slide

  62. {
    "id": 42,
    "name": "pill",
    "start_at": "2014-05-21T11:06:30+0300",
    "end_at": "2014-05-28T11:06:30+0300",
    "variants": [
    {
    "name": "red",
    "chance_weight": 5
    },
    {
    "name": "blue",
    "chance_weight": 2
    }
    ]
    },

    View Slide

  63. AB CONFIG SERVICE

    View Slide

  64. ADMIN.vinted.NET/
    EXPERIMENTS

    View Slide

  65. View Slide

  66. STORING EVENTS?

    View Slide

  67. STORING EVENTS?
    BIG DATA stack

    View Slide

  68. SIMPLE
    SCALE

    View Slide

  69. PERFECT
    SOLUTION?

    View Slide

  70. View Slide

  71. NO PERFECT
    SOLUTION

    View Slide

  72. NO PERFECT
    ARCHITECTURE

    View Slide

  73. NO PERFECT
    LANGUAGE

    View Slide

  74. EVOLVE

    View Slide

  75. THANKS!

    View Slide