Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Web界隈LT会_爆速改善_Interleaving.pdf
Search
Yuya Matsumura
July 06, 2018
Science
0
690
Web界隈LT会_爆速改善_Interleaving.pdf
A/Bテストより10-100倍効率的と言われるランキング評価手法、Interleavingについて、ザックリと概要をお話ししました。
Yuya Matsumura
July 06, 2018
Tweet
Share
More Decks by Yuya Matsumura
See All by Yuya Matsumura
生成AI・LLM時代における 機械学習エンジニアとしてのキャリア戦略・開発戦略 / my-career-and-development-strategies-for-ml-engineer-2024
yuya4
5
1.4k
新卒1年目に勢いで技術コミュニティを立ち上げたことで得たもの / tech-community-iizo-lt
yuya4
2
490
LayerXにおけるAI・機械学習技術の活用事例とその裏側 / layerx-ai-deim2024
yuya4
4
3.1k
LLM時代におけるAI-OCR機能の開発戦略 / layerx-bakuraku-ocr-llm-lt-2024
yuya4
5
3.1k
LLMを活用した爆速アウトプットのすゝめ / bakusoku-outputs-with-llm
yuya4
10
5.6k
【論文紹介】DocTr_ Document Transformer for Structured Information Extraction in Documents / iccv2023-doctr
yuya4
3
580
【論文紹介】gSASRec_Reducing Overconfidence in Sequential Recommendation Trained with Negative Sampling / recsys2023-gsasrec
yuya4
0
470
「機械学習」という技術で価値を創出する技術 〜プロダクト開発に機械学習を応用する際に考えること〜.pdf / techniques-to-create-values-with-ml-layerx-at-wi2
yuya4
15
8.1k
バクラクのAI-OCR機能の体験を支える良質なデータセット作成の仕組み / data-centric-ai-bakuraku-dataset
yuya4
2
14k
Other Decks in Science
See All in Science
構造活性フォーラム2023-山﨑担当分
yamasakih
0
310
脳とAIは似ているか ― NeuroAI の挑戦
ykamit
9
6.8k
スポーツメトリクス設計に対比較法を使いまくる / Sports metrics design using pairwise comparison method (spoana#14)
konakalab
1
710
Machine Learning for Materials (Lecture 8)
aronwalsh
0
320
Snowflake上でRを使う: RStudioセットアップとShinyアプリケーションのデプロイ
ktatsuya
0
100
Machine Learning for Materials (Lecture 2)
aronwalsh
0
570
Spark_Task_Optimization_Journey_How_I_Increased_10x_Speed_by_Performance_Tuning
tlyu0419
0
200
AI Alignment: A Comprehensive Survey
s_ota
0
180
OptimizationNight~機械学習と数理最適化の融合~
hidenari
0
290
FIBA W杯の日本代表って組み合わせ次第で2次ラウンド行けたんじゃね?をデータで検証
saltcooky12
0
200
Презентация программы бакалавриата СПбГУ "Искусственный интеллект и наука о данных"
dscs
0
100
2023-07-18_Verge_Genomics
lcolladotor
0
110
Featured
See All Featured
The World Runs on Bad Software
bkeepers
PRO
61
6.7k
Navigating Team Friction
lara
177
13k
GraphQLとの向き合い方2022年版
quramy
31
12k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
9
8.3k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
243
20k
Docker and Python
trallard
33
2.7k
StorybookのUI Testing Handbookを読んだ
zakiyama
12
4.6k
Designing on Purpose - Digital PM Summit 2013
jponch
110
6.4k
The Cost Of JavaScript in 2023
addyosmani
15
3.8k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
39
4.4k
Building Flexible Design Systems
yeseniaperezcruz
318
37k
Code Review Best Practice
trishagee
54
15k
Transcript
©2018 Wantedly, Inc. രվળʂInterleaving ABςετΑΓޮతͳϥϯΩϯάධՁΛٻΊͯ Webք۾LTձ 6.Jul.2018 - Yuya Matsumura
- @yu-ya4
©2018 Wantedly, Inc. •Yuya Matsumuraʢদଜ ༏ʣ •Wantedly, Inc. (ࣾձਓ4ϲ݄) •From
Kyoto •Interested in Information Retrieval & Recommendation Self Introduction @yu-ya4
©2018 Wantedly, Inc. ABςετͯ͠·͔͢ʁ
©2018 Wantedly, Inc. ͋Δಛఆͷظؒʹ2ύλʔϯͷࢪࡦΛ༻ҙ͠ɼͲͪΒͷ ࢪࡦͷํ͕ΑΓߴ͍ޮՌ͕ಘΒΕΔ͔Λݕূ͢Δ͜ͱ ABςετͱ WantedlyͰɼجຊతʹͯ͢ͷมߋʹରͯ͠ABςετΛߦ͔ͬͯΒ ͷਖ਼ࣜϦϦʔεΛߦ͍ͬͯ·͢ʂ
©2018 Wantedly, Inc. Wantedlyʹ͓͚ΔϥϯΩϯάͷABςετ A/B Testing ̍ ̎ ̍
̎ Group A Ranking A Control Variation Group B Ranking B
©2018 Wantedly, Inc. ༗ҙͳ͕ࠩग़Δ·Ͱʹ ABςετͷ՝ →PDCAαΠΫϧΛճ͢ͷ͕͘ͳΔʂ Ex. *ݱࡏCVR 1%ͷͷΛ10%վળ(CVR 1.1%)͢Δʹ134,379ͷαϯϓϧ͕ඞཁ
*༗ҙਫ४Ћ=5%, ݕग़ྗ1-β=95%(https://keisan.casio.jp/exec/user/1489063696) େྔͷαϯϓϧ(impression, PV)͕ඞཁ
©2018 Wantedly, Inc.
©2018 Wantedly, Inc. ͦ͜ͰInterleaving
©2018 Wantedly, Inc. 2ͭͷϥϯΩϯά͔Βަޓʹݕࡧ݁ՌΛ࠷্Ґ͔Βॱʹ બ͠1ͭͷϥϯΩϯάΛੜͯ͠Ϣʔβʹఏࣔ͢Δ Interleavingͱ ̍ ̎ ̍
̎ Ranking A Ranking B ̍ ̎ ̍ ̎ Ranking C ᶃ ᶄ ᶅ ᶆ
©2018 Wantedly, Inc. Interleavingͷੑೳ →രͰPDCAαΠΫϧ͕ճͤΔʂ *ਖ਼֬ʹʮΠϯλʔϦʔϏϯάA/BςετΑΓ100ഒޮతͳ߹͕͋Γɺ͓͓Αͦͷ߹ɺ 10~100ഒޮతͰɺA/BςετΑΓޮతͰͳ͍߹গͳ͍ʯ(Chapelle et al. 2012)
Interleaving ABςετΑΓ10ʙ100ഒޮత*
©2018 Wantedly, Inc.
©2018 Wantedly, Inc. Interleaving͕ABςετΑΓޮతͳཧ༝ ̍ ̎ ̍ ̎
Ranking A Ranking B A/B Testing ̍ Ranking A ͱ Ranking Bͷੑೳ͕ࣅ͍ͯΔ߹ ͕ࠩग़ʹ͍͘ → ̍ ̍ ̍ ̍ ̍
©2018 Wantedly, Inc. Interleaving͕ABςετΑΓޮతͳཧ༝ Interleaving ̍ Ranking A ͱ Ranking
B͕҉ʹൺֱ͞ΕΔͷͰ ͕ࠩग़͍͢ʂʂ → ̍ ̍ ̍ ̍ ̍ ̎ ̍ ̎ ̍
©2018 Wantedly, Inc. ࣮ࡍͷABςετͱInterleavingͷp-valueͷऩଋͷൺֱ (Wantedly VisitͷϥϯΩϯάվળʹͯܭଌ)
©2018 Wantedly, Inc. ͦͷ݁Ռ…
©2018 Wantedly, Inc. ৽ଔೖࣾ1ϲ݄ͰWantedly Visitͷ ৽͍͠ϥϯΩϯάࢪࡦΛຊ൪ϦϦʔε (ࠓͰܧଓͯ͠ྑ͍݁ՌΛग़͠ଓ͚͍ͯ·͢)
©2018 Wantedly, Inc. InterleavingΛ׆༻ͯ͠രϥϯΩϯάվળΛ