Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Web界隈LT会_爆速改善_Interleaving.pdf
Search
Yuya Matsumura
July 06, 2018
Science
1
760
Web界隈LT会_爆速改善_Interleaving.pdf
A/Bテストより10-100倍効率的と言われるランキング評価手法、Interleavingについて、ザックリと概要をお話ししました。
Yuya Matsumura
July 06, 2018
Tweet
Share
More Decks by Yuya Matsumura
See All by Yuya Matsumura
LLM活用の現在とこれから:LayerXにおける事例とともに 2025/1 ver. / layerx-llm-202501
yuya4
3
350
バクラクにおけるAI-UXな取り組み 2024年8月 ver. / layerx-bakuraku-ai-ux-202408
yuya4
2
790
事業会社における 機械学習・推薦システム技術の活用事例と必要な能力 / ml-recsys-in-layerx-wantedly-2024
yuya4
4
290
生成AI・LLM時代における 機械学習エンジニアとしてのキャリア戦略・開発戦略 / my-career-and-development-strategies-for-ml-engineer-2024
yuya4
6
2.2k
新卒1年目に勢いで技術コミュニティを立ち上げたことで得たもの / tech-community-iizo-lt
yuya4
2
700
LayerXにおけるAI・機械学習技術の活用事例とその裏側 / layerx-ai-deim2024
yuya4
5
4.8k
LLM時代におけるAI-OCR機能の開発戦略 / layerx-bakuraku-ocr-llm-lt-2024
yuya4
5
3.9k
LLMを活用した爆速アウトプットのすゝめ / bakusoku-outputs-with-llm
yuya4
10
6.3k
【論文紹介】DocTr_ Document Transformer for Structured Information Extraction in Documents / iccv2023-doctr
yuya4
3
830
Other Decks in Science
See All in Science
Planted Clique Conjectures are Equivalent
nobushimi
0
120
メール送信サーバの集約における透過型SMTP プロキシの定量評価 / Quantitative Evaluation of Transparent SMTP Proxy in Email Sending Server Aggregation
linyows
0
670
Online Feedback Optimization
floriandoerfler
0
920
位相的データ解析とその応用例
brainpadpr
1
980
2024-06-16-pydata_london
sofievl
0
600
FOGBoston2024
lcolladotor
0
150
構造設計のための3D生成AI-最新の取り組みと今後の展開-
kojinishiguchi
0
880
ACL読み会2024@名大 REANO: Optimising Retrieval-Augmented Reader Models through Knowledge Graph Generation
takuma_matsubara
0
150
大規模言語モデルの開発
chokkan
PRO
85
43k
20分で分かる Human-in-the-Loop 機械学習におけるアノテーションとヒューマンコンピューターインタラクションの真髄
hurutoriya
5
2.8k
WCS-LA-2024
lcolladotor
0
190
SciPyDataJapan 2025
schwalbe10
0
140
Featured
See All Featured
Git: the NoSQL Database
bkeepers
PRO
427
65k
How to Ace a Technical Interview
jacobian
276
23k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
40
2k
Making Projects Easy
brettharned
116
6k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
44
7k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
32
2.1k
Thoughts on Productivity
jonyablonski
69
4.5k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
21
2.5k
Art, The Web, and Tiny UX
lynnandtonic
298
20k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
10
1.3k
Code Reviewing Like a Champion
maltzj
521
39k
Practical Orchestrator
shlominoach
186
10k
Transcript
©2018 Wantedly, Inc. രվળʂInterleaving ABςετΑΓޮతͳϥϯΩϯάධՁΛٻΊͯ Webք۾LTձ 6.Jul.2018 - Yuya Matsumura
- @yu-ya4
©2018 Wantedly, Inc. •Yuya Matsumuraʢদଜ ༏ʣ •Wantedly, Inc. (ࣾձਓ4ϲ݄) •From
Kyoto •Interested in Information Retrieval & Recommendation Self Introduction @yu-ya4
©2018 Wantedly, Inc. ABςετͯ͠·͔͢ʁ
©2018 Wantedly, Inc. ͋Δಛఆͷظؒʹ2ύλʔϯͷࢪࡦΛ༻ҙ͠ɼͲͪΒͷ ࢪࡦͷํ͕ΑΓߴ͍ޮՌ͕ಘΒΕΔ͔Λݕূ͢Δ͜ͱ ABςετͱ WantedlyͰɼجຊతʹͯ͢ͷมߋʹରͯ͠ABςετΛߦ͔ͬͯΒ ͷਖ਼ࣜϦϦʔεΛߦ͍ͬͯ·͢ʂ
©2018 Wantedly, Inc. Wantedlyʹ͓͚ΔϥϯΩϯάͷABςετ A/B Testing ̍ ̎ ̍
̎ Group A Ranking A Control Variation Group B Ranking B
©2018 Wantedly, Inc. ༗ҙͳ͕ࠩग़Δ·Ͱʹ ABςετͷ՝ →PDCAαΠΫϧΛճ͢ͷ͕͘ͳΔʂ Ex. *ݱࡏCVR 1%ͷͷΛ10%վળ(CVR 1.1%)͢Δʹ134,379ͷαϯϓϧ͕ඞཁ
*༗ҙਫ४Ћ=5%, ݕग़ྗ1-β=95%(https://keisan.casio.jp/exec/user/1489063696) େྔͷαϯϓϧ(impression, PV)͕ඞཁ
©2018 Wantedly, Inc.
©2018 Wantedly, Inc. ͦ͜ͰInterleaving
©2018 Wantedly, Inc. 2ͭͷϥϯΩϯά͔Βަޓʹݕࡧ݁ՌΛ࠷্Ґ͔Βॱʹ બ͠1ͭͷϥϯΩϯάΛੜͯ͠Ϣʔβʹఏࣔ͢Δ Interleavingͱ ̍ ̎ ̍
̎ Ranking A Ranking B ̍ ̎ ̍ ̎ Ranking C ᶃ ᶄ ᶅ ᶆ
©2018 Wantedly, Inc. Interleavingͷੑೳ →രͰPDCAαΠΫϧ͕ճͤΔʂ *ਖ਼֬ʹʮΠϯλʔϦʔϏϯάA/BςετΑΓ100ഒޮతͳ߹͕͋Γɺ͓͓Αͦͷ߹ɺ 10~100ഒޮతͰɺA/BςετΑΓޮతͰͳ͍߹গͳ͍ʯ(Chapelle et al. 2012)
Interleaving ABςετΑΓ10ʙ100ഒޮత*
©2018 Wantedly, Inc.
©2018 Wantedly, Inc. Interleaving͕ABςετΑΓޮతͳཧ༝ ̍ ̎ ̍ ̎
Ranking A Ranking B A/B Testing ̍ Ranking A ͱ Ranking Bͷੑೳ͕ࣅ͍ͯΔ߹ ͕ࠩग़ʹ͍͘ → ̍ ̍ ̍ ̍ ̍
©2018 Wantedly, Inc. Interleaving͕ABςετΑΓޮతͳཧ༝ Interleaving ̍ Ranking A ͱ Ranking
B͕҉ʹൺֱ͞ΕΔͷͰ ͕ࠩग़͍͢ʂʂ → ̍ ̍ ̍ ̍ ̍ ̎ ̍ ̎ ̍
©2018 Wantedly, Inc. ࣮ࡍͷABςετͱInterleavingͷp-valueͷऩଋͷൺֱ (Wantedly VisitͷϥϯΩϯάվળʹͯܭଌ)
©2018 Wantedly, Inc. ͦͷ݁Ռ…
©2018 Wantedly, Inc. ৽ଔೖࣾ1ϲ݄ͰWantedly Visitͷ ৽͍͠ϥϯΩϯάࢪࡦΛຊ൪ϦϦʔε (ࠓͰܧଓͯ͠ྑ͍݁ՌΛग़͠ଓ͚͍ͯ·͢)
©2018 Wantedly, Inc. InterleavingΛ׆༻ͯ͠രϥϯΩϯάվળΛ