Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Web界隈LT会_爆速改善_Interleaving.pdf
Search
Yuya Matsumura
July 06, 2018
Science
0
720
Web界隈LT会_爆速改善_Interleaving.pdf
A/Bテストより10-100倍効率的と言われるランキング評価手法、Interleavingについて、ザックリと概要をお話ししました。
Yuya Matsumura
July 06, 2018
Tweet
Share
More Decks by Yuya Matsumura
See All by Yuya Matsumura
バクラクにおけるAI-UXな取り組み 2024年8月 ver. / layerx-bakuraku-ai-ux-202408
yuya4
3
530
事業会社における 機械学習・推薦システム技術の活用事例と必要な能力 / ml-recsys-in-layerx-wantedly-2024
yuya4
3
170
生成AI・LLM時代における 機械学習エンジニアとしてのキャリア戦略・開発戦略 / my-career-and-development-strategies-for-ml-engineer-2024
yuya4
6
1.9k
新卒1年目に勢いで技術コミュニティを立ち上げたことで得たもの / tech-community-iizo-lt
yuya4
2
600
LayerXにおけるAI・機械学習技術の活用事例とその裏側 / layerx-ai-deim2024
yuya4
5
3.9k
LLM時代におけるAI-OCR機能の開発戦略 / layerx-bakuraku-ocr-llm-lt-2024
yuya4
5
3.6k
LLMを活用した爆速アウトプットのすゝめ / bakusoku-outputs-with-llm
yuya4
10
6k
【論文紹介】DocTr_ Document Transformer for Structured Information Extraction in Documents / iccv2023-doctr
yuya4
3
710
【論文紹介】gSASRec_Reducing Overconfidence in Sequential Recommendation Trained with Negative Sampling / recsys2023-gsasrec
yuya4
0
690
Other Decks in Science
See All in Science
バックアップ『しながら』ランサムウェア検出も!? セキュリティ強化が満載 Veeam 12.1
climbteam
0
610
20分で分かる Human-in-the-Loop 機械学習におけるアノテーションとヒューマンコンピューターインタラクションの真髄
hurutoriya
3
1.5k
拡散モデルの原理紹介
brainpadpr
3
3.8k
マテリアルズ・インフォマティクスの先端で起きていること / What's Happening at the Cutting Edge of Materials Informatics
snhryt
1
110
DEIM2024 チュートリアル ~AWSで生成AIのRAGを使ったチャットボットを作ってみよう~
yamahiro
3
1.2k
Celebrate UTIG: Staff and Student Awards 2024
utig
0
250
Machine Learning for Materials (Lecture 1)
aronwalsh
1
1.8k
Pandas 2 vs Polars vs Dask (PyDataGlobal 2023 December)
ianozsvald
0
790
学術講演会中央大学学員会八王子支部
tagtag
0
210
HAS Dark Site Orientation
astronomyhouston
0
5.2k
How were Quaternion discovered
kinakomoti321
2
910
ultraArmをモニター提供してもらった話
miura55
0
170
Featured
See All Featured
Why You Should Never Use an ORM
jnunemaker
PRO
53
8.9k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
662
120k
The Cult of Friendly URLs
andyhume
76
5.9k
Being A Developer After 40
akosma
82
580k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
24
2k
KATA
mclloyd
27
13k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
230
17k
What's new in Ruby 2.0
geeforr
340
31k
Statistics for Hackers
jakevdp
793
220k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
278
13k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
502
140k
Product Roadmaps are Hard
iamctodd
PRO
48
10k
Transcript
©2018 Wantedly, Inc. രվળʂInterleaving ABςετΑΓޮతͳϥϯΩϯάධՁΛٻΊͯ Webք۾LTձ 6.Jul.2018 - Yuya Matsumura
- @yu-ya4
©2018 Wantedly, Inc. •Yuya Matsumuraʢদଜ ༏ʣ •Wantedly, Inc. (ࣾձਓ4ϲ݄) •From
Kyoto •Interested in Information Retrieval & Recommendation Self Introduction @yu-ya4
©2018 Wantedly, Inc. ABςετͯ͠·͔͢ʁ
©2018 Wantedly, Inc. ͋Δಛఆͷظؒʹ2ύλʔϯͷࢪࡦΛ༻ҙ͠ɼͲͪΒͷ ࢪࡦͷํ͕ΑΓߴ͍ޮՌ͕ಘΒΕΔ͔Λݕূ͢Δ͜ͱ ABςετͱ WantedlyͰɼجຊతʹͯ͢ͷมߋʹରͯ͠ABςετΛߦ͔ͬͯΒ ͷਖ਼ࣜϦϦʔεΛߦ͍ͬͯ·͢ʂ
©2018 Wantedly, Inc. Wantedlyʹ͓͚ΔϥϯΩϯάͷABςετ A/B Testing ̍ ̎ ̍
̎ Group A Ranking A Control Variation Group B Ranking B
©2018 Wantedly, Inc. ༗ҙͳ͕ࠩग़Δ·Ͱʹ ABςετͷ՝ →PDCAαΠΫϧΛճ͢ͷ͕͘ͳΔʂ Ex. *ݱࡏCVR 1%ͷͷΛ10%վળ(CVR 1.1%)͢Δʹ134,379ͷαϯϓϧ͕ඞཁ
*༗ҙਫ४Ћ=5%, ݕग़ྗ1-β=95%(https://keisan.casio.jp/exec/user/1489063696) େྔͷαϯϓϧ(impression, PV)͕ඞཁ
©2018 Wantedly, Inc.
©2018 Wantedly, Inc. ͦ͜ͰInterleaving
©2018 Wantedly, Inc. 2ͭͷϥϯΩϯά͔Βަޓʹݕࡧ݁ՌΛ࠷্Ґ͔Βॱʹ બ͠1ͭͷϥϯΩϯάΛੜͯ͠Ϣʔβʹఏࣔ͢Δ Interleavingͱ ̍ ̎ ̍
̎ Ranking A Ranking B ̍ ̎ ̍ ̎ Ranking C ᶃ ᶄ ᶅ ᶆ
©2018 Wantedly, Inc. Interleavingͷੑೳ →രͰPDCAαΠΫϧ͕ճͤΔʂ *ਖ਼֬ʹʮΠϯλʔϦʔϏϯάA/BςετΑΓ100ഒޮతͳ߹͕͋Γɺ͓͓Αͦͷ߹ɺ 10~100ഒޮతͰɺA/BςετΑΓޮతͰͳ͍߹গͳ͍ʯ(Chapelle et al. 2012)
Interleaving ABςετΑΓ10ʙ100ഒޮత*
©2018 Wantedly, Inc.
©2018 Wantedly, Inc. Interleaving͕ABςετΑΓޮతͳཧ༝ ̍ ̎ ̍ ̎
Ranking A Ranking B A/B Testing ̍ Ranking A ͱ Ranking Bͷੑೳ͕ࣅ͍ͯΔ߹ ͕ࠩग़ʹ͍͘ → ̍ ̍ ̍ ̍ ̍
©2018 Wantedly, Inc. Interleaving͕ABςετΑΓޮతͳཧ༝ Interleaving ̍ Ranking A ͱ Ranking
B͕҉ʹൺֱ͞ΕΔͷͰ ͕ࠩग़͍͢ʂʂ → ̍ ̍ ̍ ̍ ̍ ̎ ̍ ̎ ̍
©2018 Wantedly, Inc. ࣮ࡍͷABςετͱInterleavingͷp-valueͷऩଋͷൺֱ (Wantedly VisitͷϥϯΩϯάվળʹͯܭଌ)
©2018 Wantedly, Inc. ͦͷ݁Ռ…
©2018 Wantedly, Inc. ৽ଔೖࣾ1ϲ݄ͰWantedly Visitͷ ৽͍͠ϥϯΩϯάࢪࡦΛຊ൪ϦϦʔε (ࠓͰܧଓͯ͠ྑ͍݁ՌΛग़͠ଓ͚͍ͯ·͢)
©2018 Wantedly, Inc. InterleavingΛ׆༻ͯ͠രϥϯΩϯάվળΛ