Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Web界隈LT会_爆速改善_Interleaving.pdf
Search
Yuya Matsumura
July 06, 2018
Science
1
760
Web界隈LT会_爆速改善_Interleaving.pdf
A/Bテストより10-100倍効率的と言われるランキング評価手法、Interleavingについて、ザックリと概要をお話ししました。
Yuya Matsumura
July 06, 2018
Tweet
Share
More Decks by Yuya Matsumura
See All by Yuya Matsumura
LLM活用の現在とこれから:LayerXにおける事例とともに 2025/1 ver. / layerx-llm-202501
yuya4
4
360
バクラクにおけるAI-UXな取り組み 2024年8月 ver. / layerx-bakuraku-ai-ux-202408
yuya4
2
790
事業会社における 機械学習・推薦システム技術の活用事例と必要な能力 / ml-recsys-in-layerx-wantedly-2024
yuya4
4
300
生成AI・LLM時代における 機械学習エンジニアとしてのキャリア戦略・開発戦略 / my-career-and-development-strategies-for-ml-engineer-2024
yuya4
6
2.3k
新卒1年目に勢いで技術コミュニティを立ち上げたことで得たもの / tech-community-iizo-lt
yuya4
2
710
LayerXにおけるAI・機械学習技術の活用事例とその裏側 / layerx-ai-deim2024
yuya4
5
4.8k
LLM時代におけるAI-OCR機能の開発戦略 / layerx-bakuraku-ocr-llm-lt-2024
yuya4
5
3.9k
LLMを活用した爆速アウトプットのすゝめ / bakusoku-outputs-with-llm
yuya4
10
6.3k
【論文紹介】DocTr_ Document Transformer for Structured Information Extraction in Documents / iccv2023-doctr
yuya4
3
830
Other Decks in Science
See All in Science
論文紹介: PEFA: Parameter-Free Adapters for Large-scale Embedding-based Retrieval Models (WSDM 2024)
ynakano
0
220
ICRA2024 速報
rpc
3
6k
サメのはなし / How Sharks are born
naospon
0
2.4k
Online Feedback Optimization
floriandoerfler
0
930
大規模言語モデルの開発
chokkan
PRO
85
43k
Visual Analytics for R&D Intelligence @Funding the Commons & DeSci Tokyo 2024
hayataka88
0
140
トラブルがあったコンペに学ぶデータ分析
tereka114
2
1.4k
応用心理学Ⅰテキストマイニング講義資料講義編(2024年度)
satocos135
0
110
02_西村訓弘_プログラムディレクター_人口減少を機にひらく未来社会.pdf
sip3ristex
0
170
03_草原和博_広島大学大学院人間社会科学研究科教授_デジタル_シティズンシップシティで_新たな_学び__をつくる.pdf
sip3ristex
0
160
Coqで選択公理を形式化してみた
soukouki
0
290
All-in-One Bioinformatics Platform Realized with Snowflake ~ From In Silico Drug Discovery, Disease Variant Analysis, to Single-Cell RNA-seq
ktatsuya
0
310
Featured
See All Featured
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.5k
The Power of CSS Pseudo Elements
geoffreycrofte
75
5.5k
Why You Should Never Use an ORM
jnunemaker
PRO
55
9.2k
Embracing the Ebb and Flow
colly
84
4.6k
Build The Right Thing And Hit Your Dates
maggiecrowley
34
2.5k
How to Ace a Technical Interview
jacobian
276
23k
The Art of Programming - Codeland 2020
erikaheidi
53
13k
Automating Front-end Workflow
addyosmani
1369
200k
Large-scale JavaScript Application Architecture
addyosmani
511
110k
Adopting Sorbet at Scale
ufuk
75
9.2k
Agile that works and the tools we love
rasmusluckow
328
21k
Faster Mobile Websites
deanohume
306
31k
Transcript
©2018 Wantedly, Inc. രվળʂInterleaving ABςετΑΓޮతͳϥϯΩϯάධՁΛٻΊͯ Webք۾LTձ 6.Jul.2018 - Yuya Matsumura
- @yu-ya4
©2018 Wantedly, Inc. •Yuya Matsumuraʢদଜ ༏ʣ •Wantedly, Inc. (ࣾձਓ4ϲ݄) •From
Kyoto •Interested in Information Retrieval & Recommendation Self Introduction @yu-ya4
©2018 Wantedly, Inc. ABςετͯ͠·͔͢ʁ
©2018 Wantedly, Inc. ͋Δಛఆͷظؒʹ2ύλʔϯͷࢪࡦΛ༻ҙ͠ɼͲͪΒͷ ࢪࡦͷํ͕ΑΓߴ͍ޮՌ͕ಘΒΕΔ͔Λݕূ͢Δ͜ͱ ABςετͱ WantedlyͰɼجຊతʹͯ͢ͷมߋʹରͯ͠ABςετΛߦ͔ͬͯΒ ͷਖ਼ࣜϦϦʔεΛߦ͍ͬͯ·͢ʂ
©2018 Wantedly, Inc. Wantedlyʹ͓͚ΔϥϯΩϯάͷABςετ A/B Testing ̍ ̎ ̍
̎ Group A Ranking A Control Variation Group B Ranking B
©2018 Wantedly, Inc. ༗ҙͳ͕ࠩग़Δ·Ͱʹ ABςετͷ՝ →PDCAαΠΫϧΛճ͢ͷ͕͘ͳΔʂ Ex. *ݱࡏCVR 1%ͷͷΛ10%վળ(CVR 1.1%)͢Δʹ134,379ͷαϯϓϧ͕ඞཁ
*༗ҙਫ४Ћ=5%, ݕग़ྗ1-β=95%(https://keisan.casio.jp/exec/user/1489063696) େྔͷαϯϓϧ(impression, PV)͕ඞཁ
©2018 Wantedly, Inc.
©2018 Wantedly, Inc. ͦ͜ͰInterleaving
©2018 Wantedly, Inc. 2ͭͷϥϯΩϯά͔Βަޓʹݕࡧ݁ՌΛ࠷্Ґ͔Βॱʹ બ͠1ͭͷϥϯΩϯάΛੜͯ͠Ϣʔβʹఏࣔ͢Δ Interleavingͱ ̍ ̎ ̍
̎ Ranking A Ranking B ̍ ̎ ̍ ̎ Ranking C ᶃ ᶄ ᶅ ᶆ
©2018 Wantedly, Inc. Interleavingͷੑೳ →രͰPDCAαΠΫϧ͕ճͤΔʂ *ਖ਼֬ʹʮΠϯλʔϦʔϏϯάA/BςετΑΓ100ഒޮతͳ߹͕͋Γɺ͓͓Αͦͷ߹ɺ 10~100ഒޮతͰɺA/BςετΑΓޮతͰͳ͍߹গͳ͍ʯ(Chapelle et al. 2012)
Interleaving ABςετΑΓ10ʙ100ഒޮత*
©2018 Wantedly, Inc.
©2018 Wantedly, Inc. Interleaving͕ABςετΑΓޮతͳཧ༝ ̍ ̎ ̍ ̎
Ranking A Ranking B A/B Testing ̍ Ranking A ͱ Ranking Bͷੑೳ͕ࣅ͍ͯΔ߹ ͕ࠩग़ʹ͍͘ → ̍ ̍ ̍ ̍ ̍
©2018 Wantedly, Inc. Interleaving͕ABςετΑΓޮతͳཧ༝ Interleaving ̍ Ranking A ͱ Ranking
B͕҉ʹൺֱ͞ΕΔͷͰ ͕ࠩग़͍͢ʂʂ → ̍ ̍ ̍ ̍ ̍ ̎ ̍ ̎ ̍
©2018 Wantedly, Inc. ࣮ࡍͷABςετͱInterleavingͷp-valueͷऩଋͷൺֱ (Wantedly VisitͷϥϯΩϯάվળʹͯܭଌ)
©2018 Wantedly, Inc. ͦͷ݁Ռ…
©2018 Wantedly, Inc. ৽ଔೖࣾ1ϲ݄ͰWantedly Visitͷ ৽͍͠ϥϯΩϯάࢪࡦΛຊ൪ϦϦʔε (ࠓͰܧଓͯ͠ྑ͍݁ՌΛग़͠ଓ͚͍ͯ·͢)
©2018 Wantedly, Inc. InterleavingΛ׆༻ͯ͠രϥϯΩϯάվળΛ