Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Personalized Content Recommender System on Spring
LINE Developers
PRO
March 25, 2019
Technology
0
1.6k
Personalized Content Recommender System on Spring
LINE Developers
PRO
March 25, 2019
Tweet
Share
More Decks by LINE Developers
See All by LINE Developers
LIFF Deep Dive 2022
line_developers
PRO
0
11
LINEのData Platform室が実践する大規模分散環境のCapacity Planning
line_developers
PRO
0
470
LINEスタンプの実例紹介 小さく始める障害検知・対応・振り返りの 改善プラクティス
line_developers
PRO
3
1.6k
読者のことを考えて書いてみよう / Write with your reader in mind
line_developers
PRO
3
350
登壇者をどうやって確保するのか問題
line_developers
PRO
1
190
Machine Learning at LINE
line_developers
PRO
2
220
自分が書いた文章をレビューしてもらうときの心構え / Readiness for having your writing reviewed
line_developers
PRO
3
460
JavaScriptの進化で変わる身近なコーディング習慣
line_developers
PRO
10
11k
デザインシステムのa11y対応に役立つ新しいWeb Standard
line_developers
PRO
0
1.1k
Other Decks in Technology
See All in Technology
New Features in C# 10/11
chack411
0
950
HTTP Session Architecture Pattern
chiroito
1
400
Whats new in Android Media?
myolwin00
2
110
CADDi HCMC Technology Center
caddi_eng
0
250
220521_SFN_品質文化試論と『LEADING QUALITY』/220521_SFN_Essay_of_Quality_Culture_and_LEADING_QUALITY
mkwrd
0
250
プルリク作ったらデプロイされる仕組み on ECS / SRE NEXT 2022
carta_engineering
1
270
プロダクトグロースと技術のベースアップを両立させるRettyのアプリ開発スタイル / Achieve Product Growth and Tech Update
imaizume
1
290
[SRE NEXT 2022]組織に対してSREを適用するとはどういうことか
srenext
0
180
SRE_チーム立ち上げから1年_気づいたら_SRE_っぽくない仕事まで貢献しちゃってる説
bitkey
PRO
0
2.2k
Learning from AWS Customer Security Incidents [2022]
ramimac
0
1.1k
家の明るさ制御 / Brightness Control in My House
1024jp
0
110
0->1 フェーズで E2E 自動テストを導入した私たちの、これまでとこれから
yoyakoba
0
430
Featured
See All Featured
Navigating Team Friction
lara
175
11k
Infographics Made Easy
chrislema
233
17k
Mobile First: as difficult as doing things right
swwweet
212
7.5k
How to name files
jennybc
39
58k
Designing for humans not robots
tammielis
241
23k
What the flash - Photography Introduction
edds
61
10k
Documentation Writing (for coders)
carmenhchung
48
2.5k
In The Pink: A Labor of Love
frogandcode
130
21k
Typedesign – Prime Four
hannesfritz
33
1.3k
The Art of Programming - Codeland 2020
erikaheidi
31
5.8k
A Modern Web Designer's Workflow
chriscoyier
689
180k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
103
16k
Transcript
Personalized Content Recommender System on Spring
About me @hackmylife • ຊ໊ ᬒ थ ௨শ നถ •
։ൃ̐ηϯλʔ > AP3 team • Senior server side engineer • livedoorχϡʔε, LINEΪϑτ, LINE BLOG • LINE Ads Platform • Smart Channel • Spring fest 2018 fall
Smart Channel
• Overview • Architecture • Team Agenda
Overview
Concept Contents Contents
UX
• User͕͍ͬͯΔɺ͋Δ͍ڧ͍རӹͷ͋ΔServiceͷใ • ServiceͰ֤UserʹPersonalizeͯ͠࡞ΒΕͨͷ • ࣄલTest৴Λͯ͠ҰఆͷධՁΛ͑ͳ͚ΕͳΒͳ͍(ޙड़) Contents
• Userʹର͢ΔRecommendΛ֤Service͔ΒूΊΔ • ूΊͨ Contents ΛҰ࣌తʹอଘ͓ͯ͘͠ • RequestΛड͚ͨΒContentsΛRankingͯ͠n݅બΜͰ৴͢Δ ͜ͷαʔϏεͷಛ
Figures MAU 79M RPS 250k Servers 200+ * JP͚ͩͷ
Architecture
None
Java + Spring
• Redis • MySQL • Kafka • Spring boot 2
• lettuce5 • MyBatis + kotlin • Caffeine • fastutil, guava Spec
Application 86 pm servers 40 vm servers MySQL 3 vm
servers Redis 100+ vm instance (Redis cluster) Kafka In-house common (Kafka cluster) Log storage In-house common (Hadoop) Servers
Importer CRS Engine Event Tracker Joiner Learning worker 5ͭͷComponent
CRS Engine Flow Event Tracker Joiner Learning worker Importer LINE
App Contents Contents Imp/Click Log Log Parameter
CRS Engine 1. Importer Event Tracker Joiner Learning worker Importer
LINE App Contents Contents Imp/Click Log Log Parameter
None
• Mapping provider͔Βɺrecommend item ͷID͚ͩΒ͏ • item͕ͲΜͳσʔλΛ͍࣋ͬͯΔ͔͍ ߹ΘͤΔ • Redisʹmapping
ͱ contentsΛstore͢Δ Importer user_id1 \t item_id1, item_id2, item_id3 user_id2 \t item_id3, item_id4 user_id3 \t item_id1, item_id2, item_id6 { "items": [ { "originalId": "12345678", "title": “ࣗͷೳྗΛݟۃΊͯ", "description": "ແཧͳεέδϡʔϧͰମௐΛ่ͯ͠͠·͏ΑΓɺ ૉʹपΓʹཔΓ·͠ΐ͏ɻࠓࣗʹݫͯ͘͠͠͠·͍͕ͪɻ ࣗͷೳྗΛ͑ΔͱࢥͬͨΒঢ়گΛͯ͠ڠྗͯ͠Βͬͯ͘ ͍ͩ͞ɻ", "linkURL": “line://ch/1391620050/z/fortune/detail/2/20181011", "iconURL": “http://example.com/path/to/image1.png" }, ……. ] }
• ֤ServiceͰRecommendΛ࡞ͬͯΒ͏ • ྑ͍RecommendʹڧྗͳDomain͕ࣝඞཁ • ServiceʹΑͬͯࡉ͔͍දࣔͷͨΊͷ͕݅͋Δ Recommend
• Recommendation dataͭ͘Δͷ ML engineer • Master data ཧͯ͠Δͷ Service
side engineer • ୲෦ॺ͕ҧ͏ʢ߹ʹΑͬͯࠃҧ͏ʣ • RecommendΛ࡞ͬͨ࣌ͱ͕࣌ࠩ͋Δ Կނ2ճʹ͚ͯσʔλΛऔΔ͔
• Specͱʹ͔͘ΨνΨνʹݻΊ͓ͯ͘ • Documment͔ͬ͠Γἧ͑ͯӳޠͰॻ͍͓ͯ͘ • ServiceͷࢥతΛͪΌΜͱ͑Δ • جຊతʹࣄલTestΛͯ͠ΫϦΞ͠ͳ͍ͱ͍͚ͳ͍ ؾΛ͚͍ͭͯΔ͜ͱ
• CTRxCTRʹΑΔධՁ • ΞϯέʔτͳͲͷఆੑతධՁ • ্هޓ͍ʹڧ͍૬ؔؔ༗Γ Preliminary test
• RecommendServiceଆͰ࡞ͬͯΒ͏ • Recommend id/dataͷऔಘΛ͍ͯ͠Δ • औಘͨ͠Contentsredisʹશ෦ೖΕΔ • Service࿈ܞલʹ͍ͭ͘ͷνΣοΫ͕͋Δ Importer
·ͱΊ
CRS Engine 2. CRS Engine Event Tracker Joiner Learning worker
Importer LINE App Contents Contents Imp/Click Log Log Parameter
None
• UserຖʹRecommend contentsऔಘ • Frequency cap / mute / opt-outͳͲͰfiltering
• Recommend itemΛRanking্͔ͯ͠Βn݅ฦ͢ • Contextual bandit algorithmsΛ࠾༻ CRS Engine
• Reactive API • RedisAdvancedClusterReactiveCommands • Γ͕ Mono / Flux
• ݸਓతʹ͘͢͝εοΩϦ͔͚͖ͯ Lettuce5
• Reactive Streamsͷpublisher࣮ • Mono࠷େ1ݸͷitemΛѻ͏ • FluxෳͷitemΛѻ͏ Mono/Flux
Mono/Fluxͷྫ targetingRedis.mget(keys) // ͜͜Flux͕ؼͬͯ͘Δ .filter(KeyValue::hasValue) .map(keyValue -> Maps.immutableEntry(extractTargetingType(keyValue.getKey()), keyValue.getValue())) .collectMap(Map.Entry::getKey,
Map.Entry::getValue); // ΓɹMono<Map<String, List<String>>> Key: user_id:news Val: ["65819383620a","48d2872c86c6","293d0a33f938","c43cd18c98f4"] Key: user_id:weather Val: [“1"] -> ͜ΕΛnews, weatherΛkeyͱͨ͠Mapͱ͔ʹͯ͠ฦ͍ͨ͠ ࣮ࡍʹॲཧ͞ΕΔͷsubscribeͨ࣌͠
• Flux͔Βstreaming apiతͳॲཧΛεϜʔζʹ͔͚Δ • ଈॲཧͰͳ͘ॲཧΛޙͰߦ͑Δ • Mono.zip()ͳͲΛͬͯฒྻॲཧͰ͖Δ • ͨͩɺ·ͩͦ͜·Ͱ͍ͬͯͳ͍ Mono/FluxͷϝϦοτ
subtitle Bandit algorithms • SlotͷrewardΛ࠷େԽ͢Δ • ࣄલڞ༗ใ͕ແ͍߹͋Δ • ϥϯυຖʹͲͷarmΛબͿ͔ܾఆ͢Δ •
ϙΠϯτ୳ࡧͱ׆༻Λ͍͚Δ͜ͱ
• ࠷ॳdata͕Γͳ͍ͷͰɺͱΓ͋͑ͣճͯ͠ΈΔʢ୳ࡧʣ • ͋Δఔճ͢ͱ༏ྼ͕ղͬͯ͘Δ • Ұ൪ظ͞ΕΔreward͕ྑ͍ͱ͜ΖΛճ͢ʢ׆༻ʣ • ͬͱใु͕ྑ͍Ϛγϯ͕͋Δ͔୳͢ʢ୳ࡧʣ ୳ࡧͱ׆༻
• RewardͷఆٛUserͷreaction (CTRͱ͔xCTRͱ͔) • UserҰਓҰਓʹarmΛ͚͍͕ͨɺऩଋͣ͠Β͘ͳΔ • ͳͷͰɺ͏গ͠େཻ͖͍Ͱֶश͍ͯ͠Δ • ਫ਼Λ্͛ΔͨΊʹɺ͍͔ͭ͘ใʢ=ContextʣΛ༩͍͑ͯΔ Contextual
bandit on CRS engine
User side • Gender • Age • Interests Contents side
2-side features • Time • Id • Score
• ঁੑ > உੑ • ए͍ੈͷํ͕ؾʹ͢ΔͷͰʁ • உੑͦΜͳʹݟͳ͍ʁ User side
- ͍ Age-range x Gender
• Ͳ͏ͤΈΔͳΒேͩΑͶ • ྑ͍݁Ռͩͱclick͍͢͠ͷͰ • ѱ͍݁Ռݟͨ͘ͳ͍ʁ Contents side - ͍
Age-range x Gender x Hour x Score
LinUCB Bayesian linear regression Bayesian factorization machines Supported Algorithms Bayesian
neural factorization machines
• RedisΛ৭ʑࢀর͍ͯͯ͠Δ • Contextual bandit algorithms ͷRankerͱͯ͠n݅બͿ • BanditͷContext৭ʑ͍ͯ͠Δ CRS
engine ·ͱΊ
CRS Engine 3. Event Tracker Event Tracker Joiner Learning worker
Importer LINE App Contents Contents Imp/Click Log Log Parameter
None
• imp/click/muteͳͲͷeventΛड͚Δ • Ұ୴Kafkaʹproduce͢Δ͚ͩ • consume͢Δcomponent͚ͯͯΔ • KafkaΛ͏ϝϦοτΛ1൪ڗडͰ͖Δ Event tracker
• ߴ͍scalabilityͱavailability • Persistent data • Pub-Subܕ (kafkaͷ༻ޠͱͯ͠produce / consume)
• ෳconsumer͔ΒಡΊΔ Kafka
• eventΛอଘ͢Δॲཧͱɺ࣮ࡍʹԿ ͔͢ΔॲཧΛ͚ΒΕΔ • ผͷconsumer͔ΒಡΊΔ • ো࣌ʹӨڹΛ࠷খݶʹͰ͖Δɺ ಛఆ͘͢͠ͳΔ KafkaͷϝϦοτ
• Spring bootͱ૬ੑ͕ͱͯྑ͍ • @BeanͰKafka-clientΛཧ͢Δͱੜ/ഁغ·Ͱ໘ݟͯ͘ΕΔ • Kafka-streamsͱΈ߹ΘͤΔ͜ͱͰίʔυॻ͘ྔ͕ͱͯগͳ͍ • Spring-kafka͋Δ͚ͲެࣜKafka-client͕Φεεϝ Kafka
on spring
@Configuration @RequiredArgsConstructor @EnableConfigurationProperties({ KafkaStreamsProperties.class, KafkaTopicProperties.class }) public class KafkaStreamsConfiguration {
@Bean public KafkaStreams muteEventStream( KafkaStreamsProperties streamsProperties, KafkaTopicProperties topicProperties ) { final StreamsBuilder builder = new StreamsBuilder(); builder.stream(topicProperties.getMuteEvent(), Consumed.with(new StringSerde(), new StringSerde())) .foreach((key, value) -> { try { // Կ͔ॲཧ } catch (Exception e) { // ྫ֎ॲཧ } }); KafkaStreams streams = new KafkaStreams(builder.build(), streamsProperties.getMuteEventStream()); streams.setUncaughtExceptionHandler((thread, e) -> { log.error("Uncaught exception in StreamThread.", e); }); streams.start(); return streams; } }
kafka: servers: server1, server2, server3 metrics-reporter: com.example.awesome.kafka.metrics.PrometheusMetricsReporter topic: mute-event: awesome-mute-event-v1
stream: mute-event-stream: "bootstrap.servers": ${kafka.servers} "application.id": crs-mute-event "state.dir": /path/to/kafka-streams "num.stream.threads": 6 "replication.factor": 3 "metric.reporters": ${kafka.metrics-reporter} consumer: "max.poll.records": 100 "auto.offset.reset": earliest @ConfigurationProperties(prefix = "kafka.stream") public class KafkaStreamsProperties { @NotNull @Valid private Properties muteEventStream; }
• LINEΞϓϦ͔ΒͷEventΛड͚औΔॲཧ • KafkaΛڬΉ͜ͱͰɺߴ/োੑΛ࣮ݱͯ͠Δ • ݸਓతʹKafkaҰԡ͠Ͱ͢ Event Tracker ·ͱΊ
CRS Engine 4. Joiner Event Tracker Joiner Learning worker Importer
LINE App Contents Contents Imp/Click Log Log Parameter
None
• ϢʔβʔͷContentsʹର͢ΔԠ͕Γ͍ͨ • imp͞Εͨcontents͕click͞Ε͔ͨɺmute͞Ε͔ͨ • impͱclick/muteΛjoin͍ͨ͠ Γ͍ͨ͜ͱ
• imp͕ಧ͍ͨஈ֊Ͱclickmuteະདྷͷࣄ • click͕དྷͨ࣌Ͱimpॲཧ͞Εͯ͠·͍ͬͯΔ • impΛ͋ΔఔԆͤͯ͞ॲཧ͍ͨ͠ Ͳ͏͢Εྑ͍͔
• impͷconsumeΛ10minΒͤΔ • clickmuteͦͷ··Redisʹ • 10minޙimpΛॲཧ͢Δͱ͖ʹredisΛࢀর Ԇconsumer https://www.slideshare.net/linecorp/line-ads-platformkafka-126180436
• MLͷڭࢣσʔλͷूܭॲཧ • KafkaΛԠ༻ͨ͠ԆConsumerॲཧ Joiner ·ͱΊ
CRS Engine 5. Learning worker Event Tracker Joiner Learning worker
Importer LINE App Contents Contents Imp/Click Log Log Parameter
None
• engine͕rankerɺ͜͜trainerଆ • σʔλྔ͕ଟ͍ͷͰฒྻԽ͍ͨ͠ • ޮԽͷͨΊ·ͱΊͯॲཧ͍ͨ͠ • ಉ͡armͷσʔλಉ͡αʔόʔͰॲཧ Learning worker
• BanditͷTrainerPython • NginxͰ Consistent hashing • User trafficΛड͚ΔRankerjava Learning
worker
• NumPy/Numba͕͍͔ͨͬͨ • BanditΛ։ൃͯ͠Δͷdatalabs -> ML part • Ranker҆ఆੑΛॏࢹ͍ͨ͠ͷͰJavaͰؤுͬͯॻ͘ •
Python <-> JavaͷΠϯλʔϑΣʔεprotobufͰఆٛ Why Python
• Bandit algorithmsͷTrainer • ඇಉظ/ޮతʹֶश͢ΔͨΊʹ͍Ζ͍Ζͯ͠Δ • Java <-> PythonͰparameterΛड͚͠ •
͠ํֶशͷํͰ͍͔ͭ͘Λ͍ͯ͠Δ Learning worker·ͱΊ
Team
subtitle Developers • Server side engineer 5 ਓ • ML
team֤Service developerͱίϥϘϨʔγϣϯ • GlobalͷϝϯόʔͱͷΓऔΓසൟʹ͋Δ • ·ͩ·ͩ໘ന͍Spec͋ΔͷͰ͍͖͍ͬͯͨ
THANK YOU