Personalized Content Recommender System on Spring

1. Personalized Content Recommender System on Spring
2. About me @hackmylife • ຊ໊ ౉ᬒ ௚थ ௨শ നถ • ։ൃ̐ηϯλʔ > AP3 team • Senior server side engineer • livedoorχϡʔε, LINEΪϑτ, LINE BLOG • LINE Ads Platform • Smart Channel • Spring fest 2018 fall
3. Smart Channel
4. • Overview • Architecture • Team Agenda
5. Overview
6. Concept Contents Contents
7. UX
8. • User͕࢖͍ͬͯΔɺ͋Δ͍͸ڧ͍རӹͷ͋ΔServiceͷ৘ใ • ServiceͰ֤UserʹPersonalizeͯ͠࡞ΒΕͨ΋ͷ • ࣄલTest഑৴Λͯ͠ҰఆͷධՁΛ௒͑ͳ͚Ε͹ͳΒͳ͍(ޙड़) Contents
9. • Userʹର͢ΔRecommendΛ֤Service͔ΒूΊΔ • ूΊͨ Contents ΛҰ࣌తʹอଘ͓ͯ͘͠ • RequestΛड͚ͨΒContentsΛRankingͯ͠n݅બΜͰ഑৴͢Δ ͜ͷαʔϏεͷಛ௃
10. Figures MAU 79M RPS 250k Servers 200+ * JP͚ͩͷ਺஋
11. Architecture
13. Java + Spring
14. • Redis • MySQL • Kafka • Spring boot 2 • lettuce5 • MyBatis + kotlin • Caffeine • fastutil, guava Spec
15. Application 86 pm servers 40 vm servers MySQL 3 vm servers Redis 100+ vm instance (Redis cluster) Kafka In-house common (Kafka cluster) Log storage In-house common (Hadoop) Servers
16. Importer CRS Engine Event Tracker Joiner Learning worker 5ͭͷComponent
17. CRS Engine Flow Event Tracker Joiner Learning worker Importer LINE App Contents Contents Imp/Click Log Log Parameter
18. CRS Engine 1. Importer Event Tracker Joiner Learning worker Importer LINE App Contents Contents Imp/Click Log Log Parameter
20. • Mapping provider͔Βɺrecommend item ͷID͚ͩ΋Β͏ • item͕ͲΜͳσʔλΛ͍࣋ͬͯΔ͔໰͍ ߹ΘͤΔ • Redisʹmapping ͱ contentsΛstore͢Δ Importer user_id1 \t item_id1, item_id2, item_id3 user_id2 \t item_id3, item_id4 user_id3 \t item_id1, item_id2, item_id6 { "items": [ { "originalId": "12345678", "title": “ࣗ෼ͷೳྗΛݟۃΊͯ", "description": "ແཧͳεέδϡʔϧͰମௐΛ่ͯ͠͠·͏ΑΓɺ ૉ௚ʹपΓʹཔΓ·͠ΐ͏ɻࠓ͸ࣗ෼ʹݫͯ͘͠͠͠·͍͕ͪɻ ࣗ෼ͷೳྗΛ௒͑ΔͱࢥͬͨΒঢ়گΛ࿩ͯ͠ڠྗͯ͠΋Βͬͯ͘ ͍ͩ͞ɻ", "linkURL": “line://ch/1391620050/z/fortune/detail/2/20181011", "iconURL": “http://example.com/path/to/image1.png" }, ……. ] }
21. • ֤ServiceͰRecommendΛ࡞ͬͯ΋Β͏ • ྑ͍Recommendʹ͸ڧྗͳDomain஌͕ࣝඞཁ • ServiceʹΑͬͯ͸ࡉ͔͍දࣔͷͨΊͷ৚͕݅͋Δ Recommend
22. • Recommendation dataͭ͘Δͷ͸ ML engineer • Master data ؅ཧͯ͠Δͷ͸ Service side engineer • ୲౰෦ॺ͕ҧ͏ʢ৔߹ʹΑͬͯ͸ࠃ΋ҧ͏ʣ • RecommendΛ࡞ͬͨ࣌ͱ͕࣌ࠩ͋Δ Կނ2ճʹ෼͚ͯσʔλΛऔΔ͔
23. • Spec͸ͱʹ͔͘ΨνΨνʹݻΊ͓ͯ͘ • Documment͸͔ͬ͠Γἧ͑ͯӳޠͰॻ͍͓ͯ͘ • Serviceͷࢥ૝΍໨తΛͪΌΜͱ఻͑Δ • جຊతʹࣄલTestΛͯ͠ΫϦΞ͠ͳ͍ͱ͍͚ͳ͍ ؾΛ͚͍ͭͯΔ͜ͱ
24. • CTR΍xCTRʹΑΔ਺஋ධՁ • ΞϯέʔτͳͲͷఆੑతධՁ • ্ه͸ޓ͍ʹڧ͍૬ؔؔ܎༗Γ Preliminary test
25. • Recommend͸ServiceଆͰ࡞ͬͯ΋Β͏ • Recommend id/dataͷऔಘΛ෼཭͍ͯ͠Δ • औಘͨ͠Contents͸redisʹશ෦ೖΕΔ • Service࿈ܞલʹ͍ͭ͘΋ͷνΣοΫ͕͋Δ Importer ·ͱΊ
26. CRS Engine 2. CRS Engine Event Tracker Joiner Learning worker Importer LINE App Contents Contents Imp/Click Log Log Parameter
28. • UserຖʹRecommend contentsऔಘ • Frequency cap / mute / opt-outͳͲͰfiltering • Recommend itemΛRanking্͔ͯ͠Βn݅ฦ͢ • Contextual bandit algorithmsΛ࠾༻ CRS Engine
29. • Reactive API • RedisAdvancedClusterReactiveCommands • ໭Γ஋͕ Mono / Flux • ݸਓతʹ͸͘͢͝εοΩϦ͔͚ͯ޷͖ Lettuce5
30. • Reactive Streamsͷpublisher࣮૷ • Mono͸࠷େ1ݸͷitemΛѻ͏ • Flux͸ෳ਺ͷitemΛѻ͏ Mono/Flux
31. Mono/Fluxͷྫ targetingRedis.mget(keys) // ͜͜͸Flux͕ؼͬͯ͘Δ .filter(KeyValue::hasValue) .map(keyValue -> Maps.immutableEntry(extractTargetingType(keyValue.getKey()), keyValue.getValue())) .collectMap(Map.Entry::getKey, Map.Entry::getValue); // ໭Γ஋͸ɹMono<Map<String, List<String>>> Key: user_id:news Val: ["65819383620a","48d2872c86c6","293d0a33f938","c43cd18c98f4"] Key: user_id:weather Val: [“1"] -> ͜ΕΛnews, weatherΛkeyͱͨ͠Mapͱ͔ʹͯ͠ฦ͍ͨ͠ ࣮ࡍʹॲཧ͞ΕΔͷ͸subscribeͨ࣌͠
32. • Flux͔Βstreaming apiతͳॲཧΛεϜʔζʹ͔͚Δ • ଈॲཧͰ͸ͳ͘ॲཧΛޙͰߦ͑Δ • Mono.zip()ͳͲΛ࢖ͬͯฒྻॲཧ΋Ͱ͖Δ • ͨͩɺ·ͩͦ͜·Ͱ͸΍͍ͬͯͳ͍ Mono/FluxͷϝϦοτ
33. subtitle Bandit algorithms • SlotͷrewardΛ࠷େԽ͢Δ • ࣄલڞ༗৘ใ͕ແ͍৔߹΋͋Δ • ϥ΢ϯυຖʹͲͷarmΛબͿ͔ܾఆ͢Δ • ϙΠϯτ͸୳ࡧͱ׆༻Λ࢖͍෼͚Δ͜ͱ
34. • ࠷ॳ͸data͕଍Γͳ͍ͷͰɺͱΓ͋͑ͣճͯ͠ΈΔʢ୳ࡧʣ • ͋Δఔ౓ճ͢ͱ༏ྼ͕ղͬͯ͘Δ • Ұ൪ظ଴͞ΕΔreward͕ྑ͍ͱ͜ΖΛճ͢ʢ׆༻ʣ • ΋ͬͱใु͕ྑ͍Ϛγϯ͕͋Δ͔୳͢ʢ୳ࡧʣ ୳ࡧͱ׆༻
35. • Rewardͷఆٛ͸Userͷreaction (CTRͱ͔xCTRͱ͔) • UserҰਓҰਓʹarmΛ෼͚͍͕ͨɺऩଋͣ͠Β͘ͳΔ • ͳͷͰɺ΋͏গ͠େཻ͖͍౓Ͱֶश͍ͯ͠Δ • ਫ਼౓Λ্͛ΔͨΊʹɺ͍͔ͭ͘৘ใʢ=ContextʣΛ༩͍͑ͯΔ Contextual bandit on CRS engine
36. User side • Gender • Age • Interests Contents side 2-side features • Time • Id • Score
37. • ঁੑ > உੑ • ए͍ੈ୅ͷํ͕ؾʹ͢ΔͷͰ͸ʁ • உੑ͸ͦΜͳʹݟͳ͍ʁ User side - ઎͍ Age-range x Gender
38. • Ͳ͏ͤΈΔͳΒேͩΑͶ • ྑ͍݁Ռͩͱclick͠΍͍͢ͷͰ͸ • ѱ͍݁Ռ͸ݟͨ͘ͳ͍ʁ Contents side - ઎͍ Age-range x Gender x Hour x Score
39. LinUCB Bayesian linear regression Bayesian factorization machines Supported Algorithms Bayesian neural factorization machines
40. • RedisΛ৭ʑࢀর͍ͯͯ͠Δ • Contextual bandit algorithms ͷRankerͱͯ͠n݅બͿ • BanditͷContext͸৭ʑ޻෉͍ͯ͠Δ CRS engine ·ͱΊ
41. CRS Engine 3. Event Tracker Event Tracker Joiner Learning worker Importer LINE App Contents Contents Imp/Click Log Log Parameter
43. • imp/click/muteͳͲͷeventΛड͚Δ • Ұ୴Kafkaʹproduce͢Δ͚ͩ • consume͢Δcomponent͸෼͚ͯͯΔ • KafkaΛ࢖͏ϝϦοτΛ1൪ڗडͰ͖Δ Event tracker
44. • ߴ͍scalabilityͱavailability • Persistent data • Pub-Subܕ (kafkaͷ༻ޠͱͯ͠͸produce / consume) • ෳ਺consumer͔ΒಡΊΔ Kafka
45. • eventΛอଘ͢Δॲཧͱɺ࣮ࡍʹԿ ͔͢ΔॲཧΛ෼͚ΒΕΔ • ผͷconsumer͔Β΋ಡΊΔ • ো֐࣌ʹӨڹΛ࠷খݶʹͰ͖Δɺ ಛఆ͠΍͘͢ͳΔ KafkaͷϝϦοτ
46. • Spring bootͱ૬ੑ͕ͱͯ΋ྑ͍ • @BeanͰKafka-clientΛ؅ཧ͢Δͱੜ੒/ഁغ·Ͱ໘౗ݟͯ͘ΕΔ • Kafka-streamsͱ૊Έ߹ΘͤΔ͜ͱͰίʔυॻ͘ྔ͕ͱͯ΋গͳ͍ • Spring-kafka΋͋Δ͚ͲެࣜKafka-client͕Φεεϝ Kafka on spring
47. @Configuration @RequiredArgsConstructor @EnableConfigurationProperties({ KafkaStreamsProperties.class, KafkaTopicProperties.class }) public class KafkaStreamsConfiguration { @Bean public KafkaStreams muteEventStream( KafkaStreamsProperties streamsProperties, KafkaTopicProperties topicProperties ) { final StreamsBuilder builder = new StreamsBuilder(); builder.stream(topicProperties.getMuteEvent(), Consumed.with(new StringSerde(), new StringSerde())) .foreach((key, value) -> { try { // Կ͔ॲཧ } catch (Exception e) { // ྫ֎ॲཧ } }); KafkaStreams streams = new KafkaStreams(builder.build(), streamsProperties.getMuteEventStream()); streams.setUncaughtExceptionHandler((thread, e) -> { log.error("Uncaught exception in StreamThread.", e); }); streams.start(); return streams; } }
48. kafka: servers: server1, server2, server3 metrics-reporter: com.example.awesome.kafka.metrics.PrometheusMetricsReporter topic: mute-event: awesome-mute-event-v1 stream: mute-event-stream: "bootstrap.servers": ${kafka.servers} "application.id": crs-mute-event "state.dir": /path/to/kafka-streams "num.stream.threads": 6 "replication.factor": 3 "metric.reporters": ${kafka.metrics-reporter} consumer: "max.poll.records": 100 "auto.offset.reset": earliest @ConfigurationProperties(prefix = "kafka.stream") public class KafkaStreamsProperties { @NotNull @Valid private Properties muteEventStream; }
49. • LINEΞϓϦ͔ΒͷEventΛड͚औΔॲཧ • KafkaΛڬΉ͜ͱͰɺߴ଎/଱ো֐ੑΛ࣮ݱͯ͠Δ • ݸਓతʹ͸KafkaҰԡ͠Ͱ͢ Event Tracker ·ͱΊ
50. CRS Engine 4. Joiner Event Tracker Joiner Learning worker Importer LINE App Contents Contents Imp/Click Log Log Parameter
52. • ϢʔβʔͷContentsʹର͢Δ൓Ԡ͕஌Γ͍ͨ • imp͞Εͨcontents͕click͞Ε͔ͨɺmute͞Ε͔ͨ • impͱclick/muteΛjoin͍ͨ͠ ΍Γ͍ͨ͜ͱ
53. • imp͕ಧ͍ͨஈ֊Ͱ͸click΍mute͸ະདྷͷࣄ • click͕དྷͨ࣌఺Ͱ͸imp͸ॲཧ͞Εͯ͠·͍ͬͯΔ • impΛ͋Δఔ౓஗Ԇͤͯ͞ॲཧ͍ͨ͠ Ͳ͏͢Ε͹ྑ͍͔
54. • impͷconsumeΛ10min஗ΒͤΔ • click΍mute͸ͦͷ··Redisʹ • 10minޙimpΛॲཧ͢Δͱ͖ʹredisΛࢀর ஗Ԇconsumer https://www.slideshare.net/linecorp/line-ads-platformkafka-126180436
55. • MLͷڭࢣσʔλͷूܭॲཧ • KafkaΛԠ༻ͨ͠஗ԆConsumerॲཧ Joiner ·ͱΊ
56. CRS Engine 5. Learning worker Event Tracker Joiner Learning worker Importer LINE App Contents Contents Imp/Click Log Log Parameter
58. • engine͕rankerɺ͜͜͸trainerଆ • σʔλྔ͕ଟ͍ͷͰฒྻԽ͍ͨ͠ • ޮ཰ԽͷͨΊ·ͱΊͯॲཧ͍ͨ͠ • ಉ͡armͷσʔλ͸ಉ͡αʔόʔͰॲཧ Learning worker
59. • BanditͷTrainer͸Python • NginxͰ Consistent hashing • User trafficΛड͚ΔRanker͸java Learning worker
60. • NumPy/Numba͕࢖͍͔ͨͬͨ • BanditΛ։ൃͯ͠Δͷ͸datalabs -> ML part • Ranker͸҆ఆੑΛॏࢹ͍ͨ͠ͷͰJavaͰؤுͬͯॻ͘ • Python <-> JavaͷΠϯλʔϑΣʔε͸protobufͰఆٛ Why Python
61. • Bandit algorithmsͷTrainer • ඇಉظ/ޮ཰తʹֶश͢ΔͨΊʹ͍Ζ͍Ζ޻෉ͯ͠Δ • Java <-> PythonͰparameterΛड͚౉͠ • ౉͠ํ΍ֶशͷ࢓ํͰ͍͔ͭ͘޻෉Λ͍ͯ͠Δ Learning worker·ͱΊ
62. Team
63. subtitle Developers • Server side engineer 5 ਓ • ML team΍֤Service developerͱίϥϘϨʔγϣϯ • Globalͷϝϯόʔͱͷ΍ΓऔΓ΋සൟʹ͋Δ • ·ͩ·ͩ໘ന͍Spec͋ΔͷͰ΍͍͖͍ͬͯͨ
64. THANK YOU