Personalized Content Recommender System on Spring

Transcript

1. Personalized Content Recommender System on Spring

2. About me @hackmylife • ຊ໊ ౉ᬒ ௚थ ௨শ നถ •

   ։ൃ̐ηϯλʔ > AP3 team • Senior server side engineer • livedoorχϡʔε, LINEΪϑτ, LINE BLOG • LINE Ads Platform • Smart Channel • Spring fest 2018 fall

3. Smart Channel

4. • Overview • Architecture • Team Agenda

5. Overview

6. Concept Contents Contents

7. UX

8. • User͕࢖͍ͬͯΔɺ͋Δ͍͸ڧ͍རӹͷ͋ΔServiceͷ৘ใ • ServiceͰ֤UserʹPersonalizeͯ͠࡞ΒΕͨ΋ͷ • ࣄલTest഑৴Λͯ͠ҰఆͷධՁΛ௒͑ͳ͚Ε͹ͳΒͳ͍(ޙड़) Contents

9. • Userʹର͢ΔRecommendΛ֤Service͔ΒूΊΔ • ूΊͨ Contents ΛҰ࣌తʹอଘ͓ͯ͘͠ • RequestΛड͚ͨΒContentsΛRankingͯ͠n݅બΜͰ഑৴͢Δ ͜ͷαʔϏεͷಛ௃

10. Figures MAU 79M RPS 250k Servers 200+ * JP͚ͩͷ਺஋

11. Architecture

12. None

13. Java + Spring

14. • Redis • MySQL • Kafka • Spring boot 2

    • lettuce5 • MyBatis + kotlin • Caffeine • fastutil, guava Spec

15. Application 86 pm servers 40 vm servers MySQL 3 vm

    servers Redis 100+ vm instance (Redis cluster) Kafka In-house common (Kafka cluster) Log storage In-house common (Hadoop) Servers

16. Importer CRS Engine Event Tracker Joiner Learning worker 5ͭͷComponent

17. CRS Engine Flow Event Tracker Joiner Learning worker Importer LINE

    App Contents Contents Imp/Click Log Log Parameter

18. CRS Engine 1. Importer Event Tracker Joiner Learning worker Importer

    LINE App Contents Contents Imp/Click Log Log Parameter
19. None

20. • Mapping provider͔Βɺrecommend item ͷID͚ͩ΋Β͏ • item͕ͲΜͳσʔλΛ͍࣋ͬͯΔ͔໰͍ ߹ΘͤΔ • Redisʹmapping

    ͱ contentsΛstore͢Δ Importer user_id1 \t item_id1, item_id2, item_id3 user_id2 \t item_id3, item_id4 user_id3 \t item_id1, item_id2, item_id6 { "items": [ { "originalId": "12345678", "title": “ࣗ෼ͷೳྗΛݟۃΊͯ", "description": "ແཧͳεέδϡʔϧͰମௐΛ่ͯ͠͠·͏ΑΓɺ ૉ௚ʹपΓʹཔΓ·͠ΐ͏ɻࠓ͸ࣗ෼ʹݫͯ͘͠͠͠·͍͕ͪɻ ࣗ෼ͷೳྗΛ௒͑ΔͱࢥͬͨΒঢ়گΛ࿩ͯ͠ڠྗͯ͠΋Βͬͯ͘ ͍ͩ͞ɻ", "linkURL": “line://ch/1391620050/z/fortune/detail/2/20181011", "iconURL": “http://example.com/path/to/image1.png" }, ……. ] }

21. • ֤ServiceͰRecommendΛ࡞ͬͯ΋Β͏ • ྑ͍Recommendʹ͸ڧྗͳDomain஌͕ࣝඞཁ • ServiceʹΑͬͯ͸ࡉ͔͍දࣔͷͨΊͷ৚͕݅͋Δ Recommend

22. • Recommendation dataͭ͘Δͷ͸ ML engineer • Master data ؅ཧͯ͠Δͷ͸ Service

    side engineer • ୲౰෦ॺ͕ҧ͏ʢ৔߹ʹΑͬͯ͸ࠃ΋ҧ͏ʣ • RecommendΛ࡞ͬͨ࣌ͱ͕࣌ࠩ͋Δ Կނ2ճʹ෼͚ͯσʔλΛऔΔ͔

23. • Spec͸ͱʹ͔͘ΨνΨνʹݻΊ͓ͯ͘ • Documment͸͔ͬ͠Γἧ͑ͯӳޠͰॻ͍͓ͯ͘ • Serviceͷࢥ૝΍໨తΛͪΌΜͱ఻͑Δ • جຊతʹࣄલTestΛͯ͠ΫϦΞ͠ͳ͍ͱ͍͚ͳ͍ ؾΛ͚͍ͭͯΔ͜ͱ

24. • CTR΍xCTRʹΑΔ਺஋ධՁ • ΞϯέʔτͳͲͷఆੑతධՁ • ্ه͸ޓ͍ʹڧ͍૬ؔؔ܎༗Γ Preliminary test

25. • Recommend͸ServiceଆͰ࡞ͬͯ΋Β͏ • Recommend id/dataͷऔಘΛ෼཭͍ͯ͠Δ • औಘͨ͠Contents͸redisʹશ෦ೖΕΔ • Service࿈ܞલʹ͍ͭ͘΋ͷνΣοΫ͕͋Δ Importer

    ·ͱΊ

26. CRS Engine 2. CRS Engine Event Tracker Joiner Learning worker

    Importer LINE App Contents Contents Imp/Click Log Log Parameter
27. None

28. • UserຖʹRecommend contentsऔಘ • Frequency cap / mute / opt-outͳͲͰfiltering

    • Recommend itemΛRanking্͔ͯ͠Βn݅ฦ͢ • Contextual bandit algorithmsΛ࠾༻ CRS Engine

29. • Reactive API • RedisAdvancedClusterReactiveCommands • ໭Γ஋͕ Mono / Flux

    • ݸਓతʹ͸͘͢͝εοΩϦ͔͚ͯ޷͖ Lettuce5

30. • Reactive Streamsͷpublisher࣮૷ • Mono͸࠷େ1ݸͷitemΛѻ͏ • Flux͸ෳ਺ͷitemΛѻ͏ Mono/Flux

31. Mono/Fluxͷྫ targetingRedis.mget(keys) // ͜͜͸Flux͕ؼͬͯ͘Δ .filter(KeyValue::hasValue) .map(keyValue -> Maps.immutableEntry(extractTargetingType(keyValue.getKey()), keyValue.getValue())) .collectMap(Map.Entry::getKey,

    Map.Entry::getValue); // ໭Γ஋͸ɹMono<Map<String, List<String>>> Key: user_id:news Val: ["65819383620a","48d2872c86c6","293d0a33f938","c43cd18c98f4"] Key: user_id:weather Val: [“1"] -> ͜ΕΛnews, weatherΛkeyͱͨ͠Mapͱ͔ʹͯ͠ฦ͍ͨ͠ ࣮ࡍʹॲཧ͞ΕΔͷ͸subscribeͨ࣌͠

32. • Flux͔Βstreaming apiతͳॲཧΛεϜʔζʹ͔͚Δ • ଈॲཧͰ͸ͳ͘ॲཧΛޙͰߦ͑Δ • Mono.zip()ͳͲΛ࢖ͬͯฒྻॲཧ΋Ͱ͖Δ • ͨͩɺ·ͩͦ͜·Ͱ͸΍͍ͬͯͳ͍ Mono/FluxͷϝϦοτ

33. subtitle Bandit algorithms • SlotͷrewardΛ࠷େԽ͢Δ • ࣄલڞ༗৘ใ͕ແ͍৔߹΋͋Δ • ϥ΢ϯυຖʹͲͷarmΛબͿ͔ܾఆ͢Δ •

    ϙΠϯτ͸୳ࡧͱ׆༻Λ࢖͍෼͚Δ͜ͱ

34. • ࠷ॳ͸data͕଍Γͳ͍ͷͰɺͱΓ͋͑ͣճͯ͠ΈΔʢ୳ࡧʣ • ͋Δఔ౓ճ͢ͱ༏ྼ͕ղͬͯ͘Δ • Ұ൪ظ଴͞ΕΔreward͕ྑ͍ͱ͜ΖΛճ͢ʢ׆༻ʣ • ΋ͬͱใु͕ྑ͍Ϛγϯ͕͋Δ͔୳͢ʢ୳ࡧʣ ୳ࡧͱ׆༻

35. • Rewardͷఆٛ͸Userͷreaction (CTRͱ͔xCTRͱ͔) • UserҰਓҰਓʹarmΛ෼͚͍͕ͨɺऩଋͣ͠Β͘ͳΔ • ͳͷͰɺ΋͏গ͠େཻ͖͍౓Ͱֶश͍ͯ͠Δ • ਫ਼౓Λ্͛ΔͨΊʹɺ͍͔ͭ͘৘ใʢ=ContextʣΛ༩͍͑ͯΔ Contextual

    bandit on CRS engine

36. User side • Gender • Age • Interests Contents side

    2-side features • Time • Id • Score

37. • ঁੑ > உੑ • ए͍ੈ୅ͷํ͕ؾʹ͢ΔͷͰ͸ʁ • உੑ͸ͦΜͳʹݟͳ͍ʁ User side

    - ઎͍ Age-range x Gender

38. • Ͳ͏ͤΈΔͳΒேͩΑͶ • ྑ͍݁Ռͩͱclick͠΍͍͢ͷͰ͸ • ѱ͍݁Ռ͸ݟͨ͘ͳ͍ʁ Contents side - ઎͍

    Age-range x Gender x Hour x Score

39. LinUCB Bayesian linear regression Bayesian factorization machines Supported Algorithms Bayesian

    neural factorization machines

40. • RedisΛ৭ʑࢀর͍ͯͯ͠Δ • Contextual bandit algorithms ͷRankerͱͯ͠n݅બͿ • BanditͷContext͸৭ʑ޻෉͍ͯ͠Δ CRS

    engine ·ͱΊ

41. CRS Engine 3. Event Tracker Event Tracker Joiner Learning worker

    Importer LINE App Contents Contents Imp/Click Log Log Parameter
42. None

43. • imp/click/muteͳͲͷeventΛड͚Δ • Ұ୴Kafkaʹproduce͢Δ͚ͩ • consume͢Δcomponent͸෼͚ͯͯΔ • KafkaΛ࢖͏ϝϦοτΛ1൪ڗडͰ͖Δ Event tracker

44. • ߴ͍scalabilityͱavailability • Persistent data • Pub-Subܕ (kafkaͷ༻ޠͱͯ͠͸produce / consume)

    • ෳ਺consumer͔ΒಡΊΔ Kafka

45. • eventΛอଘ͢Δॲཧͱɺ࣮ࡍʹԿ ͔͢ΔॲཧΛ෼͚ΒΕΔ • ผͷconsumer͔Β΋ಡΊΔ • ো֐࣌ʹӨڹΛ࠷খݶʹͰ͖Δɺ ಛఆ͠΍͘͢ͳΔ KafkaͷϝϦοτ

46. • Spring bootͱ૬ੑ͕ͱͯ΋ྑ͍ • @BeanͰKafka-clientΛ؅ཧ͢Δͱੜ੒/ഁغ·Ͱ໘౗ݟͯ͘ΕΔ • Kafka-streamsͱ૊Έ߹ΘͤΔ͜ͱͰίʔυॻ͘ྔ͕ͱͯ΋গͳ͍ • Spring-kafka΋͋Δ͚ͲެࣜKafka-client͕Φεεϝ Kafka

    on spring

47. @Configuration @RequiredArgsConstructor @EnableConfigurationProperties({ KafkaStreamsProperties.class, KafkaTopicProperties.class }) public class KafkaStreamsConfiguration {

    @Bean public KafkaStreams muteEventStream( KafkaStreamsProperties streamsProperties, KafkaTopicProperties topicProperties ) { final StreamsBuilder builder = new StreamsBuilder(); builder.stream(topicProperties.getMuteEvent(), Consumed.with(new StringSerde(), new StringSerde())) .foreach((key, value) -> { try { // Կ͔ॲཧ } catch (Exception e) { // ྫ֎ॲཧ } }); KafkaStreams streams = new KafkaStreams(builder.build(), streamsProperties.getMuteEventStream()); streams.setUncaughtExceptionHandler((thread, e) -> { log.error("Uncaught exception in StreamThread.", e); }); streams.start(); return streams; } }

48. kafka: servers: server1, server2, server3 metrics-reporter: com.example.awesome.kafka.metrics.PrometheusMetricsReporter topic: mute-event: awesome-mute-event-v1

    stream: mute-event-stream: "bootstrap.servers": ${kafka.servers} "application.id": crs-mute-event "state.dir": /path/to/kafka-streams "num.stream.threads": 6 "replication.factor": 3 "metric.reporters": ${kafka.metrics-reporter} consumer: "max.poll.records": 100 "auto.offset.reset": earliest @ConfigurationProperties(prefix = "kafka.stream") public class KafkaStreamsProperties { @NotNull @Valid private Properties muteEventStream; }

49. • LINEΞϓϦ͔ΒͷEventΛड͚औΔॲཧ • KafkaΛڬΉ͜ͱͰɺߴ଎/଱ো֐ੑΛ࣮ݱͯ͠Δ • ݸਓతʹ͸KafkaҰԡ͠Ͱ͢ Event Tracker ·ͱΊ

50. CRS Engine 4. Joiner Event Tracker Joiner Learning worker Importer

    LINE App Contents Contents Imp/Click Log Log Parameter
51. None

52. • ϢʔβʔͷContentsʹର͢Δ൓Ԡ͕஌Γ͍ͨ • imp͞Εͨcontents͕click͞Ε͔ͨɺmute͞Ε͔ͨ • impͱclick/muteΛjoin͍ͨ͠ ΍Γ͍ͨ͜ͱ

53. • imp͕ಧ͍ͨஈ֊Ͱ͸click΍mute͸ະདྷͷࣄ • click͕དྷͨ࣌఺Ͱ͸imp͸ॲཧ͞Εͯ͠·͍ͬͯΔ • impΛ͋Δఔ౓஗Ԇͤͯ͞ॲཧ͍ͨ͠ Ͳ͏͢Ε͹ྑ͍͔

54. • impͷconsumeΛ10min஗ΒͤΔ • click΍mute͸ͦͷ··Redisʹ • 10minޙimpΛॲཧ͢Δͱ͖ʹredisΛࢀর ஗Ԇconsumer https://www.slideshare.net/linecorp/line-ads-platformkafka-126180436

55. • MLͷڭࢣσʔλͷूܭॲཧ • KafkaΛԠ༻ͨ͠஗ԆConsumerॲཧ Joiner ·ͱΊ

56. CRS Engine 5. Learning worker Event Tracker Joiner Learning worker

    Importer LINE App Contents Contents Imp/Click Log Log Parameter
57. None

58. • engine͕rankerɺ͜͜͸trainerଆ • σʔλྔ͕ଟ͍ͷͰฒྻԽ͍ͨ͠ • ޮ཰ԽͷͨΊ·ͱΊͯॲཧ͍ͨ͠ • ಉ͡armͷσʔλ͸ಉ͡αʔόʔͰॲཧ Learning worker

59. • BanditͷTrainer͸Python • NginxͰ Consistent hashing • User trafficΛड͚ΔRanker͸java Learning

    worker

60. • NumPy/Numba͕࢖͍͔ͨͬͨ • BanditΛ։ൃͯ͠Δͷ͸datalabs -> ML part • Ranker͸҆ఆੑΛॏࢹ͍ͨ͠ͷͰJavaͰؤுͬͯॻ͘ •

    Python <-> JavaͷΠϯλʔϑΣʔε͸protobufͰఆٛ Why Python

61. • Bandit algorithmsͷTrainer • ඇಉظ/ޮ཰తʹֶश͢ΔͨΊʹ͍Ζ͍Ζ޻෉ͯ͠Δ • Java <-> PythonͰparameterΛड͚౉͠ •

    ౉͠ํ΍ֶशͷ࢓ํͰ͍͔ͭ͘޻෉Λ͍ͯ͠Δ Learning worker·ͱΊ

62. Team

63. subtitle Developers • Server side engineer 5 ਓ • ML

    team΍֤Service developerͱίϥϘϨʔγϣϯ • Globalͷϝϯόʔͱͷ΍ΓऔΓ΋සൟʹ͋Δ • ·ͩ·ͩ໘ന͍Spec͋ΔͷͰ΍͍͖͍ͬͯͨ

64. THANK YOU