$30 off During Our Annual Pro Sale. View Details »

LINEのBot Platformの裏側 / Behind the LINE Bot Platform

LINEのBot Platformの裏側 / Behind the LINE Bot Platform

JJUG CCC 2019 Springでの登壇資料です

LINE Developers
PRO

May 18, 2019
Tweet

More Decks by LINE Developers

Other Decks in Technology

Transcript

  1. LINEͷBot Platformͷཪଆͷ࿩
    Ryosuke Hasebe, LINE Corporation
    JJUG CCC Spring 2019

    View Slide

  2. @be_hase1014 https://github.com/be-hase
    About me
    ● ௕୩෦ ྑี
    ● LINEגࣜձࣾ Account Platform։ൃࣨ AP4νʔϜ Ϛωʔδϟʔ
    ● 2013೥ ೖࣾ
    ● Server-side Engineer
    ● LINEνϟοτ (for LINEެࣜΞΧ΢ϯτ)
    ● Before:
    ● LINE Login, LINE ΧελϚʔίωΫτ, LINE Notify, 

    LINE Profile+, LINE GAME Platform, …etc

    View Slide

  3. • LINEͱJava
    • LINE Bot Platformʹ͍ͭͯ
    • LINEνϟοτ(for LINEެࣜΞΧ΢ϯτ)
    ͷ։ൃࣄྫΛ۷ΓԼ͛ͯ঺հ
    Agenda

    View Slide

  4. LINEͱJava

    View Slide

  5. ݴޠ͝ͱͷrepo਺ (ࣾ಺GHEௐ΂)

    View Slide

  6. Bot platform component overview

    View Slide

  7. Bot platform component overview
    Java ͨ͘͞Μʂ

    View Slide

  8. ● ܕ෇͖, ଎͍
    ● ๛෋ͳϥΠϒϥϦ
    ● ձࣾશମతʹੲ͔Β࢖͍ͬͯΔͷͰࢿ࢈͕ଟ͍
    ● ੈքతʹJavaΤϯδχΞ͕ଟ͍
    ● ։ൃڌ఺͸೔ຊҎ֎ʹ΋୆࿷/λΠ/ؖࠃ/தࠃ/ϕτφϜ౳ʹ͋Δ
    ● ڌ఺Λ·͍ͨͩϓϩμΫτͷҠ؅΋͋Δ
    ● ͱ͸͍͑ඞͣ͠΋JavaΛར༻͠ͳ͍ͱ͍͚ͳ͍ͱ͍͏Θ͚Ͱ͸ͳ͍
    Why Java?

    View Slide

  9. LINE Bot platformʹ͍ͭͯ

    View Slide

  10. What’s LINE Bot Platform?
    ● LINEެࣜΞΧ΢ϯτ(๏ਓ޲͚ΞΧ΢ϯταʔϏε)ʹ͓͍ͯBotΛ
    ϕʔεͱͨ͠ϏδωεɾιϦϡʔγϣϯΛఏڙ͢ΔͨΊͷPlatform
    ͷ͜ͱ
    ● ϝοηʔδૹ৴/ड৴
    ● λΠϜϥΠϯ౤ߘ
    ● … etc (ޙड़)
    اۀ΍ݸਓ Ϣʔβʔ

    View Slide

  11. What’s its VALUE?
    ѹ౗తͳϦʔνྗ ʂʂ
    ສਓ
    8000
    Percent
    86
    ೔ຊͷਓޱͷ
    60%Ҏ্
    ຖ೔ར༻͍ͯ͠Δ
    ೔ຊࠃ಺ͷϢʔβʔ

    View Slide

  12. ఏڙ͍ͯ͠Δػೳ

    View Slide

  13. ● ϝοηʔδ഑৴
    ● ࣗಈԠ౴/ΩʔϫʔυԠ౴
    ● Ϧονϝχϡʔ
    ● λΠϜϥΠϯ౤ߘ
    ● γϣοϓΧʔυ
    ● Ϋʔϙϯ
    ● ֤छΞφϦςΟΫεͷఏڙ
    ● ܾࡁ
    ● …etc
    LINE Official Account Manager
    (CMS)

    View Slide

  14. ● ΞΧ΢ϯτ؅ཧऀ༻ͷνϟοτπʔϧ
    ● εςʔλεŋλάΛ༻͍ͨސ٬؅ཧ
    ● ޙ΄Ͳ։ൃࣄྫΛ঺հ͠·͢
    LINEνϟοτ

    View Slide

  15. Messaging API
    ● ςοΫاۀ / ։ൃऀ޲͚ͷAPI܈ɻChatbotΛࣗ༝ʹ࡞੒/࣮૷Մೳ
    ● ϝοηʔδͷड৴͸Webhook
    ● ࣮͸ࣾ಺ͷ֤छαʔϏε΋࢖༻͍ͯ͠·͢
    send messages
    Receive messages via
    webhook

    View Slide

  16. ● LINE Beacon
    ● ༑ͩͪΦʔσΟΤϯε഑৴
    ● LINE LIVE࿈ܞ
    ● …etc
    Others

    View Slide

  17. ϝοηʔδ഑ૹ਺

    (݄ؒ)
    800ԯ
    ΞΧ΢ϯτ਺
    900ສ+
    LINE Bot Platformͷن໛ײ
    Messaging API
    daily requests
    2.2ԯ+
    αʔόʔ਺
    (app serverͷΈ)
    500+

    View Slide

  18. CMS server
    Dev Team
    Chat server Bot backend server
    CMS & Core web-front Chat web-front Native App
    ● ։ൃڌ఺͸೔ຊɺؖࠃɺϕτφϜ
    ● Native App͸΄΅WebViewͱ࣮ͯ͠૷͍ͯ͠ΔͷͰfrontͷਓ਺͕ଟΊ
    ● ଞʹ΋֤छઐ໳νʔϜͱ
    ● ITSC (Private cloud, LB, …), DBA (MySQL, HBase, Redis), 

    IMF (Kafka), Datalabs, QA, …etc
    6ਓ 6ਓ 8ਓ
    17ਓ 4ਓ 5ਓ

    View Slide

  19. LINEνϟοτͷ։ൃࣄྫ

    View Slide

  20. (͓͞Β͍) LINEνϟοτ

    View Slide

  21. api streaming api content
    • Spring Boot 2
    • Spring WebFlux
    • Redis Cluster
    • Redis PubSub
    • Spring Boot 2
    • Spring MVC
    • MySQL
    • Redis Cluster
    LINEνϟοτͷcomponent
    event receiver event processor chatbox
    • Spring Boot 1
    • Armeria(thrift)
    • HBase
    • Spring Boot 2
    • Spring WebFlux
    • Spring Boot 2
    • Spring WebFlux
    • Kafka
    • Spring Boot 2
    • Kafka
    • MySQL
    • Redis Cluster
    • Redis PubSub

    View Slide

  22. View Slide

  23. View Slide

  24. ● ͘͝ҰൠతͳJSON API server
    ● Φʔφʔͷૢ࡞ʹΑΔ֤छϏδωεϩδοΫ
    ● ϝοηʔδૹ৴
    ● աڈϝοηʔδऔಘ
    ● εςʔλε/λάͳͲͷσʔλอଘ
    ● MySQL΁Blocking IOΛ͍ͯ͠ΔͷͰ

    ੲͳ͕ΒͷSpring MVC(tomcat)
    api

    View Slide

  25. View Slide

  26. ● ChatΞϓϦέʔγϣϯʹ͸realtimeੑ͕ඞཁ
    ● Server-Sent events(SSE) by Spring WebFlux
    ● WebSocketΛ࠾༻͢Δ͔೰Μ͕ͩɺ૒ํ޲௨
    ৴͸ඞཁͳ͍ͷͰSSEΛ࠾༻
    ● LINE user͔ΒͷϝοηʔδΛrealtimeʹ൓ө
    ● ଞͷΦʔφʔʹΑΔ֤छૢ࡞΋
    streaming api
    Send messages
    Send message

    View Slide

  27. ● WebFlux
    ● Spring 5͔Βొ৔ͨ͠Non Blocking WebϑϨʔϜϫʔΫ
    ● SSEͷΑ͏ͳlong connectionΛ࢖༻͢ΔΞϓϦέʔγϣϯ, Gateway server
    ౳ͱ͸ಛʹ૬ੑ͕͍͍
    ● Request threadΛઐ༗͠ͳ͍
    ● ैདྷͷServlet APIͰ͸request threadͷ਺͔͠εέʔϧ͠ͳ͍
    SSEͱSpring WebFlux

    View Slide

  28. Example code
    ௕࣌ؒ, eventΛૹΒͳ͍ͱ઀ଓ͕
    ੾ΒΕΔͷͰpingૹ৴༻ͷFluxΛ
    ࡞੒ͯ͠merge͢Δ

    View Slide

  29. ● ωοτϫʔΫ઀ଓ͕੾Εͯ࠶઀ଓͨ͠ͱ͖ʹɺ

    ͦͷؒͷeventΛड৴Ͱ͖ΔΑ͏ʹ͓ͯ͘͠ඞཁ
    ͕͋Δ
    ● SSEʹ͸Last-Event-ID headerͱ͍͏

    ࢓૊Έ͕͋Γɺ࠶઀ଓ͢Δͱ͖ʹ࠷ޙʹड৴͠
    ͨevent IDΛrequest headerʹؚΊͯ͘ΕΔ
    ● RedisʹPub͢Δͱ͖ʹಉ࣌ʹͦͷdataΛListͱ
    ͯ͠΋อ͓࣋ͯ͘͠
    ● ࠶઀ଓͷࡍʹ͸ͦͷList͔ΒऔಘͰ͖ΔΑ͏ʹ
    ● Redis 5͔Β௥Ճ͞ΕͨRedis StreamsΛ࢖͏ͱ
    ָͰ͖ͦ͏ɻ

    ࠓޙࢼ͍ͨ͠ (ݱࡏ͸Redis 4࢖͍ͬͯΔ)
    SSEͷ࠶઀ଓ
    Send messages
    Send message

    View Slide

  30. ● Redis 3͔Βొ৔ͨ͠Redis Clusterʹ΋ରԠ͍ͯ͠Δ͕ҎԼͷΑ͏ͳ໰୊΋͋Δ
    Redis PubSubͷ஫ҙ఺
    In a Redis Cluster clients can subscribe to every node, and can
    also publish to every other node. The cluster will make sure that
    published messages are forwarded as needed. The current
    implementation will simply broadcast each published message
    to all other nodes, but at some point this will be optimized either
    https://redis.io/topics/cluster-spec#publishsubscribe
    ● εέʔϧ͕ඞཁʹͳͬͨΒɺผͷclusterΛ༻ҙͯ͠ৼΓ෼͚ΔΑ͏ʹͨ͠΄
    ͏͕͍͍

    View Slide

  31. View Slide

  32. ● LINE User͕Botʹରͯ͠ϝοηʔδΛૹ৴
    ͢Δ౳ͷactionΛ͢ΔͱɺWebhook͕ಧ͘
    ● WebhookΛड͚औͬͨΒଈ࠲ʹKafkaʹ
    produce
    ● KafkaΛڬΉ͜ͱͰ଱ো֐ੑΛߴΊ͍ͯΔ
    ● consumerଆͰretryՄೳ
    event receiver

    View Slide

  33. ● Kafkaࣗମͷ଱ো֐ੑ͸ͱͯ΋ߴ͍ɻͦΕͰ΋ͳʹ͔͠ΒͷτϥϒϧͰো
    ֐͕ग़Δ͜ͱ΋͋Δ
    ● ͦΕʹඋ͑ͯKafka team͸2ͭͷKafka ClusterΛ༻ҙͯ͘͠Ε͍ͯΔ
    ● Primary Cluster͕downͨ͠ͱ͖͸Secondarily Clusterʹproduce͢Δ
    Α͏ʹ͍ͯ͠Δ
    ● Primary/SecondarilyؒͰ࣮֬ͳॱ൪อূ͸Ͱ͖ͳ͘ͳΔ͕

    αʔϏεܧଓΛ༏ઌ
    KafkaͰো֐ʹͳͬͨΒ?

    View Slide

  34. View Slide

  35. ● ֤छeventΛॲཧ͢Δkafka consumer server
    ● event receiver͕produceͨ͠event
    ● ଞͷαʔϏε͕produceͨ͠event
    event processor
    Consume

    View Slide

  36. ● ଞͷαʔϏε͕อ͍࣋ͯ͠Δσʔλ͕ߋ৽͞Εͨͱ͖ʹͦͷ৘ใΛड͚औΔ͜ͱ
    ͕Ͱ͖Δͱศར
    ● ϦΞϧλΠϜʹ৘ใΛߋ৽Ͱ͖ͨΓCacheΛ࡟আͨ͠Γ͢Δ͜ͱ͕Ͱ͖Δ
    ● Kafka͸consumer groupΛ෼͚Δͱಉ͡topicʹରͯ͠ෳ਺ͷconsumerΛ༻ҙ͢Δ
    ͜ͱ͕Ͱ͖Δ
    ● ୯७ͳΩϡʔͱ͚ͯͩ͠Ͱ͸ͳͯ͘͜͏͍ͬͨ༻్ʹ΋Ԡ༻Ͱ͖Δ
    ϚΠΫϩαʔϏεؒͷ

    Data Hubͱͯ͠ͷKafka
    https://logmi.jp/tech/articles/320330

    View Slide

  37. ● มߋ͕͋ͬͨͱ͖౳ʹ֤αʔϏεʹ௨஌
    ● ີ݁߹...
    ● ར༻αʔϏε͕૿͑Δͱਏ͍
    ੲ΍ͬͯͨବ໨ͳࣄྫ

    View Slide

  38. View Slide

  39. ● image, audio, video, file messageͷProxy server
    ● సૹྔ͕େ͖͍ͨΊtomcatͩͱrequest threadΛ
    ઐ༗͕ͪ͠ɻWebFluxΛ࢖༻
    content

    View Slide

  40. View Slide

  41. ● Botʹؔ͢Δmessage΍֤छoperationΛอଘ͢Δαʔόʔ
    ● Userͷmessageͱҧͬͯอଘظݶ͕௕Ί
    ● Ϗδωε༻్Ͱ͸อଘظݶ͕௕Ί͡Όͳ͍ͱݫ͍͠
    ● LINE messengerΞϓϦͱಉ༷ʹHBaseΛར༻
    ● ݱࡏ͸ࣾ಺ར༻ͷΈ
    ● ͦͷ͏ͪ֎޲͚ʹ΋ެ։ͯ͠֎෦ͷ։ൃऀͷෛ୲ΛݮΒ͍ͨ͠
    chatbox

    View Slide

  42. Tools

    View Slide

  43. ● ࣾ಺Logging platform
    ● ֤छlog dataΛKafkaʹૹΔͱKibana౳ͰݟΔ͜ͱ͕Ͱ͖Δ
    ● Request log
    ● application log (via logback appender)
    ● ௐࠪͳͲͰ͘͢͝໾ཱ͍ͬͯΔ
    Datachain

    View Slide

  44. ● logbackʹ͸ThreadLocalϕʔεͰ࣮૷͞ΕͨMDCͱ͍͏࢓૊Έ͕͋Δ
    ● logʹ௥Ճ৘ใΛ෇༩Ͱ͖Δ
    ● ྫ͑͹ServletFilterͰrequest৘ใΛMDCʹPut͓ͯ͘͠ͱศར
    MDC࢖͏ͱ͞Βʹศར

    View Slide

  45. WebFluxͰ΋MDC࢖͍͍ͨ
    ● WebFluxͰ΋͜ͷςΫχοΫΛҾ͖ଓ͖࢖͍͍͕ͨ…
    ● ୅ΘΓʹ࣍ͷ3ͭΛ࢖͏͜ͱͰ࣮ݱͰ͖Δ
    ● WebFilter
    ● Context
    ● Hooks#onEachOperator
    ● ※ spring-cloud-sleuth΋ಉ͡Α͏ͳ͜ͱΛ͍ͯ͠Δ

    View Slide

  46. ● WebFilter Ͱ request৘ใΛContextʹ͍ΕΔ
    WebFluxͰ΋MDC࢖͍͍ͨ

    View Slide

  47. ● ֤छonXXXXϝιουΛϥοϓ͢ΔΑ͏ͳCustom SubscriberΛ࣮૷͢Δ
    ● Context͔ΒWebFilterͰొ࿥ͨ͠request৘ใΛऔಘ͢Δ
    ● subscriber#onXXXΛݺͼग़͢લʹMDC#put͠ɺݺͼग़͠ޙʹMDC#remove͢Δ
    WebFluxͰ΋MDC࢖͍͍ͨ

    View Slide

  48. ● ઌఔ࡞ͬͨCustom Subscriber͕࢖ΘΕΔΑ͏ʹHooksʹొ࿥͢Δ
    ● Spring BootͳΒConfigurationʹॻ͘
    WebFluxͰ΋MDC࢖͍͍ͨ

    View Slide

  49. ● Spring Boot 2͔ΒMicrometer͕࢖༻͞Ε͍ͯΔ
    ● ಛʹͳʹ΋ઃఆ͠ͳͯ͘΋ྑ͍ײ͡ʹMetricsΛͱͬͯ͘Ε͍ͯΔ
    ● ඞཁʹԠͯࣗ͡෼Ͱ࣮૷͢Δ
    ● Metricsͷͳ͍ίʔυ͸ςετ͕ͳ͍ίʔυҎ্ʹࠔΔͷͰ

    ͔ͬ͠ΓऔΖ͏ɻো֐࣌ʹޙͷࡇΓͱͳΔ
    Metrics Monitoring

    View Slide

  50. ● LINEͰ͸MetricsΛPrometheusʹexportͯ͠GrafanaͰݟΔ͜ͱ͕ଟ͍
    Metrics Monitoring

    View Slide

  51. ࠷ۙ͋ͬͨIssues

    View Slide

  52. ● ͪΐ͏ͲҰࡢ೔ى͖ͨλΠϜϦʔͳIssue
    ● ྫ͑͹Webflux ClientΛ༻͍ͨ͜Μͳcode͕͋Δͱ͠·͢
    OOME Issue
    ● ΋͠΋͜ͷAPI callͰtimeout͕େྔʹൃੜ͍ͯ͘͠ͱɺ

    ͳΜͱOOEMͰࢮʹ·͢

    View Slide

  53. ● netty͔ΒReadTimeoutException͕ൃੜ͢Δͱreactor core಺Ͱ͜Ε͕࣮ߦ͞ΕΔ
    ● Exceptions#propagate͸RuntimeExceptionͷαϒΫϥεͷ৔߹͸ͦͷ··ฦͯ͠
    ͍ΔɻReadTimeoutException͸RuntimeExceptionͷαϒΫϥεͳͷͰ
    ReadTimeoutExceptionࣗ਎͕ฦΔ
    ● this is ok, as re is alway a new non-singleton instanceͱॻ͍ͯ͋Δ͕…
    ReadTimeoutException
    OOME Issue
    ReadTimeoutException

    View Slide

  54. ● ReadTimeoutException͸singletonͩͬͨ
    ● ͦͷ݁Ռ, ReadTimeoutException͕อ͍࣋ͯ͠Δ
    suppressedExceptions(List)͕ͲΜͲΜͱංେԽͯ͠OOME͕ى͖Δ
    ● heapdumpΛௐ΂ͨΒؾ͍ͮͨ
    ● nettyʹGithub issue͋͛ͨΒͦͷ೔ͷ͏ͪʹPRग़ͯͨ

    https://github.com/netty/netty/pull/9152
    ● ͍ۙ͏ͪʹmaven repoʹ΋͕͋Γͦ͏
    OOME Issue

    View Slide

  55. ● LINEνϟοτͷ໰୊Ͱ͸ͳ͍͕Bot platform΍֤छࣾ಺αʔϏεͰ
    ى͖͍ͯͨ໰୊
    ● LINEνϟοτ͸lettuce(reactive api)࢖͍ͬͯΔ
    ● https://github.com/xetorthio/jedis/issues/1945
    ● connection pool͔Βऔಘ͢Δͱ͖ʹhang͢Δ
    ● Spring BootͷversionΛ͋͛Δͱ͖͸֤छϥΠϒϥϦͷversion΋͋
    ͕ΔͷͰ͔ͬ͠Γͱ֬ೝ͠·͠ΐ͏
    Jedis(2.9.1) Connection leaking

    View Slide

  56. ● Reactorͷnon-blocking threadͰblock͢ΔΑ͏ͳcodeΛॻ͍ͯ͸NG
    ● performance௿Լ͸౰વͱͯ͠ɺҙਤ͠ͳ͍deadlock΋ى͖·͢
    ● https://spring.io/blog/2019/03/28/reactor-debugging-experience#blockhound-a-
    new-kid-on-the-block
    ● Reactor nettyͰ͸InetSocketAddressΛ࢖ͬͨblocking DNS resolverΛ࢖༻
    ͍ͯ͠ΔͷͰɺnettyͷnon-blocking DNS resolverΛ࢖༻͢ΔΑ͏ʹมߋ͠
    ·͠ΐ͏
    ● (·ͩ Unstable ApiͰ͕͢)
    ● ΋͠΋DNS resolveʹ͕͔͔࣌ؒΔͱdeadlock͠·͢
    ● brennentsmith/slodns౳Λ࢖ͬͯҙਤతʹdelayͤͯ͞ΈΔͱ

    ࠶ݱ͠·͢
    ● Reactor nettyଆͰɺdefaultΛnettyͷnon-blocking DNS resolverʹ͠Α͏͔
    ͱ͍͏issue͸͋Δ
    ● https://github.com/reactor/reactor-netty/issues/569
    Blocking DNS resolver

    View Slide

  57. THANK YOU

    View Slide

  58. https://linecorp.com/ja/career/position/1643

    View Slide