Upgrade to Pro — share decks privately, control downloads, hide ads and more …

LINEのBot Platformの裏側 / Behind the LINE Bot Platform

LINEのBot Platformの裏側 / Behind the LINE Bot Platform

JJUG CCC 2019 Springでの登壇資料です

53850955f15249a1a9dc49df6113e400?s=128

LINE Developers
PRO

May 18, 2019
Tweet

Transcript

  1. LINEͷBot Platformͷཪଆͷ࿩ Ryosuke Hasebe, LINE Corporation JJUG CCC Spring 2019

  2. @be_hase1014 https://github.com/be-hase About me • ௕୩෦ ྑี • LINEגࣜձࣾ Account

    Platform։ൃࣨ AP4νʔϜ Ϛωʔδϟʔ • 2013೥ ೖࣾ • Server-side Engineer • LINEνϟοτ (for LINEެࣜΞΧ΢ϯτ) • Before: • LINE Login, LINE ΧελϚʔίωΫτ, LINE Notify, 
 LINE Profile+, LINE GAME Platform, …etc
  3. • LINEͱJava • LINE Bot Platformʹ͍ͭͯ • LINEνϟοτ(for LINEެࣜΞΧ΢ϯτ) ͷ։ൃࣄྫΛ۷ΓԼ͛ͯ঺հ

    Agenda
  4. LINEͱJava

  5. ݴޠ͝ͱͷrepo਺ (ࣾ಺GHEௐ΂)

  6. Bot platform component overview

  7. Bot platform component overview Java ͨ͘͞Μʂ

  8. • ܕ෇͖, ଎͍ • ๛෋ͳϥΠϒϥϦ • ձࣾશମతʹੲ͔Β࢖͍ͬͯΔͷͰࢿ࢈͕ଟ͍ • ੈքతʹJavaΤϯδχΞ͕ଟ͍ •

    ։ൃڌ఺͸೔ຊҎ֎ʹ΋୆࿷/λΠ/ؖࠃ/தࠃ/ϕτφϜ౳ʹ͋Δ • ڌ఺Λ·͍ͨͩϓϩμΫτͷҠ؅΋͋Δ • ͱ͸͍͑ඞͣ͠΋JavaΛར༻͠ͳ͍ͱ͍͚ͳ͍ͱ͍͏Θ͚Ͱ͸ͳ͍ Why Java?
  9. LINE Bot platformʹ͍ͭͯ

  10. What’s LINE Bot Platform? • LINEެࣜΞΧ΢ϯτ(๏ਓ޲͚ΞΧ΢ϯταʔϏε)ʹ͓͍ͯBotΛ ϕʔεͱͨ͠ϏδωεɾιϦϡʔγϣϯΛఏڙ͢ΔͨΊͷPlatform ͷ͜ͱ • ϝοηʔδૹ৴/ड৴

    • λΠϜϥΠϯ౤ߘ • … etc (ޙड़) اۀ΍ݸਓ Ϣʔβʔ
  11. What’s its VALUE? ѹ౗తͳϦʔνྗ ʂʂ ສਓ 8000 Percent 86 ೔ຊͷਓޱͷ

    60%Ҏ্ ຖ೔ར༻͍ͯ͠Δ ೔ຊࠃ಺ͷϢʔβʔ
  12. ఏڙ͍ͯ͠Δػೳ

  13. • ϝοηʔδ഑৴ • ࣗಈԠ౴/ΩʔϫʔυԠ౴ • Ϧονϝχϡʔ • λΠϜϥΠϯ౤ߘ • γϣοϓΧʔυ

    • Ϋʔϙϯ • ֤छΞφϦςΟΫεͷఏڙ • ܾࡁ • …etc LINE Official Account Manager (CMS)
  14. • ΞΧ΢ϯτ؅ཧऀ༻ͷνϟοτπʔϧ • εςʔλεŋλάΛ༻͍ͨސ٬؅ཧ • ޙ΄Ͳ։ൃࣄྫΛ঺հ͠·͢ LINEνϟοτ

  15. Messaging API • ςοΫاۀ / ։ൃऀ޲͚ͷAPI܈ɻChatbotΛࣗ༝ʹ࡞੒/࣮૷Մೳ • ϝοηʔδͷड৴͸Webhook • ࣮͸ࣾ಺ͷ֤छαʔϏε΋࢖༻͍ͯ͠·͢

    send messages Receive messages via webhook
  16. • LINE Beacon • ༑ͩͪΦʔσΟΤϯε഑৴ • LINE LIVE࿈ܞ • …etc

    Others
  17. ϝοηʔδ഑ૹ਺
 (݄ؒ) 800ԯ ΞΧ΢ϯτ਺ 900ສ+ LINE Bot Platformͷن໛ײ Messaging API

    daily requests 2.2ԯ+ αʔόʔ਺ (app serverͷΈ) 500+
  18. CMS server Dev Team Chat server Bot backend server CMS

    & Core web-front Chat web-front Native App • ։ൃڌ఺͸೔ຊɺؖࠃɺϕτφϜ • Native App͸΄΅WebViewͱ࣮ͯ͠૷͍ͯ͠ΔͷͰfrontͷਓ਺͕ଟΊ • ଞʹ΋֤छઐ໳νʔϜͱ • ITSC (Private cloud, LB, …), DBA (MySQL, HBase, Redis), 
 IMF (Kafka), Datalabs, QA, …etc 6ਓ 6ਓ 8ਓ 17ਓ 4ਓ 5ਓ
  19. LINEνϟοτͷ։ൃࣄྫ

  20. (͓͞Β͍) LINEνϟοτ

  21. api streaming api content • Spring Boot 2 • Spring

    WebFlux • Redis Cluster • Redis PubSub • Spring Boot 2 • Spring MVC • MySQL • Redis Cluster LINEνϟοτͷcomponent event receiver event processor chatbox • Spring Boot 1 • Armeria(thrift) • HBase • Spring Boot 2 • Spring WebFlux • Spring Boot 2 • Spring WebFlux • Kafka • Spring Boot 2 • Kafka • MySQL • Redis Cluster • Redis PubSub
  22. None
  23. None
  24. • ͘͝ҰൠతͳJSON API server • Φʔφʔͷૢ࡞ʹΑΔ֤छϏδωεϩδοΫ • ϝοηʔδૹ৴ • աڈϝοηʔδऔಘ

    • εςʔλε/λάͳͲͷσʔλอଘ • MySQL΁Blocking IOΛ͍ͯ͠ΔͷͰ
 ੲͳ͕ΒͷSpring MVC(tomcat) api
  25. None
  26. • ChatΞϓϦέʔγϣϯʹ͸realtimeੑ͕ඞཁ • Server-Sent events(SSE) by Spring WebFlux • WebSocketΛ࠾༻͢Δ͔೰Μ͕ͩɺ૒ํ޲௨

    ৴͸ඞཁͳ͍ͷͰSSEΛ࠾༻ • LINE user͔ΒͷϝοηʔδΛrealtimeʹ൓ө • ଞͷΦʔφʔʹΑΔ֤छૢ࡞΋ streaming api Send messages Send message
  27. • WebFlux • Spring 5͔Βొ৔ͨ͠Non Blocking WebϑϨʔϜϫʔΫ • SSEͷΑ͏ͳlong connectionΛ࢖༻͢ΔΞϓϦέʔγϣϯ,

    Gateway server ౳ͱ͸ಛʹ૬ੑ͕͍͍ • Request threadΛઐ༗͠ͳ͍ • ैདྷͷServlet APIͰ͸request threadͷ਺͔͠εέʔϧ͠ͳ͍ SSEͱSpring WebFlux
  28. Example code ௕࣌ؒ, eventΛૹΒͳ͍ͱ઀ଓ͕ ੾ΒΕΔͷͰpingૹ৴༻ͷFluxΛ ࡞੒ͯ͠merge͢Δ

  29. • ωοτϫʔΫ઀ଓ͕੾Εͯ࠶઀ଓͨ͠ͱ͖ʹɺ
 ͦͷؒͷeventΛड৴Ͱ͖ΔΑ͏ʹ͓ͯ͘͠ඞཁ ͕͋Δ • SSEʹ͸Last-Event-ID headerͱ͍͏
 ࢓૊Έ͕͋Γɺ࠶઀ଓ͢Δͱ͖ʹ࠷ޙʹड৴͠ ͨevent IDΛrequest

    headerʹؚΊͯ͘ΕΔ • RedisʹPub͢Δͱ͖ʹಉ࣌ʹͦͷdataΛListͱ ͯ͠΋อ͓࣋ͯ͘͠ • ࠶઀ଓͷࡍʹ͸ͦͷList͔ΒऔಘͰ͖ΔΑ͏ʹ • Redis 5͔Β௥Ճ͞ΕͨRedis StreamsΛ࢖͏ͱ ָͰ͖ͦ͏ɻ
 ࠓޙࢼ͍ͨ͠ (ݱࡏ͸Redis 4࢖͍ͬͯΔ) SSEͷ࠶઀ଓ Send messages Send message
  30. • Redis 3͔Βొ৔ͨ͠Redis Clusterʹ΋ରԠ͍ͯ͠Δ͕ҎԼͷΑ͏ͳ໰୊΋͋Δ Redis PubSubͷ஫ҙ఺ In a Redis Cluster

    clients can subscribe to every node, and can also publish to every other node. The cluster will make sure that published messages are forwarded as needed. The current implementation will simply broadcast each published message to all other nodes, but at some point this will be optimized either https://redis.io/topics/cluster-spec#publishsubscribe • εέʔϧ͕ඞཁʹͳͬͨΒɺผͷclusterΛ༻ҙͯ͠ৼΓ෼͚ΔΑ͏ʹͨ͠΄ ͏͕͍͍
  31. None
  32. • LINE User͕Botʹରͯ͠ϝοηʔδΛૹ৴ ͢Δ౳ͷactionΛ͢ΔͱɺWebhook͕ಧ͘ • WebhookΛड͚औͬͨΒଈ࠲ʹKafkaʹ produce • KafkaΛڬΉ͜ͱͰ଱ো֐ੑΛߴΊ͍ͯΔ •

    consumerଆͰretryՄೳ event receiver
  33. • Kafkaࣗମͷ଱ো֐ੑ͸ͱͯ΋ߴ͍ɻͦΕͰ΋ͳʹ͔͠ΒͷτϥϒϧͰো ֐͕ग़Δ͜ͱ΋͋Δ • ͦΕʹඋ͑ͯKafka team͸2ͭͷKafka ClusterΛ༻ҙͯ͘͠Ε͍ͯΔ • Primary Cluster͕downͨ͠ͱ͖͸Secondarily

    Clusterʹproduce͢Δ Α͏ʹ͍ͯ͠Δ • Primary/SecondarilyؒͰ࣮֬ͳॱ൪อূ͸Ͱ͖ͳ͘ͳΔ͕
 αʔϏεܧଓΛ༏ઌ KafkaͰো֐ʹͳͬͨΒ?
  34. None
  35. • ֤छeventΛॲཧ͢Δkafka consumer server • event receiver͕produceͨ͠event • ଞͷαʔϏε͕produceͨ͠event event

    processor Consume
  36. • ଞͷαʔϏε͕อ͍࣋ͯ͠Δσʔλ͕ߋ৽͞Εͨͱ͖ʹͦͷ৘ใΛड͚औΔ͜ͱ ͕Ͱ͖Δͱศར • ϦΞϧλΠϜʹ৘ใΛߋ৽Ͱ͖ͨΓCacheΛ࡟আͨ͠Γ͢Δ͜ͱ͕Ͱ͖Δ • Kafka͸consumer groupΛ෼͚Δͱಉ͡topicʹରͯ͠ෳ਺ͷconsumerΛ༻ҙ͢Δ ͜ͱ͕Ͱ͖Δ •

    ୯७ͳΩϡʔͱ͚ͯͩ͠Ͱ͸ͳͯ͘͜͏͍ͬͨ༻్ʹ΋Ԡ༻Ͱ͖Δ ϚΠΫϩαʔϏεؒͷ
 Data Hubͱͯ͠ͷKafka https://logmi.jp/tech/articles/320330
  37. • มߋ͕͋ͬͨͱ͖౳ʹ֤αʔϏεʹ௨஌ • ີ݁߹... • ར༻αʔϏε͕૿͑Δͱਏ͍ ੲ΍ͬͯͨବ໨ͳࣄྫ

  38. None
  39. • image, audio, video, file messageͷProxy server • సૹྔ͕େ͖͍ͨΊtomcatͩͱrequest threadΛ

    ઐ༗͕ͪ͠ɻWebFluxΛ࢖༻ content
  40. None
  41. • Botʹؔ͢Δmessage΍֤छoperationΛอଘ͢Δαʔόʔ • Userͷmessageͱҧͬͯอଘظݶ͕௕Ί • Ϗδωε༻్Ͱ͸อଘظݶ͕௕Ί͡Όͳ͍ͱݫ͍͠ • LINE messengerΞϓϦͱಉ༷ʹHBaseΛར༻ •

    ݱࡏ͸ࣾ಺ར༻ͷΈ • ͦͷ͏ͪ֎޲͚ʹ΋ެ։ͯ͠֎෦ͷ։ൃऀͷෛ୲ΛݮΒ͍ͨ͠ chatbox
  42. Tools

  43. • ࣾ಺Logging platform • ֤छlog dataΛKafkaʹૹΔͱKibana౳ͰݟΔ͜ͱ͕Ͱ͖Δ • Request log •

    application log (via logback appender) • ௐࠪͳͲͰ͘͢͝໾ཱ͍ͬͯΔ Datachain
  44. • logbackʹ͸ThreadLocalϕʔεͰ࣮૷͞ΕͨMDCͱ͍͏࢓૊Έ͕͋Δ • logʹ௥Ճ৘ใΛ෇༩Ͱ͖Δ • ྫ͑͹ServletFilterͰrequest৘ใΛMDCʹPut͓ͯ͘͠ͱศར MDC࢖͏ͱ͞Βʹศར

  45. WebFluxͰ΋MDC࢖͍͍ͨ • WebFluxͰ΋͜ͷςΫχοΫΛҾ͖ଓ͖࢖͍͍͕ͨ… • ୅ΘΓʹ࣍ͷ3ͭΛ࢖͏͜ͱͰ࣮ݱͰ͖Δ • WebFilter • Context •

    Hooks#onEachOperator • ※ spring-cloud-sleuth΋ಉ͡Α͏ͳ͜ͱΛ͍ͯ͠Δ
  46. • WebFilter Ͱ request৘ใΛContextʹ͍ΕΔ WebFluxͰ΋MDC࢖͍͍ͨ

  47. • ֤छonXXXXϝιουΛϥοϓ͢ΔΑ͏ͳCustom SubscriberΛ࣮૷͢Δ • Context͔ΒWebFilterͰొ࿥ͨ͠request৘ใΛऔಘ͢Δ • subscriber#onXXXΛݺͼग़͢લʹMDC#put͠ɺݺͼग़͠ޙʹMDC#remove͢Δ WebFluxͰ΋MDC࢖͍͍ͨ

  48. • ઌఔ࡞ͬͨCustom Subscriber͕࢖ΘΕΔΑ͏ʹHooksʹొ࿥͢Δ • Spring BootͳΒConfigurationʹॻ͘ WebFluxͰ΋MDC࢖͍͍ͨ

  49. • Spring Boot 2͔ΒMicrometer͕࢖༻͞Ε͍ͯΔ • ಛʹͳʹ΋ઃఆ͠ͳͯ͘΋ྑ͍ײ͡ʹMetricsΛͱͬͯ͘Ε͍ͯΔ • ඞཁʹԠͯࣗ͡෼Ͱ࣮૷͢Δ • Metricsͷͳ͍ίʔυ͸ςετ͕ͳ͍ίʔυҎ্ʹࠔΔͷͰ


    ͔ͬ͠ΓऔΖ͏ɻো֐࣌ʹޙͷࡇΓͱͳΔ Metrics Monitoring
  50. • LINEͰ͸MetricsΛPrometheusʹexportͯ͠GrafanaͰݟΔ͜ͱ͕ଟ͍ Metrics Monitoring

  51. ࠷ۙ͋ͬͨIssues

  52. • ͪΐ͏ͲҰࡢ೔ى͖ͨλΠϜϦʔͳIssue • ྫ͑͹Webflux ClientΛ༻͍ͨ͜Μͳcode͕͋Δͱ͠·͢ OOME Issue • ΋͠΋͜ͷAPI callͰtimeout͕େྔʹൃੜ͍ͯ͘͠ͱɺ


    ͳΜͱOOEMͰࢮʹ·͢
  53. • netty͔ΒReadTimeoutException͕ൃੜ͢Δͱreactor core಺Ͱ͜Ε͕࣮ߦ͞ΕΔ • Exceptions#propagate͸RuntimeExceptionͷαϒΫϥεͷ৔߹͸ͦͷ··ฦͯ͠ ͍ΔɻReadTimeoutException͸RuntimeExceptionͷαϒΫϥεͳͷͰ ReadTimeoutExceptionࣗ਎͕ฦΔ • this is

    ok, as re is alway a new non-singleton instanceͱॻ͍ͯ͋Δ͕… ReadTimeoutException OOME Issue ReadTimeoutException
  54. • ReadTimeoutException͸singletonͩͬͨ • ͦͷ݁Ռ, ReadTimeoutException͕อ͍࣋ͯ͠Δ suppressedExceptions(List<Throwable>)͕ͲΜͲΜͱංେԽͯ͠OOME͕ى͖Δ • heapdumpΛௐ΂ͨΒؾ͍ͮͨ • nettyʹGithub

    issue͋͛ͨΒͦͷ೔ͷ͏ͪʹPRग़ͯͨ
 https://github.com/netty/netty/pull/9152 • ͍ۙ͏ͪʹmaven repoʹ΋͕͋Γͦ͏ OOME Issue
  55. • LINEνϟοτͷ໰୊Ͱ͸ͳ͍͕Bot platform΍֤छࣾ಺αʔϏεͰ ى͖͍ͯͨ໰୊ • LINEνϟοτ͸lettuce(reactive api)࢖͍ͬͯΔ • https://github.com/xetorthio/jedis/issues/1945 •

    connection pool͔Βऔಘ͢Δͱ͖ʹhang͢Δ • Spring BootͷversionΛ͋͛Δͱ͖͸֤छϥΠϒϥϦͷversion΋͋ ͕ΔͷͰ͔ͬ͠Γͱ֬ೝ͠·͠ΐ͏ Jedis(2.9.1) Connection leaking
  56. • Reactorͷnon-blocking threadͰblock͢ΔΑ͏ͳcodeΛॻ͍ͯ͸NG • performance௿Լ͸౰વͱͯ͠ɺҙਤ͠ͳ͍deadlock΋ى͖·͢ • https://spring.io/blog/2019/03/28/reactor-debugging-experience#blockhound-a- new-kid-on-the-block • Reactor

    nettyͰ͸InetSocketAddressΛ࢖ͬͨblocking DNS resolverΛ࢖༻ ͍ͯ͠ΔͷͰɺnettyͷnon-blocking DNS resolverΛ࢖༻͢ΔΑ͏ʹมߋ͠ ·͠ΐ͏ • (·ͩ Unstable ApiͰ͕͢) • ΋͠΋DNS resolveʹ͕͔͔࣌ؒΔͱdeadlock͠·͢ • brennentsmith/slodns౳Λ࢖ͬͯҙਤతʹdelayͤͯ͞ΈΔͱ
 ࠶ݱ͠·͢ • Reactor nettyଆͰɺdefaultΛnettyͷnon-blocking DNS resolverʹ͠Α͏͔ ͱ͍͏issue͸͋Δ • https://github.com/reactor/reactor-netty/issues/569 Blocking DNS resolver
  57. THANK YOU

  58. https://linecorp.com/ja/career/position/1643