LINEのBot Platformの裏側 / Behind the LINE Bot Platform

LINEのBot Platformの裏側 / Behind the LINE Bot Platform

JJUG CCC 2019 Springでの登壇資料です

53850955f15249a1a9dc49df6113e400?s=128

LINE Developers

May 18, 2019
Tweet

Transcript

  1. 2.

    @be_hase1014 https://github.com/be-hase About me • ௕୩෦ ྑี • LINEגࣜձࣾ Account

    Platform։ൃࣨ AP4νʔϜ Ϛωʔδϟʔ • 2013೥ ೖࣾ • Server-side Engineer • LINEνϟοτ (for LINEެࣜΞΧ΢ϯτ) • Before: • LINE Login, LINE ΧελϚʔίωΫτ, LINE Notify, 
 LINE Profile+, LINE GAME Platform, …etc
  2. 8.

    • ܕ෇͖, ଎͍ • ๛෋ͳϥΠϒϥϦ • ձࣾશମతʹੲ͔Β࢖͍ͬͯΔͷͰࢿ࢈͕ଟ͍ • ੈքతʹJavaΤϯδχΞ͕ଟ͍ •

    ։ൃڌ఺͸೔ຊҎ֎ʹ΋୆࿷/λΠ/ؖࠃ/தࠃ/ϕτφϜ౳ʹ͋Δ • ڌ఺Λ·͍ͨͩϓϩμΫτͷҠ؅΋͋Δ • ͱ͸͍͑ඞͣ͠΋JavaΛར༻͠ͳ͍ͱ͍͚ͳ͍ͱ͍͏Θ͚Ͱ͸ͳ͍ Why Java?
  3. 11.
  4. 13.

    • ϝοηʔδ഑৴ • ࣗಈԠ౴/ΩʔϫʔυԠ౴ • Ϧονϝχϡʔ • λΠϜϥΠϯ౤ߘ • γϣοϓΧʔυ

    • Ϋʔϙϯ • ֤छΞφϦςΟΫεͷఏڙ • ܾࡁ • …etc LINE Official Account Manager (CMS)
  5. 18.

    CMS server Dev Team Chat server Bot backend server CMS

    & Core web-front Chat web-front Native App • ։ൃڌ఺͸೔ຊɺؖࠃɺϕτφϜ • Native App͸΄΅WebViewͱ࣮ͯ͠૷͍ͯ͠ΔͷͰfrontͷਓ਺͕ଟΊ • ଞʹ΋֤छઐ໳νʔϜͱ • ITSC (Private cloud, LB, …), DBA (MySQL, HBase, Redis), 
 IMF (Kafka), Datalabs, QA, …etc 6ਓ 6ਓ 8ਓ 17ਓ 4ਓ 5ਓ
  6. 21.

    api streaming api content • Spring Boot 2 • Spring

    WebFlux • Redis Cluster • Redis PubSub • Spring Boot 2 • Spring MVC • MySQL • Redis Cluster LINEνϟοτͷcomponent event receiver event processor chatbox • Spring Boot 1 • Armeria(thrift) • HBase • Spring Boot 2 • Spring WebFlux • Spring Boot 2 • Spring WebFlux • Kafka • Spring Boot 2 • Kafka • MySQL • Redis Cluster • Redis PubSub
  7. 22.
  8. 23.
  9. 24.

    • ͘͝ҰൠతͳJSON API server • Φʔφʔͷૢ࡞ʹΑΔ֤छϏδωεϩδοΫ • ϝοηʔδૹ৴ • աڈϝοηʔδऔಘ

    • εςʔλε/λάͳͲͷσʔλอଘ • MySQL΁Blocking IOΛ͍ͯ͠ΔͷͰ
 ੲͳ͕ΒͷSpring MVC(tomcat) api
  10. 25.
  11. 26.

    • ChatΞϓϦέʔγϣϯʹ͸realtimeੑ͕ඞཁ • Server-Sent events(SSE) by Spring WebFlux • WebSocketΛ࠾༻͢Δ͔೰Μ͕ͩɺ૒ํ޲௨

    ৴͸ඞཁͳ͍ͷͰSSEΛ࠾༻ • LINE user͔ΒͷϝοηʔδΛrealtimeʹ൓ө • ଞͷΦʔφʔʹΑΔ֤छૢ࡞΋ streaming api Send messages Send message
  12. 27.

    • WebFlux • Spring 5͔Βొ৔ͨ͠Non Blocking WebϑϨʔϜϫʔΫ • SSEͷΑ͏ͳlong connectionΛ࢖༻͢ΔΞϓϦέʔγϣϯ,

    Gateway server ౳ͱ͸ಛʹ૬ੑ͕͍͍ • Request threadΛઐ༗͠ͳ͍ • ैདྷͷServlet APIͰ͸request threadͷ਺͔͠εέʔϧ͠ͳ͍ SSEͱSpring WebFlux
  13. 29.

    • ωοτϫʔΫ઀ଓ͕੾Εͯ࠶઀ଓͨ͠ͱ͖ʹɺ
 ͦͷؒͷeventΛड৴Ͱ͖ΔΑ͏ʹ͓ͯ͘͠ඞཁ ͕͋Δ • SSEʹ͸Last-Event-ID headerͱ͍͏
 ࢓૊Έ͕͋Γɺ࠶઀ଓ͢Δͱ͖ʹ࠷ޙʹड৴͠ ͨevent IDΛrequest

    headerʹؚΊͯ͘ΕΔ • RedisʹPub͢Δͱ͖ʹಉ࣌ʹͦͷdataΛListͱ ͯ͠΋อ͓࣋ͯ͘͠ • ࠶઀ଓͷࡍʹ͸ͦͷList͔ΒऔಘͰ͖ΔΑ͏ʹ • Redis 5͔Β௥Ճ͞ΕͨRedis StreamsΛ࢖͏ͱ ָͰ͖ͦ͏ɻ
 ࠓޙࢼ͍ͨ͠ (ݱࡏ͸Redis 4࢖͍ͬͯΔ) SSEͷ࠶઀ଓ Send messages Send message
  14. 30.

    • Redis 3͔Βొ৔ͨ͠Redis Clusterʹ΋ରԠ͍ͯ͠Δ͕ҎԼͷΑ͏ͳ໰୊΋͋Δ Redis PubSubͷ஫ҙ఺ In a Redis Cluster

    clients can subscribe to every node, and can also publish to every other node. The cluster will make sure that published messages are forwarded as needed. The current implementation will simply broadcast each published message to all other nodes, but at some point this will be optimized either https://redis.io/topics/cluster-spec#publishsubscribe • εέʔϧ͕ඞཁʹͳͬͨΒɺผͷclusterΛ༻ҙͯ͠ৼΓ෼͚ΔΑ͏ʹͨ͠΄ ͏͕͍͍
  15. 31.
  16. 33.

    • Kafkaࣗମͷ଱ো֐ੑ͸ͱͯ΋ߴ͍ɻͦΕͰ΋ͳʹ͔͠ΒͷτϥϒϧͰো ֐͕ग़Δ͜ͱ΋͋Δ • ͦΕʹඋ͑ͯKafka team͸2ͭͷKafka ClusterΛ༻ҙͯ͘͠Ε͍ͯΔ • Primary Cluster͕downͨ͠ͱ͖͸Secondarily

    Clusterʹproduce͢Δ Α͏ʹ͍ͯ͠Δ • Primary/SecondarilyؒͰ࣮֬ͳॱ൪อূ͸Ͱ͖ͳ͘ͳΔ͕
 αʔϏεܧଓΛ༏ઌ KafkaͰো֐ʹͳͬͨΒ?
  17. 34.
  18. 38.
  19. 40.
  20. 42.
  21. 43.

    • ࣾ಺Logging platform • ֤छlog dataΛKafkaʹૹΔͱKibana౳ͰݟΔ͜ͱ͕Ͱ͖Δ • Request log •

    application log (via logback appender) • ௐࠪͳͲͰ͘͢͝໾ཱ͍ͬͯΔ Datachain
  22. 55.

    • LINEνϟοτͷ໰୊Ͱ͸ͳ͍͕Bot platform΍֤छࣾ಺αʔϏεͰ ى͖͍ͯͨ໰୊ • LINEνϟοτ͸lettuce(reactive api)࢖͍ͬͯΔ • https://github.com/xetorthio/jedis/issues/1945 •

    connection pool͔Βऔಘ͢Δͱ͖ʹhang͢Δ • Spring BootͷversionΛ͋͛Δͱ͖͸֤छϥΠϒϥϦͷversion΋͋ ͕ΔͷͰ͔ͬ͠Γͱ֬ೝ͠·͠ΐ͏ Jedis(2.9.1) Connection leaking
  23. 56.

    • Reactorͷnon-blocking threadͰblock͢ΔΑ͏ͳcodeΛॻ͍ͯ͸NG • performance௿Լ͸౰વͱͯ͠ɺҙਤ͠ͳ͍deadlock΋ى͖·͢ • https://spring.io/blog/2019/03/28/reactor-debugging-experience#blockhound-a- new-kid-on-the-block • Reactor

    nettyͰ͸InetSocketAddressΛ࢖ͬͨblocking DNS resolverΛ࢖༻ ͍ͯ͠ΔͷͰɺnettyͷnon-blocking DNS resolverΛ࢖༻͢ΔΑ͏ʹมߋ͠ ·͠ΐ͏ • (·ͩ Unstable ApiͰ͕͢) • ΋͠΋DNS resolveʹ͕͔͔࣌ؒΔͱdeadlock͠·͢ • brennentsmith/slodns౳Λ࢖ͬͯҙਤతʹdelayͤͯ͞ΈΔͱ
 ࠶ݱ͠·͢ • Reactor nettyଆͰɺdefaultΛnettyͷnon-blocking DNS resolverʹ͠Α͏͔ ͱ͍͏issue͸͋Δ • https://github.com/reactor/reactor-netty/issues/569 Blocking DNS resolver
  24. 57.