LINEのBot Platformの裏側 / Behind the LINE Bot Platform

by LINE Developers

Slide 1

Slide 1 text

LINEͷBot Platformͷཪଆͷ࿩ Ryosuke Hasebe, LINE Corporation JJUG CCC Spring 2019

Slide 2

Slide 2 text

@be_hase1014 https://github.com/be-hase About me ● ௕୩෦ ྑี ● LINEגࣜձࣾ Account Platform։ൃࣨ AP4νʔϜ Ϛωʔδϟʔ ● 2013೥ ೖࣾ ● Server-side Engineer ● LINEνϟοτ (for LINEެࣜΞΧ΢ϯτ) ● Before: ● LINE Login, LINE ΧελϚʔίωΫτ, LINE Notify,   LINE Profile+, LINE GAME Platform, …etc

Slide 3

Slide 3 text

• LINEͱJava • LINE Bot Platformʹ͍ͭͯ • LINEνϟοτ(for LINEެࣜΞΧ΢ϯτ) ͷ։ൃࣄྫΛ۷ΓԼ͛ͯ঺հ Agenda

Slide 4

Slide 4 text

LINEͱJava

Slide 5

Slide 5 text

ݴޠ͝ͱͷrepo਺ (ࣾ಺GHEௐ΂)

Slide 6

Slide 6 text

Bot platform component overview

Slide 7

Slide 7 text

Bot platform component overview Java ͨ͘͞Μʂ

Slide 8

Slide 8 text

● ܕ෇͖, ଎͍ ● ๛෋ͳϥΠϒϥϦ ● ձࣾશମతʹੲ͔Β࢖͍ͬͯΔͷͰࢿ࢈͕ଟ͍ ● ੈքతʹJavaΤϯδχΞ͕ଟ͍ ● ։ൃڌ఺͸೔ຊҎ֎ʹ΋୆࿷/λΠ/ؖࠃ/தࠃ/ϕτφϜ౳ʹ͋Δ ● ڌ఺Λ·͍ͨͩϓϩμΫτͷҠ؅΋͋Δ ● ͱ͸͍͑ඞͣ͠΋JavaΛར༻͠ͳ͍ͱ͍͚ͳ͍ͱ͍͏Θ͚Ͱ͸ͳ͍ Why Java?

Slide 9

Slide 9 text

LINE Bot platformʹ͍ͭͯ

Slide 10

Slide 10 text

What’s LINE Bot Platform? ● LINEެࣜΞΧ΢ϯτ(๏ਓ޲͚ΞΧ΢ϯταʔϏε)ʹ͓͍ͯBotΛ ϕʔεͱͨ͠ϏδωεɾιϦϡʔγϣϯΛఏڙ͢ΔͨΊͷPlatform ͷ͜ͱ ● ϝοηʔδૹ৴/ड৴ ● λΠϜϥΠϯ౤ߘ ● … etc (ޙड़) اۀ΍ݸਓ Ϣʔβʔ

Slide 11

Slide 11 text

What’s its VALUE? ѹ౗తͳϦʔνྗ ʂʂ ສਓ 8000 Percent 86 ೔ຊͷਓޱͷ 60%Ҏ্ ຖ೔ར༻͍ͯ͠Δ ೔ຊࠃ಺ͷϢʔβʔ

Slide 12

Slide 12 text

ఏڙ͍ͯ͠Δػೳ

Slide 13

Slide 13 text

● ϝοηʔδ഑৴ ● ࣗಈԠ౴/ΩʔϫʔυԠ౴ ● Ϧονϝχϡʔ ● λΠϜϥΠϯ౤ߘ ● γϣοϓΧʔυ ● Ϋʔϙϯ ● ֤छΞφϦςΟΫεͷఏڙ ● ܾࡁ ● …etc LINE Official Account Manager (CMS)

Slide 14

Slide 14 text

● ΞΧ΢ϯτ؅ཧऀ༻ͷνϟοτπʔϧ ● εςʔλεŋλάΛ༻͍ͨސ٬؅ཧ ● ޙ΄Ͳ։ൃࣄྫΛ঺հ͠·͢ LINEνϟοτ

Slide 15

Slide 15 text

Messaging API ● ςοΫاۀ / ։ൃऀ޲͚ͷAPI܈ɻChatbotΛࣗ༝ʹ࡞੒/࣮૷Մೳ ● ϝοηʔδͷड৴͸Webhook ● ࣮͸ࣾ಺ͷ֤छαʔϏε΋࢖༻͍ͯ͠·͢ send messages Receive messages via webhook

Slide 16

Slide 16 text

● LINE Beacon ● ༑ͩͪΦʔσΟΤϯε഑৴ ● LINE LIVE࿈ܞ ● …etc Others

Slide 17

Slide 17 text

ϝοηʔδ഑ૹ਺  (݄ؒ) 800ԯ ΞΧ΢ϯτ਺ 900ສ+ LINE Bot Platformͷن໛ײ Messaging API daily requests 2.2ԯ+ αʔόʔ਺ (app serverͷΈ) 500+

Slide 18

Slide 18 text

CMS server Dev Team Chat server Bot backend server CMS & Core web-front Chat web-front Native App ● ։ൃڌ఺͸೔ຊɺؖࠃɺϕτφϜ ● Native App͸΄΅WebViewͱ࣮ͯ͠૷͍ͯ͠ΔͷͰfrontͷਓ਺͕ଟΊ ● ଞʹ΋֤छઐ໳νʔϜͱ ● ITSC (Private cloud, LB, …), DBA (MySQL, HBase, Redis),   IMF (Kafka), Datalabs, QA, …etc 6ਓ 6ਓ 8ਓ 17ਓ 4ਓ 5ਓ

Slide 19

Slide 19 text

LINEνϟοτͷ։ൃࣄྫ

Slide 20

Slide 20 text

(͓͞Β͍) LINEνϟοτ

Slide 21

Slide 21 text

api streaming api content • Spring Boot 2 • Spring WebFlux • Redis Cluster • Redis PubSub • Spring Boot 2 • Spring MVC • MySQL • Redis Cluster LINEνϟοτͷcomponent event receiver event processor chatbox • Spring Boot 1 • Armeria(thrift) • HBase • Spring Boot 2 • Spring WebFlux • Spring Boot 2 • Spring WebFlux • Kafka • Spring Boot 2 • Kafka • MySQL • Redis Cluster • Redis PubSub

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

● ͘͝ҰൠతͳJSON API server ● Φʔφʔͷૢ࡞ʹΑΔ֤छϏδωεϩδοΫ ● ϝοηʔδૹ৴ ● աڈϝοηʔδऔಘ ● εςʔλε/λάͳͲͷσʔλอଘ ● MySQL΁Blocking IOΛ͍ͯ͠ΔͷͰ  ੲͳ͕ΒͷSpring MVC(tomcat) api

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

● ChatΞϓϦέʔγϣϯʹ͸realtimeੑ͕ඞཁ ● Server-Sent events(SSE) by Spring WebFlux ● WebSocketΛ࠾༻͢Δ͔೰Μ͕ͩɺ૒ํ޲௨ ৴͸ඞཁͳ͍ͷͰSSEΛ࠾༻ ● LINE user͔ΒͷϝοηʔδΛrealtimeʹ൓ө ● ଞͷΦʔφʔʹΑΔ֤छૢ࡞΋ streaming api Send messages Send message

Slide 27

Slide 27 text

● WebFlux ● Spring 5͔Βొ৔ͨ͠Non Blocking WebϑϨʔϜϫʔΫ ● SSEͷΑ͏ͳlong connectionΛ࢖༻͢ΔΞϓϦέʔγϣϯ, Gateway server ౳ͱ͸ಛʹ૬ੑ͕͍͍ ● Request threadΛઐ༗͠ͳ͍ ● ैདྷͷServlet APIͰ͸request threadͷ਺͔͠εέʔϧ͠ͳ͍ SSEͱSpring WebFlux

Slide 28

Slide 28 text

Example code ௕࣌ؒ, eventΛૹΒͳ͍ͱ઀ଓ͕ ੾ΒΕΔͷͰpingૹ৴༻ͷFluxΛ ࡞੒ͯ͠merge͢Δ

Slide 29

Slide 29 text

● ωοτϫʔΫ઀ଓ͕੾Εͯ࠶઀ଓͨ͠ͱ͖ʹɺ  ͦͷؒͷeventΛड৴Ͱ͖ΔΑ͏ʹ͓ͯ͘͠ඞཁ ͕͋Δ ● SSEʹ͸Last-Event-ID headerͱ͍͏  ࢓૊Έ͕͋Γɺ࠶઀ଓ͢Δͱ͖ʹ࠷ޙʹड৴͠ ͨevent IDΛrequest headerʹؚΊͯ͘ΕΔ ● RedisʹPub͢Δͱ͖ʹಉ࣌ʹͦͷdataΛListͱ ͯ͠΋อ͓࣋ͯ͘͠ ● ࠶઀ଓͷࡍʹ͸ͦͷList͔ΒऔಘͰ͖ΔΑ͏ʹ ● Redis 5͔Β௥Ճ͞ΕͨRedis StreamsΛ࢖͏ͱ ָͰ͖ͦ͏ɻ  ࠓޙࢼ͍ͨ͠ (ݱࡏ͸Redis 4࢖͍ͬͯΔ) SSEͷ࠶઀ଓ Send messages Send message

Slide 30

Slide 30 text

● Redis 3͔Βొ৔ͨ͠Redis Clusterʹ΋ରԠ͍ͯ͠Δ͕ҎԼͷΑ͏ͳ໰୊΋͋Δ Redis PubSubͷ஫ҙ఺ In a Redis Cluster clients can subscribe to every node, and can also publish to every other node. The cluster will make sure that published messages are forwarded as needed. The current implementation will simply broadcast each published message to all other nodes, but at some point this will be optimized either https://redis.io/topics/cluster-spec#publishsubscribe ● εέʔϧ͕ඞཁʹͳͬͨΒɺผͷclusterΛ༻ҙͯ͠ৼΓ෼͚ΔΑ͏ʹͨ͠΄ ͏͕͍͍

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

● LINE User͕Botʹରͯ͠ϝοηʔδΛૹ৴ ͢Δ౳ͷactionΛ͢ΔͱɺWebhook͕ಧ͘ ● WebhookΛड͚औͬͨΒଈ࠲ʹKafkaʹ produce ● KafkaΛڬΉ͜ͱͰ଱ো֐ੑΛߴΊ͍ͯΔ ● consumerଆͰretryՄೳ event receiver

Slide 33

Slide 33 text

● Kafkaࣗମͷ଱ো֐ੑ͸ͱͯ΋ߴ͍ɻͦΕͰ΋ͳʹ͔͠ΒͷτϥϒϧͰো ֐͕ग़Δ͜ͱ΋͋Δ ● ͦΕʹඋ͑ͯKafka team͸2ͭͷKafka ClusterΛ༻ҙͯ͘͠Ε͍ͯΔ ● Primary Cluster͕downͨ͠ͱ͖͸Secondarily Clusterʹproduce͢Δ Α͏ʹ͍ͯ͠Δ ● Primary/SecondarilyؒͰ࣮֬ͳॱ൪อূ͸Ͱ͖ͳ͘ͳΔ͕  αʔϏεܧଓΛ༏ઌ KafkaͰো֐ʹͳͬͨΒ?

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

● ֤छeventΛॲཧ͢Δkafka consumer server ● event receiver͕produceͨ͠event ● ଞͷαʔϏε͕produceͨ͠event event processor Consume

Slide 36

Slide 36 text

● ଞͷαʔϏε͕อ͍࣋ͯ͠Δσʔλ͕ߋ৽͞Εͨͱ͖ʹͦͷ৘ใΛड͚औΔ͜ͱ ͕Ͱ͖Δͱศར ● ϦΞϧλΠϜʹ৘ใΛߋ৽Ͱ͖ͨΓCacheΛ࡟আͨ͠Γ͢Δ͜ͱ͕Ͱ͖Δ ● Kafka͸consumer groupΛ෼͚Δͱಉ͡topicʹରͯ͠ෳ਺ͷconsumerΛ༻ҙ͢Δ ͜ͱ͕Ͱ͖Δ ● ୯७ͳΩϡʔͱ͚ͯͩ͠Ͱ͸ͳͯ͘͜͏͍ͬͨ༻్ʹ΋Ԡ༻Ͱ͖Δ ϚΠΫϩαʔϏεؒͷ  Data Hubͱͯ͠ͷKafka https://logmi.jp/tech/articles/320330

Slide 37

Slide 37 text

● มߋ͕͋ͬͨͱ͖౳ʹ֤αʔϏεʹ௨஌ ● ີ݁߹... ● ར༻αʔϏε͕૿͑Δͱਏ͍ ੲ΍ͬͯͨବ໨ͳࣄྫ

Slide 38

Slide 38 text

No content

Slide 39

Slide 39 text

● image, audio, video, file messageͷProxy server ● సૹྔ͕େ͖͍ͨΊtomcatͩͱrequest threadΛ ઐ༗͕ͪ͠ɻWebFluxΛ࢖༻ content

Slide 40

Slide 40 text

No content

Slide 41

Slide 41 text

● Botʹؔ͢Δmessage΍֤छoperationΛอଘ͢Δαʔόʔ ● Userͷmessageͱҧͬͯอଘظݶ͕௕Ί ● Ϗδωε༻్Ͱ͸อଘظݶ͕௕Ί͡Όͳ͍ͱݫ͍͠ ● LINE messengerΞϓϦͱಉ༷ʹHBaseΛར༻ ● ݱࡏ͸ࣾ಺ར༻ͷΈ ● ͦͷ͏ͪ֎޲͚ʹ΋ެ։ͯ͠֎෦ͷ։ൃऀͷෛ୲ΛݮΒ͍ͨ͠ chatbox

Slide 42

Slide 42 text

Tools

Slide 43

Slide 43 text

● ࣾ಺Logging platform ● ֤छlog dataΛKafkaʹૹΔͱKibana౳ͰݟΔ͜ͱ͕Ͱ͖Δ ● Request log ● application log (via logback appender) ● ௐࠪͳͲͰ͘͢͝໾ཱ͍ͬͯΔ Datachain

Slide 44

Slide 44 text

● logbackʹ͸ThreadLocalϕʔεͰ࣮૷͞ΕͨMDCͱ͍͏࢓૊Έ͕͋Δ ● logʹ௥Ճ৘ใΛ෇༩Ͱ͖Δ ● ྫ͑͹ServletFilterͰrequest৘ใΛMDCʹPut͓ͯ͘͠ͱศར MDC࢖͏ͱ͞Βʹศར

Slide 45

Slide 45 text

WebFluxͰ΋MDC࢖͍͍ͨ ● WebFluxͰ΋͜ͷςΫχοΫΛҾ͖ଓ͖࢖͍͍͕ͨ… ● ୅ΘΓʹ࣍ͷ3ͭΛ࢖͏͜ͱͰ࣮ݱͰ͖Δ ● WebFilter ● Context ● Hooks#onEachOperator ● ※ spring-cloud-sleuth΋ಉ͡Α͏ͳ͜ͱΛ͍ͯ͠Δ

Slide 46

Slide 46 text

● WebFilter Ͱ request৘ใΛContextʹ͍ΕΔ WebFluxͰ΋MDC࢖͍͍ͨ

Slide 47

Slide 47 text

● ֤छonXXXXϝιουΛϥοϓ͢ΔΑ͏ͳCustom SubscriberΛ࣮૷͢Δ ● Context͔ΒWebFilterͰొ࿥ͨ͠request৘ใΛऔಘ͢Δ ● subscriber#onXXXΛݺͼग़͢લʹMDC#put͠ɺݺͼग़͠ޙʹMDC#remove͢Δ WebFluxͰ΋MDC࢖͍͍ͨ

Slide 48

Slide 48 text

● ઌఔ࡞ͬͨCustom Subscriber͕࢖ΘΕΔΑ͏ʹHooksʹొ࿥͢Δ ● Spring BootͳΒConfigurationʹॻ͘ WebFluxͰ΋MDC࢖͍͍ͨ

Slide 49

Slide 49 text

● Spring Boot 2͔ΒMicrometer͕࢖༻͞Ε͍ͯΔ ● ಛʹͳʹ΋ઃఆ͠ͳͯ͘΋ྑ͍ײ͡ʹMetricsΛͱͬͯ͘Ε͍ͯΔ ● ඞཁʹԠͯࣗ͡෼Ͱ࣮૷͢Δ ● Metricsͷͳ͍ίʔυ͸ςετ͕ͳ͍ίʔυҎ্ʹࠔΔͷͰ  ͔ͬ͠ΓऔΖ͏ɻো֐࣌ʹޙͷࡇΓͱͳΔ Metrics Monitoring

Slide 50

Slide 50 text

● LINEͰ͸MetricsΛPrometheusʹexportͯ͠GrafanaͰݟΔ͜ͱ͕ଟ͍ Metrics Monitoring

Slide 51

Slide 51 text

࠷ۙ͋ͬͨIssues

Slide 52

Slide 52 text

● ͪΐ͏ͲҰࡢ೔ى͖ͨλΠϜϦʔͳIssue ● ྫ͑͹Webflux ClientΛ༻͍ͨ͜Μͳcode͕͋Δͱ͠·͢ OOME Issue ● ΋͠΋͜ͷAPI callͰtimeout͕େྔʹൃੜ͍ͯ͘͠ͱɺ  ͳΜͱOOEMͰࢮʹ·͢

Slide 53

Slide 53 text

● netty͔ΒReadTimeoutException͕ൃੜ͢Δͱreactor core಺Ͱ͜Ε͕࣮ߦ͞ΕΔ ● Exceptions#propagate͸RuntimeExceptionͷαϒΫϥεͷ৔߹͸ͦͷ··ฦͯ͠ ͍ΔɻReadTimeoutException͸RuntimeExceptionͷαϒΫϥεͳͷͰ ReadTimeoutExceptionࣗ਎͕ฦΔ ● this is ok, as re is alway a new non-singleton instanceͱॻ͍ͯ͋Δ͕… ReadTimeoutException OOME Issue ReadTimeoutException

Slide 54

Slide 54 text

● ReadTimeoutException͸singletonͩͬͨ ● ͦͷ݁Ռ, ReadTimeoutException͕อ͍࣋ͯ͠Δ suppressedExceptions(List)͕ͲΜͲΜͱංେԽͯ͠OOME͕ى͖Δ ● heapdumpΛௐ΂ͨΒؾ͍ͮͨ ● nettyʹGithub issue͋͛ͨΒͦͷ೔ͷ͏ͪʹPRग़ͯͨ  https://github.com/netty/netty/pull/9152 ● ͍ۙ͏ͪʹmaven repoʹ΋͕͋Γͦ͏ OOME Issue

Slide 55

Slide 55 text

● LINEνϟοτͷ໰୊Ͱ͸ͳ͍͕Bot platform΍֤छࣾ಺αʔϏεͰ ى͖͍ͯͨ໰୊ ● LINEνϟοτ͸lettuce(reactive api)࢖͍ͬͯΔ ● https://github.com/xetorthio/jedis/issues/1945 ● connection pool͔Βऔಘ͢Δͱ͖ʹhang͢Δ ● Spring BootͷversionΛ͋͛Δͱ͖͸֤छϥΠϒϥϦͷversion΋͋ ͕ΔͷͰ͔ͬ͠Γͱ֬ೝ͠·͠ΐ͏ Jedis(2.9.1) Connection leaking

Slide 56

Slide 56 text

● Reactorͷnon-blocking threadͰblock͢ΔΑ͏ͳcodeΛॻ͍ͯ͸NG ● performance௿Լ͸౰વͱͯ͠ɺҙਤ͠ͳ͍deadlock΋ى͖·͢ ● https://spring.io/blog/2019/03/28/reactor-debugging-experience#blockhound-a- new-kid-on-the-block ● Reactor nettyͰ͸InetSocketAddressΛ࢖ͬͨblocking DNS resolverΛ࢖༻ ͍ͯ͠ΔͷͰɺnettyͷnon-blocking DNS resolverΛ࢖༻͢ΔΑ͏ʹมߋ͠ ·͠ΐ͏ ● (·ͩ Unstable ApiͰ͕͢) ● ΋͠΋DNS resolveʹ͕͔͔࣌ؒΔͱdeadlock͠·͢ ● brennentsmith/slodns౳Λ࢖ͬͯҙਤతʹdelayͤͯ͞ΈΔͱ  ࠶ݱ͠·͢ ● Reactor nettyଆͰɺdefaultΛnettyͷnon-blocking DNS resolverʹ͠Α͏͔ ͱ͍͏issue͸͋Δ ● https://github.com/reactor/reactor-netty/issues/569 Blocking DNS resolver