LINEのBot Platformの裏側 / Behind the LINE Bot Platform

JJUG CCC 2019 Springでの登壇資料です

Transcript

1. LINEͷBot Platformͷཪଆͷ࿩ Ryosuke Hasebe, LINE Corporation JJUG CCC Spring 2019
2. @be_hase1014 https://github.com/be-hase About me • ௕୩෦ ྑี • LINEגࣜձࣾ Account Platform։ൃࣨ AP4νʔϜ Ϛωʔδϟʔ • 2013೥ ೖࣾ • Server-side Engineer • LINEνϟοτ (for LINEެࣜΞΧ΢ϯτ) • Before: • LINE Login, LINE ΧελϚʔίωΫτ, LINE Notify, 
 LINE Profile+, LINE GAME Platform, …etc
3. • LINEͱJava • LINE Bot Platformʹ͍ͭͯ • LINEνϟοτ(for LINEެࣜΞΧ΢ϯτ) ͷ։ൃࣄྫΛ۷ΓԼ͛ͯ঺հ Agenda
4. LINEͱJava
5. ݴޠ͝ͱͷrepo਺ (ࣾ಺GHEௐ΂)
6. Bot platform component overview
7. Bot platform component overview Java ͨ͘͞Μʂ
8. • ܕ෇͖, ଎͍ • ๛෋ͳϥΠϒϥϦ • ձࣾશମతʹੲ͔Β࢖͍ͬͯΔͷͰࢿ࢈͕ଟ͍ • ੈքతʹJavaΤϯδχΞ͕ଟ͍ • ։ൃڌ఺͸೔ຊҎ֎ʹ΋୆࿷/λΠ/ؖࠃ/தࠃ/ϕτφϜ౳ʹ͋Δ • ڌ఺Λ·͍ͨͩϓϩμΫτͷҠ؅΋͋Δ • ͱ͸͍͑ඞͣ͠΋JavaΛར༻͠ͳ͍ͱ͍͚ͳ͍ͱ͍͏Θ͚Ͱ͸ͳ͍ Why Java?
9. LINE Bot platformʹ͍ͭͯ
10. What’s LINE Bot Platform? • LINEެࣜΞΧ΢ϯτ(๏ਓ޲͚ΞΧ΢ϯταʔϏε)ʹ͓͍ͯBotΛ ϕʔεͱͨ͠ϏδωεɾιϦϡʔγϣϯΛఏڙ͢ΔͨΊͷPlatform ͷ͜ͱ • ϝοηʔδૹ৴/ड৴ • λΠϜϥΠϯ౤ߘ • … etc (ޙड़) اۀ΍ݸਓ Ϣʔβʔ
11. What’s its VALUE? ѹ౗తͳϦʔνྗ ʂʂ ສਓ 8000 Percent 86 ೔ຊͷਓޱͷ 60%Ҏ্ ຖ೔ར༻͍ͯ͠Δ ೔ຊࠃ಺ͷϢʔβʔ
12. ఏڙ͍ͯ͠Δػೳ
13. • ϝοηʔδ഑৴ • ࣗಈԠ౴/ΩʔϫʔυԠ౴ • Ϧονϝχϡʔ • λΠϜϥΠϯ౤ߘ • γϣοϓΧʔυ • Ϋʔϙϯ • ֤छΞφϦςΟΫεͷఏڙ • ܾࡁ • …etc LINE Official Account Manager (CMS)
14. • ΞΧ΢ϯτ؅ཧऀ༻ͷνϟοτπʔϧ • εςʔλεŋλάΛ༻͍ͨސ٬؅ཧ • ޙ΄Ͳ։ൃࣄྫΛ঺հ͠·͢ LINEνϟοτ
15. Messaging API • ςοΫاۀ / ։ൃऀ޲͚ͷAPI܈ɻChatbotΛࣗ༝ʹ࡞੒/࣮૷Մೳ • ϝοηʔδͷड৴͸Webhook • ࣮͸ࣾ಺ͷ֤छαʔϏε΋࢖༻͍ͯ͠·͢ send messages Receive messages via webhook
16. • LINE Beacon • ༑ͩͪΦʔσΟΤϯε഑৴ • LINE LIVE࿈ܞ • …etc Others
17. ϝοηʔδ഑ૹ਺
 (݄ؒ) 800ԯ ΞΧ΢ϯτ਺ 900ສ+ LINE Bot Platformͷن໛ײ Messaging API daily requests 2.2ԯ+ αʔόʔ਺ (app serverͷΈ) 500+
18. CMS server Dev Team Chat server Bot backend server CMS & Core web-front Chat web-front Native App • ։ൃڌ఺͸೔ຊɺؖࠃɺϕτφϜ • Native App͸΄΅WebViewͱ࣮ͯ͠૷͍ͯ͠ΔͷͰfrontͷਓ਺͕ଟΊ • ଞʹ΋֤छઐ໳νʔϜͱ • ITSC (Private cloud, LB, …), DBA (MySQL, HBase, Redis), 
 IMF (Kafka), Datalabs, QA, …etc 6ਓ 6ਓ 8ਓ 17ਓ 4ਓ 5ਓ
19. LINEνϟοτͷ։ൃࣄྫ
20. (͓͞Β͍) LINEνϟοτ
21. api streaming api content • Spring Boot 2 • Spring WebFlux • Redis Cluster • Redis PubSub • Spring Boot 2 • Spring MVC • MySQL • Redis Cluster LINEνϟοτͷcomponent event receiver event processor chatbox • Spring Boot 1 • Armeria(thrift) • HBase • Spring Boot 2 • Spring WebFlux • Spring Boot 2 • Spring WebFlux • Kafka • Spring Boot 2 • Kafka • MySQL • Redis Cluster • Redis PubSub
24. • ͘͝ҰൠతͳJSON API server • Φʔφʔͷૢ࡞ʹΑΔ֤छϏδωεϩδοΫ • ϝοηʔδૹ৴ • աڈϝοηʔδऔಘ • εςʔλε/λάͳͲͷσʔλอଘ • MySQL΁Blocking IOΛ͍ͯ͠ΔͷͰ
 ੲͳ͕ΒͷSpring MVC(tomcat) api
26. • ChatΞϓϦέʔγϣϯʹ͸realtimeੑ͕ඞཁ • Server-Sent events(SSE) by Spring WebFlux • WebSocketΛ࠾༻͢Δ͔೰Μ͕ͩɺ૒ํ޲௨ ৴͸ඞཁͳ͍ͷͰSSEΛ࠾༻ • LINE user͔ΒͷϝοηʔδΛrealtimeʹ൓ө • ଞͷΦʔφʔʹΑΔ֤छૢ࡞΋ streaming api Send messages Send message
27. • WebFlux • Spring 5͔Βొ৔ͨ͠Non Blocking WebϑϨʔϜϫʔΫ • SSEͷΑ͏ͳlong connectionΛ࢖༻͢ΔΞϓϦέʔγϣϯ, Gateway server ౳ͱ͸ಛʹ૬ੑ͕͍͍ • Request threadΛઐ༗͠ͳ͍ • ैདྷͷServlet APIͰ͸request threadͷ਺͔͠εέʔϧ͠ͳ͍ SSEͱSpring WebFlux
28. Example code ௕࣌ؒ, eventΛૹΒͳ͍ͱ઀ଓ͕ ੾ΒΕΔͷͰpingૹ৴༻ͷFluxΛ ࡞੒ͯ͠merge͢Δ
29. • ωοτϫʔΫ઀ଓ͕੾Εͯ࠶઀ଓͨ͠ͱ͖ʹɺ
 ͦͷؒͷeventΛड৴Ͱ͖ΔΑ͏ʹ͓ͯ͘͠ඞཁ ͕͋Δ • SSEʹ͸Last-Event-ID headerͱ͍͏
 ࢓૊Έ͕͋Γɺ࠶઀ଓ͢Δͱ͖ʹ࠷ޙʹड৴͠ ͨevent IDΛrequest headerʹؚΊͯ͘ΕΔ • RedisʹPub͢Δͱ͖ʹಉ࣌ʹͦͷdataΛListͱ ͯ͠΋อ͓࣋ͯ͘͠ • ࠶઀ଓͷࡍʹ͸ͦͷList͔ΒऔಘͰ͖ΔΑ͏ʹ • Redis 5͔Β௥Ճ͞ΕͨRedis StreamsΛ࢖͏ͱ ָͰ͖ͦ͏ɻ
 ࠓޙࢼ͍ͨ͠ (ݱࡏ͸Redis 4࢖͍ͬͯΔ) SSEͷ࠶઀ଓ Send messages Send message
30. • Redis 3͔Βొ৔ͨ͠Redis Clusterʹ΋ରԠ͍ͯ͠Δ͕ҎԼͷΑ͏ͳ໰୊΋͋Δ Redis PubSubͷ஫ҙ఺ In a Redis Cluster clients can subscribe to every node, and can also publish to every other node. The cluster will make sure that published messages are forwarded as needed. The current implementation will simply broadcast each published message to all other nodes, but at some point this will be optimized either https://redis.io/topics/cluster-spec#publishsubscribe • εέʔϧ͕ඞཁʹͳͬͨΒɺผͷclusterΛ༻ҙͯ͠ৼΓ෼͚ΔΑ͏ʹͨ͠΄ ͏͕͍͍
32. • LINE User͕Botʹରͯ͠ϝοηʔδΛૹ৴ ͢Δ౳ͷactionΛ͢ΔͱɺWebhook͕ಧ͘ • WebhookΛड͚औͬͨΒଈ࠲ʹKafkaʹ produce • KafkaΛڬΉ͜ͱͰ଱ো֐ੑΛߴΊ͍ͯΔ • consumerଆͰretryՄೳ event receiver
33. • Kafkaࣗମͷ଱ো֐ੑ͸ͱͯ΋ߴ͍ɻͦΕͰ΋ͳʹ͔͠ΒͷτϥϒϧͰো ֐͕ग़Δ͜ͱ΋͋Δ • ͦΕʹඋ͑ͯKafka team͸2ͭͷKafka ClusterΛ༻ҙͯ͘͠Ε͍ͯΔ • Primary Cluster͕downͨ͠ͱ͖͸Secondarily Clusterʹproduce͢Δ Α͏ʹ͍ͯ͠Δ • Primary/SecondarilyؒͰ࣮֬ͳॱ൪อূ͸Ͱ͖ͳ͘ͳΔ͕
 αʔϏεܧଓΛ༏ઌ KafkaͰো֐ʹͳͬͨΒ?
35. • ֤छeventΛॲཧ͢Δkafka consumer server • event receiver͕produceͨ͠event • ଞͷαʔϏε͕produceͨ͠event event processor Consume
36. • ଞͷαʔϏε͕อ͍࣋ͯ͠Δσʔλ͕ߋ৽͞Εͨͱ͖ʹͦͷ৘ใΛड͚औΔ͜ͱ ͕Ͱ͖Δͱศར • ϦΞϧλΠϜʹ৘ใΛߋ৽Ͱ͖ͨΓCacheΛ࡟আͨ͠Γ͢Δ͜ͱ͕Ͱ͖Δ • Kafka͸consumer groupΛ෼͚Δͱಉ͡topicʹରͯ͠ෳ਺ͷconsumerΛ༻ҙ͢Δ ͜ͱ͕Ͱ͖Δ • ୯७ͳΩϡʔͱ͚ͯͩ͠Ͱ͸ͳͯ͘͜͏͍ͬͨ༻్ʹ΋Ԡ༻Ͱ͖Δ ϚΠΫϩαʔϏεؒͷ
 Data Hubͱͯ͠ͷKafka https://logmi.jp/tech/articles/320330
37. • มߋ͕͋ͬͨͱ͖౳ʹ֤αʔϏεʹ௨஌ • ີ݁߹... • ར༻αʔϏε͕૿͑Δͱਏ͍ ੲ΍ͬͯͨବ໨ͳࣄྫ
39. • image, audio, video, file messageͷProxy server • సૹྔ͕େ͖͍ͨΊtomcatͩͱrequest threadΛ ઐ༗͕ͪ͠ɻWebFluxΛ࢖༻ content
41. • Botʹؔ͢Δmessage΍֤छoperationΛอଘ͢Δαʔόʔ • Userͷmessageͱҧͬͯอଘظݶ͕௕Ί • Ϗδωε༻్Ͱ͸อଘظݶ͕௕Ί͡Όͳ͍ͱݫ͍͠ • LINE messengerΞϓϦͱಉ༷ʹHBaseΛར༻ • ݱࡏ͸ࣾ಺ར༻ͷΈ • ͦͷ͏ͪ֎޲͚ʹ΋ެ։ͯ͠֎෦ͷ։ൃऀͷෛ୲ΛݮΒ͍ͨ͠ chatbox
42. Tools
43. • ࣾ಺Logging platform • ֤छlog dataΛKafkaʹૹΔͱKibana౳ͰݟΔ͜ͱ͕Ͱ͖Δ • Request log • application log (via logback appender) • ௐࠪͳͲͰ͘͢͝໾ཱ͍ͬͯΔ Datachain
44. • logbackʹ͸ThreadLocalϕʔεͰ࣮૷͞ΕͨMDCͱ͍͏࢓૊Έ͕͋Δ • logʹ௥Ճ৘ใΛ෇༩Ͱ͖Δ • ྫ͑͹ServletFilterͰrequest৘ใΛMDCʹPut͓ͯ͘͠ͱศར MDC࢖͏ͱ͞Βʹศར
45. WebFluxͰ΋MDC࢖͍͍ͨ • WebFluxͰ΋͜ͷςΫχοΫΛҾ͖ଓ͖࢖͍͍͕ͨ… • ୅ΘΓʹ࣍ͷ3ͭΛ࢖͏͜ͱͰ࣮ݱͰ͖Δ • WebFilter • Context • Hooks#onEachOperator • ※ spring-cloud-sleuth΋ಉ͡Α͏ͳ͜ͱΛ͍ͯ͠Δ
46. • WebFilter Ͱ request৘ใΛContextʹ͍ΕΔ WebFluxͰ΋MDC࢖͍͍ͨ
47. • ֤छonXXXXϝιουΛϥοϓ͢ΔΑ͏ͳCustom SubscriberΛ࣮૷͢Δ • Context͔ΒWebFilterͰొ࿥ͨ͠request৘ใΛऔಘ͢Δ • subscriber#onXXXΛݺͼग़͢લʹMDC#put͠ɺݺͼग़͠ޙʹMDC#remove͢Δ WebFluxͰ΋MDC࢖͍͍ͨ
48. • ઌఔ࡞ͬͨCustom Subscriber͕࢖ΘΕΔΑ͏ʹHooksʹొ࿥͢Δ • Spring BootͳΒConfigurationʹॻ͘ WebFluxͰ΋MDC࢖͍͍ͨ
49. • Spring Boot 2͔ΒMicrometer͕࢖༻͞Ε͍ͯΔ • ಛʹͳʹ΋ઃఆ͠ͳͯ͘΋ྑ͍ײ͡ʹMetricsΛͱͬͯ͘Ε͍ͯΔ • ඞཁʹԠͯࣗ͡෼Ͱ࣮૷͢Δ • Metricsͷͳ͍ίʔυ͸ςετ͕ͳ͍ίʔυҎ্ʹࠔΔͷͰ
 ͔ͬ͠ΓऔΖ͏ɻো֐࣌ʹޙͷࡇΓͱͳΔ Metrics Monitoring
50. • LINEͰ͸MetricsΛPrometheusʹexportͯ͠GrafanaͰݟΔ͜ͱ͕ଟ͍ Metrics Monitoring
51. ࠷ۙ͋ͬͨIssues
52. • ͪΐ͏ͲҰࡢ೔ى͖ͨλΠϜϦʔͳIssue • ྫ͑͹Webflux ClientΛ༻͍ͨ͜Μͳcode͕͋Δͱ͠·͢ OOME Issue • ΋͠΋͜ͷAPI callͰtimeout͕େྔʹൃੜ͍ͯ͘͠ͱɺ
 ͳΜͱOOEMͰࢮʹ·͢
53. • netty͔ΒReadTimeoutException͕ൃੜ͢Δͱreactor core಺Ͱ͜Ε͕࣮ߦ͞ΕΔ • Exceptions#propagate͸RuntimeExceptionͷαϒΫϥεͷ৔߹͸ͦͷ··ฦͯ͠ ͍ΔɻReadTimeoutException͸RuntimeExceptionͷαϒΫϥεͳͷͰ ReadTimeoutExceptionࣗ਎͕ฦΔ • this is ok, as re is alway a new non-singleton instanceͱॻ͍ͯ͋Δ͕… ReadTimeoutException OOME Issue ReadTimeoutException
54. • ReadTimeoutException͸singletonͩͬͨ • ͦͷ݁Ռ, ReadTimeoutException͕อ͍࣋ͯ͠Δ suppressedExceptions(List<Throwable>)͕ͲΜͲΜͱංେԽͯ͠OOME͕ى͖Δ • heapdumpΛௐ΂ͨΒؾ͍ͮͨ • nettyʹGithub issue͋͛ͨΒͦͷ೔ͷ͏ͪʹPRग़ͯͨ
 https://github.com/netty/netty/pull/9152 • ͍ۙ͏ͪʹmaven repoʹ΋͕͋Γͦ͏ OOME Issue
55. • LINEνϟοτͷ໰୊Ͱ͸ͳ͍͕Bot platform΍֤छࣾ಺αʔϏεͰ ى͖͍ͯͨ໰୊ • LINEνϟοτ͸lettuce(reactive api)࢖͍ͬͯΔ • https://github.com/xetorthio/jedis/issues/1945 • connection pool͔Βऔಘ͢Δͱ͖ʹhang͢Δ • Spring BootͷversionΛ͋͛Δͱ͖͸֤छϥΠϒϥϦͷversion΋͋ ͕ΔͷͰ͔ͬ͠Γͱ֬ೝ͠·͠ΐ͏ Jedis(2.9.1) Connection leaking
56. • Reactorͷnon-blocking threadͰblock͢ΔΑ͏ͳcodeΛॻ͍ͯ͸NG • performance௿Լ͸౰વͱͯ͠ɺҙਤ͠ͳ͍deadlock΋ى͖·͢ • https://spring.io/blog/2019/03/28/reactor-debugging-experience#blockhound-a- new-kid-on-the-block • Reactor nettyͰ͸InetSocketAddressΛ࢖ͬͨblocking DNS resolverΛ࢖༻ ͍ͯ͠ΔͷͰɺnettyͷnon-blocking DNS resolverΛ࢖༻͢ΔΑ͏ʹมߋ͠ ·͠ΐ͏ • (·ͩ Unstable ApiͰ͕͢) • ΋͠΋DNS resolveʹ͕͔͔࣌ؒΔͱdeadlock͠·͢ • brennentsmith/slodns౳Λ࢖ͬͯҙਤతʹdelayͤͯ͞ΈΔͱ
 ࠶ݱ͠·͢ • Reactor nettyଆͰɺdefaultΛnettyͷnon-blocking DNS resolverʹ͠Α͏͔ ͱ͍͏issue͸͋Δ • https://github.com/reactor/reactor-netty/issues/569 Blocking DNS resolver
57. THANK YOU
58. https://linecorp.com/ja/career/position/1643