LINEのBot Platformの裏側 / Behind the LINE Bot Platform

JJUG CCC 2019 Springでの登壇資料です

Transcript

1. LINEͷBot Platformͷཪଆͷ࿩
   Ryosuke Hasebe, LINE Corporation
   JJUG CCC Spring 2019
   

   View Slide

2. @be_hase1014 https://github.com/be-hase
   About me
   ● ௕୩෦ ྑี
   ● LINEגࣜձࣾ Account Platform։ൃࣨ AP4νʔϜ Ϛωʔδϟʔ
   ● 2013೥ ೖࣾ
   ● Server-side Engineer
   ● LINEνϟοτ (for LINEެࣜΞΧ΢ϯτ)
   ● Before:
   ● LINE Login, LINE ΧελϚʔίωΫτ, LINE Notify, 

   LINE Profile+, LINE GAME Platform, …etc
   

   View Slide

3. • LINEͱJava
   • LINE Bot Platformʹ͍ͭͯ
   • LINEνϟοτ(for LINEެࣜΞΧ΢ϯτ)
   ͷ։ൃࣄྫΛ۷ΓԼ͛ͯ঺հ
   Agenda
   

   View Slide

4. LINEͱJava
   

   View Slide

5. ݴޠ͝ͱͷrepo਺ (ࣾ಺GHEௐ΂)
   

   View Slide

6. Bot platform component overview
   

   View Slide

7. Bot platform component overview
   Java ͨ͘͞Μʂ
   

   View Slide

8. ● ܕ෇͖, ଎͍
   ● ๛෋ͳϥΠϒϥϦ
   ● ձࣾશମతʹੲ͔Β࢖͍ͬͯΔͷͰࢿ࢈͕ଟ͍
   ● ੈքతʹJavaΤϯδχΞ͕ଟ͍
   ● ։ൃڌ఺͸೔ຊҎ֎ʹ΋୆࿷/λΠ/ؖࠃ/தࠃ/ϕτφϜ౳ʹ͋Δ
   ● ڌ఺Λ·͍ͨͩϓϩμΫτͷҠ؅΋͋Δ
   ● ͱ͸͍͑ඞͣ͠΋JavaΛར༻͠ͳ͍ͱ͍͚ͳ͍ͱ͍͏Θ͚Ͱ͸ͳ͍
   Why Java?
   

   View Slide

9. LINE Bot platformʹ͍ͭͯ
   

   View Slide

10. What’s LINE Bot Platform?
    ● LINEެࣜΞΧ΢ϯτ(๏ਓ޲͚ΞΧ΢ϯταʔϏε)ʹ͓͍ͯBotΛ
    ϕʔεͱͨ͠ϏδωεɾιϦϡʔγϣϯΛఏڙ͢ΔͨΊͷPlatform
    ͷ͜ͱ
    ● ϝοηʔδૹ৴/ड৴
    ● λΠϜϥΠϯ౤ߘ
    ● … etc (ޙड़)
    اۀ΍ݸਓ Ϣʔβʔ
    

    View Slide

11. What’s its VALUE?
    ѹ౗తͳϦʔνྗ ʂʂ
    ສਓ
    8000
    Percent
    86
    ೔ຊͷਓޱͷ
    60%Ҏ্
    ຖ೔ར༻͍ͯ͠Δ
    ೔ຊࠃ಺ͷϢʔβʔ
    

    View Slide

12. ఏڙ͍ͯ͠Δػೳ
    

    View Slide

13. ● ϝοηʔδ഑৴
    ● ࣗಈԠ౴/ΩʔϫʔυԠ౴
    ● Ϧονϝχϡʔ
    ● λΠϜϥΠϯ౤ߘ
    ● γϣοϓΧʔυ
    ● Ϋʔϙϯ
    ● ֤छΞφϦςΟΫεͷఏڙ
    ● ܾࡁ
    ● …etc
    LINE Official Account Manager
    (CMS)
    

    View Slide

14. ● ΞΧ΢ϯτ؅ཧऀ༻ͷνϟοτπʔϧ
    ● εςʔλεŋλάΛ༻͍ͨސ٬؅ཧ
    ● ޙ΄Ͳ։ൃࣄྫΛ঺հ͠·͢
    LINEνϟοτ
    

    View Slide

15. Messaging API
    ● ςοΫاۀ / ։ൃऀ޲͚ͷAPI܈ɻChatbotΛࣗ༝ʹ࡞੒/࣮૷Մೳ
    ● ϝοηʔδͷड৴͸Webhook
    ● ࣮͸ࣾ಺ͷ֤छαʔϏε΋࢖༻͍ͯ͠·͢
    send messages
    Receive messages via
    webhook
    

    View Slide

16. ● LINE Beacon
    ● ༑ͩͪΦʔσΟΤϯε഑৴
    ● LINE LIVE࿈ܞ
    ● …etc
    Others
    

    View Slide

17. ϝοηʔδ഑ૹ਺

    (݄ؒ)
    800ԯ
    ΞΧ΢ϯτ਺
    900ສ+
    LINE Bot Platformͷن໛ײ
    Messaging API
    daily requests
    2.2ԯ+
    αʔόʔ਺
    (app serverͷΈ)
    500+
    

    View Slide

18. CMS server
    Dev Team
    Chat server Bot backend server
    CMS & Core web-front Chat web-front Native App
    ● ։ൃڌ఺͸೔ຊɺؖࠃɺϕτφϜ
    ● Native App͸΄΅WebViewͱ࣮ͯ͠૷͍ͯ͠ΔͷͰfrontͷਓ਺͕ଟΊ
    ● ଞʹ΋֤छઐ໳νʔϜͱ
    ● ITSC (Private cloud, LB, …), DBA (MySQL, HBase, Redis), 

    IMF (Kafka), Datalabs, QA, …etc
    6ਓ 6ਓ 8ਓ
    17ਓ 4ਓ 5ਓ
    

    View Slide

19. LINEνϟοτͷ։ൃࣄྫ
    

    View Slide

20. (͓͞Β͍) LINEνϟοτ
    

    View Slide

21. api streaming api content
    • Spring Boot 2
    • Spring WebFlux
    • Redis Cluster
    • Redis PubSub
    • Spring Boot 2
    • Spring MVC
    • MySQL
    • Redis Cluster
    LINEνϟοτͷcomponent
    event receiver event processor chatbox
    • Spring Boot 1
    • Armeria(thrift)
    • HBase
    • Spring Boot 2
    • Spring WebFlux
    • Spring Boot 2
    • Spring WebFlux
    • Kafka
    • Spring Boot 2
    • Kafka
    • MySQL
    • Redis Cluster
    • Redis PubSub
    

    View Slide

22. View Slide

23. View Slide

24. ● ͘͝ҰൠతͳJSON API server
    ● Φʔφʔͷૢ࡞ʹΑΔ֤छϏδωεϩδοΫ
    ● ϝοηʔδૹ৴
    ● աڈϝοηʔδऔಘ
    ● εςʔλε/λάͳͲͷσʔλอଘ
    ● MySQL΁Blocking IOΛ͍ͯ͠ΔͷͰ

    ੲͳ͕ΒͷSpring MVC(tomcat)
    api
    

    View Slide

25. View Slide

26. ● ChatΞϓϦέʔγϣϯʹ͸realtimeੑ͕ඞཁ
    ● Server-Sent events(SSE) by Spring WebFlux
    ● WebSocketΛ࠾༻͢Δ͔೰Μ͕ͩɺ૒ํ޲௨
    ৴͸ඞཁͳ͍ͷͰSSEΛ࠾༻
    ● LINE user͔ΒͷϝοηʔδΛrealtimeʹ൓ө
    ● ଞͷΦʔφʔʹΑΔ֤छૢ࡞΋
    streaming api
    Send messages
    Send message
    

    View Slide

27. ● WebFlux
    ● Spring 5͔Βొ৔ͨ͠Non Blocking WebϑϨʔϜϫʔΫ
    ● SSEͷΑ͏ͳlong connectionΛ࢖༻͢ΔΞϓϦέʔγϣϯ, Gateway server
    ౳ͱ͸ಛʹ૬ੑ͕͍͍
    ● Request threadΛઐ༗͠ͳ͍
    ● ैདྷͷServlet APIͰ͸request threadͷ਺͔͠εέʔϧ͠ͳ͍
    SSEͱSpring WebFlux
    

    View Slide

28. Example code
    ௕࣌ؒ, eventΛૹΒͳ͍ͱ઀ଓ͕
    ੾ΒΕΔͷͰpingૹ৴༻ͷFluxΛ
    ࡞੒ͯ͠merge͢Δ
    

    View Slide

29. ● ωοτϫʔΫ઀ଓ͕੾Εͯ࠶઀ଓͨ͠ͱ͖ʹɺ

    ͦͷؒͷeventΛड৴Ͱ͖ΔΑ͏ʹ͓ͯ͘͠ඞཁ
    ͕͋Δ
    ● SSEʹ͸Last-Event-ID headerͱ͍͏

    ࢓૊Έ͕͋Γɺ࠶઀ଓ͢Δͱ͖ʹ࠷ޙʹड৴͠
    ͨevent IDΛrequest headerʹؚΊͯ͘ΕΔ
    ● RedisʹPub͢Δͱ͖ʹಉ࣌ʹͦͷdataΛListͱ
    ͯ͠΋อ͓࣋ͯ͘͠
    ● ࠶઀ଓͷࡍʹ͸ͦͷList͔ΒऔಘͰ͖ΔΑ͏ʹ
    ● Redis 5͔Β௥Ճ͞ΕͨRedis StreamsΛ࢖͏ͱ
    ָͰ͖ͦ͏ɻ

    ࠓޙࢼ͍ͨ͠ (ݱࡏ͸Redis 4࢖͍ͬͯΔ)
    SSEͷ࠶઀ଓ
    Send messages
    Send message
    

    View Slide

30. ● Redis 3͔Βొ৔ͨ͠Redis Clusterʹ΋ରԠ͍ͯ͠Δ͕ҎԼͷΑ͏ͳ໰୊΋͋Δ
    Redis PubSubͷ஫ҙ఺
    In a Redis Cluster clients can subscribe to every node, and can
    also publish to every other node. The cluster will make sure that
    published messages are forwarded as needed. The current
    implementation will simply broadcast each published message
    to all other nodes, but at some point this will be optimized either
    https://redis.io/topics/cluster-spec#publishsubscribe
    ● εέʔϧ͕ඞཁʹͳͬͨΒɺผͷclusterΛ༻ҙͯ͠ৼΓ෼͚ΔΑ͏ʹͨ͠΄
    ͏͕͍͍
    

    View Slide

31. View Slide

32. ● LINE User͕Botʹରͯ͠ϝοηʔδΛૹ৴
    ͢Δ౳ͷactionΛ͢ΔͱɺWebhook͕ಧ͘
    ● WebhookΛड͚औͬͨΒଈ࠲ʹKafkaʹ
    produce
    ● KafkaΛڬΉ͜ͱͰ଱ো֐ੑΛߴΊ͍ͯΔ
    ● consumerଆͰretryՄೳ
    event receiver
    

    View Slide

33. ● Kafkaࣗମͷ଱ো֐ੑ͸ͱͯ΋ߴ͍ɻͦΕͰ΋ͳʹ͔͠ΒͷτϥϒϧͰো
    ֐͕ग़Δ͜ͱ΋͋Δ
    ● ͦΕʹඋ͑ͯKafka team͸2ͭͷKafka ClusterΛ༻ҙͯ͘͠Ε͍ͯΔ
    ● Primary Cluster͕downͨ͠ͱ͖͸Secondarily Clusterʹproduce͢Δ
    Α͏ʹ͍ͯ͠Δ
    ● Primary/SecondarilyؒͰ࣮֬ͳॱ൪อূ͸Ͱ͖ͳ͘ͳΔ͕

    αʔϏεܧଓΛ༏ઌ
    KafkaͰো֐ʹͳͬͨΒ?
    

    View Slide

34. View Slide

35. ● ֤छeventΛॲཧ͢Δkafka consumer server
    ● event receiver͕produceͨ͠event
    ● ଞͷαʔϏε͕produceͨ͠event
    event processor
    Consume
    

    View Slide

36. ● ଞͷαʔϏε͕อ͍࣋ͯ͠Δσʔλ͕ߋ৽͞Εͨͱ͖ʹͦͷ৘ใΛड͚औΔ͜ͱ
    ͕Ͱ͖Δͱศར
    ● ϦΞϧλΠϜʹ৘ใΛߋ৽Ͱ͖ͨΓCacheΛ࡟আͨ͠Γ͢Δ͜ͱ͕Ͱ͖Δ
    ● Kafka͸consumer groupΛ෼͚Δͱಉ͡topicʹରͯ͠ෳ਺ͷconsumerΛ༻ҙ͢Δ
    ͜ͱ͕Ͱ͖Δ
    ● ୯७ͳΩϡʔͱ͚ͯͩ͠Ͱ͸ͳͯ͘͜͏͍ͬͨ༻్ʹ΋Ԡ༻Ͱ͖Δ
    ϚΠΫϩαʔϏεؒͷ

    Data Hubͱͯ͠ͷKafka
    https://logmi.jp/tech/articles/320330
    

    View Slide

37. ● มߋ͕͋ͬͨͱ͖౳ʹ֤αʔϏεʹ௨஌
    ● ີ݁߹...
    ● ར༻αʔϏε͕૿͑Δͱਏ͍
    ੲ΍ͬͯͨବ໨ͳࣄྫ
    

    View Slide

38. View Slide

39. ● image, audio, video, file messageͷProxy server
    ● సૹྔ͕େ͖͍ͨΊtomcatͩͱrequest threadΛ
    ઐ༗͕ͪ͠ɻWebFluxΛ࢖༻
    content
    

    View Slide

40. View Slide

41. ● Botʹؔ͢Δmessage΍֤छoperationΛอଘ͢Δαʔόʔ
    ● Userͷmessageͱҧͬͯอଘظݶ͕௕Ί
    ● Ϗδωε༻్Ͱ͸อଘظݶ͕௕Ί͡Όͳ͍ͱݫ͍͠
    ● LINE messengerΞϓϦͱಉ༷ʹHBaseΛར༻
    ● ݱࡏ͸ࣾ಺ར༻ͷΈ
    ● ͦͷ͏ͪ֎޲͚ʹ΋ެ։ͯ͠֎෦ͷ։ൃऀͷෛ୲ΛݮΒ͍ͨ͠
    chatbox
    

    View Slide

42. Tools
    

    View Slide

43. ● ࣾ಺Logging platform
    ● ֤छlog dataΛKafkaʹૹΔͱKibana౳ͰݟΔ͜ͱ͕Ͱ͖Δ
    ● Request log
    ● application log (via logback appender)
    ● ௐࠪͳͲͰ͘͢͝໾ཱ͍ͬͯΔ
    Datachain
    

    View Slide

44. ● logbackʹ͸ThreadLocalϕʔεͰ࣮૷͞ΕͨMDCͱ͍͏࢓૊Έ͕͋Δ
    ● logʹ௥Ճ৘ใΛ෇༩Ͱ͖Δ
    ● ྫ͑͹ServletFilterͰrequest৘ใΛMDCʹPut͓ͯ͘͠ͱศར
    MDC࢖͏ͱ͞Βʹศར
    

    View Slide

45. WebFluxͰ΋MDC࢖͍͍ͨ
    ● WebFluxͰ΋͜ͷςΫχοΫΛҾ͖ଓ͖࢖͍͍͕ͨ…
    ● ୅ΘΓʹ࣍ͷ3ͭΛ࢖͏͜ͱͰ࣮ݱͰ͖Δ
    ● WebFilter
    ● Context
    ● Hooks#onEachOperator
    ● ※ spring-cloud-sleuth΋ಉ͡Α͏ͳ͜ͱΛ͍ͯ͠Δ
    

    View Slide

46. ● WebFilter Ͱ request৘ใΛContextʹ͍ΕΔ
    WebFluxͰ΋MDC࢖͍͍ͨ
    

    View Slide

47. ● ֤छonXXXXϝιουΛϥοϓ͢ΔΑ͏ͳCustom SubscriberΛ࣮૷͢Δ
    ● Context͔ΒWebFilterͰొ࿥ͨ͠request৘ใΛऔಘ͢Δ
    ● subscriber#onXXXΛݺͼग़͢લʹMDC#put͠ɺݺͼग़͠ޙʹMDC#remove͢Δ
    WebFluxͰ΋MDC࢖͍͍ͨ
    

    View Slide

48. ● ઌఔ࡞ͬͨCustom Subscriber͕࢖ΘΕΔΑ͏ʹHooksʹొ࿥͢Δ
    ● Spring BootͳΒConfigurationʹॻ͘
    WebFluxͰ΋MDC࢖͍͍ͨ
    

    View Slide

49. ● Spring Boot 2͔ΒMicrometer͕࢖༻͞Ε͍ͯΔ
    ● ಛʹͳʹ΋ઃఆ͠ͳͯ͘΋ྑ͍ײ͡ʹMetricsΛͱͬͯ͘Ε͍ͯΔ
    ● ඞཁʹԠͯࣗ͡෼Ͱ࣮૷͢Δ
    ● Metricsͷͳ͍ίʔυ͸ςετ͕ͳ͍ίʔυҎ্ʹࠔΔͷͰ

    ͔ͬ͠ΓऔΖ͏ɻো֐࣌ʹޙͷࡇΓͱͳΔ
    Metrics Monitoring
    

    View Slide

50. ● LINEͰ͸MetricsΛPrometheusʹexportͯ͠GrafanaͰݟΔ͜ͱ͕ଟ͍
    Metrics Monitoring
    

    View Slide

51. ࠷ۙ͋ͬͨIssues
    

    View Slide

52. ● ͪΐ͏ͲҰࡢ೔ى͖ͨλΠϜϦʔͳIssue
    ● ྫ͑͹Webflux ClientΛ༻͍ͨ͜Μͳcode͕͋Δͱ͠·͢
    OOME Issue
    ● ΋͠΋͜ͷAPI callͰtimeout͕େྔʹൃੜ͍ͯ͘͠ͱɺ

    ͳΜͱOOEMͰࢮʹ·͢
    

    View Slide

53. ● netty͔ΒReadTimeoutException͕ൃੜ͢Δͱreactor core಺Ͱ͜Ε͕࣮ߦ͞ΕΔ
    ● Exceptions#propagate͸RuntimeExceptionͷαϒΫϥεͷ৔߹͸ͦͷ··ฦͯ͠
    ͍ΔɻReadTimeoutException͸RuntimeExceptionͷαϒΫϥεͳͷͰ
    ReadTimeoutExceptionࣗ਎͕ฦΔ
    ● this is ok, as re is alway a new non-singleton instanceͱॻ͍ͯ͋Δ͕…
    ReadTimeoutException
    OOME Issue
    ReadTimeoutException
    

    View Slide

54. ● ReadTimeoutException͸singletonͩͬͨ
    ● ͦͷ݁Ռ, ReadTimeoutException͕อ͍࣋ͯ͠Δ
    suppressedExceptions(List)͕ͲΜͲΜͱංେԽͯ͠OOME͕ى͖Δ
    ● heapdumpΛௐ΂ͨΒؾ͍ͮͨ
    ● nettyʹGithub issue͋͛ͨΒͦͷ೔ͷ͏ͪʹPRग़ͯͨ

    https://github.com/netty/netty/pull/9152
    ● ͍ۙ͏ͪʹmaven repoʹ΋͕͋Γͦ͏
    OOME Issue
    

    View Slide

55. ● LINEνϟοτͷ໰୊Ͱ͸ͳ͍͕Bot platform΍֤छࣾ಺αʔϏεͰ
    ى͖͍ͯͨ໰୊
    ● LINEνϟοτ͸lettuce(reactive api)࢖͍ͬͯΔ
    ● https://github.com/xetorthio/jedis/issues/1945
    ● connection pool͔Βऔಘ͢Δͱ͖ʹhang͢Δ
    ● Spring BootͷversionΛ͋͛Δͱ͖͸֤छϥΠϒϥϦͷversion΋͋
    ͕ΔͷͰ͔ͬ͠Γͱ֬ೝ͠·͠ΐ͏
    Jedis(2.9.1) Connection leaking
    

    View Slide

56. ● Reactorͷnon-blocking threadͰblock͢ΔΑ͏ͳcodeΛॻ͍ͯ͸NG
    ● performance௿Լ͸౰વͱͯ͠ɺҙਤ͠ͳ͍deadlock΋ى͖·͢
    ● https://spring.io/blog/2019/03/28/reactor-debugging-experience#blockhound-a-
    new-kid-on-the-block
    ● Reactor nettyͰ͸InetSocketAddressΛ࢖ͬͨblocking DNS resolverΛ࢖༻
    ͍ͯ͠ΔͷͰɺnettyͷnon-blocking DNS resolverΛ࢖༻͢ΔΑ͏ʹมߋ͠
    ·͠ΐ͏
    ● (·ͩ Unstable ApiͰ͕͢)
    ● ΋͠΋DNS resolveʹ͕͔͔࣌ؒΔͱdeadlock͠·͢
    ● brennentsmith/slodns౳Λ࢖ͬͯҙਤతʹdelayͤͯ͞ΈΔͱ

    ࠶ݱ͠·͢
    ● Reactor nettyଆͰɺdefaultΛnettyͷnon-blocking DNS resolverʹ͠Α͏͔
    ͱ͍͏issue͸͋Δ
    ● https://github.com/reactor/reactor-netty/issues/569
    Blocking DNS resolver
    

    View Slide

57. THANK YOU
    

    View Slide

58. https://linecorp.com/ja/career/position/1643
    

    View Slide