LINE公式アカウントのチャットシステムにおけるSpringおよびWebFluxの活用事例 / Examples of using Spring and WebFlux in the chat system for LINE official accounts

LINE公式アカウントのチャットシステムにおけるSpringおよびWebFluxの活用事例 / Examples of using Spring and WebFlux in the chat system for LINE official accounts

LINE株式会社 長谷部 良輔
SPRING FEST ’19での発表資料です。
https://springfest2019.springframework.jp/

53850955f15249a1a9dc49df6113e400?s=128

LINE Developers

December 18, 2019
Tweet

Transcript

  1. LINEެࣜΞΧ΢ϯτͷνϟοτγεςϜ
 ʹ͓͚Δ Spring ͓Αͼ WebFlux ͷ׆༻ࣄྫ > Ryosuke Hasebe /

    LINE corporation
  2. About Me > LINEެࣜΞΧ΢ϯτ(OA: Official Account) ͷ։ൃ • LINEνϟοτ, LINE

    Official Account Manager, … ࠷ۙͷ࢓ࣄ Ҏલͷ࢓ࣄ > LINE Login, LINE Customer Connect, Profile+, LINE Notify, LINE BOT SDK Python, 
 LINE GAME PLATFORM, … ௕୩෦ ྑี (@be_hase1014) > ։ൃ4ηϯλʔ / OA Dev 2 νʔϜ > 2013೥ ೖࣾ
  3. Agenda > LINE ♥ Spring > LINEνϟοτʹ͍ͭͯ > Spring ͓Αͼ

    WebFlux ͷ׆༻ࣄྫ • γεςϜߏ੒/ઃܭͷ࿩ • WebFlux(Reactor) ͋Ε͜Ε
  4. Agenda > LINE ♥ Spring > LINEνϟοτʹ͍ͭͯ > Spring ͓Αͼ

    WebFlux ͷ׆༻ࣄྫ • γεςϜߏ੒/ઃܭͷ࿩ • WebFlux(Reactor) ͋Ε͜Ε
  5. Spring ͨ͘͞Μ
 ࢖͍ͬͯ·͢!!

  6. springframework Ͱݕࡧͯ͠Έͨ > Gradle: 3,044 > Maven POM: 2,788 >

    → 5000 project ͘Β͍͋Δ...ʁ
  7. Why Spring (boot)? > DI ίϯςφ & AOP > ๛෋ͳΠϯςάϨʔγϣϯ

    • MyBatis, FreeMarker/Thymeleaf, spring-boot-actuator, ... > Profiles & Configuration • ؀ڥຖͷίϯϑΟά؅ཧ͕ྑ͍ײ͡ > RESTful API Λ࣮૷͢ΔͨΊͷඞཁͳύʔπ͕ἧ͍ͬͯΔ • spring-webmvc, spring-webflux > ଞͷ RPC ϑϨʔϜϫʔΫͱ΋૊Έ߹ΘͤΔ͜ͱ͕Ͱ͖Δ • Spring with Armeria, Spring with gRPC
  8. LINE ♥ Spring

  9. Agenda > LINE ♥ Spring > LINEνϟοτʹ͍ͭͯ > Spring ͓Αͼ

    WebFlux ͷ׆༻ࣄྫ • γεςϜߏ੒/ઃܭͷ࿩ • WebFlux(Reactor) ͋Ε͜Ε
  10. > اۀ/ࣄۀऀ͕։ઃ͢ΔLINEΞΧ΢ϯτ > Ϗδωε޲͚ʹ༷ʑͳػೳ͕͋Δ • ϒϩʔυΩϟετ/λʔήςΟϯά഑৴ • … etc LINEެࣜΞΧ΢ϯτͬͯʁ

    >date/november 20-21 >place/grand nikko tokyo
  11. LINEνϟοτͬͯʁ > ໊শ͕େม΍΍͍͜͠Ͱ͕͢ɺ
 LINEެࣜΞΧ΢ϯτͷӡӦऀͷͨΊͷνϟοτγεςϜͷ͜ͱͰ͢ > LINEެࣜΞΧ΢ϯτ <-> LINEϢʔβʔؒͰνϟοτ͕Ͱ͖·͢ ΞΧ΢ϯτΦʔφʔ LINEϢʔβʔ

  12. Screen Shot Ƃ >november_20_first day >09:00-18:00 >november_20_first day >09:00-18:00 >date/november

    20-21 >place/grand nikko tokyo
  13. ن໛ײ αʔόʔ਺ (DB / storage ؚΉ) 150+ ಉ࣌઀ଓ਺ 5ສ+ ݄ؒϝοηʔδ


    ૹड৴਺ 4ԯ 2019/12
  14. Agenda > LINE ♥ Spring > LINEνϟοτʹ͍ͭͯ > Spring ͓Αͼ

    WebFlux ͷ׆༻ࣄྫ • γεςϜߏ੒/ઃܭͷ࿩ • WebFlux(Reactor) ͋Ε͜Ε
  15. γεςϜߏ੒/ઃܭͷ࿩

  16. νϟοτγεςϜʹٻΊΒΕΔ΋ͷ (ͦͷ1) > ϦΞϧλΠϜ௨৴ • LINE Ϣʔβʔ͔Βड৴ͨ͠ϝοηʔδ • ଞͷ ΞΧ΢ϯτΦʔφʔ

    ͕ߦͬͨૢ࡞ • → ͜ΕΒΛϦΞϧλΠϜʹΫϥΠΞϯτʹ൓ө͢Δඞཁ͕͋Δ
  17. CGI ࣌୅ͷνϟοτγεςϜ ʊਓਓਓਓਓਓਓਓਓਓਓʊ ʼɹiframe ࣗಈϦϩʔυɹʻ ʉY^Y^Y^Y^Y^Y^Y^Y^Y^Y^Y^ʉ ౰࣌͸ͱͯ΋ָ͔͚ͬͨ͠Ͳ…

  18. ϒϥ΢βʹ͓͚ΔϦΞϧλΠϜ௨৴ > ީิ • Polling • Long Polling (Comet) •

    Server Sent Events (SSE) • WebSocket • (ࠓͩͬͨΒɺgRPC-WEB ͱ͔ RSocket ΋ݕ౼͔ͨ͠ͳ...) > SSE Λ࠾༻ͨ͠ • ૒ํ޲௨৴͸ෆཁɻςΩετܗࣜͰ΋໰୊ͳ͍ • WebSocket ͱൺֱ͢Δͱγϯϓϧ (HTTP1.1 Chunked ϕʔε)
  19. spring-webmvc Ͱ Server Sent Events > ࢖͍׳Ε͍ͯΔ spring-webmvc(servlet) Ͱ
 ରԠ͞Ε͍ͯΔ͕…

    > SSE ͸αʔόʔͱ઀ଓͬ͠ͺͳ͠ • ैདྷͷ Servlet Ͱ͸εέʔϧ͠ͳ͍ (ӈਤ) • 1ϦΫΤετʹରͯ͠1εϨου ͳͨΊ • LINE Chat ͷݱࡏͷಉ࣌઀ଓ͸5ສ+ • 1ͭͷαʔόʔͰ500εϨου༻ҙͨ͠ͱͯ͠΋ɺ
 ͜Ε͚ͩͰαʔόʔ͕100୆ඞཁʹͳͬͯ͠·͏
 > Servlet 3.0 / Async Servlet Λ࢖͍ͬͯΔͷͰεϨουઐ ༗ͷ໰୊΋ͳͦ͞͏ʹݟ͑Δ͕… Thread1 Thread2 Thread3
  20. spring-webmvc Ͱ Server Sent Events ͱ͸͍͑…

  21. ͱ͸͍͑… > ͚͖ͬΐ͘ Servlet ͰͷϦΫΤετͷड৴/ૹ৴ ͸ Input/OutputStream Λ༻͍ͨ Blocking I/O

    • https://www.infoq.com/presentations/servlet- reactive-stack/ > → Spring5 ͔Β௥Ճ͞Εͨ WebFlux Λ࠾༻͢ Δ͜ͱʹ͠·ͨ͠ spring-webmvc Servlet API Tomcat, Jetty, … Blocking I/O
  22. WebFlux ͱ͸ > Spring5 ͔Βొ৔ͨ͠ Non Blocking ͳ Web ϑϨʔϜϫʔΫ

    spring-webmvc Servlet API Tomcat, Jetty, … Blocking I/O spring-webflux Spring Web API
 (Reactor, Reactive Streams) Tomcat, Jetty, …
 (Servlet 3.1) Non Blocking I/O Netty
  23. WebFlux ͱ͸ > ֤ϦΫΤετΛ Non Blocking Ͱॲཧ͢Δ͜ͱͰɺ1ͭͷ Event Loop
 Thread

    Ͱෳ਺ϦΫΤετΛࡹ͚Δ = গͳ͍αʔόʔ਺Ͱࡹ͚Δ > Block ͢Δίʔυ͸ઈରʹॻ͍ͯ͸͍͚ͳ͍ɻઈରʹμϝ Thread1 Thread2 Thread3 Event Loop Thread
  24. αʔόʔ1୆͋ͨΓͷಉ࣌઀ଓ਺ 1୆͋ͨΓͷಉ࣌઀ଓ਺͸2500ɻ 2~3ഒ͸͍͚Δ͚Ͳɺ༨༟Λ΋ͬͯɻ (ݱࡏ͸20୆࢖͍ͬͯ·͢)

  25. ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ Talk Server Event
 Receiver Kafka Event Processor Redis Cluster

    (PubSub) Redis Cluster SSE
  26. ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ Talk Server Event
 Receiver Kafka Event Processor > ϝοηʔδड৴

    Event ͕ಧ͘ > (ଞʹ΋طಡ, ༑ͩͪ௥Ճ, ...౳ͷ Event ΋) > ࣮ࡍ͸ؒʹଞͷίϯϙʔωϯτ͕͋Δ Redis Cluster (PubSub) Redis Cluster SSE
  27. ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ Talk Server Event
 Receiver Kafka Event Processor Redis Cluster

    (PubSub) Redis Cluster SSE > Event Receiver ͸ड͚औͬͨΒ Kafka ʹॻ͘ > Ұ౓ Kafka ʹॻ͍͓ͯ͘͜ͱͰɺEvent ͷ࠶ར༻΍
 ϦτϥΠΛ͠΍͘͢͢Δ > Event Receiver ΋ WebFlux Λ࢖࣮ͬͯ૷͍ͯ͠Δ
  28. ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ Talk Server Event
 Receiver Kafka Event Processor Redis Cluster

    (PubSub) Redis Cluster SSE > Event Processor Ͱ Kafka ͔Β Event ΛಡΉ > Redis ʹ Publish ͢Δ > SSE server ͕ Subscribe ͢Δ
  29. ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ Talk Server Event
 Receiver Kafka Event Processor Redis Cluster

    (PubSub) Redis Cluster SSE > ࠶઀ଓΛαϙʔτ͢ΔͨΊʹɺPublish ͱಉ࣌ʹ
 Ϧετͱͯ͠΋อ͓࣋ͯ͘͠ > ࠶઀ଓʹ࠷ޙʹऔಘͨ͠ Event ID(Last-Event-ID header) ͕౉͞ΕΔͷͰɺະऔಘͷ Event Λฦ͢
  30. ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ Talk Server Event
 Receiver Kafka Event Processor Redis Cluster

    (Stream) SSE > Redis5 ͔Β௥Ճ͞Εͨ Redis Streams Λ
 ࢖͑͹΋ͬͱ؆୯ʹͳΓͦ͏ > ※ զʑͷ Redis ͸ 4ܥ...
  31. Subscribe Redis as Flux (Reactive lettuce) ௕࣌ؒ, Event ͕ͳ͍ͱ઀ଓ͕
 ੾ΕΔͷͰ

    PING ΛૹΔ 2ͭͷ Flux Λ merge
  32. νϟοτγεςϜʹٻΊΒΕΔ΋ͷ (ͦͷ2) > ϝοηʔδͷૹ৴ > աڈϝοηʔδͷऔಘ > ϝοηʔδͷݕࡧ > ͦͷଞɺ֤छσʔλͷ

    CRUDૢ࡞ → Ϣʔβʔૢ࡞ىҼͷɺίΞͱͳΔϏδωεϩδοΫ
  33. spring-webmvc Ͱ΍͍ͬͯΔ MySQL (Sharding) RESTful API
 (spring-webmvc) Talk Server Message


    Gateway ΞΧ΢ϯτΦʔφʔ Elasticsearch HBase http jdbc http Redis Cluster
  34. Why spring-webmvc? > JDBC... (Blocking I/O) • ͓...? • Spring

    Data R2DBC goes GA • https://spring.io/blog/2019/12/06/spring-data-r2dbc-goes-ga > ͍͖ͳΓΨοπϦͱͨ͠ϏδωεϩδοΫΛ WebFlux Ͱ࣮૷͢Δͷ͸
 ͪΐͬͱϏϏͬͨɺͱ͍͏ͷ΋͋Δ • খͯ͘͞ɺ࣮֬ʹ޲͍͍ͯΔίϯϙʔωϯτ͔Βࢼͦ͏ • ։ൃͨ͠ͱ͖͸ boot 2.0.0 M1 ͱ͔ͩͬͨͷͰ...
  35. νϟοτγεςϜʹٻΊΒΕΔ΋ͷ (ͦͷ3) > LINE Ϣʔβʔ͕ૹ৴ͨ͠ը૾ɾಈըɾԻ੠ͷऔಘ • ࣾ಺ͷίϯςϯπαʔόʔ(with CDN)ʹ͋Δ • ೝূ౳ͷॾʑͷࣄ৘Ͱͦͷ··ΞΧ΢ϯτΦʔφʔ͕ΞΫηε͢Δ͜ͱ

    ͸Ͱ͖ͳ͍ͨΊɺProxy ͢Δඞཁ͕͋ͬͨ
  36. spring-webflux Ͱ΍͍ͬͯΔ Content Proxy
 (spring-webflux) Content Server ΞΧ΢ϯτΦʔφʔ http >

    ೝূͯ͠ࠨ͔Βӈʹ Proxy ͢Δ͚ͩɻ·͞ʹ Non blocking I/O ͕޲͍͍ͯΔ > εϨουΛઐ༗͢Δ͜ͱͳ͘ Proxy ͢Δ͜ͱ͕Ͱ͖Δ Auth http
  37. ͍·ৼΓฦͬͯΈͯɺͲ͏͔ > େ͖ͳো֐΋ͳ͘1೥ؒӡ༻Ͱ͖ͨͷͰɺٕज़બఆ͸੒ޭͨ͠ͱࢥ͍ͬͯΔ > WebFlux ʹ׳Ε͖ͯͨࠓͱͳͬͯ͸ RESTful API αʔόʔ ΋

    WebFlux Ͱ
 ࣮૷͢Ε͹Α͔ͬͨͳ...ͱ͸ͪΐͬͱ͚ͩࢥ͏ • Elasticsearch ͕ߴෛՙʹͳͬͯϨΠςϯγ͕ߴ·Γɺ Thread Busy ͱͳ ͬͨέʔε͕͋ͬͨ (Servlet ͋Δ͋Δ) • → Circuit breaker(resilience4j) Ͱ failing fast ͢Δ͜ͱͰରॲͯ͠Δ
  38. γεςϜߏ੒/ઃܭͷ࿩ ·ͱΊ > ίϯϙʔωϯτͷ໾ׂʹԠͯ͡ spring-webflux, webmvc Λ࢖͍෼͚͍ͯΔ > spring-webflux •

    SSE αʔόʔ (ϦΞϧλΠϜ௨৴ɺ௕࣌ؒ઀ଓ) • Event Receiver αʔόʔ (ΞΫηεଟΊ) • Content Proxy αʔόʔ > spring-webmvc • RESTful API αʔόʔ (ίΞɾϏδωεϩδοΫ)
  39. WebFlux(Reactor) ͋Ε͜Ε

  40. ࠔͬͨ࿩ ͦͷ1: Netty OOME > ྫ͑͹Webflux ClientΛ༻͍ͨ
 ͜Μͳcode͕͋Δͱ͠·͢ > ͜ͷ

    API call Ͱ timeout ͕
 େྔʹൃੜ͢Δͱɺ
 OOME Ͱαʔόʔ͕མͪΔͱ͍͏͜ͱ͕͋ͬͨ
  41. ࠔͬͨ࿩ ͦͷ1: Netty OOME > Netty ͔Β ReadTimeoutException ͕
 ൃੜ͢Δͱreactor

    core ಺Ͱ͜Ε͕࣮ߦ͞ΕΔ > Exceptions#propagate ͸RuntimeException 
 ͷαϒΫϥεͷ৔߹͸ͦͷ··ฦ͍ͯ͠Δɻ
 ReadTimeoutException ͸ RuntimeException ͷ
 αϒΫϥεͳͷͰ ReadTimeoutException
 ࣗ਎͕ฦΔɻ > ݁Ռͱͯ͠ɺReadTimeoutException#addSuppressed
 ͕࣮ߦ͞ΕΔ > this is ok, as re is alway a new non-singleton instance 
 ͱॻ͍ͯ͋Δ͕… ReadTimeoutException ͜Ε΋ReadTimeoutException
  42. ࠔͬͨ࿩ ͦͷ1: Netty OOME > ReadTimeoutException ͕ singleton… ͩͬͨ >

    ͦͷ݁Ռ, ReadTimeoutException ͕อ͍࣋ͯ͠Δ suppressedExceptions(List<Throwable>) ͕ංେԽͯ͠ OOME ͕ى͖ͨ > Issue: https://github.com/netty/netty/pull/9152 • ใࠂࡁɻݱࡏ͸मਖ਼͞Ε͍ͯ·͢
  43. ࠔͬͨ࿩ ͦͷ2: Blocking DNS Resolver > Reactor netty Ͱ͸ InetSocketAddress

    Λ࢖ͬͨ blocking DNS resolver Λ࢖༻͠ ͍ͯΔͷͰɺnetty ͷ non-blocking DNS resolver Λ࢖༻͢ΔΑ͏ʹมߋ͠·͠ ΐ͏ • ·ͩ Unstable Api Ͱ͕͢… > ۃكʹ thread ͕ hang ͯ͠͠·͏ݱ৅͕͋ͬͨͨΊௐ΂ͨ > Issue: https://github.com/reactor/reactor-netty/issues/710 • ใࠂࡁɻ0.10.x Backlog ʹೖ͍ͬͯ·͢ > σϑΥϧτΛ non blocking DNS resolver ʹ͠Α͏ɺͱ͍͏Ҋ΋ग़͍ͯ·͢ • https://github.com/reactor/reactor-netty/issues/569
  44. Tips ͦͷ1: Reactor hooks with MDC > Servlet ͷӡ༻ʹ׳Ε͍ͯΔਓ͸ MDC

    ʹ Request ৘ใΛ͍Ε͍ͯͨͱࢥ͍·͢ • ྫ: Logback ͷ MDCInsertingServletFilter • http://logback.qos.ch/xref/ch/qos/logback/classic/helpers/ MDCInsertingServletFilter.html > Request URL ౳ΛҰॹʹϩάʹग़ྗ͢Δ͜ͱ͕Ͱ͖ΔͷͰͱͯ΋ศར
  45. Tips ͦͷ1: Reactor hooks with MDC

  46. Tips ͦͷ1: Reactor hooks with MDC > WebFlux Ͱ΋ࣅͨΑ͏ͳ͜ͱΛ࣮ݱͨ͠ ͍͕...Event

    Loop Thread ͱ͸ ૬ੑ͕ѱ͍ > ࣍ͷ3ͭͷςΫχοΫΛ࢖͏͜ͱͰՄೳ • WebFilter • Reactor Context • Hooks Event Loop Thread
  47. WebFilterͰ Request ৘ใΛ Context 
 ʹೖΕ͓ͯ͘

  48. > ֤छonXXXXϝιουΛϥοϓ͢ΔΑ͏ͳ Custom SubscriberΛ࣮૷͢Δ • Context͔ΒWebFilterͰొ࿥ͨ͠ request৘ใΛऔಘ͢Δ • subscriber#onXXXΛݺͼग़͢લʹ MDC#put͠ɺݺͼग़͠ޙʹ

    MDC#remove͢Δ
  49. > ઌఔ࡞ͬͨ Custom Subscriber ͕࢖ΘΕΔΑ ͏ʹ Hooks ʹొ࿥͢Δ > (͜ͷྫͩͱSpring

    Configuration ʹ ॻ͍ͯΔ)
  50. Tips ͦͷ1: Reactor hooks with MDC

  51. Tips ͦͷ2: BlockHound > Non Blocking εϨουʹ͓͚Δ Blocking ίʔϧΛݕग़ͯ͘͠ΕΔ Java

    Τʔδ Σϯτ • ͋·Γ೔ຊޠͰݴٴ͞Ε͍ͯΔهࣄΛΈ͔͚ͳ͍ͷͰ঺հ͠·͢ • ίʔυϨϏϡʔͰ͸ൃݟͰ͖ͳ͍Մೳੑ΋͋ΔͷͰɺೖΕ͓ͯ͘ͱྑ͍ • ͠͹͠͹༧ظ͠ͳ͍ Blocking ίʔϧ΋͋ΔͷͰ... > https://github.com/reactor/BlockHound
  52. Tips ͦͷ2: BlockHound > ґଘʹ͍ΕΔ

  53. Tips ͦͷ2: BlockHound ྫ͑͹͜Μͳίʔυ͕͋Δͱ͢Δ (ྫͳͷͰத਎͸ద౰)

  54. Tips ͦͷ2: BlockHound BlockHoundΛ༗ޮʹ͢Δ ςετॻ͘

  55. Tips ͦͷ2: BlockHound Block ͯ͠͠·͍ͬͯΔͱ͍͏
 ࢫͷྫ֎͕ग़Δ

  56. WebFlux(Reactor) ͋Ε͜Ε ·ͱΊ > ࣮ࡍʹӡ༻ͯ͠Έͯɺࠔͬͨ͜ͱΛ2ͭ঺հ͠·ͨ͠ • ౎౓ɺOSS ʹϑΟʔυόοΫ΋͍ͯ͠·͢ > WebFlux

    ͷӡ༻ʹ͓͚Δ logging(MDC) ͷ Tips Λ঺հ͠·ͨ͠ > Blocking ίʔϧͷݕग़Λͯ͘͠ΕΔ BlockHound ͷ঺հΛ͠·ͨ͠
  57. (͓·͚) OA Dev Team

  58. OA Dev 1 OA Dev 2 OA(Official Account) Dev Team

    > OA Dev 1 • LINE Official Account Manager , …etc > OA Dev 2 • LINEνϟοτ, …etc > We are hiring!! → https://linecorp.com/ja/career/position/1643 6ਓ 8ਓ
  59. Thank You for Watching