Upgrade to Pro — share decks privately, control downloads, hide ads and more …

LINE公式アカウントのチャットシステムにおけるSpringおよびWebFluxの活用事例 /...

LINE公式アカウントのチャットシステムにおけるSpringおよびWebFluxの活用事例 / Examples of using Spring and WebFlux in the chat system for LINE official accounts

LINE株式会社 長谷部 良輔
SPRING FEST ’19での発表資料です。
https://springfest2019.springframework.jp/

Avatar for LINE Developers

LINE Developers

December 18, 2019
Tweet

More Decks by LINE Developers

Other Decks in Technology

Transcript

  1. About Me > LINEެࣜΞΧ΢ϯτ(OA: Official Account) ͷ։ൃ • LINEνϟοτ, LINE

    Official Account Manager, … ࠷ۙͷ࢓ࣄ Ҏલͷ࢓ࣄ > LINE Login, LINE Customer Connect, Profile+, LINE Notify, LINE BOT SDK Python, 
 LINE GAME PLATFORM, … ௕୩෦ ྑี (@be_hase1014) > ։ൃ4ηϯλʔ / OA Dev 2 νʔϜ > 2013೥ ೖࣾ
  2. Agenda > LINE ♥ Spring > LINEνϟοτʹ͍ͭͯ > Spring ͓Αͼ

    WebFlux ͷ׆༻ࣄྫ • γεςϜߏ੒/ઃܭͷ࿩ • WebFlux(Reactor) ͋Ε͜Ε
  3. Agenda > LINE ♥ Spring > LINEνϟοτʹ͍ͭͯ > Spring ͓Αͼ

    WebFlux ͷ׆༻ࣄྫ • γεςϜߏ੒/ઃܭͷ࿩ • WebFlux(Reactor) ͋Ε͜Ε
  4. Why Spring (boot)? > DI ίϯςφ & AOP > ๛෋ͳΠϯςάϨʔγϣϯ

    • MyBatis, FreeMarker/Thymeleaf, spring-boot-actuator, ... > Profiles & Configuration • ؀ڥຖͷίϯϑΟά؅ཧ͕ྑ͍ײ͡ > RESTful API Λ࣮૷͢ΔͨΊͷඞཁͳύʔπ͕ἧ͍ͬͯΔ • spring-webmvc, spring-webflux > ଞͷ RPC ϑϨʔϜϫʔΫͱ΋૊Έ߹ΘͤΔ͜ͱ͕Ͱ͖Δ • Spring with Armeria, Spring with gRPC
  5. Agenda > LINE ♥ Spring > LINEνϟοτʹ͍ͭͯ > Spring ͓Αͼ

    WebFlux ͷ׆༻ࣄྫ • γεςϜߏ੒/ઃܭͷ࿩ • WebFlux(Reactor) ͋Ε͜Ε
  6. Agenda > LINE ♥ Spring > LINEνϟοτʹ͍ͭͯ > Spring ͓Αͼ

    WebFlux ͷ׆༻ࣄྫ • γεςϜߏ੒/ઃܭͷ࿩ • WebFlux(Reactor) ͋Ε͜Ε
  7. ϒϥ΢βʹ͓͚ΔϦΞϧλΠϜ௨৴ > ީิ • Polling • Long Polling (Comet) •

    Server Sent Events (SSE) • WebSocket • (ࠓͩͬͨΒɺgRPC-WEB ͱ͔ RSocket ΋ݕ౼͔ͨ͠ͳ...) > SSE Λ࠾༻ͨ͠ • ૒ํ޲௨৴͸ෆཁɻςΩετܗࣜͰ΋໰୊ͳ͍ • WebSocket ͱൺֱ͢Δͱγϯϓϧ (HTTP1.1 Chunked ϕʔε)
  8. spring-webmvc Ͱ Server Sent Events > ࢖͍׳Ε͍ͯΔ spring-webmvc(servlet) Ͱ
 ରԠ͞Ε͍ͯΔ͕…

    > SSE ͸αʔόʔͱ઀ଓͬ͠ͺͳ͠ • ैདྷͷ Servlet Ͱ͸εέʔϧ͠ͳ͍ (ӈਤ) • 1ϦΫΤετʹରͯ͠1εϨου ͳͨΊ • LINE Chat ͷݱࡏͷಉ࣌઀ଓ͸5ສ+ • 1ͭͷαʔόʔͰ500εϨου༻ҙͨ͠ͱͯ͠΋ɺ
 ͜Ε͚ͩͰαʔόʔ͕100୆ඞཁʹͳͬͯ͠·͏
 > Servlet 3.0 / Async Servlet Λ࢖͍ͬͯΔͷͰεϨουઐ ༗ͷ໰୊΋ͳͦ͞͏ʹݟ͑Δ͕… Thread1 Thread2 Thread3
  9. ͱ͸͍͑… > ͚͖ͬΐ͘ Servlet ͰͷϦΫΤετͷड৴/ૹ৴ ͸ Input/OutputStream Λ༻͍ͨ Blocking I/O

    • https://www.infoq.com/presentations/servlet- reactive-stack/ > → Spring5 ͔Β௥Ճ͞Εͨ WebFlux Λ࠾༻͢ Δ͜ͱʹ͠·ͨ͠ spring-webmvc Servlet API Tomcat, Jetty, … Blocking I/O
  10. WebFlux ͱ͸ > Spring5 ͔Βొ৔ͨ͠ Non Blocking ͳ Web ϑϨʔϜϫʔΫ

    spring-webmvc Servlet API Tomcat, Jetty, … Blocking I/O spring-webflux Spring Web API
 (Reactor, Reactive Streams) Tomcat, Jetty, …
 (Servlet 3.1) Non Blocking I/O Netty
  11. WebFlux ͱ͸ > ֤ϦΫΤετΛ Non Blocking Ͱॲཧ͢Δ͜ͱͰɺ1ͭͷ Event Loop
 Thread

    Ͱෳ਺ϦΫΤετΛࡹ͚Δ = গͳ͍αʔόʔ਺Ͱࡹ͚Δ > Block ͢Δίʔυ͸ઈରʹॻ͍ͯ͸͍͚ͳ͍ɻઈରʹμϝ Thread1 Thread2 Thread3 Event Loop Thread
  12. ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ Talk Server Event
 Receiver Kafka Event Processor > ϝοηʔδड৴

    Event ͕ಧ͘ > (ଞʹ΋طಡ, ༑ͩͪ௥Ճ, ...౳ͷ Event ΋) > ࣮ࡍ͸ؒʹଞͷίϯϙʔωϯτ͕͋Δ Redis Cluster (PubSub) Redis Cluster SSE
  13. ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ Talk Server Event
 Receiver Kafka Event Processor Redis Cluster

    (PubSub) Redis Cluster SSE > Event Receiver ͸ड͚औͬͨΒ Kafka ʹॻ͘ > Ұ౓ Kafka ʹॻ͍͓ͯ͘͜ͱͰɺEvent ͷ࠶ར༻΍
 ϦτϥΠΛ͠΍͘͢͢Δ > Event Receiver ΋ WebFlux Λ࢖࣮ͬͯ૷͍ͯ͠Δ
  14. ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ Talk Server Event
 Receiver Kafka Event Processor Redis Cluster

    (PubSub) Redis Cluster SSE > Event Processor Ͱ Kafka ͔Β Event ΛಡΉ > Redis ʹ Publish ͢Δ > SSE server ͕ Subscribe ͢Δ
  15. ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ Talk Server Event
 Receiver Kafka Event Processor Redis Cluster

    (PubSub) Redis Cluster SSE > ࠶઀ଓΛαϙʔτ͢ΔͨΊʹɺPublish ͱಉ࣌ʹ
 Ϧετͱͯ͠΋อ͓࣋ͯ͘͠ > ࠶઀ଓʹ࠷ޙʹऔಘͨ͠ Event ID(Last-Event-ID header) ͕౉͞ΕΔͷͰɺະऔಘͷ Event Λฦ͢
  16. ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ Talk Server Event
 Receiver Kafka Event Processor Redis Cluster

    (Stream) SSE > Redis5 ͔Β௥Ճ͞Εͨ Redis Streams Λ
 ࢖͑͹΋ͬͱ؆୯ʹͳΓͦ͏ > ※ զʑͷ Redis ͸ 4ܥ...
  17. spring-webmvc Ͱ΍͍ͬͯΔ MySQL (Sharding) RESTful API
 (spring-webmvc) Talk Server Message


    Gateway ΞΧ΢ϯτΦʔφʔ Elasticsearch HBase http jdbc http Redis Cluster
  18. Why spring-webmvc? > JDBC... (Blocking I/O) • ͓...? • Spring

    Data R2DBC goes GA • https://spring.io/blog/2019/12/06/spring-data-r2dbc-goes-ga > ͍͖ͳΓΨοπϦͱͨ͠ϏδωεϩδοΫΛ WebFlux Ͱ࣮૷͢Δͷ͸
 ͪΐͬͱϏϏͬͨɺͱ͍͏ͷ΋͋Δ • খͯ͘͞ɺ࣮֬ʹ޲͍͍ͯΔίϯϙʔωϯτ͔Βࢼͦ͏ • ։ൃͨ͠ͱ͖͸ boot 2.0.0 M1 ͱ͔ͩͬͨͷͰ...
  19. spring-webflux Ͱ΍͍ͬͯΔ Content Proxy
 (spring-webflux) Content Server ΞΧ΢ϯτΦʔφʔ http >

    ೝূͯ͠ࠨ͔Βӈʹ Proxy ͢Δ͚ͩɻ·͞ʹ Non blocking I/O ͕޲͍͍ͯΔ > εϨουΛઐ༗͢Δ͜ͱͳ͘ Proxy ͢Δ͜ͱ͕Ͱ͖Δ Auth http
  20. ͍·ৼΓฦͬͯΈͯɺͲ͏͔ > େ͖ͳো֐΋ͳ͘1೥ؒӡ༻Ͱ͖ͨͷͰɺٕज़બఆ͸੒ޭͨ͠ͱࢥ͍ͬͯΔ > WebFlux ʹ׳Ε͖ͯͨࠓͱͳͬͯ͸ RESTful API αʔόʔ ΋

    WebFlux Ͱ
 ࣮૷͢Ε͹Α͔ͬͨͳ...ͱ͸ͪΐͬͱ͚ͩࢥ͏ • Elasticsearch ͕ߴෛՙʹͳͬͯϨΠςϯγ͕ߴ·Γɺ Thread Busy ͱͳ ͬͨέʔε͕͋ͬͨ (Servlet ͋Δ͋Δ) • → Circuit breaker(resilience4j) Ͱ failing fast ͢Δ͜ͱͰରॲͯ͠Δ
  21. γεςϜߏ੒/ઃܭͷ࿩ ·ͱΊ > ίϯϙʔωϯτͷ໾ׂʹԠͯ͡ spring-webflux, webmvc Λ࢖͍෼͚͍ͯΔ > spring-webflux •

    SSE αʔόʔ (ϦΞϧλΠϜ௨৴ɺ௕࣌ؒ઀ଓ) • Event Receiver αʔόʔ (ΞΫηεଟΊ) • Content Proxy αʔόʔ > spring-webmvc • RESTful API αʔόʔ (ίΞɾϏδωεϩδοΫ)
  22. ࠔͬͨ࿩ ͦͷ1: Netty OOME > ྫ͑͹Webflux ClientΛ༻͍ͨ
 ͜Μͳcode͕͋Δͱ͠·͢ > ͜ͷ

    API call Ͱ timeout ͕
 େྔʹൃੜ͢Δͱɺ
 OOME Ͱαʔόʔ͕མͪΔͱ͍͏͜ͱ͕͋ͬͨ
  23. ࠔͬͨ࿩ ͦͷ1: Netty OOME > Netty ͔Β ReadTimeoutException ͕
 ൃੜ͢Δͱreactor

    core ಺Ͱ͜Ε͕࣮ߦ͞ΕΔ > Exceptions#propagate ͸RuntimeException 
 ͷαϒΫϥεͷ৔߹͸ͦͷ··ฦ͍ͯ͠Δɻ
 ReadTimeoutException ͸ RuntimeException ͷ
 αϒΫϥεͳͷͰ ReadTimeoutException
 ࣗ਎͕ฦΔɻ > ݁Ռͱͯ͠ɺReadTimeoutException#addSuppressed
 ͕࣮ߦ͞ΕΔ > this is ok, as re is alway a new non-singleton instance 
 ͱॻ͍ͯ͋Δ͕… ReadTimeoutException ͜Ε΋ReadTimeoutException
  24. ࠔͬͨ࿩ ͦͷ1: Netty OOME > ReadTimeoutException ͕ singleton… ͩͬͨ >

    ͦͷ݁Ռ, ReadTimeoutException ͕อ͍࣋ͯ͠Δ suppressedExceptions(List<Throwable>) ͕ංେԽͯ͠ OOME ͕ى͖ͨ > Issue: https://github.com/netty/netty/pull/9152 • ใࠂࡁɻݱࡏ͸मਖ਼͞Ε͍ͯ·͢
  25. ࠔͬͨ࿩ ͦͷ2: Blocking DNS Resolver > Reactor netty Ͱ͸ InetSocketAddress

    Λ࢖ͬͨ blocking DNS resolver Λ࢖༻͠ ͍ͯΔͷͰɺnetty ͷ non-blocking DNS resolver Λ࢖༻͢ΔΑ͏ʹมߋ͠·͠ ΐ͏ • ·ͩ Unstable Api Ͱ͕͢… > ۃكʹ thread ͕ hang ͯ͠͠·͏ݱ৅͕͋ͬͨͨΊௐ΂ͨ > Issue: https://github.com/reactor/reactor-netty/issues/710 • ใࠂࡁɻ0.10.x Backlog ʹೖ͍ͬͯ·͢ > σϑΥϧτΛ non blocking DNS resolver ʹ͠Α͏ɺͱ͍͏Ҋ΋ग़͍ͯ·͢ • https://github.com/reactor/reactor-netty/issues/569
  26. Tips ͦͷ1: Reactor hooks with MDC > Servlet ͷӡ༻ʹ׳Ε͍ͯΔਓ͸ MDC

    ʹ Request ৘ใΛ͍Ε͍ͯͨͱࢥ͍·͢ • ྫ: Logback ͷ MDCInsertingServletFilter • http://logback.qos.ch/xref/ch/qos/logback/classic/helpers/ MDCInsertingServletFilter.html > Request URL ౳ΛҰॹʹϩάʹग़ྗ͢Δ͜ͱ͕Ͱ͖ΔͷͰͱͯ΋ศར
  27. Tips ͦͷ1: Reactor hooks with MDC > WebFlux Ͱ΋ࣅͨΑ͏ͳ͜ͱΛ࣮ݱͨ͠ ͍͕...Event

    Loop Thread ͱ͸ ૬ੑ͕ѱ͍ > ࣍ͷ3ͭͷςΫχοΫΛ࢖͏͜ͱͰՄೳ • WebFilter • Reactor Context • Hooks Event Loop Thread
  28. Tips ͦͷ2: BlockHound > Non Blocking εϨουʹ͓͚Δ Blocking ίʔϧΛݕग़ͯ͘͠ΕΔ Java

    Τʔδ Σϯτ • ͋·Γ೔ຊޠͰݴٴ͞Ε͍ͯΔهࣄΛΈ͔͚ͳ͍ͷͰ঺հ͠·͢ • ίʔυϨϏϡʔͰ͸ൃݟͰ͖ͳ͍Մೳੑ΋͋ΔͷͰɺೖΕ͓ͯ͘ͱྑ͍ • ͠͹͠͹༧ظ͠ͳ͍ Blocking ίʔϧ΋͋ΔͷͰ... > https://github.com/reactor/BlockHound
  29. WebFlux(Reactor) ͋Ε͜Ε ·ͱΊ > ࣮ࡍʹӡ༻ͯ͠Έͯɺࠔͬͨ͜ͱΛ2ͭ঺հ͠·ͨ͠ • ౎౓ɺOSS ʹϑΟʔυόοΫ΋͍ͯ͠·͢ > WebFlux

    ͷӡ༻ʹ͓͚Δ logging(MDC) ͷ Tips Λ঺հ͠·ͨ͠ > Blocking ίʔϧͷݕग़Λͯ͘͠ΕΔ BlockHound ͷ঺հΛ͠·ͨ͠
  30. OA Dev 1 OA Dev 2 OA(Official Account) Dev Team

    > OA Dev 1 • LINE Official Account Manager , …etc > OA Dev 2 • LINEνϟοτ, …etc > We are hiring!! → https://linecorp.com/ja/career/position/1643 6ਓ 8ਓ