LINE公式アカウントのチャットシステムにおけるSpringおよびWebFluxの活用事例 / Examples of using Spring and WebFlux in the chat system for LINE official accounts

LINE株式会社 長谷部 良輔
SPRING FEST ’19での発表資料です。
https://springfest2019.springframework.jp/

Transcript

1. LINEެࣜΞΧ΢ϯτͷνϟοτγεςϜ
 ʹ͓͚Δ Spring ͓Αͼ WebFlux ͷ׆༻ࣄྫ > Ryosuke Hasebe / LINE corporation
2. About Me > LINEެࣜΞΧ΢ϯτ(OA: Official Account) ͷ։ൃ • LINEνϟοτ, LINE Official Account Manager, … ࠷ۙͷ࢓ࣄ Ҏલͷ࢓ࣄ > LINE Login, LINE Customer Connect, Profile+, LINE Notify, LINE BOT SDK Python, 
 LINE GAME PLATFORM, … ௕୩෦ ྑี (@be_hase1014) > ։ൃ4ηϯλʔ / OA Dev 2 νʔϜ > 2013೥ ೖࣾ
3. Agenda > LINE ♥ Spring > LINEνϟοτʹ͍ͭͯ > Spring ͓Αͼ WebFlux ͷ׆༻ࣄྫ • γεςϜߏ੒/ઃܭͷ࿩ • WebFlux(Reactor) ͋Ε͜Ε
4. Agenda > LINE ♥ Spring > LINEνϟοτʹ͍ͭͯ > Spring ͓Αͼ WebFlux ͷ׆༻ࣄྫ • γεςϜߏ੒/ઃܭͷ࿩ • WebFlux(Reactor) ͋Ε͜Ε
5. Spring ͨ͘͞Μ
 ࢖͍ͬͯ·͢!!
6. springframework Ͱݕࡧͯ͠Έͨ > Gradle: 3,044 > Maven POM: 2,788 > → 5000 project ͘Β͍͋Δ...ʁ
7. Why Spring (boot)? > DI ίϯςφ & AOP > ๛෋ͳΠϯςάϨʔγϣϯ • MyBatis, FreeMarker/Thymeleaf, spring-boot-actuator, ... > Profiles & Configuration • ؀ڥຖͷίϯϑΟά؅ཧ͕ྑ͍ײ͡ > RESTful API Λ࣮૷͢ΔͨΊͷඞཁͳύʔπ͕ἧ͍ͬͯΔ • spring-webmvc, spring-webflux > ଞͷ RPC ϑϨʔϜϫʔΫͱ΋૊Έ߹ΘͤΔ͜ͱ͕Ͱ͖Δ • Spring with Armeria, Spring with gRPC
8. LINE ♥ Spring
9. Agenda > LINE ♥ Spring > LINEνϟοτʹ͍ͭͯ > Spring ͓Αͼ WebFlux ͷ׆༻ࣄྫ • γεςϜߏ੒/ઃܭͷ࿩ • WebFlux(Reactor) ͋Ε͜Ε
10. > اۀ/ࣄۀऀ͕։ઃ͢ΔLINEΞΧ΢ϯτ > Ϗδωε޲͚ʹ༷ʑͳػೳ͕͋Δ • ϒϩʔυΩϟετ/λʔήςΟϯά഑৴ • … etc LINEެࣜΞΧ΢ϯτͬͯʁ >date/november 20-21 >place/grand nikko tokyo
11. LINEνϟοτͬͯʁ > ໊শ͕େม΍΍͍͜͠Ͱ͕͢ɺ
 LINEެࣜΞΧ΢ϯτͷӡӦऀͷͨΊͷνϟοτγεςϜͷ͜ͱͰ͢ > LINEެࣜΞΧ΢ϯτ <-> LINEϢʔβʔؒͰνϟοτ͕Ͱ͖·͢ ΞΧ΢ϯτΦʔφʔ LINEϢʔβʔ
12. Screen Shot Ƃ >november_20_first day >09:00-18:00 >november_20_first day >09:00-18:00 >date/november 20-21 >place/grand nikko tokyo
13. ن໛ײ αʔόʔ਺ (DB / storage ؚΉ) 150+ ಉ࣌઀ଓ਺ 5ສ+ ݄ؒϝοηʔδ
 ૹड৴਺ 4ԯ 2019/12
14. Agenda > LINE ♥ Spring > LINEνϟοτʹ͍ͭͯ > Spring ͓Αͼ WebFlux ͷ׆༻ࣄྫ • γεςϜߏ੒/ઃܭͷ࿩ • WebFlux(Reactor) ͋Ε͜Ε
15. γεςϜߏ੒/ઃܭͷ࿩
16. νϟοτγεςϜʹٻΊΒΕΔ΋ͷ (ͦͷ1) > ϦΞϧλΠϜ௨৴ • LINE Ϣʔβʔ͔Βड৴ͨ͠ϝοηʔδ • ଞͷ ΞΧ΢ϯτΦʔφʔ ͕ߦͬͨૢ࡞ • → ͜ΕΒΛϦΞϧλΠϜʹΫϥΠΞϯτʹ൓ө͢Δඞཁ͕͋Δ
17. CGI ࣌୅ͷνϟοτγεςϜ ʊਓਓਓਓਓਓਓਓਓਓਓʊ ʼɹiframe ࣗಈϦϩʔυɹʻ ʉY^Y^Y^Y^Y^Y^Y^Y^Y^Y^Y^ʉ ౰࣌͸ͱͯ΋ָ͔͚ͬͨ͠Ͳ…
18. ϒϥ΢βʹ͓͚ΔϦΞϧλΠϜ௨৴ > ީิ • Polling • Long Polling (Comet) • Server Sent Events (SSE) • WebSocket • (ࠓͩͬͨΒɺgRPC-WEB ͱ͔ RSocket ΋ݕ౼͔ͨ͠ͳ...) > SSE Λ࠾༻ͨ͠ • ૒ํ޲௨৴͸ෆཁɻςΩετܗࣜͰ΋໰୊ͳ͍ • WebSocket ͱൺֱ͢Δͱγϯϓϧ (HTTP1.1 Chunked ϕʔε)
19. spring-webmvc Ͱ Server Sent Events > ࢖͍׳Ε͍ͯΔ spring-webmvc(servlet) Ͱ
 ରԠ͞Ε͍ͯΔ͕… > SSE ͸αʔόʔͱ઀ଓͬ͠ͺͳ͠ • ैདྷͷ Servlet Ͱ͸εέʔϧ͠ͳ͍ (ӈਤ) • 1ϦΫΤετʹରͯ͠1εϨου ͳͨΊ • LINE Chat ͷݱࡏͷಉ࣌઀ଓ͸5ສ+ • 1ͭͷαʔόʔͰ500εϨου༻ҙͨ͠ͱͯ͠΋ɺ
 ͜Ε͚ͩͰαʔόʔ͕100୆ඞཁʹͳͬͯ͠·͏
 > Servlet 3.0 / Async Servlet Λ࢖͍ͬͯΔͷͰεϨουઐ ༗ͷ໰୊΋ͳͦ͞͏ʹݟ͑Δ͕… Thread1 Thread2 Thread3
20. spring-webmvc Ͱ Server Sent Events ͱ͸͍͑…
21. ͱ͸͍͑… > ͚͖ͬΐ͘ Servlet ͰͷϦΫΤετͷड৴/ૹ৴ ͸ Input/OutputStream Λ༻͍ͨ Blocking I/O • https://www.infoq.com/presentations/servlet- reactive-stack/ > → Spring5 ͔Β௥Ճ͞Εͨ WebFlux Λ࠾༻͢ Δ͜ͱʹ͠·ͨ͠ spring-webmvc Servlet API Tomcat, Jetty, … Blocking I/O
22. WebFlux ͱ͸ > Spring5 ͔Βొ৔ͨ͠ Non Blocking ͳ Web ϑϨʔϜϫʔΫ spring-webmvc Servlet API Tomcat, Jetty, … Blocking I/O spring-webflux Spring Web API
 (Reactor, Reactive Streams) Tomcat, Jetty, …
 (Servlet 3.1) Non Blocking I/O Netty
23. WebFlux ͱ͸ > ֤ϦΫΤετΛ Non Blocking Ͱॲཧ͢Δ͜ͱͰɺ1ͭͷ Event Loop
 Thread Ͱෳ਺ϦΫΤετΛࡹ͚Δ = গͳ͍αʔόʔ਺Ͱࡹ͚Δ > Block ͢Δίʔυ͸ઈରʹॻ͍ͯ͸͍͚ͳ͍ɻઈରʹμϝ Thread1 Thread2 Thread3 Event Loop Thread
24. αʔόʔ1୆͋ͨΓͷಉ࣌઀ଓ਺ 1୆͋ͨΓͷಉ࣌઀ଓ਺͸2500ɻ 2~3ഒ͸͍͚Δ͚Ͳɺ༨༟Λ΋ͬͯɻ (ݱࡏ͸20୆࢖͍ͬͯ·͢)
25. ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ Talk Server Event
 Receiver Kafka Event Processor Redis Cluster (PubSub) Redis Cluster SSE
26. ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ Talk Server Event
 Receiver Kafka Event Processor > ϝοηʔδड৴ Event ͕ಧ͘ > (ଞʹ΋طಡ, ༑ͩͪ௥Ճ, ...౳ͷ Event ΋) > ࣮ࡍ͸ؒʹଞͷίϯϙʔωϯτ͕͋Δ Redis Cluster (PubSub) Redis Cluster SSE
27. ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ Talk Server Event
 Receiver Kafka Event Processor Redis Cluster (PubSub) Redis Cluster SSE > Event Receiver ͸ड͚औͬͨΒ Kafka ʹॻ͘ > Ұ౓ Kafka ʹॻ͍͓ͯ͘͜ͱͰɺEvent ͷ࠶ར༻΍
 ϦτϥΠΛ͠΍͘͢͢Δ > Event Receiver ΋ WebFlux Λ࢖࣮ͬͯ૷͍ͯ͠Δ
28. ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ Talk Server Event
 Receiver Kafka Event Processor Redis Cluster (PubSub) Redis Cluster SSE > Event Processor Ͱ Kafka ͔Β Event ΛಡΉ > Redis ʹ Publish ͢Δ > SSE server ͕ Subscribe ͢Δ
29. ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ Talk Server Event
 Receiver Kafka Event Processor Redis Cluster (PubSub) Redis Cluster SSE > ࠶઀ଓΛαϙʔτ͢ΔͨΊʹɺPublish ͱಉ࣌ʹ
 Ϧετͱͯ͠΋อ͓࣋ͯ͘͠ > ࠶઀ଓʹ࠷ޙʹऔಘͨ͠ Event ID(Last-Event-ID header) ͕౉͞ΕΔͷͰɺະऔಘͷ Event Λฦ͢
30. ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ Talk Server Event
 Receiver Kafka Event Processor Redis Cluster (Stream) SSE > Redis5 ͔Β௥Ճ͞Εͨ Redis Streams Λ
 ࢖͑͹΋ͬͱ؆୯ʹͳΓͦ͏ > ※ զʑͷ Redis ͸ 4ܥ...
31. Subscribe Redis as Flux (Reactive lettuce) ௕࣌ؒ, Event ͕ͳ͍ͱ઀ଓ͕
 ੾ΕΔͷͰ PING ΛૹΔ 2ͭͷ Flux Λ merge
32. νϟοτγεςϜʹٻΊΒΕΔ΋ͷ (ͦͷ2) > ϝοηʔδͷૹ৴ > աڈϝοηʔδͷऔಘ > ϝοηʔδͷݕࡧ > ͦͷଞɺ֤छσʔλͷ CRUDૢ࡞ → Ϣʔβʔૢ࡞ىҼͷɺίΞͱͳΔϏδωεϩδοΫ
33. spring-webmvc Ͱ΍͍ͬͯΔ MySQL (Sharding) RESTful API
 (spring-webmvc) Talk Server Message
 Gateway ΞΧ΢ϯτΦʔφʔ Elasticsearch HBase http jdbc http Redis Cluster
34. Why spring-webmvc? > JDBC... (Blocking I/O) • ͓...? • Spring Data R2DBC goes GA • https://spring.io/blog/2019/12/06/spring-data-r2dbc-goes-ga > ͍͖ͳΓΨοπϦͱͨ͠ϏδωεϩδοΫΛ WebFlux Ͱ࣮૷͢Δͷ͸
 ͪΐͬͱϏϏͬͨɺͱ͍͏ͷ΋͋Δ • খͯ͘͞ɺ࣮֬ʹ޲͍͍ͯΔίϯϙʔωϯτ͔Βࢼͦ͏ • ։ൃͨ͠ͱ͖͸ boot 2.0.0 M1 ͱ͔ͩͬͨͷͰ...
35. νϟοτγεςϜʹٻΊΒΕΔ΋ͷ (ͦͷ3) > LINE Ϣʔβʔ͕ૹ৴ͨ͠ը૾ɾಈըɾԻ੠ͷऔಘ • ࣾ಺ͷίϯςϯπαʔόʔ(with CDN)ʹ͋Δ • ೝূ౳ͷॾʑͷࣄ৘Ͱͦͷ··ΞΧ΢ϯτΦʔφʔ͕ΞΫηε͢Δ͜ͱ ͸Ͱ͖ͳ͍ͨΊɺProxy ͢Δඞཁ͕͋ͬͨ
36. spring-webflux Ͱ΍͍ͬͯΔ Content Proxy
 (spring-webflux) Content Server ΞΧ΢ϯτΦʔφʔ http > ೝূͯ͠ࠨ͔Βӈʹ Proxy ͢Δ͚ͩɻ·͞ʹ Non blocking I/O ͕޲͍͍ͯΔ > εϨουΛઐ༗͢Δ͜ͱͳ͘ Proxy ͢Δ͜ͱ͕Ͱ͖Δ Auth http
37. ͍·ৼΓฦͬͯΈͯɺͲ͏͔ > େ͖ͳো֐΋ͳ͘1೥ؒӡ༻Ͱ͖ͨͷͰɺٕज़બఆ͸੒ޭͨ͠ͱࢥ͍ͬͯΔ > WebFlux ʹ׳Ε͖ͯͨࠓͱͳͬͯ͸ RESTful API αʔόʔ ΋ WebFlux Ͱ
 ࣮૷͢Ε͹Α͔ͬͨͳ...ͱ͸ͪΐͬͱ͚ͩࢥ͏ • Elasticsearch ͕ߴෛՙʹͳͬͯϨΠςϯγ͕ߴ·Γɺ Thread Busy ͱͳ ͬͨέʔε͕͋ͬͨ (Servlet ͋Δ͋Δ) • → Circuit breaker(resilience4j) Ͱ failing fast ͢Δ͜ͱͰରॲͯ͠Δ
38. γεςϜߏ੒/ઃܭͷ࿩ ·ͱΊ > ίϯϙʔωϯτͷ໾ׂʹԠͯ͡ spring-webflux, webmvc Λ࢖͍෼͚͍ͯΔ > spring-webflux • SSE αʔόʔ (ϦΞϧλΠϜ௨৴ɺ௕࣌ؒ઀ଓ) • Event Receiver αʔόʔ (ΞΫηεଟΊ) • Content Proxy αʔόʔ > spring-webmvc • RESTful API αʔόʔ (ίΞɾϏδωεϩδοΫ)
39. WebFlux(Reactor) ͋Ε͜Ε
40. ࠔͬͨ࿩ ͦͷ1: Netty OOME > ྫ͑͹Webflux ClientΛ༻͍ͨ
 ͜Μͳcode͕͋Δͱ͠·͢ > ͜ͷ API call Ͱ timeout ͕
 େྔʹൃੜ͢Δͱɺ
 OOME Ͱαʔόʔ͕མͪΔͱ͍͏͜ͱ͕͋ͬͨ
41. ࠔͬͨ࿩ ͦͷ1: Netty OOME > Netty ͔Β ReadTimeoutException ͕
 ൃੜ͢Δͱreactor core ಺Ͱ͜Ε͕࣮ߦ͞ΕΔ > Exceptions#propagate ͸RuntimeException 
 ͷαϒΫϥεͷ৔߹͸ͦͷ··ฦ͍ͯ͠Δɻ
 ReadTimeoutException ͸ RuntimeException ͷ
 αϒΫϥεͳͷͰ ReadTimeoutException
 ࣗ਎͕ฦΔɻ > ݁Ռͱͯ͠ɺReadTimeoutException#addSuppressed
 ͕࣮ߦ͞ΕΔ > this is ok, as re is alway a new non-singleton instance 
 ͱॻ͍ͯ͋Δ͕… ReadTimeoutException ͜Ε΋ReadTimeoutException
42. ࠔͬͨ࿩ ͦͷ1: Netty OOME > ReadTimeoutException ͕ singleton… ͩͬͨ > ͦͷ݁Ռ, ReadTimeoutException ͕อ͍࣋ͯ͠Δ suppressedExceptions(List<Throwable>) ͕ංେԽͯ͠ OOME ͕ى͖ͨ > Issue: https://github.com/netty/netty/pull/9152 • ใࠂࡁɻݱࡏ͸मਖ਼͞Ε͍ͯ·͢
43. ࠔͬͨ࿩ ͦͷ2: Blocking DNS Resolver > Reactor netty Ͱ͸ InetSocketAddress Λ࢖ͬͨ blocking DNS resolver Λ࢖༻͠ ͍ͯΔͷͰɺnetty ͷ non-blocking DNS resolver Λ࢖༻͢ΔΑ͏ʹมߋ͠·͠ ΐ͏ • ·ͩ Unstable Api Ͱ͕͢… > ۃكʹ thread ͕ hang ͯ͠͠·͏ݱ৅͕͋ͬͨͨΊௐ΂ͨ > Issue: https://github.com/reactor/reactor-netty/issues/710 • ใࠂࡁɻ0.10.x Backlog ʹೖ͍ͬͯ·͢ > σϑΥϧτΛ non blocking DNS resolver ʹ͠Α͏ɺͱ͍͏Ҋ΋ग़͍ͯ·͢ • https://github.com/reactor/reactor-netty/issues/569
44. Tips ͦͷ1: Reactor hooks with MDC > Servlet ͷӡ༻ʹ׳Ε͍ͯΔਓ͸ MDC ʹ Request ৘ใΛ͍Ε͍ͯͨͱࢥ͍·͢ • ྫ: Logback ͷ MDCInsertingServletFilter • http://logback.qos.ch/xref/ch/qos/logback/classic/helpers/ MDCInsertingServletFilter.html > Request URL ౳ΛҰॹʹϩάʹग़ྗ͢Δ͜ͱ͕Ͱ͖ΔͷͰͱͯ΋ศར
45. Tips ͦͷ1: Reactor hooks with MDC
46. Tips ͦͷ1: Reactor hooks with MDC > WebFlux Ͱ΋ࣅͨΑ͏ͳ͜ͱΛ࣮ݱͨ͠ ͍͕...Event Loop Thread ͱ͸ ૬ੑ͕ѱ͍ > ࣍ͷ3ͭͷςΫχοΫΛ࢖͏͜ͱͰՄೳ • WebFilter • Reactor Context • Hooks Event Loop Thread
47. WebFilterͰ Request ৘ใΛ Context 
 ʹೖΕ͓ͯ͘
48. > ֤छonXXXXϝιουΛϥοϓ͢ΔΑ͏ͳ Custom SubscriberΛ࣮૷͢Δ • Context͔ΒWebFilterͰొ࿥ͨ͠ request৘ใΛऔಘ͢Δ • subscriber#onXXXΛݺͼग़͢લʹ MDC#put͠ɺݺͼग़͠ޙʹ MDC#remove͢Δ
49. > ઌఔ࡞ͬͨ Custom Subscriber ͕࢖ΘΕΔΑ ͏ʹ Hooks ʹొ࿥͢Δ > (͜ͷྫͩͱSpring Configuration ʹ ॻ͍ͯΔ)
50. Tips ͦͷ1: Reactor hooks with MDC
51. Tips ͦͷ2: BlockHound > Non Blocking εϨουʹ͓͚Δ Blocking ίʔϧΛݕग़ͯ͘͠ΕΔ Java Τʔδ Σϯτ • ͋·Γ೔ຊޠͰݴٴ͞Ε͍ͯΔهࣄΛΈ͔͚ͳ͍ͷͰ঺հ͠·͢ • ίʔυϨϏϡʔͰ͸ൃݟͰ͖ͳ͍Մೳੑ΋͋ΔͷͰɺೖΕ͓ͯ͘ͱྑ͍ • ͠͹͠͹༧ظ͠ͳ͍ Blocking ίʔϧ΋͋ΔͷͰ... > https://github.com/reactor/BlockHound
52. Tips ͦͷ2: BlockHound > ґଘʹ͍ΕΔ
53. Tips ͦͷ2: BlockHound ྫ͑͹͜Μͳίʔυ͕͋Δͱ͢Δ (ྫͳͷͰத਎͸ద౰)
54. Tips ͦͷ2: BlockHound BlockHoundΛ༗ޮʹ͢Δ ςετॻ͘
55. Tips ͦͷ2: BlockHound Block ͯ͠͠·͍ͬͯΔͱ͍͏
 ࢫͷྫ֎͕ग़Δ
56. WebFlux(Reactor) ͋Ε͜Ε ·ͱΊ > ࣮ࡍʹӡ༻ͯ͠Έͯɺࠔͬͨ͜ͱΛ2ͭ঺հ͠·ͨ͠ • ౎౓ɺOSS ʹϑΟʔυόοΫ΋͍ͯ͠·͢ > WebFlux ͷӡ༻ʹ͓͚Δ logging(MDC) ͷ Tips Λ঺հ͠·ͨ͠ > Blocking ίʔϧͷݕग़Λͯ͘͠ΕΔ BlockHound ͷ঺հΛ͠·ͨ͠
57. (͓·͚) OA Dev Team
58. OA Dev 1 OA Dev 2 OA(Official Account) Dev Team > OA Dev 1 • LINE Official Account Manager , …etc > OA Dev 2 • LINEνϟοτ, …etc > We are hiring!! → https://linecorp.com/ja/career/position/1643 6ਓ 8ਓ
59. Thank You for Watching