LINE公式アカウントのチャットシステムにおけるSpringおよびWebFluxの活用事例 / Examples of using Spring and WebFlux in the chat system for LINE official accounts

by LINE Developers

Slide 1

Slide 1 text

LINEެࣜΞΧ΢ϯτͷνϟοτγεςϜ  ʹ͓͚Δ Spring ͓Αͼ WebFlux ͷ׆༻ࣄྫ > Ryosuke Hasebe / LINE corporation

Slide 2

Slide 2 text

About Me > LINEެࣜΞΧ΢ϯτ(OA: Official Account) ͷ։ൃ • LINEνϟοτ, LINE Official Account Manager, … ࠷ۙͷ࢓ࣄ Ҏલͷ࢓ࣄ > LINE Login, LINE Customer Connect, Profile+, LINE Notify, LINE BOT SDK Python,   LINE GAME PLATFORM, … ௕୩෦ ྑี (@be_hase1014) > ։ൃ4ηϯλʔ / OA Dev 2 νʔϜ > 2013೥ ೖࣾ

Slide 3

Slide 3 text

Agenda > LINE ♥ Spring > LINEνϟοτʹ͍ͭͯ > Spring ͓Αͼ WebFlux ͷ׆༻ࣄྫ • γεςϜߏ੒/ઃܭͷ࿩ • WebFlux(Reactor) ͋Ε͜Ε

Slide 4

Slide 4 text

Agenda > LINE ♥ Spring > LINEνϟοτʹ͍ͭͯ > Spring ͓Αͼ WebFlux ͷ׆༻ࣄྫ • γεςϜߏ੒/ઃܭͷ࿩ • WebFlux(Reactor) ͋Ε͜Ε

Slide 5

Slide 5 text

Spring ͨ͘͞Μ  ࢖͍ͬͯ·͢!!

Slide 6

Slide 6 text

springframework Ͱݕࡧͯ͠Έͨ > Gradle: 3,044 > Maven POM: 2,788 > → 5000 project ͘Β͍͋Δ...ʁ

Slide 7

Slide 7 text

Why Spring (boot)? > DI ίϯςφ & AOP > ๛෋ͳΠϯςάϨʔγϣϯ • MyBatis, FreeMarker/Thymeleaf, spring-boot-actuator, ... > Profiles & Configuration • ؀ڥຖͷίϯϑΟά؅ཧ͕ྑ͍ײ͡ > RESTful API Λ࣮૷͢ΔͨΊͷඞཁͳύʔπ͕ἧ͍ͬͯΔ • spring-webmvc, spring-webflux > ଞͷ RPC ϑϨʔϜϫʔΫͱ΋૊Έ߹ΘͤΔ͜ͱ͕Ͱ͖Δ • Spring with Armeria, Spring with gRPC

Slide 8

Slide 8 text

LINE ♥ Spring

Slide 9

Slide 9 text

Agenda > LINE ♥ Spring > LINEνϟοτʹ͍ͭͯ > Spring ͓Αͼ WebFlux ͷ׆༻ࣄྫ • γεςϜߏ੒/ઃܭͷ࿩ • WebFlux(Reactor) ͋Ε͜Ε

Slide 10

Slide 10 text

> اۀ/ࣄۀऀ͕։ઃ͢ΔLINEΞΧ΢ϯτ > Ϗδωε޲͚ʹ༷ʑͳػೳ͕͋Δ • ϒϩʔυΩϟετ/λʔήςΟϯά഑৴ • … etc LINEެࣜΞΧ΢ϯτͬͯʁ >date/november 20-21 >place/grand nikko tokyo

Slide 11

Slide 11 text

LINEνϟοτͬͯʁ > ໊শ͕େม΍΍͍͜͠Ͱ͕͢ɺ  LINEެࣜΞΧ΢ϯτͷӡӦऀͷͨΊͷνϟοτγεςϜͷ͜ͱͰ͢ > LINEެࣜΞΧ΢ϯτ <-> LINEϢʔβʔؒͰνϟοτ͕Ͱ͖·͢ ΞΧ΢ϯτΦʔφʔ LINEϢʔβʔ

Slide 12

Slide 12 text

Screen Shot Ƃ >november_20_first day >09:00-18:00 >november_20_first day >09:00-18:00 >date/november 20-21 >place/grand nikko tokyo

Slide 13

Slide 13 text

ن໛ײ αʔόʔ਺ (DB / storage ؚΉ) 150+ ಉ࣌઀ଓ਺ 5ສ+ ݄ؒϝοηʔδ  ૹड৴਺ 4ԯ 2019/12

Slide 14

Slide 14 text

Agenda > LINE ♥ Spring > LINEνϟοτʹ͍ͭͯ > Spring ͓Αͼ WebFlux ͷ׆༻ࣄྫ • γεςϜߏ੒/ઃܭͷ࿩ • WebFlux(Reactor) ͋Ε͜Ε

Slide 15

Slide 15 text

γεςϜߏ੒/ઃܭͷ࿩

Slide 16

Slide 16 text

νϟοτγεςϜʹٻΊΒΕΔ΋ͷ (ͦͷ1) > ϦΞϧλΠϜ௨৴ • LINE Ϣʔβʔ͔Βड৴ͨ͠ϝοηʔδ • ଞͷ ΞΧ΢ϯτΦʔφʔ ͕ߦͬͨૢ࡞ • → ͜ΕΒΛϦΞϧλΠϜʹΫϥΠΞϯτʹ൓ө͢Δඞཁ͕͋Δ

Slide 17

Slide 17 text

CGI ࣌୅ͷνϟοτγεςϜ ʊਓਓਓਓਓਓਓਓਓਓਓʊ ʼɹiframe ࣗಈϦϩʔυɹʻ ʉY^Y^Y^Y^Y^Y^Y^Y^Y^Y^Y^ʉ ౰࣌͸ͱͯ΋ָ͔͚ͬͨ͠Ͳ…

Slide 18

Slide 18 text

ϒϥ΢βʹ͓͚ΔϦΞϧλΠϜ௨৴ > ީิ • Polling • Long Polling (Comet) • Server Sent Events (SSE) • WebSocket • (ࠓͩͬͨΒɺgRPC-WEB ͱ͔ RSocket ΋ݕ౼͔ͨ͠ͳ...) > SSE Λ࠾༻ͨ͠ • ૒ํ޲௨৴͸ෆཁɻςΩετܗࣜͰ΋໰୊ͳ͍ • WebSocket ͱൺֱ͢Δͱγϯϓϧ (HTTP1.1 Chunked ϕʔε)

Slide 19

Slide 19 text

spring-webmvc Ͱ Server Sent Events > ࢖͍׳Ε͍ͯΔ spring-webmvc(servlet) Ͱ  ରԠ͞Ε͍ͯΔ͕… > SSE ͸αʔόʔͱ઀ଓͬ͠ͺͳ͠ • ैདྷͷ Servlet Ͱ͸εέʔϧ͠ͳ͍ (ӈਤ) • 1ϦΫΤετʹରͯ͠1εϨου ͳͨΊ • LINE Chat ͷݱࡏͷಉ࣌઀ଓ͸5ສ+ • 1ͭͷαʔόʔͰ500εϨου༻ҙͨ͠ͱͯ͠΋ɺ  ͜Ε͚ͩͰαʔόʔ͕100୆ඞཁʹͳͬͯ͠·͏  > Servlet 3.0 / Async Servlet Λ࢖͍ͬͯΔͷͰεϨουઐ ༗ͷ໰୊΋ͳͦ͞͏ʹݟ͑Δ͕… Thread1 Thread2 Thread3

Slide 20

Slide 20 text

spring-webmvc Ͱ Server Sent Events ͱ͸͍͑…

Slide 21

Slide 21 text

ͱ͸͍͑… > ͚͖ͬΐ͘ Servlet ͰͷϦΫΤετͷड৴/ૹ৴ ͸ Input/OutputStream Λ༻͍ͨ Blocking I/O • https://www.infoq.com/presentations/servlet- reactive-stack/ > → Spring5 ͔Β௥Ճ͞Εͨ WebFlux Λ࠾༻͢ Δ͜ͱʹ͠·ͨ͠ spring-webmvc Servlet API Tomcat, Jetty, … Blocking I/O

Slide 22

Slide 22 text

WebFlux ͱ͸ > Spring5 ͔Βొ৔ͨ͠ Non Blocking ͳ Web ϑϨʔϜϫʔΫ spring-webmvc Servlet API Tomcat, Jetty, … Blocking I/O spring-webflux Spring Web API  (Reactor, Reactive Streams) Tomcat, Jetty, …  (Servlet 3.1) Non Blocking I/O Netty

Slide 23

Slide 23 text

WebFlux ͱ͸ > ֤ϦΫΤετΛ Non Blocking Ͱॲཧ͢Δ͜ͱͰɺ1ͭͷ Event Loop  Thread Ͱෳ਺ϦΫΤετΛࡹ͚Δ = গͳ͍αʔόʔ਺Ͱࡹ͚Δ > Block ͢Δίʔυ͸ઈରʹॻ͍ͯ͸͍͚ͳ͍ɻઈରʹμϝ Thread1 Thread2 Thread3 Event Loop Thread

Slide 24

Slide 24 text

αʔόʔ1୆͋ͨΓͷಉ࣌઀ଓ਺ 1୆͋ͨΓͷಉ࣌઀ଓ਺͸2500ɻ 2~3ഒ͸͍͚Δ͚Ͳɺ༨༟Λ΋ͬͯɻ (ݱࡏ͸20୆࢖͍ͬͯ·͢)

Slide 25

Slide 25 text

ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ Talk Server Event  Receiver Kafka Event Processor Redis Cluster (PubSub) Redis Cluster SSE

Slide 26

Slide 26 text

ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ Talk Server Event  Receiver Kafka Event Processor > ϝοηʔδड৴ Event ͕ಧ͘ > (ଞʹ΋طಡ, ༑ͩͪ௥Ճ, ...౳ͷ Event ΋) > ࣮ࡍ͸ؒʹଞͷίϯϙʔωϯτ͕͋Δ Redis Cluster (PubSub) Redis Cluster SSE

Slide 27

Slide 27 text

ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ Talk Server Event  Receiver Kafka Event Processor Redis Cluster (PubSub) Redis Cluster SSE > Event Receiver ͸ड͚औͬͨΒ Kafka ʹॻ͘ > Ұ౓ Kafka ʹॻ͍͓ͯ͘͜ͱͰɺEvent ͷ࠶ར༻΍  ϦτϥΠΛ͠΍͘͢͢Δ > Event Receiver ΋ WebFlux Λ࢖࣮ͬͯ૷͍ͯ͠Δ

Slide 28

Slide 28 text

ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ Talk Server Event  Receiver Kafka Event Processor Redis Cluster (PubSub) Redis Cluster SSE > Event Processor Ͱ Kafka ͔Β Event ΛಡΉ > Redis ʹ Publish ͢Δ > SSE server ͕ Subscribe ͢Δ

Slide 29

Slide 29 text

ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ Talk Server Event  Receiver Kafka Event Processor Redis Cluster (PubSub) Redis Cluster SSE > ࠶઀ଓΛαϙʔτ͢ΔͨΊʹɺPublish ͱಉ࣌ʹ  Ϧετͱͯ͠΋อ͓࣋ͯ͘͠ > ࠶઀ଓʹ࠷ޙʹऔಘͨ͠ Event ID(Last-Event-ID header) ͕౉͞ΕΔͷͰɺະऔಘͷ Event Λฦ͢

Slide 30

Slide 30 text

ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ Talk Server Event  Receiver Kafka Event Processor Redis Cluster (Stream) SSE > Redis5 ͔Β௥Ճ͞Εͨ Redis Streams Λ  ࢖͑͹΋ͬͱ؆୯ʹͳΓͦ͏ > ※ զʑͷ Redis ͸ 4ܥ...

Slide 31

Slide 31 text

Subscribe Redis as Flux (Reactive lettuce) ௕࣌ؒ, Event ͕ͳ͍ͱ઀ଓ͕  ੾ΕΔͷͰ PING ΛૹΔ 2ͭͷ Flux Λ merge

Slide 32

Slide 32 text

νϟοτγεςϜʹٻΊΒΕΔ΋ͷ (ͦͷ2) > ϝοηʔδͷૹ৴ > աڈϝοηʔδͷऔಘ > ϝοηʔδͷݕࡧ > ͦͷଞɺ֤छσʔλͷ CRUDૢ࡞ → Ϣʔβʔૢ࡞ىҼͷɺίΞͱͳΔϏδωεϩδοΫ

Slide 33

Slide 33 text

spring-webmvc Ͱ΍͍ͬͯΔ MySQL (Sharding) RESTful API  (spring-webmvc) Talk Server Message  Gateway ΞΧ΢ϯτΦʔφʔ Elasticsearch HBase http jdbc http Redis Cluster

Slide 34

Slide 34 text

Why spring-webmvc? > JDBC... (Blocking I/O) • ͓...? • Spring Data R2DBC goes GA • https://spring.io/blog/2019/12/06/spring-data-r2dbc-goes-ga > ͍͖ͳΓΨοπϦͱͨ͠ϏδωεϩδοΫΛ WebFlux Ͱ࣮૷͢Δͷ͸  ͪΐͬͱϏϏͬͨɺͱ͍͏ͷ΋͋Δ • খͯ͘͞ɺ࣮֬ʹ޲͍͍ͯΔίϯϙʔωϯτ͔Βࢼͦ͏ • ։ൃͨ͠ͱ͖͸ boot 2.0.0 M1 ͱ͔ͩͬͨͷͰ...

Slide 35

Slide 35 text

νϟοτγεςϜʹٻΊΒΕΔ΋ͷ (ͦͷ3) > LINE Ϣʔβʔ͕ૹ৴ͨ͠ը૾ɾಈըɾԻ੠ͷऔಘ • ࣾ಺ͷίϯςϯπαʔόʔ(with CDN)ʹ͋Δ • ೝূ౳ͷॾʑͷࣄ৘Ͱͦͷ··ΞΧ΢ϯτΦʔφʔ͕ΞΫηε͢Δ͜ͱ ͸Ͱ͖ͳ͍ͨΊɺProxy ͢Δඞཁ͕͋ͬͨ

Slide 36

Slide 36 text

spring-webflux Ͱ΍͍ͬͯΔ Content Proxy  (spring-webflux) Content Server ΞΧ΢ϯτΦʔφʔ http > ೝূͯ͠ࠨ͔Βӈʹ Proxy ͢Δ͚ͩɻ·͞ʹ Non blocking I/O ͕޲͍͍ͯΔ > εϨουΛઐ༗͢Δ͜ͱͳ͘ Proxy ͢Δ͜ͱ͕Ͱ͖Δ Auth http

Slide 37

Slide 37 text

͍·ৼΓฦͬͯΈͯɺͲ͏͔ > େ͖ͳো֐΋ͳ͘1೥ؒӡ༻Ͱ͖ͨͷͰɺٕज़બఆ͸੒ޭͨ͠ͱࢥ͍ͬͯΔ > WebFlux ʹ׳Ε͖ͯͨࠓͱͳͬͯ͸ RESTful API αʔόʔ ΋ WebFlux Ͱ  ࣮૷͢Ε͹Α͔ͬͨͳ...ͱ͸ͪΐͬͱ͚ͩࢥ͏ • Elasticsearch ͕ߴෛՙʹͳͬͯϨΠςϯγ͕ߴ·Γɺ Thread Busy ͱͳ ͬͨέʔε͕͋ͬͨ (Servlet ͋Δ͋Δ) • → Circuit breaker(resilience4j) Ͱ failing fast ͢Δ͜ͱͰରॲͯ͠Δ

Slide 38

Slide 38 text

γεςϜߏ੒/ઃܭͷ࿩ ·ͱΊ > ίϯϙʔωϯτͷ໾ׂʹԠͯ͡ spring-webflux, webmvc Λ࢖͍෼͚͍ͯΔ > spring-webflux • SSE αʔόʔ (ϦΞϧλΠϜ௨৴ɺ௕࣌ؒ઀ଓ) • Event Receiver αʔόʔ (ΞΫηεଟΊ) • Content Proxy αʔόʔ > spring-webmvc • RESTful API αʔόʔ (ίΞɾϏδωεϩδοΫ)

Slide 39

Slide 39 text

WebFlux(Reactor) ͋Ε͜Ε

Slide 40

Slide 40 text

ࠔͬͨ࿩ ͦͷ1: Netty OOME > ྫ͑͹Webflux ClientΛ༻͍ͨ  ͜Μͳcode͕͋Δͱ͠·͢ > ͜ͷ API call Ͱ timeout ͕  େྔʹൃੜ͢Δͱɺ  OOME Ͱαʔόʔ͕མͪΔͱ͍͏͜ͱ͕͋ͬͨ

Slide 41

Slide 41 text

ࠔͬͨ࿩ ͦͷ1: Netty OOME > Netty ͔Β ReadTimeoutException ͕  ൃੜ͢Δͱreactor core ಺Ͱ͜Ε͕࣮ߦ͞ΕΔ > Exceptions#propagate ͸RuntimeException   ͷαϒΫϥεͷ৔߹͸ͦͷ··ฦ͍ͯ͠Δɻ  ReadTimeoutException ͸ RuntimeException ͷ  αϒΫϥεͳͷͰ ReadTimeoutException  ࣗ਎͕ฦΔɻ > ݁Ռͱͯ͠ɺReadTimeoutException#addSuppressed  ͕࣮ߦ͞ΕΔ > this is ok, as re is alway a new non-singleton instance   ͱॻ͍ͯ͋Δ͕… ReadTimeoutException ͜Ε΋ReadTimeoutException

Slide 42

Slide 42 text

ࠔͬͨ࿩ ͦͷ1: Netty OOME > ReadTimeoutException ͕ singleton… ͩͬͨ > ͦͷ݁Ռ, ReadTimeoutException ͕อ͍࣋ͯ͠Δ suppressedExceptions(List) ͕ංେԽͯ͠ OOME ͕ى͖ͨ > Issue: https://github.com/netty/netty/pull/9152 • ใࠂࡁɻݱࡏ͸मਖ਼͞Ε͍ͯ·͢

Slide 43

Slide 43 text

ࠔͬͨ࿩ ͦͷ2: Blocking DNS Resolver > Reactor netty Ͱ͸ InetSocketAddress Λ࢖ͬͨ blocking DNS resolver Λ࢖༻͠ ͍ͯΔͷͰɺnetty ͷ non-blocking DNS resolver Λ࢖༻͢ΔΑ͏ʹมߋ͠·͠ ΐ͏ • ·ͩ Unstable Api Ͱ͕͢… > ۃكʹ thread ͕ hang ͯ͠͠·͏ݱ৅͕͋ͬͨͨΊௐ΂ͨ > Issue: https://github.com/reactor/reactor-netty/issues/710 • ใࠂࡁɻ0.10.x Backlog ʹೖ͍ͬͯ·͢ > σϑΥϧτΛ non blocking DNS resolver ʹ͠Α͏ɺͱ͍͏Ҋ΋ग़͍ͯ·͢ • https://github.com/reactor/reactor-netty/issues/569

Slide 44

Slide 44 text

Tips ͦͷ1: Reactor hooks with MDC > Servlet ͷӡ༻ʹ׳Ε͍ͯΔਓ͸ MDC ʹ Request ৘ใΛ͍Ε͍ͯͨͱࢥ͍·͢ • ྫ: Logback ͷ MDCInsertingServletFilter • http://logback.qos.ch/xref/ch/qos/logback/classic/helpers/ MDCInsertingServletFilter.html > Request URL ౳ΛҰॹʹϩάʹग़ྗ͢Δ͜ͱ͕Ͱ͖ΔͷͰͱͯ΋ศར

Slide 45

Slide 45 text

Tips ͦͷ1: Reactor hooks with MDC

Slide 46

Slide 46 text

Tips ͦͷ1: Reactor hooks with MDC > WebFlux Ͱ΋ࣅͨΑ͏ͳ͜ͱΛ࣮ݱͨ͠ ͍͕...Event Loop Thread ͱ͸ ૬ੑ͕ѱ͍ > ࣍ͷ3ͭͷςΫχοΫΛ࢖͏͜ͱͰՄೳ • WebFilter • Reactor Context • Hooks Event Loop Thread

Slide 47

Slide 47 text

WebFilterͰ Request ৘ใΛ Context   ʹೖΕ͓ͯ͘

Slide 48

Slide 48 text

> ֤छonXXXXϝιουΛϥοϓ͢ΔΑ͏ͳ Custom SubscriberΛ࣮૷͢Δ • Context͔ΒWebFilterͰొ࿥ͨ͠ request৘ใΛऔಘ͢Δ • subscriber#onXXXΛݺͼग़͢લʹ MDC#put͠ɺݺͼग़͠ޙʹ MDC#remove͢Δ

Slide 49

Slide 49 text

> ઌఔ࡞ͬͨ Custom Subscriber ͕࢖ΘΕΔΑ ͏ʹ Hooks ʹొ࿥͢Δ > (͜ͷྫͩͱSpring Configuration ʹ ॻ͍ͯΔ)

Slide 50

Slide 50 text

Tips ͦͷ1: Reactor hooks with MDC

Slide 51

Slide 51 text

Tips ͦͷ2: BlockHound > Non Blocking εϨουʹ͓͚Δ Blocking ίʔϧΛݕग़ͯ͘͠ΕΔ Java Τʔδ Σϯτ • ͋·Γ೔ຊޠͰݴٴ͞Ε͍ͯΔهࣄΛΈ͔͚ͳ͍ͷͰ঺հ͠·͢ • ίʔυϨϏϡʔͰ͸ൃݟͰ͖ͳ͍Մೳੑ΋͋ΔͷͰɺೖΕ͓ͯ͘ͱྑ͍ • ͠͹͠͹༧ظ͠ͳ͍ Blocking ίʔϧ΋͋ΔͷͰ... > https://github.com/reactor/BlockHound

Slide 52

Slide 52 text

Tips ͦͷ2: BlockHound > ґଘʹ͍ΕΔ

Slide 53

Slide 53 text

Tips ͦͷ2: BlockHound ྫ͑͹͜Μͳίʔυ͕͋Δͱ͢Δ (ྫͳͷͰத਎͸ద౰)

Slide 54

Slide 54 text

Tips ͦͷ2: BlockHound BlockHoundΛ༗ޮʹ͢Δ ςετॻ͘

Slide 55

Slide 55 text

Tips ͦͷ2: BlockHound Block ͯ͠͠·͍ͬͯΔͱ͍͏  ࢫͷྫ֎͕ग़Δ

Slide 56

Slide 56 text

WebFlux(Reactor) ͋Ε͜Ε ·ͱΊ > ࣮ࡍʹӡ༻ͯ͠Έͯɺࠔͬͨ͜ͱΛ2ͭ঺հ͠·ͨ͠ • ౎౓ɺOSS ʹϑΟʔυόοΫ΋͍ͯ͠·͢ > WebFlux ͷӡ༻ʹ͓͚Δ logging(MDC) ͷ Tips Λ঺հ͠·ͨ͠ > Blocking ίʔϧͷݕग़Λͯ͘͠ΕΔ BlockHound ͷ঺հΛ͠·ͨ͠

Slide 57

Slide 57 text

(͓·͚) OA Dev Team

Slide 58

Slide 58 text

OA Dev 1 OA Dev 2 OA(Official Account) Dev Team > OA Dev 1 • LINE Official Account Manager , …etc > OA Dev 2 • LINEνϟοτ, …etc > We are hiring!! → https://linecorp.com/ja/career/position/1643 6ਓ 8ਓ

Slide 59

Slide 59 text

Thank You for Watching