LINE公式アカウントのチャットシステムにおけるSpringおよびWebFluxの活用事例 / Examples of using Spring and WebFlux in the chat system for LINE official accounts

LINE株式会社 長谷部 良輔
SPRING FEST ’19での発表資料です。
https://springfest2019.springframework.jp/

Transcript

1. LINEެࣜΞΧ΢ϯτͷνϟοτγεςϜ

   ʹ͓͚Δ Spring ͓Αͼ WebFlux ͷ׆༻ࣄྫ
   > Ryosuke Hasebe / LINE corporation
   

   View Slide

2. About Me
   > LINEެࣜΞΧ΢ϯτ(OA: Official Account) ͷ։ൃ
   • LINEνϟοτ, LINE Official Account Manager, …
   ࠷ۙͷ࢓ࣄ
   Ҏલͷ࢓ࣄ
   > LINE Login, LINE Customer Connect, Profile+, LINE Notify, LINE BOT SDK Python, 

   LINE GAME PLATFORM, …
   ௕୩෦ ྑี (@be_hase1014)
   > ։ൃ4ηϯλʔ / OA Dev 2 νʔϜ
   > 2013೥ ೖࣾ
   

   View Slide

3. Agenda
   > LINE ♥ Spring
   > LINEνϟοτʹ͍ͭͯ
   > Spring ͓Αͼ WebFlux ͷ׆༻ࣄྫ
   • γεςϜߏ੒/ઃܭͷ࿩
   • WebFlux(Reactor) ͋Ε͜Ε
   

   View Slide

4. Agenda
   > LINE ♥ Spring
   > LINEνϟοτʹ͍ͭͯ
   > Spring ͓Αͼ WebFlux ͷ׆༻ࣄྫ
   • γεςϜߏ੒/ઃܭͷ࿩
   • WebFlux(Reactor) ͋Ε͜Ε
   

   View Slide

5. Spring ͨ͘͞Μ

   ࢖͍ͬͯ·͢!!
   

   View Slide

6. springframework Ͱݕࡧͯ͠Έͨ
   > Gradle: 3,044
   > Maven POM: 2,788
   > → 5000 project ͘Β͍͋Δ...ʁ
   

   View Slide

7. Why Spring (boot)?
   > DI ίϯςφ & AOP
   > ๛෋ͳΠϯςάϨʔγϣϯ
   • MyBatis, FreeMarker/Thymeleaf, spring-boot-actuator, ...
   > Profiles & Configuration
   • ؀ڥຖͷίϯϑΟά؅ཧ͕ྑ͍ײ͡
   > RESTful API Λ࣮૷͢ΔͨΊͷඞཁͳύʔπ͕ἧ͍ͬͯΔ
   • spring-webmvc, spring-webflux
   > ଞͷ RPC ϑϨʔϜϫʔΫͱ΋૊Έ߹ΘͤΔ͜ͱ͕Ͱ͖Δ
   • Spring with Armeria, Spring with gRPC
   

   View Slide

8. LINE ♥ Spring
   

   View Slide

9. Agenda
   > LINE ♥ Spring
   > LINEνϟοτʹ͍ͭͯ
   > Spring ͓Αͼ WebFlux ͷ׆༻ࣄྫ
   • γεςϜߏ੒/ઃܭͷ࿩
   • WebFlux(Reactor) ͋Ε͜Ε
   

   View Slide

10. > اۀ/ࣄۀऀ͕։ઃ͢ΔLINEΞΧ΢ϯτ
    > Ϗδωε޲͚ʹ༷ʑͳػೳ͕͋Δ
    • ϒϩʔυΩϟετ/λʔήςΟϯά഑৴
    • … etc
    LINEެࣜΞΧ΢ϯτͬͯʁ
    >date/november 20-21
    >place/grand nikko tokyo
    

    View Slide

11. LINEνϟοτͬͯʁ
    > ໊শ͕େม΍΍͍͜͠Ͱ͕͢ɺ

    LINEެࣜΞΧ΢ϯτͷӡӦऀͷͨΊͷνϟοτγεςϜͷ͜ͱͰ͢
    > LINEެࣜΞΧ΢ϯτ LINEϢʔβʔؒͰνϟοτ͕Ͱ͖·͢
    ΞΧ΢ϯτΦʔφʔ LINEϢʔβʔ
    

    View Slide

12. Screen Shot
    Ƃ
    >november_20_first day
    >09:00-18:00
    >november_20_first day
    >09:00-18:00
    >date/november 20-21
    >place/grand nikko tokyo
    

    View Slide

13. ن໛ײ
    αʔόʔ਺
    (DB / storage ؚΉ)
    150+
    ಉ࣌઀ଓ਺
    5ສ+
    ݄ؒϝοηʔδ

    ૹड৴਺
    4ԯ
    2019/12
    

    View Slide

14. Agenda
    > LINE ♥ Spring
    > LINEνϟοτʹ͍ͭͯ
    > Spring ͓Αͼ WebFlux ͷ׆༻ࣄྫ
    • γεςϜߏ੒/ઃܭͷ࿩
    • WebFlux(Reactor) ͋Ε͜Ε
    

    View Slide

15. γεςϜߏ੒/ઃܭͷ࿩
    

    View Slide

16. νϟοτγεςϜʹٻΊΒΕΔ΋ͷ (ͦͷ1)
    > ϦΞϧλΠϜ௨৴
    • LINE Ϣʔβʔ͔Βड৴ͨ͠ϝοηʔδ
    • ଞͷ ΞΧ΢ϯτΦʔφʔ ͕ߦͬͨૢ࡞
    • → ͜ΕΒΛϦΞϧλΠϜʹΫϥΠΞϯτʹ൓ө͢Δඞཁ͕͋Δ
    

    View Slide

17. CGI ࣌୅ͷνϟοτγεςϜ
    ʊਓਓਓਓਓਓਓਓਓਓਓʊ
    ʼɹiframe ࣗಈϦϩʔυɹʻ
    ʉY^Y^Y^Y^Y^Y^Y^Y^Y^Y^Y^ʉ
    ౰࣌͸ͱͯ΋ָ͔͚ͬͨ͠Ͳ…
    

    View Slide

18. ϒϥ΢βʹ͓͚ΔϦΞϧλΠϜ௨৴
    > ީิ
    • Polling
    • Long Polling (Comet)
    • Server Sent Events (SSE)
    • WebSocket
    • (ࠓͩͬͨΒɺgRPC-WEB ͱ͔ RSocket ΋ݕ౼͔ͨ͠ͳ...)
    > SSE Λ࠾༻ͨ͠
    • ૒ํ޲௨৴͸ෆཁɻςΩετܗࣜͰ΋໰୊ͳ͍
    • WebSocket ͱൺֱ͢Δͱγϯϓϧ (HTTP1.1 Chunked ϕʔε)
    

    View Slide

19. spring-webmvc Ͱ Server Sent Events
    > ࢖͍׳Ε͍ͯΔ spring-webmvc(servlet) Ͱ

    ରԠ͞Ε͍ͯΔ͕…
    > SSE ͸αʔόʔͱ઀ଓͬ͠ͺͳ͠
    • ैདྷͷ Servlet Ͱ͸εέʔϧ͠ͳ͍ (ӈਤ)
    • 1ϦΫΤετʹରͯ͠1εϨου ͳͨΊ
    • LINE Chat ͷݱࡏͷಉ࣌઀ଓ͸5ສ+
    • 1ͭͷαʔόʔͰ500εϨου༻ҙͨ͠ͱͯ͠΋ɺ

    ͜Ε͚ͩͰαʔόʔ͕100୆ඞཁʹͳͬͯ͠·͏

    > Servlet 3.0 / Async Servlet Λ࢖͍ͬͯΔͷͰεϨουઐ
    ༗ͷ໰୊΋ͳͦ͞͏ʹݟ͑Δ͕…
    Thread1
    Thread2
    Thread3
    

    View Slide

20. spring-webmvc Ͱ Server Sent Events
    ͱ͸͍͑…
    

    View Slide

21. ͱ͸͍͑…
    > ͚͖ͬΐ͘ Servlet ͰͷϦΫΤετͷड৴/ૹ৴
    ͸ Input/OutputStream Λ༻͍ͨ Blocking I/O
    • https://www.infoq.com/presentations/servlet-
    reactive-stack/
    > → Spring5 ͔Β௥Ճ͞Εͨ WebFlux Λ࠾༻͢
    Δ͜ͱʹ͠·ͨ͠
    spring-webmvc
    Servlet API
    Tomcat, Jetty, …
    Blocking I/O
    

    View Slide

22. WebFlux ͱ͸
    > Spring5 ͔Βొ৔ͨ͠ Non Blocking ͳ Web ϑϨʔϜϫʔΫ
    spring-webmvc
    Servlet API
    Tomcat, Jetty, …
    Blocking I/O
    spring-webflux
    Spring Web API

    (Reactor, Reactive Streams)
    Tomcat, Jetty, …

    (Servlet 3.1)
    Non Blocking I/O
    Netty
    

    View Slide

23. WebFlux ͱ͸
    > ֤ϦΫΤετΛ Non Blocking Ͱॲཧ͢Δ͜ͱͰɺ1ͭͷ Event Loop

    Thread Ͱෳ਺ϦΫΤετΛࡹ͚Δ = গͳ͍αʔόʔ਺Ͱࡹ͚Δ
    > Block ͢Δίʔυ͸ઈରʹॻ͍ͯ͸͍͚ͳ͍ɻઈରʹμϝ
    Thread1
    Thread2
    Thread3
    Event Loop Thread
    

    View Slide

24. αʔόʔ1୆͋ͨΓͷಉ࣌઀ଓ਺
    1୆͋ͨΓͷಉ࣌઀ଓ਺͸2500ɻ
    2~3ഒ͸͍͚Δ͚Ͳɺ༨༟Λ΋ͬͯɻ
    (ݱࡏ͸20୆࢖͍ͬͯ·͢)
    

    View Slide

25. ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ
    Talk
    Server
    Event

    Receiver
    Kafka
    Event
    Processor
    Redis Cluster
    (PubSub)
    Redis Cluster
    SSE
    

    View Slide

26. ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ
    Talk
    Server
    Event

    Receiver
    Kafka
    Event
    Processor
    > ϝοηʔδड৴ Event ͕ಧ͘
    > (ଞʹ΋طಡ, ༑ͩͪ௥Ճ, ...౳ͷ Event ΋)
    > ࣮ࡍ͸ؒʹଞͷίϯϙʔωϯτ͕͋Δ
    Redis Cluster
    (PubSub)
    Redis Cluster
    SSE
    

    View Slide

27. ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ
    Talk
    Server
    Event

    Receiver
    Kafka
    Event
    Processor
    Redis Cluster
    (PubSub)
    Redis Cluster
    SSE
    > Event Receiver ͸ड͚औͬͨΒ Kafka ʹॻ͘
    > Ұ౓ Kafka ʹॻ͍͓ͯ͘͜ͱͰɺEvent ͷ࠶ར༻΍

    ϦτϥΠΛ͠΍͘͢͢Δ
    > Event Receiver ΋ WebFlux Λ࢖࣮ͬͯ૷͍ͯ͠Δ
    

    View Slide

28. ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ
    Talk
    Server
    Event

    Receiver
    Kafka
    Event
    Processor
    Redis Cluster
    (PubSub)
    Redis Cluster
    SSE
    > Event Processor Ͱ Kafka ͔Β Event ΛಡΉ
    > Redis ʹ Publish ͢Δ
    > SSE server ͕ Subscribe ͢Δ
    

    View Slide

29. ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ
    Talk
    Server
    Event

    Receiver
    Kafka
    Event
    Processor
    Redis Cluster
    (PubSub)
    Redis Cluster
    SSE
    > ࠶઀ଓΛαϙʔτ͢ΔͨΊʹɺPublish ͱಉ࣌ʹ

    Ϧετͱͯ͠΋อ͓࣋ͯ͘͠
    > ࠶઀ଓʹ࠷ޙʹऔಘͨ͠ Event ID(Last-Event-ID
    header) ͕౉͞ΕΔͷͰɺະऔಘͷ Event Λฦ͢
    

    View Slide

30. ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ
    Talk
    Server
    Event

    Receiver
    Kafka
    Event
    Processor
    Redis Cluster
    (Stream)
    SSE
    > Redis5 ͔Β௥Ճ͞Εͨ Redis Streams Λ

    ࢖͑͹΋ͬͱ؆୯ʹͳΓͦ͏
    > ※ զʑͷ Redis ͸ 4ܥ...
    

    View Slide

31. Subscribe Redis as Flux
    (Reactive lettuce)
    ௕࣌ؒ, Event ͕ͳ͍ͱ઀ଓ͕

    ੾ΕΔͷͰ PING ΛૹΔ
    2ͭͷ Flux Λ merge
    

    View Slide

32. νϟοτγεςϜʹٻΊΒΕΔ΋ͷ (ͦͷ2)
    > ϝοηʔδͷૹ৴
    > աڈϝοηʔδͷऔಘ
    > ϝοηʔδͷݕࡧ
    > ͦͷଞɺ֤छσʔλͷ CRUDૢ࡞
    → Ϣʔβʔૢ࡞ىҼͷɺίΞͱͳΔϏδωεϩδοΫ
    

    View Slide

33. spring-webmvc Ͱ΍͍ͬͯΔ
    MySQL
    (Sharding)
    RESTful API

    (spring-webmvc)
    Talk
    Server
    Message

    Gateway
    ΞΧ΢ϯτΦʔφʔ
    Elasticsearch
    HBase
    http
    jdbc
    http
    Redis Cluster
    

    View Slide

34. Why spring-webmvc?
    > JDBC... (Blocking I/O)
    • ͓...?
    • Spring Data R2DBC goes GA
    • https://spring.io/blog/2019/12/06/spring-data-r2dbc-goes-ga
    > ͍͖ͳΓΨοπϦͱͨ͠ϏδωεϩδοΫΛ WebFlux Ͱ࣮૷͢Δͷ͸

    ͪΐͬͱϏϏͬͨɺͱ͍͏ͷ΋͋Δ
    • খͯ͘͞ɺ࣮֬ʹ޲͍͍ͯΔίϯϙʔωϯτ͔Βࢼͦ͏
    • ։ൃͨ͠ͱ͖͸ boot 2.0.0 M1 ͱ͔ͩͬͨͷͰ...
    

    View Slide

35. νϟοτγεςϜʹٻΊΒΕΔ΋ͷ (ͦͷ3)
    > LINE Ϣʔβʔ͕ૹ৴ͨ͠ը૾ɾಈըɾԻ੠ͷऔಘ
    • ࣾ಺ͷίϯςϯπαʔόʔ(with CDN)ʹ͋Δ
    • ೝূ౳ͷॾʑͷࣄ৘Ͱͦͷ··ΞΧ΢ϯτΦʔφʔ͕ΞΫηε͢Δ͜ͱ
    ͸Ͱ͖ͳ͍ͨΊɺProxy ͢Δඞཁ͕͋ͬͨ
    

    View Slide

36. spring-webflux Ͱ΍͍ͬͯΔ
    Content Proxy

    (spring-webflux)
    Content Server
    ΞΧ΢ϯτΦʔφʔ
    http
    > ೝূͯ͠ࠨ͔Βӈʹ Proxy ͢Δ͚ͩɻ·͞ʹ Non blocking I/O ͕޲͍͍ͯΔ
    > εϨουΛઐ༗͢Δ͜ͱͳ͘ Proxy ͢Δ͜ͱ͕Ͱ͖Δ
    Auth
    http
    

    View Slide

37. ͍·ৼΓฦͬͯΈͯɺͲ͏͔
    > େ͖ͳো֐΋ͳ͘1೥ؒӡ༻Ͱ͖ͨͷͰɺٕज़બఆ͸੒ޭͨ͠ͱࢥ͍ͬͯΔ
    > WebFlux ʹ׳Ε͖ͯͨࠓͱͳͬͯ͸ RESTful API αʔόʔ ΋ WebFlux Ͱ

    ࣮૷͢Ε͹Α͔ͬͨͳ...ͱ͸ͪΐͬͱ͚ͩࢥ͏
    • Elasticsearch ͕ߴෛՙʹͳͬͯϨΠςϯγ͕ߴ·Γɺ Thread Busy ͱͳ
    ͬͨέʔε͕͋ͬͨ (Servlet ͋Δ͋Δ)
    • → Circuit breaker(resilience4j) Ͱ failing fast ͢Δ͜ͱͰରॲͯ͠Δ
    

    View Slide

38. γεςϜߏ੒/ઃܭͷ࿩ ·ͱΊ
    > ίϯϙʔωϯτͷ໾ׂʹԠͯ͡ spring-webflux, webmvc Λ࢖͍෼͚͍ͯΔ
    > spring-webflux
    • SSE αʔόʔ (ϦΞϧλΠϜ௨৴ɺ௕࣌ؒ઀ଓ)
    • Event Receiver αʔόʔ (ΞΫηεଟΊ)
    • Content Proxy αʔόʔ
    > spring-webmvc
    • RESTful API αʔόʔ (ίΞɾϏδωεϩδοΫ)
    

    View Slide

39. WebFlux(Reactor) ͋Ε͜Ε
    

    View Slide

40. ࠔͬͨ࿩ ͦͷ1: Netty OOME
    > ྫ͑͹Webflux ClientΛ༻͍ͨ

    ͜Μͳcode͕͋Δͱ͠·͢
    > ͜ͷ API call Ͱ timeout ͕

    େྔʹൃੜ͢Δͱɺ

    OOME Ͱαʔόʔ͕མͪΔͱ͍͏͜ͱ͕͋ͬͨ
    

    View Slide

41. ࠔͬͨ࿩ ͦͷ1: Netty OOME
    > Netty ͔Β ReadTimeoutException ͕

    ൃੜ͢Δͱreactor core ಺Ͱ͜Ε͕࣮ߦ͞ΕΔ
    > Exceptions#propagate ͸RuntimeException 

    ͷαϒΫϥεͷ৔߹͸ͦͷ··ฦ͍ͯ͠Δɻ

    ReadTimeoutException ͸ RuntimeException ͷ

    αϒΫϥεͳͷͰ ReadTimeoutException

    ࣗ਎͕ฦΔɻ
    > ݁Ռͱͯ͠ɺReadTimeoutException#addSuppressed

    ͕࣮ߦ͞ΕΔ
    > this is ok, as re is alway a new non-singleton instance 

    ͱॻ͍ͯ͋Δ͕…
    ReadTimeoutException
    ͜Ε΋ReadTimeoutException
    

    View Slide

42. ࠔͬͨ࿩ ͦͷ1: Netty OOME
    > ReadTimeoutException ͕ singleton… ͩͬͨ
    > ͦͷ݁Ռ, ReadTimeoutException ͕อ͍࣋ͯ͠Δ
    suppressedExceptions(List) ͕ංେԽͯ͠ OOME ͕ى͖ͨ
    > Issue: https://github.com/netty/netty/pull/9152
    • ใࠂࡁɻݱࡏ͸मਖ਼͞Ε͍ͯ·͢
    

    View Slide

43. ࠔͬͨ࿩ ͦͷ2: Blocking DNS Resolver
    > Reactor netty Ͱ͸ InetSocketAddress Λ࢖ͬͨ blocking DNS resolver Λ࢖༻͠
    ͍ͯΔͷͰɺnetty ͷ non-blocking DNS resolver Λ࢖༻͢ΔΑ͏ʹมߋ͠·͠
    ΐ͏
    • ·ͩ Unstable Api Ͱ͕͢…
    > ۃكʹ thread ͕ hang ͯ͠͠·͏ݱ৅͕͋ͬͨͨΊௐ΂ͨ
    > Issue: https://github.com/reactor/reactor-netty/issues/710
    • ใࠂࡁɻ0.10.x Backlog ʹೖ͍ͬͯ·͢
    > σϑΥϧτΛ non blocking DNS resolver ʹ͠Α͏ɺͱ͍͏Ҋ΋ग़͍ͯ·͢
    • https://github.com/reactor/reactor-netty/issues/569
    

    View Slide

44. Tips ͦͷ1: Reactor hooks with MDC
    > Servlet ͷӡ༻ʹ׳Ε͍ͯΔਓ͸ MDC ʹ Request ৘ใΛ͍Ε͍ͯͨͱࢥ͍·͢
    • ྫ: Logback ͷ MDCInsertingServletFilter
    • http://logback.qos.ch/xref/ch/qos/logback/classic/helpers/
    MDCInsertingServletFilter.html
    > Request URL ౳ΛҰॹʹϩάʹग़ྗ͢Δ͜ͱ͕Ͱ͖ΔͷͰͱͯ΋ศར
    

    View Slide

45. Tips ͦͷ1: Reactor hooks with MDC
    

    View Slide

46. Tips ͦͷ1: Reactor hooks with MDC
    > WebFlux Ͱ΋ࣅͨΑ͏ͳ͜ͱΛ࣮ݱͨ͠
    ͍͕...Event Loop Thread ͱ͸ ૬ੑ͕ѱ͍
    > ࣍ͷ3ͭͷςΫχοΫΛ࢖͏͜ͱͰՄೳ
    • WebFilter
    • Reactor Context
    • Hooks Event Loop Thread
    

    View Slide

47. WebFilterͰ Request ৘ใΛ Context 

    ʹೖΕ͓ͯ͘
    

    View Slide

48. > ֤छonXXXXϝιουΛϥοϓ͢ΔΑ͏ͳ
    Custom SubscriberΛ࣮૷͢Δ
    • Context͔ΒWebFilterͰొ࿥ͨ͠
    request৘ใΛऔಘ͢Δ
    • subscriber#onXXXΛݺͼग़͢લʹ
    MDC#put͠ɺݺͼग़͠ޙʹ
    MDC#remove͢Δ
    

    View Slide

49. > ઌఔ࡞ͬͨ Custom Subscriber ͕࢖ΘΕΔΑ
    ͏ʹ Hooks ʹొ࿥͢Δ
    > (͜ͷྫͩͱSpring Configuration ʹ ॻ͍ͯΔ)
    

    View Slide

50. Tips ͦͷ1: Reactor hooks with MDC
    

    View Slide

51. Tips ͦͷ2: BlockHound
    > Non Blocking εϨουʹ͓͚Δ Blocking ίʔϧΛݕग़ͯ͘͠ΕΔ Java Τʔδ
    Σϯτ
    • ͋·Γ೔ຊޠͰݴٴ͞Ε͍ͯΔهࣄΛΈ͔͚ͳ͍ͷͰ঺հ͠·͢
    • ίʔυϨϏϡʔͰ͸ൃݟͰ͖ͳ͍Մೳੑ΋͋ΔͷͰɺೖΕ͓ͯ͘ͱྑ͍
    • ͠͹͠͹༧ظ͠ͳ͍ Blocking ίʔϧ΋͋ΔͷͰ...
    > https://github.com/reactor/BlockHound
    

    View Slide

52. Tips ͦͷ2: BlockHound
    > ґଘʹ͍ΕΔ
    

    View Slide

53. Tips ͦͷ2: BlockHound
    ྫ͑͹͜Μͳίʔυ͕͋Δͱ͢Δ
    (ྫͳͷͰத਎͸ద౰)
    

    View Slide

54. Tips ͦͷ2: BlockHound
    BlockHoundΛ༗ޮʹ͢Δ
    ςετॻ͘
    

    View Slide

55. Tips ͦͷ2: BlockHound
    Block ͯ͠͠·͍ͬͯΔͱ͍͏

    ࢫͷྫ֎͕ग़Δ
    

    View Slide

56. WebFlux(Reactor) ͋Ε͜Ε ·ͱΊ
    > ࣮ࡍʹӡ༻ͯ͠Έͯɺࠔͬͨ͜ͱΛ2ͭ঺հ͠·ͨ͠
    • ౎౓ɺOSS ʹϑΟʔυόοΫ΋͍ͯ͠·͢
    > WebFlux ͷӡ༻ʹ͓͚Δ logging(MDC) ͷ Tips Λ঺հ͠·ͨ͠
    > Blocking ίʔϧͷݕग़Λͯ͘͠ΕΔ BlockHound ͷ঺հΛ͠·ͨ͠
    

    View Slide

57. (͓·͚) OA Dev Team
    

    View Slide

58. OA Dev 1 OA Dev 2
    OA(Official Account) Dev Team
    > OA Dev 1
    • LINE Official Account Manager , …etc
    > OA Dev 2
    • LINEνϟοτ, …etc
    > We are hiring!! → https://linecorp.com/ja/career/position/1643
    6ਓ
    8ਓ
    

    View Slide

59. Thank You for Watching
    

    View Slide