Upgrade to Pro — share decks privately, control downloads, hide ads and more …

LINE公式アカウントのチャットシステムにおけるSpringおよびWebFluxの活用事例 / Examples of using Spring and WebFlux in the chat system for LINE official accounts

LINE公式アカウントのチャットシステムにおけるSpringおよびWebFluxの活用事例 / Examples of using Spring and WebFlux in the chat system for LINE official accounts

LINE株式会社 長谷部 良輔
SPRING FEST ’19での発表資料です。
https://springfest2019.springframework.jp/

LINE Developers
PRO

December 18, 2019
Tweet

More Decks by LINE Developers

Other Decks in Technology

Transcript

  1. LINEެࣜΞΧ΢ϯτͷνϟοτγεςϜ

    ʹ͓͚Δ Spring ͓Αͼ WebFlux ͷ׆༻ࣄྫ
    > Ryosuke Hasebe / LINE corporation

    View Slide

  2. About Me
    > LINEެࣜΞΧ΢ϯτ(OA: Official Account) ͷ։ൃ
    • LINEνϟοτ, LINE Official Account Manager, …
    ࠷ۙͷ࢓ࣄ
    Ҏલͷ࢓ࣄ
    > LINE Login, LINE Customer Connect, Profile+, LINE Notify, LINE BOT SDK Python, 

    LINE GAME PLATFORM, …
    ௕୩෦ ྑี (@be_hase1014)
    > ։ൃ4ηϯλʔ / OA Dev 2 νʔϜ
    > 2013೥ ೖࣾ

    View Slide

  3. Agenda
    > LINE ♥ Spring
    > LINEνϟοτʹ͍ͭͯ
    > Spring ͓Αͼ WebFlux ͷ׆༻ࣄྫ
    • γεςϜߏ੒/ઃܭͷ࿩
    • WebFlux(Reactor) ͋Ε͜Ε

    View Slide

  4. Agenda
    > LINE ♥ Spring
    > LINEνϟοτʹ͍ͭͯ
    > Spring ͓Αͼ WebFlux ͷ׆༻ࣄྫ
    • γεςϜߏ੒/ઃܭͷ࿩
    • WebFlux(Reactor) ͋Ε͜Ε

    View Slide

  5. Spring ͨ͘͞Μ

    ࢖͍ͬͯ·͢!!

    View Slide

  6. springframework Ͱݕࡧͯ͠Έͨ
    > Gradle: 3,044
    > Maven POM: 2,788
    > → 5000 project ͘Β͍͋Δ...ʁ

    View Slide

  7. Why Spring (boot)?
    > DI ίϯςφ & AOP
    > ๛෋ͳΠϯςάϨʔγϣϯ
    • MyBatis, FreeMarker/Thymeleaf, spring-boot-actuator, ...
    > Profiles & Configuration
    • ؀ڥຖͷίϯϑΟά؅ཧ͕ྑ͍ײ͡
    > RESTful API Λ࣮૷͢ΔͨΊͷඞཁͳύʔπ͕ἧ͍ͬͯΔ
    • spring-webmvc, spring-webflux
    > ଞͷ RPC ϑϨʔϜϫʔΫͱ΋૊Έ߹ΘͤΔ͜ͱ͕Ͱ͖Δ
    • Spring with Armeria, Spring with gRPC

    View Slide

  8. LINE ♥ Spring

    View Slide

  9. Agenda
    > LINE ♥ Spring
    > LINEνϟοτʹ͍ͭͯ
    > Spring ͓Αͼ WebFlux ͷ׆༻ࣄྫ
    • γεςϜߏ੒/ઃܭͷ࿩
    • WebFlux(Reactor) ͋Ε͜Ε

    View Slide

  10. > اۀ/ࣄۀऀ͕։ઃ͢ΔLINEΞΧ΢ϯτ
    > Ϗδωε޲͚ʹ༷ʑͳػೳ͕͋Δ
    • ϒϩʔυΩϟετ/λʔήςΟϯά഑৴
    • … etc
    LINEެࣜΞΧ΢ϯτͬͯʁ
    >date/november 20-21
    >place/grand nikko tokyo

    View Slide

  11. LINEνϟοτͬͯʁ
    > ໊শ͕େม΍΍͍͜͠Ͱ͕͢ɺ

    LINEެࣜΞΧ΢ϯτͷӡӦऀͷͨΊͷνϟοτγεςϜͷ͜ͱͰ͢
    > LINEެࣜΞΧ΢ϯτ LINEϢʔβʔؒͰνϟοτ͕Ͱ͖·͢
    ΞΧ΢ϯτΦʔφʔ LINEϢʔβʔ

    View Slide

  12. Screen Shot
    Ƃ
    >november_20_first day
    >09:00-18:00
    >november_20_first day
    >09:00-18:00
    >date/november 20-21
    >place/grand nikko tokyo

    View Slide

  13. ن໛ײ
    αʔόʔ਺
    (DB / storage ؚΉ)
    150+
    ಉ࣌઀ଓ਺
    5ສ+
    ݄ؒϝοηʔδ

    ૹड৴਺

    2019/12

    View Slide

  14. Agenda
    > LINE ♥ Spring
    > LINEνϟοτʹ͍ͭͯ
    > Spring ͓Αͼ WebFlux ͷ׆༻ࣄྫ
    • γεςϜߏ੒/ઃܭͷ࿩
    • WebFlux(Reactor) ͋Ε͜Ε

    View Slide

  15. γεςϜߏ੒/ઃܭͷ࿩

    View Slide

  16. νϟοτγεςϜʹٻΊΒΕΔ΋ͷ (ͦͷ1)
    > ϦΞϧλΠϜ௨৴
    • LINE Ϣʔβʔ͔Βड৴ͨ͠ϝοηʔδ
    • ଞͷ ΞΧ΢ϯτΦʔφʔ ͕ߦͬͨૢ࡞
    • → ͜ΕΒΛϦΞϧλΠϜʹΫϥΠΞϯτʹ൓ө͢Δඞཁ͕͋Δ

    View Slide

  17. CGI ࣌୅ͷνϟοτγεςϜ
    ʊਓਓਓਓਓਓਓਓਓਓਓʊ
    ʼɹiframe ࣗಈϦϩʔυɹʻ
    ʉY^Y^Y^Y^Y^Y^Y^Y^Y^Y^Y^ʉ
    ౰࣌͸ͱͯ΋ָ͔͚ͬͨ͠Ͳ…

    View Slide

  18. ϒϥ΢βʹ͓͚ΔϦΞϧλΠϜ௨৴
    > ީิ
    • Polling
    • Long Polling (Comet)
    • Server Sent Events (SSE)
    • WebSocket
    • (ࠓͩͬͨΒɺgRPC-WEB ͱ͔ RSocket ΋ݕ౼͔ͨ͠ͳ...)
    > SSE Λ࠾༻ͨ͠
    • ૒ํ޲௨৴͸ෆཁɻςΩετܗࣜͰ΋໰୊ͳ͍
    • WebSocket ͱൺֱ͢Δͱγϯϓϧ (HTTP1.1 Chunked ϕʔε)

    View Slide

  19. spring-webmvc Ͱ Server Sent Events
    > ࢖͍׳Ε͍ͯΔ spring-webmvc(servlet) Ͱ

    ରԠ͞Ε͍ͯΔ͕…
    > SSE ͸αʔόʔͱ઀ଓͬ͠ͺͳ͠
    • ैདྷͷ Servlet Ͱ͸εέʔϧ͠ͳ͍ (ӈਤ)
    • 1ϦΫΤετʹରͯ͠1εϨου ͳͨΊ
    • LINE Chat ͷݱࡏͷಉ࣌઀ଓ͸5ສ+
    • 1ͭͷαʔόʔͰ500εϨου༻ҙͨ͠ͱͯ͠΋ɺ

    ͜Ε͚ͩͰαʔόʔ͕100୆ඞཁʹͳͬͯ͠·͏

    > Servlet 3.0 / Async Servlet Λ࢖͍ͬͯΔͷͰεϨουઐ
    ༗ͷ໰୊΋ͳͦ͞͏ʹݟ͑Δ͕…
    Thread1
    Thread2
    Thread3

    View Slide

  20. spring-webmvc Ͱ Server Sent Events
    ͱ͸͍͑…

    View Slide

  21. ͱ͸͍͑…
    > ͚͖ͬΐ͘ Servlet ͰͷϦΫΤετͷड৴/ૹ৴
    ͸ Input/OutputStream Λ༻͍ͨ Blocking I/O
    • https://www.infoq.com/presentations/servlet-
    reactive-stack/
    > → Spring5 ͔Β௥Ճ͞Εͨ WebFlux Λ࠾༻͢
    Δ͜ͱʹ͠·ͨ͠
    spring-webmvc
    Servlet API
    Tomcat, Jetty, …
    Blocking I/O

    View Slide

  22. WebFlux ͱ͸
    > Spring5 ͔Βొ৔ͨ͠ Non Blocking ͳ Web ϑϨʔϜϫʔΫ
    spring-webmvc
    Servlet API
    Tomcat, Jetty, …
    Blocking I/O
    spring-webflux
    Spring Web API

    (Reactor, Reactive Streams)
    Tomcat, Jetty, …

    (Servlet 3.1)
    Non Blocking I/O
    Netty

    View Slide

  23. WebFlux ͱ͸
    > ֤ϦΫΤετΛ Non Blocking Ͱॲཧ͢Δ͜ͱͰɺ1ͭͷ Event Loop

    Thread Ͱෳ਺ϦΫΤετΛࡹ͚Δ = গͳ͍αʔόʔ਺Ͱࡹ͚Δ
    > Block ͢Δίʔυ͸ઈରʹॻ͍ͯ͸͍͚ͳ͍ɻઈରʹμϝ
    Thread1
    Thread2
    Thread3
    Event Loop Thread

    View Slide

  24. αʔόʔ1୆͋ͨΓͷಉ࣌઀ଓ਺
    1୆͋ͨΓͷಉ࣌઀ଓ਺͸2500ɻ
    2~3ഒ͸͍͚Δ͚Ͳɺ༨༟Λ΋ͬͯɻ
    (ݱࡏ͸20୆࢖͍ͬͯ·͢)

    View Slide

  25. ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ
    Talk
    Server
    Event

    Receiver
    Kafka
    Event
    Processor
    Redis Cluster
    (PubSub)
    Redis Cluster
    SSE

    View Slide

  26. ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ
    Talk
    Server
    Event

    Receiver
    Kafka
    Event
    Processor
    > ϝοηʔδड৴ Event ͕ಧ͘
    > (ଞʹ΋طಡ, ༑ͩͪ௥Ճ, ...౳ͷ Event ΋)
    > ࣮ࡍ͸ؒʹଞͷίϯϙʔωϯτ͕͋Δ
    Redis Cluster
    (PubSub)
    Redis Cluster
    SSE

    View Slide

  27. ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ
    Talk
    Server
    Event

    Receiver
    Kafka
    Event
    Processor
    Redis Cluster
    (PubSub)
    Redis Cluster
    SSE
    > Event Receiver ͸ड͚औͬͨΒ Kafka ʹॻ͘
    > Ұ౓ Kafka ʹॻ͍͓ͯ͘͜ͱͰɺEvent ͷ࠶ར༻΍

    ϦτϥΠΛ͠΍͘͢͢Δ
    > Event Receiver ΋ WebFlux Λ࢖࣮ͬͯ૷͍ͯ͠Δ

    View Slide

  28. ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ
    Talk
    Server
    Event

    Receiver
    Kafka
    Event
    Processor
    Redis Cluster
    (PubSub)
    Redis Cluster
    SSE
    > Event Processor Ͱ Kafka ͔Β Event ΛಡΉ
    > Redis ʹ Publish ͢Δ
    > SSE server ͕ Subscribe ͢Δ

    View Slide

  29. ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ
    Talk
    Server
    Event

    Receiver
    Kafka
    Event
    Processor
    Redis Cluster
    (PubSub)
    Redis Cluster
    SSE
    > ࠶઀ଓΛαϙʔτ͢ΔͨΊʹɺPublish ͱಉ࣌ʹ

    Ϧετͱͯ͠΋อ͓࣋ͯ͘͠
    > ࠶઀ଓʹ࠷ޙʹऔಘͨ͠ Event ID(Last-Event-ID
    header) ͕౉͞ΕΔͷͰɺະऔಘͷ Event Λฦ͢

    View Slide

  30. ϝοηʔδ͕ϦΞϧλΠϜʹಧ͘·ͰͷྲྀΕ
    Talk
    Server
    Event

    Receiver
    Kafka
    Event
    Processor
    Redis Cluster
    (Stream)
    SSE
    > Redis5 ͔Β௥Ճ͞Εͨ Redis Streams Λ

    ࢖͑͹΋ͬͱ؆୯ʹͳΓͦ͏
    > ※ զʑͷ Redis ͸ 4ܥ...

    View Slide

  31. Subscribe Redis as Flux
    (Reactive lettuce)
    ௕࣌ؒ, Event ͕ͳ͍ͱ઀ଓ͕

    ੾ΕΔͷͰ PING ΛૹΔ
    2ͭͷ Flux Λ merge

    View Slide

  32. νϟοτγεςϜʹٻΊΒΕΔ΋ͷ (ͦͷ2)
    > ϝοηʔδͷૹ৴
    > աڈϝοηʔδͷऔಘ
    > ϝοηʔδͷݕࡧ
    > ͦͷଞɺ֤छσʔλͷ CRUDૢ࡞
    → Ϣʔβʔૢ࡞ىҼͷɺίΞͱͳΔϏδωεϩδοΫ

    View Slide

  33. spring-webmvc Ͱ΍͍ͬͯΔ
    MySQL
    (Sharding)
    RESTful API

    (spring-webmvc)
    Talk
    Server
    Message

    Gateway
    ΞΧ΢ϯτΦʔφʔ
    Elasticsearch
    HBase
    http
    jdbc
    http
    Redis Cluster

    View Slide

  34. Why spring-webmvc?
    > JDBC... (Blocking I/O)
    • ͓...?
    • Spring Data R2DBC goes GA
    • https://spring.io/blog/2019/12/06/spring-data-r2dbc-goes-ga
    > ͍͖ͳΓΨοπϦͱͨ͠ϏδωεϩδοΫΛ WebFlux Ͱ࣮૷͢Δͷ͸

    ͪΐͬͱϏϏͬͨɺͱ͍͏ͷ΋͋Δ
    • খͯ͘͞ɺ࣮֬ʹ޲͍͍ͯΔίϯϙʔωϯτ͔Βࢼͦ͏
    • ։ൃͨ͠ͱ͖͸ boot 2.0.0 M1 ͱ͔ͩͬͨͷͰ...

    View Slide

  35. νϟοτγεςϜʹٻΊΒΕΔ΋ͷ (ͦͷ3)
    > LINE Ϣʔβʔ͕ૹ৴ͨ͠ը૾ɾಈըɾԻ੠ͷऔಘ
    • ࣾ಺ͷίϯςϯπαʔόʔ(with CDN)ʹ͋Δ
    • ೝূ౳ͷॾʑͷࣄ৘Ͱͦͷ··ΞΧ΢ϯτΦʔφʔ͕ΞΫηε͢Δ͜ͱ
    ͸Ͱ͖ͳ͍ͨΊɺProxy ͢Δඞཁ͕͋ͬͨ

    View Slide

  36. spring-webflux Ͱ΍͍ͬͯΔ
    Content Proxy

    (spring-webflux)
    Content Server
    ΞΧ΢ϯτΦʔφʔ
    http
    > ೝূͯ͠ࠨ͔Βӈʹ Proxy ͢Δ͚ͩɻ·͞ʹ Non blocking I/O ͕޲͍͍ͯΔ
    > εϨουΛઐ༗͢Δ͜ͱͳ͘ Proxy ͢Δ͜ͱ͕Ͱ͖Δ
    Auth
    http

    View Slide

  37. ͍·ৼΓฦͬͯΈͯɺͲ͏͔
    > େ͖ͳো֐΋ͳ͘1೥ؒӡ༻Ͱ͖ͨͷͰɺٕज़બఆ͸੒ޭͨ͠ͱࢥ͍ͬͯΔ
    > WebFlux ʹ׳Ε͖ͯͨࠓͱͳͬͯ͸ RESTful API αʔόʔ ΋ WebFlux Ͱ

    ࣮૷͢Ε͹Α͔ͬͨͳ...ͱ͸ͪΐͬͱ͚ͩࢥ͏
    • Elasticsearch ͕ߴෛՙʹͳͬͯϨΠςϯγ͕ߴ·Γɺ Thread Busy ͱͳ
    ͬͨέʔε͕͋ͬͨ (Servlet ͋Δ͋Δ)
    • → Circuit breaker(resilience4j) Ͱ failing fast ͢Δ͜ͱͰରॲͯ͠Δ

    View Slide

  38. γεςϜߏ੒/ઃܭͷ࿩ ·ͱΊ
    > ίϯϙʔωϯτͷ໾ׂʹԠͯ͡ spring-webflux, webmvc Λ࢖͍෼͚͍ͯΔ
    > spring-webflux
    • SSE αʔόʔ (ϦΞϧλΠϜ௨৴ɺ௕࣌ؒ઀ଓ)
    • Event Receiver αʔόʔ (ΞΫηεଟΊ)
    • Content Proxy αʔόʔ
    > spring-webmvc
    • RESTful API αʔόʔ (ίΞɾϏδωεϩδοΫ)

    View Slide

  39. WebFlux(Reactor) ͋Ε͜Ε

    View Slide

  40. ࠔͬͨ࿩ ͦͷ1: Netty OOME
    > ྫ͑͹Webflux ClientΛ༻͍ͨ

    ͜Μͳcode͕͋Δͱ͠·͢
    > ͜ͷ API call Ͱ timeout ͕

    େྔʹൃੜ͢Δͱɺ

    OOME Ͱαʔόʔ͕མͪΔͱ͍͏͜ͱ͕͋ͬͨ

    View Slide

  41. ࠔͬͨ࿩ ͦͷ1: Netty OOME
    > Netty ͔Β ReadTimeoutException ͕

    ൃੜ͢Δͱreactor core ಺Ͱ͜Ε͕࣮ߦ͞ΕΔ
    > Exceptions#propagate ͸RuntimeException 

    ͷαϒΫϥεͷ৔߹͸ͦͷ··ฦ͍ͯ͠Δɻ

    ReadTimeoutException ͸ RuntimeException ͷ

    αϒΫϥεͳͷͰ ReadTimeoutException

    ࣗ਎͕ฦΔɻ
    > ݁Ռͱͯ͠ɺReadTimeoutException#addSuppressed

    ͕࣮ߦ͞ΕΔ
    > this is ok, as re is alway a new non-singleton instance 

    ͱॻ͍ͯ͋Δ͕…
    ReadTimeoutException
    ͜Ε΋ReadTimeoutException

    View Slide

  42. ࠔͬͨ࿩ ͦͷ1: Netty OOME
    > ReadTimeoutException ͕ singleton… ͩͬͨ
    > ͦͷ݁Ռ, ReadTimeoutException ͕อ͍࣋ͯ͠Δ
    suppressedExceptions(List) ͕ංେԽͯ͠ OOME ͕ى͖ͨ
    > Issue: https://github.com/netty/netty/pull/9152
    • ใࠂࡁɻݱࡏ͸मਖ਼͞Ε͍ͯ·͢

    View Slide

  43. ࠔͬͨ࿩ ͦͷ2: Blocking DNS Resolver
    > Reactor netty Ͱ͸ InetSocketAddress Λ࢖ͬͨ blocking DNS resolver Λ࢖༻͠
    ͍ͯΔͷͰɺnetty ͷ non-blocking DNS resolver Λ࢖༻͢ΔΑ͏ʹมߋ͠·͠
    ΐ͏
    • ·ͩ Unstable Api Ͱ͕͢…
    > ۃكʹ thread ͕ hang ͯ͠͠·͏ݱ৅͕͋ͬͨͨΊௐ΂ͨ
    > Issue: https://github.com/reactor/reactor-netty/issues/710
    • ใࠂࡁɻ0.10.x Backlog ʹೖ͍ͬͯ·͢
    > σϑΥϧτΛ non blocking DNS resolver ʹ͠Α͏ɺͱ͍͏Ҋ΋ग़͍ͯ·͢
    • https://github.com/reactor/reactor-netty/issues/569

    View Slide

  44. Tips ͦͷ1: Reactor hooks with MDC
    > Servlet ͷӡ༻ʹ׳Ε͍ͯΔਓ͸ MDC ʹ Request ৘ใΛ͍Ε͍ͯͨͱࢥ͍·͢
    • ྫ: Logback ͷ MDCInsertingServletFilter
    • http://logback.qos.ch/xref/ch/qos/logback/classic/helpers/
    MDCInsertingServletFilter.html
    > Request URL ౳ΛҰॹʹϩάʹग़ྗ͢Δ͜ͱ͕Ͱ͖ΔͷͰͱͯ΋ศར

    View Slide

  45. Tips ͦͷ1: Reactor hooks with MDC

    View Slide

  46. Tips ͦͷ1: Reactor hooks with MDC
    > WebFlux Ͱ΋ࣅͨΑ͏ͳ͜ͱΛ࣮ݱͨ͠
    ͍͕...Event Loop Thread ͱ͸ ૬ੑ͕ѱ͍
    > ࣍ͷ3ͭͷςΫχοΫΛ࢖͏͜ͱͰՄೳ
    • WebFilter
    • Reactor Context
    • Hooks Event Loop Thread

    View Slide

  47. WebFilterͰ Request ৘ใΛ Context 

    ʹೖΕ͓ͯ͘

    View Slide

  48. > ֤छonXXXXϝιουΛϥοϓ͢ΔΑ͏ͳ
    Custom SubscriberΛ࣮૷͢Δ
    • Context͔ΒWebFilterͰొ࿥ͨ͠
    request৘ใΛऔಘ͢Δ
    • subscriber#onXXXΛݺͼग़͢લʹ
    MDC#put͠ɺݺͼग़͠ޙʹ
    MDC#remove͢Δ

    View Slide

  49. > ઌఔ࡞ͬͨ Custom Subscriber ͕࢖ΘΕΔΑ
    ͏ʹ Hooks ʹొ࿥͢Δ
    > (͜ͷྫͩͱSpring Configuration ʹ ॻ͍ͯΔ)

    View Slide

  50. Tips ͦͷ1: Reactor hooks with MDC

    View Slide

  51. Tips ͦͷ2: BlockHound
    > Non Blocking εϨουʹ͓͚Δ Blocking ίʔϧΛݕग़ͯ͘͠ΕΔ Java Τʔδ
    Σϯτ
    • ͋·Γ೔ຊޠͰݴٴ͞Ε͍ͯΔهࣄΛΈ͔͚ͳ͍ͷͰ঺հ͠·͢
    • ίʔυϨϏϡʔͰ͸ൃݟͰ͖ͳ͍Մೳੑ΋͋ΔͷͰɺೖΕ͓ͯ͘ͱྑ͍
    • ͠͹͠͹༧ظ͠ͳ͍ Blocking ίʔϧ΋͋ΔͷͰ...
    > https://github.com/reactor/BlockHound

    View Slide

  52. Tips ͦͷ2: BlockHound
    > ґଘʹ͍ΕΔ

    View Slide

  53. Tips ͦͷ2: BlockHound
    ྫ͑͹͜Μͳίʔυ͕͋Δͱ͢Δ
    (ྫͳͷͰத਎͸ద౰)

    View Slide

  54. Tips ͦͷ2: BlockHound
    BlockHoundΛ༗ޮʹ͢Δ
    ςετॻ͘

    View Slide

  55. Tips ͦͷ2: BlockHound
    Block ͯ͠͠·͍ͬͯΔͱ͍͏

    ࢫͷྫ֎͕ग़Δ

    View Slide

  56. WebFlux(Reactor) ͋Ε͜Ε ·ͱΊ
    > ࣮ࡍʹӡ༻ͯ͠Έͯɺࠔͬͨ͜ͱΛ2ͭ঺հ͠·ͨ͠
    • ౎౓ɺOSS ʹϑΟʔυόοΫ΋͍ͯ͠·͢
    > WebFlux ͷӡ༻ʹ͓͚Δ logging(MDC) ͷ Tips Λ঺հ͠·ͨ͠
    > Blocking ίʔϧͷݕग़Λͯ͘͠ΕΔ BlockHound ͷ঺հΛ͠·ͨ͠

    View Slide

  57. (͓·͚) OA Dev Team

    View Slide

  58. OA Dev 1 OA Dev 2
    OA(Official Account) Dev Team
    > OA Dev 1
    • LINE Official Account Manager , …etc
    > OA Dev 2
    • LINEνϟοτ, …etc
    > We are hiring!! → https://linecorp.com/ja/career/position/1643
    6ਓ
    8ਓ

    View Slide

  59. Thank You for Watching

    View Slide