Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Web service development at LINE / LINE Campus Talk in UC Berkeley by Koji Lin

Web service development at LINE / LINE Campus Talk in UC Berkeley by Koji Lin

19.02.2019 Campus Talk in UC Berkeley
Presented by Koji Lin


LINE Developers

February 19, 2019


  1. None
  2. About Me ྛ ߁࢘ LINE Fukuoka Corporation Manager of Development

    Center Dept. 1 KOJI LIN
  3. Web Service at LINE LINE Fukuoka Dept. 1 • Sticker/Theme/Emoji

    Shop • Wallet tab • UIT • Verda
  4. Web Service at LINE Sticker/Theme/Emoji Shop and Web Store

  5. Web Service at LINE Shop Team • Provide Sticker/Theme/Emoji product

    in LINE • Ownership management • Artifact downloading/uploading • Hot/New ranking • Auto suggest keyword management
  6. Web Service at LINE High traffic • Sticker/Theme/Emoji is highly

    used by LINE users • Many corporation's official account provides sticker for free • Million followers • 00:00 at January 1st • People usually buy and reply happy new year sticker when they 
 receive from friend • 5 times larger than usual • Trending stickers by TV show • Especially requests are easily to be amplified in microservices architecture
  7. Web Service at LINE Overview Shop Server(s) MySQL/ MongoD B

    Elasticsearch Redis MySQL/ MongoDB LEGY/ Talk Server 
 Other Team
  8. Web Service at LINE Lots of server to server requests

    Shop API 
 Data Team 
 Object storage 
 Ownership Legacy Shop API
  9. Web Service at LINE What we use • Java •

    Thrift • Armeria • LINE’s open-source Async RPC library • Based on Netty and supports HTTP/2 • Spring Boot
  10. Web Service at LINE What we use • MySQL and

    MongoDB • Elasticsearch • Redis • Grafana/Prometheus • Zipkin
  11. Web Service at LINE MySQL and MongoDB • Store product

    data and user ownership data • More than 100 million users related records. • At first we use in-house sharding library for MySQL • Recently we use sharding in MongoDB for most case
  12. Web Service at LINE Elasticsearch • Using Elasticsearch’s score function

 Aggregation to provide different type of rankings.
  13. Web Service at LINE Redis • Ranking and product related

    data is not changed frequently • Cache re-usable response for same request parameters
  14. Web Service at LINE Monitoring • Grafana and Prometheus •

  15. Web Service at LINE Zipkin tracing

  16. Web Service at LINE Our responsibility • Feature development •

    Enhance the codebase • Providing reliable service
  17. Web Service at LINE Feature development • Editor’s pick •

    Popup sticker • Emoji enhancement • Browsing history
  18. Web Service at LINE Enhance the codebase • Migrate framework/library

    • Separate to fine-grained service • Refactor
  19. Web Service at LINE Armeria and async • Before •

    Nifty Thrift client/server • Spring web and Spark web framework • Logics are usually implemented in synchronous way • After • Armeria • All blocking code change to asynchronous way
  20. Web Service at LINE Why asynchrony ? • Synchronous calls

    don’t scale • Sending/Processing requests 1 by 1 • Max concurrent requests less than max threads • Inefficient cpu usage • Fast and priority requests blocked by others
  21. Web Service at LINE Shop server asynchronification • MySQL JDBC

    access • Executor Pool and wrap in ListenableFuture<T> • MongoDB • Async Driver • Retrofit • RxJava Adapter • Redis • Lettuce Future API
  22. Web Service at LINE Armeria and async programming • Reduce

    latency • Non-blocking Event Loop + async task • Encourage us to write concurrent async calls • Less resource usage • From thread per request to event loop • CPU, memory by less threads • Less tcp connections • Armeria supports HTTP/2
  23. Web Service at LINE From Blocking to Non-blocking • Phase

    1 • Switching to Armeria from other framework • Using ListenableFuture and Guava’s Futures#transformAsync
  24. Web Service at LINE Using Dagger • Phase 1.5 •

    Hand written transform/transformAsync is not easy to maintain • Using Dagger Producers to compose async dependency
  25. Web Service at LINE

  26. Web Service at LINE Using Dagger • Dagger will generate

    code for us, we just need to call it directly
  27. Web Service at LINE From Guava to RxJava2 • Phase

    2 • Guava only has transform/transformAsync • When logic become bigger, dagger graph is not easy to read • RxJava has many operators • zip, retry, filter, etc.
  28. Web Service at LINE From RxJava2 to Kotlin Coroutines •

    Phase 3 • RxJava has high learning curve • When to use observeOn? • When is the data produced? • Kotlin Coroutines is more readable
  29. Web Service at LINE Providing reliable service • Monitoring •

    On call shifts • 2 members per week
  30. Web Service at LINE Outage handling • Receive alert from

    slack/LINE/mail/call depends on the urgency • 1st/2nd responder need to check the metrics and decide to rollback/throttling
  31. Web Service at LINE What metrics/logs do we check ?

    • Node • CPU/Memory/Network • JVM • Heap/Thread/GC • Application • Latency/Response size/Error count
  32. Web Service at LINE Dump script jcmd $PID Thread.print >

    /home/www/logs/etc/thread_$DATE.txt jcmd $PID GC.heap_dump /home/www/logs/etc/heap_dump_$DATE.txt jcmd $PID GC.class_histogram > /home/www/logs/etc/class_histogram_$DATE.txt iostat -x > /home/www/logs/etc/io_stats_$DATE.txt lsof > /home/www/logs/etc/lsof_$DATE.txt w > /home/www/logs/etc/w_$DATE.txt free > /home/www/logs/etc/free_$DATE.txt df -h > /home/www/logs/etc/df_$DATE.txt netstat -an > /home/www/logs/etc/netstat_$DATE.txt
  33. Web Service at LINE Outage handling • Receive alert from

    slack/LINE/mail depends on the urgency • 1st/2nd responder need to check the metrics and decide to rollback/throttling • Create Outage report(postmortem) for future enhancement
  34. Web Service at LINE Many teams are involved • Shop

    Team • Server • Client • UIT • Verda • Z part • B part • Datalab • and more
  35. Web Service at LINE Shop Client • Manage sticker/theme/emoji in

    LINE client • Purchase and download products • Send sticker/emoji in chatroom • Apply theme
  36. Web Service at LINE UIT • Frontend team • Sticker/Theme

    shop in native client is web application • Web Store • CMS for partner
  37. Web Service at LINE Verda • Develop/Maintain Common/Fundamental Function for

    Private Cloud (IaaS) • OpenStack + α • Object storage • Load balancer • MySQL/Elasticsearch/kafka…etc • K8s
  38. Web Service at LINE Z part • Provide reliable HBase/Redis/Kafka

    for talk server • Shop team produces purchase event • Other team can consume events to provide campaign
  39. Web Service at LINE B part • Provide Bot related

    API • Send campaign/recommend product message
  40. Web Service at LINE Datalab • Data scientists/Data engineers/Machine learning

    engineers • Provides data processing platform • Hive/Presto/HDFS…etc • Processing sales count/similarities/keyword for ranking and recommendations • Prepare and analyze A/B testing
  41. Web Service at LINE And more • Billing • Game

    • Other countries’ campaign • LINE Search
  42. Web Service at LINE At Last • LINE has many

    services and apps • Camera, Games, Music …etc • Different team/project may use different stacks/tools • Backend services are usually written in Java • Monitoring/Provisioning/Data Processing