Web service development at LINE / LINE Campus Talk in UC Berkeley by Koji Lin

19.02.2019 Campus Talk in UC Berkeley
Presented by Koji Lin

2. About Me ྛ ߁࢘ LINE Fukuoka Corporation Manager of Development Center Dept. 1 KOJI LIN
3. Web Service at LINE LINE Fukuoka Dept. 1 • Sticker/Theme/Emoji Shop • Wallet tab • UIT • Verda
4. Web Service at LINE Sticker/Theme/Emoji Shop and Web Store
5. Web Service at LINE Shop Team • Provide Sticker/Theme/Emoji product in LINE • Ownership management • Artifact downloading/uploading • Hot/New ranking • Auto suggest keyword management
6. Web Service at LINE High traffic • Sticker/Theme/Emoji is highly used by LINE users • Many corporation's official account provides sticker for free • Million followers • 00:00 at January 1st • People usually buy and reply happy new year sticker when they 
 receive from friend • 5 times larger than usual • Trending stickers by TV show • Especially requests are easily to be amplified in microservices architecture
7. Web Service at LINE Overview Shop Server(s) MySQL/ MongoD B Elasticsearch Redis MySQL/ MongoDB LEGY/ Talk Server 
 Other Team
8. Web Service at LINE Lots of server to server requests Shop API 
 Billing 
 Data Team 
 
 Object storage 
 
 Ownership Legacy Shop API
9. Web Service at LINE What we use • Java • Thrift • Armeria • LINE’s open-source Async RPC library • Based on Netty and supports HTTP/2 • Spring Boot
10. Web Service at LINE What we use • MySQL and MongoDB • Elasticsearch • Redis • Grafana/Prometheus • Zipkin
11. Web Service at LINE MySQL and MongoDB • Store product data and user ownership data • More than 100 million users related records. • At first we use in-house sharding library for MySQL • Recently we use sharding in MongoDB for most case
12. Web Service at LINE Elasticsearch • Using Elasticsearch’s score function and 
 Aggregation to provide different type of rankings.
13. Web Service at LINE Redis • Ranking and product related data is not changed frequently • Cache re-usable response for same request parameters
14. Web Service at LINE Monitoring • Grafana and Prometheus • Promgen
15. Web Service at LINE Zipkin tracing
16. Web Service at LINE Our responsibility • Feature development • Enhance the codebase • Providing reliable service
17. Web Service at LINE Feature development • Editor’s pick • Popup sticker • Emoji enhancement • Browsing history
18. Web Service at LINE Enhance the codebase • Migrate framework/library • Separate to fine-grained service • Refactor
19. Web Service at LINE Armeria and async • Before • Nifty Thrift client/server • Spring web and Spark web framework • Logics are usually implemented in synchronous way • After • Armeria • All blocking code change to asynchronous way
20. Web Service at LINE Why asynchrony ? • Synchronous calls don’t scale • Sending/Processing requests 1 by 1 • Max concurrent requests less than max threads • Inefficient cpu usage • Fast and priority requests blocked by others
21. Web Service at LINE Shop server asynchronification • MySQL JDBC access • Executor Pool and wrap in ListenableFuture<T> • MongoDB • Async Driver • Retrofit • RxJava Adapter • Redis • Lettuce Future API
22. Web Service at LINE Armeria and async programming • Reduce latency • Non-blocking Event Loop + async task • Encourage us to write concurrent async calls • Less resource usage • From thread per request to event loop • CPU, memory by less threads • Less tcp connections • Armeria supports HTTP/2
23. Web Service at LINE From Blocking to Non-blocking • Phase 1 • Switching to Armeria from other framework • Using ListenableFuture and Guava’s Futures#transformAsync
24. Web Service at LINE Using Dagger • Phase 1.5 • Hand written transform/transformAsync is not easy to maintain • Using Dagger Producers to compose async dependency
25. Web Service at LINE
26. Web Service at LINE Using Dagger • Dagger will generate code for us, we just need to call it directly
27. Web Service at LINE From Guava to RxJava2 • Phase 2 • Guava only has transform/transformAsync • When logic become bigger, dagger graph is not easy to read • RxJava has many operators • zip, retry, filter, etc.
28. Web Service at LINE From RxJava2 to Kotlin Coroutines • Phase 3 • RxJava has high learning curve • When to use observeOn? • When is the data produced? • Kotlin Coroutines is more readable
29. Web Service at LINE Providing reliable service • Monitoring • On call shifts • 2 members per week
30. Web Service at LINE Outage handling • Receive alert from slack/LINE/mail/call depends on the urgency • 1st/2nd responder need to check the metrics and decide to rollback/throttling
31. Web Service at LINE What metrics/logs do we check ? • Node • CPU/Memory/Network • JVM • Heap/Thread/GC • Application • Latency/Response size/Error count
32. Web Service at LINE Dump script jcmd $PID Thread.print > /home/www/logs/etc/thread_$DATE.txt jcmd $PID GC.heap_dump /home/www/logs/etc/heap_dump_$DATE.txt jcmd $PID GC.class_histogram > /home/www/logs/etc/class_histogram_$DATE.txt iostat -x > /home/www/logs/etc/io_stats_$DATE.txt lsof > /home/www/logs/etc/lsof_$DATE.txt w > /home/www/logs/etc/w_$DATE.txt free > /home/www/logs/etc/free_$DATE.txt df -h > /home/www/logs/etc/df_$DATE.txt netstat -an > /home/www/logs/etc/netstat_$DATE.txt
33. Web Service at LINE Outage handling • Receive alert from slack/LINE/mail depends on the urgency • 1st/2nd responder need to check the metrics and decide to rollback/throttling • Create Outage report(postmortem) for future enhancement
34. Web Service at LINE Many teams are involved • Shop Team • Server • Client • UIT • Verda • Z part • B part • Datalab • and more
35. Web Service at LINE Shop Client • Manage sticker/theme/emoji in LINE client • Purchase and download products • Send sticker/emoji in chatroom • Apply theme
36. Web Service at LINE UIT • Frontend team • Sticker/Theme shop in native client is web application • Web Store • CMS for partner
37. Web Service at LINE Verda • Develop/Maintain Common/Fundamental Function for Private Cloud (IaaS) • OpenStack + α • Object storage • Load balancer • MySQL/Elasticsearch/kafka…etc • K8s
38. Web Service at LINE Z part • Provide reliable HBase/Redis/Kafka for talk server • Shop team produces purchase event • Other team can consume events to provide campaign
39. Web Service at LINE B part • Provide Bot related API • Send campaign/recommend product message
40. Web Service at LINE Datalab • Data scientists/Data engineers/Machine learning engineers • Provides data processing platform • Hive/Presto/HDFS…etc • Processing sales count/similarities/keyword for ranking and recommendations • Prepare and analyze A/B testing
41. Web Service at LINE And more • Billing • Game • Other countries’ campaign • LINE Search
42. Web Service at LINE At Last • LINE has many services and apps • Camera, Games, Music …etc • Different team/project may use different stacks/tools • Backend services are usually written in Java • Monitoring/Provisioning/Data Processing