Enable Resiliency and Efficiency at LINE Invoice

活動：LINE Developers Meetup 開發者小聚 #18 - TECH FRESH 畢業展
活動頁面：https://linegroup.kktix.cc/events/20230627-tech-fresh-gratudation
講者：Matt Chang, Steve Yu

Transcript

1. Enable Resiliency and
   Efficiency at LINE Invoice
   Matt Chang, Steve Yu
   2023.06
   

   View Slide

2. Matt Chang
   FullStack Engineer Intern @ LINE Invoice
   

   View Slide

3. • Introduce LINE Invoice
   • Share my intern experience
   • Share some tech knowledge
   • Mainly focus on Backend.
   Introduction
   

   View Slide

4. LINE 發票管家
   

   View Slide

5. Backend: CRUD Boy
   • CRUD (Create, Read, Update, Delete)
   • Example: TODO List
   • What's CRUD Boy?
   

   View Slide

6. Backend: CRUD on Distributed System
   • CRUD on Distributed System
   • 寫的 code 變成跑在多台 server 上
   • 不同 server 間彼此需要緊密溝通
   • 需要考慮的事情變多
   • Race condition
   • Unreliable connection
   

   View Slide

7. Resilliency and Efficiency
   • Resilliency
   • Infra:
   • E.g. self-healing, rolling upgrade
   • Backend:
   • E.g. Retry, Timeout and Rate Limit
   • Efficiency:
   • Infra:
   • Vertical Scaling
   • Horizontal Scaling
   • Backend:
   • E.g. Time complexity, concurrent/multithread programming, System Design
   

   View Slide

8. Resilliency: Unreliable connection
   • Be careful with Input & Output.
   • Retry:
   • What happened when requested service is temporarily unavailable?
   • Timeout:
   • What happened when requested service is unresponsive?
   • Rate limit:
   • In-bound: Restrict others from accessing service too frequently.
   • Out-bound: Restrict request frequency to other service.
   

   View Slide

9. Efficiency: Concurrent Programming
   Real world example: 7-11
   

   View Slide

10. Efficiency: System Design
    • Usage of Cache and Why ?
    • Memory access is faster than disk.
    • No need to do complex computation again.
    • Real world example: Human Brain
    • In-memory cache Problem
    • Cache stored in server S1 can't be accessed
    by server S2.
    • Try In-memory database
    • E.g. Redis
    

    View Slide

11. Short Summary
    • CRUD / CRUD in distributed System
    • Resiilliency
    • Retry / Timeout / Rate Limiting
    • Efficiency
    • Concurrent programming
    • System Design: Cache
    

    View Slide

12. SteveYi Yu
    Site Reliability Engineering Intern @ LINE
    12
    ● Service maintenance (k8s)
    ● Deployment (containerization & YAML)
    ● Monitoring (Grafana)
    

    View Slide

13. 簡單來說就是 CI/CD 一條龍
    Alert & Monitor
    13
    CI/CD
    Docker-compose / containerization & ArgoCD
    ● CI/CD
    ○ 自動化 build 成
    Image
    ○ 上傳到內部的
    Docker Registry
    ● ArgoCD
    ○ 以自動規模化
    (Auto Scale) 的方
    式部署
    ● Alert & Monitor
    ○ Grafana 上查看視
    覺化面板
    ○ 異常情況發送通
    知到 Slack
    

    View Slide

14. 簡單來說就是 CI/CD 一條龍
    14
    

    View Slide

15. Deploy to ArgoCD
    15
    ● Build to Docker Image
    ● Upload to Harbor
    (Internal Image Registry)
    ● Update k8s Deploy YAML
    

    View Slide

16. Deploy to ArgoCD
    16
    

    View Slide

17. How LINE Invoice Work?
    17
    ● Kubernetes
    ○ 根據流量自動擴展 Pod
    ○ rolling update 會將流量分散
    ，不影響用戶訪問
    

    View Slide

18. How LINE Invoice Work?
    18
    ● Treafik
    ○ 負責處理用戶的請求
    ○ 將流量分散的分配給不
    同的 Pod
    

    View Slide

19. THANK YOU
    

    View Slide