從混亂到掌控：使用 Observability 揭開管家的神秘面紗

Speaker: Gary Hu, Tristan Wu
Event: LINE TECH FRESH 畢業分享會

Transcript

1. 從混亂到掌控：使用 Observability 揭開管家的神秘面紗
2. Tristan Education • B.B.A in Finance @ NTU Experience • 2023 - 2024 | TECH FRESH @ LINE Taiwan • 2022 - 2023 | Software Engineer Intern @ Junyiacademy • 2022 | Backend Trainee @ AppWorks School
3. LINE INVOICE 發票管家
4. LINE OAP
5. 01 02 03 Three Pillars of Observability Case Study: LINE INVOICE Introduction to Observability CONTENT 04 Conclusion
6. SECTION 01 Introduction to Observability
7. What should be taken when system errors occurs?
8. Purpose of Observability
9. SECTION 02 Three Pillars of Observability
10. Three Pillars of Observability
11. Logs 1. Immutable / Timestamped record of discrete events 2. Record necessary info for each request Source: https://grafana.com/products/cloud/logs/
12. • Unstructured - PlainText • Structured - JSON format • Binary ◦ MySQL binlogs ◦ systemd journal logs Logs Format
13. • Unstructured - PlainText • Structured - JSON format • Binary ◦ MySQL binlogs ◦ systemd journal logs Logs Format
14. • Unstructured - PlainText • Structured - JSON format • Binary ◦ MySQL binlogs ◦ systemd journal logs Logs Format Source: https://www.percona.com/blog/binlog-encryption-percona-server-mysql/
15. Logs Collection Flow
16. Metrics Quantitative insight into system performance and resource utilization Source: https://grafana.com/products/cloud/metrics/
17. Metrics Supported Data Types Counter Gauge Histogram Summary
18. Metrics Supported Data Types Counter Gauge Histogram Summary • Only increases, never decreases • Application: HTTP request times
19. Metrics Supported Data Types Counter Gauge Histogram Summary • Increase or decrease at any time • Application: num of concurrent reqs
20. Metrics Supported Data Types Counter Gauge Histogram Summary • Only increases, never decreases • Application: request durations
21. Metrics Supported Data Types Counter Gauge Histogram Summary • Provides precise sampling of observations • Application: request durations
22. Metrics Collection Flow
23. Traces • Record and visualize the complete path of a request through the system • Identify specific points Source: https://grafana.com/docs/grafana/latest/panels-visualizations/visualizations/traces/ https://www.oreilly.com/library/view/distributed-systems-observability/9781492033431/ch04.html
24. Traces Span
25. Traces Span Structure
26. Traces Collection Flow
27. SECTION 03 Case Study LINE INVOICE
28. Gary Hu Education • M.S. in Computer Science @ NTU • B.B.A in Information Management @ NTU Experience • 2023 - 2024 | TECH FRESH @ LINE Taiwan • 2022 - 2023 | Software Engineer Intern @ KKCompany • 2022 | Research Assistant @ Academia Sinica
29. LINE INVOICE 發票管家
30. LINE Sticker 貼圖傳送任務
31. SECTION 03 Case Study LINE INVOICE
32. Case 1: Mystery Behind the Blank Screen Scenario Thousands of users simultaneously accessing our system Problem Users are met with blank screens and error messages. Challenges We need to investigate the error, and identify its cause.
33. Case 1: Mystery Behind the Blank Screen Steps 1. Centralized Log Collection 2. Log Search 3. Identify Log Locations 4. Impact Tracking
34. Case 2: Peak Traffic Monitoring Scenario Thousands of users simultaneously accessing our system Problem 1. Server cannot handle all requests 2. Timeouts and poor user experience Challenges Identify bottlenecks and optimize server performance
35. Case 2: Peak Traffic Monitoring Source: https://pixotech.com/blog/what-a-performance-how-site-speed-affects-ux/ https://smallbusinessweb.co/impact-of-website-loading-speed/ 3.7s Users start getting frustrated 75% Speed affects user experience 53% Users abandon after three seconds
36. Case 2: Peak Traffic Monitoring Steps 1. Collect metrics from all services 2. Visualize metrics to understand system behavior 3. Monitor traffic volume and response time
37. Case 2: Peak Traffic Monitoring Steps 4. Collect CPU and memory usage 5. Identify issues 6. Address inappropriate configurations
38. About Our System
39. Case 3: Mystery of the 5-Minute Workflow Scenario Many workflows are executed daily to fetch user invoices Problem We discovered that numerous workflows are taking over 5 minutes to complete. Challenges Identify the cause of the delays and optimize the workflow performance.
40. Case 3: Mystery of the 5-Minute Workflow Steps 1. Collect traces 2. Visualize traces 3. Analyze each spans v
41. Case 3: Mystery of the 5-Minute Workflow Findings • Fetching invoices from the government takes 27 seconds. • Storing the invoices in the database, however, takes 54 seconds. 54 secs 27 secs
42. Conclusion Precision Detection Optimization
43. Thank You Tristan Wu Gary Hu