The journey to the automated deployment in OpenChat server

Seunghwan Joo
LINE SQ Part OpenChat Dev Backend Software Engineer
https://linedevday.linecorp.com/2020/ja/sessions/1742
https://linedevday.linecorp.com/2020/en/sessions/1742

Transcript

2. Agenda › OpenChat Server Release Process › Problem / Objective › How We Automated › Result › Future Works
3. OpenChat Server Release Process
4. OpenChat Server Release Process › Release Master oversees the entire release. › Weekly release › Each member is in charge of 1 or 2 components. › Takes about 1H ~ 1.5H
5. OpenChat Server Release Process
6. OpenChat Server Release Process Share the beginning of release
7. OpenChat Server Release Process › LINE OpenChat chatroom, LINE Group chatroom
8. OpenChat Server Release Process Deploy & Restart
9. OpenChat Server Release Process › PMC: A tool which build and deploy LINE's major services.
10. OpenChat Server Release Process Monitor
11. OpenChat Server Release Process Share release completion
12. OpenChat Server Release Process
13. Problem
14. Cause human mistake
15. Waste human resource
16. Many repetitive manual operations
17. How many manual operations we repeated?
18. # of Manual Operations › Currently, the total number of components is 7. › 2 of components are the target of Canary Release. (Restart Canary Group => Restart All Servers)
19. # of Manual Operations Sharing Deployment Monitoring
20. # of Manual Operations Sharing Deployment Monitoring # of Sharing = 7 (Start) + 7 (Finish) + 2 (Canary Release) = 16 # of Deployment = 7 (Deploy) + 7 (Restart) + 2 (Canary Release) = 16 # of Monitoring = 7 + 2 (Canary Release) = 9 # of Total Manual Operations = 16 + 16 + 9 = 41 16 16 9
21. Objective › Reduce human resource : All member => ? › Prevent human mistake › Reduce manual operations : 41 => ?
22. How we automated
23. Automate Sharing › LINE OpenChat chatroom: OpenChat Bot Account & OpenChat Server API › LINE Group chatroom: LINE Notify › https://notify-bot.line.me/doc/en/
24. Sharing - LINE OpenChat
25. Sharing - LINE Group
26. Automate Deployment › Deployment API, Restart API, … › Use REST API provided by PMC
27. Automate Deployment PMC API Server Python Script Call Deployment API Call Restart API Polling to check task is finished Polling to check task is finished
28. Automate Monitoring?
29. How do we usually monitor?
30. How we monitor after release › System Layer: E-mail Notification › Service Layer: Check ERROR / WARN logs on IMON ElasticSearch Kibana › IMON: A system that monitors LINE services in an integrated manner
31. IMON ElasticSearch Kibana
32. How we monitor › System Layer: E-mail Notification › Service Layer: Check ERROR / WARN logs on IMON ElasticSearch Kibana
33. When do we detect the problem during monitoring?
34. A. When there are new error logs we’ve never seen before B. When the existing error log increases rapidly When we think the problem occurs
35. Automate Monitoring › Use IMON ElasticSearch API to collect error logs during specific period › Aggregate number of logs by error type › Compare the number of logs before release with after release
36. Automate Monitoring Suppose we want to collect error logs from DAY1 to DAY3 IMON ElasticSearch Python Script
37. Automate Monitoring IMON ElasticSearch Get error logs through search API Return result Python Script Suppose we want to collect error logs from DAY1 to DAY3 A : 20 B : 40 C : 10 A : 20 D : 15 A : 5 B : 10 E : 60 DAY1 DAY2 DAY3
38. Automate Monitoring Python Script Suppose we want to import error logs from DAY1 to DAY3 A: 20 B: 40 C: 10 A : 20 D : 15 A : 5 B : 10 E : 60 Aggregate by error type A : 45 B : 50 C : 10 D : 15 E : 60
39. Automate Monitoring Suppose that the deployment start time is T, A :45 B : 50 C : 10 D : 15 E : 60 A :20 B : 5 D : 2 (T - 10 Days) ~ T T ~ now
40. Automate Monitoring Suppose that the deployment start time is T, A : 20 B : 5 D : 2 F : 3 (T - 10 Days) ~ T T ~ now New error log A :45 B : 50 C : 10 D : 15 E : 60
41. Automate Monitoring Suppose that the deployment start time is T, A : 1 B : 250 C : 5 D : 2 (T - 10 Days) ~ T T ~ now Increased old error log A :45 B : 50 C : 10 D : 15 E : 60 50 < 250
42. Automate Monitoring New error logs? or Increased old logs? Finish deployment Finish monitoring Wait for 5 minutes Share errors to team chatroom Over 3 times? Y Y N N
43. Automate Monitoring
44. Integration Deployment Share Monitoring
45. Integration OpenChat Server API & LINE Notify PMC API iMON ElasticSearch API OpenChat Server API & LINE Notify Build Automated!
46. Result
47. Result › Weekly release? Just Click the “Build” button!
48. Before Human Mistake Possible Human Resource All members Manual Operations 41
49. After Human Mistake Impossible Human Resource Only Release Master Manual Operations 1
50. We are FREE!
51. Future Works › Automate Rollback › Strengthen Monitoring › Pipelining (ex. Jenkins Pipeline) › Introduce Interactive Bot (ex. Slack Bot)
52. To Be Continued…