Sharing experience of LINE Infrastructure

Younghee Park
LINE IT Service Center Executive Officer
https://linedevday.linecorp.com/2020/ja/sessions/1830
https://linedevday.linecorp.com/2020/en/sessions/1830

2. Park Younghee LINE Executive Officer Head of ITService Center
3. COVID-19 is redefining IT infrastructure
4. COVID-19 is redefining IT infrastructure Our preparation was “not enough”
5. LINE User 0 50000 100000 150000 200000 250000 300000 350000 400000 450000 2011/6/1 2011/08/23/ 2011/10/14/ 2011/10/17/ 2011/11/08/ 2011/11/29/ 2012/01/27/ 2012/03/05/ 2012/03/27/ 2012/04/18/ 2012/06/06/ 2012/10/25/ 2012/11/30/ 2013/05/01/ 2013/11/25/ 2014/04/02/ ※The number of downloads until June 2011, and the number of registered users after March 2012. Winter is coming
6. LINE MAU ※MAU (4 major countries) 0 10000 20000 30000 40000 50000 60000 70000 80000 90000 2014.2Q 2014.4Q 2015.2Q 2015.4Q 2016.2Q 2016.4Q 2017.2Q 2017.4Q 2018.2Q 2018.4Q 2019.2Q 2019.4Q 2020.2Q Network x 6 CDN x 12 Server x 5 Storage x 4 IDC JP*2, SG, US, DE, KR, TW*2 Office x3 Infrastructure
7. Speed Global Scale
8. • Problem: difficulties of understanding current status of infrastructure Period of rapid growth of messaging
9. • Problem: difficulties of understanding current status of infrastructure • LINE engineers need to understand event alarm, server fault latency of database, long query traffic of L4/L7 load balance DNS setting, certificate setting blocking by network, firewall ACL inter IDC traffic, ISP traffic Period of rapid growth of messaging
10. Infrastructure Information Dashboard - Infra Management Console (IMC)
11. Infrastructure Information Dashboard - Infra Management Console (IMC) Network Traffic CMDB Server Monitoring SSL Cert Load Balancer IMC DB Monitoring ACL
12. Transparency of information is the start of collaboration.
13. • Problem: late delivery, operation in large scale Period of emergence of new services as Life Portal
14. • Problem: late delivery, operation in large scale We decided changing workflow to cloud • OpenStack maturity • Change in motivation Work as a developer to solve user’s problem Implement new features to user Provide values to end users and App developers • Cost consideration Period of emergence of new services as Life Portal
15. vm baremetal SSO OpenStack IaaS
16. vm baremetal SSO OpenStack IaaS
17. Verda is a Private Cloud for LINE is based on OpenStack. since 2016~ FaaS PaaS IaaS NAT Load Balancer Baremetal
18. Verda is a Private Cloud for LINE is based on OpenStack. since 2016~ FaaS PaaS IaaS NAT Load Balancer Baremetal Hypervisor 2,000+ Virtual Machine 55,000+ Physical Machine 20,000+
19. Verda is a Private Cloud for LINE
20. Verda is a Private Cloud for LINE
21. Easy way to save infrastructure cost
22. Easy way to save infrastructure cost Define low usage server Set about 15% line by cpu, mem, filesystem, network 1 High utilization 85% low utilization 15% Low usage server All barematal & vm
23. Easy way to save infrastructure cost Define low usage server Set about 15% line by cpu, mem, filesystem, network 1 Decrease low usage servers • return • baremetal à vm • Spec down • containerizing • change architecture 2 High utilization 85% low utilization 15% Low usage server All barematal & vm
24. Easy way to save infrastructure cost Define low usage server Set about 15% line by cpu, mem, filesystem, network 1 Decrease low usage servers • return • baremetal à vm • Spec down • containerizing • change architecture 2 Strict standard when low usage server < 10% 3 High utilization 85% low utilization 15% Low usage server All barematal & vm
25. Next challenge : Productivity in both of Dev, Infrastructure
26. Next challenge : Productivity in both of Dev, Infrastructure • Easily movable application (operation) • Declarative use of infrastructure & platform (ease of use) • Utilize resources of entire datacenter (cost optimization)
27. Technology needs to understand the business
28. Project Plano Physical/Virtual Machines Applications Development Application Operation Deployment Infrastructure Operations App Developers Infrastructure Operators AS-IS Responsibilities
29. Project Plano Resource Abstraction Physical/Virtual Machines Applications TO-BE Responsibilities Physical/Virtual Machines Applications Development Application Operation Deployment Infrastructure Operations App Developers Infrastructure Operators AS-IS Responsibilities
30. Responsibility as a social infrastructure • BCP/DR • IPv6
31. BCP/DR “text messaging”
32. 1st IDC 2nd IDC 7ms, 500km Over 100 Gbps Message front Message backend Redis Hbase Message front internet SSL VPN DNS NTP Kerberos JIRA Wiki Git
33. 1st IDC 2nd IDC 7ms, 500km Over 100 Gbps Message front Message backend Redis Hbase Message front internet SSL VPN DNS NTP Kerberos JIRA Wiki Git SSL VPN DNS NTP Kerberos JIRA Wiki Git
34. 1st IDC 2nd IDC 7ms, 500km Over 100 Gbps Message front Message backend Redis Hbase Message front internet SSL VPN DNS NTP Kerberos JIRA Wiki Git SSL VPN DNS NTP Kerberos JIRA Wiki Git
35. “text messaging” but 3hour “Zero time, all messaging”
36. “text messaging” but 3hour • Multi IDC • Active / Active • Failback “Zero time, all messaging”
37. Current LINE’s IPv6 DNS64 NAT64 IPv6 IPv4
38. 2021 LINE will support “server” side IPv6
39. Thank you