Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Sharing experience of LINE Infrastructure

Sharing experience of LINE Infrastructure

Eebedc2ee7ff95ffb9d9102c6d4a065c?s=128

LINE DevDay 2020

November 25, 2020
Tweet

Transcript

  1. None
  2. Park Younghee LINE Executive Officer Head of ITService Center

  3. COVID-19 is redefining IT infrastructure

  4. COVID-19 is redefining IT infrastructure Our preparation was “not enough”

  5. LINE User 0 50000 100000 150000 200000 250000 300000 350000

    400000 450000 2011/6/1 2011/08/23/ 2011/10/14/ 2011/10/17/ 2011/11/08/ 2011/11/29/ 2012/01/27/ 2012/03/05/ 2012/03/27/ 2012/04/18/ 2012/06/06/ 2012/10/25/ 2012/11/30/ 2013/05/01/ 2013/11/25/ 2014/04/02/ ※The number of downloads until June 2011, and the number of registered users after March 2012. Winter is coming
  6. LINE MAU ※MAU (4 major countries) 0 10000 20000 30000

    40000 50000 60000 70000 80000 90000 2014.2Q 2014.4Q 2015.2Q 2015.4Q 2016.2Q 2016.4Q 2017.2Q 2017.4Q 2018.2Q 2018.4Q 2019.2Q 2019.4Q 2020.2Q Network x 6 CDN x 12 Server x 5 Storage x 4 IDC JP*2, SG, US, DE, KR, TW*2 Office x3 Infrastructure
  7. Speed Global Scale

  8. • Problem: difficulties of understanding current status of infrastructure Period

    of rapid growth of messaging
  9. • Problem: difficulties of understanding current status of infrastructure •

    LINE engineers need to understand event alarm, server fault latency of database, long query traffic of L4/L7 load balance DNS setting, certificate setting blocking by network, firewall ACL inter IDC traffic, ISP traffic Period of rapid growth of messaging
  10. Infrastructure Information Dashboard - Infra Management Console (IMC)

  11. Infrastructure Information Dashboard - Infra Management Console (IMC) Network Traffic

    CMDB Server Monitoring SSL Cert Load Balancer IMC DB Monitoring ACL
  12. Transparency of information is the start of collaboration.

  13. • Problem: late delivery, operation in large scale Period of

    emergence of new services as Life Portal
  14. • Problem: late delivery, operation in large scale We decided

    changing workflow to cloud • OpenStack maturity • Change in motivation Work as a developer to solve user’s problem Implement new features to user Provide values to end users and App developers • Cost consideration Period of emergence of new services as Life Portal
  15. vm baremetal SSO OpenStack IaaS

  16. vm baremetal SSO OpenStack IaaS

  17. Verda is a Private Cloud for LINE is based on

    OpenStack. since 2016~ FaaS PaaS IaaS NAT Load Balancer Baremetal
  18. Verda is a Private Cloud for LINE is based on

    OpenStack. since 2016~ FaaS PaaS IaaS NAT Load Balancer Baremetal Hypervisor 2,000+ Virtual Machine 55,000+ Physical Machine 20,000+
  19. Verda is a Private Cloud for LINE

  20. Verda is a Private Cloud for LINE

  21. Easy way to save infrastructure cost

  22. Easy way to save infrastructure cost Define low usage server

    Set about 15% line by cpu, mem, filesystem, network 1 High utilization 85% low utilization 15% Low usage server All barematal & vm
  23. Easy way to save infrastructure cost Define low usage server

    Set about 15% line by cpu, mem, filesystem, network 1 Decrease low usage servers • return • baremetal à vm • Spec down • containerizing • change architecture 2 High utilization 85% low utilization 15% Low usage server All barematal & vm
  24. Easy way to save infrastructure cost Define low usage server

    Set about 15% line by cpu, mem, filesystem, network 1 Decrease low usage servers • return • baremetal à vm • Spec down • containerizing • change architecture 2 Strict standard when low usage server < 10% 3 High utilization 85% low utilization 15% Low usage server All barematal & vm
  25. Next challenge : Productivity in both of Dev, Infrastructure

  26. Next challenge : Productivity in both of Dev, Infrastructure •

    Easily movable application (operation) • Declarative use of infrastructure & platform (ease of use) • Utilize resources of entire datacenter (cost optimization)
  27. Technology needs to understand the business

  28. Project Plano Physical/Virtual Machines Applications Development Application Operation Deployment Infrastructure

    Operations App Developers Infrastructure Operators AS-IS Responsibilities
  29. Project Plano Resource Abstraction Physical/Virtual Machines Applications TO-BE Responsibilities Physical/Virtual

    Machines Applications Development Application Operation Deployment Infrastructure Operations App Developers Infrastructure Operators AS-IS Responsibilities
  30. Responsibility as a social infrastructure • BCP/DR • IPv6

  31. BCP/DR “text messaging”

  32. 1st IDC 2nd IDC 7ms, 500km Over 100 Gbps Message

    front Message backend Redis Hbase Message front internet SSL VPN DNS NTP Kerberos JIRA Wiki Git
  33. 1st IDC 2nd IDC 7ms, 500km Over 100 Gbps Message

    front Message backend Redis Hbase Message front internet SSL VPN DNS NTP Kerberos JIRA Wiki Git SSL VPN DNS NTP Kerberos JIRA Wiki Git
  34. 1st IDC 2nd IDC 7ms, 500km Over 100 Gbps Message

    front Message backend Redis Hbase Message front internet SSL VPN DNS NTP Kerberos JIRA Wiki Git SSL VPN DNS NTP Kerberos JIRA Wiki Git
  35. “text messaging” but 3hour “Zero time, all messaging”

  36. “text messaging” but 3hour • Multi IDC • Active /

    Active • Failback “Zero time, all messaging”
  37. Current LINE’s IPv6 DNS64 NAT64 IPv6 IPv4

  38. 2021 LINE will support “server” side IPv6

  39. Thank you