Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introduction of software engineer(Infra Platform)

Introduction of software engineer(Infra Platform)

53850955f15249a1a9dc49df6113e400?s=128

LINE Developers
PRO

May 18, 2019
Tweet

Transcript

  1. Introduction of "Software Engineer (Infrastructure)” LINE Corporation Luke

  2. About me • Name: Luke • 2011 ~ 2015 Devops

    engineer at Rakuten • 2016/01 ~ Cloud engineer at LINE • Baremetal • K8s
  3. Service Architecture Redis HBase Talkserver Object Storage Image, File, Video

    VoIP BOT Japan,Taiwan Users Southeast Asia Users North America Users Gateway Gateway Gateway JP SG US
  4. Physical Infrastructure Internet Datacenter Datacenter Server Datacenter Network Datacenter Interconnect

    Rack Cable Load Balancer Router
  5. LINE Infrastructure Scale • 1Tbps+ User Traffic • 40,000+ Servers

    • Multiple datacenters in 5+ countries
  6. Responsibilities Physical Infrastructure Infrastructure Platform Application Application Application Application Application

    Application Application Application • Private Cloud • IaaS • Managed Service • PaaS API CLI WebUI
  7. Private Cloud Components Network Controller VM Baremetal DNS Load Balancer

    Object Storage Block Storage Managed Database Managed Pub/Sub Managed Kubernetes Nucleo FaaS IaaS Managed Service PaaS OpenStack Kubernetes
  8. BEFORE KAAS VM Kubernetes Kubernetes Container Container Container Container Container

    Container Container Container Container Container Container Container Developers A in Japan Developers B in Taiwan Private Cloud Private Cloud Developers Application Developer OS VM OS VM OS BM OS BM OS BM OS IaaS Responsibility border
  9. ! Different Toolset ◦ Different Version ◦ Different Add-on ◦

    Different Plugin ! Distributing Knowledge PROBLEMS WE FACED ! Large Operation Cost ◦ Deploy Cluster ◦ Extend Cluster ◦ Troubleshooting Cluster ! Quality of Cluster ◦ No HA Configuration ◦ No Performance Consideration ◦ No Private Cloud Collaboration From Application Developer point of view From Company-wide point of view
  10. For more than 2000 developers ( 100+ clusters ) KaaS

    Kubernetes Operator Kubernetes Cluster Performance High Availability Private Cloud Collaboration OUR MISSION
  11. Kubernetes Operator Kubernetes Cluster Performance High Availability Private Cloud Collaboration

    Performance Consideration
  12. VM Network (L3 Reachability ensured) Kubernetes Cluster VM VM VM

    vxlan Container Network Private Cloud Agnostic Encapsulate Overhead Container network case
  13. VM Network (L3 Reachability ensured) Kubernetes Cluster vxlan Container Network

    Less encapsulate overhead Build container network with performance optimization Tenant Network For Container (L2 Reachability ensured) VM VM VM host-gw VXLAN Offload Container network case
  14. Kubernetes Operator Kubernetes Cluster Performance High Availability Private Cloud Collaboration

    HA Consideration
  15. etcd etcd etcd 3 •etcd 1 controller controller controller 3

    •kube-apiserver •kube-controller-manager •kube-scheduler •kubelet •kube-proxy × × × 2 worker worker worker N •kubelet •kube-proxy - Toleration Limit Our Deployment HA Design
  16. etcd etcd etcd controller controller controller worker worker worker ×

    × × etcd controller controller When we detect VM shutdown/error in Private Cloud, Unavailable node will be deleted from cluster and new node will be added when the number of unavailable node is lower than toleration limit Delete Delete Delete New New New Replace Solution
  17. Snapshot of etcd Keep at least 2 days etcd etcd

    etcd 14:30 14:45 etcd etcd etcd Snapshot as of 14:30 14:40 etcd etcd etcd × × × Object Storage New New New Kubernetes Cluster Kubernetes Cluster Kubernetes Cluster Backup And Restore
  18. Kubernetes Operator Kubernetes Cluster Performance High Availability Private Cloud Collaboration

    Private Cloud Collaboration
  19. DNS Pod Pod Create/Configure Pass access information Kubernetes Cluster Use

    Cloud Services from k8s
  20. DNS Pod Pod kind: Redis spec: mode: sentinel node: normal

    kind: Service ==== Redis Instance ==== kind: Secret ==== Redis Credential ==== Kubernetes Cluster Develop Add-on: Custom Controller for Managed Service Custom Controller for Managed Service Make cloud services available inside k8s
  21. csi-cinder-provisioner csi-attacher-cinderplugin csi-nodeplugin-cinderplugin csi-nodeplugin-cinderplugin csi-nodeplugin-cinderplugin kind: StorageClass metadata: name: cinder-ssd

    provisioner: csi-cinderplugin Create volume Attach volume to VM kind: PersistentVolumeClaim spec: storageClassName: cinder-ssd resources: requests: storage: 1G Verified Add-on: Cinder Standalone Persistent Volume Provider Kubernetes Cluster Support persistent volume
  22. Custom Controller for Type Loadbalancer In-house Loadbalancer kind: Service spec:

    selector: app: nginx ports: - protocol: TCP port: 80 targetPort: 80 nodePort: 30000 type: LoadBalancer Create New VIP Add Real Server Develop Add-on: Custom Controller for in-house LB Kubernetes Cluster Implement type load balancer
  23. Cluster Operation - Cluster Create - Cluster Update - Add

    Worker Use Cluster - Deploy application - Scale application Manage Cluster - Deploy - Monitor - Update Private Cloud User Automate Operating Multiple Cluster API Kubernetes Kubernetes Kubernetes Architecture Overview
  24. Jobs • Kubernetes Platform • https://linecorp.com/ja/career/position/1358 • Private Cloud Platform

    (OpenStack) • https://linecorp.com/ja/career/position/827
  25. Jobs in Detail • recent & future challenges • https://speakerdeck.com/line_developers/introduction-of-private-cloud-in-line

    • cloud native challenges • https://www.slideshare.net/linecorp/lines-private-cloud-meet-cloud-native-world
  26. Thank You