Introduction of software engineer(Infra Platform)

Luke
LINE Developer Recruitment Day at Taiwan #1
https://engineering.linecorp.com/zh-hant/blog/2019-line-taiwan-technical-recruitment-day/a

1. Introduction of "Software Engineer (Infrastructure)” LINE Corporation Luke
2. About me • Name: Luke • 2011 ~ 2015 Devops engineer at Rakuten • 2016/01 ~ Cloud engineer at LINE • Baremetal • K8s
3. Service Architecture Redis HBase Talkserver Object Storage Image, File, Video VoIP BOT Japan,Taiwan Users Southeast Asia Users North America Users Gateway Gateway Gateway JP SG US
4. Physical Infrastructure Internet Datacenter Datacenter Server Datacenter Network Datacenter Interconnect Rack Cable Load Balancer Router
5. LINE Infrastructure Scale • 1Tbps+ User Traffic • 40,000+ Servers • Multiple datacenters in 5+ countries
6. Responsibilities Physical Infrastructure Infrastructure Platform Application Application Application Application Application Application Application Application • Private Cloud • IaaS • Managed Service • PaaS API CLI WebUI
7. Private Cloud Components Network Controller VM Baremetal DNS Load Balancer Object Storage Block Storage Managed Database Managed Pub/Sub Managed Kubernetes Nucleo FaaS IaaS Managed Service PaaS OpenStack Kubernetes
8. BEFORE KAAS VM Kubernetes Kubernetes Container Container Container Container Container Container Container Container Container Container Container Container Developers A in Japan Developers B in Taiwan Private Cloud Private Cloud Developers Application Developer OS VM OS VM OS BM OS BM OS BM OS IaaS Responsibility border
9. ! Different Toolset ◦ Different Version ◦ Different Add-on ◦ Different Plugin ! Distributing Knowledge PROBLEMS WE FACED ! Large Operation Cost ◦ Deploy Cluster ◦ Extend Cluster ◦ Troubleshooting Cluster ! Quality of Cluster ◦ No HA Configuration ◦ No Performance Consideration ◦ No Private Cloud Collaboration From Application Developer point of view From Company-wide point of view
10. For more than 2000 developers ( 100+ clusters ) KaaS Kubernetes Operator Kubernetes Cluster Performance High Availability Private Cloud Collaboration OUR MISSION
11. Kubernetes Operator Kubernetes Cluster Performance High Availability Private Cloud Collaboration Performance Consideration
12. VM Network (L3 Reachability ensured) Kubernetes Cluster VM VM VM vxlan Container Network Private Cloud Agnostic Encapsulate Overhead Container network case
13. VM Network (L3 Reachability ensured) Kubernetes Cluster vxlan Container Network Less encapsulate overhead Build container network with performance optimization Tenant Network For Container (L2 Reachability ensured) VM VM VM host-gw VXLAN Offload Container network case
14. Kubernetes Operator Kubernetes Cluster Performance High Availability Private Cloud Collaboration HA Consideration
15. etcd etcd etcd 3 •etcd 1 controller controller controller 3 •kube-apiserver •kube-controller-manager •kube-scheduler •kubelet •kube-proxy × × × 2 worker worker worker N •kubelet •kube-proxy - Toleration Limit Our Deployment HA Design
16. etcd etcd etcd controller controller controller worker worker worker × × × etcd controller controller When we detect VM shutdown/error in Private Cloud, Unavailable node will be deleted from cluster and new node will be added when the number of unavailable node is lower than toleration limit Delete Delete Delete New New New Replace Solution
17. Snapshot of etcd Keep at least 2 days etcd etcd etcd 14:30 14:45 etcd etcd etcd Snapshot as of 14:30 14:40 etcd etcd etcd × × × Object Storage New New New Kubernetes Cluster Kubernetes Cluster Kubernetes Cluster Backup And Restore
18. Kubernetes Operator Kubernetes Cluster Performance High Availability Private Cloud Collaboration Private Cloud Collaboration
19. DNS Pod Pod Create/Configure Pass access information Kubernetes Cluster Use Cloud Services from k8s
20. DNS Pod Pod kind: Redis spec: mode: sentinel node: normal kind: Service ==== Redis Instance ==== kind: Secret ==== Redis Credential ==== Kubernetes Cluster Develop Add-on: Custom Controller for Managed Service Custom Controller for Managed Service Make cloud services available inside k8s
21. csi-cinder-provisioner csi-attacher-cinderplugin csi-nodeplugin-cinderplugin csi-nodeplugin-cinderplugin csi-nodeplugin-cinderplugin kind: StorageClass metadata: name: cinder-ssd provisioner: csi-cinderplugin Create volume Attach volume to VM kind: PersistentVolumeClaim spec: storageClassName: cinder-ssd resources: requests: storage: 1G Verified Add-on: Cinder Standalone Persistent Volume Provider Kubernetes Cluster Support persistent volume
22. Custom Controller for Type Loadbalancer In-house Loadbalancer kind: Service spec: selector: app: nginx ports: - protocol: TCP port: 80 targetPort: 80 nodePort: 30000 type: LoadBalancer Create New VIP Add Real Server Develop Add-on: Custom Controller for in-house LB Kubernetes Cluster Implement type load balancer
23. Cluster Operation - Cluster Create - Cluster Update - Add Worker Use Cluster - Deploy application - Scale application Manage Cluster - Deploy - Monitor - Update Private Cloud User Automate Operating Multiple Cluster API Kubernetes Kubernetes Kubernetes Architecture Overview
24. Jobs • Kubernetes Platform • https://linecorp.com/ja/career/position/1358 • Private Cloud Platform (OpenStack) • https://linecorp.com/ja/career/position/827
25. Jobs in Detail • recent & future challenges • https://speakerdeck.com/line_developers/introduction-of-private-cloud-in-line • cloud native challenges • https://www.slideshare.net/linecorp/lines-private-cloud-meet-cloud-native-world
26. Thank You