LINE Fukuoka 2017.09 - Present • Experienced ◦ Develop/Operate In-house deployment/monitoring system ◦ DevOps/SRE-ish role • Current ◦ A member of private cloud "Verda" developers in LINE group 2
cloud service in LINE • Helpful services for LINE developers • Many kind of as-a-service ◦ EC2-like VM / Bare metal service ◦ Object Storage/CDN ◦ Databases (MySQL, Elasticsearch, Redis) ◦ Load balancer ◦ Kubernetes ◦ Heroku-ish service ◦ Functions a.k.a Serverless ◦ And more
Developers had “toils” to control the configurations. • e.g, URL routing • Update Server certs • Their own layer 7 for their own. Project Overview - Past in the day 4
Started on 2016 • Software-based load balancer • Containerized by Docker ◦ Linux namespace / cgroup • Orchestration by Docker Swarm ◦ Standalone mode (not swarm mode) ◦ Low cost of learning and development • Packet processing by XDP on L4 ◦ ソフトウェアでのパケット処理あれこれ〜何故我々はロードバランサを自作す るに至ったのか〜 13
Swarm ◦ Auto-Scalability of container ▪ No support out of the box ◦ Docker Integrated with Kubernetes ▪ Docker captain said “Swarm is alive and well.” but… • Implementation ◦ 1VIP in 1Container ◦ Resource efficiency issue again
VIPs in a single container as many as possible ◦ Configure like as Virtual Host • Noisy neighbor? ◦ Deploy many pods in low cost machine ◦ Incoming traffic is supposed to be balnaced 18 Before After
◦ In progress... • Difficulties ◦ Handle unpredicted situation ◦ Lightning fast scaling ◦ “Graceful” shutdown ▪ Graceful…? • Keep connection even when the Pod is going down • Share communication resources such as socket…? 19
Japan ▪ Google Cloud Platform • “Pure” L3 network ◦ No overlay ◦ BGP based IP routing • Direct reachability to Pods from out of cluster Project Story - Phase 2. Kubernetes - Calico 21
Better accommodations ◦ Virtual Host approach • (In progress) Enable to auto-scale ◦ Need more time... • (Solve) Migration from Docker Swarm to Kubernetes ◦ Direct connectivity to Pods by Calico • New challenges ◦ Intelligent resource scheduling ◦ Better communication between L4 and L7 ◦ Graceful upstream draining ◦ and more... 22