gRPC service development in private Kubernetes cluster

Keiichiro Ui
LINE Development Team H Server Side Engineer
https://linedevday.linecorp.com/jp/2019/sessions/E1-1

Transcript

1. 2019 DevDay gRPC Service Development in Private Kubernetes Cluster > Keiichiro Ui > LINE Development Team H Server Side Engineer
2. Kubernetes User without Cloud Platforms?
3. About Me > Started with LINE LIVE projects > Developing new live commerce service > Leveraging new technology • With LINE development environments
4. Live Commerce Service with > LINE LIVE technologies > New Technologies
5. Why Use Kubernetes?
6. Traffic Spike in Live Cast Service > Requiring large scale out • For popular artists, performer, etc.
7. Scale Out Operations in LINE LIVE > Create VM instances > Run some setup scripts/ansible > Register with: • Deployment system • Metrics observer • etc. > Deploy > Register with load balancers
8. Why Use Kubernetes? With Kubernetes > Pod/Cluster autoscaler > Open • Ecosystem for OSS Existing Way > Manual scaling > Developed for LINE devs • Ecosystem for LINE services
9. Agenda > Kubernetes Service for LINE Services > Our Service Architecture Using gRPC > Service Metrics/Observations > Log Aggregation
10. IaaS/PaaS in LINE: Verda > Kubernetes > OpenStack > MySQL > Elasticsearch > Redis > Object Storage > LoadBalancer > etc.
11. Kubernetes for LINE Dev > Resources (Pod, Deployments, etc...) > Namespace > Kubernetes Cluster > OpenStack > Infrastructure + Log + Observation Service Dev Kubernetes Team
12. Agenda > Kubernetes Service for LINE Services > Our Service Architecture Using gRPC > Service Metrics/Observations > Log Aggregation
13. Overview > Microservices with gRPC > Spring Boot > Using Envoy to convert various protocols to gRPC
14. App Interactions > gRPC over Internet > L7 load balancer > NodePort as L4 load balancer > Envoy for L7 load balancer > Server application as headless service
15. Why Envoy? > Load balancing does not work appropriately • External LB cannot identify pod locations > NodePort works as L4 LB • TCP connections are long-lived in
 gRPC/HTTP2
16. Why Envoy?
17. Internal Interactions > Spring Boot > grpc-java • With headless service • Client-side load balancing
18. gRPC-Web > For JavaScript on web browsers • Admin system • Service landing pages > gRPC-Web supports any HTTP/* • no-dependency on HTTP/2
19. REST API > For other LINE services • Envoy gRPC-JSON transcoding > Protocol Buffers can be serialized into JSON
20. Why Avoid Istio? https://istio.io/
21. Why Avoid Istio? > Spring has the same functionalities • Metrics for traffics • Distributed tracing • Fault injection, Circuit breaker, Retry > Performance
22. Why Not Use Istio https://istio.io/docs/concepts/performance-and-scalability/ 13.7ms 1.89ms
23. Database > LINE has managed MySQL • with ACL
24. ACL Is Incompatible with Scaler
25. ACL Manager > API for managing DB ACL > Hooks autoscaler to add or delete a node
26. Agenda > Kubernetes Service for LINE Services > Our Service Architecture Using gRPC > Service Metrics/Observations > Log Aggregation
27. Prometheus > Placed in our Kubernetes cluster > Works as StatefulSet
28. Prometheus > Persistent Volume is hard to use > We try using a remote storage instead • Developed TSDB in-house
29. Too Many Metrics > Too many metrics in Prometheus with Kubernetes • How to visualize? • Which metrics should we watch? > Head time series: 240,638 Prometheus With Kubernetes
30. Too Many Metrics > kubernetes-mixin adds many settings • Grafana dashboards • Pod crush loop detection • Node hanging up • Volume usage predication for PersistentVolumes • etc. Prometheus with Kubernetes
31. Spring Application Metrics > Spring Boot Actuator > OpenCensus • Metrics • Distributed traces • Various Language
32. OpenCensus Built into gRPC OpenCensus > Metrics > Distributed tracing
33. Agenda > Kubernetes Service for LINE Services > Our Service Architecture Using gRPC > Service Metrics/Observations > Log Aggregation
34. Log Aggregations > Fluentd + Elasticsearch + Kibana • For generic logs > IMON • For error log alerts
35. Log Aggregation in Kubernetes Cluster > Old pod logs may be removed > Logs should be stored in external storage
36. EFK (Elasticsearch + Fluentd + Kibana)
37. Fluentd > Referred to • knative requirements • kubernetes addon • not for production • manifests in Helm Chart > plugins • fluent-plugin-elasticsearch • fluent-plugin-kubernetes_metadata_filter • fluent-plugin-detect-exceptions • fluent-plugin-multi-format-parser
38. IMON > Log monitoring system > Developed in-house > Sending alerts for warning/error logs
39. Agenda > Kubernetes Service for LINE Services > Our Service Architecture Using gRPC > Service Metrics/Observations > Log Aggregation
40. Our Booth > Kotlin coroutine > CI/CD with Drone CI + Kubernetes + Github > Development flow with sharing proto files with server/client
41. Related Sessions Kubernetes Client TSDB