gRPC service development in private Kubernetes cluster

gRPC service development in private Kubernetes cluster

Keiichiro Ui
LINE Development Team H Server Side Engineer
https://linedevday.linecorp.com/jp/2019/sessions/E1-1

Be4518b119b8eb017625e0ead20f8fe7?s=128

LINE DevDay 2019

November 20, 2019
Tweet

Transcript

  1. 2019 DevDay gRPC Service Development in Private Kubernetes Cluster >

    Keiichiro Ui > LINE Development Team H Server Side Engineer
  2. Kubernetes User without Cloud Platforms?

  3. About Me > Started with LINE LIVE projects > Developing

    new live commerce service > Leveraging new technology • With LINE development environments
  4. Live Commerce Service with > LINE LIVE technologies > New

    Technologies
  5. Why Use Kubernetes?

  6. Traffic Spike in Live Cast Service > Requiring large scale

    out • For popular artists, performer, etc.
  7. Scale Out Operations in LINE LIVE > Create VM instances

    > Run some setup scripts/ansible > Register with: • Deployment system • Metrics observer • etc. > Deploy > Register with load balancers
  8. Why Use Kubernetes? With Kubernetes > Pod/Cluster autoscaler > Open

    • Ecosystem for OSS Existing Way > Manual scaling > Developed for LINE devs • Ecosystem for LINE services
  9. Agenda > Kubernetes Service for LINE Services > Our Service

    Architecture Using gRPC > Service Metrics/Observations > Log Aggregation
  10. IaaS/PaaS in LINE: Verda > Kubernetes > OpenStack > MySQL

    > Elasticsearch > Redis > Object Storage > LoadBalancer > etc.
  11. Kubernetes for LINE Dev > Resources (Pod, Deployments, etc...) >

    Namespace > Kubernetes Cluster > OpenStack > Infrastructure + Log + Observation Service Dev Kubernetes Team
  12. Agenda > Kubernetes Service for LINE Services > Our Service

    Architecture Using gRPC > Service Metrics/Observations > Log Aggregation
  13. Overview > Microservices with gRPC > Spring Boot > Using

    Envoy to convert various protocols to gRPC
  14. App Interactions > gRPC over Internet > L7 load balancer

    > NodePort as L4 load balancer > Envoy for L7 load balancer > Server application as headless service
  15. Why Envoy? > Load balancing does not work appropriately •

    External LB cannot identify pod locations > NodePort works as L4 LB • TCP connections are long-lived in
 gRPC/HTTP2
  16. Why Envoy?

  17. Internal Interactions > Spring Boot > grpc-java • With headless

    service • Client-side load balancing
  18. gRPC-Web > For JavaScript on web browsers • Admin system

    • Service landing pages > gRPC-Web supports any HTTP/* • no-dependency on HTTP/2
  19. REST API > For other LINE services • Envoy gRPC-JSON

    transcoding > Protocol Buffers can be serialized into JSON
  20. Why Avoid Istio? https://istio.io/

  21. Why Avoid Istio? > Spring has the same functionalities •

    Metrics for traffics • Distributed tracing • Fault injection, Circuit breaker, Retry > Performance
  22. Why Not Use Istio https://istio.io/docs/concepts/performance-and-scalability/ 13.7ms 1.89ms

  23. Database > LINE has managed MySQL • with ACL

  24. ACL Is Incompatible with Scaler

  25. ACL Manager > API for managing DB ACL > Hooks

    autoscaler to add or delete a node
  26. Agenda > Kubernetes Service for LINE Services > Our Service

    Architecture Using gRPC > Service Metrics/Observations > Log Aggregation
  27. Prometheus > Placed in our Kubernetes cluster > Works as

    StatefulSet
  28. Prometheus > Persistent Volume is hard to use > We

    try using a remote storage instead • Developed TSDB in-house
  29. Too Many Metrics > Too many metrics in Prometheus with

    Kubernetes • How to visualize? • Which metrics should we watch? > Head time series: 240,638 Prometheus With Kubernetes
  30. Too Many Metrics > kubernetes-mixin adds many settings • Grafana

    dashboards • Pod crush loop detection • Node hanging up • Volume usage predication for PersistentVolumes • etc. Prometheus with Kubernetes
  31. Spring Application Metrics > Spring Boot Actuator > OpenCensus •

    Metrics • Distributed traces • Various Language
  32. OpenCensus Built into gRPC OpenCensus > Metrics > Distributed tracing

  33. Agenda > Kubernetes Service for LINE Services > Our Service

    Architecture Using gRPC > Service Metrics/Observations > Log Aggregation
  34. Log Aggregations > Fluentd + Elasticsearch + Kibana • For

    generic logs > IMON • For error log alerts
  35. Log Aggregation in Kubernetes Cluster > Old pod logs may

    be removed > Logs should be stored in external storage
  36. EFK (Elasticsearch + Fluentd + Kibana)

  37. Fluentd > Referred to • knative requirements • kubernetes addon

    • not for production • manifests in Helm Chart > plugins • fluent-plugin-elasticsearch • fluent-plugin-kubernetes_metadata_filter • fluent-plugin-detect-exceptions • fluent-plugin-multi-format-parser
  38. IMON > Log monitoring system > Developed in-house > Sending

    alerts for warning/error logs
  39. Agenda > Kubernetes Service for LINE Services > Our Service

    Architecture Using gRPC > Service Metrics/Observations > Log Aggregation
  40. Our Booth > Kotlin coroutine > CI/CD with Drone CI

    + Kubernetes + Github > Development flow with sharing proto files with server/client
  41. Related Sessions Kubernetes Client TSDB