gRPC service development in private Kubernetes cluster

gRPC service development in private Kubernetes cluster

Keiichiro Ui
LINE Development Team H Server Side Engineer
https://linedevday.linecorp.com/jp/2019/sessions/E1-1

Be4518b119b8eb017625e0ead20f8fe7?s=128

LINE DevDay 2019

November 20, 2019
Tweet

Transcript

  1. 1.

    2019 DevDay gRPC Service Development in Private Kubernetes Cluster >

    Keiichiro Ui > LINE Development Team H Server Side Engineer
  2. 3.

    About Me > Started with LINE LIVE projects > Developing

    new live commerce service > Leveraging new technology • With LINE development environments
  3. 6.

    Traffic Spike in Live Cast Service > Requiring large scale

    out • For popular artists, performer, etc.
  4. 7.

    Scale Out Operations in LINE LIVE > Create VM instances

    > Run some setup scripts/ansible > Register with: • Deployment system • Metrics observer • etc. > Deploy > Register with load balancers
  5. 8.

    Why Use Kubernetes? With Kubernetes > Pod/Cluster autoscaler > Open

    • Ecosystem for OSS Existing Way > Manual scaling > Developed for LINE devs • Ecosystem for LINE services
  6. 9.

    Agenda > Kubernetes Service for LINE Services > Our Service

    Architecture Using gRPC > Service Metrics/Observations > Log Aggregation
  7. 10.

    IaaS/PaaS in LINE: Verda > Kubernetes > OpenStack > MySQL

    > Elasticsearch > Redis > Object Storage > LoadBalancer > etc.
  8. 11.

    Kubernetes for LINE Dev > Resources (Pod, Deployments, etc...) >

    Namespace > Kubernetes Cluster > OpenStack > Infrastructure + Log + Observation Service Dev Kubernetes Team
  9. 12.

    Agenda > Kubernetes Service for LINE Services > Our Service

    Architecture Using gRPC > Service Metrics/Observations > Log Aggregation
  10. 13.

    Overview > Microservices with gRPC > Spring Boot > Using

    Envoy to convert various protocols to gRPC
  11. 14.

    App Interactions > gRPC over Internet > L7 load balancer

    > NodePort as L4 load balancer > Envoy for L7 load balancer > Server application as headless service
  12. 15.

    Why Envoy? > Load balancing does not work appropriately •

    External LB cannot identify pod locations > NodePort works as L4 LB • TCP connections are long-lived in
 gRPC/HTTP2
  13. 17.
  14. 18.

    gRPC-Web > For JavaScript on web browsers • Admin system

    • Service landing pages > gRPC-Web supports any HTTP/* • no-dependency on HTTP/2
  15. 19.

    REST API > For other LINE services • Envoy gRPC-JSON

    transcoding > Protocol Buffers can be serialized into JSON
  16. 21.

    Why Avoid Istio? > Spring has the same functionalities •

    Metrics for traffics • Distributed tracing • Fault injection, Circuit breaker, Retry > Performance
  17. 25.

    ACL Manager > API for managing DB ACL > Hooks

    autoscaler to add or delete a node
  18. 26.

    Agenda > Kubernetes Service for LINE Services > Our Service

    Architecture Using gRPC > Service Metrics/Observations > Log Aggregation
  19. 28.

    Prometheus > Persistent Volume is hard to use > We

    try using a remote storage instead • Developed TSDB in-house
  20. 29.

    Too Many Metrics > Too many metrics in Prometheus with

    Kubernetes • How to visualize? • Which metrics should we watch? > Head time series: 240,638 Prometheus With Kubernetes
  21. 30.

    Too Many Metrics > kubernetes-mixin adds many settings • Grafana

    dashboards • Pod crush loop detection • Node hanging up • Volume usage predication for PersistentVolumes • etc. Prometheus with Kubernetes
  22. 31.

    Spring Application Metrics > Spring Boot Actuator > OpenCensus •

    Metrics • Distributed traces • Various Language
  23. 33.

    Agenda > Kubernetes Service for LINE Services > Our Service

    Architecture Using gRPC > Service Metrics/Observations > Log Aggregation
  24. 34.

    Log Aggregations > Fluentd + Elasticsearch + Kibana • For

    generic logs > IMON • For error log alerts
  25. 35.

    Log Aggregation in Kubernetes Cluster > Old pod logs may

    be removed > Logs should be stored in external storage
  26. 37.

    Fluentd > Referred to • knative requirements • kubernetes addon

    • not for production • manifests in Helm Chart > plugins • fluent-plugin-elasticsearch • fluent-plugin-kubernetes_metadata_filter • fluent-plugin-detect-exceptions • fluent-plugin-multi-format-parser
  27. 39.

    Agenda > Kubernetes Service for LINE Services > Our Service

    Architecture Using gRPC > Service Metrics/Observations > Log Aggregation
  28. 40.

    Our Booth > Kotlin coroutine > CI/CD with Drone CI

    + Kubernetes + Github > Development flow with sharing proto files with server/client