Upgrade to Pro — share decks privately, control downloads, hide ads and more …

gRPC service development in private Kubernetes cluster

gRPC service development in private Kubernetes cluster

Keiichiro Ui
LINE Development Team H Server Side Engineer
https://linedevday.linecorp.com/jp/2019/sessions/E1-1

LINE DevDay 2019

November 20, 2019
Tweet

More Decks by LINE DevDay 2019

Other Decks in Technology

Transcript

  1. 2019 DevDay gRPC Service Development in Private Kubernetes Cluster >

    Keiichiro Ui > LINE Development Team H Server Side Engineer
  2. About Me > Started with LINE LIVE projects > Developing

    new live commerce service > Leveraging new technology • With LINE development environments
  3. Traffic Spike in Live Cast Service > Requiring large scale

    out • For popular artists, performer, etc.
  4. Scale Out Operations in LINE LIVE > Create VM instances

    > Run some setup scripts/ansible > Register with: • Deployment system • Metrics observer • etc. > Deploy > Register with load balancers
  5. Why Use Kubernetes? With Kubernetes > Pod/Cluster autoscaler > Open

    • Ecosystem for OSS Existing Way > Manual scaling > Developed for LINE devs • Ecosystem for LINE services
  6. Agenda > Kubernetes Service for LINE Services > Our Service

    Architecture Using gRPC > Service Metrics/Observations > Log Aggregation
  7. IaaS/PaaS in LINE: Verda > Kubernetes > OpenStack > MySQL

    > Elasticsearch > Redis > Object Storage > LoadBalancer > etc.
  8. Kubernetes for LINE Dev > Resources (Pod, Deployments, etc...) >

    Namespace > Kubernetes Cluster > OpenStack > Infrastructure + Log + Observation Service Dev Kubernetes Team
  9. Agenda > Kubernetes Service for LINE Services > Our Service

    Architecture Using gRPC > Service Metrics/Observations > Log Aggregation
  10. Overview > Microservices with gRPC > Spring Boot > Using

    Envoy to convert various protocols to gRPC
  11. App Interactions > gRPC over Internet > L7 load balancer

    > NodePort as L4 load balancer > Envoy for L7 load balancer > Server application as headless service
  12. Why Envoy? > Load balancing does not work appropriately •

    External LB cannot identify pod locations > NodePort works as L4 LB • TCP connections are long-lived in
 gRPC/HTTP2
  13. gRPC-Web > For JavaScript on web browsers • Admin system

    • Service landing pages > gRPC-Web supports any HTTP/* • no-dependency on HTTP/2
  14. REST API > For other LINE services • Envoy gRPC-JSON

    transcoding > Protocol Buffers can be serialized into JSON
  15. Why Avoid Istio? > Spring has the same functionalities •

    Metrics for traffics • Distributed tracing • Fault injection, Circuit breaker, Retry > Performance
  16. ACL Manager > API for managing DB ACL > Hooks

    autoscaler to add or delete a node
  17. Agenda > Kubernetes Service for LINE Services > Our Service

    Architecture Using gRPC > Service Metrics/Observations > Log Aggregation
  18. Prometheus > Persistent Volume is hard to use > We

    try using a remote storage instead • Developed TSDB in-house
  19. Too Many Metrics > Too many metrics in Prometheus with

    Kubernetes • How to visualize? • Which metrics should we watch? > Head time series: 240,638 Prometheus With Kubernetes
  20. Too Many Metrics > kubernetes-mixin adds many settings • Grafana

    dashboards • Pod crush loop detection • Node hanging up • Volume usage predication for PersistentVolumes • etc. Prometheus with Kubernetes
  21. Spring Application Metrics > Spring Boot Actuator > OpenCensus •

    Metrics • Distributed traces • Various Language
  22. Agenda > Kubernetes Service for LINE Services > Our Service

    Architecture Using gRPC > Service Metrics/Observations > Log Aggregation
  23. Log Aggregations > Fluentd + Elasticsearch + Kibana • For

    generic logs > IMON • For error log alerts
  24. Log Aggregation in Kubernetes Cluster > Old pod logs may

    be removed > Logs should be stored in external storage
  25. Fluentd > Referred to • knative requirements • kubernetes addon

    • not for production • manifests in Helm Chart > plugins • fluent-plugin-elasticsearch • fluent-plugin-kubernetes_metadata_filter • fluent-plugin-detect-exceptions • fluent-plugin-multi-format-parser
  26. Agenda > Kubernetes Service for LINE Services > Our Service

    Architecture Using gRPC > Service Metrics/Observations > Log Aggregation
  27. Our Booth > Kotlin coroutine > CI/CD with Drone CI

    + Kubernetes + Github > Development flow with sharing proto files with server/client