Pro Yearly is on sale from $80 to $50! »

The Guide to Load Test gRPC Application

3b36493b4296ebeb219bcd3ffab3aa2b?s=47 Kenju Wagatsuma
September 28, 2018

The Guide to Load Test gRPC Application

You can understand …
- How to perform load testing to gRPC application
- What is “github.com/bojang/ghz” and How to use it
- Master/Minion Patterns
- TICK Stack & InfluxDB basis for Reporting the load testing results

3b36493b4296ebeb219bcd3ffab3aa2b?s=128

Kenju Wagatsuma

September 28, 2018
Tweet

Transcript

  1. Guide to Load Test gRPC application 2018/09/28 Kenju Wagatsuma @

    Cookpad Inc.
  2. TOC Overview Specification for Load Testing Overall Load Testing tools

    (HTTP & gRPC) github.com/bojang/ghz Master/Minion Patterns (Multi-Threaded ghz wrapper) Reporting (InfluxDB & Grafana)
  3. Overview

  4. Overview You can understand … - How to perform load

    testing to gRPC application - What is “bojang/ghz” and How to use it - Master/Minion Patterns - TICK Stack & InfluxDB basis for Reporting the load testing results
  5. Specification for Load Testing

  6. Specification for Load Testing gRPC supported. 3 RPC will be

    called at the same time: - 1,200 RPS - 800 RPS - 150 RPS
  7. Overall Load Testing tools (HTTP & gRPC)

  8. Load Testing Tools - gRPC supports are rare - bojand/ghz

    is the very nice tool I was looking for https://github.com/httperf/httperf - one of major tool for HTTP stress testing, no support for gRPC https://github.com/loadimpact/k6 - also one of major HTTP stress testing tool, no support for gRPC https://github.com/tsenart/vegeta - yet another HTTP load testing, no support for gRPC https://grpc.io/docs/guides/benchmarking.html - benchmark test at gRPC https://gist.github.com/voluntas/5c9e0f778e36c1e934e83611a94ffdfa#id 5 - Shigredo (時雨堂) Oribe, no support for gRPC https://github.com/bojand/ghz - gRPC-oriented stress testing tool
  9. github.com/bojang/ghz

  10. github.com/bojang/ghz gRPC benchmarking and load testing tool inspired by “rakyll/hey/”

    or “fullstorydev/grpcurl”
  11. simple usage, nice reporting can be configured… - num of

    concurrency (implemented via goroutines internally) - rate limit (= QPS / Query Per Second) - max duration (sec) - HTML/JSON/CSV/InfluxDB Line Protocol output
  12. Example Usage Template Action supported - implemented by text/template (Go

    standard API) - {{.RequestNumber}}, {{.MethodName}}, {{.ServiceName}}, .etc. Fixtures can be file - handy & reusable $ cat testdata/Heart.json { "episode_id": "{{.RequestNumber}}", "heart_key": "heart key", "user_id": 999, "count": 9999 } $ cat testdata/metadata.json {} $ ghz \ -proto protobuf-definitions/myapp/v1/mydefinition.proto \ -call myapp.v1.Mydefinition.Heart \ -D testdata/Heart.json -M testdata/metadata.json \ -o log/Heart.log # output log file path -n 10000 \ # number of requests -q 5000 \ # QPS -insecure \ # for non TLS connection -O influx-details \ # output type format (e.g. csv/json/pretty/html/influx-summary) 127.0.0.1:5555
  13. Master/Minion Patterns (Multi-Threaded ghz wrapper)

  14. Master/Minion Patterns 時雨堂 シナリオ負荷試験ツール Oribe - Master has N minions

    - Scenario can be written in Lua - HTTP/1.1 - No support for reporting https://gist.github.com/voluntas/5c9e0f778e36c1e934e83611a94ffdfa#id5
  15. Master/Minion Patterns Master has responsibility for: - common options (host

    & port, log, .etc) - handles multiple minions - reporting Minion has responsibility for: - run `ghz` to a single RPC - delegate results to the master class Minion def run system "ghz #{options}" end def rpc raise NotImplementedError end end class HelloRPCMinion < Minion def rpc; "Hello" end end class PingRPCMinion < Minion def rpc; "Ping" end end class Master def load_test minions = [ HelloRPCMinion.new, PingRPCMinion.new ] threads = minions.map {|m| Thread.new { m.run } } threads.each(&:join) end end
  16. Sample Code Check it out -> “Run Multiple gRPC Load

    Testing using ghz” http://itiskj.hatenablog.com/entry/2018/09/14/122110
  17. Reporting (InfluxDB & Grafana)

  18. Reporting - InfluxDB + Grafana Dashboard - InfluxDB … Time-series

    oriented - bojand/ghz supports InfluxDB Line Protocol - https://github.com/bojand/ghz/pull/26
  19. TICK Stack Telegraf (Golang) - similar to Fluentd agent -

    pluggable architecture (AWS Cloudwatch, Consul, ElasticSearch, Fluentd, HAProxy,, Nginx, ...) Kapacitor (Golang) - streaming data processing engine - similar to AWS Kinesis or Apache Kafka Chronograf (TypeScript, Golang) - monitoring + visualization https://www.influxdata.com/time-series-platform/
  20. InfluxDB Line Protocol - whitespace are separator of each components

    - support floats, integers, string or booleans as field values type - use HTTP API with curl or CLI to write to InfluxDB with Line Protocol Syntax) weather,location=us-midwest temperature=82 1465839830100400200 | --------------------------------- ------------------------- | | | | | | | | | +---------------------+-----------+-+-----------------+-+------------------+ |measurement|,tag_set| | field_set | |timestamp| +---------------------+-----------+-+-----------------+-+------------------+ Example) ghz_detail,proto="/testdata/greeter.proto",call="helloworld.Greeter.SayHello",host="0.0.0. 0:50051",n=1000,c=50,qps=0,z=0,timeout=20,dial_timeout=10,keepalive=0,data="{\"name\ ":\"{{.InputName}}\"}",metadata="{\"rn\":\"{{.RequestNumber}}\"}",hasError=false latency=5157328,error=,status=OK 681023506
  21. Thank you!