Guide to Load Test gRPC application 2018/09/28 Kenju Wagatsuma @ Cookpad Inc. 

TOC Overview Specification for Load Testing Overall Load Testing tools (HTTP & gRPC) github.com/bojang/ghz Master/Minion Patterns (Multi-Threaded ghz wrapper) Reporting (InfluxDB & Grafana) 

Overview 

Overview You can understand … - How to perform load testing to gRPC application - What is “bojang/ghz” and How to use it - Master/Minion Patterns - TICK Stack & InfluxDB basis for Reporting the load testing results 

Specification for Load Testing 

Specification for Load Testing gRPC supported. 3 RPC will be called at the same time: - 1,200 RPS - 800 RPS - 150 RPS 

Overall Load Testing tools (HTTP & gRPC) 

Load Testing Tools - gRPC supports are rare - bojand/ghz is the very nice tool I was looking for https://github.com/httperf/httperf - one of major tool for HTTP stress testing, no support for gRPC https://github.com/loadimpact/k6 - also one of major HTTP stress testing tool, no support for gRPC https://github.com/tsenart/vegeta - yet another HTTP load testing, no support for gRPC https://grpc.io/docs/guides/benchmarking.html - benchmark test at gRPC https://gist.github.com/voluntas/5c9e0f778e36c1e934e83611a94ffdfa#id 5 - Shigredo (時雨堂) Oribe, no support for gRPC https://github.com/bojand/ghz - gRPC-oriented stress testing tool 

github.com/bojang/ghz 

github.com/bojang/ghz gRPC benchmarking and load testing tool inspired by “rakyll/hey/” or “fullstorydev/grpcurl” 

simple usage, nice reporting can be configured… - num of concurrency (implemented via goroutines internally) - rate limit (= QPS / Query Per Second) - max duration (sec) - HTML/JSON/CSV/InfluxDB Line Protocol output 

Example Usage Template Action supported - implemented by text/template (Go standard API) - {{.RequestNumber}}, {{.MethodName}}, {{.ServiceName}}, .etc. Fixtures can be file - handy & reusable $ cat testdata/Heart.json { "episode_id": "{{.RequestNumber}}", "heart_key": "heart key", "user_id": 999, "count": 9999 } $ cat testdata/metadata.json {} $ ghz \ -proto protobuf-definitions/myapp/v1/mydefinition.proto \ -call myapp.v1.Mydefinition.Heart \ -D testdata/Heart.json -M testdata/metadata.json \ -o log/Heart.log # output log file path -n 10000 \ # number of requests -q 5000 \ # QPS -insecure \ # for non TLS connection -O influx-details \ # output type format (e.g. csv/json/pretty/html/influx-summary) 127.0.0.1:5555 

Master/Minion Patterns (Multi-Threaded ghz wrapper) 

Master/Minion Patterns 時雨堂 シナリオ負荷試験ツール Oribe - Master has N minions - Scenario can be written in Lua - HTTP/1.1 - No support for reporting https://gist.github.com/voluntas/5c9e0f778e36c1e934e83611a94ffdfa#id5 

Master/Minion Patterns Master has responsibility for: - common options (host & port, log, .etc) - handles multiple minions - reporting Minion has responsibility for: - run `ghz` to a single RPC - delegate results to the master class Minion def run system "ghz #{options}" end def rpc raise NotImplementedError end end class HelloRPCMinion < Minion def rpc; "Hello" end end class PingRPCMinion < Minion def rpc; "Ping" end end class Master def load_test minions = [ HelloRPCMinion.new, PingRPCMinion.new ] threads = minions.map {|m| Thread.new { m.run } } threads.each(&:join) end end 

Sample Code Check it out -> “Run Multiple gRPC Load Testing using ghz” http://itiskj.hatenablog.com/entry/2018/09/14/122110 

Reporting (InfluxDB & Grafana) 

Reporting - InfluxDB + Grafana Dashboard - InfluxDB … Time-series oriented - bojand/ghz supports InfluxDB Line Protocol - https://github.com/bojand/ghz/pull/26 

TICK Stack Telegraf (Golang) - similar to Fluentd agent - pluggable architecture (AWS Cloudwatch, Consul, ElasticSearch, Fluentd, HAProxy,, Nginx, ...) Kapacitor (Golang) - streaming data processing engine - similar to AWS Kinesis or Apache Kafka Chronograf (TypeScript, Golang) - monitoring + visualization https://www.influxdata.com/time-series-platform/ 

InfluxDB Line Protocol - whitespace are separator of each components - support floats, integers, string or booleans as field values type - use HTTP API with curl or CLI to write to InfluxDB with Line Protocol Syntax) weather,location=us-midwest temperature=82 1465839830100400200 | --------------------------------- ------------------------- | | | | | | | | | +---------------------+-----------+-+-----------------+-+------------------+ |measurement|,tag_set| | field_set | |timestamp| +---------------------+-----------+-+-----------------+-+------------------+ Example) ghz_detail,proto="/testdata/greeter.proto",call="helloworld.Greeter.SayHello",host="0.0.0. 0:50051",n=1000,c=50,qps=0,z=0,timeout=20,dial_timeout=10,keepalive=0,data="{\"name\ ":\"{{.InputName}}\"}",metadata="{\"rn\":\"{{.RequestNumber}}\"}",hasError=false latency=5157328,error=,status=OK 681023506 

Thank you!