Quick and dirty load testing with Vegeta

Slide 1

Slide 1 text

quick and dirty load and soak testing with Vegeta @coldclimate

Slide 2

Slide 2 text

How the hell do I say it? “veg Eater” source: https://www.youtube.com/watch?v=O082H7FhRt8

Slide 3

Slide 3 text

Aim 1. What is Vegeta 2. Get you up and running 3. How does it work? 4. Prep-work 5. Ramp up testing 6. Load testing 7. Soak testing 8. Painkillers

Slide 4

Slide 4 text

what is it?

Slide 5

Slide 5 text

vegeta is 1. HTTP based load testing tool designed for constant rate of hits 2. Written in Go (+1 hipster point, +2 easy install) 3. Can be used as a library or a command line script 4. Great for running “fixed size” tests

Slide 6

Slide 6 text

what isn’t it?

Slide 7

Slide 7 text

vegeta is not 1. Synthetic user testing 2. Able to react to responses 3. Good at realtime feedback

Slide 8

Slide 8 text

getting up and running

Slide 9

Slide 9 text

Install it 1. Spin up an instance with multiple CPU + RAM (M3.xLarge) 2. Download it https://github.com/tsenart/vegeta/releases 3. Permission it: chmod u+x vegeta 4. Set some limits a. process limits b. file descriptor limits c. https://gist.github.com/sage-oli-wood/2d2e748d25a3cad6b86c

Slide 10

Slide 10 text

How’s it work

Slide 11

Slide 11 text

How’s it work? 1. Reads a list of calls from targets.txt 2. Launches a fixed rate of requests against your targets 3. Results written out to binary file 4. Once finished….Binary file piped through Vegeta to get results

Slide 12

Slide 12 text

The basic setup for launching vegeta HTTP Target system targets.txt payload.json output.bin

Slide 13

Slide 13 text

The basic setup for reporting vegeta Report output.bin knowledge bombs

Slide 14

Slide 14 text

Example? 1. echo “GET http://www.google.com” >> targets.txt 2. vegeta -cpus=16 attack -targets=targets.txt -workers=64 - rate=200 -duration=20s >> temp.bin 3. cat temp.bin | vegeta report -reporter text a. json b. histogram c. plot

Slide 15

Slide 15 text

Options 1. cpus - how many CPUs to use 2. workers - start number - will increase as needed 3. rate - desired rate per second 4. duration - how long to run for See https://github.com/tsenart/vegeta

Slide 16

Slide 16 text

Output Requests [total, rate] 4000, 200.05 Duration [total, attack, wait] 20.075626747s, 19.994999805s, 80.626942ms Latencies [mean, 50, 95, 99, max] 87.221968ms, 85.30566 ms, 96.614723ms, 154.43155ms, 319.217301ms Bytes In [total, mean] 76221015, 19055.25 Bytes Out [total, mean] 0, 0.00 Success [ratio] 100.00% Status Codes [code:count] 200:4000 Error Set:

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

Output 1. JSON can be programmatically read 2. CSV because Excel because enterprise 3. histogram (bucket based)

Slide 19

Slide 19 text

Output bonus Run once, report many

Slide 20

Slide 20 text

Prep Work

Slide 21

Slide 21 text

Generating targets.txt 1. URLs 2. POST data 3. Frequency

Slide 22

Slide 22 text

URLs and Frequency from Elastic Load Balancers 1. Log to an S3 bucket 2. Download 3. awk out the right column 4. Replace any GUIDs using sed 5. sort | uniq -c | sort -rn See https://www.theapproachablegeek.co.uk/blog/parsing-elb- logs/

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

Example more complex targets.txt POST http://example.com:8080/things X-Account-ID: 8675309 Authorization: Token DEADBEEF @8675309-DEADBEEF.json Custom header, custom body

Slide 25

Slide 25 text

POST data and preparatory steps Script up the preparatory steps, capture to files as you go. 1. User creation 2. Data clean up or creation 3. Session identifier (cookie etc) capture 4. Write out data to JSON files 5. I used BASH (and some Python)

Slide 26

Slide 26 text

Three types of test

Slide 27

Slide 27 text

Soak test what happens with constant load over time

Slide 28

Slide 28 text

ramp up test how quickly can we add actions

Slide 29

Slide 29 text

load test what’s the maximum before it explodes

Slide 30

Slide 30 text

Soak testing

Slide 31

Slide 31 text

Run a constant rate over a protracted time Perfect use for Vegeta ● Degradation in performance over time ● Crashing (memory leaks etc) ● Performance of DB with table size ● Error rates over time ● Disk usage (log files etc) ● Finding optimal boxes

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

Soak testing gotchas ● Monitoring Vegeta output ○ Realtime trick ○ ELB statistics ○ Other tools ● Size of data involved ○ Post processing is tough ● Boring ○ Easy to get distracted ○ Needs checking - fail in the first hour etc

Slide 34

Slide 34 text

ramp up testing

Slide 35

Slide 35 text

What’s the fastest we can add requests before something dies? ● Launch multiple instances of Vegeta in a controlled manner ● Monitor in real time until ○ Errors ○ Latency (define in advance) ● Use this info in your other tests

Slide 36

Slide 36 text

Time Rate

Slide 37

Slide 37 text

Launching in a controlled manner BASH to the rescue! while true do vegeta -cpus=16 attack -targets=target.txt -workers=64 - rate=50 > result$(date +%s).bin & sleep 60 done

Slide 38

Slide 38 text

load testing

Slide 39

Slide 39 text

How much can we throw at it before it explodes ● Define “explodes” - latency/errors ● Add at a rate that doesn’t break prematurely ● Watch for premature errors ● Repeat, repeat and repeat more

Slide 40

Slide 40 text

Other tools give yourself all the chances

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

See all the things not just what you get back ● sar/iostat/ et al on box ● CloudWatch (ELB info) ● DB tools (MONyog) ● See http://techblog.netflix.com/2015/11/linux- performance-analysis-in-60s.html

Slide 43

Slide 43 text

Scaling up

Slide 44

Slide 44 text

When you max out a load generator 1. Use multiple boxes 2. Build hostname and timestamp into binary results 3. Consolidate output binaries to one place 4. CSSH for coordination (also alt-~) 5. EC2 to EC2 instance transfer for speed

Slide 45

Slide 45 text

Like this but more biggerer vegeta HTTP Target system targets.txt payload.json output.bin

Slide 46

Slide 46 text

Yeah... HTTP Target system vegeta targets.txt payload.json output.bin vegeta targets.txt payload.json output.bin vegeta targets.txt payload.json output.bin vegeta targets.txt payload.json output.bin vegeta targets.txt payload.json output.bin vegeta targets.txt payload.json output.bin vegeta targets.txt payload.json output.bin vegeta targets.txt payload.json output.bin vegeta targets.txt payload.json output.bin

Slide 47

Slide 47 text

Expect pain vegeta output.bin output.bin output.bin output.bin output.bin output.bin output.bin output.bin output.bin output.bin output.bin output.bin output.bin output.bin output.bin output.bin output.bin output.bin reporting knowledge

Slide 48

Slide 48 text

Realtime results

Slide 49

Slide 49 text

Dorty Hack ● Peek into Vegeta as is runs ● Not 100% accurate but good enough ● Not quite as simple as *.bin ● Export name=”output”;REPORT=`ls -1 $name | sed "s/^/$name\//g" |sed 's/$/,/g' | tr -d "\n" | sed 's/.$//g'` ● vegeta report -inputs="$REPORT" --reporter text > $name. txt ● https://gist.github.com/coldclimate/e7905f93079f8c19b6e0

Slide 50

Slide 50 text

Giant result sets

Slide 51

Slide 51 text

Can be a pain ● 20GB of binary outputs ● Forget nice HTML view ● Use other tools ● vegeta dump -dumper csv ● BASH it hard - sort - reduce - summerise ○ sort it (not sorted by default) ○ Consolidate by timestamp ○ Sum or average response times ○ Count response codes ● R?

Slide 52

Slide 52 text

How big can you go?

Slide 53

Slide 53 text

Depends... ● Vegeta is rarely the problem ● Load generating boxes (file descriptors etc) ● 20k req/s using 4x c3.2xlarge (8 cores, 15GB RAM)

Slide 54

Slide 54 text

FIN