Quick and dirty load testing with Vegeta

using and abusing an excellent tool

Transcript

1. quick and dirty load and soak testing with Vegeta @coldclimate
2. How the hell do I say it? “veg Eater” source: https://www.youtube.com/watch?v=O082H7FhRt8
3. Aim 1. What is Vegeta 2. Get you up and running 3. How does it work? 4. Prep-work 5. Ramp up testing 6. Load testing 7. Soak testing 8. Painkillers
4. what is it?
5. vegeta is 1. HTTP based load testing tool designed for constant rate of hits 2. Written in Go (+1 hipster point, +2 easy install) 3. Can be used as a library or a command line script 4. Great for running “fixed size” tests
6. what isn’t it?
7. vegeta is not 1. Synthetic user testing 2. Able to react to responses 3. Good at realtime feedback
8. getting up and running
9. Install it 1. Spin up an instance with multiple CPU + RAM (M3.xLarge) 2. Download it https://github.com/tsenart/vegeta/releases 3. Permission it: chmod u+x vegeta 4. Set some limits a. process limits b. file descriptor limits c. https://gist.github.com/sage-oli-wood/2d2e748d25a3cad6b86c
10. How’s it work
11. How’s it work? 1. Reads a list of calls from targets.txt 2. Launches a fixed rate of requests against your targets 3. Results written out to binary file 4. Once finished….Binary file piped through Vegeta to get results
12. The basic setup for launching vegeta HTTP Target system targets.txt payload.json output.bin
13. The basic setup for reporting vegeta Report output.bin knowledge bombs
14. Example? 1. echo “GET http://www.google.com” >> targets.txt 2. vegeta -cpus=16 attack -targets=targets.txt -workers=64 - rate=200 -duration=20s >> temp.bin 3. cat temp.bin | vegeta report -reporter text a. json b. histogram c. plot
15. Options 1. cpus - how many CPUs to use 2. workers - start number - will increase as needed 3. rate - desired rate per second 4. duration - how long to run for See https://github.com/tsenart/vegeta
16. Output Requests [total, rate] 4000, 200.05 Duration [total, attack, wait] 20.075626747s, 19.994999805s, 80.626942ms Latencies [mean, 50, 95, 99, max] 87.221968ms, 85.30566 ms, 96.614723ms, 154.43155ms, 319.217301ms Bytes In [total, mean] 76221015, 19055.25 Bytes Out [total, mean] 0, 0.00 Success [ratio] 100.00% Status Codes [code:count] 200:4000 Error Set:
18. Output 1. JSON can be programmatically read 2. CSV because Excel because enterprise 3. histogram (bucket based)
19. Output bonus Run once, report many
20. Prep Work
21. Generating targets.txt 1. URLs 2. POST data 3. Frequency
22. URLs and Frequency from Elastic Load Balancers 1. Log to an S3 bucket 2. Download 3. awk out the right column 4. Replace any GUIDs using sed 5. sort | uniq -c | sort -rn See https://www.theapproachablegeek.co.uk/blog/parsing-elb- logs/
24. Example more complex targets.txt POST http://example.com:8080/things X-Account-ID: 8675309 Authorization: Token DEADBEEF @8675309-DEADBEEF.json Custom header, custom body
25. POST data and preparatory steps Script up the preparatory steps, capture to files as you go. 1. User creation 2. Data clean up or creation 3. Session identifier (cookie etc) capture 4. Write out data to JSON files 5. I used BASH (and some Python)
26. Three types of test
27. Soak test what happens with constant load over time
28. ramp up test how quickly can we add actions
29. load test what’s the maximum before it explodes
30. Soak testing
31. Run a constant rate over a protracted time Perfect use for Vegeta • Degradation in performance over time • Crashing (memory leaks etc) • Performance of DB with table size • Error rates over time • Disk usage (log files etc) • Finding optimal boxes
33. Soak testing gotchas • Monitoring Vegeta output ◦ Realtime trick ◦ ELB statistics ◦ Other tools • Size of data involved ◦ Post processing is tough • Boring ◦ Easy to get distracted ◦ Needs checking - fail in the first hour etc
34. ramp up testing
35. What’s the fastest we can add requests before something dies? • Launch multiple instances of Vegeta in a controlled manner • Monitor in real time until ◦ Errors ◦ Latency (define in advance) • Use this info in your other tests
36. Time Rate
37. Launching in a controlled manner BASH to the rescue! while true do vegeta -cpus=16 attack -targets=target.txt -workers=64 - rate=50 > result$(date +%s).bin & sleep 60 done
38. load testing
39. How much can we throw at it before it explodes • Define “explodes” - latency/errors • Add at a rate that doesn’t break prematurely • Watch for premature errors • Repeat, repeat and repeat more
40. Other tools give yourself all the chances
42. See all the things not just what you get back • sar/iostat/ et al on box • CloudWatch (ELB info) • DB tools (MONyog) • See http://techblog.netflix.com/2015/11/linux- performance-analysis-in-60s.html
43. Scaling up
44. When you max out a load generator 1. Use multiple boxes 2. Build hostname and timestamp into binary results 3. Consolidate output binaries to one place 4. CSSH for coordination (also alt-~) 5. EC2 to EC2 instance transfer for speed
45. Like this but more biggerer vegeta HTTP Target system targets.txt payload.json output.bin
46. Yeah... HTTP Target system vegeta targets.txt payload.json output.bin vegeta targets.txt payload.json output.bin vegeta targets.txt payload.json output.bin vegeta targets.txt payload.json output.bin vegeta targets.txt payload.json output.bin vegeta targets.txt payload.json output.bin vegeta targets.txt payload.json output.bin vegeta targets.txt payload.json output.bin vegeta targets.txt payload.json output.bin
47. Expect pain vegeta output.bin output.bin output.bin output.bin output.bin output.bin output.bin output.bin output.bin output.bin output.bin output.bin output.bin output.bin output.bin output.bin output.bin output.bin reporting knowledge
48. Realtime results
49. Dorty Hack • Peek into Vegeta as is runs • Not 100% accurate but good enough • Not quite as simple as *.bin • Export name=”output”;REPORT=`ls -1 $name | sed "s/^/$name\//g" |sed 's/$/,/g' | tr -d "\n" | sed 's/.$//g'` • vegeta report -inputs="$REPORT" --reporter text > $name. txt • https://gist.github.com/coldclimate/e7905f93079f8c19b6e0
50. Giant result sets
51. Can be a pain • 20GB of binary outputs • Forget nice HTML view • Use other tools • vegeta dump -dumper csv • BASH it hard - sort - reduce - summerise ◦ sort it (not sorted by default) ◦ Consolidate by timestamp ◦ Sum or average response times ◦ Count response codes • R?
52. How big can you go?
53. Depends... • Vegeta is rarely the problem • Load generating boxes (file descriptors etc) • 20k req/s using 4x c3.2xlarge (8 cores, 15GB RAM)
54. FIN