Save 37% off PRO during our Black Friday Sale! »

Quick and dirty load testing with Vegeta

6f44c2cc7923c18417bb5a4126152fb2?s=47 coldclimate
January 05, 2016

Quick and dirty load testing with Vegeta

using and abusing an excellent tool

6f44c2cc7923c18417bb5a4126152fb2?s=128

coldclimate

January 05, 2016
Tweet

Transcript

  1. quick and dirty load and soak testing with Vegeta @coldclimate

  2. How the hell do I say it? “veg Eater” source:

    https://www.youtube.com/watch?v=O082H7FhRt8
  3. Aim 1. What is Vegeta 2. Get you up and

    running 3. How does it work? 4. Prep-work 5. Ramp up testing 6. Load testing 7. Soak testing 8. Painkillers
  4. what is it?

  5. vegeta is 1. HTTP based load testing tool designed for

    constant rate of hits 2. Written in Go (+1 hipster point, +2 easy install) 3. Can be used as a library or a command line script 4. Great for running “fixed size” tests
  6. what isn’t it?

  7. vegeta is not 1. Synthetic user testing 2. Able to

    react to responses 3. Good at realtime feedback
  8. getting up and running

  9. Install it 1. Spin up an instance with multiple CPU

    + RAM (M3.xLarge) 2. Download it https://github.com/tsenart/vegeta/releases 3. Permission it: chmod u+x vegeta 4. Set some limits a. process limits b. file descriptor limits c. https://gist.github.com/sage-oli-wood/2d2e748d25a3cad6b86c
  10. How’s it work

  11. How’s it work? 1. Reads a list of calls from

    targets.txt 2. Launches a fixed rate of requests against your targets 3. Results written out to binary file 4. Once finished….Binary file piped through Vegeta to get results
  12. The basic setup for launching vegeta HTTP Target system targets.txt

    payload.json output.bin
  13. The basic setup for reporting vegeta Report output.bin knowledge bombs

  14. Example? 1. echo “GET http://www.google.com” >> targets.txt 2. vegeta -cpus=16

    attack -targets=targets.txt -workers=64 - rate=200 -duration=20s >> temp.bin 3. cat temp.bin | vegeta report -reporter text a. json b. histogram c. plot
  15. Options 1. cpus - how many CPUs to use 2.

    workers - start number - will increase as needed 3. rate - desired rate per second 4. duration - how long to run for See https://github.com/tsenart/vegeta
  16. Output Requests [total, rate] 4000, 200.05 Duration [total, attack, wait]

    20.075626747s, 19.994999805s, 80.626942ms Latencies [mean, 50, 95, 99, max] 87.221968ms, 85.30566 ms, 96.614723ms, 154.43155ms, 319.217301ms Bytes In [total, mean] 76221015, 19055.25 Bytes Out [total, mean] 0, 0.00 Success [ratio] 100.00% Status Codes [code:count] 200:4000 Error Set:
  17. None
  18. Output 1. JSON can be programmatically read 2. CSV because

    Excel because enterprise 3. histogram (bucket based)
  19. Output bonus Run once, report many

  20. Prep Work

  21. Generating targets.txt 1. URLs 2. POST data 3. Frequency

  22. URLs and Frequency from Elastic Load Balancers 1. Log to

    an S3 bucket 2. Download 3. awk out the right column 4. Replace any GUIDs using sed 5. sort | uniq -c | sort -rn See https://www.theapproachablegeek.co.uk/blog/parsing-elb- logs/
  23. None
  24. Example more complex targets.txt POST http://example.com:8080/things X-Account-ID: 8675309 Authorization: Token

    DEADBEEF @8675309-DEADBEEF.json Custom header, custom body
  25. POST data and preparatory steps Script up the preparatory steps,

    capture to files as you go. 1. User creation 2. Data clean up or creation 3. Session identifier (cookie etc) capture 4. Write out data to JSON files 5. I used BASH (and some Python)
  26. Three types of test

  27. Soak test what happens with constant load over time

  28. ramp up test how quickly can we add actions

  29. load test what’s the maximum before it explodes

  30. Soak testing

  31. Run a constant rate over a protracted time Perfect use

    for Vegeta • Degradation in performance over time • Crashing (memory leaks etc) • Performance of DB with table size • Error rates over time • Disk usage (log files etc) • Finding optimal boxes
  32. None
  33. Soak testing gotchas • Monitoring Vegeta output ◦ Realtime trick

    ◦ ELB statistics ◦ Other tools • Size of data involved ◦ Post processing is tough • Boring ◦ Easy to get distracted ◦ Needs checking - fail in the first hour etc
  34. ramp up testing

  35. What’s the fastest we can add requests before something dies?

    • Launch multiple instances of Vegeta in a controlled manner • Monitor in real time until ◦ Errors ◦ Latency (define in advance) • Use this info in your other tests
  36. Time Rate

  37. Launching in a controlled manner BASH to the rescue! while

    true do vegeta -cpus=16 attack -targets=target.txt -workers=64 - rate=50 > result$(date +%s).bin & sleep 60 done
  38. load testing

  39. How much can we throw at it before it explodes

    • Define “explodes” - latency/errors • Add at a rate that doesn’t break prematurely • Watch for premature errors • Repeat, repeat and repeat more
  40. Other tools give yourself all the chances

  41. None
  42. See all the things not just what you get back

    • sar/iostat/ et al on box • CloudWatch (ELB info) • DB tools (MONyog) • See http://techblog.netflix.com/2015/11/linux- performance-analysis-in-60s.html
  43. Scaling up

  44. When you max out a load generator 1. Use multiple

    boxes 2. Build hostname and timestamp into binary results 3. Consolidate output binaries to one place 4. CSSH for coordination (also alt-~) 5. EC2 to EC2 instance transfer for speed
  45. Like this but more biggerer vegeta HTTP Target system targets.txt

    payload.json output.bin
  46. Yeah... HTTP Target system vegeta targets.txt payload.json output.bin vegeta targets.txt

    payload.json output.bin vegeta targets.txt payload.json output.bin vegeta targets.txt payload.json output.bin vegeta targets.txt payload.json output.bin vegeta targets.txt payload.json output.bin vegeta targets.txt payload.json output.bin vegeta targets.txt payload.json output.bin vegeta targets.txt payload.json output.bin
  47. Expect pain vegeta output.bin output.bin output.bin output.bin output.bin output.bin output.bin

    output.bin output.bin output.bin output.bin output.bin output.bin output.bin output.bin output.bin output.bin output.bin reporting knowledge
  48. Realtime results

  49. Dorty Hack • Peek into Vegeta as is runs •

    Not 100% accurate but good enough • Not quite as simple as *.bin • Export name=”output”;REPORT=`ls -1 $name | sed "s/^/$name\//g" |sed 's/$/,/g' | tr -d "\n" | sed 's/.$//g'` • vegeta report -inputs="$REPORT" --reporter text > $name. txt • https://gist.github.com/coldclimate/e7905f93079f8c19b6e0
  50. Giant result sets

  51. Can be a pain • 20GB of binary outputs •

    Forget nice HTML view • Use other tools • vegeta dump -dumper csv • BASH it hard - sort - reduce - summerise ◦ sort it (not sorted by default) ◦ Consolidate by timestamp ◦ Sum or average response times ◦ Count response codes • R?
  52. How big can you go?

  53. Depends... • Vegeta is rarely the problem • Load generating

    boxes (file descriptors etc) • 20k req/s using 4x c3.2xlarge (8 cores, 15GB RAM)
  54. FIN