Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Quick and dirty load testing with Vegeta

Quick and dirty load testing with Vegeta

using and abusing an excellent tool

coldclimate

January 05, 2016
Tweet

More Decks by coldclimate

Other Decks in Technology

Transcript

  1. How the hell do I say it? “veg Eater” source:

    https://www.youtube.com/watch?v=O082H7FhRt8
  2. Aim 1. What is Vegeta 2. Get you up and

    running 3. How does it work? 4. Prep-work 5. Ramp up testing 6. Load testing 7. Soak testing 8. Painkillers
  3. vegeta is 1. HTTP based load testing tool designed for

    constant rate of hits 2. Written in Go (+1 hipster point, +2 easy install) 3. Can be used as a library or a command line script 4. Great for running “fixed size” tests
  4. vegeta is not 1. Synthetic user testing 2. Able to

    react to responses 3. Good at realtime feedback
  5. Install it 1. Spin up an instance with multiple CPU

    + RAM (M3.xLarge) 2. Download it https://github.com/tsenart/vegeta/releases 3. Permission it: chmod u+x vegeta 4. Set some limits a. process limits b. file descriptor limits c. https://gist.github.com/sage-oli-wood/2d2e748d25a3cad6b86c
  6. How’s it work? 1. Reads a list of calls from

    targets.txt 2. Launches a fixed rate of requests against your targets 3. Results written out to binary file 4. Once finished….Binary file piped through Vegeta to get results
  7. Example? 1. echo “GET http://www.google.com” >> targets.txt 2. vegeta -cpus=16

    attack -targets=targets.txt -workers=64 - rate=200 -duration=20s >> temp.bin 3. cat temp.bin | vegeta report -reporter text a. json b. histogram c. plot
  8. Options 1. cpus - how many CPUs to use 2.

    workers - start number - will increase as needed 3. rate - desired rate per second 4. duration - how long to run for See https://github.com/tsenart/vegeta
  9. Output Requests [total, rate] 4000, 200.05 Duration [total, attack, wait]

    20.075626747s, 19.994999805s, 80.626942ms Latencies [mean, 50, 95, 99, max] 87.221968ms, 85.30566 ms, 96.614723ms, 154.43155ms, 319.217301ms Bytes In [total, mean] 76221015, 19055.25 Bytes Out [total, mean] 0, 0.00 Success [ratio] 100.00% Status Codes [code:count] 200:4000 Error Set:
  10. Output 1. JSON can be programmatically read 2. CSV because

    Excel because enterprise 3. histogram (bucket based)
  11. URLs and Frequency from Elastic Load Balancers 1. Log to

    an S3 bucket 2. Download 3. awk out the right column 4. Replace any GUIDs using sed 5. sort | uniq -c | sort -rn See https://www.theapproachablegeek.co.uk/blog/parsing-elb- logs/
  12. POST data and preparatory steps Script up the preparatory steps,

    capture to files as you go. 1. User creation 2. Data clean up or creation 3. Session identifier (cookie etc) capture 4. Write out data to JSON files 5. I used BASH (and some Python)
  13. Run a constant rate over a protracted time Perfect use

    for Vegeta • Degradation in performance over time • Crashing (memory leaks etc) • Performance of DB with table size • Error rates over time • Disk usage (log files etc) • Finding optimal boxes
  14. Soak testing gotchas • Monitoring Vegeta output ◦ Realtime trick

    ◦ ELB statistics ◦ Other tools • Size of data involved ◦ Post processing is tough • Boring ◦ Easy to get distracted ◦ Needs checking - fail in the first hour etc
  15. What’s the fastest we can add requests before something dies?

    • Launch multiple instances of Vegeta in a controlled manner • Monitor in real time until ◦ Errors ◦ Latency (define in advance) • Use this info in your other tests
  16. Launching in a controlled manner BASH to the rescue! while

    true do vegeta -cpus=16 attack -targets=target.txt -workers=64 - rate=50 > result$(date +%s).bin & sleep 60 done
  17. How much can we throw at it before it explodes

    • Define “explodes” - latency/errors • Add at a rate that doesn’t break prematurely • Watch for premature errors • Repeat, repeat and repeat more
  18. See all the things not just what you get back

    • sar/iostat/ et al on box • CloudWatch (ELB info) • DB tools (MONyog) • See http://techblog.netflix.com/2015/11/linux- performance-analysis-in-60s.html
  19. When you max out a load generator 1. Use multiple

    boxes 2. Build hostname and timestamp into binary results 3. Consolidate output binaries to one place 4. CSSH for coordination (also alt-~) 5. EC2 to EC2 instance transfer for speed
  20. Yeah... HTTP Target system vegeta targets.txt payload.json output.bin vegeta targets.txt

    payload.json output.bin vegeta targets.txt payload.json output.bin vegeta targets.txt payload.json output.bin vegeta targets.txt payload.json output.bin vegeta targets.txt payload.json output.bin vegeta targets.txt payload.json output.bin vegeta targets.txt payload.json output.bin vegeta targets.txt payload.json output.bin
  21. Expect pain vegeta output.bin output.bin output.bin output.bin output.bin output.bin output.bin

    output.bin output.bin output.bin output.bin output.bin output.bin output.bin output.bin output.bin output.bin output.bin reporting knowledge
  22. Dorty Hack • Peek into Vegeta as is runs •

    Not 100% accurate but good enough • Not quite as simple as *.bin • Export name=”output”;REPORT=`ls -1 $name | sed "s/^/$name\//g" |sed 's/$/,/g' | tr -d "\n" | sed 's/.$//g'` • vegeta report -inputs="$REPORT" --reporter text > $name. txt • https://gist.github.com/coldclimate/e7905f93079f8c19b6e0
  23. Can be a pain • 20GB of binary outputs •

    Forget nice HTML view • Use other tools • vegeta dump -dumper csv • BASH it hard - sort - reduce - summerise ◦ sort it (not sorted by default) ◦ Consolidate by timestamp ◦ Sum or average response times ◦ Count response codes • R?
  24. Depends... • Vegeta is rarely the problem • Load generating

    boxes (file descriptors etc) • 20k req/s using 4x c3.2xlarge (8 cores, 15GB RAM)
  25. FIN