Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Quick and dirty load testing with Vegeta

coldclimate
January 05, 2016

Quick and dirty load testing with Vegeta

using and abusing an excellent tool

coldclimate

January 05, 2016
Tweet

More Decks by coldclimate

Other Decks in Technology

Transcript

  1. quick and dirty load and soak
    testing with Vegeta
    @coldclimate

    View full-size slide

  2. How the hell do I say it?
    “veg Eater”
    source: https://www.youtube.com/watch?v=O082H7FhRt8

    View full-size slide

  3. Aim
    1. What is Vegeta
    2. Get you up and running
    3. How does it work?
    4. Prep-work
    5. Ramp up testing
    6. Load testing
    7. Soak testing
    8. Painkillers

    View full-size slide

  4. vegeta is
    1. HTTP based load testing tool designed for constant rate
    of hits
    2. Written in Go (+1 hipster point, +2 easy install)
    3. Can be used as a library or a command line script
    4. Great for running “fixed size” tests

    View full-size slide

  5. what isn’t it?

    View full-size slide

  6. vegeta is not
    1. Synthetic user testing
    2. Able to react to responses
    3. Good at realtime feedback

    View full-size slide

  7. getting up and running

    View full-size slide

  8. Install it
    1. Spin up an instance with multiple CPU + RAM (M3.xLarge)
    2. Download it https://github.com/tsenart/vegeta/releases
    3. Permission it: chmod u+x vegeta
    4. Set some limits
    a. process limits
    b. file descriptor limits
    c. https://gist.github.com/sage-oli-wood/2d2e748d25a3cad6b86c

    View full-size slide

  9. How’s it work

    View full-size slide

  10. How’s it work?
    1. Reads a list of calls from targets.txt
    2. Launches a fixed rate of requests against your targets
    3. Results written out to binary file
    4. Once finished….Binary file piped through Vegeta to get
    results

    View full-size slide

  11. The basic setup for launching
    vegeta
    HTTP
    Target
    system
    targets.txt payload.json
    output.bin

    View full-size slide

  12. The basic setup for reporting
    vegeta
    Report
    output.bin
    knowledge bombs

    View full-size slide

  13. Example?
    1. echo “GET http://www.google.com” >> targets.txt
    2. vegeta -cpus=16 attack -targets=targets.txt -workers=64 -
    rate=200 -duration=20s >> temp.bin
    3. cat temp.bin | vegeta report -reporter text
    a. json
    b. histogram
    c. plot

    View full-size slide

  14. Options
    1. cpus - how many CPUs to use
    2. workers - start number - will increase as needed
    3. rate - desired rate per second
    4. duration - how long to run for
    See https://github.com/tsenart/vegeta

    View full-size slide

  15. Output
    Requests [total, rate] 4000, 200.05
    Duration [total, attack, wait] 20.075626747s,
    19.994999805s, 80.626942ms
    Latencies [mean, 50, 95, 99, max] 87.221968ms, 85.30566
    ms, 96.614723ms, 154.43155ms, 319.217301ms
    Bytes In [total, mean] 76221015, 19055.25
    Bytes Out [total, mean] 0, 0.00
    Success [ratio] 100.00%
    Status Codes [code:count] 200:4000
    Error Set:

    View full-size slide

  16. Output
    1. JSON can be programmatically read
    2. CSV because Excel because enterprise
    3. histogram (bucket based)

    View full-size slide

  17. Output bonus
    Run once, report many

    View full-size slide

  18. Generating targets.txt
    1. URLs
    2. POST data
    3. Frequency

    View full-size slide

  19. URLs and Frequency from Elastic Load Balancers
    1. Log to an S3 bucket
    2. Download
    3. awk out the right column
    4. Replace any GUIDs using sed
    5. sort | uniq -c | sort -rn
    See https://www.theapproachablegeek.co.uk/blog/parsing-elb-
    logs/

    View full-size slide

  20. Example more complex targets.txt
    POST http://example.com:8080/things
    X-Account-ID: 8675309
    Authorization: Token DEADBEEF
    @8675309-DEADBEEF.json
    Custom header, custom body

    View full-size slide

  21. POST data and preparatory steps
    Script up the preparatory steps, capture to files as you go.
    1. User creation
    2. Data clean up or creation
    3. Session identifier (cookie etc) capture
    4. Write out data to JSON files
    5. I used BASH (and some Python)

    View full-size slide

  22. Three types of test

    View full-size slide

  23. Soak test
    what happens with constant load over time

    View full-size slide

  24. ramp up test
    how quickly can we add actions

    View full-size slide

  25. load test
    what’s the maximum before it explodes

    View full-size slide

  26. Soak testing

    View full-size slide

  27. Run a constant rate over a protracted time
    Perfect use for Vegeta
    ● Degradation in performance over time
    ● Crashing (memory leaks etc)
    ● Performance of DB with table size
    ● Error rates over time
    ● Disk usage (log files etc)
    ● Finding optimal boxes

    View full-size slide

  28. Soak testing gotchas
    ● Monitoring Vegeta output
    ○ Realtime trick
    ○ ELB statistics
    ○ Other tools
    ● Size of data involved
    ○ Post processing is tough
    ● Boring
    ○ Easy to get distracted
    ○ Needs checking - fail in the first hour etc

    View full-size slide

  29. ramp up testing

    View full-size slide

  30. What’s the fastest we can add requests before something
    dies?
    ● Launch multiple instances of Vegeta in a controlled
    manner
    ● Monitor in real time until
    ○ Errors
    ○ Latency (define in advance)
    ● Use this info in your other tests

    View full-size slide

  31. Launching in a controlled manner
    BASH to the rescue!
    while true
    do
    vegeta -cpus=16 attack -targets=target.txt -workers=64 -
    rate=50 > result$(date +%s).bin &
    sleep 60
    done

    View full-size slide

  32. load testing

    View full-size slide

  33. How much can we throw at it before it explodes
    ● Define “explodes” - latency/errors
    ● Add at a rate that doesn’t break prematurely
    ● Watch for premature errors
    ● Repeat, repeat and repeat more

    View full-size slide

  34. Other tools
    give yourself all the chances

    View full-size slide

  35. See all the things not just what you get back
    ● sar/iostat/ et al on box
    ● CloudWatch (ELB info)
    ● DB tools (MONyog)
    ● See http://techblog.netflix.com/2015/11/linux-
    performance-analysis-in-60s.html

    View full-size slide

  36. When you max out a load generator
    1. Use multiple boxes
    2. Build hostname and timestamp into binary results
    3. Consolidate output binaries to one place
    4. CSSH for coordination (also alt-~)
    5. EC2 to EC2 instance transfer for speed

    View full-size slide

  37. Like this but more biggerer
    vegeta
    HTTP
    Target
    system
    targets.txt payload.json
    output.bin

    View full-size slide

  38. Yeah...
    HTTP
    Target
    system
    vegeta
    targets.txt
    payload.json
    output.bin
    vegeta
    targets.txt
    payload.json
    output.bin
    vegeta
    targets.txt
    payload.json
    output.bin
    vegeta
    targets.txt
    payload.json
    output.bin
    vegeta
    targets.txt
    payload.json
    output.bin
    vegeta
    targets.txt
    payload.json
    output.bin
    vegeta
    targets.txt
    payload.json
    output.bin
    vegeta
    targets.txt
    payload.json
    output.bin
    vegeta
    targets.txt
    payload.json
    output.bin

    View full-size slide

  39. Expect pain
    vegeta
    output.bin
    output.bin
    output.bin
    output.bin
    output.bin
    output.bin
    output.bin
    output.bin
    output.bin
    output.bin
    output.bin
    output.bin
    output.bin
    output.bin
    output.bin
    output.bin
    output.bin
    output.bin
    reporting knowledge

    View full-size slide

  40. Realtime results

    View full-size slide

  41. Dorty Hack
    ● Peek into Vegeta as is runs
    ● Not 100% accurate but good enough
    ● Not quite as simple as *.bin
    ● Export name=”output”;REPORT=`ls -1 $name | sed
    "s/^/$name\//g" |sed 's/$/,/g' | tr -d "\n" | sed
    's/.$//g'`
    ● vegeta report -inputs="$REPORT" --reporter text > $name.
    txt
    ● https://gist.github.com/coldclimate/e7905f93079f8c19b6e0

    View full-size slide

  42. Giant result sets

    View full-size slide

  43. Can be a pain
    ● 20GB of binary outputs
    ● Forget nice HTML view
    ● Use other tools
    ● vegeta dump -dumper csv
    ● BASH it hard - sort - reduce - summerise
    ○ sort it (not sorted by default)
    ○ Consolidate by timestamp
    ○ Sum or average response times
    ○ Count response codes
    ● R?

    View full-size slide

  44. How big can you go?

    View full-size slide

  45. Depends...
    ● Vegeta is rarely the problem
    ● Load generating boxes (file descriptors etc)
    ● 20k req/s using 4x c3.2xlarge (8 cores, 15GB RAM)

    View full-size slide