Load Testing with Gatling - SQUADCO2015

Transcript

1. Load Testing with Gatling Alan Baird QA Lead at LivingSocial @bluegrasscoder abaird@bairdsnet.net ! https://goo.gl/Ka0PPr
2. How do you QA for: Black Friday Mother’s Day deal New Feature rollout
3. Load testing!
4. JMeter Gatling Tsung The Grinder LoadUI LoadRunner Neoload Which should you use? ¯\_(ツ)_/¯
5. My Reasons • Gatling lets you write code • It’s not XML • Open Source • I don’t work with Java developers, and there are Scala developers where I work
6. …other reasons • Nice charts! • no heavy IDE (works with Vim!) ! ! ! ! • yeah...there’s a recorder too
7. Gatling is… …a load test DSL written in Scala ! it leverages Scala’s concurrency so it’s fast, efficient, and takes advantage of other functional things I don’t understand...
8. …but this is easy enough to understand https://blog.flood.io/stress-testing-jmeter-and-gatling/
9. A Gatling Simulation is made up of Virtual Users who carry out repeated Scenarios over time
10. Simulation ./bin/gatling.sh -s LoadTest 1 class LoadTest extends Simulation {! 2 ...! 3 }!
11. Scenario 1 val edit = exec(http("Form")! 2 .get("/computers/new"))! 3 .pause(1)! 4 .exec(http("Post")! 5 .post("/computers")! 6 .headers(headers_10)! 7 .formParam("name", "Computer")! 8 .formParam("company", "37"))! 9 }!
12. Virtual Users 1 setUp( ! 2 scenario1.inject(atOnceusers(1)),! 3 scenario2.inject(rampUsers(10) over (2 ! 4 seconds)),! 5 scenario3.inject(constantUsersPerSec(2) ! 6 over (10 seconds))! 7 )!
13. And now for an extended example...
14. The Pizza API Where is your favorite pizza? A project to track the the best pizza places in the world! (unless you like deep dish*) ! https://github.com/stevekinney/pizza * this repository’s hating on deep dish pizza does not represent the views of the author of this presentation
15. INDEX: returns all locations api/v1/pizzerias ! SHOW: returns specific location by id api/v1/pizzerias/:some_pizzeria_id ! SEARCH: returns all properties for valid params api/v1/properties/search?property=value live at: http://pizza-dev.elasticbeanstalk.com* *Please don’t break my server until after the presentation
16. 1 class LoadTest extends Simulation {! 2 val httpConf = http! 3 .baseURL("http://pizza-dev.elasticbeanstalk.com")! 4 ! 5 val scn = scenario("Pizzerias")! 6 .exec(http("get all pizzerias")! 7 .get("/api/v1/pizzerias"))! 8 ! 9 setUp( 10 scn.inject(atOnceUsers(100)) 11 ).protocols(httpConf) 12 } ! A simple index test
18. Great...so we got the list page test coded, what about individual pizzerias?
19. 1 class Pizzerias2 extends Simulation {! 2 ! 3 val httpProtocol = http! 4 .baseURL("http://pizza-dev.elasticbeanstalk.com")! 5 ! 6 val scnA = scenario("Pizza Place 1")! 7 .exec(http("get Pizza Place 1")! 8 .get("/api/v1/pizzerias/1"))! 9 ! 10 val scnB = scenario("Pizza Place 2")! 11 .exec(http("get Pizza Place 2")! 12 .get("/api/v1/pizzerias/2"))! 13 ! 14 setUp(! 15 scnA.inject(constantUsersPerSec(10) during(30 seconds)),! 16 scnB.inject(constantUsersPerSec(10) during(30 seconds))! 17 ).protocols(httpProtocol)! 18 }! 19 ! 20 !
20. • some applications cache the answers to certain requests • if the same request is made over and over again, the cache is used to reply quickly • if you want to characterize the actual service, caching will skew your results • if you are going for real world load, this might not be bad Aside: caching
21. • We need a feeder • Feeders are gatling’s way of setting up an iterator that will make data available to a Scenario in a map • You can make an iterator yourself • You can make an iterator from a CSV • You can make an iterator from a JSON source • and others... How do we fix this?
22. 1 Iterator.continually(Map("id" -> (Random.nextInt(133))))
23. 1 class PizzeriasRandom extends Simulation {! 2 ! 3 val httpProtocol = http! 4 .baseURL("http://pizza-dev.elasticbeanstalk.com")! 5 ! 6 val feeder = Iterator.continually(Map("id" ->! 7 (Random.nextInt(133)))) ! 8 ! 9 val scnR = scenario("Pizza Place Random")! 10 .feed(feeder)! 11 .exec(http("get Pizza Place 1")! 12 .get("/api/v1/pizzerias/${id}"))! 13 ! 14 setUp(scnR! 15 .inject(constantUsersPerSec(2) during(60 seconds)))! 16 .protocols(httpProtocol)! 17 }
24. remember what the API looks like: ! SEARCH: returns all properties for valid params ! api/v1/properties/search?some_param=some query And now for search
25. 1 class PizzeriasQuery extends Simulation {! 2 ! 3 val httpProtocol = http! 4 .baseURL("http://pizza-dev.elasticbeanstalk.com")! 5 ! 6 val scn = scenario("Pizza Place Searches")! 7 .exec(http("search for pizza place")! 8 .get("/api/v1/properties/search")! 9 .queryParam("city", "Oklahoma City"))! 10 ! 11 setUp(scn.inject(constantUsersPerSec(2) ! 12 during(10 seconds)))! 13 .protocols(httpProtocol)! 14 }! 15 !
26. 1 property,value! 2 city,Oklahoma City! 3 city,Boston! 4 city,Dallas! 5 city,Atlantis!
27. 1 class PizzeriasQueryWithCSV extends Simulation {! 2 ! 3 val httpProtocol = http! 4 .baseURL("http://pizza-dev.elasticbeanstalk.com")! 5 ! 6 val searches = csv("pizza_search.csv").circular! 7 ! 8 val scn = scenario("Pizza Place Searches")! 9 .feed(searches)! 10 .exec(http("search for pizza place")! 11 .get("/api/v1/properties/search")! 12 .queryParam("${property}","${value}"))! 13 ! 14 setUp(scn.inject(constantUsersPerSec(2) ! 15 during(10 seconds)))! 16 .protocols(httpProtocol)! 17 } !
28. Debugging 1 <?xml version="1.0" encoding="UTF-8"?>! 2 <configuration>! 3 ! 4 <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">! 5 <encoder>! 6 <pattern>%d{HH:mm:ss.SSS} [%-5level] %logger{15} - %msg%n%rEx</pattern>! 7 <immediateFlush>false</immediateFlush>! 8 </encoder>! 9 </appender>! 10 ! 11 <!-- Uncomment for logging ALL HTTP request and responses -->! 12 <!-- <logger name="io.gatling.http.ahc" level="TRACE" /> -->! 13 <!-- <logger name="io.gatling.http.response" level="TRACE" /> -->! 14 <!-- Uncomment for logging ONLY FAILED HTTP request and responses -->! 15 <!-- <logger name="io.gatling.http.ahc" level="DEBUG" /> -->! 16 <!-- <logger name="io.gatling.http.response" level="DEBUG" /> -->! 17 ! 18 <root level="WARN">! 19 <appender-ref ref="CONSOLE" />! 20 </root>! 21 ! 22 </configuration>!
29. 1 Request: ! 2 search for pizza place: OK ! 3 =========================! 4 Session:! 5 Session(Pizza Place Searches,1480189078212752348-! 6 19,Map(property -> city, value -> Atlantis),14421! 7 13956702,0,OK,List(),<function1>)! 8 =========================! 9 HTTP request:! 10 GET http://pizza-dev.elasticbeanstalk.com/api/v1! 11 /properties/search?city=Atlantis! 12 headers=! 13 Connection: keep-alive! 14 Host: pizza-dev.elasticbeanstalk.com! 15 Accept: */*! 16 =========================! 17 HTTP response:! 18 status=! 19 200 OK! 20 headers= ! 21 Content-Type: [application/json]! 22 Date: [Sun, 13 Sep 2015 03:14:12 GMT]! 23 Server: [nginx/1.6.2]! 24 X-Content-Type-Options: [nosniff]! 25 Content-Length: [2]! 26 Connection: [keep-alive]! 27 ! 28 body=! 29 []!
30. Validation 1 class PizzeriasQueryWithCheck extends Simulation {! 2 ! 3 val httpProtocol = http! 4 .baseURL("http://pizza-dev.elasticbeanstalk.com")! 5 ! 6 val searches = csv("pizza_search.csv").circular! 7 ! 8 val scn = scenario("Pizza Place Searches")! 9 .feed(searches)! 10 .exec(http("search for pizza place")! 11 .get("/api/v1/properties/search")! 12 .queryParam("${property}","${value}")! 13 .check(bodyString.not("[]")))! 14 ! 15 setUp(scn.inject(constantUsersPerSec(2) ! 16 during(10 seconds)))! 17 .protocols(httpProtocol)! 18 }! 19 !
31. Special Cases Or, how to know when your tests aren’t measuring what you think they’re measuring • Excessive Queuing • Testing caching instead of your service • Cannot assign requested address errors
32. Excessive Queuing • Symptom • You are sending more packets than your system says it’s receiving • You don’t see any errors • Fix • Back off the rate you are sending
33. Testing caching instead of your service • Symptom • system responds faster than expected no matter what load you put on it • Fix • vary the types of requests so they are more random
34. Cannot assign requested address errors Symptom • You get “Cannot assign requested address” errors during a load test • Other out of memory errors Fix • see note about how Gatling uses ports on the host OS: http://gatling.io/ docs/2.1.7/general/simulation_setup.html?highlight=simulation#injection • Follow tuning guidelines: http://gatling.io/docs/2.1.7/general/ operations.html?highlight=tuning#os-tuning • use a repeat statement in your scenarios to make connections last longer
35. How to get help with Gatling • Documentation: http://gatling.io/docs/2.1.7/ index.html • Gatling User Group: https://groups.google.com/forum/#!forum/gatling
36. The Pizza API and my Gatling repo • pizza api: https://github.com/stevekinney/pizza • gatling examples: https://github.com/abaird/ gatling_examples • running in Elastic Beanstalk: ?? • script to setup loadtest machine in AWS: ??
37. Happy Loadtesting!