Optimizing Selenium for build performance

Tricks and tips for creating a fast and scalable test codebase.

Transcript

1. Optimizing Selenium for build performance Tricks and tips for creating a fast and scalable test codebase http://bit.ly/optimizing-selenium
2. Intro Santiago Suarez Ordoñez >5 years with Selenium and test automation Selenium committer Sauce Ninja at Sauce Labs @santiycr sso@saucelabs.com
3. Sauce Labs
4. Agenda • Ensure data independence • Don't DDOS your own QA server • Write atomic tests • Focus on the essential • Generate app state • Use guinea pigs • Remove third party • Get rid of static pauses • Split your build • Watch your build
5. The main secret to faster Selenium tests
6. Parallelization Hardware == cheap Engineers == $$$$
7. It takes some work to get there... Paralellization is not easy
8. Don't DDOS Your Server Serial tests == very low load 50 concurrent browsers == lots of requests Pay attention and when needed, get a web server that better resembles production
9. Data Independence Execution order => Data dependency Parallel tests == no order Never use data sets more than once! Wipe your DB on every build
10. Data Dependency
11. Data Collision
12. Atomic Tests TestSignupLoginChangePasswdLogout :,( Small tests: • run in seconds • report failures precisely • have higher ROI in parallel Small tests are good tests
13. Focus on the Essential Don't test content during functional tests Write integrity tests Avoid Selenium for plain content when possible Don't test performance during functional tests Write performance tests Test performance in a controlled environment
14. Generate Application State Generate state out of the browser The UI is inefficient Causes interdependence (Y can't run unless X works) Keep manual versions for special cases
15. Generate State via DB/Memcache
16. Generate State via Web Server
17. Use Guinea Pigs Test widgets in isolation Test extensively on an empty page Run smoke tests on any real page
18. Guinea Pig Example
19. Remove Third Party Where Possible Anything outside your control slows you down Adds non-determinism (brittleness) Test third party with specific third party tests Run functional tests including third party on a separate schedule
20. Get Rid of all Static Pauses Every pause is there for a purpose Find that purpose and wait smartly Every second spent waiting for something that already happened, is a second wasted by your full development team
21. Break Your Build in Test Steps Not all tests are worth parallelizing :,( Doesn't mean all of them need to go serially Have a serial tests and a parallel tests step
22. Watch your build Sit at your build while it runs Check logs Find bottlenecks
23. QA
24. sso@saucelabs.com @santiycr Thanks!