What is a flaky test?
A test that produces ‘random’ result when ran
multiple times agains the same application.
@EmanuilSlavov
Slide 3
Slide 3 text
Why bother?
@EmanuilSlavov
Slide 4
Slide 4 text
Adding 1 test a day.
Assuming 5% of tests are flaky.
Increasing time to fix a test - from 1 to 11 minutes.
0
35
70
105
140
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
@EmanuilSlavov
Slide 5
Slide 5 text
Adding 1 test a day.
Assuming 5% of tests are flaky.
Increasing time to fix a test - from 1 to 11 minutes.
0
35
70
105
140
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
@EmanuilSlavov
Slide 6
Slide 6 text
Why listen to this presentation?
Google’s flaky test rate: 1.5%*
Komfo’s flaky test rate: 0.13%
*Blog Post: Flaky Tests at Google and How We Mitigate Them
@EmanuilSlavov
Slide 7
Slide 7 text
How to detect a flaky test?
After the tests run is done, rerun the failing ones, if any.
If the failing ones pass on the second try, then they are flaky.
@EmanuilSlavov
Slide 8
Slide 8 text
Where do they occur?
Unit Tests - run only in memory
API Tests - fully deployed system
UI Tests - fully deployed system
@EmanuilSlavov
Slide 9
Slide 9 text
Flaky Tests Kata
Slide 10
Slide 10 text
Immediately investigate the flaky test
Consider deleting the test if fix is impossible
The cause may may not be the test
@EmanuilSlavov
Slide 11
Slide 11 text
78%of all tests are flaky the
first time they are written*
*An Empirical Analysis of Flaky Tests, Qingzhou Luo et al., University of Illinois 2014
When you are done with the fix…
@EmanuilSlavov
Slide 12
Slide 12 text
for i in {1..100}; do if ! execute_test ; then break; fi; done;
single test execution command
stop if the test fails even once
run it 100 times
@EmanuilSlavov
Slide 13
Slide 13 text
The Ideal Test
At the right level
Independent
Tests one thing only
@EmanuilSlavov
Slide 14
Slide 14 text
What causes flaky tests?
@EmanuilSlavov
Slide 15
Slide 15 text
Lack of APIs forces testing only through the UI
Most business logic in not in the UI
Architecture
@EmanuilSlavov
Slide 16
Slide 16 text
No content
Slide 17
Slide 17 text
24%of our test failures were
due to shared resources
Shared Resources
@EmanuilSlavov
Slide 18
Slide 18 text
39%of the test failures were
due to async causes
Handling of Async Operations
@EmanuilSlavov
Slide 19
Slide 19 text
result = @user.search(items)
result.first[‘channel_id'].should == channel_id
sleeping(1).seconds.between_tries.failing_after(5).tries do
end
async operation
assert
retry up to 5 times, in case of exception
@EmanuilSlavov
Slide 20
Slide 20 text
3rd Party Services
Unreliable Network
Throttling
Costly
Can’t Control Output
@EmanuilSlavov
The time needed to create data for a single test:
And then the test starts
Call 12 API endpoints
Modify data in 11 tables
This setup takes
about 1.2 seconds
@EmanuilSlavov
Slide 28
Slide 28 text
The Trouble with UI tests
Slide 29
Slide 29 text
Too many moving parts
The only “true” testing?!?
Dev/QA division, outsource
UI is made for human consumption
@EmanuilSlavov
Slide 30
Slide 30 text
AUTOMATION ≠ SELENIUM
@EmanuilSlavov
Slide 31
Slide 31 text
UI Tests
Unit Tests API Tests
UI Tests
@EmanuilSlavov
Slide 32
Slide 32 text
If you still need to write UI tests….
@EmanuilSlavov
Slide 33
Slide 33 text
Basic Fixes
@EmanuilSlavov
Slide 34
Slide 34 text
Don’t use XPath locators
No record and replay
Smart Wait for UI elements
Wait for all Ajax calls completion
@EmanuilSlavov
Slide 35
Slide 35 text
Advanced Fixes
@EmanuilSlavov
Slide 36
Slide 36 text
Skip Login
5 seconds
22 seconds
Use token in the URL, cookie, browser plugins
@EmanuilSlavov
Slide 37
Slide 37 text
No Setup in the UI
Insert test data directly with API or in the DB
@EmanuilSlavov
Slide 38
Slide 38 text
No Navigation
Insert test data directly with API or in the DB
@EmanuilSlavov
Slide 39
Slide 39 text
Conclusion
Test at the right level
Full control of the test environment
Immediately investigate flaky tests
Collaborate with developers
UI test should have minimal interactions
@EmanuilSlavov
Slide 40
Slide 40 text
Recommended Reading
@EmanuilSlavov
Slide 41
Slide 41 text
Please, rate this talk.
EmanuilSlavov.com
@EmanuilSlavov
speakerdeck.com/emanuil