NEED FOR SPEED
accelerate tests from 3 hours to 3 minutes
[email protected]
Slide 2
Slide 2 text
3
hours
3
minutes
600 API tests
Slide 3
Slide 3 text
Before After
The
3 Minute
Goal
Slide 4
Slide 4 text
It’s not about the numbers or
techniques you’ll see.
It’s all about continuous
improvement.
Slide 5
Slide 5 text
Dedicated
Environment
Slide 6
Slide 6 text
Execution Time in Minutes
180
123 New Environment
Slide 7
Slide 7 text
Empty Databases
Slide 8
Slide 8 text
The time needed to create data for one test:
And then the test starts
Call 12 API endpoints
Modify data in 11 tables Takes about 1.2 seconds
Slide 9
Slide 9 text
180
123
Execution Time in Minutes
89 Empty Databases
Slide 10
Slide 10 text
Simulate
Dependencies
Slide 11
Slide 11 text
+Some
More
STUB
STUB
STUB
STUB
STUB
STUB
STUB
Stub all external dependencies
Core
API
Slide 12
Slide 12 text
Transparent
Fake SSL certs
Dynamic Responses
Local Storage
Return Binary Data
Regex URL match
Existing Tools (March 2016)
Stubby4J
WireMock
Wilma
soapUI
MockServer
mounteback
Hoverfly
Mirage
We created project Nagual,
open source soon.
Slide 13
Slide 13 text
180
123
89
Execution Time in Minutes
65 Stub Dependencies
Slide 14
Slide 14 text
Move to Containers
Slide 15
Slide 15 text
180
123
89
65
Execution Time in Minutes
104 Using Containers
Slide 16
Slide 16 text
Run Databases
in Memory
Slide 17
Slide 17 text
180
123
89
65
104
Execution Time in Minutes
61 Run Databases in Memory
Slide 18
Slide 18 text
Don’t Clean
Test Data
Slide 19
Slide 19 text
180
123
89
65
104
61
Execution Time in Minutes
46 Don’t delete test data
Slide 20
Slide 20 text
Run in Parallel
Slide 21
Slide 21 text
4 6 8 10 12
14 16
Time to execute 12 9 7
5 8 12 17
The Sweet Spot
Slide 22
Slide 22 text
180
123
89
65
104
61
46
Execution Time in Minutes
5 Run in Parallel
Slide 23
Slide 23 text
Equalize Workload
Slide 24
Slide 24 text
No content
Slide 25
Slide 25 text
No content
Slide 26
Slide 26 text
180
123
89
65
104
61
46
5
Execution Time in Minutes
3 Equal Batches
Run in Parallel
Don’t delete test data
Run Databases in Memory
Using Containers
Stub Dependencies
Empty Databases
New Environment
Slide 27
Slide 27 text
After Hardware Upgrade
The Outcome
2:15 min.
1:38 min.
Slide 28
Slide 28 text
The tests are slow
The tests are unreliable
The tests can’t exactly pinpoint the problem
High Level Tests Problems
3 Minutes
No external dependencies
It’s cheap to run all tests after every change
Slide 29
Slide 29 text
In a couple of years, running all your
automated tests, after every code
change, for less than 3 minutes, will be
standard development practice.