Waiting is boring - Speaker Deck

Slide 1

Slide 1 text

Waiting is boring Filipe Freire - 30 September 2017

Slide 2

Slide 2 text

Quick intro Husband, learner, tester, developer  (new) OS contributor 2y work as a “coding” tester  1y work as a developer 2 Currently @

Slide 3

Slide 3 text

Disclaimer This talk is about waiting in software testing. Expect an imperfect personal account on problems and solutions. 3

Slide 4

Slide 4 text

What you can take from this Tips on testing async scenarios in: A) services with messaging systems  B) frontend apps  C) distributed stream processing apps. 4

Slide 5

Slide 5 text

1) Intro 5

Slide 6

Slide 6 text

–Venkat Subramaniam (twitter, 1 September 2017) “It’s foolish to wait forever for something to happen. Both in life and in asynchronous programming, timeouts are critical.” 6

Slide 7

Slide 7 text

Slide 8

Slide 8 text

What about software? (testing-wise) 8

Slide 9

Slide 9 text

Why do we test? 9

Slide 10

Slide 10 text

“To verify something  can work” 10

Slide 11

Slide 11 text

(Or “Fact checks”) Automated tests Testing effort 11

Slide 12

Slide 12 text

12 source: martinfowler.com

Slide 13

Slide 13 text

Theres more to it, but lets focus on time 13

Slide 14

Slide 14 text

Syncing test’s actions with results, typical first approach… 14

Slide 15

Slide 15 text

Thread.sleep(ms); usleep(microseconds); time.sleep(time_ms) System.Threading.Thread.Sleep(msecs); 15

Slide 16

Slide 16 text

We let the computer wait a) 5 seconds b) 10 seconds c) 1 minute d) 1 hour … 16

Slide 17

Slide 17 text

If we’re lucky with our “estimate” 17

Slide 18

Slide 18 text

Slide 19

Slide 19 text

It’s never just X seconds. 19

Slide 20

Slide 20 text

How do we fix avoid it? 20

Slide 21

Slide 21 text

2) Example: API & messaging 21

Slide 22

Slide 22 text

The context Warehousing & Logistics project 60 person project Tons of Microservices & Monoliths (different teams) Comms: REST and AMQP (RabbitMQ) 22

Slide 23

Slide 23 text

Typical “End-to-End” Event triggered Expected state ? Producer Consumer Exchange Queue Live Live 23

Slide 24

Slide 24 text

Different teams & test tools design patterns  containers dependency injection magic java “scripts” magic A B Hammered sleeps 24

Slide 25

Slide 25 text

One more thing… Inverted test pyramid And the Hammered sleeps 25

Slide 26

Slide 26 text

“Inverted pyramid?  Not my money!” 26

Slide 27

Slide 27 text

So the sleeps then… 27

Slide 28

Slide 28 text

“Typical day” 20 different async comm flows each w/ 20 variations in payload  (+ “inf” minor variations) 28

Slide 29

Slide 29 text

“Typical day” Some math: 20 flows times   20 variations times   5 sec sleep = half hour (+/-) 29

Slide 30

Slide 30 text

“Typical day” Some math: 20 flows times   20 variations times   10 sec sleep = (not uncommon) 1 hour (+/-) 30

Slide 31

Slide 31 text

Solutions 1. Invert the damn pyramid 2. Don’t hammer a wait 31

Slide 32

Slide 32 text

Solutions 1. Invert the damn pyramid 2. Don’t hammer a sleep 32

Slide 33

Slide 33 text

How do we avoid a hammered sleep? 33

Slide 34

Slide 34 text

The “scholar”-way 34

Slide 35

Slide 35 text

The “scholar”-way 35

Slide 36

Slide 36 text

The “do it for me”-way 36

Slide 37

Slide 37 text

The “do it for me”-way + AssertJ 37

Slide 38

Slide 38 text

3) Example: Frontend 38

Slide 39

Slide 39 text

The context Back-office projects ( >1 project) “Latest frameworks & tools” No FE unit tests. No FE tests. “Let’s make end-to-end FE tests” 39

Slide 40

Slide 40 text

Sorry, what? Back-office projects ( >1 project) “Latest frameworks & tools” No FE unit tests. No FE tests. “Let’s make end-to-end FE tests” 40

Slide 41

Slide 41 text

? No unit tests. No tests. 41

Slide 42

Slide 42 text

??? No unit tests. No tests. 42

Slide 43

Slide 43 text

Scream disappointed. 43

Slide 44

Slide 44 text

Now let’s pretend we don’t have that problem, and focus   on end-to-end tests 44

Slide 45

Slide 45 text

And the other problem Elements visibility takes time Interactions take time Animations take time Requests take time 45

Slide 46

Slide 46 text

How do we fight this “time”? 46

Slide 47

Slide 47 text

The “spit upwards” approach >60 tests Hammer big waits Wait for everything to be ready Result: 2h test suite 47

Slide 48

Slide 48 text

Should we run the tests in   parallel? Not yet. 48

Slide 49

Slide 49 text

The “moderate” approach 45 tests (clean-up) Small individual fixed waits Result: 40min test suite 49

Slide 50

Slide 50 text

The “surgeon” approach 30 tests (keep the “good” ones) Small individual waits Waits are counter-like Result: 20min test suite 50

Slide 51

Slide 51 text

The “Duke Nukem” approach 10 tests (less is more) Fluent waits Result: < 6min test suite 51

Slide 52

Slide 52 text

Fluent waits? “Implicit… explicit… mayonese” We don’t need to reinvent the wheel. Selenium libs have this for ages. 52

Slide 53

Slide 53 text

Fluent waits! Note: If you’re using WebDriverWait  you’re using FluentWait 53

Slide 54

Slide 54 text

Expected conditions Check the doc: https://goo.gl/J17NVD 54

Slide 55

Slide 55 text

Google it today. It’s worth it. 55

Slide 56

Slide 56 text

3) Example: Stream Processing  (i.e. Apache Storm) 56

Slide 57

Slide 57 text

The context (new) project 10 person team Process half million critical messages p/day This is just the beginning (MVP) 57

Slide 58

Slide 58 text

A recent problem Storm + Scala project 1 small integration (e2e) test suite Critical sanity/regression checks Needs to take less than 2-3 min. How do we handle waits? 58

Slide 59

Slide 59 text

How a typical e2e test works Topology Consumer Event triggered Expected state ? 59

Slide 60

Slide 60 text

Do we hammer 10 sec waits? Maybe 20 sec? 60

Slide 61

Slide 61 text

Nope. We use what we have at hand. 61

Slide 62

Slide 62 text

Scalatest’s “eventually” 62

Slide 63

Slide 63 text

How you can do it 63

Slide 64

Slide 64 text

Why is time that important? 64

Slide 65

Slide 65 text

If e2e take 3 mins, Our current pipeline takes 10 mins 65

Slide 66

Slide 66 text

If e2e take 6 mins (+3) Our CI/CD pipeline takes 13 mins   10 runs per day Time wasted per day: 30 mins Per month: 10 hours 66

Slide 67

Slide 67 text

4) Conclusions 67

Slide 68

Slide 68 text

–Michael Bolton (twitter, September 2017) “If you’re not scrutinising test code, why do you trust it any more than your production code? Especially when no problems are reported?” 68

Slide 69

Slide 69 text

–Michael Bolton (twitter, September 2017) “Lesson: don’t blindly trust your test code or your automated checks, lest they fail to reveal important problems in production code.” 69