Waiting is boring

64999497a1bea392bb5f23c9353e6c3a?s=47 Filipe Freire
September 30, 2017

Waiting is boring

Talk about waiting in software testing and display problems and solutions found to test async scenarios in services with messaging systems, frontend apps and distributed stream processing apps.

64999497a1bea392bb5f23c9353e6c3a?s=128

Filipe Freire

September 30, 2017
Tweet

Transcript

  1. 2.

    Quick intro Husband, learner, tester, developer
 (new) OS contributor 2y

    work as a “coding” tester
 1y work as a developer 2 Currently @
  2. 3.

    Disclaimer This talk is about waiting in software testing. Expect

    an imperfect personal account on problems and solutions. 3
  3. 4.

    What you can take from this Tips on testing async

    scenarios in: A) services with messaging systems
 B) frontend apps
 C) distributed stream processing apps. 4
  4. 6.

    –Venkat Subramaniam (twitter, 1 September 2017) “It’s foolish to wait

    forever for something to happen. Both in life and in asynchronous programming, timeouts are critical.” 6
  5. 7.

    7

  6. 16.

    We let the computer wait a) 5 seconds b) 10

    seconds c) 1 minute d) 1 hour … 16
  7. 18.

    18

  8. 22.

    The context Warehousing & Logistics project 60 person project Tons

    of Microservices & Monoliths (different teams) Comms: REST and AMQP (RabbitMQ) 22
  9. 24.

    Different teams & test tools design patterns
 containers dependency injection

    magic java “scripts” magic A B Hammered sleeps 24
  10. 28.

    “Typical day” 20 different async comm flows each w/ 20

    variations in payload
 (+ “inf” minor variations) 28
  11. 29.

    “Typical day” Some math: 20 flows times 
 20 variations

    times 
 5 sec sleep = half hour (+/-) 29
  12. 30.

    “Typical day” Some math: 20 flows times 
 20 variations

    times 
 10 sec sleep = (not uncommon) 1 hour (+/-) 30
  13. 39.

    The context Back-office projects ( >1 project) “Latest frameworks &

    tools” No FE unit tests. No FE tests. “Let’s make end-to-end FE tests” 39
  14. 40.

    Sorry, what? Back-office projects ( >1 project) “Latest frameworks &

    tools” No FE unit tests. No FE tests. “Let’s make end-to-end FE tests” 40
  15. 45.

    And the other problem Elements visibility takes time Interactions take

    time Animations take time Requests take time 45
  16. 47.

    The “spit upwards” approach >60 tests Hammer big waits Wait

    for everything to be ready Result: 2h test suite 47
  17. 50.

    The “surgeon” approach 30 tests (keep the “good” ones) Small

    individual waits Waits are counter-like Result: 20min test suite 50
  18. 51.
  19. 52.
  20. 57.

    The context (new) project 10 person team Process half million

    critical messages p/day This is just the beginning (MVP) 57
  21. 58.

    A recent problem Storm + Scala project 1 small integration

    (e2e) test suite Critical sanity/regression checks Needs to take less than 2-3 min. How do we handle waits? 58
  22. 66.

    If e2e take 6 mins (+3) Our CI/CD pipeline takes

    13 mins 
 10 runs per day Time wasted per day: 30 mins Per month: 10 hours 66
  23. 68.

    –Michael Bolton (twitter, September 2017) “If you’re not scrutinising test

    code, why do you trust it any more than your production code? Especially when no problems are reported?” 68
  24. 69.

    –Michael Bolton (twitter, September 2017) “Lesson: don’t blindly trust your

    test code or your automated checks, lest they fail to reveal important problems in production code.” 69