Waiting is boring

64999497a1bea392bb5f23c9353e6c3a?s=47 Filipe Freire
September 30, 2017

Waiting is boring

Talk about waiting in software testing and display problems and solutions found to test async scenarios in services with messaging systems, frontend apps and distributed stream processing apps.

64999497a1bea392bb5f23c9353e6c3a?s=128

Filipe Freire

September 30, 2017
Tweet

Transcript

  1. Waiting is boring Filipe Freire - 30 September 2017

  2. Quick intro Husband, learner, tester, developer
 (new) OS contributor 2y

    work as a “coding” tester
 1y work as a developer 2 Currently @
  3. Disclaimer This talk is about waiting in software testing. Expect

    an imperfect personal account on problems and solutions. 3
  4. What you can take from this Tips on testing async

    scenarios in: A) services with messaging systems
 B) frontend apps
 C) distributed stream processing apps. 4
  5. 1) Intro 5

  6. –Venkat Subramaniam (twitter, 1 September 2017) “It’s foolish to wait

    forever for something to happen. Both in life and in asynchronous programming, timeouts are critical.” 6
  7. 7

  8. What about software? (testing-wise) 8

  9. Why do we test? 9

  10. “To verify something
 can work” 10

  11. (Or “Fact checks”) Automated tests Testing effort 11

  12. 12 source: martinfowler.com

  13. Theres more to it, but lets focus on time 13

  14. Syncing test’s actions with results, typical first approach… 14

  15. Thread.sleep(ms); usleep(microseconds); time.sleep(time_ms) System.Threading.Thread.Sleep(msecs); 15

  16. We let the computer wait a) 5 seconds b) 10

    seconds c) 1 minute d) 1 hour … 16
  17. If we’re lucky with our “estimate” 17

  18. 18

  19. It’s never just X seconds. 19

  20. How do we fix avoid it? 20

  21. 2) Example: API & messaging 21

  22. The context Warehousing & Logistics project 60 person project Tons

    of Microservices & Monoliths (different teams) Comms: REST and AMQP (RabbitMQ) 22
  23. Typical “End-to-End” Event triggered Expected state ? Producer Consumer Exchange

    Queue Live Live 23
  24. Different teams & test tools design patterns
 containers dependency injection

    magic java “scripts” magic A B Hammered sleeps 24
  25. One more thing… Inverted test pyramid And the Hammered sleeps

    25
  26. “Inverted pyramid?
 Not my money!” 26

  27. So the sleeps then… 27

  28. “Typical day” 20 different async comm flows each w/ 20

    variations in payload
 (+ “inf” minor variations) 28
  29. “Typical day” Some math: 20 flows times 
 20 variations

    times 
 5 sec sleep = half hour (+/-) 29
  30. “Typical day” Some math: 20 flows times 
 20 variations

    times 
 10 sec sleep = (not uncommon) 1 hour (+/-) 30
  31. Solutions 1. Invert the damn pyramid 2. Don’t hammer a

    wait 31
  32. Solutions 1. Invert the damn pyramid 2. Don’t hammer a

    sleep 32
  33. How do we avoid a hammered sleep? 33

  34. The “scholar”-way 34

  35. The “scholar”-way 35

  36. The “do it for me”-way 36

  37. The “do it for me”-way + AssertJ 37

  38. 3) Example: Frontend 38

  39. The context Back-office projects ( >1 project) “Latest frameworks &

    tools” No FE unit tests. No FE tests. “Let’s make end-to-end FE tests” 39
  40. Sorry, what? Back-office projects ( >1 project) “Latest frameworks &

    tools” No FE unit tests. No FE tests. “Let’s make end-to-end FE tests” 40
  41. ? No unit tests. No tests. 41

  42. ??? No unit tests. No tests. 42

  43. Scream disappointed. 43

  44. Now let’s pretend we don’t have that problem, and focus

    
 on end-to-end tests 44
  45. And the other problem Elements visibility takes time Interactions take

    time Animations take time Requests take time 45
  46. How do we fight this “time”? 46

  47. The “spit upwards” approach >60 tests Hammer big waits Wait

    for everything to be ready Result: 2h test suite 47
  48. Should we run the tests in 
 parallel? Not yet.

    48
  49. The “moderate” approach 45 tests (clean-up) Small individual fixed waits

    Result: 40min test suite 49
  50. The “surgeon” approach 30 tests (keep the “good” ones) Small

    individual waits Waits are counter-like Result: 20min test suite 50
  51. The “Duke Nukem” approach 10 tests (less is more) Fluent

    waits Result: < 6min test suite 51
  52. Fluent waits? “Implicit… explicit… mayonese” We don’t need to reinvent

    the wheel. Selenium libs have this for ages. 52
  53. Fluent waits! Note: If you’re using WebDriverWait
 you’re using FluentWait

    53
  54. Expected conditions Check the doc: https://goo.gl/J17NVD 54

  55. Google it today. It’s worth it. 55

  56. 3) Example: Stream Processing
 (i.e. Apache Storm) 56

  57. The context (new) project 10 person team Process half million

    critical messages p/day This is just the beginning (MVP) 57
  58. A recent problem Storm + Scala project 1 small integration

    (e2e) test suite Critical sanity/regression checks Needs to take less than 2-3 min. How do we handle waits? 58
  59. How a typical e2e test works Topology Consumer Event triggered

    Expected state ? 59
  60. Do we hammer 10 sec waits? Maybe 20 sec? 60

  61. Nope. We use what we have at hand. 61

  62. Scalatest’s “eventually” 62

  63. How you can do it 63

  64. Why is time that important? 64

  65. If e2e take 3 mins, Our current pipeline takes 10

    mins 65
  66. If e2e take 6 mins (+3) Our CI/CD pipeline takes

    13 mins 
 10 runs per day Time wasted per day: 30 mins Per month: 10 hours 66
  67. 4) Conclusions 67

  68. –Michael Bolton (twitter, September 2017) “If you’re not scrutinising test

    code, why do you trust it any more than your production code? Especially when no problems are reported?” 68
  69. –Michael Bolton (twitter, September 2017) “Lesson: don’t blindly trust your

    test code or your automated checks, lest they fail to reveal important problems in production code.” 69
  70. Smarter waits don’t solve… 70

  71. Immutable poor culture 71

  72. Immutable poor decisions 72

  73. Waiting for change is boring. Stop waiting. 73

  74. Thank you. Questions? filfreire filrfreire 74