Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Flaky Tests - Fighting Nightmares

Flaky Tests - Fighting Nightmares

"A liar will not be believed even if he tells the truth" - Aesop

A test can also be such a liar: Unreliable tests are living nightmares for anyone who writes automated tests or even pays attention to their results. Sometimes they pass, sometimes they don't, and therefore do not give a valid statement about the state of the software. In the worst case, they can impair the credibility of the entire test suite. It doesn't matter whether you write unit, integration or end-to-end tests, you can counter them in any way you like.

I've spent hours and hours on such tests, and I want to share my path through them and the lessons learned from them: So that together we can avoid or even get rid of these nightmares.

36751965e79dca4618cccc3c08efc912?s=128

Ramona Schwering

September 25, 2020
Tweet

Transcript

  1. Flaky tests - Fighting nightmares @leichteckig Flaky tests Flaky tests

    Fighting nightmares Fighting nightmares
  2. Flaky tests - Fighting nightmares @leichteckig

  3. Flaky tests - Fighting nightmares @leichteckig Our Test Automation Stack

    Our Test Automation Stack
  4. Flaky tests - Fighting nightmares @leichteckig The boy who cried

    wolf The boy who cried wolf
  5. Flaky tests - Fighting nightmares @leichteckig A liar will not

    be believed, even when he speaks the truth. (Aesop)
  6. Flaky tests - Fighting nightmares @leichteckig An ugly pattern emerges

    An ugly pattern emerges
  7. Flaky tests - Fighting nightmares @leichteckig “oh that test, yeah

    it sometimes fails, kick the build o again ” - Some dev from my team
  8. Flaky tests - Fighting nightmares @leichteckig Spotting the causes Spotting

    the causes
  9. Flaky tests - Fighting nightmares @leichteckig Test-sided causes Test-sided causes

  10. Flaky tests - Fighting nightmares @leichteckig Hardcoded IDs or other

    data
  11. Flaky tests - Fighting nightmares @leichteckig

  12. Flaky tests - Fighting nightmares @leichteckig Random- or generated demo

    data
  13. Flaky tests - Fighting nightmares @leichteckig

  14. Flaky tests - Fighting nightmares @leichteckig Tests with cross dependencies

  15. Flaky tests - Fighting nightmares @leichteckig

  16. Flaky tests - Fighting nightmares @leichteckig Incorrect assumptions about order

    and time
  17. Flaky tests - Fighting nightmares @leichteckig Environment-sided causes Environment-sided causes

  18. Flaky tests - Fighting nightmares @leichteckig Resource leaks

  19. Flaky tests - Fighting nightmares @leichteckig

  20. Flaky tests - Fighting nightmares @leichteckig Incompatibilities between dependencies

  21. Flaky tests - Fighting nightmares @leichteckig

  22. Flaky tests - Fighting nightmares @leichteckig Full application stack

  23. Flaky tests - Fighting nightmares @leichteckig Product-sided causes Product-sided causes

  24. Flaky tests - Fighting nightmares @leichteckig In some cases when

    fixing a flaky test, the fix is in the app, not in the test. (Sam Sa ron, 2019)
  25. Flaky tests - Fighting nightmares @leichteckig Finding a way Finding

    a way
  26. Flaky tests - Fighting nightmares @leichteckig

  27. Flaky tests - Fighting nightmares @leichteckig Start an honest discussion

    with your team
  28. Flaky tests - Fighting nightmares @leichteckig Coping strategies Coping strategies

  29. Flaky tests - Fighting nightmares @leichteckig Coping strategies Coping strategies

  30. Flaky tests - Fighting nightmares @leichteckig Coping strategies Coping strategies

  31. Flaky tests - Fighting nightmares @leichteckig Coping strategies Coping strategies

  32. Flaky tests - Fighting nightmares @leichteckig Coping strategies Coping strategies

  33. Flaky tests - Fighting nightmares @leichteckig Start an honest discussion

    with your team
  34. Flaky tests - Fighting nightmares @leichteckig

  35. Flaky tests - Fighting nightmares @leichteckig If the build is

    not green nothing gets deployed
  36. Flaky tests - Fighting nightmares @leichteckig

  37. Flaky tests - Fighting nightmares @leichteckig Try running a flaky

    test in a loop
  38. Flaky tests - Fighting nightmares @leichteckig Run test in a

    loop Run test in a loop
  39. Flaky tests - Fighting nightmares @leichteckig Try running a flaky

    test in a loop
  40. Flaky tests - Fighting nightmares @leichteckig

  41. Flaky tests - Fighting nightmares @leichteckig Invest in a fast

    test suite with good resources
  42. Flaky tests - Fighting nightmares @leichteckig

  43. Flaky tests - Fighting nightmares @leichteckig Keep tests isolated

  44. Flaky tests - Fighting nightmares @leichteckig Test data management Test

    data management
  45. Flaky tests - Fighting nightmares @leichteckig Keep tests isolated

  46. Flaky tests - Fighting nightmares @leichteckig

  47. Flaky tests - Fighting nightmares @leichteckig Use smaller tests

  48. Flaky tests - Fighting nightmares @leichteckig

  49. Flaky tests - Fighting nightmares @leichteckig Automatic retries

  50. Flaky tests - Fighting nightmares @leichteckig Retries in Gitlab Retries

    in Gitlab
  51. Flaky tests - Fighting nightmares @leichteckig Retries in Cypress Retries

    in Cypress
  52. Flaky tests - Fighting nightmares @leichteckig Automatic retries

  53. Flaky tests - Fighting nightmares @leichteckig

  54. Flaky tests - Fighting nightmares @leichteckig Add purpose built diagnostic

    code
  55. Flaky tests - Fighting nightmares @leichteckig Add purpose built diagnostic

    code Add purpose built diagnostic code
  56. Flaky tests - Fighting nightmares @leichteckig Add purpose built diagnostic

    code
  57. Flaky tests - Fighting nightmares @leichteckig

  58. Flaky tests - Fighting nightmares @leichteckig Matching waiting strategy

  59. Flaky tests - Fighting nightmares @leichteckig Exit! Exit!

  60. Flaky tests - Fighting nightmares @leichteckig Thank you! <3 Thank

    you! <3
  61. Flaky tests - Fighting nightmares @leichteckig Concerning image copyright Concerning

    image copyright 1: Photo by hp koch on Unsplash 2: Photo by Pelly Benassi on Unsplash on Unsplash 3: Photo by Paul Green on Unsplash 4: Photo by Franck V. on Unsplash 5: Photo by Michael Jasmund on Unsplash 6: Photo by Wilhelm Gunktel on Unsplash