Flaky Tests - Fighting Nightmares

"A liar will not be believed even if he tells the truth" - Aesop

A test can also be such a liar: Unreliable tests are living nightmares for anyone who writes automated tests or even pays attention to their results. Sometimes they pass, sometimes they don't, and therefore do not give a valid statement about the state of the software. In the worst case, they can impair the credibility of the entire test suite. It doesn't matter whether you write unit, integration or end-to-end tests, you can counter them in any way you like.

I've spent hours and hours on such tests, and I want to share my path through them and the lessons learned from them: So that together we can avoid or even get rid of these nightmares.

Transcript

1. View Slide

2. View Slide

3. View Slide

4. View Slide

5. View Slide

6. View Slide

7. View Slide

8. View Slide

9. View Slide

10. View Slide

11. View Slide

12. View Slide

13. View Slide

14. View Slide

15. View Slide

16. View Slide

17. View Slide

18. View Slide

19. View Slide

20. View Slide

21. View Slide

22. View Slide

23. View Slide

24. View Slide

25. View Slide

26. View Slide

27. View Slide

28. View Slide

29. View Slide

30. View Slide

31. View Slide

32. View Slide

33. View Slide

34. View Slide

35. View Slide

36. View Slide

37. View Slide

38. Cypress._.times(10, () => {
    describe('Description', () => {
    it('runs 10 times', () => {
    //...
    });
    });
    });
    

    View Slide

39. View Slide

40. describe('Customer login', () => {
    let customer = {};
    beforeEach(() => {
    // Set application to clean state
    cy.setInitialState()
    .then(() => {
    // Create test data
    return cy.setFixture('customer');
    })
    }):
    // … tests will start below
    })
    

    View Slide

41. View Slide

42. View Slide

43. {
    "retries": {
    // Configure retry attempts for `cypress run`
    // Default is 0
    "runMode": 2,
    // Configure retry attempts for `cypress open`
    // Default is 0
    "openMode": 0
    }
    }
    

    View Slide

44. View Slide

45. View Slide

46. View Slide

47. View Slide

48. View Slide

49. View Slide

50. View Slide

51. View Slide