No more flaky tests!

Slides of my talk at TestJS Summit 2023 in Berlin.

Transcript

1. NO MORE FLAKY TESTS! TALKING ABOUT TESTING WALMYR LIMA E SILVA FILHO
3. But what about that test that passed locally on your computer, and when it started running in the continuous integration pipeline, it started failing (sometimes?)
4. A FLAKY TEST IS LIKA A GRANADE... Waiting to explode.
5. MY DEFINITION OF A DETERMINISTIC TEST Given the same initial system conditions, when a test is run with the same inputs, it will ALWAYS return the same result.
6. MY DEFINITION OF A NON-DETERMINISTIC TEST Given the “same” initial system conditions, when a test is run with the same inputs, it returns DIFFERENT RESULTS.
7. “Flaky tests are more harmful than no tests at all”
8. SIDE EFFECTS OF A FLAKY TEST SUITE Decreases confidence in tests Reduces confidence in the team Increases the number of unfound bugs Increases debugging time Increases software engineering costs Increases the stress level of those involved Increases delivery time Decreases the perceived software quality
9. FOR WHAT REASONS CAN TESTS BE (OR BECOME) NON-DETERMINISTIC? Shared environment Manual and automated tests running against the same environment. Rendering order cy.get().first() vs. cy.get().should(‘have.length, n).first() Waiting times (network) cy.wait(10000) vs. cy.wait(‘@alias’) Local vs. CI Different computing resources Component state Example: Disabled button Not visible field Animations/transitions The Cypress way: cy.get(‘button’).should(‘be.e nabled’).click() Lack of tool knowledge cy.visit() cy.intercept().as(‘alias’) cy.wait(‘@alias’) vs. cy.intercept().as(‘alias’) cy.visit() cy.wait(‘@alias’) Dependency between tests describe(‘Test suite’, () => { it(‘visits the page’, () => { cy.visit(‘/’) }) it(‘checks the title’, () => { cy.get(‘h1’) .should(‘be.visible’) }) }) Parallelism Data creation that cannot be repeated Tests using shared data API limitations
10. TEST RETRIES Use as a way to identify non-deterministic tests. Not as a way to mask them.
11. DIFFERENT WAYS to “burn” tests before making them part of the “official” test suite
12. Cypress & Lodash context('Cypress._', () => { Cypress._.times(3, () => { it('runs', () => {}) }) })
13. @cypress/grep // newTest.cy.js it('new test case', { tags: '@burn' }, () => { ... }) // And in the command line terminal, run: cypress run --env grepTags=@burn,burn=10
14. Coming soon to the Cypress Cloud CYPRESS BURN-IN
15. WHAT TO DO WHEN ENCOUNTERING A FLAKY TEST? Here are some suggestions. 01. Identify the root cause and fix it immediately 02. Quarantine 03. Rewrite or move the test to another level (e.g. component instead of e2e)
16. WHAT NOT TO DO WHEN ENCOUNTERING A FLAKY TEST? Here are some tips on what not to do. 01. Don't ignore flaky testing 02. Don't use something like cy.wait(Number) 03. Don’t just rely on retries (you’ll just be “sweeping the dust under the rug”)
17. WE DO NOT ACCEPT FLAKY TESTS! Trust is everything, and just as you don't like the phrase “it works on my machine,” understand that there is no point in passing the test just on your machine. Automated tests must be deterministic and also pass in the continuous integration environment.
18. CONTENT SUGGESTIONS https://medium.com/the-making-of-whereby/the- importance-of-dealing-with-flaky-tests- 1b69145dee0f#.g2p671vvx https://dev.to/walmyrlimaesilv/how-to-run-a-test- multiple-times-with-cypress-to-prove-it-is-stable-3c0k https://youtu.be/SK2D18m52Cs?si=dJkeMeYTi8R76325 https://cypress.io/blog/tag/flake https://docs.cypress.io/guides/cloud/flaky-test- management
19. CYPRESS, FROM ZERO TO THE CLOUD
20. QUESTIONS?
21. THANKS! TALKING ABOUT TESTING HTTPS://WALMYR.DEV