[mercari GEARS 2025] Running 1000 End-To-End Web Tests Daily

Transcript

1. Running 1000 End-To-End Web Tests Daily
 Gleb Bahmutov
 Mercari US / Sr Director of Engineering

2. Talk Contents
 • End-to-End tests • Test speed • Test tags • Picking tests to run • AI for end-to-end testing
3. Gleb Bahmutov
 
 
 Gleb Bahmutov is a JavaScript ninja, image processing expert, and software quality fanatic.
 • https://gleb.dev
 • https://github.com/bahmutov
 • https://slides.com/bahmutov
 • https://youtube.com/@gleb
 Sr Director of Engineering

4. Mercari.com

5. Example Mercari web E2E test 📺
6. Testing
 Unit
 Focus on the code
 Fast
 API
 Focus on computer interfaces
 E2E
 Focus on the human user
 Can be fast

7. E2E Test Size At Mercari.com

8. Writing E2E Cypress Tests With Copilot’s Help
 GitHub Copilot inline code suggestion 👍👍👍
9. Example Cypress E2E Test
 test tags page objects accessing page elements
10. • Use API to set the state (create seller, item, etc)
 • Cache data
 • Intelligent waiting (few hardcoded waits)
 Individual Test Speed 🐢 🏎
 Making each test faster
11. Test Speed 🐢 🏎
 • 1 E2E Cypress test = 1 second to 4 minutes • 1000 E2E Cypress tests = TOO SLOW TO RUN ON EACH COMMIT • Run all tests every 8 hours
12. Test Speed 🐢 🏎
 39 min across 
 20 machines using Cypress Cloud

13. Test Speed 🐢 🏎
 Vary the number of machines

14. Do We Need To Run All The Tests All The Time?
15. Feature-specific testing
 • “I am working on items search, what tests do we have?”
 • “We received a bug report, is the search service working correctly?”
 • “I have opened a PR for the search service, I need to test it”

16. Test Tags

17. Test Tags
 Feature Tags
 • @search
 • @sell
 • @offer
 • …
 Group tags
 • @sanity • @regression • @mobile • …
18. Test Tags

19. Pick Tests To Run on Pull Request (Testing Repo)
 Runs a few tests across all features
 And all tests for the given tag(s)

20. Pick Tests To Run on Pull Request (Web Repo)
 Dev opens a web pull request

21. Pick Tests To Run on Pull Request (Web Repo)
 Preview environment is deployed

22. Pick Tests To Run on Pull Request (Web Repo)
 Dev can run E2E tests using tags
 115 specs have tests tagged @sanity or @profile

23. Main API and service repos can test PRs using the “/cypress” comment
 Pick Tests To Run on Pull Request (Any Repo)

24. Flexibility And Power • Pick tests using test tags: /cypress tags=@sanity
 • Pick tests visiting specific page: /cypress url=/mypage/purchase/in_progress/
 • Pick tests calling specific API: /cypress graphql=newLister
 

25. Even Better: Target The Web Tests To Changes 🎯

26. <div data-testid=”Greeting”>
 … // code changes
 </div>
 Web Pull Request
 the changed frontend source file

27. E2E specs that use “Greeting” test id
 cy.byTestId(‘Greeting’)
 .should(‘be.visible’)
 …
 new-user.cy.js
 cy.byTestId(‘Greeting’)
 …
 home.cy.js

28. Pick Tests Using Changed Source Code (Web Repo)
 1 spec that exercises the elements with test ids “MenuItems” and “MenuItemsComp”
 test ids

29. Which Test Tag(s)?

30. Picking Test Tags Using AI
 Which E2E tests should we run for this service PR?

31. AI suggests test tags based on the PR title and body text vs test tag descriptions
 Picking Test Tags Using AI

32. Picking Test Tags Using AI
 Confidence with each picked test tag

33. Bonus: Automated Testing
 for GitHub Bug Reports

34. When Someone Opens An Issue Tagged “bug” 🐞
 Picked tests
 to run based
 on the bug title and description

35. When Someone Opens An Issue Tagged “bug” 🐞
 Completed test run status: all current tests ✅

36. GitHub Copilot PR Reviews 👀

37. copilot-instructions.md file
 When performing a code review:
 
 - confirm that there are no hard-coded magic numbers.
 Prefer using named constants.
 - do not allow unreachable code
 - check each HTML element that shows any unique application data,
 like prices, values, names, address, etc to have a `data-testid`
 attribute to be used in end-to-end tests. If the attribute is missing,
 add a `data-testid` attribute with a meaningful value.
 Also add `data-testid` attributes to the top level forms, pages,
 large components.

38. Copilot review can detect page elements without “data-testid” attributes and even suggest good attribute names

39. Using AI for QA: What we learned so far
 Generation
 • Sometimes works for smaller inline coding
 • Hard to express all application knowledge as context for each prompt
 • Long waiting loops
 • Human review is hard

40. Using AI for QA: What we learned so far
 Generation
 • Sometimes works for smaller inline coding
 • Hard to express all application knowledge as context for each prompt
 • Long waiting loops
 • Human review is hard
 Review
 • AI can help debug simple problems
 • Works well explaining and fixing edge cases with specific tools and languages
 • Copilot reviews work well

41. Using AI for QA: What we learned so far
 Generation
 • Sometimes works for smaller inline coding
 • Hard to express all application knowledge as context for each prompt
 • Long waiting loops
 • Human review is hard
 Review
 • AI can help debug simple problems
 • Works well explaining and fixing edge cases with specific tools and languages
 • Copilot reviews work well
 “Picking”
 • Works really well for “pick the text most similar to X”, example is picking the appropriate test tags

42. Conclusions
 
 • End-to-end testing is extremely important
 • Do not run ALL the tests ALL the time
 • Let AI help

43. Thank You!
 Gleb Bahmutov
 Mercari US / Sr Director of Engineering
 gleb.dev