Merpay Tech Fest 2021 / Frontend Testing_Cypress as a Testing Platform

Merpay Tech Fest 2021は5日間のオンライン技術カンファレンスです。

IT企業で働くソフトウェアエンジニアおよびメルペイの技術スタックに興味がある方々を対象に2021年7月26日（月）から7月30日（金）までの5日間、開催します。 Merpay Tech Festは事業との関わりから技術への興味を深め、プロダクトやサービスを支えるエンジニアリングを知れるお祭りです。 セッションでは事業を支える組織・技術・課題などへの試行錯誤やアプローチを紹介予定です。お楽しみに！

■イベント関連情報
- 公式ウェブサイト：https://events.merpay.com/techfest-2021/
- 申し込みページ：https://mercari.connpass.com/event/215035/
- Twitterハッシュタグ： #MerpayTechFest

■リンク集
- メルカリ・メルペイイベント一覧：https://mercari.connpass.com/
- メルカリキャリアサイト：https://careers.mercari.com/
- メルカリエンジニアリングブログ：https://engineering.mercari.com/blog/
- メルカリエンジニア向けTwitterアカウント：https://twitter.com/mercaridevjp
- 株式会社メルペイ：https://jp.merpay.com/

Transcript

1. #MerpayTechFest Session Title Frontend Testing: Cypress as a Testing Platform Wilson Lau Frontend Engineer, Merpay
2. #MerpayTechFest Frontend Engineer, Merpay Wilson Lau Originally from Canada, I joined Merpay in the beginning of 2020 as a frontend engineer. Since then, I've worked across a number of Merpay's web products - including our internal merchant and customer Support tools, our Merpay partner site, and new merchant application process. Currently, I'm leading the push for better frontend testing within Merpay, and hopefully beyond!
3. #MerpayTechFest Background: Merpay 2019.2–3 iD Payments/ QR Code Payments Available at 2.06 million iD and Merpay merchants nationwide, including convenience stores, restaurants, drug stores, and fast food *As of March 31, 2021 (merchants with both iD/QR counted once only) 2020.7 Merpay Smart Payments Allows users to defer payment for items based on past usage of Mercari and Merpay Online Payments Enables use of Merpay for not only offline purchases, but online as well 2020.9 Send/receive Enables users to send their Merpay balance/points to family and friends No-contact service 2019.4–5 Growing Wallet Enables users to utilize their Merpay balance on the peer-to-peer investment service Funds *Second investment fund went live in January 2021 2020.11 2021.3 Shared d Payment/ Merpay QR code Enables use of single QR code for both d Payment and Merpay Virtual card Enables users to easily issue a card number which can be used to pay using Merpay balance at many online stores Identity verification using Japan’s Individual Number Card (JPKI) Launched support for identity verification using Japan’s public Individual Number Card identification service (JPKI) Merpay Smart Payments (fixed-amount) Allows for “fixed-amount payments,” whereby users pay in installments from the following month onwards Mercari Donation Enables donation of Merpay balance, charged from Mercari sales balance or bank account, etc., to local governments and other organizations of the user’s choice
4. #MerpayTechFest Background • Reduce manual QA time requirements • Increase confidence when doing maintenance tasks and refactoring • Onboarding new team members / learning new products ◦ Using tests as a reference, new members can quickly understand specs • Increase velocity in new feature development ◦ Fully mocked backend APIs lets us develop at the same time as backend ◦ Implementation-agnostic tests allows us to write tests and product code simultaneously • Reduce bugs and issues in general How do we maintain our development velocity?
5. #MerpayTechFest Cypress Testing Stack Cypress API Mocks Integration Testing E2E Regression Testing Accessibility Testing Shared Utilities Performance Testing Component Testing Visual Regression Testing Reporting / Dashboards FUTURE
6. #MerpayTechFest Cypress Testing Stack Cypress
7. #MerpayTechFest Cypress What is Cypress?
8. #MerpayTechFest Cypress Why Cypress instead of other testing libraries? • Less mocking; real browser environment ◦ Using jest-based testing libraries that rely on jsdom requires a lot of mocking out of key components of our application ◦ Not true integration tests with so many components mocked out ◦ Significant issues with asynchronous behaviour • Immediate visual feedback ◦ Easier to debug tests when there is visual feedback available ◦ Can develop in a TDD-like way and create different scenarios • Enforces testing of behaviour instead of implementation
9. #MerpayTechFest Cypress Testing Stack Cypress API Mocks
10. #MerpayTechFest export default function handler( req: GetUserRequest | null = null, modification: Partial<GetUserRequest> ): GetUserResponse { return { id: req.user.id || “default-id”, name: “Test User", email: “test@email.com”, ...modification } } API Mocks Full set of type-safe mocks lets us setup any scenario possible
11. #MerpayTechFest API Mocks • Our mocks are response generator / request handler functions written in TS ◦ Every mock returns a default response, and accept a second parameter that allows for modifications of the response ◦ Optionally, it’s possible to write custom handlers that respond different based on request body / query parameters, which Cypress makes available • Combinations of these mocks are used as the ‘setup’ before loading a page ◦ cy.intercept also allows us to change the responses throughout a test • These mocks can also be used on local development – we have implementations using mock service workers / custom node.js server for mocking Full set of type-safe mocks lets us setup any scenario possible
12. #MerpayTechFest Cypress Testing Stack Cypress API Mocks Shared Utilities
13. #MerpayTechFest Shared Utilities describe(“Form”, () => { beforeEach(() => { cy.mock(UserAPI.getUser, { email: ‘test@email.com’ }); }); it(“should render the form properly”, () => { cy.getByTestId(“testForm”) .assertTextInput(“Email”, “default@email.com”, { placeholder: “Email” }) .assertSelect(“True/False”, null, { values: [“True”, “False”] }) .updateTextInput(“Email”, “newEmail@email.com”); }); }); We write custom utilities to make tests easier to read and write
14. #MerpayTechFest Shared Utilities • Make tests easy to read ◦ Tests are very readable and can act as a specification for the product ◦ Liberal use of Cypress chaining pattern to group sections together ◦ Use of file structure, sections and test descriptions for organization • Make tests easy to write ◦ Abstractions enforce patterns in both the test code and product code ◦ Fairly strict test structure helps us make sure that we’re testing ◦ Tests are based on visible DOM elements as much as possible We write custom utilities to make tests easier to read and write
15. #MerpayTechFest Cypress Testing Stack Cypress API Mocks Shared Utilities Reporting / Dashboards
16. #MerpayTechFest Reporting & Dashboards What gets measured get managed • All of our tests generate their own reports ◦ @cypress/code-coverage (nyc reporters) ◦ Custom cy.task / cy.log reports for a11y • Tests are run by cron jobs either once a day / on every commit in CircleCI • All test runs done in CircleCI have a job that uploads these reports ◦ Upload all test reports via GCP Cloud Logging / BigQuery ◦ Generate dashboards using Looker ▪ Individual project dashboards ▪ Overview dashboards for type of test
17. #MerpayTechFest Reporting & Dashboards
18. #MerpayTechFest Reporting & Dashboards
19. #MerpayTechFest Cypress Testing Stack Cypress API Mocks Shared Utilities Reporting / Dashboards E2E Regression Testing
20. #MerpayTechFest 1. E2E Regression Testing E2E Testing in live test environment • We build and maintain E2E regression tests in collaboration with our QA team ◦ These tests do not mock anything at all and run in our test environment and interface with backend microservices deployed in test. ◦ Tests are run during QA period, before every release • In order to create test data: ◦ QA team maintains test requests as Postman JSON – we built a helper to use Postman JSON to create requests in Cypress. ◦ There is also some existing test data that we use if we do not feel like we need to create new data for the test.
21. #MerpayTechFest Cypress Testing Stack Cypress API Mocks Shared Utilities Reporting / Dashboards E2E Regression Testing Integration Testing
22. #MerpayTechFest 2. Integration Testing What is frontend integration testing? • Integration of all frontend code (logic, store, components) • API Responses are fully mocked, but nothing else. ◦ Input (Mocks): API Responses + user actions in the DOM ◦ Output (Assertions): Rendered DOM & API Requests • We test behaviour, not implementation: ◦ Rendering: Given a response, do we render the correct elements? ◦ Permissions: Given a response + auth, is what is displayed / allowed correct? ◦ Validation: Do we correctly validate users’ inputs and actions? ◦ Actions: Does a given user flow generate the correct request?
23. #MerpayTechFest 2. Integration Testing Example Test Structure describe(“/dashboard”, () => { describe(“rendering:”, () => { it(“should render the form correctly”, () => { ... }); }); describe(“validation:”, () => { it(“should validate the name input correctly”, () => { ... }); it(“should not be able submit the form when errors exist”, () => { ... }); }); describe(“actions:”, () = { it(“should be able to submit the form successfully”, () => { ... }); it(“should be able to navigate to the next page correctly”, () => { ... }); }); });
24. #MerpayTechFest Cypress Testing Stack Cypress API Mocks Shared Utilities Reporting / Dashboards E2E Regression Testing Integration Testing Accessibility Testing
25. #MerpayTechFest 3. Accessibility Testing • axe is one of the standard accessibility tests that we’ve chosen to adopt • cypress-axe is convenient integration with some custom code ◦ Render a full page using standard mocked response ◦ Run axe tests against the rendered page ◦ Use cy.task and cy.log to generate our own custom reports • Measurements ◦ Number of accessibility violations per page ◦ % of pages passing threshold number of accessibility violations • Custom cypress commands / config can be shared between all projects cypress-axe as key integration
26. #MerpayTechFest Cypress Testing Stack Cypress API Mocks Shared Utilities Reporting / Dashboards E2E Regression Testing Integration Testing Accessibility Testing Performance Testing
27. #MerpayTechFest 4. Performance Testing • We’ve chosen to use Lighthouse as our performance testing tool • Cypress allows us to run these tests in multiple environments: ◦ Build app as SSR vs SPA ◦ Use mocked APIs vs using APIs in test environment ◦ Allows us to diagnose performance issues in more detail • Measurements ◦ Typical performance metrics: LCP, TBT, CLS, TTI ◦ Metrics measured on a per-page level cypress-audit & cypress-lighthouse
28. #MerpayTechFest Cypress Testing Stack Cypress API Mocks Integration Testing E2E Regression Testing Accessibility Testing Shared Utilities Performance Testing Visual Regression Testing Reporting / Dashboards FUTURE
29. #MerpayTechFest 5. Visual Regression Testing (Future) • Current tests do not give us any feedback about any unwanted visual changes • QA visual inspections are still required • cy.screenshot can be used to produce screenshots of each page ◦ Render a full page using standard mocked response ◦ Take screenshots of the page ◦ Use library to diff the screenshots • Tried using reg-cli for comparisons, but still have not standardized approach • Some other options to consider: applitools, cypress-plugin-snapshots, cypress-visual-regression, cypress-image-diff Currently investigating various options
30. #MerpayTechFest Cypress Testing Stack Cypress API Mocks Integration Testing E2E Regression Testing Accessibility Testing Shared Utilities Performance Testing Component Testing Visual Regression Testing Reporting / Dashboards FUTURE
31. #MerpayTechFest 6. Component Testing (Future?) Introduced in v.7.0.0 in April 2021 (in alpha) • Potential alternative to Storybook + Jest • Mount components instead of rendering an entire page ◦ Visually inspect component in a browser environment ◦ Use Cypress query / action APIs to manipulate and test various actions ◦ Use Cypress assertions to write tests • Cypress is building out support for frameworks to mock global variables ◦ Ex. $store, $i18n, etc. • Still early and missing a lot of features available in Storybook
32. #MerpayTechFest Cypress Testing Stack Cypress API Mocks Integration Testing E2E Regression Testing Accessibility Testing Shared Utilities Performance Testing Component Testing Visual Regression Testing Reporting / Dashboards FUTURE
33. #MerpayTechFest Thank you! Any questions? @wilsonplau @wilsonplau