Cross-Functional Strategy for Visual Regression Testing, JSConf JP 2025

These slides are for 10 min. LT at JSConf JP 2025 on November 16. (The actual presentation was made in Japanese. These are for non-Japanese speakers.)

In the B2B area at Money Forward, more than 20 products use Chromatic for Visual Regression Testing (VRT). As usage grew, the cost increased much faster than expected, and we began to face issues with governance and unclear guidelines.

In this talk, I share how our cross-functional team is defining the purpose and scope of VRT, clarifying responsibilities between product teams and the design system, exploring alternative testing methods, and optimizing Chromatic usage to build a sustainable VRT strategy across the organization.

https://jsconf.jp/2025/en/talks/visual-regression-testing-chromatic

Transcript

1. Cross-Functional Strategy for Visual Regression Testing Taiga Kiyokawa 2025.11.16 JSConf JP 2025 Track C LT ̶ Should we keep using Chromatic? ̶
2. Cross-functional Strategy for VRT Clarify the “Strategy” before relaying on VRT! Use Chromatic respectively based on “usage scale”! Operate VRT with a focus on “what & when to do”! Consider alternative testing methods! Cover pattern-level UI mainly. Leave component-level UI to design system! Small usage: Boost VRT with easy use Chromatic! Large usage: Consider open source alternatives! Be prepared for DIY ops! In Chromatic, adopt best practices to optimize the cost: TurboSnap, disable by default, consolidate stories, and control the workflow! #jsconfjp
3. Taiga @taigakiyokawa Web frontend engineer at Money Forward, Inc. Design System, Accessibility, Technical Standardization Props in my room: DeLorean, Master Sword, Kanedaʼs Motorcycle
4. Background Knowledge
5. What is Visual Regression Testing (VRT)?
6. Visual Regression Testing (VRT) VRT is a test compares snapshot diffs for UI renderings. It benefits to check visual changes not covering API tests.
7. What is Chromatic?
8. Chromatic is a platform for VRT, UI review, etc. Itʼs in harmony with Storybook and easy to integrate. Chromatic
9. Pricing for Chromatic You can take a fixed number of snapshots for each plan. Excess snapshots are charged based on usage.
10. Case Study
11. Whatʼ happening in Money Forward?
12. Dev Situation of Money Forward Cloud Over 40 products and their teams are developing. There's not much connection or control across teams.
13. 4 years ago...
14. Start using Chromatic A product team tried Chromatic. It grows VRT momentum and is gradually spreading. … … … … …
15. A few years later...
16. Costs skyrocketed in no time!
17. 20+ products are using Chromatic It's so easy to use, has gotten popular, and costs shot up. Thereʼs no governance and policies across teams.
18. Upgraded Starter → Pro → Enterprise just in one year...
19. “Why does it cost so much?”
20. The committee has been established.
21. Develop a VRT strategy We align perspectives and standardize a policy. Weʼre considering how we should cost for VRT.
22. “Considering How Should We Cost for VRT”
23. Thatʼs tough.
24. VRT brings great peace of mind UIʼs direct inspection feels like a “last line of defence.” The more coverage, the better (it seems).
25. How much attention should be given to which tests?
26. The Testing Trophy and Testing Classifications
27. “Testing is all about ROI”
28. VRT is about perspective VRT can be considered either integration/unit testing. Assertion method matters, not classifications. Component-level UI Pattern-level UI Page-level UI VRT
29. Current Strategy
30. First, consider other methods than VRT.
31. Shift-left to other testing VRT is like a “GK (Goalkeeper).” Position other testing methods as “DF (Defenders).” VRT Static (ESLint) Accessibility (axe-core) Interaction (testing-library) Integration (@storybook/test-runner)
32. e.g. @storybook/test-runner Tests for interactive UI are not suitable for VRT. A product team has cut about 60% of VRT cost! Chromatic 100% Chromatic 40% @storybook/test-runner 60%
33. Distinguish responsibilities for VRT between products and the design system.
34. VRT for products: Pattern-level UI Ensure combinations of UI components (patterns) mainly. Consider E2E to ensure page-level UI if you really want.
35. VRT for design system: component-level UI Ensure general UI components on the design system. Consolidate VRT costs from each product.
36. Q. Should we keep using Chromatic?
37. A. It should be decided by “usage scale of each product.”
38. Small usage: Boost with Chromatic Itʼs more cost-effective than DIY ops with open source. Reducing the barrier of VRT makes it ideal for beginners.
39. Large usage: Consider open source Be prepared DIY ops because itʼs getting “easy < money.” It might be around 10,000 snapshots/month.
40. VRT with open sources It needs to maintain infrastructure or can burden a repo. Maybe itʼs not that hard...? (Please let me know.)
41. 4 Tactics for Saving Chromatic Cost
42. Enable TurboSnap Itʼs an option to reduce redundant snapshots. It can cut about 30% costs as experienced. steps: # ... - name: Run Chromatic uses: chromaui/action@v1 with: projectToken: ${{ secrets.CHROMATIC_PROJECT_TOKEN }} onlyChanged: true
43. Disable snapshot by default Configure disableSnapshot in preview.ts. Make “what stories should be taken snapshots” explicit. const preview: Preview = { parameters: { chromatic: { disableSnapshot: true }, }, }; export const Foo: Story = { parameters: { chromatic: { disableSnapshot: false }, }, }; .storybook/preview.ts Foo.stories.tsx
44. Consolidate variants into 1 story Reduce target stories by consolidating possible patterns. Itʼs for small components. 9 stories 1 story
45. Control the workflow execution pushする度に実⾏していると不要なスナップショットが増えがち ラベルなどを使い、適切なタイミングでトリガーする chromatic-run push push push
46. Results by tactics A product team reduced 70,000ss/month → 3,000ss/month! In the design system, the number of target stories reduced to 1/3! 3,000 snapshots/month 70,000 snapshots/month
47. Optimize operation costs by reviewing “target stories” and “frequency.” What When
48. Summary
49. Cross-functional Strategy for VRT Clarify the “Strategy” before relaying on VRT! Use Chromatic respectively based on “usage scale”! Operate VRT with a focus on “what & when to do”! Consider alternative testing methods! Cover pattern-level UI mainly. Leave component-level UI to design system! Small usage: Boost VRT with easy use Chromatic! Large usage: Consider open source alternatives! Be prepared for DIY ops! In Chromatic, adopt best practices to optimize the cost: TurboSnap, disable by default, consolidate stories, and control the workflow! #jsconfjp