Visual Regression Testing with PhantomCSS

Website are visually more complex than ever, automated visual regression testing is quickly becoming as important as functional end-to-end and unit tests. PhantomCSS is one of many new tools support this paradigm shift in UI testing.

(Original presentation included video examples)

Presented at London JS meetup 19th Aug 2013.

Transcript

1. PhantomCSS Visual Regression Testing @jamescryer UI developer at Huddle Wednesday, 21 August 13
2. Visual regression testing? Testing visual changes Wednesday, 21 August 13
3. Visual regression testing? • Implicit test coverage of CSS and HTML • Not coupled to the implementation • Verifiable contract of design Wednesday, 21 August 13
4. Why is it important? • We are already testing visuals, manual testing is a time consuming process • The risk of regression and the cost of manual testing is higher with responsive layouts and complex applications • CSS refactoring is necessary for long life projects, we need automated support • Visual regression can often be difficult to identify Wednesday, 21 August 13
5. Wednesday, 21 August 13
6. Existing web tests? • Functional testing still has a place in our workflow • But test assertions with XPath and CSS selectors aren’t enough • Testing that “.red-button” exists doesn’t mean that the button is red. • There’s more than one way to skin a cat website. Tests shouldn’t care about the implementation Wednesday, 21 August 13
7. Available tools • Huxley (Selenium, Python) • Wraith (PhantomJS or SlimerJS, Ruby) • CSSCritic (PhantomJS & In browser) • Needle (Selenium & Nose/Python) • Sikuli Script Wednesday, 21 August 13
8. PhantomJS Scriptable headless Webkit browser CasperJS Navigation scripting and testing framework PhantomCSS Screenshot comparison library for CasperJS PhantomCSS? Wednesday, 21 August 13
9. PhantomCSS? • Takes screenshots, compares them to existing images and reports if they’re different • PhantomJS is the only binary dependency • Agnostic to build stack Wednesday, 21 August 13
10. Example Wednesday, 21 August 13
11. Wednesday, 21 August 13
12. But there is no pixel perfect!? • Antialiasing • OS and graphics card dependent • PhantomCSS needs clever analysis with Resemble.js • Animations • Optionally removes jQuery animations, CSS3 keyframes & transitions • CSS3 keyframes & transitions Wednesday, 21 August 13
13. Mutable content • It’s not worth visually testing pages that have regular content changes .. • Unless you can fake the content for your tests?! • Live style guides (with static content) are perfect for visual testing Wednesday, 21 August 13
14. Full page screenshots • It’s better to test UI components, rather than pages • As with unit testing, too many asserts in one test result in poor feedback • Larger screenshot tests will fail more often Wednesday, 21 August 13
15. The future of UI testing? • Visual regression as a first-class citizen in UI testing • Faster test feedback • Better reporting for devs • Better transparency for stakeholders Wednesday, 21 August 13
16. Wednesday, 21 August 13
17. More info • https://github.com/Huddle/PhantomCSS • https://github.com/Huddle/Resemble.js • https://github.com/Huddle/PhantomFlow Thanks. Wednesday, 21 August 13