Visual Regression Testing with PhantomCSS

Slide 1

Slide 1 text

PhantomCSS Visual Regression Testing @jamescryer UI developer at Huddle Wednesday, 21 August 13

Slide 2

Slide 2 text

Visual regression testing? Testing visual changes Wednesday, 21 August 13

Slide 3

Slide 3 text

Visual regression testing? • Implicit test coverage of CSS and HTML • Not coupled to the implementation • Veriﬁable contract of design Wednesday, 21 August 13

Slide 4

Slide 4 text

Why is it important? • We are already testing visuals, manual testing is a time consuming process • The risk of regression and the cost of manual testing is higher with responsive layouts and complex applications • CSS refactoring is necessary for long life projects, we need automated support • Visual regression can often be difﬁcult to identify Wednesday, 21 August 13

Slide 5

Slide 5 text

Wednesday, 21 August 13

Slide 6

Slide 6 text

Existing web tests? • Functional testing still has a place in our workﬂow • But test assertions with XPath and CSS selectors aren’t enough • Testing that “.red-button” exists doesn’t mean that the button is red. • There’s more than one way to skin a cat website. Tests shouldn’t care about the implementation Wednesday, 21 August 13

Slide 7

Slide 7 text

Available tools • Huxley (Selenium, Python) • Wraith (PhantomJS or SlimerJS, Ruby) • CSSCritic (PhantomJS & In browser) • Needle (Selenium & Nose/Python) • Sikuli Script Wednesday, 21 August 13

Slide 8

Slide 8 text

PhantomJS Scriptable headless Webkit browser CasperJS Navigation scripting and testing framework PhantomCSS Screenshot comparison library for CasperJS PhantomCSS? Wednesday, 21 August 13

Slide 9

Slide 9 text

PhantomCSS? • Takes screenshots, compares them to existing images and reports if they’re different • PhantomJS is the only binary dependency • Agnostic to build stack Wednesday, 21 August 13

Slide 10

Slide 10 text

Example Wednesday, 21 August 13

Slide 11

Slide 11 text

Wednesday, 21 August 13

Slide 12

Slide 12 text

But there is no pixel perfect!? • Antialiasing • OS and graphics card dependent • PhantomCSS needs clever analysis with Resemble.js • Animations • Optionally removes jQuery animations, CSS3 keyframes & transitions • CSS3 keyframes & transitions Wednesday, 21 August 13

Slide 13

Slide 13 text

Mutable content • It’s not worth visually testing pages that have regular content changes .. • Unless you can fake the content for your tests?! • Live style guides (with static content) are perfect for visual testing Wednesday, 21 August 13

Slide 14

Slide 14 text

Full page screenshots • It’s better to test UI components, rather than pages • As with unit testing, too many asserts in one test result in poor feedback • Larger screenshot tests will fail more often Wednesday, 21 August 13

Slide 15

Slide 15 text

The future of UI testing? • Visual regression as a ﬁrst-class citizen in UI testing • Faster test feedback • Better reporting for devs • Better transparency for stakeholders Wednesday, 21 August 13

Slide 16

Slide 16 text

Wednesday, 21 August 13

Slide 17

Slide 17 text

More info • https://github.com/Huddle/PhantomCSS • https://github.com/Huddle/Resemble.js • https://github.com/Huddle/PhantomFlow Thanks. Wednesday, 21 August 13