Visual Regression Testing with PhantomCSS

An introduction to visual regression testing as a way to automatically detect regressions in the appearance of your website.

Transcript

1. Visual Regression Testing with PhantomCSS Pavels Jelisejevs, C.T.Co
2. By "visual regression testing" (VRT) I mean: Tools that allow to automatically check if the UI is displayed correctly, including the dimensions of elements, colors, images and etc.
4. Why would we want to automate this? Because people are bad at detecting visual changes. This phenomenon is referred to as "change blindness".
13. Major use cases for VRT are: •Testing the appearance of UI elements on the page: headers, footers, button etc for visual defects •Testing responsive designs •Testing complex visualizations (think charts, diagrams, graphs) which cannot be easily covered with unit or functional tests
14. How can we test visual aspects? We can try to retrieve certain CSS properties from elements and compare them to the benchmark: assert(element.getCss("color") === "red") Hardly an option due to the amount of code need to be written and complex maintenance.
15. How can we test visualizations? <path fill="#7cb5ec" d="M 562 25.899999999999977 C 567.328 25.899999999999977 567.328 33.89999999999998 562 33.89999999999998 C 556.672 33.89999999999998 556.672 25.899999999999977 562 25.899999999999977 Z" stroke-width="1"></ path> Ain't no testing framework gonna help your here
16. Introducing PhantomCSS PhantomCSS is a tool that allows you to: 1. Open a page and take a screenshot of an element 2. Store the screenshot as a baseline 3. After a change is done, take another screenshot and compare it to the baseline
17. What's under the hood? PhantomCSS uses: •PhantomJS 2 or SlimerJS for browsing •CasperJS for interacting with the browser •Resemble.js for screenshot comparison •Python required for CasperJS
18. How does the test look like? casper.
 start( url )
 .then(function() {
 // Take a screenshot of the UI component
 phantomcss.screenshot('#my-elemenmt', 'a screenshot of my element');
 
 })
 .then(function() {
 // compare screenshots
 phantomcss.compareAll();
 })
 .run(function() {
 casper.test.done();
 });
19. Where can I try it out? You can try out a demo available at GitHub Run: npm install casperjs test demo/testsuite.js
20. Now what? New screenshot at C:/PhantomCSS/screenshots\coffee machine dialog_1.png
21. What if I change something? FAIL Visual change found for screenshot C:/PhantomCSS/screenshots\coffee machine dialog_1.png (21.81% mismatch) # type: fail # file: demo/testsuite.js # subject: false
22. How do I accept the change? casperjs test demo/testsuite.js --rebase
23. Additional configuration are supported You can set pass additional parameters to CasperJS PhantomCSS. Most popular ones are: - Resolution - Tolerance - Delays
24. A word of advice •PhantomCSS is not a replacement for functional tests •Take screenshots of certain elements rather than whole pages •Use your VCS system to store baseline screenshots
25. A word of advice •Set up a testing environment: •If you’re testing a whole website, mock all dynamic data •If you’re testing components - showcase them on a separate page (style guide)
26. Problems Running PhantomCSS on CI environment occasionally leads to unexpected errors.
27. Problems Changes in the dynamic content provided by the backend will cause the tests to break. How do we easily decouple them?
28. Popular questions Q: At what project stages should we use VRT? A: When you have something to take pictures of. Q: How to organise VRT in a team? A: In the same manner as all other tests.
29. Thank you! • PhantomCSS on GitHub - https://github.com/ Huddle/PhantomCSS • The phenomenon of Change Blindness - https:// en.wikipedia.org/wiki/Change_blindness
30. You’re welcome to join our team. Contact pavel.yeliseev@ctco.lv