Visual Regression Testing - from a tool to a process

We work with many JavaScript based tools in our day to day lives.

But how do we take those tools from being used by a single person on a machine and integrate it into a team's workflow?

How can we link those tool into chains and go from a tool to a process? :)

In this talk I will discuss how I converted PhantomCSS, a tool for visual regression testing, and integrated it into our workflow. I will also cover the challenges we faced which ranged from integrating the tests into our CI, test execution taking too long, ability to browse the test results and notifying developers about failed regressions.

Transcript

1. VISUAL REGRESSION TESTING from a tool to a process Nikhil Verma
2. Nikhil Verma Team Lead / JS Developer MOBILE WEB TEAM @NikhilVerma
3. badoo.com team.badoo.com techblog.badoo.com
4. VISUAL REGRESSION TESTING
5. VISUAL
6. It’s all about change which can be obvious or subtle
7. Spot the difference in 15 seconds
10. How can we easily detect change?
11. Let the computer do it.
13. REGRESSION TESTING
14. Why do we need visual regression testing?
15. Swiss cheese model https://wikipedia.org/wiki/Swiss_cheese_model
16. Unit testing Automation testing Visual Regression testing Human testing Change
17. The process today
18. https://github.com/Huddle/PhantomCSS CasperJS PhantomJS = + ResembleJS +
19. Local • Used by developers • Choose the tests to run • Manage your own “baseline” • Develop/Refactor/Debug
21. • Feature/Staging branches • Runs the entire test suite • Baseline from production • Catch regressions and bugs Automated
22. 1. Download baseline from server 2. Generate and compare screenshots 3. Upload results back to the server 4. Send notifications to developer/team How does it work?
25. Some issues caught by VRT
30. How it started
31. didoo.net / @areaweb Cristiano Rastelli Front-End / CSS Developer MOBILE WEB TEAM
32. Tests were slow Can’t run individual tests No integration with our process
33. Let’s start improving it! one problem at a time
34. Tests were slow • Don’t know when the page will be ready • Wait for images to download
35. Don’t know when the content will be ready
36. // On the client, when a page is rendered $page.attr('data-vrt-ready', 'true'); Client adds an attribute when ready
37. // In the VRT test waitForSelector('[data-vrt-ready="true"]') .then(() => { // take screenshot }); Wait for selector
38. Wait for images to download
39. Inline all images with base64 const PLACEHOLDER_90x90 = ‘data:image/ png;base64,iVBORw0KGgo…’; $('img').attr('src', PLACEHOLDER_90x90);
40. Before: 6.8 mins After: 1.2 mins 5x faster!
41. Tests were slow Can’t run individual tests No integration with our process
42. Can’t run individual tests
44. casperjs -- --config=config/phantomjs.json —concise target=https://m.badoo.com test instructions/mobileweb/tests/ login.js instructions/mobileweb/tests/chat.js instructions/ mobileweb/tests/gifts.js instructions/mobileweb/tests/liked- you.js --pre=instructions/mobileweb/pre.js -- post=instructions/mobileweb/post.js
45. https://www.npmjs.com/package/inquirer
46. Convert an array of questions module.exports = [{ type: 'list', name: 'suite', message: 'What do you want to test?', choices: […], default: 'styleguide' }, { type: 'list', name: 'environment', message: 'Which environment do you want to use?', choices: function (answers) {…} }];
47. To an interactive questionnaire
48. Tests were slow Can’t run individual tests No integration with our process
49. How to integrate it into the process?
50. Local Feature Staging Production
51. +
52. Hook • Git/Mercurial post-receive hooks • Deploy scripts • JIRA/Trello webhooks • No hook! wat?
53. Trigger • Run a bash script • Start a CI agent (TeamCity/Jenkins/CircleCI) • Boot a docker VM/Remote machine • Cron job (no hook needed!)
54. + !
55. Notification • Email • HipChat / Slack etc • Sound!
56. See the results • Email • Hosted website with results • Commit to repository • Hard Disk/Network Attached Storage
58. A hook in our deploy scripts which triggers VRT on a TeamCity agent which sends a notification via HipChat and lets us see the results on a website
59. Tests were slow Can’t run individual tests No integration with our process
60. Is everything perfect?
61. • Stability of tests • Test for different conditions • User agents • Resolutions • Expand test coverage
62. These are trivial problems and they only need to be solved once
63. Thank you @NikhilVerma joind.in/talk/bfb96