Visual Regression Testing - from a tool to a process

Visual Regression Testing - from a tool to a process

We work with many JavaScript based tools in our day to day lives.

But how do we take those tools from being used by a single person on a machine and integrate it into a team's workflow?

How can we link those tool into chains and go from a tool to a process? :)

In this talk I will discuss how I converted PhantomCSS, a tool for visual regression testing, and integrated it into our workflow. I will also cover the challenges we faced which ranged from integrating the tests into our CI, test execution taking too long, ability to browse the test results and notifying developers about failed regressions.

620ecdb7215ec072617bfd7f75c98760?s=128

Nikhil Verma

May 10, 2017
Tweet

Transcript

  1. VISUAL REGRESSION TESTING from a tool to a process Nikhil

    Verma
  2. Nikhil Verma Team Lead / JS Developer MOBILE WEB TEAM

    @NikhilVerma
  3. badoo.com team.badoo.com techblog.badoo.com

  4. VISUAL REGRESSION TESTING

  5. VISUAL

  6. It’s all about change which can be obvious or subtle

  7. Spot the difference in 15 seconds

  8. None
  9. None
  10. How can we easily detect change?

  11. Let the computer do it.

  12. None
  13. REGRESSION TESTING

  14. Why do we need visual regression testing?

  15. Swiss cheese model https://wikipedia.org/wiki/Swiss_cheese_model

  16. Unit testing Automation testing Visual Regression testing Human testing Change

  17. The process today

  18. https://github.com/Huddle/PhantomCSS CasperJS PhantomJS = + ResembleJS +

  19. Local • Used by developers • Choose the tests to

    run • Manage your own “baseline” • Develop/Refactor/Debug
  20. None
  21. • Feature/Staging branches • Runs the entire test suite •

    Baseline from production • Catch regressions and bugs Automated
  22. 1. Download baseline from server 2. Generate and compare screenshots

    3. Upload results back to the server 4. Send notifications to developer/team How does it work?
  23. None
  24. None
  25. Some issues caught by VRT

  26. None
  27. None
  28. None
  29. None
  30. How it started

  31. didoo.net / @areaweb Cristiano Rastelli Front-End / CSS Developer MOBILE

    WEB TEAM
  32. Tests were slow Can’t run individual tests No integration with

    our process
  33. Let’s start improving it! one problem at a time

  34. Tests were slow • Don’t know when the page will

    be ready • Wait for images to download
  35. Don’t know when the content will be ready

  36. // On the client, when a page is rendered $page.attr('data-vrt-ready',

    'true'); Client adds an attribute when ready
  37. // In the VRT test waitForSelector('[data-vrt-ready="true"]') .then(() => { //

    take screenshot }); Wait for selector
  38. Wait for images to download

  39. Inline all images with base64 const PLACEHOLDER_90x90 = ‘data:image/ png;base64,iVBORw0KGgo…’;

    $('img').attr('src', PLACEHOLDER_90x90);
  40. Before: 6.8 mins After: 1.2 mins 5x faster!

  41. Tests were slow Can’t run individual tests No integration with

    our process
  42. Can’t run individual tests

  43. None
  44. casperjs -- --config=config/phantomjs.json —concise target=https://m.badoo.com test instructions/mobileweb/tests/ login.js instructions/mobileweb/tests/chat.js instructions/

    mobileweb/tests/gifts.js instructions/mobileweb/tests/liked- you.js --pre=instructions/mobileweb/pre.js -- post=instructions/mobileweb/post.js
  45. https://www.npmjs.com/package/inquirer

  46. Convert an array of questions module.exports = [{ type: 'list',

    name: 'suite', message: 'What do you want to test?', choices: […], default: 'styleguide' }, { type: 'list', name: 'environment', message: 'Which environment do you want to use?', choices: function (answers) {…} }];
  47. To an interactive questionnaire

  48. Tests were slow Can’t run individual tests No integration with

    our process
  49. How to integrate it into the process?

  50. Local Feature Staging Production

  51. +

  52. Hook • Git/Mercurial post-receive hooks • Deploy scripts • JIRA/Trello

    webhooks • No hook! wat?
  53. Trigger • Run a bash script • Start a CI

    agent (TeamCity/Jenkins/CircleCI) • Boot a docker VM/Remote machine • Cron job (no hook needed!)
  54. + !

  55. Notification • Email • HipChat / Slack etc • Sound!

  56. See the results • Email • Hosted website with results

    • Commit to repository • Hard Disk/Network Attached Storage
  57. None
  58. A hook in our deploy scripts which triggers VRT on

    a TeamCity agent which sends a notification via HipChat and lets us see the results on a website
  59. Tests were slow Can’t run individual tests No integration with

    our process
  60. Is everything perfect?

  61. • Stability of tests • Test for different conditions •

    User agents • Resolutions • Expand test coverage
  62. These are trivial problems and they only need to be

    solved once
  63. Thank you @NikhilVerma joind.in/talk/bfb96