Visual Regression Testing - from a tool to a process

We work with many JavaScript based tools in our day to day lives.

But how do we take those tools from being used by a single person on a machine and integrate it into a team's workflow?

How can we link those tool into chains and go from a tool to a process? :)

In this talk I will discuss how I converted PhantomCSS, a tool for visual regression testing, and integrated it into our workflow. I will also cover the challenges we faced which ranged from integrating the tests into our CI, test execution taking too long, ability to browse the test results and notifying developers about failed regressions.

Transcript

1. VISUAL
   REGRESSION
   TESTING
   from a tool to a process
   Nikhil Verma
   

   View Slide

2. Nikhil Verma
   Team Lead / JS Developer
   MOBILE WEB TEAM
   @NikhilVerma
   

   View Slide

3. badoo.com
   team.badoo.com
   techblog.badoo.com
   

   View Slide

4. VISUAL
   REGRESSION
   TESTING
   

   View Slide

5. VISUAL
   

   View Slide

6. It’s all about
   change
   which can be obvious or
   subtle
   

   View Slide

7. Spot the difference
   in 15 seconds
   

   View Slide

8. View Slide

9. View Slide

10. How can we easily
    detect change?
    

    View Slide

11. Let the computer
    do it.
    

    View Slide

12. View Slide

13. REGRESSION
    TESTING
    

    View Slide

14. Why do we need visual
    regression testing?
    

    View Slide

15. Swiss cheese model
    https://wikipedia.org/wiki/Swiss_cheese_model
    

    View Slide

16. Unit testing
    Automation testing
    Visual Regression testing
    Human testing
    Change
    

    View Slide

17. The process
    today
    

    View Slide

18. https://github.com/Huddle/PhantomCSS
    CasperJS PhantomJS
    = +
    ResembleJS
    +
    

    View Slide

19. Local
    • Used by developers
    • Choose the tests to run
    • Manage your own “baseline”
    • Develop/Refactor/Debug
    

    View Slide

20. View Slide

21. • Feature/Staging branches
    • Runs the entire test suite
    • Baseline from production
    • Catch regressions and bugs
    Automated
    

    View Slide

22. 1. Download baseline from server
    2. Generate and compare screenshots
    3. Upload results back to the server
    4. Send notifications to developer/team
    How does it work?
    

    View Slide

23. View Slide

24. View Slide

25. Some issues caught by
    VRT
    

    View Slide

26. View Slide

27. View Slide

28. View Slide

29. View Slide

30. How it started
    

    View Slide

31. didoo.net / @areaweb
    Cristiano Rastelli
    Front-End / CSS Developer
    MOBILE WEB TEAM
    

    View Slide

32. Tests were slow
    Can’t run individual tests
    No integration with our process
    

    View Slide

33. Let’s start
    improving it!
    one problem at a time
    

    View Slide

34. Tests were slow
    • Don’t know when the page will be ready
    • Wait for images to download
    

    View Slide

35. Don’t know when the content will be
    ready
    

    View Slide

36. // On the client, when a page is rendered
    $page.attr('data-vrt-ready', 'true');
    Client adds an attribute when ready
    

    View Slide

37. // In the VRT test
    waitForSelector('[data-vrt-ready="true"]')
    .then(() => {
    // take screenshot
    });
    Wait for selector
    

    View Slide

38. Wait for images to download
    

    View Slide

39. Inline all images with base64
    const PLACEHOLDER_90x90 = ‘data:image/
    png;base64,iVBORw0KGgo…’;
    $('img').attr('src', PLACEHOLDER_90x90);
    

    View Slide

40. Before: 6.8 mins
    After: 1.2 mins
    5x faster!
    

    View Slide

41. Tests were slow
    Can’t run individual tests
    No integration with our process
    

    View Slide

42. Can’t run individual tests
    

    View Slide

43. View Slide

44. casperjs -- --config=config/phantomjs.json —concise
    target=https://m.badoo.com test instructions/mobileweb/tests/
    login.js instructions/mobileweb/tests/chat.js instructions/
    mobileweb/tests/gifts.js instructions/mobileweb/tests/liked-
    you.js --pre=instructions/mobileweb/pre.js --
    post=instructions/mobileweb/post.js
    

    View Slide

45. https://www.npmjs.com/package/inquirer
    

    View Slide

46. Convert an array of questions
    module.exports = [{
    type: 'list',
    name: 'suite',
    message: 'What do you want to test?',
    choices: […],
    default: 'styleguide'
    }, {
    type: 'list',
    name: 'environment',
    message: 'Which environment do you want to use?',
    choices: function (answers) {…}
    }];
    

    View Slide

47. To an interactive questionnaire
    

    View Slide

48. Tests were slow
    Can’t run individual tests
    No integration with our process
    

    View Slide

49. How to integrate it into
    the process?
    

    View Slide

50. Local
    Feature
    Staging
    Production
    

    View Slide

51. +
    

    View Slide

52. Hook
    • Git/Mercurial post-receive hooks
    • Deploy scripts
    • JIRA/Trello webhooks
    • No hook! wat?
    

    View Slide

53. Trigger
    • Run a bash script
    • Start a CI agent (TeamCity/Jenkins/CircleCI)
    • Boot a docker VM/Remote machine
    • Cron job (no hook needed!)
    

    View Slide

54. +
    !
    

    View Slide

55. Notification
    • Email
    • HipChat / Slack etc
    • Sound!
    

    View Slide

56. See the results
    • Email
    • Hosted website with results
    • Commit to repository
    • Hard Disk/Network Attached Storage
    

    View Slide

57. View Slide

58. A hook in our deploy scripts
    which triggers VRT on a TeamCity agent
    which sends a notification via HipChat
    and lets us see the results on a website
    

    View Slide

59. Tests were slow
    Can’t run individual tests
    No integration with our process
    

    View Slide

60. Is everything
    perfect?
    

    View Slide

61. • Stability of tests
    • Test for different conditions
    • User agents
    • Resolutions
    • Expand test coverage
    

    View Slide

62. These are trivial
    problems
    and
    they only need to be
    solved once
    

    View Slide

63. Thank you
    @NikhilVerma
    joind.in/talk/bfb96
    

    View Slide