$30 off During Our Annual Pro Sale. View Details »

Visual Regression Testing - from a tool to a process

Visual Regression Testing - from a tool to a process

We work with many JavaScript based tools in our day to day lives.

But how do we take those tools from being used by a single person on a machine and integrate it into a team's workflow?

How can we link those tool into chains and go from a tool to a process? :)

In this talk I will discuss how I converted PhantomCSS, a tool for visual regression testing, and integrated it into our workflow. I will also cover the challenges we faced which ranged from integrating the tests into our CI, test execution taking too long, ability to browse the test results and notifying developers about failed regressions.

Nikhil Verma

May 10, 2017
Tweet

More Decks by Nikhil Verma

Other Decks in Technology

Transcript

  1. VISUAL
    REGRESSION
    TESTING
    from a tool to a process
    Nikhil Verma

    View Slide

  2. Nikhil Verma
    Team Lead / JS Developer
    MOBILE WEB TEAM
    @NikhilVerma

    View Slide

  3. badoo.com
    team.badoo.com
    techblog.badoo.com

    View Slide

  4. VISUAL
    REGRESSION
    TESTING

    View Slide

  5. VISUAL

    View Slide

  6. It’s all about
    change
    which can be obvious or
    subtle

    View Slide

  7. Spot the difference
    in 15 seconds

    View Slide

  8. View Slide

  9. View Slide

  10. How can we easily
    detect change?

    View Slide

  11. Let the computer
    do it.

    View Slide

  12. View Slide

  13. REGRESSION
    TESTING

    View Slide

  14. Why do we need visual
    regression testing?

    View Slide

  15. Swiss cheese model
    https://wikipedia.org/wiki/Swiss_cheese_model

    View Slide

  16. Unit testing
    Automation testing
    Visual Regression testing
    Human testing
    Change

    View Slide

  17. The process
    today

    View Slide

  18. https://github.com/Huddle/PhantomCSS
    CasperJS PhantomJS
    = +
    ResembleJS
    +

    View Slide

  19. Local
    • Used by developers
    • Choose the tests to run
    • Manage your own “baseline”
    • Develop/Refactor/Debug

    View Slide

  20. View Slide

  21. • Feature/Staging branches
    • Runs the entire test suite
    • Baseline from production
    • Catch regressions and bugs
    Automated

    View Slide

  22. 1. Download baseline from server
    2. Generate and compare screenshots
    3. Upload results back to the server
    4. Send notifications to developer/team
    How does it work?

    View Slide

  23. View Slide

  24. View Slide

  25. Some issues caught by
    VRT

    View Slide

  26. View Slide

  27. View Slide

  28. View Slide

  29. View Slide

  30. How it started

    View Slide

  31. didoo.net / @areaweb
    Cristiano Rastelli
    Front-End / CSS Developer
    MOBILE WEB TEAM

    View Slide

  32. Tests were slow
    Can’t run individual tests
    No integration with our process

    View Slide

  33. Let’s start
    improving it!
    one problem at a time

    View Slide

  34. Tests were slow
    • Don’t know when the page will be ready
    • Wait for images to download

    View Slide

  35. Don’t know when the content will be
    ready

    View Slide

  36. // On the client, when a page is rendered
    $page.attr('data-vrt-ready', 'true');
    Client adds an attribute when ready

    View Slide

  37. // In the VRT test
    waitForSelector('[data-vrt-ready="true"]')
    .then(() => {
    // take screenshot
    });
    Wait for selector

    View Slide

  38. Wait for images to download

    View Slide

  39. Inline all images with base64
    const PLACEHOLDER_90x90 = ‘data:image/
    png;base64,iVBORw0KGgo…’;
    $('img').attr('src', PLACEHOLDER_90x90);

    View Slide

  40. Before: 6.8 mins
    After: 1.2 mins
    5x faster!

    View Slide

  41. Tests were slow
    Can’t run individual tests
    No integration with our process

    View Slide

  42. Can’t run individual tests

    View Slide

  43. View Slide

  44. casperjs -- --config=config/phantomjs.json —concise
    target=https://m.badoo.com test instructions/mobileweb/tests/
    login.js instructions/mobileweb/tests/chat.js instructions/
    mobileweb/tests/gifts.js instructions/mobileweb/tests/liked-
    you.js --pre=instructions/mobileweb/pre.js --
    post=instructions/mobileweb/post.js

    View Slide

  45. https://www.npmjs.com/package/inquirer

    View Slide

  46. Convert an array of questions
    module.exports = [{
    type: 'list',
    name: 'suite',
    message: 'What do you want to test?',
    choices: […],
    default: 'styleguide'
    }, {
    type: 'list',
    name: 'environment',
    message: 'Which environment do you want to use?',
    choices: function (answers) {…}
    }];

    View Slide

  47. To an interactive questionnaire

    View Slide

  48. Tests were slow
    Can’t run individual tests
    No integration with our process

    View Slide

  49. How to integrate it into
    the process?

    View Slide

  50. Local
    Feature
    Staging
    Production

    View Slide

  51. +

    View Slide

  52. Hook
    • Git/Mercurial post-receive hooks
    • Deploy scripts
    • JIRA/Trello webhooks
    • No hook! wat?

    View Slide

  53. Trigger
    • Run a bash script
    • Start a CI agent (TeamCity/Jenkins/CircleCI)
    • Boot a docker VM/Remote machine
    • Cron job (no hook needed!)

    View Slide

  54. +
    !

    View Slide

  55. Notification
    • Email
    • HipChat / Slack etc
    • Sound!

    View Slide

  56. See the results
    • Email
    • Hosted website with results
    • Commit to repository
    • Hard Disk/Network Attached Storage

    View Slide

  57. View Slide

  58. A hook in our deploy scripts
    which triggers VRT on a TeamCity agent
    which sends a notification via HipChat
    and lets us see the results on a website

    View Slide

  59. Tests were slow
    Can’t run individual tests
    No integration with our process

    View Slide

  60. Is everything
    perfect?

    View Slide

  61. • Stability of tests
    • Test for different conditions
    • User agents
    • Resolutions
    • Expand test coverage

    View Slide

  62. These are trivial
    problems
    and
    they only need to be
    solved once

    View Slide

  63. Thank you
    @NikhilVerma
    joind.in/talk/bfb96

    View Slide