Visual Regression Testing with Huxley

Huxley is a tool crated by Pete Hunt at Instagram (@floydophone).

Transcript

1. Visual Regression
   Testing with Huxley
   @nathanbirrell
   !
   15m
   

   View Slide

2. Visual Regression Testing with Huxley
   Outline
   Why?
   Workflows and Demo
   Limitations
   1
   2
   3
   

   View Slide

3. Visual Regression Testing with Huxley
   Problem
   Testing the user
   interface can be slow
   and fragile.

   Unit testing does not
   test the browser-
   rendered UI.

   ○
   A style change for
   one component can
   have unforeseen
   repercussions.
   

   View Slide

4. Visual Regression Testing with Huxley
   Huxley
   !
   !
   !
   !
   Record your
   actions
   Take
   screenshots
   Action
   playback
   Update
   screenshots
   with changes
   

   View Slide

5. Visual Regression Testing with Huxley
   How it Works
   Selenium Server
   Selenium WebDriver (browser-specific plugin)
   A server (Middleman)
   Huxleyfile
   

   View Slide

6. Visual Regression Testing with Huxley
   Huxleyfile
   [Test01]
   url=http://0.0.0.0:4567/PageToTest1.html
   !
   [Test02]
   url=http://0.0.0.0:4567/PageToTest2.html
   screensize=1024x768
   sleepfactor=0.5
   browser=firefox
   !
   ...
   Name URL
   Screen size
   (for responsive testing)
   Browser selection
   (currently unreliable)
   Speed
   

   View Slide

7. Visual Regression Testing with Huxley
   Files
   Test01/diff.png
   Test01/record.json
   Test01/
   

   View Slide

8. Visual Regression Testing with Huxley
   Workflows
   Default:
   2. Make normal code changes
   3. Run Huxley [Screenshots
   overwritten to show visual
   changes]
   4. Commit changes to git
   5. Compare screenshots to
   previous commits (using an
   image comparison app)
   1. Record all tests
   Screenshots and click events registered to the millisecond
   Alternative:
   2. Make normal code changes in
   separate branches
   3. Keep a master branch for a
   base to test from
   4. Run Huxley on the base
   branch, then on the working
   branch for a pixel-perfect
   comparison.
   

   View Slide

9. Visual Regression Testing with Huxley
   Workflows
   Default:
   2. Make normal code changes
   3. Run Huxley [Screenshots
   overwritten to show visual
   changes]
   4. Commit changes to git
   5. Compare screenshots to
   previous commits (using an
   image comparison app)
   1. Record all tests
   Screenshots and click events registered to the millisecond
   Alternative:
   2. Make normal code changes in
   separate branches
   3. Keep a master branch for a
   base to test from
   4. Run Huxley on the base
   branch, then on the working
   branch for a pixel-perfect
   comparison.
   

   View Slide

10. Visual Regression Testing with Huxley
    Limitations
    Time
    Recordings take a while to
    get through on large
    projects.
    ☼
    Browsers
    Browser selection is
    supported, but seems a
    little complicated.
    Animation
    Sometimes .gif spinners
    and JS movements throw
    errors.
    

    View Slide

11. Visual Regression Testing with Huxley
    To come…
    Integration with VirtualBox VMs
    Better browser integration (IE and Chrome)
    

    View Slide

12. Demo
    

    View Slide

13. Visual Regression Testing with Huxley
    More
    Slides: speakerdeck.com/nathanbirrell
    Github: github.com/facebook/huxley
    Jira guide: bit.ly/ddhuxley
    Podcast on Huxley: bit.ly/LS7iTL
    

    View Slide