PhantomCSS within web architecture

Presented on 21st March 2017 for @webdeldn meetup

https://www.eventbrite.co.uk/e/webdeldn-5-visual-regression-testing-tickets-32104099225#

Some great questions were asked, here they are:

Q. How long did it take to amass 1532 tests
A. Huddle began writing PhantomCSS tests (VRT) from the beginning of 2012, though PhantomCSS wasn't open-sourced until November of that year. Huddle have continued to use PhantomCSS ever since.

Q. Do you share VRTs with other developers, if so, how do you share the baseline.
A. We commit the baseline images into the git repo(s) along with the tests and the code they're testing, this works well for us because all the developers at Huddle use Windows machines (not all the same version though), as do our CI build agents. We don't do environment "snapshot" testing, it fully supportive of Huddles Continuous deployment strategy - we sometimes release two of three times a week.

Q. Do you often have flaky tests?
A. Yes, but in proportion to the amount of test we have the number of flaky tests are quite small. Usual reasons for flakyness are caused by assumptions in tests about the order of events/callbacks, but we've also seen reflow issues caused by image and font loading. Flakyness is heavily mitigated by proper SUT isolation.

Q. Do you test responsive layout.
A. Not much, but yes we do. PhantomJS/CasperJS provides methods for changing the viewport.

Q. How do you test mutable states like contextual date/time stamps, e.g. "Just now", "two minutes ago", "a day ago" etc.
A. We use the following library to fake the Date object https://github.com/sinonjs/lolex. (FYI we also use https://github.com/sinonjs/sinon for our XHR fakes)

Q. How long does it take to run all these tests?
A. We run the VRTs along with the functional tests. They're run against four git repos, five test suites. In CI they take less that ten minutes (depending on npm installs) , on local they're a lot quicker - though locally we can't parallelise each repo's test suite.

Q. What about browsers?
A. Huddle don't do any automated browser testing, we believe there is not enough value in doing so. Anecdotally In the five years of not doing
automated browser testing we have never had a critical x-browser regression. Our cross functional teams include QA which may be a factor in the early discovery and fixes of bugs.

Transcript

1. P H A N T O M C S S W I T H I N W E B U I A R C H I T E C T U R E
2. PhantomCSS is a CasperJS module for automating visual regression testing with PhantomJS and Resemble.js. For testing Web apps, live style guides and responsive layouts
3. Test coverage of UI without implementation coupling. Tests that reflect what the user sees, not what the developer writes
4. • Manual testing is a time consuming process • The risk of regression is higher with responsive layouts and complex applications • CSS refactoring is necessary for long life projects • Visual regression identification can often be difficult and subjective W H Y ?
5. casper.start( url ).then(function(){ casper.click(‘button’); phantomCSS.screenshot( ‘.modal’, ‘Modal dialog’ ); });
6. - More logs casperjs test demo/testsuite.js --verbose —log-level=debug - Use SlimerJS instead of PhantomJS casperjs test demo/testsuite.js --engine=slimerjs - Use debugger; statements with Webkit remote console casperjs test demo/testsuite.js —remote-debugger-port=9000 CO M M A N D L I N E
7. CO M M A N D L I N E
8. • Write the test • Run the test • Check the baseline image • Git ignore the ‘diff’ images • Commit the baseline image B A S I C WO R K F LO W
9. S Y S T E M U N D E R T E S T J A VA S C R I P T H T M L C S S F I X T U R E I N T E R FA C E PhantomCSS provides implicit test coverage of behaviour, template and style regardless of technology choices.
10. – M A R T I N F O W L E R “A classic test only cares about the final state - not how that state was derived. ” J A VA S C R I P T H T M L C S S F I X T U R E I N T E R FA C E Visual regression tests can also be Unit tests. They’re fast, and can be run individually and in any order.
11. Isolating the system under test makes feedback faster and tests less fragile. Confidence in tests makes applications easier and cheaper to change, extend and refactor J A VA S C R I P T H T M L C S S F I X T U R E I N T E R FA C E
12. Visual regression testing as isolated Unit tests is core to Huddle’s front-end Web application architecture J A VA S C R I P T H T M L C S S F I X T U R E I N T E R FA C E
13. Developed at Huddle; tools that help people gather, review, create and share content. A complex Javascript Web application
14. 1532 visual tests in total Visual Functional
15. • Simple UI components do not speak directly with backend services • Complex UI components handle request/response with backend services. They are composed of other complex and simple UI components. • Complex and simple components require different test strategies, but both benefit from PhantomCSS coverage S I M P L E & C O M P L E X U I C O M P O N E N T S
16. S I M P L E & C O M P L E X U I C O M P O N E N T S Encapsulated behaviours with declarative inputs
17. S I M P L E & C O M P L E X U I C O M P O N E N T S Variations are generated automatically for both documentation and tests
18. S I M P L E & C O M P L E X U I C O M P O N E N T S External dependencies are faked in code
19. S I M P L E & C O M P L E X U I C O M P O N E N T S Fake setups can be evaluated from tests as well as in a hot-reload development environment
20. S I M P L E & C O M P L E X U I C O M P O N E N T S Fake setups can be evaluated from tests to support visual regression testing with PhantomCSS
21. https://github.com/huddle/phantomcss @ J A M E S C RY E R P H A N T O M C S S W I T H I N W E B U I A R C H I T E C T U R E