PhantomCSS - From a tool to a process

I talked about how we at Badoo used PhantomCSS and VRT from a tool used by a single developer on a machine into a process which is used by the team and is integrated into our continuous delivery system.

Transcript

1. PhantomCSS from a tool to a process Nikhil Verma
2. Nikhil Verma Front-End / JS Developer for 4 years MOBILE WEB TEAM @
3. Why do we need VRT? We already have unit testing, automation testing and manual testing process
4. Swiss cheese model
5. Unit testing Automation testing Visual Regression testing Human testing Change
6. The process today
7. Local Visual Regression Testing • Used by developers • Choose what to test • Manage your own baseline • Develop/Refactor/Debug
8. Automated Visual Regression Testing • Feature/Staging • Runs the entire test suite • Baseline from production • Catch regressions and bugs
9. Running tests locally
11. git checkout master git checkout -b MW-1234_fix_bugs git ls-files | xargs git rm -r git add -A git commit -m “fixed all bugs”
12. • Run visual regression locally if needed Feature Website Staging Website On Production Develop
13. • Download baseline images from server • PhantomCSS: Generate and compare images • Upload screenshots to server • Send notifications to developer Feature Website Staging Website On Production Develop
16. Feature Website Staging Website On Production Develop • Download baseline images from server • PhantomCSS: Generate and compare images • Upload screenshots to server • Send notifications to release room
18. Regenerate baseline from production Feature Website Staging Website On Production Develop
19. Examples of issues caught by VRT
24. How it started
25. Cristiano Rastelli Front-End / CSS Developer didoo.net / @areaweb MOBILE WEB TEAM
26. Tests were slow Hard to run individual tests Hardcoded to a specific user No integration with our process
27. Let’s start improving it! one problem at a time
28. Tests were slow .thenEvaluate(function() { require('Core/History').navigate('popularity'); }) .wait(5000) // wait for page to render .thenEvaluate(function() { $('.brick__image').css('background-image','url(http:// placehold.it/90x90)'); }) .wait(5000) // ensure placeholder images are downloaded .then(function() { phantomcss.screenshot(".page.popularity", 5, '', "Pages/Popularity"); })
29. Wait for page to render
30. Add data attributes on the client readyForVrt() { this.parent.attr('data-vrt-ready', 'true'); }
31. Add data attributes on the client // Inside a page implementation someNetworkRequest() .then(data => { this.$el.html(someTemplate.render(data)); }) .then(() => this.readyForVrt());
32. Ensure placeholder images are downloaded
33. Replace placehold.it images with base64 module.exports = { PLACEHOLDER_90x90: 'data:image/png;base64,iVBORw0KGgoA PLACEHOLDER_180x180: 'data:image/png;base64,iVBORw0KGg PLACEHOLDER_400x400: 'data:image/png;base64,iVBORw0KGg RAINBOW: 'data:image/png;base64,iVBORw0KGgoAAAAN...' };
34. Replace placehold.it images with base64 var PLACEHOLDER_90x90 = require('config/ constants').PLACEHOLDER_90x90; .thenEvaluate(function (placeholder90) { $(‘.brick__image') .css( 'background-image', 'url("' + placeholder90 + '")' ); }, PLACEHOLDER_90x90)
35. Replace placehold.it images with base64
36. navigate('popularity') .thenEvaluate(function (placeholder90) { $('.brick__image') .css( 'background-image', 'url("' + placeholder90 + '")' ); }, PLACEHOLDER_90x90) .then(function () { phantomcss.screenshot(...); });
37. Before: 6.8 mins After: 1.2 mins ~6x faster!
38. Tests were slow Hard to run individual tests Hardcoded to a specific user No integration with our process
39. Hard to run individual tests
40. Inquirer - Convert an array of objects https://www.npmjs.com/package/inquirer
41. Inquirer - To a list of questions
42. inquirer.prompt(questions).then(function (answers) { var testList = answers.split ? answers.tests : getFullTestList(answers.suite); launchSuite( answers.host, answers.suite, testList, answers.login, answers.password ); });
43. casperjs -- --config=config/phantomjs.json --concise --login=nikhil10 —password=***** --target=https:// mw-1245.mshot.badoo.com test instructions/mobileweb/ tests/login.js instructions/mobileweb/tests/chat.js instructions/mobileweb/tests/gifts.js instructions/ mobileweb/tests/liked-you.js --pre=instructions/ mobileweb/pre.js --post=instructions/mobileweb/post.js This is what one would have to type
44. Tests were slow Hard to run individual tests Hardcoded to a specific user No integration with our process
45. Hardcoded to a specific user
46. Enter: QAAPI QAAPI is an internal testing API in Badoo exposed to developers and QAs. It allows them to modify test accounts and trigger various functionality in the application.
47. http://qaapi/userRegister http://qaapi/forcePromoNotification http://qaapi/generateWantYou http://qaapi/userAddFriend http://qaapi/userAddFavorite http://qaapi/chatSendMessage and many more…
48. Convert QAAPI calls to JS functions
49. Generate a brand new test user
50. Login with the test user casper.user = user; // in tests .then(function () { this.fillSelectors('.form-login', { 'input[name=name]': this.user.email, 'input[name=password]': this.user.password }, false); })
51. Tests were slow Hard to run individual tests Unreliable due to depending on a specific user No integration with our process
52. • Plug into existing CI system • Allow developers to access the results
53. Plug into existing CI system • After deploying feature/staging we want to trigger a build agent to run VRT • Build agent checks out our visual regression repo • Run a script
54. Triggering a build: TeamCity http://teamcity/httpAuth/action.html? add2Queue=Mobileweb_VisualRegressionTests&name= build.shot_name&value={hostName} https://www.jetbrains.com/teamcity/
55. Build agent: Checkout git and execute script git clone git@git.mlan:mobileweb/vrt.git cd vrt/ ./run.sh --host={hostName}
56. npm install # make directories mkdir screenshots mkdir screenshots/$VRT_URL # download baseline ./utils/downloadMasterResultScreens.sh $VRT_URL # run tests ./visualregression.js --suite=styleguide --host=$VRT_URL ./visualregression.js --suite=mobileweb --host=$VRT_URL # upload results ./utils/uploadResultScreens.sh $VRT_URL # check for errors and notify developers ERRORS_FILE="screenshots/$VRT_URL/error.log" if [ -f $ERRORS_FILE ]; then php ./utils/notifyIssueUsers.php "$(cat $ERRORS_FILE)" $VRT_URL fi
57. Allow developers to access the results • Results are stored in one folder • Read the folder and match file names to generate a JS object *.fail.png - Test failed *.diff.png - Test passed *.png - New screenshot • Use the JS object to render a template and serve it over http
58. A simple webapp using Koa which is a frontend to the “screenshots” folder http://koajs.com/
59. The final architecture
60. Smart Prompt > Build Agent Node application PhantomCSS CasperJS ResembleJS PhantomJS Scripts Screenshots Diffs Results Web Application Baseline - Master Results - MW1234 Results - MW9876 Results - MW5432 Node.js App GIT Results Baseline Diffs TeamCity App Trigger Event HipChat
61. Tests were slow Hard to run individual tests Unreliable due to depending on a specific user No integration with our process
63. Branch naming conventions VRT-1234_my_feature MW-1234_my_feature Use a matching VRT branch to run tests in case of refactoring
64. Javascript Error logging Log all JS errors during the text execution in a file then upload it along with the regression results to make debugging easier.
65. Utilise npm config to store your user npm config set vrtuser xxxxxxxx npm config set vrtpass xxxxxxxx Now we don’t have to enter our test user details all the time
66. Q: Is everything perfect? Answer : Nope
67. • Stability of tests (It’s not unit testing) • Reliability of QAAPI/data received • Test for different conditions (user agents, viewports) • Expand test coverage
68. Thank you Answer : Nope