PhantomCSS - From a tool to a process

Slide 1

Slide 1 text

PhantomCSS from a tool to a process Nikhil Verma

Slide 2

Slide 2 text

Nikhil Verma Front-End / JS Developer for 4 years MOBILE WEB TEAM @

Slide 3

Slide 3 text

Why do we need VRT? We already have unit testing, automation testing and manual testing process

Slide 4

Slide 4 text

Swiss cheese model

Slide 5

Slide 5 text

Unit testing Automation testing Visual Regression testing Human testing Change

Slide 6

Slide 6 text

The process today

Slide 7

Slide 7 text

Local Visual Regression Testing • Used by developers • Choose what to test • Manage your own baseline • Develop/Refactor/Debug

Slide 8

Slide 8 text

Automated Visual Regression Testing • Feature/Staging • Runs the entire test suite • Baseline from production • Catch regressions and bugs

Slide 9

Slide 9 text

Running tests locally

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

git checkout master git checkout -b MW-1234_fix_bugs git ls-files | xargs git rm -r git add -A git commit -m “fixed all bugs”

Slide 12

Slide 12 text

• Run visual regression locally if needed Feature Website Staging Website On Production Develop

Slide 13

Slide 13 text

• Download baseline images from server • PhantomCSS: Generate and compare images • Upload screenshots to server • Send notiﬁcations to developer Feature Website Staging Website On Production Develop

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

Feature Website Staging Website On Production Develop • Download baseline images from server • PhantomCSS: Generate and compare images • Upload screenshots to server • Send notiﬁcations to release room

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

Regenerate baseline from production Feature Website Staging Website On Production Develop

Slide 19

Slide 19 text

Examples of issues caught by VRT

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

No content

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

How it started

Slide 25

Slide 25 text

Cristiano Rastelli Front-End / CSS Developer didoo.net / @areaweb MOBILE WEB TEAM

Slide 26

Slide 26 text

Tests were slow Hard to run individual tests Hardcoded to a speciﬁc user No integration with our process

Slide 27

Slide 27 text

Let’s start improving it! one problem at a time

Slide 28

Slide 28 text

Tests were slow .thenEvaluate(function() { require('Core/History').navigate('popularity'); }) .wait(5000) // wait for page to render .thenEvaluate(function() { $('.brick__image').css('background-image','url(http:// placehold.it/90x90)'); }) .wait(5000) // ensure placeholder images are downloaded .then(function() { phantomcss.screenshot(".page.popularity", 5, '', "Pages/Popularity"); })

Slide 29

Slide 29 text

Wait for page to render

Slide 30

Slide 30 text

Add data attributes on the client readyForVrt() { this.parent.attr('data-vrt-ready', 'true'); }

Slide 31

Slide 31 text

Add data attributes on the client // Inside a page implementation someNetworkRequest() .then(data => { this.$el.html(someTemplate.render(data)); }) .then(() => this.readyForVrt());

Slide 32

Slide 32 text

Ensure placeholder images are downloaded

Slide 33

Slide 33 text

Replace placehold.it images with base64 module.exports = { PLACEHOLDER_90x90: 'data:image/png;base64,iVBORw0KGgoA PLACEHOLDER_180x180: 'data:image/png;base64,iVBORw0KGg PLACEHOLDER_400x400: 'data:image/png;base64,iVBORw0KGg RAINBOW: 'data:image/png;base64,iVBORw0KGgoAAAAN...' };

Slide 34

Slide 34 text

Replace placehold.it images with base64 var PLACEHOLDER_90x90 = require('config/ constants').PLACEHOLDER_90x90; .thenEvaluate(function (placeholder90) { $(‘.brick__image') .css( 'background-image', 'url("' + placeholder90 + '")' ); }, PLACEHOLDER_90x90)

Slide 35

Slide 35 text

Replace placehold.it images with base64

Slide 36

Slide 36 text

navigate('popularity') .thenEvaluate(function (placeholder90) { $('.brick__image') .css( 'background-image', 'url("' + placeholder90 + '")' ); }, PLACEHOLDER_90x90) .then(function () { phantomcss.screenshot(...); });

Slide 37

Slide 37 text

Before: 6.8 mins After: 1.2 mins ~6x faster!

Slide 38

Slide 38 text

Tests were slow Hard to run individual tests Hardcoded to a speciﬁc user No integration with our process

Slide 39

Slide 39 text

Hard to run individual tests

Slide 40

Slide 40 text

Inquirer - Convert an array of objects https://www.npmjs.com/package/inquirer

Slide 41

Slide 41 text

Inquirer - To a list of questions

Slide 42

Slide 42 text

inquirer.prompt(questions).then(function (answers) { var testList = answers.split ? answers.tests : getFullTestList(answers.suite); launchSuite( answers.host, answers.suite, testList, answers.login, answers.password ); });

Slide 43

Slide 43 text

casperjs -- --config=config/phantomjs.json --concise --login=nikhil10 —password=***** --target=https:// mw-1245.mshot.badoo.com test instructions/mobileweb/ tests/login.js instructions/mobileweb/tests/chat.js instructions/mobileweb/tests/gifts.js instructions/ mobileweb/tests/liked-you.js --pre=instructions/ mobileweb/pre.js --post=instructions/mobileweb/post.js This is what one would have to type

Slide 44

Slide 44 text

Tests were slow Hard to run individual tests Hardcoded to a speciﬁc user No integration with our process

Slide 45

Slide 45 text

Hardcoded to a speciﬁc user

Slide 46

Slide 46 text

Enter: QAAPI QAAPI is an internal testing API in Badoo exposed to developers and QAs. It allows them to modify test accounts and trigger various functionality in the application.

Slide 47

Slide 47 text

http://qaapi/userRegister http://qaapi/forcePromoNotiﬁcation http://qaapi/generateWantYou http://qaapi/userAddFriend http://qaapi/userAddFavorite http://qaapi/chatSendMessage and many more…

Slide 48

Slide 48 text

Convert QAAPI calls to JS functions

Slide 49

Slide 49 text

Generate a brand new test user

Slide 50

Slide 50 text

Login with the test user casper.user = user; // in tests .then(function () { this.fillSelectors('.form-login', { 'input[name=name]': this.user.email, 'input[name=password]': this.user.password }, false); })

Slide 51

Slide 51 text

Tests were slow Hard to run individual tests Unreliable due to depending on a speciﬁc user No integration with our process

Slide 52

Slide 52 text

• Plug into existing CI system • Allow developers to access the results

Slide 53

Slide 53 text

Plug into existing CI system • After deploying feature/staging we want to trigger a build agent to run VRT • Build agent checks out our visual regression repo • Run a script

Slide 54

Slide 54 text

Triggering a build: TeamCity http://teamcity/httpAuth/action.html? add2Queue=Mobileweb_VisualRegressionTests&name= build.shot_name&value={hostName} https://www.jetbrains.com/teamcity/

Slide 55

Slide 55 text

Build agent: Checkout git and execute script git clone [email protected]:mobileweb/vrt.git cd vrt/ ./run.sh --host={hostName}

Slide 56

Slide 56 text

npm install # make directories mkdir screenshots mkdir screenshots/$VRT_URL # download baseline ./utils/downloadMasterResultScreens.sh $VRT_URL # run tests ./visualregression.js --suite=styleguide --host=$VRT_URL ./visualregression.js --suite=mobileweb --host=$VRT_URL # upload results ./utils/uploadResultScreens.sh $VRT_URL # check for errors and notify developers ERRORS_FILE="screenshots/$VRT_URL/error.log" if [ -f $ERRORS_FILE ]; then php ./utils/notifyIssueUsers.php "$(cat $ERRORS_FILE)" $VRT_URL fi

Slide 57

Slide 57 text

Allow developers to access the results • Results are stored in one folder • Read the folder and match ﬁle names to generate a JS object *.fail.png - Test failed *.diff.png - Test passed *.png - New screenshot • Use the JS object to render a template and serve it over http

Slide 58

Slide 58 text

A simple webapp using Koa which is a frontend to the “screenshots” folder http://koajs.com/

Slide 59

Slide 59 text

The ﬁnal architecture

Slide 60

Slide 60 text

Smart Prompt > Build Agent Node application PhantomCSS CasperJS ResembleJS PhantomJS Scripts Screenshots Diffs Results Web Application Baseline - Master Results - MW1234 Results - MW9876 Results - MW5432 Node.js App GIT Results Baseline Diffs TeamCity App Trigger Event HipChat

Slide 61

Slide 61 text

Tests were slow Hard to run individual tests Unreliable due to depending on a speciﬁc user No integration with our process

Slide 62

Slide 62 text

No content

Slide 63

Slide 63 text

Branch naming conventions VRT-1234_my_feature MW-1234_my_feature Use a matching VRT branch to run tests in case of refactoring

Slide 64

Slide 64 text

Javascript Error logging Log all JS errors during the text execution in a ﬁle then upload it along with the regression results to make debugging easier.

Slide 65

Slide 65 text

Utilise npm conﬁg to store your user npm config set vrtuser xxxxxxxx npm config set vrtpass xxxxxxxx Now we don’t have to enter our test user details all the time

Slide 66

Slide 66 text

Q: Is everything perfect? Answer : Nope

Slide 67

Slide 67 text

• Stability of tests (It’s not unit testing) • Reliability of QAAPI/data received • Test for different conditions (user agents, viewports) • Expand test coverage

Slide 68

Slide 68 text

Thank you Answer : Nope