#LDNSE 12 : Advanced Visual Test Automation With Selenium

Slides from talk on Advanced Visual Test Automation with Selenium at the London Selenium Meetup

Transcript

1. Adam Carmi (@carmiadam) Co-Founder and VP R&D at Applitools adam.carmi@applitools.com ADVANCED VISUAL TEST AUTOMATION WITH SELENIUM
2. YOU CAN AND SHOULD AUTOMATE YOUR VISUAL TESTS!
4. AGENDA  Why automated visual testing?  Tools & Technology  Where does it fit?
5. WHAT IS VISUAL TESTING? A quality assurance activity aimed to verify that a Graphical User Interface appears correctly to users
6. A VISUAL BUG
7. AND ANOTHER…
8. WHY SHOULD IT BE AUTOMATED? THE TEST MATRIX IS TOO BIG TO APPROXIMATE MANUALLY  Web browsers  Devices  Operating systems  Screen resolutions  Responsive designs  L10n
9. WHY SHOULD IT BE AUTOMATED?
10. WHY SHOULD IT BE AUTOMATED? NATIVE / HYBRID MOBILE APPS  Harder to roll back changes  Can’t push daily  Updates take battery and data  Higher quality bar
11. WHY SHOULD IT BE AUTOMATED? Many are already doing it (and sharing)… PhantomCSS Fighting Layout Bugs CSS Critc Wraith Needle Grunt PhotoBox dpdxt WebdriverCSS Eyes Huxley FBSnapshotTestCase Gemini SeleniumVisual Diff VisualCeption Specter Snap And Compare kobold
12. AGENDA  Why automated visual testing?  Tools & Technology  Where does it fit?
13. THE WORKFLOW Drive the AUT and take screenshots Compare screenshots with baseline images Report differences Update the baseline
14. DEMO https://github.com/webdriverio/webdrivercss
15. THE WORKFLOW Drive the AUT and take screenshots Compare screenshots with baseline images Report differences Update the baseline
16. DRIVING AND CAPTURING OTHER • Wraith (URLs) • PhantomCSS (CJS) • dpdxt (URLs) • Grunt PhotoBox (URLs) • Snap And Compare (URLs) • Specter (JS) • Gemini (JS) • Grunt-Vigo (URLs) • WebDriverCSS (JS) • Fighting Layout Bugs (Java) • Wraith-Selenium (Ruby) • Selenium Visual Diff (Java) • Needle (Python) • VisualCeption (PHP) • Huxley (RP) • Gemini (JS) • Vizregress (.NET) • Visualci (Java) • Viff (JS) • Pix-Diff (JS) • Applitools Eyes (All) • CSS Critic (URLs) • FBSnapshotTestCase (XCTest) • Kobold (folders) • Applitools Eyes (All)
17. SCREENSHOTS  Real browsers?  Full page?  Frames?  Regions?  Viewport size?  Page stabilization?  Page preparation?
18. THE WORKFLOW Drive the AUT and take screenshots Compare screenshots with baseline images Report differences Update the baseline
20. WHY NOT? FALSE POSITIVES
21. ANTI ALIASING 1/2
22. ANTI ALIASING 2/2
23. BRIGHTNESS 1/2
24. BRIGHTNESS 2/2
25. SUB PIXEL SHIFTS 1/2
26. SUB PIXEL SHIFTS 2/2
27. IMAGE SCALING 1/2
28. IMAGE SCALING 2/2
29. AND MORE… 1 pixel offsets in element positioning Dynamic content Moving elements Images of different size Performance
31. Image Comparison APIs
32. ImageMagick  A powerful command line tool for image processing.  APIs are available for most programming languages.  Fuzzing is used to eliminate slight color differences  An error ratio is usually used to determine a match $ compare –metric AE –fuzz 5% img1.png img2.png diff.png 2246
33. Resemble.js  An image comparison library implemented in Javascript  Used by PhantomCSS and other tools.  Good antialiasing support  An error ratio is usually used to determine a match  http://huddle.github.io/Resemble.js/
34. Blink-Diff  An image comparison library implemented in Javascript  Can be operated from the command line using node  Perceptual color distance computation  Anti-aliasing support  An error ratio is used to determine a match  https://github.com/yahoo/blink-diff/
35. Applitools Eyes  A specialized image processing stack designed to compare computer generated UI images  Anti-aliasing, pixel offsets, and image scaling with low false negative rate  Dynamic and moving content  Compare images of different sizes  No error ratio configuration required  Fast!  Layout matching  Available as a cloud service
36. DEMO
37. THE WORKFLOW Drive the AUT and take screenshots Compare screenshots with baseline images Report differences Update the baseline
38. REPORT DIFFERENCES As files on the file system (combined with source control)
39. REPORT DIFFERENCES As a Gallery (example from Selenium Visual Diff)
40. THE WORKFLOW Drive the AUT and take screenshots Compare screenshots with baseline images Report differences Update the baseline
41. UPDATE THE BASELINE  Rename or commit individual image files  Overwrite mode
42. UPDATE THE BASELINE  GUI (Gemini GUI)  Automatic maintenance (Applitools Eyes)
43. AGENDA  Why automated visual testing?  Tools & Technology  Where does it fit?
44. WHERE DOES IT FIT? • Components • Code review • Developers • Designers • QA
45. WHERE DOES IT FIT? • Pages • Page sections • Developers • Designers • QA • Others
46. WHERE DOES IT FIT? • Staging vs. Production • Ops • QA
47. WHERE DOES IT FIT? • Monitoring • Ops • QA
48. QUESTIONS?
49. Adam Carmi (@carmiadam) Co-Founder and VP R&D at Applitools adam.carmi@applitools.com THANK YOU