Adam Carmi (Applitools), Transform Your Automated Tests with Visual Testing, CodeFest 2017

https://2017.codefest.ru/lecture/1142

Automated visual testing is a major emerging trend in the dev / test community. It allows you to automatically verify that your app's GUI renders correctly across all devices, browsers, and screen resolutions. In this session, we will take a deep dive into some of the technological challenges involved with visual test automation and show how modern tools address them. We will review available open-source and commercial visual testing tools, demo cutting edge technologies that enable running cross browser and cross device visual tests at large scale, and show how visual test automation fits in the development / deployment lifecycle.

If you don't know what visual testing is, if you think that Sikuli is a visual test automation tool, if you are already automating your visual tests and want to learn more on what else is out there, if you are on your way to implement Continuous Deployment or just interested in seeing how cool image processing algorithms can be, this talk is for you!

Transcript

1. Automating visual software testing TRANSFORM YOUR AUTOMATED TESTS WITH VISUAL TESTING Adam Carmi CTO Applitools
2. Automating visual software testing YOU CAN AND SHOULD AUTOMATE YOUR VISUAL TESTS!
3. Automating visual software testing
4. Automating visual software testing AGENDA • Why automated visual testing? • Tools & Technology • Where does it fit? • Q & A
5. Automating visual software testing WHAT IS VISUAL TESTING? A quality assurance activity aimed to verify that a Graphical User Interface appears correctly to users
6. Automating visual software testing
7. Automating visual software testing
8. Automating visual software testing
9. Automating visual software testing WHY AUTOMATE? The test matrix is too big to cover manually • Web browsers • Devices • Operating systems • Screen resolutions • Responsive design • L10n • 3rd Party upgrades
10. Automating visual software testing WHY AUTOMATE? Mobile apps quality is critical • Harder to roll back changes • Can’t push daily • Updates take battery and data • Updates are not mandatory • Higher quality bar
11. Automating visual software testing WHY AUTOMATE? Release cycles keep getting shorter
12. Automating visual software testing AGENDA • Why automated visual testing? • Tools & Technology • Where does it fit? • Q & A
13. Automating visual software testing HOW DOES IT WORK? Drive the AUT and take screenshots Compare screenshots with baseline images Report differences Update the baseline
14. Automating visual software testing DEMO WebDriverCSS
15. Automating visual software testing HOW DOES IT WORK? Drive the AUT and take screenshots Compare screenshots with baseline images Report differences Update the baseline
16. Automating visual software testing
17. Automating visual software testing
18. Automating visual software testing
19. Automating visual software testing
20. Automating visual software testing
21. Automating visual software testing AND MORE… • 1 pixel offsets in element positioning • Dynamic content • Moving elements • Images of different size • Performance
22. Automating visual software testing
23. Automating visual software testing IMAGE COMPARISON ENGINES
24. Automating visual software testing IMAGE MAGICK • A command line tool for bitmap processing • Fuzzing is used to eliminate slight color differences • An error ratio is usually used to determine a match $ compare –metric AE –fuzz 5% img1.png img2.png diff.png 2246
25. Automating visual software testing THE JAVASCRIPT ENGINES • Resemble.js Pixel by pixel + error ratio + anti-aliasing http://huddle.github.io/Resemble.js • Blink-Diff Pixel by pixel + error ratio + anti-aliasing + perceptual color distance computation https://github.com/yahoo/blink-diff • Looks-Same Pixel by pixel + perceptual color distance computation + ignore Caret https://github.com/gemini-testing/looks-same
26. Automating visual software testing APPLITOOLS EYES • Handles anti-aliasing, pixel offsets, color similarity, and image scaling • Dynamic and moving content • Compare images of different sizes • No error ratio configuration required • Validates full UI pages • Fast! • Supports layout / structure matching
27. Automating visual software testing DEMO VISUAL LAYOUT MATCHING
28. Automating visual software testing HOW DOES IT WORK? Drive the AUT and take screenshots Compare screenshots with baseline images Report differences Update the baseline
29. Automating visual software testing QUICK FEEDBACK TOOLS MOTIVATION • Get fast feedback on code changes • Run tests locally in the background CONS • Partial coverage • Chrome, IE? • Limited navigation • Stale browser versions SETUP • Render screenshots with a headless browser • PhatomJS, SlimerJS • Configuration file driven tests
30. Automating visual software testing VISUAL COVERAGE TOOLS MOTIVATION • Verify that the app / site looks right in all real execution environments • Test as many UI states as possible • Execute many tests in parallel CONS • Requires test infrastructure • Implement and maintain test code SETUP • Render screenshots using real browsers on a variety of operating systems and devices in parallel • WebDriver, Grid • Code driven tests • WebDriver, DSL
31. Automating visual software testing SAMPLE TOOLS CODE / SCRIPT • Needle (Python WD) • Gemini (JS DSL) • Selenium Visual Diff (Java WD) • VisualCeption (PHP CodeCeption) • Shoov (JS WebDriverIO) • Vizregress (.NET WD) • Rspec Page Regression (Capibara) • Applitools Eyes (All WD + Appium) • PhantomCSS (JS CasperJS) • Specter (JS DSL) • FBSnapshotTestCase (XCTest) CONFIGURATION • Viff • CSS Visual Test • Wraith • Dpdxt • Grunt PhotoBox • Snap And Compare • BackstopJS • CSSCritic • Kobold • DiffCop • SUCCSS
32. Automating visual software testing HOW DOES IT WORK? Drive the AUT and take screenshots Compare screenshots with baseline images Report differences Update the baseline
33. Automating visual software testing REPORT DIFFERENCES As files on the file system
34. Automating visual software testing REPORT DIFFERENCES As a gallery
35. Automating visual software testing REPORT DIFFERENCES In your CI build report
36. Automating visual software testing HOW DOES IT WORK? Drive the AUT and take screenshots Compare screenshots with baseline images Report differences Update the baseline
37. Automating visual software testing UPDATE THE BASELINE Gemini
38. Automating visual software testing UI VERSION CONTROL • Keep CI tests green • Multiple app versions • Revert to previous baselines • A visual change log (documentation) 1 1 1 2 1 2 3 3 2 4 4
39. Automating visual software testing
40. Automating visual software testing DEMO AUTOMATED MAINTENANCE
41. Automating visual software testing AGENDA • Why automated visual testing? • Tools & Technology • Where does it fit? • Q & A
42. Automating visual software testing WHERE DOES IT FIT? CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION Before committing / merging code: • See how UI changes appear in other execution environments • Make sure there are no unintended changes
43. Automating visual software testing WHERE DOES IT FIT? CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION Visual testing of frontend components by frontend developers
44. Automating visual software testing WHERE DOES IT FIT? CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION Regression, Localization, Accessibility, Cross browser/device, responsive design testing and more… All team members can review and maintain A collaboration tool
45. Automating visual software testing WHERE DOES IT FIT? CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION Validate your staging deployment using your production deployment as a baseline
46. Automating visual software testing WHERE DOES IT FIT? CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION No missing resources in production No corruption due to 3rd party data, upgrades, hackers, etc.
47. Automating visual software testing Adam Carmi (@carmiadam) Co-Founder and CTO at Applitools QUESTIONS?