TMPA-2021: Algorithm for Mapping Layout Grids in User Interfaces: Automating the “Squint Test”

Maxim Bakaev and Maxim Shirokov

Algorithm for Mapping Layout Grids in User Interfaces: Automating the “Squint Test”

TMPA is an annual International Conference on Software Testing, Machine Learning and Complex Process Analysis. The conference will focus on the application of modern methods of data science to the analysis of software quality.

To learn more about Exactpro, visit our website https://exactpro.com/

Follow us on
LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro

Transcript

1. Algorithm for Mapping Layout Grids in User Interfaces: Automating the “Squint Test” Maxim Bakaev, Maxim Shirokov bakaev@corp.nstu.ru Novosibirsk State Technical University, Russia Tools and Methods for Program Analysis (TMPA-2021)
2. TMPA 2021 Introduction – UI Testing ▪ Number of potential interaction contexts for a human-computer system nowadays? ❑ Websites (not even web pages!): 108 ❑ Mobile and web users: 109 ❑ Devices (office, home, mobile, etc.): 1010 ▪ 100% “manual” design and testing of UIs are no longer feasible – automation is needed ▪ Traditional approach: code-based UI analysis ❑ corresponds to “static” testing ❑ particularly suitable for web UIs (HTML/CSS code is easily accessible and well defined in standards and guidelines) ❑ widely used for validating web UIs’ technical quality, compliance, accessibility, etc.
3. TMPA 2021 Introduction – Visual Analysis ▪ Gaining in popularity recently: Image-based (visual) UI analysis ❑ corresponds to “dynamic” testing ▪ Powered by recent advances in image recognition technologies Still computationally expensive (order of 100s seconds per UI) Needs lots of training data for identifying and classifying UI elements (and still not entirely accurate) ▪ Can “see” what the target user sees in the environment ⊕ Good for layouts, UI elements’ sizes, color-related aspects, etc. ⊕ Naturally considers the interaction context, i.e. screen size ⊕ Can be used to predict visual complexity, aesthetical and even trust impressions in target users
4. TMPA 2021 Our Previous Work ▪ We proposed Coarsening Method for converting real web interfaces into coded representations of layouts ❑ the method was inspired by the informal “Squint Test” widely used in usability engineering for estimation of visual organization ❑ today, all websites are designed with layout grids (Bootstrap has it) ▪ Overlaying 2D grid over web interface image and assigning 0 or 1 to each cell of the grid ❑ The coded representations did predict visual complexity perception in both model grids and real websites ▪ M. Bakaev et al. (2019) Data compression algorithms in analysis of UI layouts visual complexity. Lecture Notes in Computer Science, 11964: Perspectives of System Informatics. P. 167-184
5. TMPA 2021 The Grid and Coding Example
6. TMPA 2021 The Human Annotators ▪ The grid overlaying and coding was implemented manually, by 8 students (averaged) for 21 websites
7. APPROACH AND METHODS: Automated construction of layout grids on top of visual representations (screenshots) of existing web user interfaces TMPA 2021 RESEARCH OBJECTIVE: • Software tool implementing the specially developed computer vision based algorithm • OpenCV: edge detection, pixelation, etc. • Microsoft.NET with C++/CLI: the algorithm programming and the tool’s UI implementation • Validation of the outcome with 497 real websites • Ongoing work: calibration of the algorithm’s parameters
8. TMPA 2021 Algorithm – 1) Edge Detection ▪ We used Canny edge detector from the OpenCV library to perform the following steps: 1. removing noise and clutter from the image; 2. calculating the image gradient; 3. edge thinning; 4. edge linking. CVAPI(void) cvCanny( const CvArr* image, CvArr* edges, double threshold1, double threshold2, int aperture_size CV_DEFAULT(3) ); aperture_size is the parameter for Sobel’s operator
9. TMPA 2021 Algorithm – 1) Edge Detection
10. TMPA 2021 Algorithm – 1) Edge Detection
11. TMPA 2021 Algorithm – 2) Image Pixelation ▪ The goal of the step is to make the elements in the image more obvious for the algorithm 1. The image size is reduced by the factor of 3 2. The image is recovered to the original size ▪ We used cvResize() function from the OpenCV library: CVAPI(void) cvResize( const CvArr* src, CvArr* dst, int interpolation CV_DEFAULT( CV_INTER_LINEAR )); interpolation is bilinear interpolation method (by default) – requires the minimal time for processing
12. TMPA 2021 Algorithm – 1) Edge Detection
13. TMPA 2021 Algorithm – 1) Edge Detection
14. TMPA 2021 Algorithm – 3) Grid Overlaying 1. The analysis starts by columns (vertical lines) ❑ looking for continuous segments of black and white pixels ❑ depending on the grid accuracy parameter (allowed ratio of the segment’s length to the column length) and the allowed noise – error (allowed percentage of irregular white or black pixels in the segment), decide on making a column ❑ the next column is checked only after the minimum grid step (as the percentage of the whole image dimension) 2. The analysis continues by rows (horizontal lines) ❑ the principle is the same, with the grid accuracy, the error and the minimum grid step parameters independently specified for the horizontal dimension ❑ but the analysis is performed only within a given column
15. TMPA 2021 Algorithm – 1) Edge Detection
16. TMPA 2021 Algorithm – 1) Edge Detection
17. TMPA 2021 Algorithm – 4) Cell Coding ▪ For each cell in the grid we assign 0 or 1 value ❑ the threshold for the decision is the ratio of white pixels in the cell ▪ This is a fuzzy part, linked to human perception ❑ the work on calibrating the threshold parameter is still ongoing in our project (currently we empirically chose 3%) ❑ we are also working on auto-tuning of the parameters depending on the image set characteristics ▪ After the analyses, the original image is recovered, with the grid overlayed and the cells coded ❑ The algorithm can work in “batch” mode, taking the list of the screenshot filenames and outputting the grids and the coding data
18. TMPA 2021 Algorithm – 1) Edge Detection
19. TMPA 2021 Algorithm – 1) Edge Detection
20. TMPA 2021 Algorithm – 1) Edge Detection
21. TMPA 2021 Algorithm – Parameters’ Effects (1)
22. TMPA 2021 Algorithm – Parameters’ Effects (1)
23. TMPA 2021 Algorithm – Parameters’ Effects (2)
24. TMPA 2021 Algorithm – Parameters’ Effects (2)
25. TMPA 2021 Validation – Experiment Description ▪ Material: 497 screenshots of university websites’ homepages ▪ Subjects: 70 human annotators ❑ NSTU students and IT specialists ▪ Dependent variables (collected with Likert scale survey): ❑ Complexity: how visually complex the WUI appears ❑ Aesthetics: how aesthetically pleasant the WUI appears ❑ Orderliness: how orderly the WUI appears ▪ Independent variables (produced by our software): ▪ Columns: the number of columns in the overlaid grid, ▪ Rows: the number of rows in the overlaid grid, ▪ Cells: the total number of cells in the overlaid grid (note that due to spanning of some cells this number is less or equal to Rows multiplied by Columns), ▪ Whitespace: the number of cells with the 0 value divided by the total number of cells in the overlaid grid.
26. TMPA 2021 Validation – Results ▪ We obtained 4235 full evaluations, per the 3 scales each ❑ each screenshot was evaluated by 8-10 participants ▪ Linear regression equations for all three scales ❑ with the Backwards variable selection method ❑ the Whitespace factor is significant in all the equations ❑ all the equations are highly significant
27. TMPA 2021 Conclusions ▪ UI testing/analysis automation is a growing trend ❑ code-based UI analysis is fast and accurate, already widely used (e.g. in accessibility validation), but it can not consider layouts ❑ image-based UI analysis is slower and generally requires training data for ML, but can “see” what user sees in any device ▪ Previously we proposed an analogue of the “Squint Test” (reduction of visual fidelity) that proved meaningful for VC ❑ and in the current work we present the algorithm for automating it ▪ The grid overlaying algorithm for web UIs: ❑ A sort of reverse engineering (grids are used in design mockups) ❑ Based on Computer Vision techniques (OpenCV library) ❑ Has numerous adjustable parameters that influence the outcome ▪ Ongoing work: use the collected data to implement auto-adjustment of the parameters to match human perception