Giacomo Zecchini - Rendering, Style, and Layout When Things Go Wrong

Transcript

1. Rendering, Style, and Layout When Things Go Wrong
2. ABOUT ME • Giacomo Zecchini • R&D Director at Merj • Based in Tuscany, operates globally @giacomozecchini
3. MERJ IN A NUTSHELL We are a technical SEO, AEO and data engineering partner. We shape resilient, discoverable platforms designed for good bots, search engines, and customers alike. @giacomozecchini
4. TRUSTED BY INDUSTRY GIANTS INCLUDING @giacomozecchini
5. • How rendering, style, and layout shape what search engine crawlers and AI crawlers actually see • A few edge cases where implementation mistakes can cause content mismatches, leading crawlers to hide important elements or misjudge their relevance IN THE NEXT 30 MINUTES, WE’LL EXPLORE @giacomozecchini
6. @giacomozecchini DISCLAIMER - EXPERIMENTAL CONTENT When this badge appears on a slide, the content is based on testing, personal assumption, or non-official sources. Implementation details may change without notice at any time.
7. IDEAL WORLD VS REALITY
8. In an ideal world, information would be easily retrievable, text-based, well-structured, and semantic. @giacomozecchini THE IDEAL WORLD
9. The web is a vast, dynamic landscape, constantly shifting and updating. @giacomozecchini THE REALITY
10. Information exists in many formats, websites employ numerous rendering patterns, and must support a vast array of features, and the list goes on. @giacomozecchini THE REALITY
11. The primary challenge is implementing ideal solutions within the constraints of reality. @giacomozecchini THE CHALLENGE
12. Search engines have addressed some of the problems of the modern web by rendering web pages. @giacomozecchini ALL MAJOR SEARCH ENGINES RENDER WEB PAGES
13. Excluding AI agents, most AI data crawlers skip client-side page rendering, as shown in our joint research with Vercel. @giacomozecchini AI CRAWLERS Read more: https://vercel.com/blog/the-rise-of-the-ai-crawler
14. @giacomozecchini AI DATA CRAWLERS RENDERING STATUS UPDATED (DEC 2025) Crawler Rendering Googlebot Bingbot Yandex Naver Baidu Applebot AdsBot-Google adidxbot Applebot Crawler Rendering CCBot GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-User Claude-SearchBot PerplexityBot Perplexity-User Crawler Rendering GrokBot meta-externalagent MistralAI-User Bytespider DuckAssistBot * Render capability does not imply that rendering occurs consistently on every crawl.
15. The rise of AI has forced a re-evaluation of content serving, mirroring the industry shift previously seen with Search Engines. @giacomozecchini CONTENT DELIVERY
16. @giacomozecchini THINGS ARE RAPIDLY CHANGING ChatGPT Atlas Perplexity Comet
17. @giacomozecchini NOW ABLE TO SEE FULLY RENDERED PAGE CONTENT
18. THE RENDERED PAGE CONTENT IS ACCESSIBLE FROM THE WEBSITE @giacomozecchini
19. By default, we don’t use the content you browse to train our models. If you choose to opt-in this content, you can enable “include web browsing” in your data controls settings. @giacomozecchini CHATGPT ATLAS CONTENT VISIBILITY AND TRAINING Source: https://openai.com/index/introducing-chatgpt-atlas/
20. @giacomozecchini WIRESHARK NETWORK TRAFFIC ANALYSIS
21. However, it’s good to remember that direct crawling is not the only source of data for AI tools. @giacomozecchini SOURCE OF DATA FOR AI CRAWLERS
22. LET’S TALK ABOUT RENDERING
23. @giacomozecchini SEARCH ENGINE RENDERING Read more: https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics Search engine rendering has long been a known challenge, yet we continue to uncover interesting edge cases and poor implementations.
24. The key differences between automated and user rendering stem from the lack of user interaction, the use of internal asset-caching heuristics, and the way each handles certain edge cases. @giacomozecchini SEARCH ENGINE RENDERING VS USER RENDERING
25. As an output of the rendering process, we have access to the DOM Tree, the Computed Styles, and the Layout Tree. @giacomozecchini OUTPUT OF RENDERING PROCESS
26. The Dom Tree contains the content of the page. @giacomozecchini DOM TREE
27. The Computed Styles contains the style of elements of the page. @giacomozecchini COMPUTED STYLES
28. The Layout Tree contains the dimensions and positions of each element on the page. @giacomozecchini LAYOUT TREE
29. One thing that really bugs me is how little our industry leverages the Computed Style and Layout Tree information. @giacomozecchini WHY ARE YOU NOT USING IT?
30. We’ve been discussing content “above the fold” and similar stuff for years, but very little has actually been done about it during this time (publicly, at least). @giacomozecchini ABOVE THE FOLD?!
31. I get it, there are new shiny and more interesting things around 👀 👀 👀 @giacomozecchini SHINY STUFF
32. But this is a cumulative game, and in competitive markets, every detail matters. A good platform can only be built on strong foundations. @giacomozecchini THIS IS A CUMULATIVE GAME
33. VIEWPORT DIMENSIONS
34. @giacomozecchini VIEWPORT DIMENSIONS During page rendering, search engines and AI crawlers execute automated browser instances to emulate real user behavior, including specific parameters like viewport dimensions.
35. The viewport of a browser is the visible area on the screen where the web content is displayed. @giacomozecchini WHAT DO WE MEAN BY “VIEWPORT”?
36. @giacomozecchini MAIN CRAWLERS MOST USED VIEWPORT DIMENSIONS NAME VIEWPORT (Width x Height) Google Desktop 1024 x 1024 Google Smartphone 412 x 732 Bingbot Desktop 1024 x 569 Bingbot Smartphone 320 x 568 Applebot 1920 x 1080
37. GOOGLE’S VIEWPORT EXPANSION
38. The “viewport expansion” is a technique used by Google to ensure it gets the most content possible from a page by changing the height of the viewport. @giacomozecchini GOOGLE’S VIEWPORT EXPANSION
39. When rendering a web page, Google generally initiates the browser instance with one of two specific sets of viewports, depending on the origin crawler format. @giacomozecchini VIEWPORT DIMENSIONS [1/2]
40. @giacomozecchini VIEWPORT DIMENSIONS [2/2]
41. After the initial page load, the page height is checked. If the page is taller than the initial viewport, the viewport height is updated to match the height of the page (note: this is not a scroll event!). @giacomozecchini VIEWPORT HEIGHT IS DYNAMIC [1/4]
42. @giacomozecchini VIEWPORT HEIGHT IS DYNAMIC [2/4]
43. With these new viewport dimensions, lazy-loaded content (when implemented correctly) is triggered and may add additional elements to the page. @giacomozecchini VIEWPORT HEIGHT IS DYNAMIC [3/4]
44. @giacomozecchini VIEWPORT HEIGHT IS DYNAMIC [4/4]
45. If the page is not taller than the initial viewport, the viewport height is not updated. The viewport height is resized only once. @giacomozecchini ADDITIONAL NOTES
46. Let’s remember that changing the viewport during rendering is not a common thing on the internet, that is why we have some edge cases. @giacomozecchini CHANGING VIEWPORT IS NOT A COMMON THING
47. WHEN THINGS GO WRONG
48. We can group rendering and layout issues into two main categories: • Content mismatches • Value mismatches @giacomozecchini WHEN THINGS GO WRONG
49. The crawler sees more or less content than you intended. @giacomozecchini CONTENT MISMATCHES
50. The crawler sees the content but misunderstands its position or value. @giacomozecchini VALUE MISMATCHES
51. LAZY LOADING
52. @giacomozecchini LAZY LOADING [1/4] Lazy loading is used to keep the initial page light and load additional content only if needed.
53. @giacomozecchini LAZY LOADING [2/4] As we just seen, the viewport expansion is primarily meant to get the lazy loaded content by changing the viewport size.
54. @giacomozecchini LAZY LOADING [3/4] Bots generally don’t perform actions on a web site. If your website waits for a scroll, wheel, or touchstart event to load additional content, that event may never occur.
55. @giacomozecchini LAZY LOADING [4/4] • Don’t wait for a scroll, wheel, or touchstart event to load additional content. • From a web-performance point of view, it’s generally a bad idea to attach handlers to the scroll, wheel, or touchstart events (without using passive event listeners). • The recommended solution is to use the Intersection Observer API, which is now widely supported.
56. ::before AND ::after
57. The CSS ::before and ::after pseudo element can be used to add “cosmetic” content to an element. @giacomozecchini ::before AND ::after PSEUDO ELEMENTS [1/4] Read more: https://css-tricks.com/7-practical-uses-for-the-before-and-after-pseudo-elements-in-css/
58. This content is not part of the DOM Tree and is not indexed by the crawler. @giacomozecchini ::before AND ::after PSEUDO ELEMENTS [2/4]
59. On some websites, the content placed in those pseudo elements changes the semantic meaning of the text or hides important information. @giacomozecchini ::before AND ::after PSEUDO ELEMENTS [3/4]
60. Do not use the ::before and ::after pseudo element to include important information. @giacomozecchini ::before AND ::after PSEUDO ELEMENTS [4/4]
61. HTML HEADING STYLE
62. The Computed Style of HTML heading tags (H1, H2, H3, etc) are used to determine the relative importance of the heading tags compared to each other. @giacomozecchini HTML HEADING STYLE [1/3] Mentioned: https://www.youtube.com/watch?v=65rE4mvzCX0
63. @giacomozecchini HTML HEADING STYLE [2/3] The HTML heading hierarchy often fails to align with the page’s visual semantic structure.
64. • Where possible, respect the “visual semantic” of the elements. • If you can’t do that, just remember that the most relevant heading isn’t always your H1 (especially if it’s just that small, keyword-stuffed line at the top of the page). @giacomozecchini HTML HEADING STYLE [3/3]
65. DISPLAY: NONE
66. When you use display: none on an element it is omitted from the Render Tree even if the content is still present in the DOM Tree. @giacomozecchini DISPLAY: NONE [1/4]
67. The content has no position and no dimensions, only the textual value. @giacomozecchini DISPLAY: NONE [2/4]
68. If search engines or AI crawlers factor element position into their scoring, any element rendered without coordinates or dimensions may receive a lower relevance score. @giacomozecchini DISPLAY: NONE [3/4]
69. • Display: none elements had no position and dimensions. • Element with no position might be treated as less “important”. • Using instead visibility: hidden, an element retains its dimensions and position but is simply not rendered visibly. @giacomozecchini DISPLAY: NONE [4/4]
70. INFINITE SCROLLING
71. @giacomozecchini INFINITE SCROLLING [1/8] Many websites implement infinite scrolling to dynamically append new content as the user reaches the end of the page. Mentioned: https://www.youtube.com/watch?v=XgkhNF1gDKA
72. @giacomozecchini INFINITE SCROLLING [2/8] Websites that use infinite scrolling for textual content often rely on the History API to update the URL when new content becomes fully visible.
73. @giacomozecchini INFINITE SCROLLING [3/8]
74. @giacomozecchini INFINITE SCROLLING [4/8] The issue is that even if you change the URL using the History API during rendering, Google will still use the first URL, the one it started rendering.
75. @giacomozecchini INFINITE SCROLLING [5/8] When Viewport Expansion is taken into account, during rendering of Article #1 the next article may be triggered to load and become part of the initial URL’s rendered content.
76. @giacomozecchini INFINITE SCROLLING [6/8] Some implementations replace the existing content instead of appending the new content.
77. @giacomozecchini INFINITE SCROLLING [7/8] Viewport expansion can trigger the second article to load, resulting in Article #2 being rendered on the initial URL.
78. • With infinite scrolling, you may end up showing content from two different articles under a single URL. • In certain implementations, when content is replaced instead of appended, the initial URL may even display only the content of the second loaded article. • Google’s Web Rendering Service supports the History API, but together with infinite scrolling may produce undesirable side effects. • To exclude specific bots, use the User-Agent string for identification - this can be done either server-side or client-side. @giacomozecchini INFINITE SCROLLING [8/8]
79. THE VIEWPORT HEIGHT TRAP
80. @giacomozecchini THE VIEWPORT HEIGHT TRAP [1/6] Numerous websites render the primary image at 100% of the viewport width and height. While typical of fullscreen layouts, this approach is not limited to them.
81. @giacomozecchini THE VIEWPORT HEIGHT TRAP [2/6] This is typically achieved by using the viewport height (for example, 100vh) to dynamically scale the media so that it fits the device’s display viewport. Mentioned: https://www.youtube.com/watch?v=l2E3gBDP_qY
82. @giacomozecchini THE VIEWPORT HEIGHT TRAP [3/6] See more: https://www.awwwards.com/websites/fullscreen/
83. @giacomozecchini THE VIEWPORT HEIGHT TRAP [4/6] When a bot that uses viewport expansion renders a page containing a 100vh element, the 100vh value is recomputed from the expanded viewport size, which causes that element to scale to the increased height matching the new viewport dimensions.
84. If search engines or AI crawlers use above the fold positioning into their ranking models, a 100vh hero image that pushes the primary content down the page introduces a potential problem. @giacomozecchini THE VIEWPORT HEIGHT TRAP [5/6]
85. • Using 100vh for the main image of the page means that when the viewport is resized, the element expands to match the new viewport height, which can push the main content further down the page. • Specifying a max-height preserves the fullscreen aesthetic while preventing issues with the positioning of the page’s main content. @giacomozecchini THE VIEWPORT HEIGHT TRAP [6/6]
86. THE “VH” TRAP + LAZY LOADING
87. @giacomozecchini THE “VH” TRAP + LAZY LOADING [1/2] In addition to the previous case, the use of viewport expansion together with a 100vh section may keep the lazy-loading threshold from being reached, causing the extra content to never load.
88. Specifying a max-height preserves the fullscreen aesthetic while preventing issues with the positioning of the page’s main content and lazy loaded elements. @giacomozecchini THE “VH” TRAP + LAZY LOADING [2/2]
89. *Bonus* Edge Case for AI Crawlers
90. Even AI agents with full rendering capabilities can miss portions of a page’s content when that content is revealed only through user-initiated events, such as “scroll” based lazy loading. @giacomozecchini AI AGENT
91. Rendering is more than “content”
92. Layout information can be used to verify if key content is visible and prioritised above the fold (or at least not 10.000px down the page). @giacomozecchini ADDITIONAL USE CASES (1) - ABOVE THE FOLD ANALYSIS
93. Layout information can be used to check Web Page Visual Similarity, which, while beneficial for user familiarity and navigation ease, should be balanced against the risk that excessive similarity diminishes brand differentiation and negatively impacts business outcomes. @giacomozecchini ADDITIONAL USE CASES (2) - WEB PAGE VISUAL SIMILARITY
94. Layout information can be used on a 1:1 migration, ensuring that web pages are 100% visually similar across domains helps to avoid the introduction of new rendering errors. @giacomozecchini ADDITIONAL USE CASES (3) - 1:1 WEBSITE MIGRATION
95. @giacomozecchini ADDITIONAL USE CASES (4) - IDENTIFY INTRUSIVE INTERSTITIALS Layout information can be used to identify if web pages have intrusive interstitials and dialogs that may interfere with the understanding of the content.
96. Conclusions
97. • Not all crawlers render web pages. • Even when a crawler can render a page, it doesn’t always do so and it may still miss parts of the content. • Google’s viewport expansion isn’t a standard technique and can break web pages. • Layout information and computed styles are often underestimated. @giacomozecchini CLOSING REMARKS
98. @giacomozecchini LINK TO THE DECK
99. Thanks for your time!