Tweet
Tweet

Slide 1

Slide 1 text

@badams #FOS22 (Almost) Everything You Need To Know About Crawling, Indexing, and Rendering in Google Barry Adams June 2022

Slide 2

Slide 2 text

@badams #FOS22 What does Google do?

Slide 3

Slide 3 text

@badams #FOS22 Crawler Indexer Ranker Google Processes

Slide 4

Slide 4 text

@badams #FOS22 Crawler Indexer Ranker 1. Crawler (Googlebot)

Slide 5

Slide 5 text

@badams #FOS22 @badams #FOS22 Crawling: Discovery

Slide 6

Slide 6 text

@badams #FOS22 Crawling: Queue Management URL Deduplication

Slide 7

Slide 7 text

@badams #FOS22 Crawling: Queue Management Prioritisation & Scheduling

Slide 8

Slide 8 text

@badams #FOS22 @badams #FOS22 Crawling: Fetch & Parse

Slide 9

Slide 9 text

@badams #FOS22 @badams #FOS22 Crawl Politeness

Slide 10

Slide 10 text

@badams #FOS22 @badams #FOS22 Optimise Crawling • Server Response Time

Slide 11

Slide 11 text

@badams #FOS22 @badams #FOS22 GSC Crawl Stats

Slide 12

Slide 12 text

@badams #FOS22 @badams #FOS22 Page Resource Load

Slide 13

Slide 13 text

@badams #FOS22 @badams #FOS22 Googlebot & AdsBot

Slide 14

Slide 14 text

@badams #FOS22 @badams #FOS22 Optimise Crawling • Serve correct HTTP status codes; ➢ 200 OK ➢ 301 / 302 Redirects ➢ 304 Not Modified ➢ 401 / 403 Permission Issues ➢ 404 / 410 Not Found/Gone ➢ 5xx Error

Slide 15

Slide 15 text

@badams #FOS22 @badams #FOS22 Optimise Crawling • ALL resources consume crawl budget; ➢ Not just HTML pages ➢ Reduce HTTP requests per page

Slide 16

Slide 16 text

@badams #FOS22 @badams #FOS22 Optimise Crawling • ALL resources consume crawl budget; ➢ Not just HTML pages ➢ Reduce HTTP requests per page • AdsBot can consume crawl budget; ➢ Double-check your Google Ads campaigns

Slide 17

Slide 17 text

@badams #FOS22 @badams #FOS22 Optimise Crawling • ALL resources consume crawl budget; ➢ Not just HTML pages ➢ Reduce HTTP requests per page • AdsBot can consume crawl budget; ➢ Double-check your Google Ads campaigns • Link equity (PageRank) impacts crawl budget; ➢ More link equity = more crawl budget

Slide 18

Slide 18 text

@badams #FOS22 2. Indexer Crawler Indexer Ranker

Slide 19

Slide 19 text

@badams #FOS22 @badams #FOS22 Two Stages* of Indexing Crawler Indexer Ranker 1 2 *At least – indexing is a collection of interconnected processes

Slide 20

Slide 20 text

@badams #FOS22 @badams #FOS22 Indexing: HTML Lexer & Tokenizer

Slide 21

Slide 21 text

@badams #FOS22 @badams #FOS22 Indexing: Selection

Slide 22

Slide 22 text

@badams #FOS22 @badams #FOS22 Indexing: HTML Source

Slide 23

Slide 23 text

@badams #FOS22 @badams #FOS22 Indexing: Rendering

Slide 24

Slide 24 text

@badams #FOS22 Indexing: Index Integrity Deduplication & Canonicalisation

Slide 25

Slide 25 text

@badams #FOS22 @badams #FOS22 Rendering

Slide 26

Slide 26 text

@badams #FOS22 @badams #FOS22 Evergreen Chrome

Slide 27

Slide 27 text

@badams #FOS22 @badams #FOS22 What happens during Rendering in your Browser? HTML CSS HTML Parser CSS Parser DOM Tree CSSOM Render Tree Painting Display Layout

Slide 28

Slide 28 text

@badams #FOS22 @badams #FOS22 JavaScript HTML CSS HTML Parser CSS Parser DOM Tree CSSOM Render Tree Painting Display JavaScript Layout

Slide 29

Slide 29 text

@badams #FOS22 @badams #FOS22 JavaScript… HTML CSS HTML Parser CSS Parser DOM Tree CSSOM Render Tree Painting Display JavaScript Layout JavaScript

Slide 30

Slide 30 text

@badams #FOS22 @badams #FOS22 JavaScript… HTML CSS HTML Parser CSS Parser DOM Tree CSSOM Render Tree Painting Display JavaScript Layout JavaScript JavaScript

Slide 31

Slide 31 text

@badams #FOS22 @badams #FOS22 JavaScript… HTML CSS HTML Parser CSS Parser DOM Tree CSSOM Render Tree Painting Display JavaScript Layout JavaScript JavaScript JavaScript

Slide 32

Slide 32 text

@badams #FOS22 @badams #FOS22 Google’s Rendering as part of Indexing HTML CSS HTML Parser CSS Parser DOM Tree CSSOM Render Tree Painting Display JavaScript Layout JavaScript JavaScript

Slide 33

Slide 33 text

@badams #FOS22 @badams #FOS22 Google does not perform actions

Slide 34

Slide 34 text

@badams #FOS22 Why Rendering?

Slide 35

Slide 35 text

@badams #FOS22 @badams #FOS22 Raw HTML:

Slide 36

Slide 36 text

@badams #FOS22 @badams #FOS22 Rendered DOM:

Slide 37

Slide 37 text

@badams #FOS22 @badams #FOS22 Rendering allows Google to… • … load all meta data, content, and links on a webpage • … understand the page’s layout and content hierarchy • … evaluate the usability and quality of the webpage

Slide 38

Slide 38 text

@badams #FOS22 Rendering Issues

Slide 39

Slide 39 text

@badams #FOS22 @badams #FOS22 Possible Rendering Issues in GSC

Slide 40

Slide 40 text

@badams #FOS22 @badams #FOS22 Rendering Issues • Inaccessible Resources; ➢ Make sure all page resources can be crawled

Slide 41

Slide 41 text

@badams #FOS22 @badams #FOS22 Rendering Issues • JavaScript inserts invalid HTML in the ; ➢ tags in the break Google’s processing of meta tags

Slide 42

Slide 42 text

@badams #FOS22 @badams #FOS22 Rendering Issues • JavaScript inserts invalid HTML in the ; ➢ tags in the break Google’s processing of meta tags

Slide 43

Slide 43 text

@badams #FOS22 @badams #FOS22 https://developers.google.com/search/docs/advanced/guidelines/valid-html

Slide 44

Slide 44 text

@badams #FOS22 @badams #FOS22 Rendering Issues • HTML vs Render mismatch; ➢ Different content in raw HTML vs fully rendered page

Slide 45

Slide 45 text

@badams #FOS22 @badams #FOS22 https://chrome.google.com/webstore/detail/view-rendered- source/ejgngohbdedoabanmclafpkoogegdpob

Slide 46

Slide 46 text

@badams #FOS22 @badams #FOS22 SEO Crawlers Can Also Render

Slide 47

Slide 47 text

@badams #FOS22 @badams #FOS22 Google Tools *ALWAYS* Render

Slide 48

Slide 48 text

@badams #FOS22 @badams #FOS22 Optimise Rendering • Don’t rely on Google’s rendering; ➢ Use SSR & CDN caching • Minimise page weight; ➢ Fewer page resources = better use of crawl budget faster load speed & CWV less chance of rendering issues • Optimise your HTML source; ➢ Think about where tags exist and what they do when their code is executed

Slide 49

Slide 49 text

@badams #FOS22 @badams #FOS22 Optimise Indexing • Optimise your page layouts; ➢ Prominent content & links are more valuable for users & Google • Improve internal linking; ➢ More PageRank = higher chance of indexing • Improve your content; ➢ Google has no obligation to index all your pages ➢ Make it worth Google’s while…

Slide 50

Slide 50 text

@badams #FOS22 Bypassing Rendering* with Edge SEO *sort of

Slide 51

Slide 51 text

@badams #FOS22 @badams #FOS22 Edge SEO Your Webserver Cloud CDNs Users

Slide 52

Slide 52 text

@badams #FOS22 @badams #FOS22 Edge SEO Your Webserver Cloud CDNs Googlebot

Slide 53

Slide 53 text

@badams #FOS22 @badams #FOS22 Edge SEO Your Webserver Cloud CDNs Googlebot Change your webpages here

Slide 54

Slide 54 text

@badams #FOS22 @badams #FOS22 Edge SEO • CDNs store cached versions of your webpages; ➢ Global coverage with edge nodes worldwide ➢ Usually also results in faster crawling and better CWV • You manipulate your CDN cached pages; ➢ Cloud Workers enable a range of functionality • Googlebot crawls & indexes the changed CDN-cached pages; ➢ Your ‘original’ website remains unchanged ➢ Google only sees the changed CDN webpages

Slide 55

Slide 55 text

@badams #FOS22 @badams #FOS22 Why Edge SEO? • Faster deployment; ➢ Bypass your developers’ lengthy queues ➢ ‘Ask forgiveness, not permission’ ➢ No reliance on client-side JavaScript • No CMS constraints; ➢ Change pages directly regardless of your CMS capabilities • Testing; ➢ Perform narrow tests on specific site sections ➢ A/B testing for SEO

Slide 56

Slide 56 text

@badams #FOS22 @badams #FOS22 SEO A/B Split Testing

Slide 57

Slide 57 text

@badams #FOS22 @badams #FOS22 SEO Split Testing Case Studies https://www.searchpilot.com/resources/newsletter/

Slide 58

Slide 58 text

@badams #FOS22 @badams #FOS22 Barry Adams ➢ Doing SEO since 1998 ➢ Specialist in Technical SEO & News SEO ➢ Newsletter: SEOforGoogleNews.com

Slide 59

Slide 59 text

@badams #FOS22 Thank You [email protected] @badams