Serge Bezborodov - Evergreen Crawl: See Your Site Exactly as Googlebot Saw It

1. The crawl you don't need to run Serge Bezborodov ✦ Tech SEO Connect
3. Technical SEO Google Search Console? Meta tags? Indexation? Structured data? Logs? Redirect chains? 03
5. Serge Bezborodov 2008 Built my first online shop 2011 Started in SEO 10 years JetOctopus - a crawler for big websites, especially e-commerce 2025 EdgeComet - delivering fixes, not just reports ✦ Google's patents are a must-read 05
7. There are tons of crawlers 07
10. Someone has to run it It takes time What's wrong with a crawl It goes out of date fast Big websites are hard to crawl - partial data It puts extra load on the server 10
11. 1 1
13. Googlebot and AI bots are not SEO crawlers 13
16. SEO crawler Homepage Depth 1 Depth 2 Depth 3 An SEO crawler finds pages through links, moving from the homepage deeper into the site. It simply follows the website structure. 16
17. Googlebot Crawled by Googlebot Pages with demand Pages with content Some trash Uncrawled Googlebot is interested in valuable pages that serve demand and have content - and it is very selective about trashy and low-value pages. 17
18. Googlebot vs crawl 18
19. Googlebot vs crawl 19
22. The ability to look at a website the way Googlebot and AI bots see it 22
23. You don't need to run it It's always available No additional server load A true time machine 23
25. Boring theory part How does the internet work? Request Client Server Response 25
26. Request GET https://edgecomet.com/ User-Agent: MyFancyCrawler/1.0 Client Server 26
27. Response HTTP/2 200 content-type: text/html Client Server HTML page 27
28. The whole idea: as they are served to bots 28
29. Response HTTP/2 200 content-type: text/html Client HTTP/2 200 content-type: text/html Middleman HTML page Server HTML page 29
30. Response HTTP/2 200 content-type: text/html Client HTTP/2 200 content-type: text/html Middleman HTML page Server HTML page Reads the page, parses and analyzes it 30
31. Response HTTP/2 200 content-type: text/html Client HTTP/2 200 content-type: text/html Middleman HTML page Server HTML page Reads the page, parses and analyzes it Database 31
32. Claude + the database Claude Database Claude works well with the database and can do the job natively. 32
35. Cloudflare Worker HTML page Cloudflare Worker reads the page D1 Database 35
36. Cloudflare Worker HTML page Cloudflare Worker reads the page Comet Trail App D1 Database Claude Custom dashboards 36
37. Comet Trail App 37
38. Comet Trail App 38
39. Comet Trail App github.com/EdgeComet/comet-trail 39
41. Evergreen Crawl Artificial Crawl Built from real bot traffic A crawler you run yourself You don't need to run it VS Test new pages before deploy It's always available Staging crawls No additional server load It's convenient A true time machine Finds orphan and deep pages See the website as Googlebot does No need to wait for bots 41
42. They work together Staging Artificial Crawl Evergreen Crawl Test pages before they ship See what bots get, every day Deploy Live site Artificial Crawl checks a page before release. Evergreen Crawl watches it after. 42
43. Evergreen Crawl Artificial Crawl A daily driver M T Every day W T F S S + Big trips M T W T F S S When the job calls for it 43
45. Serge Bezborodov Tech SEO Connect ✦ edgecomet.com LinkedIn