Rebuilding the SEO stack: From crawl budget to AI-assisted decision making

In a world where AI Overviews are disrupting the SERPs and traditional metrics are losing relevance, how should SEOs adapt? François Goube, CEO & Founder of Oncrawl, explores how AI-driven interfaces like ChatGPT Search are redefining how visibility, clicks, and relevance are measured.

This insightful session walks through the evolution of SEO from crawl budget management to AI-assisted analysis, revealing how server logs and bot behavior (like OAI-SearchBot and ChatGPT-user) are the new keys to tracking indexation and relevance in the age of AI search. You’ll learn how to analyze traffic quality, identify “relevant” content through OpenAI crawlers, and build a modern SEO stack that responds to changing visibility rules.

Transcript

1. Rebuilding the SEO stack: From crawl budget to AI- assisted decision making F r a n c o i s G o u b e , C E O & F o u n d e r, O n c r a w l 2 4 M AY 2 0 2 5
2. ▪ CEO & Founder @Oncrawl A little intro @FrancoisGoube Technical SEO data for competitive websites Helping you analyze, quantify and increase ROI from SEO
3. ▪ CEO & Founder @Oncrawl ▪ Weird passion about SEO and Patents A little intro @FrancoisGoube
4. ▪ CEO & Founder @Oncrawl ▪ Weird passion about SEO and Patents ▪ Investor A little intro @FrancoisGoube
5. What shou ld we do i n an SEO Lan dscape re defi ne d by AI? Identify the SEO challenge
6. How big is the challenge? ▪ AI Overview is taking more space
8. Google impressions have surged nearly 50% since AI Overviews And clicks are continuously dropping down
9. Is it the end of an era? 0 100 200 300 400 500 600 700 0 10000 20000 30000 40000 50000 60000 2/2 1/25 2/2 2/25 2/2 3/25 2/2 4/25 2/2 5/25 2/2 6/25 2/2 7/25 2/2 8/25 3/1 / 2 5 3/2 / 2 5 3/3 / 2 5 3/4 / 2 5 3/5 / 2 5 3/6 / 2 5 3/7 / 2 5 3/8 / 2 5 3/9 / 2 5 3/1 0/25 3/1 1/25 3/1 2/25 3/1 3/25 3/1 4/25 3/1 5/25 3/1 6/25 3/1 7/25 3/1 8/25 3/1 9/25 3/2 0/25 3/2 1/25 3/2 2/25 3/2 3/25 3/2 4/25 3/2 5/25 3/2 6/25 3/2 7/25 3/2 8/25 3/2 9/25 3/3 0/25 3/3 1/25 4/1 / 2 5 4/2 / 2 5 4/3 / 2 5 4/4 / 2 5 4/5 / 2 5 4/6 / 2 5 4/7 / 2 5 4/8 / 2 5 4/9 / 2 5 4/1 0/25 4/1 1/25 4/1 2/25 4/1 3/25 4/1 4/25 4/1 5/25 4/1 6/25 4/1 7/25 4/1 8/25 4/1 9/25 4/2 0/25 4/2 1/25 4/2 2/25 4/2 3/25 4/2 4/25 4/2 5/25 4/2 6/25 4/2 7/25 4/2 8/25 4/2 9/25 4/3 0/25 5/1 / 2 5 5/2 / 2 5 5/3 / 2 5 5/4 / 2 5 5/5 / 2 5 5/6 / 2 5 5/7 / 2 5 5/8 / 2 5 5/9 / 2 5 5/1 0/25 5/1 1/25 5/1 2/25 5/1 3/25 5/1 4/25 5/1 5/25 5/1 6/25 5/1 7/25 5/1 8/25 5/1 9/25 5/2 0/25 Impressions vs Clics Impressions Clics
10. But there is un untold story…
11. While AI is boosting on-page engagement, it’s also keeping users inside Google’s walls
12. Measure the quality of traffic 0.00% 2.00% 4.00% 6.00% 8.00% 10.00% 12.00% 14.00% 16.00% 0 100 200 300 400 500 600 700 2/21/25 2/23/25 2/25/25 2/27/25 3/1/25 3/3/25 3/5/25 3/7/25 3/9/25 3/11/25 3/13/25 3/15/25 3/17/25 3/19/25 3/21/25 3/23/25 3/25/25 3/27/25 3/29/25 3/31/25 4/2/25 4/4/25 4/6/25 4/8/25 4/10/25 4/12/25 4/14/25 4/16/25 4/18/25 4/20/25 4/22/25 4/24/25 4/26/25 4/28/25 4/30/25 5/2/25 5/4/25 5/6/25 5/8/25 5/10/25 5/12/25 5/14/25 5/16/25 5/18/25 5/20/25 Clicks vs Conversion Rate AVG Conversion Rate Clics
13. If you’re measuring success the old way, you may already be behind.
14. Change Your SEO Approach
15. So Does your SEO Stack
16. AI Search has not fully evolved yet
17. From Prompt to Click
18. Ranking in ChatGPT Search
19. Measuring and monitoring your SEO efforts on ChatGPT Search is challenging: • There's no equivalent of Google Search Console for ChatGPT Search • Traditional SEO metrics are not available or meaningless: ◦ No keyword search volume ◦ No average position, impressions, clicks and CTR ◦ No state of indexation The challenge of reliable data
20. What SEOs usually do when they don’t know?
21. Log analysis might be the best source of insights for indexation and visibility tracking of your website in ChatGPT Search.
22. GPTBot OAI-SearchBot ChatGPT-User OpenAI is using different bots/crawlers for different tasks and features: Crawling webpages for model training. ChatGPT “offline” ChatGPT Search Crawling webpages to answer a user query in real-time. Crawling webpages to improve Search features asynchronously. Meet OpenAI’s crawlers
23. You can allow or disallow OpenAI’s crawlers case-by-case. To have your website appear in ChatGPT Search results, you have to allow both OAI-Searchbot and ChatGPT-user in the robots.txt. #Allow AI Search ChatGPT-User: OAI-Searchbot: Allow: / You can disallow OpenAI’s training bot to prevent ChatGPT’s “offline” models from being trained on your content but still appear in ChatGPT Search results. #Disallow AI Training ChatGPT: Disallow: / OpenAI’s bots and robots.txt
24. OAI-SearchBot ChatGPT-User ChatGPT-user and OAI-SearchBot are the key bots to monitor. ChatGPT Search system relies on them: • ChatGPT-user crawls webpages to answer a user in real-time. • OAI-SearchBot crawls webpages to “index” and improve the search process. Useful to track visibility OpenAI’s AI search crawlers Useful to track indexation
25. How log analysis works? Every time a bot or a user visits a page on your website, it leaves a new line in your server log files. Each line contains the following information used for log analysis in SEO: ▪ URL visited ▪ Date and time ▪ User-agent (to identify the botname) ▪ Referrer (to identify the source of the user visit) ▪ Status code of the URL
26. GPTBot user-agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot OAI-SearchBot user-agent: OAI-SearchBot/1.0; +https://openai.com/searchbot ChatGPT-user user-agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot How to identify OpenAI’s crawlers
27. ChatGPT clicks / referral traffic ▪ You can also use server logs to track referral traffic from ChatGPT ▪ When a user clicks on a link on chatgpt.com pointing to your website, your server log will register a new line with : ▪ Visited URL ▪ Status code ▪ Date and time ▪ Referer value: “https://chatgpt.com” (or “https://chat.openai.com”)
28. Metrics that matters for AI bots tracking ▪ Key metrics ▪ Volume of bot hits ▪ Volume of unique pages crawled ▪ Crawl Frequency ▪ Volume of referral traffic ▪ Volume of unique pages with referral traffic
29. Compare OpenAI’s bots behavior
30. Compare OpenAI’s bots behavior
31. Identify accessibility issues
32. What types of pages are crawled by bots?
33. What types of pages are generating clicks?
34. How can you use this data ? ▪ Make sure ChatGPT can access “useful” pages ▪ Identify pages crawled by ChatGPT that are not accessible ▪ Identify which type of content ChatGPT is crawling or not crawling ▪ Identify which type of content ChatGPT is crawling the most ▪ Identify the last time a page was crawled by ChatGPT
35. Key takeaways ▪ ChatGPT-user and OAI-SearchBot are the key bots to monitor. ▪ You can block GPTBot but still appear in ChatGPT Search results. ▪ You can use ChatGPT-user visits as a key visibility metric. ▪ You can track OAI-SearchBot to prevent indexation issues ▪ Traditional technical SEO indexability / accessibility still matters.
36. Relevance is ChatGPT’s “ranking factor” “Relevant” is the wording used by OpenAI’s documentation and support team define web content that is used to answer a prompt with Search.
37. ChatGPT Search conversation json file You can access the json file of a conversation with ChatGPT Search activated to find the full list of links : 1.Login to chatgpt.com and write a prompt with Search activated 2.Right click inspect page 3.Open Network tab 4.Reload page 5.Find json file based url id
38. ▪ search_result: Search results urls gathered by ChatGPT for the prompt. ▪ sources_footnote: Main citations urls used directly in the answer. ▪ supporting_websites: Additional citations urls available on hover when multiple citations are used. ▪ safe_urls: Urls that were verified as safe for the user to visit. They are included to provide users with relevant and trustworthy resources. ▪ blocked_urls: Urls ChatGPT is not permitted to access or include in responses for safety, content, or policy reasons. Types of URLs available in the json
39. How to identify relevant urls
40. Relevant URLs are the URLs getting the highest crawl frequency from OAI-SearchBot
41. Study what defines “relevant” content 1. Test prompts and parse the json to pull the URLs list 2. Use your favorite crawler to extract: ▪ Title and description ▪ Hn structure ▪ Content ▪ Structured data ▪ Ngrams 3. Compare and analyze: ▪ Content gap ▪ Unique content ▪ Content freshness (Last update) Identify the common Pattern
42. Use ChatGPT to build ”Intent analysis” ▪ Crawl your website ▪ Extract ▪ GSC Keyword Data ▪ H1, H2, H3 ▪ Titles ▪ Upload and prompt:
43. Wanna go back to the Future of SEO?
44. Use Next-Gen Tools
45. Unlock your SEO Potential
46. Thank you! Please Follow @Oncrawl E a r l y Access O n c r aw l N e x t