Masterclass: Advanced Technical SEO - BrightonSEO UK October 2026

1h20 to dive into Technical SEO in the age of AI: what do AI bots change in the ways we audit websites, common pitfalls to avoid, best practices and tips from 15+ years of real world experience working with large websites.

Masterclass during BrightonSEO UK October 2026.

Companion video might come later, stay tuned!

Transcript

1. Masterclass: Advanced Technical SEO Julien Deneuville Databulle @diije
2. Hello, world • Reims, Champagne 🍾 • 15+ years of XP • Freelancer since 2017 • Tech SEO / Data / Python • Bluesky : @dii.je • Blog : databulle.com/blog • Bluesky : speakerdeck.com/diije @diije #brightonSEO
3. This morning - Some theory - Some examples - Please ask questions - Then we can get some lunch @diije #brightonSEO
4. Today’s playground
5. @diije #brightonSEO
6. SEO test ground ~1M SEO visits per year ~2M Googlebot hits per month ~200k indexed pages (1.2M known) @diije #brightonSEO
7. First things first
8. SEO, GEO? @diije #brightonSEO
9. Same old sh*t! @diije #brightonSEO
10. Are you sure? @diije #brightonSEO
11. Technical SEO @diije #brightonSEO
12. What do we want? @diije #brightonSEO
13. Make Googlebot’s job faster @diije #brightonSEO
14. Make robots job faster @diije #brightonSEO
15. Share of traffic from AI robots over the past 12 months Source: https:///radar.cloudflare.com/ai-insights @diije #brightonSEO
16. How many crawl hits for one human visit? Source: https:///radar.cloudflare.com/ai-insights @diije #brightonSEO
17. Types of content served Source: https:///radar.cloudflare.com/ai-insights @diije #brightonSEO
18. Datadome, Cloudflare & co Which bots are allowed or not? Can they really access the website? Is the allowlist up to date? @diije #brightonSEO
19. UA & IP to allow Owner User-agents IPs Amazon Amazonbot (indexing) https:///developer.amazon.com/amazonbot/ Anthropic Claude-SearchBot (search), Claude-User (user), ClaudeBot (training) https:///platform.claude.com/docs/en/api/ip-addresses Apple Applebot (indexing), Applebot-Extended (training) https:///search.developer.apple.com/applebot.json Bing (Microsoft) Bingbot (indexing) https:///wwwww.bing.com/toolbox/bingbot.json CommonCrawl CCBot (indexing) https:///index.commoncrawl.org/ccbot.json Google Googlebot (indexing), Gemini-User (user), Google-Extended (training) GoogleOther (divers) https:///wwwww.gstatic.com/ipranges/goog.json (the most exhaustive) Meta FacebookBot (divers), Meta-ExternalAgent (training) AS32934 OpenAI ChatGPT-User (user), GPTBot (training), OAI-SearchBot (search) https:///developers.openai.com/api/docs/bots Perplexity Perplexity-User (user), PerplexityBot (indexing) https:///docs.perplexity.ai/docs/resources/perplexitycrawlers @diije #brightonSEO
20. Tricksters @diije #brightonSEO
21. UA Spoofing on ourecycler.fr @diije #brightonSEO
22. @diije What's to see? à robots.txt Indexability à Server conf Crawlability Accessibility Who's in or out? What's indexable? à Meta robots #brightonSEO
23. AI bot = black box @diije #brightonSEO
24. Not all crawlers are equal Source : think.resoneo.com/sentinel/geo-llm-crawler-report.html @diije #brightonSEO
25. Let’s get to work!
26. Step 0 : prepare data @diije #brightonSEO
27. Let’s crawl! @diije #brightonSEO
28. How do I do that? @diije #brightonSEO
29. Get prepared Whitelist (IP, HTTP header, …) Scope (subdomains, subfolders, …) Scrap (number of products, dates, …) @diije #brightonSEO
30. Protip : test run Test your setup: • A few hundred pages / a few minutes • Goal: make sure your setup works before launching your « real » crawl @diije #brightonSEO
31. Check the volumes In GSC : 219k indexed pages ✅ Ideally, also check the real inventory @diije #brightonSEO
32. Step 1 : fix wrong links @diije #brightonSEO
33. Where do my links go? @diije #brightonSEO
34. Avoid redirects Waste of time for everybody Virtually increase the nb of URLs à Link directly to the destination ! @diije #brightonSEO
35. Don’t do redirects Update your internal links acordingly! @diije #brightonSEO
36. Avoid errors Fix broken links Define a strategy for expired content @diije #brightonSEO
37. @diije #brightonSEO
38. 404, 410? @diije #brightonSEO
39. Protip: create your own error Does the server answer with the appropriate response code? $ curl -I https://ourecycler.fr/qsdf HTTP/2 404 @diije #brightonSEO
40. Soft 404 $ curl -I https://ourecycler.fr/materiaux/piles/44230/st-sebastien-loire HTTP/2 200 @diije #brightonSEO
41. Noindex pages Waste of time for robots à Avoid links to noindex pages à Consider blocking with robots.txt @diije #brightonSEO
42. Triple directives On the same page: <meta name="robots" content="index, follow"/> … <meta name="robots" content="noindex"/> … <meta name="robots" content="index, follow, max-videopreview:-1, max-image-preview:large, max-snippet:-1"/> @diije #brightonSEO
43. Be consistent! @diije #brightonSEO
44. Empty pages @diije #brightonSEO
45. Step 2 : analyze @diije #brightonSEO
46. PageRank @diije #brightonSEO
47. PageRank is correlated to traffic @diije #brightonSEO
48. Calculate PageRank Your crawler already does Link Score in Screaming Frog • InRank in Oncrawl • … • Otherwise do it yourself • @diije databulle.com/blog/code/python-pagerank-benchmark.html #brightonSEO
49. Best linked pages @diije #brightonSEO
50. Leaks! @diije #brightonSEO
51. Compare values @diije #brightonSEO
52. Split the inventory Strategic pages Mid-tier Long tail Garbage @diije #brightonSEO
53. Heatmaps: display data over multiple segments Example: % of active pages per category and per year @diije #brightonSEO
54. Data Logs Traffic Sales … @diije #brightonSEO
55. Find the truth in log files @diije #brightonSEO
56. Anatomy of a logfile line 83.114.51.220 - - [11/Apr/2016:06:51:45 +0200] "GET /guide/creer-une-entreprise-de-transport-toutes-lesetapes/ HTTP/1.1" 200 32054 "https://www.google.fr/" "Mozilla/5.0 (iPad; CPU OS 9_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) GSA/13.1.72140 Mobile/13E237 Safari/600.1.4” IP Response code Date Response weight Method Referer URL User agent @diije #brightonSEO
57. It’s a mess $ wc -l access.log 2798 $ wc -l ourecycler.access.log 425806 $ wc -l ourecycler.custom.log 425909 @diije #brightonSEO
58. Compare volumes (again) $ zcat ourecycler.custom.log-2026-09-14.gz | grep ooglebot | grep 66.249 | wc –l Compare over a few days with GSC : Settings > Crawl stats You can also check user traffic. @diije #brightonSEO
59. Example: tracking issue @diije #brightonSEO
60. Never trust data @diije #brightonSEO
61. Don’t do this at home In the Nginx config: location = /robots.txt { access_log off; } location = /sitemap.xml { access_log off; } Not a good idea. @diije #brightonSEO
62. Protip : script awk $ nano parse.awk #!/usr/bin/awk -f BEGIN { FPAT = "([^ ]+)|(\"[^\"]+\")" } { print $1, $4, $5, $6, $7, $8, $9 } $ chmod +x parse.awk $ ./parse.awk /path/to/access.log > output.log @diije #brightonSEO
63. What now? Before Now $ jupyter notebook $ claude @diije #brightonSEO
64. A few tips It’s all about context: client, targets, datasources, … Get your projects organized: scripts, data, output, … Split: small projects, small tasks Plan first, explore afterwards @diije #brightonSEO
65. Fun stuff @diije #brightonSEO
66. Check for false bots! OuRecycler.fr: 3% false hits @diije #brightonSEO
67. Where do they come from? @diije #brightonSEO
68. Who do they tell you they are? @diije #brightonSEO
69. Server errors Only one issue on July 21st: @diije #brightonSEO
70. Are you sure? Server errors by hour and weekday, excluding July 21st incident à What’s happening every Tuesday at noon? @diije #brightonSEO
71. CCBot: can it access your website? Response codes: @diije #brightonSEO
72. CCBot: what does it actually see? Hits per page type: @diije #brightonSEO
73. About Brave https:///safe.search.brave.com/help/brave-search-crawler @diije #brightonSEO
74. BraveBot isn’t who you think Protip: a good idea for a honeypot to find bad bots ;-) @diije #brightonSEO
75. Who crawls what? % of indexable pages crawled at least once @diije #brightonSEO
76. How often? Median time between 2 hits on same page @diije #brightonSEO
77. Impact of page depth @diije #brightonSEO
78. Impact of PageRank @diije #brightonSEO
79. Orfan pages Googlebot is the most curious @diije #brightonSEO
80. Types of 404 errors @diije #brightonSEO
81. Llms.txt @diije #brightonSEO
82. Llms.txt @diije #brightonSEO
83. Hits on XML sitemaps Most AI bots don’t care… But do they know these URLs? @diije #brightonSEO
84. Pagination @diije #brightonSEO
85. Webperfs: are your strategic pages actually fast? @diije #brightonSEO
86. JavaScript rendering @diije #brightonSEO
87. % of rendered pages @diije #brightonSEO
88. Delay between HTML crawl and JavaScript rendering @diije #brightonSEO
89. Keep It Simple, Stupid https:///motherfuckingwebsite.com/ @diije #brightonSEO
90. Protip: No-JS browser session Create a session where JavaScript is disabled by default. Example in Google Chrome: Settings – Privacy and security > Site settings @diije #brightonSEO
91. What about traffic? @diije #brightonSEO
92. Which page types generate traffic? @diije #brightonSEO
93. Which pages generate traffic? @diije #brightonSEO
94. @diije #brightonSEO
95. Landing error rate @diije #brightonSEO
96. Where do these errors come from? @diije #brightonSEO
97. Step 3 : optimize @diije #brightonSEO
98. Overview @diije #brightonSEO
99. Prioritize! @diije #brightonSEO
100. About link obfuscation Example: 166,648 /signaler URLs Example: on every location page @diije #brightonSEO
101. Learn more Slides Replay @diije #brightonSEO
102. We improve only what we measure @diije #brightonSEO
103. Follow up Did the crawl metrics improve? Did the rankings / traffic improve? Did the user experience KPIs improve? @diije #brightonSEO
104. @diije #brightonSEO
105. In a nutshell #0 Back to basics #1 Test it yourself #2 Split #3 Rinse & repeat @diije #brightonSEO
106. Keep in touch! @diije / databulle.com @diije #brightonSEO