Retrievable Beats Rankable: Build for Retrieval-Based AI

This was so much fun to do at AthenSEO (or as I have been calling it, AthensSEO). I covered how technical SEO is essential (if not critical) for ranking. Whether we look at speed, spiderability, or other forms of optimisation, the essentials of TechSEO are so critical for ranking well in AI that without them, you likely won't appear in the responses AI makes. Add to that the fact that models can and do use search results, this means good optimisation for AI is good SEO.
This session covered a bit about how AI works and how there is not storage of facts, what AI needs, llms.txt, and more. It all boiled down to technical SEO in the end.

Transcript

1. #AthensSEO Judith Lewis Founder, Decabbit Consultancy Retrievable Beats Rankable: Build for Retrieval-Based AI decabbit@gmail.com
2. #AthensSEO
3. #AthensSEO Expert trainer, speaker, fractional CMO, MC and compère Run DeCabbit Consultancy specialising in global digital marketing consultancy Non-Executive Director (NED) of the Academy of Chocolate Technical Lead of the WBA think tank in Switzerland working on AI in viticulture Contact: decabbit@gmail.com Who I Am & What I Do
4. #AthensSEO I have CHOCOLATE for questions relevant to the session so remember to ask!
5. #AthensSEO
6. #AthensSEO
7. #AthensSEO The company running a version of an LLM (like Claude) releases a bot images by ChatGPT
8. #AthensSEO The LLM ingests content including webpages images by ChatGPT
9. #AthensSEO The LLM chunks and tokenises the content images by ChatGPT
10. #AthensSEO Content is no longer recognisable images by ChatGPT
11. #AthensSEO Tokens placed within vectors images by ChatGPT
12. #AthensSEO Entities and their relationships built images by ChatGPT
13. #AthensSEO LLMs do not store facts the way we assume they do images by ChatGPT
14. #AthensSEO Factual recall in LLMs is additive rather than driven by one single “fact neuron” https://www.themoonlight.io/en/review/summing-up-the-facts-additive-mechanisms-behind-factual-recall-in-llms
15. #AthensSEO It treats facts as triples like (subject, relation, attribute) https://www.themoonlight.io/en/review/summing-up-the-facts-additive-mechanisms-behind-factual-recall-in-llms
16. #AthensSEO The right answer appears because these mechanisms reinforce one another https://www.themoonlight.io/en/review/summing-up-the-facts-additive-mechanisms-behind-factual-recall-in-llms
17. #AthensSEO LLMs combine several partial clues until one answer stands out https://www.themoonlight.io/en/review/summing-up-the-facts-additive-mechanisms-behind-factual-recall-in-llms
18. #AthensSEO
19. #AthensSEO image by ChatGPT
20. #AthensSEO
21. #AthensSEO What does the machine REALLY want?
22. #AthensSEO Building Brand Trust Through AI Readiness Generally on Site https://www.aleydasolis.com/en/ai-search/ai-search-winning-brands-characteristics/
23. #AthensSEO Aleyda’s 10 points specifically for AI content ingestion for positive results https://www.aleydasolis.com/en/ai-search/ai-search-optimization-checklist/
24. #AthensSEO So just SEO?
25. #AthensSEO Rank well on Google, surface well in AI
26. #AthensSEO This is why content structure is important https://brandwell.ai/blog/seo-content-template/
27. #AthensSEO Single page, single topic
28. #AthensSEO It helps to have answers to popular questions already within content, structured to make it easy to understand and deliver in a result https://alsoasked.com/
29. #AthensSEO FAQs are really for PAA (and people) https://alsoasked.com/
30. #AthensSEO Be the answer
31. #AthensSEO On every platform https://chatgpt.com/
32. #AthensSEO
33. #AthensSEO Optimise content for crawlability Are you using a CDN and if so, has it auto-blocked AI bots or are they allowed? (This is a known issue) Avoid Client Side Rendered JavaScript – AI cannot see it. Check that none of the pages are noindexing via meta robots any of your valuable content which should be surfaced. Avoid using a “nosnippet” rule which will prevent the content from being used as a direct input for AI Overviews and AI Mode (covered recently by MS at a conference – interesting news) Optimise internal linking so make sure if there is a relevant associated page or article on your site, you link to it in other articles.
34. #AthensSEO llms.txt https://www.linkedin.com/posts/crystal-carter-digital_llmstxt-is-for-agents-yall- thats-activity-7462528887948836864-XyxI
35. #AthensSEO llms.txt https://developer.chrome.com/docs/l ighthouse/agentic-browsing/llms-txt
36. #AthensSEO llms.txt Google search say they don’t use it but… agents are different… Agentic llms.txt is different Tells agents *how to do specific things* on your website Points to specific tools and APIs Points to pages for completing tasks like making a booking, filling out a form, or *purchasing a product* There are many use cases for llms.txt Wix websites have agentic llms.txt – check Crystal Carters posts on this for more information!
37. #AthensSEO https://www.geeksforgeeks.org/blogs/what-is-technical-seo/
38. #AthensSEO Teaching the AI: Topic Breadth and Depth AI search breaks complex queries into multiple related searches This behaviour rewards sites with detailed content pieces (but note examples are about topics likely to have general search volume). For example, a detailed article about clonal selection relating to the plantings of pinot noir a winery chose to plant in its vineyard. Or an article about the history of Gyros and how it came to be one of the most popular foods. Stick to the same topic though so depth on topic, not breadth. Don’t cover all different vines and their parentage in a single article unless it is your pillar article lining out to detailed ones. Or an article about the history of Gyros start with the assumed point of invention
39. #AthensSEO Teaching: Topic Breadth and Depth Create pillar (hub) pages, main broad topics, and then cluster pages around specific facets: Pillar (broad/general) pages should summarize each topic element, covering key sections briefly, with links to deeper cluster pages. Cluster (indepth) pages should target specific facets, that should be covered in-depth, ensuring unique purpose and query intent for each page. Cross-link between cluster pages where relevant and back to the hub, as a central resource, establishing semantic relationships across content, and helping AI understand the full context and span connections between topics.
40. #AthensSEO
41. #AthensSEO Optimise for “Chunk” Storage & Retrieval CHUNKS ARE WHAT ARE DONE TO TEXT, not what we do to them! AI search engines don’t index or retrieve whole pages They break content into chunks as part of their ingestion process. You should optimise each section/paragraph like a standalone snippet (for PAA, and other SEO reasons). Don’t rely on AI using the whole page for context, each segment should be independently understandable but still work within the whole… for PEOPLE and SEO not just AI One idea per section: keep each passage tightly focused on a single concept. Use structured, accessible, and well-formatted HTML with clear subheadings (H2/H3) for every subtopic.
42. #AthensSEO Be the Answer from Fan-out & RAG AI search engines put together multiple results from its analysis from different sources into a single (usually) coherent response. This means your content must be easy to extract and logically structured to fit into a multi-source answer. Summarise complex ideas clearly, then expand. Start answers with a direct, concise sentence. Favor plain, factual, non-promotional tone. Use Structured Data to help search engines and by extension AI better understand your page (LLMs don’t use schema themselves – it’s more complicated than that). Use natural language Q&A format.
43. #AthensSEO Be Citation Worthy LLMs will cite content when it’s perceived as factually accurate, up- to-date, well-structured, and authoritative. Your content must meet higher trust and clarity criteria to be included in a result with a link within an LLM (or be outside certain training data which does not get a link). Use specific, up-to-date, verifiable claims, fact-based statements, not vague generalities. Include source citations (link to studies, stats, or experts) when you include facts and figures (or specify these are from your data/info). Show authorship and/or credentials for EEAT signals such as “certified tour guide for 10 years in Athens” or “viticulturalist graduated from Geisenheim 2018 but working on Alpha Estate since 2015”. Use author, organization, and other structured data to really dominate search results outside LLMs.
44. #AthensSEO Content Authoritativeness Signals Authority increases the likelihood that your content will be included in AI-generated answers. Use clear authority signals such as expert bylines, structured data (dev help), external citations (links), and mentions on reputable sites Optimise your brand presence across web platforms, including social channels, in a consistent way, linking back to your main site, engaging with your community, answering reviews, etc. Publish original research, reports, or data studies, conduct surveys, compile unique datasets, or run industry studies. Promote them to journalists and bloggers who create content roundups. Secure coverage in industry and expert publications, contribute quotes or guest content to respected newsletters and blogs in your industry.
45. #AthensSEO So it’s just SEO
46. #AthensSEO Ensure Multi-Modal Elements On-Page AI search systems are increasingly retrieving and synthesizing multimodal content, including images, charts, tables, videos, to better answer user queries. This then creates opportunities to provide more useful, scannable and engaging answers for users. Ensure images and videos have well optimised names. Images should use descriptive alt text that includes topic context. Add captions to images and videos with explanation right below or beside the visual.
47. #AthensSEO
48. #AthensSEO You can win brand recognition, help people learn about your brand (and win at TikTok) https://medium.com/better-marketing/how-i-create-a-video-first-content- strategy-cb304837d7b9
49. #AthensSEO So now we need some frameworks
50. #AthensSEO https://claude.com/resources/tutorials
51. #AthensSEO And you don’t always need to have a framework you made (Note: This is created with a history of Hotel Chocolat analysis so it already has URLs and other information in memory like my requests for a priority checklist and other requests regarding this website)
52. #AthensSEO This enables us to quickly populate a complex audit https://tinyurl.com/AuditFrame-0505
53. #AthensSEO And even help us make decisions!
54. #AthensSEO But we need to make sure we ask the LLM the right things
55. #AthensSEO Or what we get back will be worse than bad…
56. #AthensSEO We can use prompt engineering to improve the output
57. #AthensSEO How to create a framework Structure your prompts using a systematic approach, such as the C.R.I.S.P.Y. framework, to ensure the AI has all necessary information Context: Set the background and provide relevant information. Role: Define the persona or perspective the AI/LLM should adopt (e.g., "Act as an expert data scientist"). Instructions: Clearly state the action required. Specifics: Add constraints (e.g. "keep it under 200 words" "use bullet points"). Parameters: Specify tone, format, or style. Yielding: Describe the expected output or result.
58. #AthensSEO Lightweight evaluation method Create a prompt library as you work Ensure you are logging everything, including failures in responses to improve your prompting Once you have a lot of personas, scenarios, and data, versioning is essential (remember to clear memory and start a new chat) Then finally evaluation of the results by a human is critical Basically… spreadsheet everything with dates and times and be the words most boring person…
59. #AthensSEO https://claude.com/resources/tutorials
60. #AthensSEO Other frameworks Chain of Thought (CoT): Instruct the model to "think step by step" to increase logical accuracy. Ask Before Answering: Instruct the AI to ask clarifying questions before providing a final answer to reduce ambiguity. Persona Assignment: Assigning a specific role instantly alters the depth, perspective, and tone of the response, yielding better, more tailored results. Define Output Format: Clearly state if you want a table, JSON, code block, or a summary, which reduces post- processing work.
61. #AthensSEO Ways to improve output Iterative Generation/Evaluation: Use a "Draft-Evaluate- Refine" system where the AI generates an initial response, self-evaluates against criteria, and improves it. Meta Control Framework (MCF): When stakes are high, break down the goal into smaller, manageable sub-tasks and create specific, testable components for each. Reality-Anchored Framework (RAF): For enterprise use, instruct the AI to use specific data sources, documents, or external references to prevent hallucinations.
62. #AthensSEO And finally… Define Metrics: Evaluate answers based on accuracy, relevance, coherence, and helpfulness. Human in the Loop (HILCS): For critical tasks, ensure a human reviews the AI's output to own the risk and make final decisions. Continuous Improvement: Standardise successful prompts and create a "system prompt" to avoid repeating instructions.
63. #AthensSEO AEO and GEO are SEO in a red dress
64. #AthensSEO
65. #AthensSEO Remember… even ChatGPT uses search results https://www.searchenginejournal.com/ chatgpt-appears-to-use-google-search- as-a-fallback/552089/
66. #AthensSEO https://www.nngroup.com/articles/ai-changing- search-behaviors/ Google is a hard habit to break – and now pushes to AI
67. #AthensSEO Which means optimising for Google
68. #AthensSEO Because the AIO pulls from search results
69. #AthensSEO As does ChatGPT
70. #AthensSEO How people search will continue to evolve
71. #AthensSEO Good SEO is all you really need.
72. #AthensSEO
73. #AthensSEO Ask questions
74. #AthensSEO You Can Find Me Online Bluesky/X: @JudithLewis Threads/Insta: @Decabbit LinkedIn: https://uk.linkedin.com/in/judithlewis Web: https://www.decabbit.com Food Blog: https://MostlyAboutChocolate.com Slides: https://speakerdeck.com/decabbit https://www.slideshare.net/DeCabbit/