The Hunt for Digital Ghosts

This presentation introduces llms.txt and MCPs and why they are needed to tame the flood of "digital ghosts and demons".

Transcript

1. The Hunt for Digital Ghosts Taming LLMs with llms.txt and MCP
2. Magical Institute for Commons and Energy (MICE)
3. The Wizards
4. Techlington
5. Techlington Website Server World Wide Web Network Protocol
7. Whisper Artifact
8. Whisper Artifact (Website)
9. Wizards’ Whispering Weave (World Wide Web)
10. Entities of the Lower Whispering Dimensions
11. Wizards’ Whispering Weave (World Wide Web)
12. Crawler Artifact The Problem (Website) (Crawler?)
13. Crawler Artifact The Problem (Website) (Crawler?)
14. Crawler The Problem (Crawler?) Artifact (Website)
15. Wizards’ Whispering Weave (World Wide Web)
16. Wizards’ Whispering Weave (World Wide Web)
17. Wizards’ Whispering Weave (World Wide Web)
18. Wizards’ Whispering Weave (World Wide Web)
19. Seeking Stone (Search Engine)
20. Seeking Stone (Search Engine)
21. Crawler(Crawler?) Artifact Solutions (Website) Crawler(Crawler?) robots.txt
22. Wizards’ Whispering Weave (World Wide Web) robots.txt robots.txt
24. The Luminous Language Mirror (LLM)
25. The Luminous Language Mirror (LLM) Large Language Model
26. A New Era
31. A New Era
32. A New Era
33. Jeremy Howard !
34. Grand Wizard Jeremiah Howardus
35. Grand Wizard Jeremiah Howardus llms.txt
36. llms.txt # Title > Optional description goes here Optional details go here ## Section name - [Link title](https://link_url): Optional link details ## Optional - [Link title](https://link_url)
37. llms.txt # FastHTML > FastHTML is a python library which brings together Starlette, Uvicorn, HTMX, and fastcore's `FT` "FastTags" into a library for creating server-rendered hypermedia applications. Important notes: - Although parts of its API are inspired by FastAPI, .... ## Docs - [FastHTML quick start](https://fastht.ml/docs/tutorials/quickstart_for_web_devs.html.md): A brief overview of many FastHTML features - [HTMX reference]( https://github.com/bigskysoftware/htmx/blob/master/www/content/reference.md): Brief description of all HTMX attributes, CSS classes, headers, events, extensions, js lib methods, and config options ## Examples - [Todo list application] (https://github.com/AnswerDotAI/fasthtml/blob/main/examples/adv_app.py): Detailed walk-thru of a ....
38. llms.txt robots.txt sitemap.xml
39. llms.txt robots.txt sitemap.xml
40. llms.txt robots.txt sitemap.xml
41. llms.txt robots.txt sitemap.xml
42. llms.txt robots.txt sitemap.xml
43. llms.txt robots.txt sitemap.xml
44. llms.txt robots.txt sitemap.xml
45. llms.txt
46. Implementing llms.txt llms.txt llms.txt llms.txt llms.txt llms.txt llms.txt llms.txt llms.txt
47. llms.txt llms.txt llms.txt llms.txt llms.txt Implementing llms.txt
48. Zoozle Feta Macrosoft The Adoption Problem
49. Zoozle Feta Macrosoft llms.txt ? llms.txt ? llms.txt ? The Adoption Problem
50. Zoozle Feta Macrosoft Some Feta? The Adoption Problem
52. Realm of Anthropia (Anthropic)
53. Mirror Conduit Protocol (Model Context Protocol) The Lore-Weaver Fairy (Client Application)
54. Mirror Conduit Protocol (Model Context Protocol) The Lore-Weaver Fairy (Client Application) The Nexus Core Fairy (Server Application)
55. Mirror Conduit Protocol (Model Context Protocol) The Lore-Weaver Fairy (Client Application) The Nexus Core Fairy (Server Application) “.....”
56. Mirror Conduit Protocol (Model Context Protocol) The Lore-Weaver Fairy (Client Application) The Nexus Core Fairy (Server Application) “.....”
57. Mirror Conduit Protocol (Model Context Protocol) The Lore-Weaver Fairy (Client Application) The Nexus Core Fairy (Server Application) “.....”
58. Mirror Conduit Protocol (Model Context Protocol) The Lore-Weaver Fairy (Client Application) The Nexus Core Fairy (Server Application) “.....”
59. Mirror Conduit Protocol (Model Context Protocol) The Lore-Weaver Fairy (Client Application) The Nexus Core Fairy (Server Application) “.....”
60. Mirror Conduit Protocol (Model Context Protocol) The Lore-Weaver Fairy (Client Application)
61. Realm of Anthropia (Anthropic)
62. Implementing MCPs
63. Implementing MCPs
64. Zoozle Feta Macrosoft The Adoption Problem
65. Zoozle Feta Macrosoft The Adoption Problem MCP MCP MCP
66. Zoozle Feta Macrosoft The Adoption Problem Some Feta?
67. The End
68. RECAP “Traditional” crawlers → handled by firewalls, IP-blocking, robots.txt New crawlers for: LLM training data on-demand, up-to-date information for LLM tools
69. RECAP New crawlers less “regulated”: come back more often look into more details more capable than old generation might neglect robots.txt
70. RECAP llms.txt as new (additional) guidance Github Repo simple markdown format gives high-level information of website content helps LLM find relevant sites Problem: Low Adaption in Industry
71. RECAP MCP as universal interface main focus on tool usage high acceptance rate in industry needs MCP server for each website
72. RECAP llms.txt MCP simple markdown high-level info for LLMs slow adoption in industry custom server universal applications high adoption in industry
73. RECAP llms.txt MCP courses: microsoft huggingface examples: https://llmstxt.site/ generator
74. Thank You!