T3DD26: From RAGs to Riches

AI assistants are only as smart as the content they were trained with — which may or (more likely) may not include the content managed in your CMS. Retrieval-Augmented Generation (RAG) changes that by making your content available to any AI model at query time, without having to retrain or fine-tune any models.

This talk breaks down how RAG pipelines actually work: how documents get chunked and embedded into vector representations, how semantic search retrieves the right content, and how to make that knowledge available to AI assistants. We’ll then explore several different strategies to build such a pipeline for the content managed in your TYPO3 CMS.

Transcript

1. S G A R M FRO S E H C I R TO d l a w t t i m @ H C I M L E H N I T MAR
2. N I T R A M H C I M L HE mittwald Chief Technology Evangelist Head of Developer Relations TYPO3 Association Board Member PHWT Lecturer, Software Engineering & Cloud Computing
3. NEVER HAVE I EVER
4. F T W NO SUCH THING G N O R W
5. — „LET’S JUST TRAIN OUR OWN AI MODEL YOUR MANAGER, PROBABLY Eli Missing https://unsplash.com/photos/an-ostrich-looking-at-the-camera-with-trees-in-the-background-C6TmvLdcdtQ
6. FRONTIER MODEL TRAINING
7. FINE TUNING
8. THE PITFALLS OF FINE TUNING YOUR OWN AI MODELS
9. heym ittwald my serverz kinda slow could u take a look k thx bye Bob Hello Daniel! No worries! You’ve had a stuck MySQL query running on your server. We’ve restarted it, should be all good now. THE PITFALLS OF FINE TUNING YOUR OWN AI MODELS Cheerio, Your mittwald support team
10. Dearest mittwald team, I’m writing to inform you that my last invoice looked a bit odd. Might I inquire that you have a look? Yours sincerely, Mary Hello Daniel! THE PITFALLS OF No worries! You’ve had a stuck MySQL query running on your server. We’ve restarted it, should be all good now. YOUR OWN AI MODELS Cheerio, Your mittwald support team FINE TUNING
11. THE PITFALLS OF FINE TUNING YOUR OWN AI MODELS OVERFITTING TRAINING COSTS INFERENCE COSTS CONTINUOUS RETRAINING
12. BUT FIRST, THE BASICS
13. QUERY RESPONSE LLM
14. MUCH BETTER QUERY RESPONSE LLM RELEVANT CONTENT CONTENT REPOSITORY
15. that cute rodent from south america No results found. :( SEARCH
16. that cute rodent from south america Did you mean Capybara? SEARCH
17. PRECISION & RECALL ( ( : ) ( : ) ( Jurafsky & Martin 2026 "Speech and Language Processing" 3rd ed. draft) https://web.stanford.edu/~jurafsky/slp3/ Rijsbergen 1979 "Information Retrieval" 2nd ed)
18. PRECISION = & RECALL Relevant found documents Found documents "Are the found documents relevant?"
19. PRECISION Relevant found documents = Relevant documents & RECALL "Are the relevant documents found?"
20. RODENT + SOUTH AMERICA + CUDDLY = CAPYBARA THE MAGIC OF EMBEDDINGS Anna Roberts https://unsplash.com/photos/a-close-up-of-a-capybara-laying-in-the-grass-near-a-body-of-g8OQxw5ZwyY
21. CANADIAN BEAVER THE MAGIC OF EMBEDDINGS CAPYBARA GEOGRAPHY DENSE VECTOR REPRESENTATION ES S EURASIAN BEAVER TE N CU PENGUINS LAND ANIMAL
22. THE MAGIC OF EMBEDDINGS CANADIAN BEAVER CAPYBARA short vector space distance = high conceptual similarity
23. THE MAGIC OF EMBEDDINGS CANADIAN BEAVER CAPYBARA GERMAN TAX LAW long vector space distance = no conceptual similarity
24. THE MAGIC OF EMBEDDINGS $client = OpenAI factory() withApiKey('sk') withBaseUri('https: llm.aihosting.mittwald.de/v1') make(); $response = $client embeddings() create([ 'model' 'Qwen3-Embedding-8B', 'input' 'cuddly rodent from south america', ]); > - > embeddings[0] - / / > - . . > . - : : > > = = > > > - - - $embedding = $response embedding;
25. $client = OpenAI $response = $client > / . . . . > var_dump($embedding); array(4096) { } / / THE MAGIC OF EMBEDDINGS
26. MUCH BETTER QUERY RESPONSE LLM RELEVANT CONTENT CONTENT REPOSITORY
27. MUCH BETTER QUERY RESPONSE LLM EMBEDDING MODEL RELEVANT CONTENT CONTENT REPOSITORY VECTOR DATABASE
28. var_dump($embedding); array(4096) { } DIMENSIONALITY MATTERS Chunks, Vector Store, Search queries need the same dimensionality. WORKAROUND Some models Qwen3 Embedding-8B have been trained with Matryoshka Representation Learning, which allows you to reduce dimensionality afterwards. https://developer.mittwald.de/docs/v2/platform/aihosting/models/qwen3-embedding-8b/ ) . . - . ( : ) ( Kusupati et al. 2024 "Matryoshka Representation Learning" 36th Conference on Neural Information Processing Systems https://arxiv.org/abs/2205.13147 / / THE MAGIC OF EMBEDDINGS
29. RERANKING $client = new GuzzleHttp\Client(); $documents = [ ' ']; $response = $client post('https: llm.aihosting.mittwald.de/v1/rerank', [ 'headers' ['Authorization' 'Bearer sk'], 'json' [ 'model' 'Qwen3-VL-Reranker-2B', 'query' 'Can I keep a Capybara at home?', 'documents' $documents, ], 'timeout' 30, ]); $data = json_decode($response getBody() getContents(), true); > = < . . . > - / / > = > - > = > - . . > . = > > > > = = = = > = Sort results by relevance score, descending $results = $data['results']; usort($results, fn($a, $b) $b['relevance_score'] / / FOR MORE PRECISION: $a['relevance_score']);
30. FOR MORE PRECISION: RERANKING MUCH BETTER QUERY RESPONSE LLM EMBEDDING MODEL RELEVANT CONTENT CONTENT REPOSITORY VECTOR DATABASE
31. FOR MORE PRECISION: RERANKING MUCH BETTER QUERY RESPONSE LLM EMBEDDING MODEL CONTENT REPOSITORY VECTOR DATABASE RELEVANT CONTENT RERANKING MODEL RANKED RELEVANT CONTENT
32. FOR MORE PRECISION: RERANKING VECTOR DATABASE RELEVANT CONTENT HIGH RECALL LOW PRECISION RERANKING MODEL RANKED RELEVANT CONTENT HIGH RECALL HIGH PRECISION
33. CHAT WITH YOUR CONTENT Difficulty: EASY* * Well, kind of
34. PART 1: SEARCH PART 2: CHAT
35. PART 1: SEARCH DON'T REINVENT THE WHEEL martin@box $ composer require 'apache-solr-for-typo3/solr:14.0.0-RC1'
36. martin@box $ composer require 'apache-solr-for-typo3/solr:14.0.0-RC1' martin@box $ docker compose up # mw stack deploy compose.yml: services: solr: image: solr:10 ports: ["8983:8983"] environment: SOLR_OPTS: > -Dsolr.vector.dimension=4096 -Dsolr.max.booleanClauses=8192 volumes: solr-data:/var/solr/data This depends on the embedding model that you intend to use. Changing this requires a complete reindex!
37. Repeat this for every core! PUT /solr/core_en/schema/text-to-vector-model-store HTTP/1.1 Content-Type: application/json BUG! To work around #4650, re-register the same model as language-models { "class": "dev.langchain4j.model.openai.OpenAiEmbeddingModel", "name": "llm", "params": { "baseUrl": "https: llm.aihosting.mittwald.de/v1", "apiKey": "sk", "modelName": "Qwen3-Embedding-8B", "timeout": 5, The dimensionality of this model "logRequests": true, must match the solr.vector.dimension property "logResponses": true, "maxRetries": 2 } REGISTERING EMBEDDING MODELS IN SOLR / / . . . }
38. setup.typoscript plugin.tx_solr.search.query { type = 1 vectorSearch { minimumSimilarity = 0.75 topK = 1000 } } # 1 = pure vector (KNN) search # minimum cosine similarity (0 Minimum cosine distance ∈ 0;1 ENABLING VECTOR SEARCH . . ] https://docs.typo3.org/p/apache-solr-for-typo3/solr/main/en-us/Configuration/Reference/TxSolrSearch.html#query-type [ 1)
39. PART 1: SEARCH DON'T REINVENT THE WHEEL VARIANT B martin@box $ composer require \ lochmueller/seal-ai \ symfony/ai-generic-platform \ symfony/ai-redis-store
40. THE AGONY OF CHOICE
41. PART 1: SEARCH PART 2: CHAT
42. PART 2: CHAT SELECT DIFFICULTY DON'T REINVENT THE WHEEL ROLL YOUR OWN
43. PART 2: CHAT DON'T REINVENT THE WHEEL martin@box $ docker compose up # mw stack deploy
44. martin@box $ docker compose up compose.yml: . . . services: n8n: image: n8nio/n8n:latest ports: ["5678:5678"] environment: # volumes: n8n-data:/home/node/.n8n # mw stack deploy
45. O M E D ME I T Beyza Kaplan https://www.pexels.com/photo/juggler-in-hat-performing-11629564/
48. PART 2: CHAT DON'T REINVENT THE WHEEL
49. PART 2: CHAT REINVENT THE WHEEL . . create('sk- . : : $platform = PlatformFactory ');
50. PART 2: CHAT REINVENT THE WHEEL $platform = PlatformFactory $toolbox = new Toolbox([new TYPO3SearchTool(rootPageId: 1)]); $processor = new AgentProcessor($toolbox); . $agent = new Agent( $platform, 'Qwen3.6-35B-A3B-FP8', inputProcessors: [$processor], outputProcessors: [$processor], );
51. $platform = PlatformFactory $toolbox = new Toolbox([new TYPO3SearchTool(rootPageId: 1)]); $processor = new AgentProcessor($toolbox); $agent = new Agent( $platform, 'Qwen3.6-35B-A3B-FP8', inputProcessors: [$processor], outputProcessors: [$processor], ); $messages = new MessageBag( Message forSystem('Answer only from search_typo3 results. Cite the url.'), Message ofUser($userMessage), ); . > call($messages) - > - : : : : return (string) $agent getContent();
52. O M E D ME I T Rick Hadley https://www.pexels.com/photo/fire-performer-juggling-flaming-staff-at-night-36393905/
56. USER What clothing should I bring on a hike to the Camino? ASSISTANT GROUND TRUTH - T‑shirts or shirts 2 3, quick‑dry fabric - Lightweight pants or shorts 2 pairs - Long‑sleeve shirt 1 (for sun protection or cooler evenings) - Lightweight fleece or jacket 1 - Underwear 1 2 quick‑dry sets - Hat or cap 1 for sun protection - Buff or bandana – useful for sweat, dust, or extra sun shade - Waterproof jacket or poncho – essential, as rain is common, especially in Galicia - T-shirts or shirts: 2 3, quick-dry fabric - Lightweight pants or shorts: 2 pairs - Long-sleeve shirt: 1, for sun protection or chilly evenings - Lightweight fleece or jacket: 1 - Underwear: 1 2 sets, quick-dry - Hat or cap: 1, for sun protection - Waterproof jacket or poncho: essential — rain is common, especially in Galicia - Buff or bandana: useful for sweat, dust, or extra sun protection Thinking... For the Camino you should pack only lightweight, quick‑dry clothing and avoid cotton: All items should be made of quick‑dry materials and kept to a minimum to keep your pack light. – – – – – – > – – – – > Source: “Packing List” page – [/packing-list] Pack light and stick to quick-dry fabrics — cotton traps moisture and is best avoided on the trail:
57. USER What clothing should I bring on a hike to the Camino? ASSISTANT GROUND TRUTH For the Camino you should pack only lightweight, quick‑dry clothing and avoid cotton: Pack light and stick to quick-dry fabrics — cotton traps moisture and is best avoided on the trail: - T‑shirts or shirts 2 3, quick‑dry fabric - Lightweight pants or shorts 2 pairs - Long‑sleeve shirt 1 (for sun protection or cooler evenings) - Lightweight fleece or jacket 1 - Underwear 1 2 quick‑dry sets - Hat or cap 1 for sun protection - Buff or bandana – useful for sweat, dust, or extra sun shade - Waterproof jacket or poncho – essential, as rain is common, especially in Galicia - T-shirts or shirts: 2 3, quick-dry fabric - Lightweight pants or shorts: 2 pairs - Long-sleeve shirt: 1, for sun protection or chilly evenings - Lightweight fleece or jacket: 1 - Underwear: 1 2 sets, quick-dry - Hat or cap: 1, for sun protection - Waterproof jacket or poncho: essential — rain is common, especially in Galicia - Buff or bandana: useful for sweat, dust, or extra sun protection Precision ≅ 0,53 ≅ 0,74 Recall ≅ 0,61 F1 All items should be made of quick‑dry materials and kept to a minimum to keep your pack light. PRECISION, RECALL, F-MEASURE Source: “Packing List” page – [/packing-list] ( ( – – – : ) – > – – ( – : ) – – – ( > Jurafsky & Martin 2026 "Speech and Language Processing" 3rd ed. draft) https://web.stanford.edu/~jurafsky/slp3/ Rijsbergen 1979 "Information Retrieval" 2nd ed)
58. Please act as an impartial judge and evaluate the quality of the response provided by an AI assistant to the user question displayed below. Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of the response. Begin your evaluation by providing a short explanation. Be as objective as possible. After providing your explanation, please rate the response on a scale of 1 to 10 by strictly following this format: "[[rating]]", for example: "Rating: 5 ". Question] {question} The Start of Assistant’s Answer] {answer} The End of Assistant’s Answer] LLM-AS-A-JUDGE ] ] [ [ ) ( - : ) > Zheng et al 2023 "Judging LLM-as-a-Judge with MT Bench and Chatbot Arena", 37th Conference on Neural Information Processing Systems NeurIPS 2023 Track on Datasets and Benchmarks. https://arxiv.org/pdf/2306.05685 ( [ [ [ SYSTEM
59. LLM OBSERVABILITY TOOL RECOMMENDATIONS LANGSMITH or GRAFANA AGENT O11Y SaaS only; easy to use; no PHP instrumentation LANGFUSE Open-Source; easy setup, but resource-intensive; no PHP instrumentation OpenTelemetry-based) ( ( TRULENS Open-Source; lean, but needs integration; no PHP instrumentation OpenTelemetry-based)
60. MARTIN HELMICH m.helmich@mittwald.de martin.helmich@typo3.org WE’RE HIRING Senior Kubernetes Engineer Tech-Lead Kubernetes (all genders, on-site or remote) https://www.mittwald.de/karriere