From Search Results to Insights

GenAI services have been adopted successfully in no time across various digital business models, but what if your data has the better answers? How could this innovative technology be combined with a companies knowledge and data?

In this talk, we delve into the intricacies of Large Language Models (LLMs) and their augmentation with custom data through the use of Retrieval-Augmented Generation (RAG). Learn about Statista’s pioneering journey in moving from extensive search results to concise and well-founded answers, using their LLM-based application, ResearchAI. We will tackle the challenges faced, including building a skilled team for such an emerging technology, the impact of exclusive data sources on answer quality, high product costs and latency per request, and the tendency of LLMs to produce hallucinations despite the availability of accurate data. This session offers a realistic look at the hurdles encountered, and the strategies employed, providing valuable lessons on building and optimizing RAG applications in the real world.

Transcript

1. From Search Results to Insights Learnings from Statista’s GenerativeAI Journey
2. Matthias Lau #machinelearning #developer #founder Ingo Schellhammer #biztech #cto Bene Stemmildt #socio-technical arch #networker #cto #velominatus
4. #131 (majestic million) 23m (content views/m) 500k (content downloads/m) >1m (statistics) 70k (reports)
5. 66-80%
6. 2/2023: Explore 1 Engineer 4/2023: Prototype 7/2023: Develop +PM +External 11/2023: Invest Full team 5/2024: Launch
7. Optimization Process
8. Retrieval Reranking Answering Rating Query Answer Answering Workflow.
9. Retrieval Reranking Answering Rating Query Answer Answering Workflow. 40 1 1
11. Retrieval Reranking Answering Rating Query Answer Answering Workflow. 44% 5% 49% 3% - 13% 78% 9%
12. Latency Costs Quality 1 2 3 < 30s < 5c Metrics Relevance.
13. Reference Data.
14. Test Runner.
15. 27.58 8.1c 30%
16. Many Experiments.
17. -10% -65% +140% 24.7 2.8c 72%
18. 1. Add Traceability 2. Define your Metrics Relevance 3. Create a Reference Dataset 4. Measure a Baseline 5. Experiment and Measure Delta Recap Optimization Playbook.
19. Technical Learnings
20. Retrieval Query
21. Retrieval Query Rewrite Query
22. Retrieval Query Rewrite Query How tall is the Eiffel Tower? It looked so high when I was there last year? What is the height of the Eiffel Tower?
23. Retrieval Query Rewrite Query Variants Retrieval Retrieval Reranking
24. Retrieval Query Rewrite Query Variants Retrieval Retrieval Reranking Which company had more revenue 2015 to 2020, Apple or Microsoft? Apple revenue from 2015 to 2020. Microsoft revenue from 2015 to 2020. Microsoft and Apple revenue comparison from 2015 to 2020.
25. Retrieval HyDE Query Retrieve with an answer, not a question
26. Retrieval HyDE Query Which company had more revenue 2015 to 2020, Apple or Microsoft? Between 2015 and 2020, Apple consistently had higher revenue than Microsoft. Apple’s revenue grew from approximately $233.7 billion in 2015 to $274.5 billion in 2020, while Microsoft’s revenue increased from about $93.6 billion in 2015 to $143 billion in 2020.
27. Query Retrieval HyDE Retrieval Rewrite Query Variants Retrieval Retrieval Reranking
28. -10% -65% +140% 24.7 2.8c 72%
29. Recap Optimization Workflow. What about the Model?
35. There’s no One-Fits-All. Query with good Retrievals Llama 3.1 8B 👍 Query with the need for complex conclusions Llama 3.1 8B 👎
36. Business Learnings
37. From prototype…
38. to product… Pages/session Research AI: 13 keyword search: 9 Bounce rate 16,9% Requests Aug: 11k Nov: 29k
39. 24.7 -10% 2.8c -65% 72% +140% 14.6 -41% 1.6c -43% 72% +/- 0% POST HEUREKA IMPROVEMENTS TODAY (Sep. ‘24) 27.6 8.1c 30% START (Sep. ‘23)
40. Meeting our quality ambition in a high-growth game 3,1 bn (+9%) 275 mn (-4%) 72 mn (+16%) 29 k (+11%) Quality Monthly visits
41. On micro-level: Go upstream where the real traffic is….
42. Customer On macro level: Where is the hottest (LLM) party? Research AI API LLM
43. Customer Research AI API LLM On macro level: Where is the hottest (LLM) party?
44. Customer Research AI API LLM On macro level: Where is the hottest (LLM) party?
45. From Project to Product 💻 ⏱ 📅 → ☁ 🔄 💎
48. 👋