When your data warehouse talks back: How conversational analytics changes the analytics game.

Transcript

1. When your data warehouse talks back How conversational analytics changes the analytics game, Mark Schep, Big Data Expo, 10 September 2026 Mark Your Data B.V.
2. Over mij Mark Schep Mark Schep | Data & AI Engineer | Aquarel hobbyist Less then two years ago, when my daughter was born, I picked up watercolor painting. Today, in a time where the whole world goes a little AI-crazy, it's nice to have an offline hobby to relax and persist some memories that puts a smile on your face. Role Data & AI Engineer Focus Driving value with data & AI Location Utrecht, Nederland Customers Bol.com, Winparts, Snelstart, Tikkie, Heineken
3. Me: "What was the total revenue in 2023?" Conversational analytics Agent: SELECT SUM(total_amount) FROM fct_orders WHERE order_year_num = 2023 AND is_net_sale = TRUE; → € 338,361.59 Me: Thanks! Now I don’t have to ask Pete our analyst who is already busy! Mark Your Data B.V.
4. How does conversational analytics change our work and the way we do business? And … Toelichting To answer this question, I combined my professional, with my hobby craft: Can we trust an AI data analyst? I simulated a crafts e-commerce business, from orders to shipments, to help me understand the implications of conversational analytics on my role as Data & AI engineer, and adjecent analytics and business roles. Today I share what I’ve learned
5. Can we trust an AI data analyst? SITUATION Good data analytics means fast, reliable, consistent insight: pipelines, models, a warehouse, dashboards for business users, direct SQL for analysts, all built on a solid foundation and real domain knowledge. COMPLICATION What if everyone had their own AI data analyst, turning any plain-language question straight into SQL? Can we trust it, or will it confidently send us in the wrong direction with SQL that runs but misses the nuance an analyst wouldn't have missed? ANSWER Yes, and here is the shape of it. 1. Conversational analytics is already good enough to be a valuable new tool. 2. It will change how business people, analysts and data engineers work with data. 3. You can start today, as long as you focus and get the foundation right first.
6. 1 Conversational analytics is already good enough to be a valuable new tool. A realistic test of what conversational analytics can already do, where it quietly breaks, and how you can fix it.
7. Three things I found 1 2 3 Conversational Analytics works • Real questions, a real database, correct SQL back But it can be confidently wrong • Hidden traps in ordinary business data Simple tricks makes it reliable • No new tooling, just writing down what a good analyst knows
8. Conversational Analytics works I built a realistic test: a simulated online store selling paint and paper, three years of sales, shipping and customer data spread across separate systems, the way real company data usually is. I asked it a plain question and it answered straight from the live database. Me: "What was the total revenue in 2023?" Agent: SELECT SUM(total_amount) FROM fct_orders WHERE order_year_num = 2023 AND is_net_sale = TRUE; → € 338,361.59 Me: Woa, thats correct!
9. But it can be confidently wrong Without guidance With guidance SUM(signed_amount) on the payment log → € 331,044.22. Cash basis: the date money moved. SUM(total_amount) on the orders table → € 338,361.59. Order basis: the date the sale was placed. Neither number is wrong, but only one matches what finance expects.
10. A simple fix makes it reliable Toelichting From ±90% to 100% accuracy on 15 real business questions The fix wasn't more tooling. It was writing down what a good analyst already knows: Using a simple skill (text file): • A short, plain-language note about the domain's gotchas, sitting right next to the data model.
11. 2 It changes how we work For the people who ask the questions, the people who already knew how to ask them, and the people who used to control every answer.
12. Three roles, three changes 1 2 Business: your jobs gets easier • Quick lookups get an instant answer Analysts: your jobs gets easier • For existing data models, you will be faster, which saves time for new or more complex things. 3 Engineers: your job is getting harder! • You now have many more users, and you don’t know in advance how they will ask their questions to your agent! Also the agents LLM might change overnight
13. For simple questions, I get answers faster via my analytics agent Business: fewer questions stuck in the queue Analysts: faster access to the deep data Roughly half of analyst time typically goes to ad hoc questions, split between quick lookups and real strategic work. A conversational layer takes the quick lookups off that queue: an instant answer instead of waiting on Slack. An analyst who already knows how to query directly doesn't lose anything. They gain faster access to more granular data in the warehouse, for the harder questions a dashboard was never built to answer. A shortcut, not a replacement. For simple questions, I get answers faster via my analytics agent For simple questions, I get answers faster via my analytics agent
14. Data engineering roles become harder The old platform The conversational agent Same source in, same transformations, same dashboard out. Ask the same question twice, get the same answer. You control every step. Many more people, asking in their own words. The model behind it can change overnight. "Sales" already means something different to finance, marketing and logistics. Today, the agent might just pick one, confidently. Ask the same question twice? Now someone has to check. I only need to worry about my data pipelines, and data models, which are totally in my control. And a handful of analysts using them. I now need to worry about how more and less data savvy users ask questions to my agent. Which is not in my control.
15. 3 Start today, but focus The good news: if your foundation is solid, you're closer to ready than you think. Here's where to begin, and where not to.
16. Foundation & Focus First 1 2 3 Foundation first 1. Data pipelines 2. Data models 3. Agents One domain at a time • Pick a goal, go through the whole chain, then iterate Privacy by design • Decide what the agent may see, before it can see anything
17. Foundation first: Part I – Get the right tools Collect data from source systems in one central place
18. Foundation first: Part II – Model your data Model your data into easy to use, understandable reusable “blocks”
19. Foundation first: Part III – Build your agent Build your agent on top of your data models The agent is last!
20. One domain at a time Solve a small problem Don’t just implement an agent Without focus, you’ll have to check data models, semantic layers, evaluation sets, and a way to big group of users to be able to deliver reliable conversational analytics. Start small!
21. Privacy by design You only get to do this once, properly You cannot just feed all your data to an AI. Decide what it may see: ideally only aggregated, anonymous data. Confirm it isn't stored or used to train a vendor's model. Check, and if needed update, your privacy statement. Build this in line with GDPR from day one. To start simple, limit access to only aggregated, non PII data. This should be sufficient for most business users. Data analysts might want more granular data access. But configure data access in a way that you comply with privacy laws and compliance departments.
22. The shape of it, once more Three things you should remember
23. My takeaways 1 Conversational Analytics works • The technology is ready to become adopted • You should preventing common pitfalls, Me: "What was the total revenue in 2023?" which does not need to be hard 2 3 It changes how we work • Business and analyst roles become easier • Data engineering roles become harder Start today, but focus • Start with the foundation • Implement privacy by design • Rollout one use case / domain at a time Agent: SELECT SUM(total_amount) FROM fct_orders WHERE order_year_num = 2023 AND is_net_sale = TRUE; → € 338,361.59 Me: Woa, thats correct!
24. Thank you! Lets connect on LinkedIn ->
25. + Appendix: the artifacts Four real pieces from the project, if you want to see them: how it's organized, what the skill knows, how the skill behaves, and the evaluation set.
26. How the project is organized dlt → dbt → skill files → CLI Nothing exotic: the same stack (dlt, dbt, DuckDB) I'd set up for a client, plus two markdown files and a small Python eval script. data-ingestion/ four dlt pipelines, one per source system data-modelling/ dbt project models/staging/ rename + cast, per source models/marts/ dims, facts, lifecycle models skill-craft-analytics.md <- domain map + gotchas skill-craft-analysis-process.md <- step-by-step process src/craft_demo/ the CLI: question in, SQL + answer out evals.py <- the 15-question evaluation set
27. The skill, part 1: what it knows skill-craft-analytics.md No new infrastructure. Just the two sentences a good analyst would already say out loud, written down once instead of repeated in every conversation. is_net_sale, not status != 'returned'. An order can be cancelled (never fulfilled) or returned (fulfilled then reversed). Both are non-revenue, but a filter that only excludes 'returned' still counts cancelled orders. signed_amount, not amount. fct_payment_transactions is a ledger: amount is an unsigned magnitude for every row. SUM(amount) adds refunds and charges together and overstates revenue. Use SUM(signed_amount).
28. The skill, part 2: how it behaves skill-craft-analysis-process.md Two files make up “the skill”: one says what to look up, the other says how to behave, ask once if unclear, check its own work, and say plainly how sure it is. Clarify if genuinely ambiguous. Ask one specific question, once, only if truly needed. Apply the documented filters before finalizing the SQL. A silently wrong filter is worse than a query that visibly fails. Self-review before answering. Right date column? Right filter? Right join keys? Does the result actually match what was asked? Report source and confidence, honestly. e.g. "fct_payment_transactions (raw SQL), confidence: medium"
29. The evaluation set: a sample of the 15 questions evals.py, scored with and without the skill file Each question has a hand-verified answer, run against the live database. Score with the skill file loaded, score again without it, compare. That's the whole method behind the 87–93% → 100% number in the main talk. No cherry-picking: every question and its expected answer is in the repo. "What was total revenue in 2023?" → 338,361.59 "Which product category has the highest return rate?" → Accessories "How many customers churned in 2024?" → 111 "Sum all the amounts in payment transactions for 2023" → 331,044.22