Easy start to data prompting

Transcript

1. EASY START TO DATA PROMPTING Data Expo · 9 September 2026
2. “ “ “Can you quickly check how many parent orders DKF had last week?” “What is the quality of the customers we acquired through our activation last month?” Slack, #data Asked again every week, with a new market, a new date range
4. What we tried first, and why it failed The setup The result • Connect the model straight to BigQuery • Give people access and let them ask • No definitions, no routing, no eval • Total build time: one afternoon • Fluent SQL, wrong numbers • Invented column names • Wrong revenue definition, wrong VATs • Long consideration time • Not hallucination - inconsistency and lack of understanding
5. “ “Can you quickly check how many parent orders DKF had last week?” Slack, #data Asked again every week, with a new market, a new date range
6. THE THESIS DATA PROMPTING IS 10% PROMPT, 90% CONTEXT. * You don't train people to prompt better. You make the warehouse answerable.
7. Five things the model needs 1 2 3 4 5 Access Vocabulary Semantics Routing Trust Governed, inherited. Managed in one place What our words and abbreviations mean What our metrics mean, in writing Where to look, and in what order Proof it is right before you roll out
8. Layer 1: Access governed through Google Business user Claude + BigQuery connector Answer queries run as the user, not as a service account Google IAM PII policies No extracts Existing roles and groups. No second permission system Column-level masking stays in BigQuery No copies, no shadow warehouse, no stale data Nobody sees anything they did not already have access to before
9. Layer 2: Prompts use shorthand Parent order One-off vs renewal OTIF Documented in Slack threads, not in the schema Distinction lives in Asana task titles Ops shorthand — defined in a Drive doc DKJ / DKF eCLV I am sure you can come up with 10 for you company on the spot Two brands, four markets — never spelled out Used daily in prompts, defined nowhere queryable Almost all abbreviations, most specific terminology and inter department wording was pulled from Slack and Asana through Claude and matched against existing views and definitions.
10. TAKEAWAY YOUR ABBREVIATIONS ARE UNDOCUMENTED. * Indexing Slack, Asana and Drive costs nothing and fixes the question before it reaches the data.
11. Layer 3: Comment & Explanation over tool Metric Our definition (abbreviated) Where it lives Parent order Every new subscription started with an order vw_business_data New customer Any first order in the selected period vw_business_data Renewal The recurring order from a subscription starting from n>=2 vw_subscriptions OTIF orders with 100% on time in full shipments / orders vw_ops_otif Attributed order An order attributed to a ad, campaign channel by the MTAs attribution model vw_mta_model_…
12. “Do I need dbt for this?” What dbt gives you What we did instead • schema.yml descriptions per model and column • A rendered docs site with lineage • MetricFlow: define a metric once, query it via API • CI that fails when a description is missing • At scale, this is the right home for definitions • Definitions written straight into views • Column comments set directly in BigQuery • Routing rules as plain markdown files • Extra tooling cost: zero • If we outgrow it, the definitions port over
13. Be honest about the trade-off FINE FOR US + 30 views, small team WHAT WE GAVE UP − One person can hold the whole semantic layer in their head. + Changes land same-day We cannot see what breaks when a source column changes. − No PR cycle between a definition being wrong and being fixed. + Nothing new to learn SQL and markdown. The business can read both. No lineage graph No tests on definitions Nothing stops a definition and its comment drifting apart. − Does not scale to 300 models At that point the tooling earns its cost. We are not there.
14. THE POINT THE MODEL READS WAREHOUSE METADATA. * dbt persist_docs is one way to fill it. We filled it directly. Pick what suits your business
15. Layer 4: Routing - the waterfall 1 2 3 Curated metric views Department views Raw tables First stop. Commented, tested, one definition per metric. Second stop. Narrower scope, still governed. Fallback only. Never the entry point.
16. Layer 4: A different map per department 1 3 Marketing 2 Operations Kickbite attribution, spend, CPO, ROAS, campaign level. Raw ad platform tables out of scope. OTIF, carriers, reships, shipment volume. Fixed thresholds so the answer never drifts. Finance Business / general Revenue, margin, budget vs actual. One revenue definition, no exceptions. 4 Parent orders, new customers, renewals, tickets. The 80% of questions everyone asks.
17. TAKEAWAY ROUTE BEFORE YOU RETRIEVE. * Narrowing the search space was the single highest-leverage change we made.
18. Layer 5: Earned trust with 40 questions Evaluation on result What is the number we expect and is the answer within an acceptable range. <1% 40 Fixed questions in the eval set Evaluation on methodolgy How did we get to the number. Was the question correctly interpreted, was the right skill or information selected to tackle the problem and was the approach well executed DWH and semantics errors Does our infrastructure hold up. Some of the errors are interpretation and skill, other errors are definition drifts and DWH setup. We have invested almost as much in debt as we have in developing the skills
19. What the eval caught, and what it became Unclear definition of revenue sources (marketplace vs retail vs direct) Double counted order value from shipments Became a column comment on the revenue view Became a routing rule: never aggregate from the raw shipments table VAT hardcoded in two place 4 brand naming conventions Dynamically set & rule based Became a entry in the skill and is being resolved
20. TAKEAWAY THE EVAL STARTED AS A GRAD AND BECAME THE BACKLOG. * A right number reached the wrong way will not survive the next question.
21. Upkeep: the business maintains the context Gap spotted in Slack Knowledge base Next answer is correct one definition, one place, version-controlled Marketing context Ops context Finance context Updated by marketing Updated by ops Updated by finance [FILL: who maintains it, how often it changes, how long a change takes to land]
22. Where we are now [FILL] [FILL] [FILL] €0 Questions per week answered without the data team Time to answer Total build time Extra tooling cost Note: [FILL: measurement period]
23. Start this week Mon Tue Wed Thu Fri Check permissions Write 10 definitions Build one view One markdown file Ten eval questions In your IdP, before you connect anything Your most-asked metrics. That list is the roadmap Wrap them. Comment every single column Which source to use first, per question type Answers you already trust. Score the method too
24. Three things to take away 1 2 3 Definitions, not prompts Route before you retrieve Measure, then roll out Better prompts do not fix an undocumented warehouse. The written definition is the asset. Curated views first. Raw tables are the fallback, never the entry point. Score the answer and the method. Every failure becomes a rule.
25. Questions? Lukas Michalitsch, lukas@klukinsights.eu