From Demo to Production_Building Enterprise AI Agents on Databricks.

Transcript

1. ENTERPRISE AGENTIC AI From Demo to Production Building Enterprise AI Agents on Databricks Paul Karikari Principal Engineer & Solutions Architect ,Data & AI - Databricks thedatalead.ai 60-minute session • 30 min talk + 30 minlab
2. A SHORT STORY AI agents are projected to drive up to $4.4 trillion in annual global business value and automate 15% to 50% of business tasks by 2027 From Demo to Production • Enterprise Agents on Databricks - Paul Karikari 2 / 22
3. A SHORT STORY 95% of enterprise AI pilots fail to deliver measurable ROI. From Demo to Production • Enterprise Agents on Databricks - Paul Karikari 2 / 22
4. A SHORT STORY McDonald's + IBM drive-thru AI McDonald's and IBM partnered in 2021 on an automated voice-ordering system, deployed across more than 100 US drive-thrus. Today: why most enterprise agent projects don't fail in production. they fail before production, because they were never the right tool for the job. From Demo to Production • Enterprise Agents on Databricks - Paul Karikari 2 / 22
5. A SHORT STORY McDonald's + IBM drive-thru AI From Demo to Production • Enterprise Agents on Databricks - Paul Karikari 2 / 22
6. A SHORT STORY McDonald's + IBM drive-thru AI The pilot covered three years and over 100 locations, but viral social media posts documented frequent misorders adding unwanted items, mixing adjacent lane orders, ignoring corrections before McDonald's confirmed it was ending the IBM partnership in June 2024. Today: why most enterprise agent projects don't fail in production — they fail before production, because they were never the right tool for the job. From Demo to Production • Enterprise Agents on Databricks - Paul Karikari 2 / 22
7. WHO THIS IS FOR If you've shipped ML before, this is for you ML / AI engineers Data engineers Tech leads & architects Building or about to build agents Owning the platform agents will run on Picking the patterns the team adopts Familiar with LLMs. New to agents in production. From Demo to Production • Enterprise Agents on Databricks -- Paul Karikari 3 / 22
8. AGENDA The next 60 minutes 1 When does an agent earn its keep? 6 min A 5-question rubric to filter ideas 2 Seven things that break in production 12 min And how Databricks addresses each 3 Reference architecture 5 min How the pieces fit together 4 Hands-on: Mortgage Advisor 30 min Build, trace, evaluate — in your workspace From Demo to Production • Enterprise Agents on Databricks - Paul Karikari 4 / 22
9. PART 1 When does an agent earn its keep? Most enterprise "agent" ideas are workflows in disguise. From Demo to Production • Enterprise Agents on Databricks 5 / 22
10. FRAMING How does AI agents work ? Reasoning LLM OpenAI (o1, o3, & o4 families) Anthropic (Claude 3.7 Sonnet & Opus Series) Google (Gemini Pro/Flash Thinking) If the steps are fixed, you don't need an agent — you need a workflow. From Demo to Production • Enterprise Agents on Databricks - Paul Karikari 6 / 22
11. FRAMING Not everything that talks to an LLM is an agent If the steps are fixed, you don't need an agent — you need a workflow. From Demo to Production • Enterprise Agents on Databricks - Paul Karikari 6 / 22
12. FRAMING Not everything that talks to an LLM is an agent Prompt + LLM One call. Stateless. Summarization, classification, generation RAG Retrieve, then generate. Q&A over docs, knowledge grounding Workflow / chain Fixed steps, LLM in the middle. Most enterprise pipelines Agent Dynamic decisions over tools. Triage, research, multi-step ops If the steps are fixed, you don't need an agent — you need a workflow. From Demo to Production • Enterprise Agents on Databricks - Paul Karikari 6 / 22
13. FRAMING Not everything that talks to an LLM is an agent Prompt + LLM One call. Stateless. Summarization, classification, generation RAG Retrieve, then generate. Q&A over docs, knowledge grounding Workflow / chain Fixed steps, LLM in the middle. Most enterprise pipelines Agent Dynamic decisions over tools. Triage, research, multi-step ops If the steps are fixed, you don't need an agent — you need a workflow. From Demo to Production • Enterprise Agents on Databricks - Paul Karikari 6 / 22
14. RUBRIC Five questions before you build an agent 1 Dynamic decisions? Does the task require choosing which steps to take, based on intermediate results? 2 Multiple tools? Does it call several systems whose use depends on what came before? 3 Worth the cost? 5–50× the latency and price of a deterministic alternative — is the value there? 4 Can you evaluate it? If you can't write 20 eval examples, you can't ship an agent — you can only demo one. 5 Acceptable blast radius? When it's confidently wrong, what breaks? Lawsuits and lost customers need a human in the loop. Need YES on at least 1, 2, and 3. From Demo to Production • Enterprise Agents on Databricks - Paul Karikari 7 / 22
15. WHEN AN AGENT FITS Two-axis filter: complexity × tool breadth Tool / system breadth → Workflow AGENT Prompt + LLM RAG Only the top-right quadrant is genuinely agent territory. Everything else is a workflow, a RAG app, or a function call. Decision complexity → From Demo to Production • Enterprise Agents on Databricks - Paul Karikari 8 / 22
16. PART 2 Seven things that break in production The hard problems aren't the model — they're everything around it. From Demo to Production • Enterprise Agents on Databricks 9 / 22
17. THE SEVEN What gets you stuck on the road to production Orchestration Deciding what step comes next. Observability Seeing what really happened. Tool governance Who can call what, safely Deployment Shipping without breaking prod. Retrieval Getting the right context in. Evaluation Knowing if it's actually better. Cost & latency Cheap enough, fast enough. We'll walk each row — challenge → Databricks capability → what to verify before you trust it. From Demo to Production • Enterprise Agents on Databricks - Paul Karikari 10 / 22
18. MAPPING Challenge → Databricks capability Challenge Databricks capability Agent orchestration & tool calling Mosaic AI Agent Framework / Open ai framework Tool governance (who can call what) Unity Catalog functions as agent tools Retrieval over enterprise data Mosaic AI Vector Search Quality, safety, regression eval Mosaic AI Agent Evaluation Tracing & observability MLflow Tracing for GenAI Versioning & deployment MLflow Registry + Model Serving Cost & latency control Provisioned throughput, model routing Security & data isolation Unity Catalog + workspace controls Human review & feedback Review Apps / evaluation review UI From Demo to Production • Enterprise Agents on Databricks - Paul Karikari 11 / 22
19. DEEP DIVE • TOOL GOVERNANCE An agent is only as safe as its tools The problem • Tools are arbitrary code with credentials. • Agents call them dynamically — at runtime, not design time. • Permissions, audit, lineage usually missing. • Security review blocks the launch. From Demo to Production • Enterprise Agents on Databricks - Paul Karikari Unity Catalog functions as tools • Tools live as governed UC objects (SQL or Python). • GRANT EXECUTE controls who/what can call them. • Lineage and audit are inherited from the catalog. • Same permission model your data already uses. 12 / 22
20. DEEP DIVE • EVALUATION If you can't measure it, you can't ship it "If you can't write 20 eval examples, you can't ship an agent — you can only demo one." Curate Judge Iterate Build a small (20–50) golden set per task. Include known failure modes. Mix deterministic checks (tool was called, output schema valid) and LLM-as-judge for quality. Treat eval scores like unit-test results. Prompt or tool change → re-run → ship or revert. From Demo to Production • Enterprise Agents on Databricks - Paul Karikari 13 / 22
21. DEEP DIVE • OBSERVABILITY You debug what you can see What MLflow Tracing gives you trace: account_health_query agent.run llm.call → plan 2.4s 0.6s tool.search_tickets (vector) 0.3s tool.get_usage_metrics (uc.func) 0.2s tool.get_renewal_status (uc.func) 0.2s llm.call → synthesize 1.1s From Demo to Production • Enterprise Agents on Databricks - Paul Karikari • Every LLM call, tool call, retrieval step • Inputs, outputs, latency, cost per span • Linked to evaluation runs and deployments • Replayable for debugging and regressions 14 / 22
22. PART 3 Reference architecture You don't need a new stack. Agents slot into the one you have. From Demo to Production • Enterprise Agents on Databricks 15 / 22
23. ARCHITECTURE Agents slot into the platform you already have Client / app Model Serving OpenAI/ Mosaic AI Agent Framework Foundation Models Unity Catalog tools Vector Search Provisioned throughput, routing Governed SQL / Python functions Retrieval over enterprise data CROSS-CUTTING MLflow Tracing Agent Evaluation From Demo to Production • Enterprise Agents on Databricks- Paul Karikari MLflow Registry Unity Catalog (governance) 16 / 22
24. HANDS-ON LAB Build an AI Agent in 30 minutes From Demo to Production • Enterprise Agents on Databricks 17 / 22
25. THE USE CASE A Mortgage Advisor agent for client meetings Sarah Chen is coming in tomorrow about mortgage advice. What are her options and what should I recommend?" Search History Pull Finance Data Check rates Synthesize Vector Search over past advisor notes and client calls UC SQL function on account and credit data UC Python function calling CRM stub Recommend refinancing options. From Demo to Production • Enterprise Agents on Databricks - Paul Karikari 18 / 22
26. Thank you Paul Karikari, Principal Engineer & Solutions Architect-Databricks thedatalead.ai https://www.linkedin.com/in/paulkarikari/ Agents are software. Treat them like it.