Building Multi-Agent AI Systems on Databricks

Transcript

1. Meetup · PyData Bradford Building Multi-Agent AI Systems on Databricks Patterns & Pitfalls Paul Karikari - Principal Engineer & Solutions Architect - Data & AI TheDataLead AI 1
2. What we'll cover 60 minutes · half talk, half lab 01 What changes with multiple agents 02 The 6 patterns worth knowing Why teams of agents, when one would do, and when not. Supervisor, parallel specialists, critic, tool worker. 03 04 Doing it on Databricks Pitfalls & the lab Agent Framework, MLflow, Vector Search, Unity Catalog. What breaks in practice — then build one yourself. Building Multi-Agent Systems on Databricks - Paul Karikari 2
3. A SHORT STORY AI agents are projected to drive up to $4.4 trillion in annual global business value and automate 15% to 50% of business tasks by 2027 Building Multi-Agent Systems on Databricks - Paul Karikari 3
4. A SHORT STORY 95% of enterprise AI pilots fail to deliver measurable ROI. Building Multi-Agent Systems on Databricks - Paul Karikari 4
5. A SHORT STORY Gartner reported a 1,445% surge in multi-agent system inquiries between Q1 2024 and Q2 2025. Organizations already use an average of 12 agents, and that number is projected to climb 67% within two years. But 40% of multi-agent pilots fail within six months of production deployment. The pattern isn't that multi-agent systems don't work. It's that teams pick the wrong orchestration pattern for their problem, or they pick the right one without understanding how it breaks. Building Multi-Agent Systems on Databricks - Paul Karikari 5
6. How does a single agent work? Before multi agents, lets understand a single agent. Building Multi-Agent Systems on Databricks - Paul Karikari 6
7. How does an AI agent work ? Reasoning LLM OpenAI (o1, o3, & o4 families) Anthropic (Claude 3.7 Sonnet & Opus Series) Google (Gemini Pro/Flash Thinking) If the steps are fixed, you don't need an agent — you need a workflow. Building Multi-Agent Systems on Databricks - Paul Karikari 7
8. What is a Multi-Agent System? Building Multi-Agent Systems on Databricks - Paul Karikari 8
9. Why multi-agent at all? Most systems should start as one agent. Use more only when one stops scaling. One agent Many agents • Single prompt, tool set • Specialized roles & tool scopes • Easy to reason about & debug • Parallelism on independent tasks • Might Work for most use cases • Better at long, branching workflows • Breaks down when scope grows • More moving parts to govern & evaluate Rule of thumb: don't reach for multi-agent until your single agent is probably the bottleneck. Building Multi-Agent Systems on Databricks - Paul Karikari 9
10. 6 patterns worth knowing Most production systems are one of these or a combination. Supervisor Sequential Pipeline Fan-out/Fan in Router agent decides which specialist handles each turn. Agents process tasks in a linear, step-by-step chain where each output feeds the next. Independent agents work concurrently; results are merged. Critic / reviewer(Refiner) Dynamic Handoff Adaptive Planning One agent generates, another critiques and revises. Agents autonomously transfer execution control directly to another specialist mid-workflow. A planner agent continuously revises and re-shapes its execution steps based on tool feedback. Building Multi-Agent Systems on Databricks - Paul Karikari 10
11. Pattern 1 — Supervisor A planner routes each request to the right specialist; specialists don't talk to each other. Use when Supervisor • Tasks split cleanly by domain • You want one entry point • Routing logic is non-trivial Watch out for Retriever Analyst Writer Supervisor decides → calls one specialist → returns result Building Multi-Agent Systems on Databricks - Paul Karikari • Supervisor becomes the bottleneck • Routing prompt grows unboundedly • Hidden costs from extra hops 11
12. Pattern 2 — Sequential Pipeline Fixed linear chain where each agent process the previous output Use when • Procedural safety is mandatory: The business logic requires strict, irreversible step-by-step executions. • Incremental data modification: Each layer adds specific value (e.g., translation, compliance checking, formatting). • Predictability is favored: System operations must remain easy to follow, trace, and troubleshoot. Watch out for • Upstream corruption cascading: A broken payload at Step 1 completely compromises all downstream workers. • Compounding latencies: Because operations are completely blocked until the step before finishes, total runtime can spike. • Inflexible structural design: Sudden deviations or fallback requirements ruin the linear progression layout. Building Multi-Agent Systems on Databricks - Paul Karikari 12
13. Pattern 3 — Fan-Out / Fan-in Parallel agents execute simultaneously, with a single aggregation point Use when • Multi-perspective analysis (financial analysis with fundamental, technical, sentiment, and ESG agents running in parallel). • Concurrent code review across security, style, and performance. • Any scenario with four or more independent tasks Watch out for Building Multi-Agent Systems on Databricks - Paul Karikari • API rate limits. • Race conditions on shared state scale quadratically • The aggregation step itself introduces error. LLM-based synthesis can hallucinate. 13
14. Pattern 4 — Critic-Reviewer (Refiner) An internal correction loop continually checks generated outputs against an evaluation agent until quality criteria are hit. Use when • Rigid output rules apply: Perfect adherence to programming structures or complex legal rules is necessary. • Unsupervised accuracy counts: The engine must self-correct bugs or logical flaws before delivering to production. • Handling high-stakes math/logic: Validating step-by-step thinking eliminates hidden hallucinations. Watch out for • • • Building Multi-Agent Systems on Databricks - Paul Karikari Infinite execution loops: Agents can easily bounce corrections back and forth endlessly without a strict breakout point. Diminishing quality returns: Repeated corrections often consume excessive compute tokens for minor polish improvements. Drastically increased latency: Multi-turn internal evaluations make this pattern unviable for real-time customer chats. 14
15. Pattern 5 — Dynamic Handoff Agents organically transition context and task control to a peer based on runtime context Use when changes, bypassing a central supervisor. • Workflows are highly fluid: Complex, • • open-ended tasks make it impossible to chart execution flows early. Decentralized system designs: Microservice-like layouts function independently without a master supervisor. Cross-organizational operations: Different applications handle specific portions of work across unique platform ecosystems. Watch out for • • • Building Multi-Agent Systems on Databricks - Paul Karikari Total loss of tracking trace: Missing central control makes recording system states difficult to debug. Agent loop traps: Two models can pass control back and forth indefinitely if task boundaries overlap. State sync degradation: Passing context variables peer-to-peer without central verification risks losing vital system history. 15
16. Pattern 6 — Adaptive Planning Manager discovers the plan through specialist consultation and iteration. Use when • Open-ended problems with no predetermined solution path. • Complex migrations where scope changes during execution. Watch out for Building Multi-Agent Systems on Databricks - Paul Karikari • Slow to converge. • The pattern optimizes for correctness over speed. • Time-sensitive tasks are poor fits. 16
17. Pattern Selection Select the simplest pattern that fits your problem structure. Most teams over-architect , don't pick a complex pattern when a simple one will do. Problem Requirement Recommended Pattern Primary Mechanism Fixed linear steps Sequential Pipeline Order never changes; each step depends directly on the previous output. Known task decomposition Orchestrator-Worker (Supervisor) You know subtasks at design time; a supervisor acts as the single accountability point. Independent parallel work Fan-Out / Fan-In Subtasks have no dependencies; tasks execute concurrently and merge results. Quality verification needed Multi-Agent Debate (Critic) Maker-checker loops where output accuracy matters more than latency. Unpredictable routing Dynamic Handoff Specialist selection cannot be known until the conversation unfolds. Open-ended problem Adaptive Planning The execution plan must be dynamically discovered and reshaped via tool feedback. Building Multi-Agent Systems on Databricks - Paul Karikari 17
18. Organizational Structure to Agent Pattern Mapping Organizational design provides a natural blueprint for selecting multi-agent patterns. This connection is driven by Conway’s Law, which states that systems naturally reflect the communication structures of the organizations that design them. When building multi-agent systems, treating agents as digital team members ensures that agent topologies align directly with operational boundaries, domain ownership, and governance models. Building Multi-Agent Systems on Databricks - Paul Karikari 18
19. Pattern Selection Match your multi-agent architecture to your organization's human workflow. If a human team couldn't efficiently operate under a specific delegation model, your AI agents won't either. Organizational Structure Corresponding Agent Pattern Why It Fits Centralized Command / PMO Sequential Pipeline Mirrors a lead manager delegating subtasks to specialists while maintaining full oversight, auditability, and single-point accountability. Stage-Gated Operations Orchestrator-Worker (Supervisor) Mirrors cross-departmental handoffs (e.g., Sales -> Legal ->Finance), where each step depends directly on the output of the prior team. Front-Desk Triage / Support Hub Fan-Out / Fan-In Mirrors a receptionist or triage desk that inspects incoming requests and routes them directly to a dedicated subject matter expert. Compliance & QA Multi-Agent Debate (Critic/Reviewer) Mirrors strict separation of duties (e.g., Developer + Code Reviewer or Writer + Legal Compliance), prioritizing accuracy and verification over speed.. Building Multi-Agent Systems on Databricks - Paul Karikari 19
20. Multi-Agent System Implementation Workflow A 5-Phase Roadmap: From Single-Agent Baseline to Production 1 2 3 4 5 Single-Agent First Objective Action Establish core capabilities before adding system complexity. Validate the Baseline Objective Pattern Selection Determine routing, delegation, or orchestration. Objective Scope & Governance Enforce boundaries, tool access, and authorization limits. Objective Assembly & Control Connect agents, memory systems, and middleware pipelines. Objective Observability & Eval Building Multi-Agent Systems on Databricks - Paul Karikari Measure reliability, latency, cost, and task accuracy. Action Pick Simplest Architecture Action Isolate Roles & Permissions Action Wire System Layers Action Trace & Evaluate 20
21. The Databricks building blocks Four primitives cover most multi-agent needs out of the box. Mosaic AI Agent Framework MLflow Vector Search Unity Catalog Define agents, tools, and orchestration as code. Trace runs, log prompts, evaluate quality. Retrieval grounded in your governed data. Permissions on data, models, and tools. Building Multi-Agent Systems on Databricks - Paul Karikari 21
22. Mosaic AI Agent Framework At the center of Databricks’ AI story sits Mosaic AI, the umbrella brand for all of its machine learning and generative AI tooling. Mosaic AI is not a single product but a layered stack that includes access to foundation models, vector search, model serving, agent frameworks, evaluation tooling, and LLMOps infrastructure. Everything inside Mosaic AI is wired into Unity Catalog for governance, which is what makes the platform feel genuinely enterprise-grade rather than just a collection of good tools. Building Multi-Agent Systems on Databricks - Paul Karikari 22
23. How they fit together. Client / app Model Serving Mosaic AI Agent Framework Foundation Models Unity Catalog tools Vector Search Provisioned throughput, routing Governed SQL / Python functions Retrieval over enterprise data CROSS-CUTTING MLflow Tracing Agent Evaluation Building Multi-Agent Systems on Databricks - Paul Karikari MLflow Registry Unity Catalog (governance) 23
24. Agent implementation frameworks Building Multi-Agent Systems on Databricks - Paul Karikari 24
25. Reference architecture How a supervisor-style multi-agent system maps onto Databricks. ORCHESTRATION Supervisor agent Specialist agents Tool router CAPABILITIES LLM endpoints MLflow tracing & eval Vector Search retrieval Unity Catalog tables Functions as tools Volumes / files GOVERNED DATA & TOOLS Each layer is independently observable, governed, and swappable. Building Multi-Agent Systems on Databricks - Paul Karikari 25
26. Pitfalls you'll hit Every team I've seen ship multi-agent systems has tripped on at least three of these. Premature multi-agent Context bleed Splitting before a single agent is proven to be the limit. Agents inheriting each other's reasoning, multiplying errors. Untraceable failures Eval as an afterthought No per-agent traces means you can't tell which one failed. Shipping without a way to score the system end-to-end. Tool sprawl Governance debt Twenty tools, three of them used; the rest just confuse the planner. Tools hitting data the agent shouldn't be able to reach. Building Multi-Agent Systems on Databricks - Paul Karikari 26
27. LAB Build a research agent & mortgage advisor in prod 1 2 3 Set up Wire agents Trace & eval Import the notebook, attach a cluster, configure a model serving endpoint. Define tools, register a retriever, and bolt on a supervisor. Run sample queries; inspect the trace tree; score with MLflow. Notebook + sample data: link in chat 27
28. Before you ship: a quick checklist If you can't tick all six, your system probably isn't ready for users yet. One agent has been ruled out Each agent has a narrow role You proved the simpler design hits a real wall. Single responsibility, scoped tools, clear input/output. Traces are inspectable End-to-end evals exist MLflow shows what each agent saw, said, and called. An offline eval set scores the whole system, not parts. Tools are governed Failure modes are documented Unity Catalog permissions match the agent's intended scope. You know what breaks first, and what the user sees when it does. Building Multi-Agent Systems on Databricks - Paul Karikari 28
29. Thanks — go build something. Three things to take away: • Start with one agent. Add more only when you must. • Pick a pattern that fits the problem, not the hype. • Trace and evaluate from day one — not after launch. 29
30. Let’s connect Paul Karikari, Principal Engineer & Solutions Architect, Data & AI -Databricks thedatalead.ai https://www.linkedin.com/in/paulkarikari/ 30