Affiliation: Services Computing Research Dept. Research & Development Group, Hitachi, Ltd. Activity: ⚫ Engaged in R&D on blockchain and distributed ledger technologies ⬦ Including Self-sovereign Identity/Decentralized Identifier and Non-Fungible Token use cases ⚫ Since 2022, contributing to Open-source projects under the Linux Foundation ⬦ e.g., Hyperledger Fabric (consortium blockchain) ⚫ Engaged in R&D on generative AI Self-Introduction LinkedIn: @satoshi-ito
incident workflows and single-LLM limits 2. From a single agent to multi-agent (A2A) and its cost 3. Hybrid inference with on-prem and cloud LLMs 4. Platform design and practical lessons with LangChain + KServe
Real-world tasks follow multi-step workflows, with reasoning and decisions at each step ⚫ We must collect and correlate data across many systems and tools ⚫ Based on the analysis, we need to apply fixes, config changes, and recovery actions Single-LLM approach ⚫ Try to solve everything in one shot with a huge prompt ⬦ Example: send alerts, logs, traces, runbooks, and manuals to a single LLM and ask “What should I do?” Problems ⚫ Context and token limits are quickly saturated, so reasoning degrades ⚫ Latency and cost per incident become unstable and hard to predict ⚫ There are no explicit steps, so it is hard to review, debug, and trust the answer Complex Reasoning and Single LLM Limits
Response Workflow ⚫ Detection: notice the incident from chat or alerts ⚫ Triage: understand the situation and set severity and impact ⚫ Diagnosis: collect logs, alerts, and deploy info to find the root cause ⚫ Planning: create a remediation plan based on manuals and runbooks ⚫ Execution: apply config changes and recovery actions according to the plan ⚫ Communication: inform affected users and stakeholders about status and recovery ⚫ Post-incident review: validate fixes and improve monitoring and runbooks Multiple data sources ⚫ alerts, logs, traces, deploy diffs, runbooks / manuals, chat history, etc. Key point ⚫ Some of these steps can be handled by agents, but if one single LLM tries to own the whole workflow, the control logic and context size become very large Example: Incident Response as a Complex Workflow
single “super-agent” ⚫ One “super-agent” handling the whole workflow runs into the same issues as a single LLM: huge context and opaque reasoning Moving to multi-agent ⚫ Define specialized agents per task and step in the workflow ⚫ Use the Agent2Agent Protocol (A2A) to connect these agents with a consistent, typed message format ⚫ Each step has clear inputs and outputs, so intermediate reasoning becomes visible and reviewable Trade-offs: More agents mean more things to manage: configs, versions, monitoring, and testing overhead. From a Single Agent to Multi-Agent (A2A) Single Agent Multi-Agent(a2a) Please recovery system I clear DB Please recovery system Recovery system
Interaction is mostly between the user and one agent ⚫ Sometimes we retry the same agent if the answer is not good enough Multi-agent case ⚫ In addition to user agent, we now have many agent agent messages ⚫ These internal calls can also trigger retries, depending on the output ⚫ More agents usually mean more tokens and more chances to retry Cost of Multi-Agent Systems Input: 1,000 Output: 2,000 Input: 1,000 Output: 2,000 Input: 500 Output: 1,000 Input: 500 Output: 1,000 Input: 500 Output: 1,000 Input: 500 Output: 1,000 Single Agent Multi-Agent(a2a) Total Token: 3,000 Total cost: $ 3.75 (0.00125 $/Token) Total Token: 15,000 Total Cost: $ 18.75 (0.00125 $/Token) Output: 500, Input: 1,000 Output: 500, Input: 1,000 Output: 500, Input: 1,000 Output: 500, Input: 1,000 15000 (Numbers are illustrative; actual pricing depends on the provider and model.)
simpler or high-volume agent tasks on open-source models deployed on-prem, instead of always using cloud models ⚫ Keep complex, high-quality tasks on cloud LLMs On-prem LLMs ⚫ Tasks that touch sensitive data (e.g., detailed logs) ⚫ Log analysis and structured extraction that follow clear patterns ⚫ Recursive or frequently repeated steps inside loops ⚫ Latency-sensitive tasks where round-trip to the cloud is too slow Cloud LLMs ⚫ Tasks that need long context and rich generation, e.g., incident reports and user-facing announcements ⚫ Communication that needs better tone, style, or multiple languages ⚫ A few reasoning steps where we really want the best model quality Hybrid Inference: On-Prem + Cloud LLMs
a multi-agent (A2A) workflow for incidents ⚫ A routing layer chooses between on-prem LLMs (KServe + llm-d) and cloud LLMs ⚫ On-prem: open models on a Kubernetes GPU cluster for sensitive and high-volume tasks ⚫ Cloud: Azure OpenAI, Amazon Bedrock, OpenAI API, etc. for high-quality, long-form outputs ⚫ Agents call a single OpenAI-style API; backends can change without touching agent logic Platform Design with LangChain + KServe Cloud LLM Providers On-prem LLM (KServe + llm-d) Kubernetes GPU Cluster Azure Open AI OpenAI API Amazon Bedrock Coordinator Agent (LangChain) Gemini Claude User Manual Agent LangChain Log Analysis Agent LangChain A2A Protocol etc.
Single agent (cloud LLM only): 560 * 0.00125 = $ 0.7 ⚫ Multi-agent (cloud LLM only): 2800 * 0.00125 = $ 3.5 ⚫ Multi-agent (cloud + on-prem LLMs): 680 * 0.00125 (cloud) + $ 1.49 (on-prem) = $ 2.34 (Numbers are approximate and depend on model and provider.) Challenges with multi-agent & model selection ⚫ More agents = more model choices and routing rules to maintain. ⚫ Choosing “the right LLM per task” is non-trivial: ⬦ different models have different strengths and weaknesses ⬦ especially with open models, quality varies a lot by task Lessons from hybrid LLM infrastructure ⚫ Exposing on-prem LLMs via an OpenAI-compatible API made switching easy ⚫ We used KServe + llm-d to serve open models on a Kubernetes GPU cluster ⚫ Agents call one OpenAI-style endpoint; routing decides whether the call goes to cloud or on-prem Practical Challenges and Lessons
single LLM or single agent is not enough for complex, multi-step workflows ⚫ Multi-agent workflows add structure, but also increase LLM cost and system complexity Approach ⚫ Use Agent2Agent (A2A) to decompose the workflow into specialized agents ⚫ Use hybrid inference with on-prem (KServe + llm-d) and cloud LLMs to place each task on the right side Lessons ⚫ Choosing the right model per task is hard and must be guided by experiments and telemetry ⚫ Standard protocols (A2A) and OpenAI-compatible serving make agents and models reusable across frameworks and infrastructure Summary and Takeaways
and Kubernetes are registered trademarks or trademarks of The Linux Foundation. ⚫ GitHub is a registered trademark or trademark of GitHub Inc. ⚫ All other company names, product names, service names and other proper nouns are registered trademarks or trademarks of their respective companies. ⚫ The TM and 🄬 marks are not shown in the text and figures in this slide. Trademarks
sato4
.png){kind=avatar}
satai
yukinobaba
skoyamalab
akademiya2063
koide3
ks91
culturaltransition
flekschas
ozekinote
karakurist
polipoli
eumesy
keithpitt
sabderemane
philnash
skipperchong
kimpetersen
asonas
ryanjones
szymonslowik
tonyrice
mayatellez
garrettdimon
