Building AI Agent Observability with OpenTelemetry EN

Transcript

1. Building AI Agent Observability with OpenTelemetry Japan Community Day 2026 / OpenTelemetry Meetup Japan · 2026/07/28 Joe / Yuzuru Ohira @joe_yuzupi
2. About Me Joe (Yuzuru Ohira) LayerX Inc. / Ai Workforce Division Group Manager, Platform Enablement Observability Architect for the AI Agent Platform Favorite technology: OpenTelemetry Span Links © LayerX Inc. 2
3. © LayerX Inc. 3
4. Business Overview © LayerX Inc. 4
5. Product Architecture © LayerX Inc. 5
6. Agenda Agenda Why Observe AI Agents? About GenAI Semantic Conventions How to Use OTel for Evaluation and Improvement Today, We Will Focus on Traces © LayerX Inc. 6
7. Why Observe AI Agents? Why AI Agent Observability Matters 1. Dynamic Execution Paths The agent decides which tools and retrieval operations to call at runtime. The execution path cannot be determined from code alone. © LayerX Inc. 2. Non-Deterministic Behavior The same input can produce different paths and outcomes. A debugger cannot reliably reproduce them. 3. Failures Do Not Surface as Exceptions The operation succeeds, but the output quality fails. 7
8. Why Observe AI Agents? Example: An Answer Is Generated Without Sufficient Evidence User request Answer with supporting evidence 200 HTTP status © LayerX Inc. → Retrieval 0 results → 0 Unhandled errors LLM Generate answer → Answer Does not disclose insufficient evidence 1 Answer generated 8
9. What to Record with OTel Observe an End-to-End Execution with a Trace Time Trace: answer a question end-to-end 2.4 s ecarT Span: invoke agent agent operation 2.3 s Span: retrieval 0.6 s Span: chat model inference 1.4 s © LayerX Inc. 9
10. What to Record with OTel A Trace Records One Complete Sequence of Agent Operations Time Trace: answer a question end-to-end 2.4 s ecarT Span: invoke agent agent operation 2.3 s Span: retrieval 0.6 s Span: chat model inference 1.4 s Phoenix Representative tools that use traces for evaluation and improvement © LayerX Inc. 10
11. GenAI Semantic Conventions What Are GenAI Semantic Conventions A common OpenTelemetry specification for recording generative AI and Agent executions Model spans: spans for model calls Agent spans: spans for Agent and Tool executions Metrics: token usage and latency Events: model inputs and outputs Exceptions: exceptions raised during execution © LayerX Inc. open-telemetry/semantic-conventions-genai 11
12. Spans in GenAI Semantic Conventions Specification Example Attribute gen_ai.operation.name gen_ai.provider.name gen_ai.request.model gen_ai.usage.input_tokens gen_ai.input.messages © LayerX Inc. Requirement Description Level Required Type of operation performed GenAI provider identified by the Required instrumentation Conditionally Model name specified in the request Required Recommended Number of input tokens used Opt-In Chat history passed to the model Not collected by default Example Values chat generate_content openai gcp.vertex_ai gpt-4 100 [{"role":"user", ...}] GenAI spans specification 12
13. Current State of GenAI Semantic Conventions Breaking Changes Are Possible While the Specification Is in Development Message Event Deprecation System-specific naming policy Deprecated Replacement Before After gen_ai.system_instructions or gen_ai.input.messages gen_ai.system gen_ai.provider.name gen_ai.openai.* openai.* gen_ai.input.messages az.ai.* azure.ai.* gen_ai.system.message gen_ai.user.message gen_ai.assistant.message gen_ai.tool.message gen_ai.choice © LayerX Inc. gen_ai.output.messages v1.37 release notes · #2046 13
14. Evolution of GenAI Semantic Conventions Continuous Changes and Release Transitions v1.37 2024-04 v1.37 v1.38-39 v1.42 SIG formed Breaking Change MCP SemConv Evaluation event Moved to a dedicated repo Chat history revamp: message events aggregated into gen_ai.input.messages / gen_ai.output.messages v1.38 v1.39 v1.40 v1.41 v1.42 © LayerX Inc. Evaluation event, tool definitions and call details, span kind guidance MCP semantic conventions added Retrieval span and cache token attributes span names require the tool name, reasoning tokens, , streaming metrics moved to semantic-conventions-genai (2026-06) invoke_agent execute_tool invoke_workflow gen_ai.* 14
15. Current State of GenAI Semantic Conventions Telemetry Varies by Framework and Provider Implementation LangChain LangGraph Claude Agent SDK Google ADK (Python) Strands Agents SDK (Python) OpenAI Agents SDK © LayerX Inc. SemConv Support Partial support langsmith.* also used Partial support mainly claude_code.* Supported experimental opt-in mixed stable telemetry Supported latest conventions opt-in No documented SemConv support proprietary tracing Representative Recorded Data langsmith.span.kind gen_ai.system (legacy name) gen_ai.system (legacy name) input_tokens / agent_id gen_ai.agent.name gen_ai.usage.* gen_ai.agent.name gen_ai.operation.name gen_ai.usage.* agent / generation / function model / usage / tools OTel Export OTLP Trace Traces (Beta) Metrics / Logs Trace / Metrics / Logs OTLP Trace / Metrics OpenAI Traces API custom processor LangChain OTel · Claude Agent SDK · Google ADK · Strands Agents SDK · OpenAI Agents SDK 15
16. GenAI Implementation Support Use the OTel Collector to Normalize Implementation Differences Data Sources Backends OpenTelemetry Collector Grafana® collector.yaml ADK OpenAI Agents SDK © LayerX Inc. → → Splunk® Observability Cloud 16
17. Observability Platform Support GenAI Support Across Observability Platforms Platform Datadog Langfuse Arize Phoenix New Relic Grafana Cloud Splunk © LayerX Inc. OTLP Ingestion Supported Supported over HTTP Supported (15.10.0+) Supported Supported Supported Handling of GenAI Attributes Automatically maps SemConv v1.37+ to Agent Observability Converts OTel Spans to the Langfuse format, prioritizing Automatically converts to OpenInference Preserves OTel attributes for querying with NRQL Ingests OTLP; GenAI dashboards are configured through OpenLIT Identifies Agent Spans using langfuse.* gen_ai.* gen_ai.operation.name Datadog · Langfuse · Phoenix · New Relic · Grafana Cloud · Splunk 17
18. Instrumenting with GenAI Semantic Conventions Example: A Trace Instrumented with GenAI Semantic Conventions Time invoke_agent research-agent plan research-agent chat Claude Sonnet 5 ecarT search_memory user-memory retrieval knowledge-base execute_tool web-search chat Claude Sonnet 5 © LayerX Inc. Invoke Agent span · Memory span · Retrieval span · Inference span 18
19. Instrumenting with GenAI Semantic Conventions gen_ai.operation.name Identifies the Role of Each Span Time invoke_agent invoke_agent plan plan chat chat ecarT search_memory search_memory retrieval retrieval execute_tool execute_tool One complete agent execution Planning and task decomposition before execution LLM generation of responses and tool calls Search the agent's memory store Retrieve external knowledge or context Execute the tool selected by the agent chat * When multiple Agents are orchestrated together, the overall execution can be recorded as (for example, ). invoke_workflow gen_ai.workflow.name: multi_agent_rag © LayerX Inc. Agent spans · GenAI spans · Memory span 19
20. Tracing the Execution Start by Reading It Like a Conventional Trace IDENTIFY FAILURES Status: Error LLM API rate limits and timeouts Tool execution failures Connection failures to the Retrieval backend IDENTIFY SLOW EXECUTIONS duration Total duration of Identify slow Spans and operations Drill down into child Spans, such as expensive DB calls invoke_agent W3C Trace Context links GenAI Spans to existing distributed traces © LayerX Inc. https://opentelemetry.io/docs/specs/semconv/general/recording-errors/ 20
21. Tracing the Execution Even Successful Runs Can Contain Wasteful Loops STATUS: OK AND THE ANSWER IS CORRECT HOW TO IDENTIFY THEM Repeated calls to the same tool (the number and order of child spans) duration is an outlier Cumulative / are unusually high Time invoke_agent 42 s chat invoke_agent web-search ecarT gen_ai.usage.input_tokens chat output_tokens web-search chat web-search… chat © LayerX Inc. 21
22. Tracing the Execution This Is the Limit of Conventional Trace Analysis Conventional Quality Retrieval Status: unset trace → → assessment 0 results / low → operation analysis relevance completed Cannot detect quality failures © LayerX Inc. Evaluation https://opentelemetry.io/docs/specs/semconv/general/recording-errors/ 22
23. Core Use Case — Linking Evaluation Link Evaluation Results to the Execution Being Evaluated An Evaluation result alone does not reveal which execution or path produced a poor score. Evaluator Human evaluation / LLM-as-a-Judge → Evaluation Event gen_ai.evaluation.result What OTel provides Correlation between the result and the target operation © LayerX Inc. → GenAI operation Target Span What the Evaluator provides The assessment itself, such as groundedness GenAI evaluation result event specification 23
24. Lessons from Coding Agents Coding Agents Make Execution Paths Easier to Explore Excerpt from the Trace generated while proofreading this deck Claude Code Trace claude_code.interaction claude_code.llm_request claude_code.tool · Bash claude_code.tool.execution claude_code.tool · preview_start claude_code.tool.blocked_on_user claude_code.llm_request claude_code.tool · get_page_text claude_code.tool.execution claude_code.llm_request © LayerX Inc. Time → 162.4 s 8.2 s 0.29 s 0.29 s 2.28 s 2.15 s 5.0 s 4 ms 3 ms 4.3 s 24
25. Lessons from Coding Agents Recording the Trace Reveals the Causal Chain Behind a Failure parent agent Implement CSV export handoff: pagination Delegate to a Subagent reference: old helper offset method (position specified by item count) subagent: implementation Implemented using the offset method ↓ suspect Cause The Agent referred to an outdated implementation approach © LayerX Inc. https://www.langchain.com/blog/your-coding-agents-are-a-black-box-heres-how-to-crack-them-open 25
26. Lessons from Coding Agents Accumulated Traces Let Other Agents Improve Execution 1. Run the Agent → 2. Accumulate Trace / usage ↑ Improvement Loop ↓ Modify Prompt / Tool / Workflow ← Process the task and invoke Tool / Handoff 4. Improve execution latency / token usage / error 3. Analyze with an Agent Baseline deviation Token-cost spike Tool / Handoff bias Compare each new run using the same metrics. © LayerX Inc. 26
27. Lessons from Coding Agents Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses Model remains fixed Only its surroundings change → Edit the Harness Manage Prompt / Tool / Middleware as files → Execute and observe Prepare execution data for Agent analysis → Validate or revert Keep effective edits or revert them Reference: Terminal-Bench 2 pass@1 improved from 69.7% to 77.0%, surpassing the human-designed harness (71.9%). © LayerX Inc. https://arxiv.org/abs/2604.25850 27
28. Lessons from Coding Agents The Key Is Three Observability Dimensions Component Represent the Harness as a collection of files Each error type maps to a file to fix. Track edits in Git history and revert at any time. © LayerX Inc. Experience Turn execution trajectories into analysis data an Agent can read Execution trajectories (millions of tokens) ↓ Compress Failure analysis by task ↓ Aggregate Benchmark-wide overview The Agent starts from the overview and drills down only where needed. Decision Attach predictions to edits and validate them empirically Edit + prediction “Tasks 3 and 7 should improve” ↓ Measure in the next round ✓ Correct → adopt ✗ Incorrect → revert Determine impact by comparing against measured results, not self-reported claims. 28
29. Lessons from Coding Agents The Impact of an Improvement Can Only Be Determined Empirically Improvements do happen SWE-bench Verified achieved a 75.6% success rate with 32% fewer tokens Switching to another model still yields a 5–10 pp improvement An Agent’s self-assessment is unreliable Predictions of where an edit will cause regressions have only 11.8% precision—nearly blind “It should have worked” is not enough Giving the Agent the ability to observe its own impact is the foundation of the improvement loop © LayerX Inc. 29
30. Lessons from Coding Agents We Are Starting to Use OTel to Evaluate and Improve Coding Agents The workflow development and improvement cycle used by FDE. We are building observability for evaluation © LayerX Inc. and improvement using OTel. https://speakerdeck.com/cipepser/layerx-fde-practices?slide=15 30
31. Content Capture and Confidentiality Caution: The More You Observe, the More Sensitive Data Enters the Trace Do not record full Prompt / Output content, Source code / Shell output, full Tool inputs and outputs, or environment variables by default (default-off) Only when recording is necessary, design allowlists, redaction, retention, access control, and references to external artifacts © LayerX Inc. 31
32. A Minimal Starting Point Start with Traces, Then Expand to Metrics and Logs Use Traces to capture the execution paths of Agents, Workflows, LLMs, and Tools Use Metrics to continuously monitor trends in latency, token usage, and errors Use Logs to correlate detailed individual events with their Traces © LayerX Inc. 32
33. Summary Summary AI Agents follow dynamic paths; final results and Status alone cannot reveal failures, waste, or quality issues Record Agent, LLM, Tool, and Retrieval Spans with GenAI Semantic Conventions, then connect them to distributed Traces through W3C Trace Context Correlate Status, duration, usage, and Evaluation to find failure origins and wasteful loops Standardize observability data as a shared asset for incident response, quality evaluation, improvement, and analysis by other Agents © LayerX Inc. 33
34. Topics Not Covered Today Topics Not Covered Today Instrumentation code for each SDK / Agent Framework; OpenTelemetry Collector / Exporter / Backend architecture; Sampling, Batching, Retention, and Redaction Evaluation implementation and dataset design; LLM judge selection and assessment; Dashboard / Query / Alert design © LayerX Inc. 34
35. Appendix Appendix — Sources OpenTelemetry GenAI Semantic Conventions — commit c26a2c2 OpenTelemetry Semantic Conventions — Recording errors LangChain — How to Debug Coding Agents with LangSmith Traces LangSmith — Coding agent metadata contract Agentic Harness Engineering — arXiv v4 OpenLLMetry — Issue #3515 (migration from removed attributes pending) OpenInference — Semantic conventions spec OpenLIT — semcov (sdk/python) Datadog — LLM Observability and OTel semantic conventions Langfuse — OpenTelemetry integration Google ADK — Observability / Traces Vercel AI SDK — Telemetry Pydantic Logfire — Issue #1586 (acknowledging mixed legacy attributes) © LayerX Inc. 35
36. Appendix — Metrics Metrics : token usage gen_ai.client.operation.duration : duration of a client operation gen_ai.client.operation.time_to_first_chunk / time_per_output_chunk : streaming response characteristics gen_ai.invoke_agent.duration / inference_calls / tool_calls : Agent execution duration and call counts gen_ai.execute_tool.duration : Tool execution duration gen_ai.server.request.duration / time_to_first_token / time_per_output_token : 36 gen-ai-metrics.md © LayerX Inc. inference server metrics gen_ai.client.token.usage
37. Appendix — Events Events gen_ai.client.inference.operation.details parameters : records inference inputs, outputs, and : records evaluation results Main attributes: gen_ai.input.messages / gen_ai.output.messages / gen_ai.system_instructions / gen_ai.tool.definitions Message content is Opt-In; it may contain PII, so instrumentations do not record it by default gen_ai.evaluation.result © LayerX Inc. gen-ai-events.md 37
38. Appendix — Exceptions Exceptions gen_ai.client.operation.exception : records exceptions raised during client operations Attributes: exception.type / exception.message / exception.stacktrace Recorded at WARN severity (severity number 13) © LayerX Inc. gen-ai-exceptions.md 38